From zunzun at zunzun.com Fri Oct 1 06:01:03 2010 From: zunzun at zunzun.com (James Phillips) Date: Fri, 1 Oct 2010 05:01:03 -0500 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Thu, Sep 30, 2010 at 7:20 PM, wrote: > > ppf is (very) expensive to calculate for some distributions. and > different starting values end up with different maximum likellihood > fit results. As we here in Alabama sometimes say, "Well, dang it!" So there is no (even slow) general method to use at the moment. For the time being, it seems the zunzun.com web site will have to cut back on the number of distributions. James From zunzun at zunzun.com Fri Oct 1 11:59:51 2010 From: zunzun at zunzun.com (James Phillips) Date: Fri, 1 Oct 2010 05:59:51 -1000 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Thu, Sep 30, 2010 at 2:20 PM, wrote: > ppf is (very) expensive to calculate for some distributions. > But I think using a global optimizer will be quite a bit more robust > for several distributions where the likelihood doesn't have a well > behaved shape. Version Two of Robert Kern's edited code and my example file are attached, this runs much faster. It seems to work considerably better that the first versions I posted. I have folded optimize.fmin() into Robert's code, among several other changes. The example works for powerlaw, beta, gamma, and pareto, although some of the distros are very parameter sensitive as you might see at the end of the example. This code would work much faster if it did not always solve for loc and scale, in some distributions these are known from the data and should not be solved for. Please give this test code a try. James -------------- next part -------------- A non-text attachment was scrubbed... Name: example.py Type: text/x-python Size: 2487 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: diffev.py Type: text/x-python Size: 14580 bytes Desc: not available URL: From zunzun at zunzun.com Fri Oct 1 12:11:05 2010 From: zunzun at zunzun.com (James Phillips) Date: Fri, 1 Oct 2010 06:11:05 -1000 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Thu, Sep 30, 2010 at 2:20 PM, ? wrote: > ppf is (very) expensive to calculate for some distributions. I forgot to mention that the code uses the log likelihood nnlf() method now. Thank you for letting me know about the computational expense. James From josef.pktd at gmail.com Fri Oct 1 12:55:57 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 1 Oct 2010 12:55:57 -0400 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Fri, Oct 1, 2010 at 12:11 PM, James Phillips wrote: > On Thu, Sep 30, 2010 at 2:20 PM, ? wrote: >> ppf is (very) expensive to calculate for some distributions. > > I forgot to mention that the code uses the log likelihood nnlf() > method now. ?Thank you for letting me know about the computational > expense. Which version of scipy are you using for testing? I had to fix some python 2.5 incompatibilities (*args), but when I ran the script it seemed to get stuck after printing 4 or 5 iterations. I let it run for a few minutes but then killed the process. Josef > > ? ? James > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From zunzun at zunzun.com Fri Oct 1 13:21:56 2010 From: zunzun at zunzun.com (James Phillips) Date: Fri, 1 Oct 2010 07:21:56 -1000 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Fri, Oct 1, 2010 at 6:55 AM, wrote: > Which version of scipy are you using for testing? Scipy 0.7.0 on Ubuntu Lucid Lynx with Python 2.6.5. > I had to fix some python 2.5 incompatibilities (*args), but when I ran > the script it seemed to get stuck after printing 4 or 5 iterations. I > let it run for a few minutes but then killed the process. At the top of diffev.py's solve() method is a loop that runs per generation of the genetic algorithm, you might consider placing a "print gen" statement there to see if it is running or stuck somehow. The virtual server I'm using has 4 CPUs so I plan to run the distribution fitting in parallel. Here are current timing results fitting both loc and scale, in parallel these would have taken 207 seconds (gamma time): distribution: powerlaw took 69.5095369816 seconds to run de parameters [ 1.31332767 0.55933617 2.45766383] nnlf: 23.3610045864 distribution: beta took 47.2591450214 seconds to run de parameters [ 0.70567183 0.6011756 0.607 2.41 ] nnlf: 7.45846316363 distribution: gamma took 206.991358995 seconds to run de parameters [ 1.40727512e+02 -5.83784812e+00 5.55091421e-02] nnlf: 26.7597491247 distribution: pareto took 86.9141609669 seconds to run de parameters [ 7.75987131e+13 -1.05369726e+14 1.05369726e+14] nnlf: 35.2599606822 James From zunzun at zunzun.com Fri Oct 1 13:58:05 2010 From: zunzun at zunzun.com (James Phillips) Date: Fri, 1 Oct 2010 07:58:05 -1000 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Fri, Oct 1, 2010 at 7:21 AM, James Phillips wrote: > > Here are current timing results fitting both loc and scale... I'm iterating over all continuous distributions now, and the genetic algorithm results are showing which distributions can be run with loc = min(data) and scale = max(data) - min(data). With that information in hand I can then speed up the overall fitting considerably by not fitting those parameters. James From josef.pktd at gmail.com Fri Oct 1 14:07:20 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 1 Oct 2010 14:07:20 -0400 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Fri, Oct 1, 2010 at 1:21 PM, James Phillips wrote: > On Fri, Oct 1, 2010 at 6:55 AM, ? wrote: >> Which version of scipy are you using for testing? > > Scipy 0.7.0 on Ubuntu Lucid Lynx with Python 2.6.5. > > >> I had to fix some python 2.5 incompatibilities (*args), but when I ran >> the script it seemed to get stuck after printing 4 or 5 iterations. I >> let it run for a few minutes but then killed the process. > > At the top of diffev.py's solve() method is a loop that runs per > generation of the genetic algorithm, you might consider placing a > "print gen" statement there to see if it is running or stuck somehow. > > The virtual server I'm using has 4 CPUs so I plan to run the > distribution fitting in parallel. ?Here are current timing results > fitting both loc and scale, in parallel these would have taken 207 > seconds (gamma time): much better than my old notebook > > distribution: powerlaw took 69.5095369816 seconds to run > de parameters [ 1.31332767 ?0.55933617 ?2.45766383] > nnlf: 23.3610045864 > > distribution: beta took 47.2591450214 seconds to run > de parameters [ 0.70567183 ?0.6011756 ? 0.607 ? ? ? 2.41 ? ? ?] > nnlf: 7.45846316363 > > distribution: gamma took 206.991358995 seconds to run > de parameters [ ?1.40727512e+02 ?-5.83784812e+00 ? 5.55091421e-02] > nnlf: 26.7597491247 > > distribution: pareto took 86.9141609669 seconds to run > de parameters [ ?7.75987131e+13 ?-1.05369726e+14 ? 1.05369726e+14] > nnlf: 35.2599606822 I did manage to run gamma, the parameters change a lot from generation to generation, it could also be the case that with your small sample some parameters are not well identified. However, with my scipy version, I'm getting some nans for which I don't see any reason at all. >>> scipy.stats.gamma.pdf(np.linspace(0,5,11), 300.5609591140931425, loc=0, scale=0.5) array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]) >>> scipy.stats.gamma.pdf(np.linspace(0,5,11), 300.5609591140931425, loc=0, scale=0.25) array([ 0., 0., 0., 0., 0., 0., NaN, NaN, NaN, NaN, NaN]) Given that you are doing a randomized search, there might not be enough restrictions that the parameters make sense. I did some fuzz testing when I got started with distributions but haven't run them in a long time. Your random parameters might hit some range where the numerical accuracy and correct results have never been checked. It's possible that fmin could get stuck with some nans in the results. I don't know how many dark corners are left. Josef > > > ? ? James > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From zunzun at zunzun.com Fri Oct 1 14:18:52 2010 From: zunzun at zunzun.com (James Phillips) Date: Fri, 1 Oct 2010 08:18:52 -1000 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Fri, Oct 1, 2010 at 8:07 AM, wrote: > On Fri, Oct 1, 2010 at 1:21 PM, James Phillips wrote: >> >> distribution: gamma took 206.991358995 seconds to run >> de parameters [ ?1.40727512e+02 ?-5.83784812e+00 ? 5.55091421e-02] >> nnlf: 26.7597491247 > >>>> scipy.stats.gamma.pdf(np.linspace(0,5,11), 300.5609591140931425, loc=0, scale=0.5) > array([ 0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0.]) >>>> scipy.stats.gamma.pdf(np.linspace(0,5,11), 300.5609591140931425, loc=0, scale=0.25) > array([ ?0., ? 0., ? 0., ? 0., ? 0., ? 0., ?NaN, ?NaN, ?NaN, ?NaN, ?NaN]) > > Given that you are doing a randomized search, there might not be > enough restrictions that the parameters make sense. I did some fuzz > testing when I got started with distributions but haven't run them in > a long time. Your random parameters might hit some range where the > numerical accuracy and correct results have never been checked. > It's possible that fmin could get stuck with some nans in the results. > I don't know how many dark corners are left. You have loc=0 where the GA fit the example data set to -5.8 or so, and I see from your results that the NaNs are dependent on your scale value of 0.5 or 0.25. James From josef.pktd at gmail.com Fri Oct 1 14:24:20 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 1 Oct 2010 14:24:20 -0400 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Fri, Oct 1, 2010 at 1:58 PM, James Phillips wrote: > On Fri, Oct 1, 2010 at 7:21 AM, James Phillips wrote: >> >> Here are current timing results fitting both loc and scale... > > I'm iterating over all continuous distributions now, and the genetic > algorithm results are showing which distributions can be run with loc > = min(data) and scale = max(data) - min(data). ?With that information > in hand I can then speed up the overall fitting considerably by not > fitting those parameters. good to hear I used loc = min(data) - 1e-6 to avoid evaluating the log likelihood at the lower bound, in some cases this might be infinite. (I picked 1e-6 arbitrarily) >>> scipy.stats.gamma.logpdf(np.linspace(0,1,3), 300.5609591140931425, loc=0, scale=0.25) array([ -Inf, -1205.37511721, -999.735283 ]) One more: vonmises doesn't define the bounds because it can be used as a circular distribution. If you want to include it as a regular distribution, then you should set the bound a and b. stats.distributions.vonmises.a = -np.pi stats.distributions.vonmises.b = np.pi (Even more fun would be to specify a proper Bayesian prior on all distribution parameters to make the random starting values match the distribution.) Josef > > ? ? James > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From zunzun at zunzun.com Fri Oct 1 14:41:48 2010 From: zunzun at zunzun.com (James Phillips) Date: Fri, 1 Oct 2010 08:41:48 -1000 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Fri, Oct 1, 2010 at 8:24 AM, wrote: > I used loc = min(data) - 1e-6 ?to avoid evaluating the log likelihood > at the lower bound, in some cases this might be infinite. > (I picked 1e-6 arbitrarily) > >>>> scipy.stats.gamma.logpdf(np.linspace(0,1,3), 300.5609591140931425, loc=0, scale=0.25) > array([ ? ? ? ? ?-Inf, -1205.37511721, ?-999.735283 ?]) Ah - understood. Thank you. > One more: vonmises doesn't define the bounds because it can be used as > a circular distribution. If you want to include it as a regular > distribution, then you should set the bound a and b. > stats.distributions.vonmises.a = -np.pi > stats.distributions.vonmises.b = np.pi I shall try this. James From stefan at sun.ac.za Sat Oct 2 18:01:06 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 3 Oct 2010 00:01:06 +0200 Subject: [SciPy-Dev] scikits.scipy.org now active Message-ID: Hi all, Thanks to Aaron from Enthought, the SciKits homepage is now available on http://scikits.scipy.org (Google still hosts the application, but it answers to this name as well). Regards St?fan From matthew.brett at gmail.com Mon Oct 4 15:04:29 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 4 Oct 2010 12:04:29 -0700 Subject: [SciPy-Dev] Scipy git? Message-ID: Hi, > I'd recommend going for git already. Some instructions how to get set up > using Scipy's SVN mirror can be found here: > > http://projects.scipy.org/numpy/wiki/GitMirror Speaking of which ;) Is it too soon to ask whether there's a plan to switch scipy over to git? It would be a happy day for me. Thanks a lot, Matthew From pav at iki.fi Mon Oct 4 15:41:06 2010 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 4 Oct 2010 19:41:06 +0000 (UTC) Subject: [SciPy-Dev] Scipy switching to Git? References: Message-ID: Mon, 04 Oct 2010 12:04:29 -0700, Matthew Brett wrote: >> I'd recommend going for git already. Some instructions how to get set >> up using Scipy's SVN mirror can be found here: >> >> http://projects.scipy.org/numpy/wiki/GitMirror > > Speaking of which ;) > > Is it too soon to ask whether there's a plan to switch scipy over to > git? It would be a happy day for me. To my understanding there is a tacit plan to switch at "some point". From my POV, the sooner the better. The repository conversion is mostly done: http://github.com/pv/numpy2git/ But it still needs some sanity checking (i.e. check dangling branches). Pauli From millman at berkeley.edu Mon Oct 4 16:01:31 2010 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 4 Oct 2010 13:01:31 -0700 Subject: [SciPy-Dev] Scipy switching to Git? In-Reply-To: References: Message-ID: On Mon, Oct 4, 2010 at 12:41 PM, Pauli Virtanen wrote: > To my understanding there is a tacit plan to switch at "some point". From > my POV, the sooner the better. +1 Now that numpy has moved over to git, it would be great to get scipy moved as well (and the sooner the better). Jarrod From matthew.brett at gmail.com Mon Oct 4 16:29:58 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 4 Oct 2010 13:29:58 -0700 Subject: [SciPy-Dev] OSX build failing at gfortran link Message-ID: Hi, I am sorry for my ignorance, but I am running into trouble building SVN current scipy on OSX: macpython python 2.6.6 OSX 10.6.4 XCode 3.2.3 numpy 1.5.0 from sourceforge binary gfortran from ATT - 4.2.3 export CXX=/usr/bin/c++-4.0 Specifically, my build ends like this: collect2: ld returned 1 exit status ld: warning: object file compiled with -mlong-branch which is no longer needed. To remove this warning, recompile without -mlong-branch: /usr/local/lib/gcc/powerpc-apple-darwin8/4.2.3/crt3.o ld: warning: in build/temp.macosx-10.3-fat-2.6/libdfftpack.a, file was built for unsupported file format which is not the architecture being linked (ppc) ld: warning: in build/temp.macosx-10.3-fat-2.6/libfftpack.a, file was built for unsupported file format which is not the architecture being linked (ppc) lipo: can't open input file: /var/folders/jg/jgfZ12ZXHwGSFKD85xLpLk+++TI/-Tmp-//ccLDwo7E.out (No such file or directory) error: Command "/usr/local/bin/gfortran-4.0 -Wall -arch ppc -arch i686 -arch x86_64 -arch ppc64 -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/_fftpackmodule.o build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfft.o build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/drfft.o build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zrfft.o build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfftnd.o build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/src/dct.o build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/fortranobject.o -Lbuild/temp.macosx-10.3-fat-2.6 -ldfftpack -lfftpack -lgfortran -o build/lib.macosx-10.3-fat-2.6/scipy/fftpack/_fftpack.so" failed with exit status 1 If I remove the "-arch x86_64 -arch ppc64" from the above line, it compiles OK. I suppose this is because my macpython 2.6 is a 32 bit binary. I tried setting: export FFLAGS='-arch i386 -arch ppc' but it looks as if ('python setup.py config_fc --help-fcompiler') numpy distutils ignores the FFLAGS in generating the link flags for gfortran. My only option then is the rather unpleasant: export LDFLAGS="-Wall -arch ppc -arch i386 -undefined dynamic_lookup -bundle" Thus the combination of: export FFLAGS='-arch i386 -arch ppc' export LDFLAGS="-Wall -arch ppc -arch i386 -undefined dynamic_lookup -bundle" does work, and generates scipy with no test failures. Looking at numpy/distutils/fcompiler/gnu.py, I see that the function ``_can_target`` cycles through the arch flags seeing whether they (on their own) raise an error when compiling an empty .f file. I guess the problem here is that, although gfortran will compile a file with '-arch x86_64' - it can't link a python extension with '-arch x86_64'. Would it make any sense to get the -arch flags somehow from the default C linking flags? Is that sensible? Is it easy? Thanks a lot, Matthew From ralf.gommers at googlemail.com Thu Oct 7 05:45:30 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 7 Oct 2010 17:45:30 +0800 Subject: [SciPy-Dev] OSX build failing at gfortran link In-Reply-To: References: Message-ID: On Tue, Oct 5, 2010 at 4:29 AM, Matthew Brett wrote: > Hi, > > I am sorry for my ignorance, but I am running into trouble building > SVN current scipy on OSX: > > macpython python 2.6.6 > OSX 10.6.4 > XCode 3.2.3 > numpy 1.5.0 from sourceforge binary > gfortran from ATT - 4.2.3 > export CXX=/usr/bin/c++-4.0 > > Specifically, my build ends like this: > > collect2: ld returned 1 exit status > ld: warning: object file compiled with -mlong-branch which is no > longer needed. To remove this warning, recompile without > -mlong-branch: /usr/local/lib/gcc/powerpc-apple-darwin8/4.2.3/crt3.o > ld: warning: in build/temp.macosx-10.3-fat-2.6/libdfftpack.a, file was > built for unsupported file format which is not the architecture being > linked (ppc) > ld: warning: in build/temp.macosx-10.3-fat-2.6/libfftpack.a, file was > built for unsupported file format which is not the architecture being > linked (ppc) > lipo: can't open input file: > /var/folders/jg/jgfZ12ZXHwGSFKD85xLpLk+++TI/-Tmp-//ccLDwo7E.out (No > such file or directory) > error: Command "/usr/local/bin/gfortran-4.0 -Wall -arch ppc -arch i686 > -arch x86_64 -arch ppc64 -Wall -undefined dynamic_lookup -bundle > > build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/_fftpackmodule.o > build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfft.o > build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/drfft.o > build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zrfft.o > build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfftnd.o > > build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/src/dct.o > > build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/fortranobject.o > -Lbuild/temp.macosx-10.3-fat-2.6 -ldfftpack -lfftpack -lgfortran -o > build/lib.macosx-10.3-fat-2.6/scipy/fftpack/_fftpack.so" failed with > exit status 1 > > If I remove the "-arch x86_64 -arch ppc64" from the above line, it compiles > OK. > > I suppose this is because my macpython 2.6 is a 32 bit binary. Correct, see #1399. > > I tried setting: > > export FFLAGS='-arch i386 -arch ppc' > > but it looks as if ('python setup.py config_fc --help-fcompiler') > numpy distutils ignores the FFLAGS in generating the link flags for > gfortran. > > My only option then is the rather unpleasant: > > export LDFLAGS="-Wall -arch ppc -arch i386 -undefined dynamic_lookup > -bundle" > Your other option is to remove "ppc64" from line 257 of numpy/distutils/fcompiler/gnu.py. > > Thus the combination of: > > export FFLAGS='-arch i386 -arch ppc' > export LDFLAGS="-Wall -arch ppc -arch i386 -undefined dynamic_lookup > -bundle" > > does work, and generates scipy with no test failures. > > Looking at numpy/distutils/fcompiler/gnu.py, I see that the function > ``_can_target`` cycles through the arch flags seeing whether they (on > their own) raise an error when compiling an empty .f file. I guess > the problem here is that, although gfortran will compile a file with > '-arch x86_64' - it can't link a python extension with '-arch x86_64'. > Would it make any sense to get the -arch flags somehow from the > default C linking flags? Is that sensible? Is it easy? > I think it makes sense but I'm not sure it's easy. I'll try to figure that out before 1.5.1rc1, if it's not easy then the patch from #1399 should just be reverted. Mea culpa again for causing this problem. Ralf > > Thanks a lot, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Oct 7 23:01:01 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Oct 2010 21:01:01 -0600 Subject: [SciPy-Dev] SVN access for Wes McKinney Message-ID: Hi All, Can we get SVN access fo Wes McKinney. I'm not sure who handles this now. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Thu Oct 7 23:06:44 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Thu, 7 Oct 2010 23:06:44 -0400 Subject: [SciPy-Dev] SVN access for Wes McKinney In-Reply-To: References: Message-ID: <796F1512-95A6-451D-8156-8D0A32481892@enthought.com> Hopefully we can move development to Git for Scipy as well. Then that part cam be managed more transparently and efficiently as well. Travis -- (mobile phone of) Travis Oliphant Enthought, Inc. 1-512-536-1057 http://www.enthought.com On Oct 7, 2010, at 11:01 PM, Charles R Harris wrote: > Hi All, > > Can we get SVN access fo Wes McKinney. I'm not sure who handles this now. > > Chuck > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From pav at iki.fi Fri Oct 8 03:35:27 2010 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 8 Oct 2010 07:35:27 +0000 (UTC) Subject: [SciPy-Dev] SVN access for Wes McKinney References: Message-ID: Thu, 07 Oct 2010 21:01:01 -0600, Charles R Harris wrote: > Can we get SVN access fo Wes McKinney. I'm not sure who handles this > now. It requires someone with root access on new.scipy.org -- I can't do that. Pauli From ariver at enthought.com Fri Oct 8 10:46:11 2010 From: ariver at enthought.com (Aaron River) Date: Fri, 8 Oct 2010 09:46:11 -0500 Subject: [SciPy-Dev] SVN access for Wes McKinney In-Reply-To: References: Message-ID: Hello Chuck, Wes now has write access to the repo. Let me know if there is anything else you need. Thanks, -- Aaron On Thu, Oct 7, 2010 at 22:01, Charles R Harris wrote: > Hi All, > > Can we get SVN access fo Wes McKinney. I'm not sure who handles this now. > > Chuck > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > From vincent at vincentdavis.net Sun Oct 10 13:45:27 2010 From: vincent at vincentdavis.net (Vincent Davis) Date: Sun, 10 Oct 2010 11:45:27 -0600 Subject: [SciPy-Dev] wig_import_helper error Message-ID: This ticket might be related http://projects.scipy.org/scipy/ticket/1210 This is on a fresh install from source of py2.7, numpy, scipy 64bit on osx 10.6 ====================================================================== ERROR: Failure: ImportError (dynamic module does not define init function (init_csr)) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.2-py2.7.egg/nose/loader.py", line 382, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.2-py2.7.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.2-py2.7.egg/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/stats/__init__.py", line 7, in from stats import * File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/stats/stats.py", line 207, in import distributions File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/stats/distributions.py", line 13, in from scipy import optimize File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/__init__.py", line 14, in from nonlin import * File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/nonlin.py", line 115, in import scipy.sparse.linalg File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/__init__.py", line 6, in from csr import * File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/csr.py", line 12, in from sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \ File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/sparsetools/__init__.py", line 4, in from csr import * File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/sparsetools/csr.py", line 25, in _csr = swig_import_helper() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/sparsetools/csr.py", line 21, in swig_import_helper _mod = imp.load_module('_csr', fp, pathname, description) ImportError: dynamic module does not define init function (init_csr) ---------------------------------------------------------------------- Ran 1801 tests in 7.235s FAILED (KNOWNFAIL=6, SKIP=29, errors=9) -- Thanks Vincent Davis 720-301-3003 From zunzun at zunzun.com Mon Oct 11 15:45:20 2010 From: zunzun at zunzun.com (James Phillips) Date: Mon, 11 Oct 2010 09:45:20 -1000 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: The genetic algorithm approach is not working as a general solution to the problem of finding starting parameters for fmin() for statistical distributions, presumably due to extreme parameter sensitivity. I do not see a general solution to the problem given these results. See the attached Python file, also copied below. My results: Digits of precision test for the beta distribution nnlf native = inf nnlf 16 digits = 10.14091764 nnlf 15 digits = 10.3222074111 nnlf 14 digits = 10.977829575 nnlf 13 digits = inf nnlf 12 digits = 13.198954184 James import scipy, scipy.stats data = scipy.array([ 3.017,2.822,2.632,2.287,2.207,2.048, 1.963,1.784,1.712,2.972,2.719,2.495, 2.070,1.969,1.768,1.677,1.479,1.387, 2.843,2.485,2.163,1.687,1.408,1.279, 1.016,0.742,0.607]) # parameters p1 = 7.69589403034175001E-01 p2 = 5.52884409849620395E-01 p3 = 6.06094740472452820E-01 p4 = 2.41090525952754753E+00 print "Digits of precision test for the beta distribution" print "nnlf native =", scipy.stats.beta.nnlf([p1, p2, p3, p3], data) print "nnlf 16 digits =", scipy.stats.beta.nnlf([float("%.16E" % p1), float("%.16E" % p2), float("%.16E" % p3), float("%.16E" % p4)], data) print "nnlf 15 digits =", scipy.stats.beta.nnlf([float("%.15E" % p1), float("%.15E" % p2), float("%.15E" % p3), float("%.15E" % p4)], data) print "nnlf 14 digits =", scipy.stats.beta.nnlf([float("%.14E" % p1), float("%.14E" % p2), float("%.14E" % p3), float("%.14E" % p4)], data) print "nnlf 13 digits =", scipy.stats.beta.nnlf([float("%.13E" % p1), float("%.13E" % p2), float("%.13E" % p3), float("%.13E" % p4)], data) print "nnlf 12 digits =", scipy.stats.beta.nnlf([float("%.12E" % p1), float("%.12E" % p2), float("%.12E" % p3), float("%.12E" % p4)], data) On Fri, Oct 1, 2010 at 7:58 AM, James Phillips wrote: > On Fri, Oct 1, 2010 at 7:21 AM, James Phillips wrote: >> >> Here are current timing results fitting both loc and scale... > > I'm iterating over all continuous distributions now, and the genetic > algorithm results are showing which distributions can be run with loc > = min(data) and scale = max(data) - min(data). ?With that information > in hand I can then speed up the overall fitting considerably by not > fitting those parameters. > > ? ? James > -------------- next part -------------- A non-text attachment was scrubbed... Name: sensitivity.py Type: text/x-python Size: 1178 bytes Desc: not available URL: From josef.pktd at gmail.com Mon Oct 11 16:10:06 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 11 Oct 2010 16:10:06 -0400 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Mon, Oct 11, 2010 at 3:45 PM, James Phillips wrote: > The genetic algorithm approach is not working as a general solution to > the problem of finding starting parameters for fmin() for statistical > distributions, presumably due to extreme parameter sensitivity. ?I do > not see a general solution to the problem given these results. ?See > the attached Python file, also copied below. > > My results: > > Digits of precision test for the beta distribution > nnlf native ? ?= inf > nnlf 16 digits = 10.14091764 > nnlf 15 digits = 10.3222074111 > nnlf 14 digits = 10.977829575 > nnlf 13 digits = inf > nnlf 12 digits = 13.198954184 > > > ? ? James > > import scipy, scipy.stats > > data = scipy.array([ > 3.017,2.822,2.632,2.287,2.207,2.048, > 1.963,1.784,1.712,2.972,2.719,2.495, > 2.070,1.969,1.768,1.677,1.479,1.387, > 2.843,2.485,2.163,1.687,1.408,1.279, > 1.016,0.742,0.607]) > > # parameters > p1 = 7.69589403034175001E-01 > p2 = 5.52884409849620395E-01 > p3 = 6.06094740472452820E-01 > p4 = 2.41090525952754753E+00 > > print "Digits of precision test for the beta distribution" > print "nnlf native ? ?=", scipy.stats.beta.nnlf([p1, p2, p3, p3], data) typo should be p4 > print "nnlf 16 digits =", scipy.stats.beta.nnlf([float("%.16E" % p1), > float("%.16E" % p2), float("%.16E" % p3), float("%.16E" % p4)], data) > print "nnlf 15 digits =", scipy.stats.beta.nnlf([float("%.15E" % p1), > float("%.15E" % p2), float("%.15E" % p3), float("%.15E" % p4)], data) > print "nnlf 14 digits =", scipy.stats.beta.nnlf([float("%.14E" % p1), > float("%.14E" % p2), float("%.14E" % p3), float("%.14E" % p4)], data) > print "nnlf 13 digits =", scipy.stats.beta.nnlf([float("%.13E" % p1), > float("%.13E" % p2), float("%.13E" % p3), float("%.13E" % p4)], data) > print "nnlf 12 digits =", scipy.stats.beta.nnlf([float("%.12E" % p1), > float("%.12E" % p2), float("%.12E" % p3), float("%.12E" % p4)], data) If I remember correctly, you have observations that are too close to the upper boundary. >>> data.max() -(p3+p4) -4.4408920985006262e-016 If you have an observation at the boundary, the loglikelihood is inf >>> scipy.stats.beta.nnlf([p1, p2, p3, p4], data) 10.14091764000147 reduce variance a bit, shrinks the support >>> scipy.stats.beta.nnlf([p1, p2, p3, p4-1e-15], data) 1.#INF I think in these cases you have to keep the boundary of the support away from the max and min of the data. Similar in other distributions, as I mentioned before. If MLE doesn't work for a distribution then a global optimizer wouldn't help either. In these cases, usually another estimation method is recommended in the literature. For example matching quantiles similar to your initial version. Are all distributions that have support in the entire real line working ? Josef > > > > > On Fri, Oct 1, 2010 at 7:58 AM, James Phillips wrote: >> On Fri, Oct 1, 2010 at 7:21 AM, James Phillips wrote: >>> >>> Here are current timing results fitting both loc and scale... >> >> I'm iterating over all continuous distributions now, and the genetic >> algorithm results are showing which distributions can be run with loc >> = min(data) and scale = max(data) - min(data). ?With that information >> in hand I can then speed up the overall fitting considerably by not >> fitting those parameters. >> >> ? ? James >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > From zunzun at zunzun.com Mon Oct 11 16:24:17 2010 From: zunzun at zunzun.com (James Phillips) Date: Mon, 11 Oct 2010 10:24:17 -1000 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Mon, Oct 11, 2010 at 10:10 AM, wrote: > > typo should be p4 Oops - thank you. > If I remember correctly, you have observations that are too close to > the upper boundary. > > If you have an observation at the boundary, the loglikelihood is inf > > I think in these cases you have to keep the boundary of the support > away from the max and min of the data. Similar in other distributions, > as I mentioned before. Thank you. > If MLE doesn't work for a distribution then a global optimizer > wouldn't help either. In these cases, usually another estimation > method is recommended in the literature. For example matching > quantiles similar to your initial version. Thank you again. > Are all distributions that have support in the entire real line working ? Stopped at beta. I'll pick up with this work again after having read your discussion above. James From josef.pktd at gmail.com Mon Oct 11 17:02:54 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 11 Oct 2010 17:02:54 -0400 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com> <4CA22FA3.5080402@gmail.com> Message-ID: On Mon, Oct 11, 2010 at 4:24 PM, James Phillips wrote: > On Mon, Oct 11, 2010 at 10:10 AM, ? wrote: >> >> typo should be p4 > > Oops - thank you. > > >> If I remember correctly, you have observations that are too close to >> the upper boundary. >> >> If you have an observation at the boundary, the loglikelihood is inf >> >> I think in these cases you have to keep the boundary of the support >> away from the max and min of the data. Similar in other distributions, >> as I mentioned before. > > Thank you. > > >> If MLE doesn't work for a distribution then a global optimizer >> wouldn't help either. In these cases, usually another estimation >> method is recommended in the literature. For example matching >> quantiles similar to your initial version. > > Thank you again. > > >> Are all distributions that have support in the entire real line working ? > > Stopped at beta. ?I'll pick up with this work again after having read > your discussion above. I spent 1 to 2 weeks on the pareto family this summer. It's easy to get lost while trying to figure out specific distributions. I would recommend that you concentrate on the "easy" distributions first, and then work your way to the more difficult ones. I hope that eventually I also have enough time again to finish up the other (non-mle) estimators. Josef > > > ? ? James > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From matthew.brett at gmail.com Mon Oct 11 19:33:09 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 11 Oct 2010 16:33:09 -0700 Subject: [SciPy-Dev] OSX build failing at gfortran link In-Reply-To: References: Message-ID: Hi Ralf, >> Looking at numpy/distutils/fcompiler/gnu.py, I see that the function >> ``_can_target`` cycles through the arch flags seeing whether they (on >> their own) raise an error when compiling an empty .f file. ? ?I guess >> the problem here is that, although gfortran will compile a file with >> '-arch x86_64' - it can't link a python extension with '-arch x86_64'. >> ?Would it make any sense to get the -arch flags somehow from the >> default C linking flags? ?Is that sensible? ?Is it easy? > > I think it makes sense but I'm not sure it's easy. I'll try to figure that > out before 1.5.1rc1, if it's not easy then the patch from #1399 should just > be reverted. Mea culpa again for causing this problem. I had a go - always unpleasant with distutils. I think this works: http://github.com/matthew-brett/numpy/compare/numpy:master...farchs-from-c At least the standard 'CXX=/usr/bin/g++-4.0 python setup.py install' routine gave a working scipy passing all tests. Does it look OK to you? If so I'll submit a pull request to numpy. Cheers, Matthew From Per.Brodtkorb at ffi.no Tue Oct 12 04:17:11 2010 From: Per.Brodtkorb at ffi.no (Per.Brodtkorb at ffi.no) Date: Tue, 12 Oct 2010 10:17:11 +0200 Subject: [SciPy-Dev] Subversion scipy.stats irregular problem with source code example In-Reply-To: References: <4CA22B5A.3020601@gmail.com><4CA22FA3.5080402@gmail.com> Message-ID: <1ED225FF18AA8B48AC192F7E1D032C6E0115F731@hbu-posten.ffi.no> On Mon, Oct 11, 2010 at 4:24 PM, James Phillips wrote: > On Mon, Oct 11, 2010 at 10:10 AM, ? wrote: >> >> typo should be p4 > > Oops - thank you. > > >> If I remember correctly, you have observations that are too close to >> the upper boundary. >> >> If you have an observation at the boundary, the loglikelihood is inf >> >> I think in these cases you have to keep the boundary of the support >> away from the max and min of the data. Similar in other distributions, >> as I mentioned before. > > Thank you. > > >> If MLE doesn't work for a distribution then a global optimizer >> wouldn't help either. In these cases, usually another estimation >> method is recommended in the literature. For example matching >> quantiles similar to your initial version. > The maximum product of spacings (MPS) method is a general method of estimating parameters in continuous univariate distributions that in many cases solves this problem. It is especially suited to cases where one of the parameters is an unknown shifted origin. This occurs, for example, in the three-parameter lognormal, gamma, Generalized Extreme Value Generalized Pareto and Weibull models. For such distributions it is known that maximum likelihood (ML) estimation can break down because the likelihood is unbounded and this can lead to inconsistent estimators. In particular MPS is shown to give consistent estimators with asymptotic efficiency equal to ML estimators when these exist. Moreover it gives consistent, asymptoti- cally efficient estimators in situations where ML fails. Finally, as a by-product of the MPS, a goodness of ?t statistic, Moran?s statistic, is available for evaluating the fit to the selected distribution. Two years ago I implemented this method + some other enhancements. The source code is available here: http://code.google.com/p/joepython/source/browse/trunk/joepython/scipystats/enhance/per/distributions_per.py The parameters are estimated by minimizing the method nlogps in the rv_continous class. You will find more details on the method in the following references: Estimating Parameters in Continuous Univariate Distributions with a Shifted Origin R. C. H. Cheng and N. A. K. Amin (http://links.jstor.org/sici?sici=0035-9246%281983%2945%3A3%3C394%3AEPICUD%3E2.0.CO%3B2-N) A Note on the Estimation of Extreme Value Distributions Using Maximum Product of Spacings T. S. T. Wong and W. K. Li Lecture Notes-Monograph Series Vol. 52, Time Series and Related Topics: In Memory of Ching-Zong Wei (2006), pp. 272-283 (article consists of 12 pages) Published by: Institute of Mathematical Statistics Stable URL: http://www.jstor.org/stable/20461444 A Goodness-Of-Fit Test Using Moran's Statistic with Estimated Parameters R. C. H. Cheng and M. A. Stephens (http://links.jstor.org/sici?sici=0006-3444%28198906%2976%3A2%3C385%3AAGTUMS%3E2.0.CO%3B2-1 ) From ralf.gommers at googlemail.com Tue Oct 12 10:58:37 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 12 Oct 2010 22:58:37 +0800 Subject: [SciPy-Dev] OSX build failing at gfortran link In-Reply-To: References: Message-ID: Hi Matthew, On Tue, Oct 12, 2010 at 7:33 AM, Matthew Brett wrote: > Hi Ralf, > > >> Looking at numpy/distutils/fcompiler/gnu.py, I see that the function > >> ``_can_target`` cycles through the arch flags seeing whether they (on > >> their own) raise an error when compiling an empty .f file. I guess > >> the problem here is that, although gfortran will compile a file with > >> '-arch x86_64' - it can't link a python extension with '-arch x86_64'. > >> Would it make any sense to get the -arch flags somehow from the > >> default C linking flags? Is that sensible? Is it easy? > > > > I think it makes sense but I'm not sure it's easy. I'll try to figure > that > > out before 1.5.1rc1, if it's not easy then the patch from #1399 should > just > > be reverted. Mea culpa again for causing this problem. > > I had a go - always unpleasant with distutils. I think this works: > > http://github.com/matthew-brett/numpy/compare/numpy:master...farchs-from-c > > At least the standard 'CXX=/usr/bin/g++-4.0 python setup.py install' > routine gave a working scipy passing all tests. > > Does it look OK to you? If so I'll submit a pull request to numpy. > > Thanks for diving in! That works for me, and looks like the right thing to do. The methods you removed are clearly marked as private in the base class and not used anywhere else as far as I can tell, but it would be good to get a confirmation from someone familiar with the history / use cases of fcompiler that this is fine. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Oct 12 16:22:15 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 12 Oct 2010 13:22:15 -0700 Subject: [SciPy-Dev] matlab reading / writing changes - review anyone? Message-ID: Hi, In the spirit of the development workflow discussions on the numpy list - I'd be very grateful for a review of changes I've made to the matlab reader. The changes are to: a) Allow the reader to read some malformed files that matlab is able to read b) Prevent the writer needlessly up-casting some numpy array types (uint8, uint32) I'm sorry - I probably have probably made the comparison a little too large to be taken in in an instant, but I'd be very grateful for feedback before merge. http://github.com/matthew-brett/scipy-work/compare/master...mio-read-fix Thanks, Matthew From pav at iki.fi Tue Oct 12 17:31:56 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 12 Oct 2010 21:31:56 +0000 (UTC) Subject: [SciPy-Dev] matlab reading / writing changes - review anyone? References: Message-ID: Tue, 12 Oct 2010 13:22:15 -0700, Matthew Brett wrote: [clip] > I'm sorry - I probably have probably made the comparison a little too > large to be taken in in an instant, but I'd be very grateful for > feedback before merge. Based on a very shallow glance, looks mostly good to me, although: - Using UserWarning sounds a bit nasty -- maybe use class MatlabReaderWarning(UserWarning): pass instead? - Is there a test for "varmats_from_mat"? - Breaks on Python 3: ====================================================================== ERROR: test_mio.test_round_types ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.1/dist-packages/nose/case.py", line 177, in runTest self.test(*self.arg) File "/home/pauli/prj/scipy/scipy/dist/linux/lib/python3.1/site- packages/scipy/io/matlab/tests/test_mio.py", line 825, in test_round_types vars = loadmat(stream) File "/home/pauli/prj/scipy/scipy/dist/linux/lib/python3.1/site- packages/scipy/io/matlab/mio.py", line 149, in loadmat matfile_dict = MR.get_variables(variable_names) File "/home/pauli/prj/scipy/scipy/dist/linux/lib/python3.1/site- packages/scipy/io/matlab/mio4.py", line 304, in get_variables mdict[name] = self.read_var_array(hdr) File "/home/pauli/prj/scipy/scipy/dist/linux/lib/python3.1/site- packages/scipy/io/matlab/mio4.py", line 283, in read_var_array return self._matrix_reader.array_from_header(header, process) File "/home/pauli/prj/scipy/scipy/dist/linux/lib/python3.1/site- packages/scipy/io/matlab/mio4.py", line 132, in array_from_header raise TypeError('No reader for class code %s' % mclass) TypeError: No reader for class code 4 ====================================================================== ERROR: test_mio5_utils.test_zero_byte_string ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.1/dist-packages/nose/case.py", line 177, in runTest self.test(*self.arg) File "/home/pauli/prj/scipy/scipy/dist/linux/lib/python3.1/site- packages/scipy/io/matlab/tests/test_mio5_utils.py", line 174, in test_zero_byte_string _write_stream(str_io, tag.tostring() + ' ') TypeError: can't concat bytes to str ---------------------------------------------------------------------- Ran 402 tests in 2.391s -- Pauli Virtanen From matthew.brett at gmail.com Tue Oct 12 23:40:06 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 12 Oct 2010 20:40:06 -0700 Subject: [SciPy-Dev] matlab reading / writing changes - review anyone? In-Reply-To: References: Message-ID: Hi, >> I'm sorry - I probably have probably made the comparison a little too >> large to be taken in in an instant, but I'd be very grateful for >> feedback before merge. > > Based on a very shallow glance, looks mostly good to me, although: > > - Using UserWarning sounds a bit nasty -- maybe use > ?class MatlabReaderWarning(UserWarning): pass instead? Thanks, good idea. > - Is there a test for "varmats_from_mat"? No, only the doctest, thanks for the reminder, I'll put some in. > - Breaks on Python 3: Aha - that's sort of what I was expecting ;) At last I will set myself up with a python 3 rig and test it properly. Thanks a lot for the feedback, it's always cheering, See you, Matthew From ralf.gommers at googlemail.com Thu Oct 14 08:50:00 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 14 Oct 2010 20:50:00 +0800 Subject: [SciPy-Dev] OSX build failing at gfortran link In-Reply-To: References: Message-ID: On Tue, Oct 12, 2010 at 10:58 PM, Ralf Gommers wrote: > Hi Matthew, > > On Tue, Oct 12, 2010 at 7:33 AM, Matthew Brett wrote: > >> Hi Ralf, >> >> >> Looking at numpy/distutils/fcompiler/gnu.py, I see that the function >> >> ``_can_target`` cycles through the arch flags seeing whether they (on >> >> their own) raise an error when compiling an empty .f file. I guess >> >> the problem here is that, although gfortran will compile a file with >> >> '-arch x86_64' - it can't link a python extension with '-arch x86_64'. >> >> Would it make any sense to get the -arch flags somehow from the >> >> default C linking flags? Is that sensible? Is it easy? >> > >> > I think it makes sense but I'm not sure it's easy. I'll try to figure >> that >> > out before 1.5.1rc1, if it's not easy then the patch from #1399 should >> just >> > be reverted. Mea culpa again for causing this problem. >> >> I had a go - always unpleasant with distutils. I think this works: >> >> http://github.com/matthew-brett/numpy/compare/numpy:master...farchs-from-c >> >> At least the standard 'CXX=/usr/bin/g++-4.0 python setup.py install' >> routine gave a working scipy passing all tests. >> >> Does it look OK to you? If so I'll submit a pull request to numpy. >> >> Thanks for diving in! That works for me, and looks like the right thing to > do. The methods you removed are clearly marked as private in the base class > and not used anywhere else as far as I can tell, but it would be good to get > a confirmation from someone familiar with the history / use cases of > fcompiler that this is fine. > David W.F. tested on two non-standard Python installs, all good: http://projects.scipy.org/numpy/ticket/1399#comment:8. So time to apply your fix I think. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Oct 16 06:43:59 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 16 Oct 2010 18:43:59 +0800 Subject: [SciPy-Dev] OSX build failing at gfortran link In-Reply-To: References: Message-ID: On Thu, Oct 14, 2010 at 8:50 PM, Ralf Gommers wrote: > > > On Tue, Oct 12, 2010 at 10:58 PM, Ralf Gommers > wrote: >> >> Hi Matthew, >> >> On Tue, Oct 12, 2010 at 7:33 AM, Matthew Brett >> wrote: >>> >>> Hi Ralf, >>> >>> >> Looking at numpy/distutils/fcompiler/gnu.py, I see that the function >>> >> ``_can_target`` cycles through the arch flags seeing whether they (on >>> >> their own) raise an error when compiling an empty .f file. ? ?I guess >>> >> the problem here is that, although gfortran will compile a file with >>> >> '-arch x86_64' - it can't link a python extension with '-arch x86_64'. >>> >> ?Would it make any sense to get the -arch flags somehow from the >>> >> default C linking flags? ?Is that sensible? ?Is it easy? >>> > >>> > I think it makes sense but I'm not sure it's easy. I'll try to figure >>> > that >>> > out before 1.5.1rc1, if it's not easy then the patch from #1399 should >>> > just >>> > be reverted. Mea culpa again for causing this problem. >>> >>> I had a go - always unpleasant with distutils. ?I think this works: >>> >>> >>> http://github.com/matthew-brett/numpy/compare/numpy:master...farchs-from-c >>> >>> At least the standard 'CXX=/usr/bin/g++-4.0 python setup.py install' >>> routine gave a working scipy passing all tests. >>> >>> Does it look OK to you? ?If so I'll submit a pull request to numpy. >>> >> Thanks for diving in! That works for me, and looks like the right thing to >> do. The methods you removed are clearly marked as private in the base class >> and not used anywhere else as far as I can tell, but it would be good to get >> a confirmation from someone familiar with the history / use cases of >> fcompiler that this is fine. > > David W.F. tested on two non-standard Python installs, all good: > http://projects.scipy.org/numpy/ticket/1399#comment:8. So time to apply your > fix I think. > Committed now, and http://projects.scipy.org/numpy/ticket/1399 closed. Thanks again Matthew! Cheers, Ralf From matthew.brett at gmail.com Sat Oct 16 06:47:13 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 16 Oct 2010 03:47:13 -0700 Subject: [SciPy-Dev] OSX build failing at gfortran link In-Reply-To: References: Message-ID: Hi, >> David W.F. tested on two non-standard Python installs, all good: >> http://projects.scipy.org/numpy/ticket/1399#comment:8. So time to apply your >> fix I think. >> > Committed now, and http://projects.scipy.org/numpy/ticket/1399 closed. > Thanks again Matthew! Ah - good - thanks for applying. I suppose we now have to wait to see what obscure thing this one breaks! Best, Matthew From ralf.gommers at googlemail.com Sat Oct 16 07:21:37 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 16 Oct 2010 19:21:37 +0800 Subject: [SciPy-Dev] OSX build failing at gfortran link In-Reply-To: References: Message-ID: On Sat, Oct 16, 2010 at 6:47 PM, Matthew Brett wrote: > Hi, > >>> David W.F. tested on two non-standard Python installs, all good: >>> http://projects.scipy.org/numpy/ticket/1399#comment:8. So time to apply your >>> fix I think. >>> >> Committed now, and http://projects.scipy.org/numpy/ticket/1399 closed. >> Thanks again Matthew! > > Ah - good - thanks for applying. ?I suppose we now have to wait to see > what obscure thing this one breaks! We'll probably find out the day after the release.... From stefan at sun.ac.za Sat Oct 16 18:53:29 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sun, 17 Oct 2010 00:53:29 +0200 Subject: [SciPy-Dev] Scipy.org down? Message-ID: Hi all, It looks like projects.scipy.org and scipy.org are down. Can someone confirm? Cheers St?fan From robert.kern at gmail.com Sat Oct 16 20:08:55 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 16 Oct 2010 19:08:55 -0500 Subject: [SciPy-Dev] Scipy.org down? In-Reply-To: References: Message-ID: 2010/10/16 St?fan van der Walt : > Hi all, > > It looks like projects.scipy.org and scipy.org are down. ?Can someone confirm? Yes. The monitors have caught it. Aaron should be on the case. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From wardefar at iro.umontreal.ca Sat Oct 16 20:20:26 2010 From: wardefar at iro.umontreal.ca (David Warde-Farley) Date: Sat, 16 Oct 2010 20:20:26 -0400 Subject: [SciPy-Dev] OSX build failing at gfortran link In-Reply-To: References: Message-ID: <06132ED1-B09A-418E-B956-DE2D1ABAFFC7@iro.umontreal.ca> On 2010-10-16, at 6:47 AM, Matthew Brett wrote: > Ah - good - thanks for applying. I suppose we now have to wait to see > what obscure thing this one breaks! Let's hope not much. :) Thanks again for that fix, Matthew. David From ariver at enthought.com Sat Oct 16 20:23:26 2010 From: ariver at enthought.com (Aaron River) Date: Sat, 16 Oct 2010 19:23:26 -0500 Subject: [SciPy-Dev] Scipy.org down? In-Reply-To: References: Message-ID: Hello St?fan, The SciPy servers are okay. The problem is that I'm having to do some fixin' on our DNS setup. It will flicker a bit over the next hour or so, but it will be better after that. Thanks, -- Aaron River 2010/10/16 St?fan van der Walt > > Hi all, > > It looks like projects.scipy.org and scipy.org are down. ?Can someone confirm? > > Cheers > St?fan > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From ndbecker2 at gmail.com Tue Oct 19 19:40:13 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 19 Oct 2010 19:40:13 -0400 Subject: [SciPy-Dev] scipy svn + python3.1.2? Message-ID: Is this supposed to work? I got as far as: env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 LAPACK=/usr/lib64 python3 setup.py config_fc --fcompiler=gnu95 build Converting to Python3 via 2to3... ... RefactoringTool: /home/nbecker/scipy/build/py3k/scipy/sparse/linalg/dsolve/umfpack/setup.py RefactoringTool: /home/nbecker/scipy/build/py3k/scipy/sparse/linalg/isolve/setup.py /home/nbecker/scipy/build/py3k/scipy/linalg/basic.py /home/nbecker/scipy/build/py3k/scipy/linalg/decomp.py ... /home/nbecker/scipy/build/py3k/scipy/sparse/linalg/isolve/iterative.py Traceback (most recent call last): File "setup.py", line 179, in setup_package() File "setup.py", line 144, in setup_package shutil.copy(site_cfg, src_path) NameError: global name 'shutil' is not defined From ndbecker2 at gmail.com Tue Oct 19 19:53:30 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 19 Oct 2010 19:53:30 -0400 Subject: [SciPy-Dev] scipy svn + python3.1.2? References: Message-ID: Neal Becker wrote: > Is this supposed to work? > > I got as far as: > env ATLAS=/usr/lib64 FFTW=/usr/lib64 BLAS=/usr/lib64 LAPACK=/usr/lib64 > python3 setup.py config_fc --fcompiler=gnu95 build > Converting to Python3 via 2to3... > > ... > RefactoringTool: > /home/nbecker/scipy/build/py3k/scipy/sparse/linalg/dsolve/umfpack/setup.py > RefactoringTool: > /home/nbecker/scipy/build/py3k/scipy/sparse/linalg/isolve/setup.py > /home/nbecker/scipy/build/py3k/scipy/linalg/basic.py > /home/nbecker/scipy/build/py3k/scipy/linalg/decomp.py > ... > /home/nbecker/scipy/build/py3k/scipy/sparse/linalg/isolve/iterative.py > Traceback (most recent call last): > File "setup.py", line 179, in > setup_package() > File "setup.py", line 144, in setup_package > shutil.copy(site_cfg, src_path) > NameError: global name 'shutil' is not defined Well, that was easy: Index: setup.py =================================================================== --- setup.py (revision 6848) +++ setup.py (working copy) @@ -141,6 +141,7 @@ site_cfg = os.path.join(local_path, 'site.cfg') if os.path.isfile(site_cfg): + import shutil shutil.copy(site_cfg, src_path) os.chdir(local_path) From pav at iki.fi Tue Oct 19 19:53:33 2010 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 19 Oct 2010 23:53:33 +0000 (UTC) Subject: [SciPy-Dev] scipy svn + python3.1.2? References: Message-ID: Tue, 19 Oct 2010 19:40:13 -0400, Neal Becker wrote: > NameError: global name 'shutil' is not defined Missing import. Apparently nobody tried having a site.cfg on Python 3 before. From josef.pktd at gmail.com Fri Oct 22 22:49:28 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 22 Oct 2010 22:49:28 -0400 Subject: [SciPy-Dev] stats.distributions ticket 1291 Message-ID: http://projects.scipy.org/scipy/ticket/1291 Warren, if you want, there are a few lines in stats.distributions to change (my scipy is out of date, and I don't have time to update.) As explained in the ticket, the bug cannot be really tested in the test suite, since it shows up only on the first call to the distribution. Thanks, Josef From warren.weckesser at enthought.com Sun Oct 24 16:55:10 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sun, 24 Oct 2010 15:55:10 -0500 Subject: [SciPy-Dev] Added "Checkin Checklist" to the developer's wiki Message-ID: Back in May I sent a "Checkin Checklist" to this list. I've finally added it to the developer's wiki; see the first link under "Guidelines" here: http://projects.scipy.org/scipy/wiki Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sun Oct 24 17:53:05 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 24 Oct 2010 23:53:05 +0200 Subject: [SciPy-Dev] [ANN]: FEMTEC 2011: track on Open-source projects and Python in scientific computing. Message-ID: <20101024215305.GG18057@phare.normalesup.org> The 3rd International Conference on Finite Element Methods in Engineering and Science (FEMTEC 2011, http://hpfem.org/events/femtec-2011/) will have a track on Open-source projects and Python in scientific computing. FEMTEC 2011 is co-organized by the University of Nevada (Reno), Desert Reseach Institute (Reno), Idaho National Laboratory (Idaho Falls, Idaho), and U.S. Army Engineer Research and Development Center (Vicksburg, Mississippi). The objective of the meeting is to strengthen the interaction between researchers who develop new computational methods, and scientists and engineers from various fields who employ numerical methods in their research. Specific focus areas of FEMTEC 2011 include, but are not limited to, the following: * Computational methods in hydrology, atmospheric modeling, and other earth sciences. * Computational methods in nuclear, mechanical, civil, electrical, and other engineering fields. * Mesh generation and scientific visualization. * Open-source projects and Python in scientific computing. Part of the conference will be a software afternoon featuring open source projects of participants. Proceedings Proceedings of FEMTEC 2011 will appear as a special issue of Journal of Computational and Applied Mathematics (2008 SCI impact factor 1.292), and additional high-impact international journals as needed. ____________ On a personnal note, I was at ESCO 2010 this year, the European version organized by the same community, and it was really a refreshing and friendly conference during which I learned a lot. From zunzun at zunzun.com Mon Oct 25 15:36:51 2010 From: zunzun at zunzun.com (James Phillips) Date: Mon, 25 Oct 2010 14:36:51 -0500 Subject: [SciPy-Dev] scipy.stats.distributions: note on initial parameters for fitting the beta distribution Message-ID: Here is a link on estimating initial parameters for fitting the beta distribution. I'm posting as it seems a good bit different from the current calculation in scipy.stats.distribution, and it calculates only the sample mean and biased sample variance. I am not competent to judge which of the two methods is better for scipy, and hope someone on the mailing list can do so. http://www.xycoon.com/beta_parameterestimation.htm James From josef.pktd at gmail.com Mon Oct 25 16:14:25 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 25 Oct 2010 16:14:25 -0400 Subject: [SciPy-Dev] scipy.stats.distributions: note on initial parameters for fitting the beta distribution In-Reply-To: References: Message-ID: On Mon, Oct 25, 2010 at 3:36 PM, James Phillips wrote: > Here is a link on estimating initial parameters for fitting the beta > distribution. ?I'm posting as it seems a good bit different from the > current calculation in scipy.stats.distribution, and it calculates > only the sample mean and biased sample variance. ?I am not competent > to judge which of the two methods is better for scipy, and hope > someone on the mailing list can do so. > http://www.xycoon.com/beta_parameterestimation.htm This is exactly the same as the `fit` method in the scipy trunk version when loc and scale are fixed. The problem is only if you also want to estimate loc and scale, in that case standard mle doesn't really work. I'm not sure what the estimator in the _fitstart is, looks like matching skew and kurtosis. As Per mentioned in the other thread the best candidates for generic estimation will be other methods, like maximum product spacings (I never remember the official name), or maybe matching moments or similar, which is also one recommendation for extreme value distributions. Are you handling fixed loc and scale in your code? In that case, it might be possible just to restrict the usage of the beta distribution for data between zero and one, or fix loc=x.min()-eps, scale=(x.max() - x.min() + 2*eps) or something like this, if you don't want to get another estimation method. (I don't remember if I tried this for the beta distribution.) Josef > > ? ? James > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From aman.thakral at gmail.com Mon Oct 25 20:55:45 2010 From: aman.thakral at gmail.com (Aman Thakral) Date: Mon, 25 Oct 2010 20:55:45 -0400 Subject: [SciPy-Dev] Partial Least Squares Regression (PLSR) implementation In-Reply-To: References: Message-ID: Hi, I've recently implemented PLSR in Python. Is there currently an implementation in scipy? If not, i'd be happy to share it (and get it reviewed :)) I'm working on including SPE, VIP, and Hotelling's T squared as part of the analysis. Cheers, Aman -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Oct 25 21:01:23 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 25 Oct 2010 19:01:23 -0600 Subject: [SciPy-Dev] Partial Least Squares Regression (PLSR) implementation In-Reply-To: References: Message-ID: On Mon, Oct 25, 2010 at 6:55 PM, Aman Thakral wrote: > Hi, > > I've recently implemented PLSR in Python. Is there currently an > implementation in scipy? If not, i'd be happy to share it (and get it > reviewed :)) > > I'm working on including SPE, VIP, and Hotelling's T squared as part of the > analysis. > > I'd be interested in it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aman.thakral at gmail.com Mon Oct 25 21:36:24 2010 From: aman.thakral at gmail.com (Aman Thakral) Date: Mon, 25 Oct 2010 21:36:24 -0400 Subject: [SciPy-Dev] Partial Least Squares Regression (PLSR) implementation In-Reply-To: References: Message-ID: Great. What is the best way to share it? Scikits? Create something in Git Hub/PyPI? On Mon, Oct 25, 2010 at 9:01 PM, Charles R Harris wrote: > > > On Mon, Oct 25, 2010 at 6:55 PM, Aman Thakral wrote: > >> Hi, >> >> I've recently implemented PLSR in Python. Is there currently an >> implementation in scipy? If not, i'd be happy to share it (and get it >> reviewed :)) >> >> I'm working on including SPE, VIP, and Hotelling's T squared as part of >> the analysis. >> >> > I'd be interested in it. > > Chuck > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Oct 25 22:24:48 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 25 Oct 2010 22:24:48 -0400 Subject: [SciPy-Dev] Partial Least Squares Regression (PLSR) implementation In-Reply-To: References: Message-ID: On Mon, Oct 25, 2010 at 9:36 PM, Aman Thakral wrote: > Great.? What is the best way to share it? Scikits? Create something in Git > Hub/PyPI? Putting it in a public code repository for whatever is your favorite version control system would be a good first step to share the code. A pypi package or a scikits would be good depending on your plans for it. Josef > > On Mon, Oct 25, 2010 at 9:01 PM, Charles R Harris > wrote: >> >> >> On Mon, Oct 25, 2010 at 6:55 PM, Aman Thakral >> wrote: >>> >>> Hi, >>> >>> I've recently implemented PLSR in Python. Is there currently an >>> implementation in scipy? If not, i'd be happy to share it (and get it >>> reviewed :)) >>> >>> I'm working on including SPE, VIP, and Hotelling's T squared as part of >>> the analysis. >>> >> >> I'd be interested in it. >> >> Chuck >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > From gael.varoquaux at normalesup.org Tue Oct 26 01:23:57 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 26 Oct 2010 07:23:57 +0200 Subject: [SciPy-Dev] Partial Least Squares Regression (PLSR) implementation In-Reply-To: References: Message-ID: <20101026052357.GA27034@phare.normalesup.org> On Mon, Oct 25, 2010 at 10:24:48PM -0400, josef.pktd at gmail.com wrote: > Putting it in a public code repository for whatever is your favorite > version control system would be a good first step to share the code. A > pypi package or a scikits would be good depending on your plans for > it. It mights be good to have it in statsmodel, which is the reference scikit for statistical model :) Ga?l From zunzun at zunzun.com Tue Oct 26 05:37:07 2010 From: zunzun at zunzun.com (James Phillips) Date: Tue, 26 Oct 2010 04:37:07 -0500 Subject: [SciPy-Dev] scipy.stats.distributions: note on initial parameters for fitting the beta distribution In-Reply-To: References: Message-ID: Thank you, this is helpful to me. James On Mon, Oct 25, 2010 at 3:14 PM, wrote: > This is exactly the same as the `fit` method in the scipy trunk... From josef.pktd at gmail.com Tue Oct 26 10:21:12 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 26 Oct 2010 10:21:12 -0400 Subject: [SciPy-Dev] Partial Least Squares Regression (PLSR) implementation In-Reply-To: <20101026052357.GA27034@phare.normalesup.org> References: <20101026052357.GA27034@phare.normalesup.org> Message-ID: On Tue, Oct 26, 2010 at 1:23 AM, Gael Varoquaux wrote: > On Mon, Oct 25, 2010 at 10:24:48PM -0400, josef.pktd at gmail.com wrote: >> Putting it in a public code repository for whatever is your favorite >> version control system would be a good first step to share the code. A >> pypi package or a scikits would be good depending on your plans for >> it. > > It mights be good to have it in statsmodel, which is the reference scikit > for statistical model :) That was my first thought, and I would like to get this for statsmodels. However, I'm not sure there is an easy way to integrate this right now, and Skipper is busy with time series analysis and work, and I have still a big backlog of things to get out of the sandbox. It would be an extension, both to the support for control charts and to regression with dimension reduction. I looked a bit at the control chart literature when I was working on structural break tests (e.g cusum test), and other stats packages have support for this that we could or should also provide. For regression with dimension reduction, we only have a principal component regression example in the sandbox, but we haven't discussed yet how it will fit in. One option would be to add it to statsmodels.miscmodels which is for models that look good (tested for standard usage cases), but don't necessarily fit in yet with the rest. If Aman sees a way to fit it into statsmodels, then I would be happy to see something like this included, but otherwise I would prefer to postpone this until we have cleaned up the current code sufficiently for a new release. Josef > > Ga?l > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From aman.thakral at gmail.com Tue Oct 26 11:54:01 2010 From: aman.thakral at gmail.com (Aman Thakral) Date: Tue, 26 Oct 2010 11:54:01 -0400 Subject: [SciPy-Dev] Partial Least Squares Regression (PLSR) implementation In-Reply-To: References: <20101026052357.GA27034@phare.normalesup.org> Message-ID: I was browsing the scikits repo (http://projects.scipy.org/scikits/browser) but didn't see the statsmodel in /trunk. Just let me know how/when I should pass along the code. Aman On Tue, Oct 26, 2010 at 10:21 AM, wrote: > On Tue, Oct 26, 2010 at 1:23 AM, Gael Varoquaux > wrote: > > On Mon, Oct 25, 2010 at 10:24:48PM -0400, josef.pktd at gmail.com wrote: > >> Putting it in a public code repository for whatever is your favorite > >> version control system would be a good first step to share the code. A > >> pypi package or a scikits would be good depending on your plans for > >> it. > > > > It mights be good to have it in statsmodel, which is the reference scikit > > for statistical model :) > > That was my first thought, and I would like to get this for statsmodels. > However, I'm not sure there is an easy way to integrate this right > now, and Skipper is busy with time series analysis and work, and I > have still a big backlog of things to get out of the sandbox. > > It would be an extension, both to the support for control charts and > to regression with dimension reduction. > I looked a bit at the control chart literature when I was working on > structural break tests (e.g cusum test), and other stats packages have > support for this that we could or should also provide. > For regression with dimension reduction, we only have a principal > component regression example in the sandbox, but we haven't discussed > yet how it will fit in. > > One option would be to add it to statsmodels.miscmodels which is for > models that look good (tested for standard usage cases), but don't > necessarily fit in yet with the rest. > > If Aman sees a way to fit it into statsmodels, then I would be happy > to see something like this included, but otherwise I would prefer to > postpone this until we have cleaned up the current code sufficiently > for a new release. > > Josef > > > > > Ga?l > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -- Aman Thakral B.Eng & Biosci, M.Eng Design -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Oct 26 12:17:11 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 26 Oct 2010 12:17:11 -0400 Subject: [SciPy-Dev] Partial Least Squares Regression (PLSR) implementation In-Reply-To: References: <20101026052357.GA27034@phare.normalesup.org> Message-ID: On Tue, Oct 26, 2010 at 11:54 AM, Aman Thakral wrote: > I was browsing the scikits repo (http://projects.scipy.org/scikits/browser) > but didn't see the statsmodel in /trunk.? Just let me know how/when I should > pass along the code. here are the links for scikits.statsmodels http://pypi.python.org/pypi/scikits.statsmodels http://groups.google.com/group/pystatsmodels?hl=en https://code.launchpad.net/statsmodels Here is the list of scikits: http://scikits.appspot.com/scikits The actual source of the scikits is pretty spread out across, scipy svn, github, launchpad, ... Josef > > Aman > > On Tue, Oct 26, 2010 at 10:21 AM, wrote: >> >> On Tue, Oct 26, 2010 at 1:23 AM, Gael Varoquaux >> wrote: >> > On Mon, Oct 25, 2010 at 10:24:48PM -0400, josef.pktd at gmail.com wrote: >> >> Putting it in a public code repository for whatever is your favorite >> >> version control system would be a good first step to share the code. A >> >> pypi package or a scikits would be good depending on your plans for >> >> it. >> > >> > It mights be good to have it in statsmodel, which is the reference >> > scikit >> > for statistical model :) >> >> That was my first thought, and I would like to get this for statsmodels. >> However, I'm not sure there is an easy way to integrate this right >> now, and Skipper is busy with time series analysis and work, and I >> have still a big backlog of things to get out of the sandbox. >> >> It would be an extension, both to the support for control charts and >> to regression with dimension reduction. >> I looked a bit at the control chart literature when I was working on >> structural break tests (e.g cusum test), and other stats packages have >> support for this that we could or should also provide. >> For regression with dimension reduction, we only have a principal >> component regression example in the sandbox, but we haven't discussed >> yet how it will fit in. >> >> One option would be to add it to statsmodels.miscmodels which is for >> models that look good (tested for standard usage cases), but don't >> necessarily fit in yet with the rest. >> >> If Aman sees a way to fit it into statsmodels, then I would be happy >> to see something like this included, but otherwise I would prefer to >> postpone this until we have cleaned up the current code sufficiently >> for a new release. >> >> Josef >> >> > >> > Ga?l >> > _______________________________________________ >> > SciPy-Dev mailing list >> > SciPy-Dev at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-dev >> > >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > -- > Aman Thakral > B.Eng & Biosci, M.Eng Design > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > From zunzun at zunzun.com Sat Oct 30 12:37:35 2010 From: zunzun at zunzun.com (James Phillips) Date: Sat, 30 Oct 2010 11:37:35 -0500 Subject: [SciPy-Dev] scipy.stats.distributions: note on initial parameters for fitting the beta distribution In-Reply-To: References: Message-ID: Attached is a version of your file "matchdist.py" that is working well in my tests. It tests a half-dozen-ish different starting parameters for each continuous distribution - although this is often redundant - and uses the best fit from them based on the ks_pval statistic. It seems to run fast enough for use on my web site if I do not use the distributions ncf or kstwobign, so those are excluded in the file. I'll parallelize this code and make a few more tweaks, and then add it to my web site. I am concerned about the large number of warnings from scipy since they will fill up the web server error logs, but my understanding is that those warning messages can be suppressed or at least not sent to stderr. James > On Mon, Oct 25, 2010 at 3:14 PM, ? wrote: >> This is exactly the same as the `fit` method in the scipy trunk... -------------- next part -------------- A non-text attachment was scrubbed... Name: matchdist.py Type: text/x-python Size: 9895 bytes Desc: not available URL: From zunzun at zunzun.com Sat Oct 30 12:44:18 2010 From: zunzun at zunzun.com (James Phillips) Date: Sat, 30 Oct 2010 11:44:18 -0500 Subject: [SciPy-Dev] scipy.stats.distributions: note on initial parameters for fitting the beta distribution In-Reply-To: References: Message-ID: My apologies, that is an old version of the file. Correct version attached. James 2010/10/30 James Phillips : > Attached is a version of your file "matchdist.py" that is working well > in my tests. -------------- next part -------------- A non-text attachment was scrubbed... Name: matchdist.py Type: text/x-python Size: 9829 bytes Desc: not available URL: From pav at iki.fi Sat Oct 30 13:49:56 2010 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 30 Oct 2010 17:49:56 +0000 (UTC) Subject: [SciPy-Dev] Google custom search for ask.scipy.org? Message-ID: Hi, A current (killer) problem with ask.scipy.org is that it does not have search. As a band-aid solution, it should be possible to add a Google custom search box, like on this page: http://scikit-learn.sourceforge.net/ Google searches "site:ask.scipy.org/en/topic KEYWORDS" work well, so that a custom search should also work OK. -- Pauli Virtanen From robert.kern at gmail.com Sat Oct 30 22:48:28 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 30 Oct 2010 21:48:28 -0500 Subject: [SciPy-Dev] Google custom search for ask.scipy.org? In-Reply-To: References: Message-ID: On Sat, Oct 30, 2010 at 12:49, Pauli Virtanen wrote: > Hi, > > A current (killer) problem with ask.scipy.org is that it does not have > search. As a band-aid solution, it should be possible to add a Google > custom search box, like on this page: http://scikit-learn.sourceforge.net/ > > Google searches "site:ask.scipy.org/en/topic KEYWORDS" work well, so that > a custom search should also work OK. Patches welcome. http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/solace -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From zunzun at zunzun.com Sun Oct 31 09:33:33 2010 From: zunzun at zunzun.com (James Phillips) Date: Sun, 31 Oct 2010 08:33:33 -0500 Subject: [SciPy-Dev] scipy.stats.distributions: note on initial parameters for fitting the beta distribution In-Reply-To: References: Message-ID: Here is a more polished and quite smaller version of your example file matchdist.py that uses either nnlf or residuals for ranking, and includes checks for NaN, +inf and -inf. I think this has all of the logic and range checks that it needs. James 2010/10/30 James Phillips : > I'll parallelize this code and make a few more tweaks, and then add it > to my web site. From zunzun at zunzun.com Sun Oct 31 09:34:04 2010 From: zunzun at zunzun.com (James Phillips) Date: Sun, 31 Oct 2010 08:34:04 -0500 Subject: [SciPy-Dev] scipy.stats.distributions: note on initial parameters for fitting the beta distribution In-Reply-To: References: Message-ID: File attached. On Sun, Oct 31, 2010 at 8:33 AM, James Phillips wrote: > Here is a more polished and quite smaller version of your example file > matchdist.py that uses either nnlf or residuals for ranking, and > includes checks for NaN, +inf and -inf. ?I think this has all of the > logic and range checks that it needs. > > ? ? James > > 2010/10/30 James Phillips : >> I'll parallelize this code and make a few more tweaks, and then add it >> to my web site. > -------------- next part -------------- A non-text attachment was scrubbed... Name: matchdist_nnlf.py Type: text/x-python Size: 7962 bytes Desc: not available URL: