From mforbes at physics.ubc.ca Sat Mar 1 05:46:02 2008 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Sat, 1 Mar 2008 19:46:02 +0900 Subject: [SciPy-dev] scipy trac ticket #600 UnivariateSpline error. Message-ID: Hi, Could someone please reclassify ticket #600: I did not classify it properly when I submitted it and now I can't figure out how to reclassify it. It has a small patch to ensure that spline evaluation preserves the shape of the input array: The following demonstrates the problem: >>> from scipy.interpolate import UnivariateSpline >>> from numpy import shape >>> f = UnivariateSpline([0,1,2,3],[0,1,2,3]) >>> shape(4) () >>> shape(f(4)) (1,) Thanks, Michael. From mforbes at physics.ubc.ca Sat Mar 1 09:53:09 2008 From: mforbes at physics.ubc.ca (Michael McNeil Forbes) Date: Sat, 1 Mar 2008 23:53:09 +0900 Subject: [SciPy-dev] Subtle fitpack2 bug: resize (but only in doctest) Message-ID: <716F74E4-5C0E-4D31-8A36-D08CAB9DB221@physics.ubc.ca> Hi, I have been running into a very strange and hard to track down bug with fitpack2.py. When I run certain tests as doctests, then I get the following error: File "/data/apps/Python-2.5.1/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/ interpolate/fitpack2.py", line 91, in __init__ data = self._reset_nest(data) File "/data/apps/Python-2.5.1/Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/scipy/ interpolate/fitpack2.py", line 128, in _reset_nest t.resize(nest) ValueError: cannot resize an array that has been referenced or is referencing another array in this way. Use the resize function I am doing a simple 1d interpolation, but the tricky part is that this only occurs when I am running under the doctest module: if I import the file at an interpreter prompt and then run the doctests, there is no problem. The problem is at line 128 in fitpack2.py which occurs in this context: 128 t,c,fpint,nrdata = data[8].copy(),data[9].copy(),\ 129 data[11].copy(),data[12].copy() 130 t.resize(nest) 131 c.resize(nest) 132 fpint.resize(nest) 133 nrdata.resize(nest) ... Thus, t should be a fresh copy and should be resizable, but somehow, when running the code under a doctest, there is a reference created that messes up the call to resize. I have no idea what is causing this problem - it also seems to be very fickle: I can cause the problem on my macbook pro (Core Duo) with the following program: err.py: """ >>> x = [-1.,-0.65016502,-0.58856235,-0.26903553,-0.17370892, ... -0.10011001,0.,0.10011001,0.17370892,0.26903553,0.58856235, ... 0.65016502,1.] >>> y = [ 1.,0.62928599,0.5797223, 0.39965815,0.36322694,0.3508061, ... 0.35214793,0.3508061, 0.36322694,0.39965815,0.5797223, ... 0.62928599,1.] >>> w = [1.00000000e+12, 6.88875973e+02, 4.89314737e+02, 4.26864807e +02, ... 6.07746770e+02, 4.51341444e+02, 3.17480210e+02, 4.51341444e+02, ... 6.07746770e+02, 4.26864807e+02, 4.89314737e+02, 6.88875973e+02, ... 1.00000000e+12] >>> from scipy.interpolate import UnivariateSpline >>> sp = UnivariateSpline(x=x,y=y,w=w,s=None) """ import doctest doctest.testmod() But when I run it on my linux box (64 bit), it runs fine. In any case, there is an easy patch. Just replace the lines above with 128 t,c,fpint,nrdata = [resize(data[n],nest) for n in 129 [8,9,11,12]] and add resize to the numpy imports 21 from numpy import ... resize The bug is so strange, though, that I though it should be investigated a bit more. Any ideas what could be causing this? Michael. From ggellner at uoguelph.ca Sat Mar 1 14:17:09 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 01 Mar 2008 12:17:09 -0700 Subject: [SciPy-dev] ANN: Enthought Python Distribution - Beta In-Reply-To: <47C7AB02.1010107@enthought.com> References: <733AFCE4-50FA-4A79-B032-B0965F75E434@enthought.com> <5b9ba9310802272027u689d755by4af6ca07b4b20baa@mail.gmail.com> <3d375d730802272118t26ea170cr8a63023d4a5243cb@mail.gmail.com> <5b9ba9310802281021v2478dd59ycaabe21bb347c5ed@mail.gmail.com> <3d375d730802281124v4246fff7o7e6d02dd83d770f5@mail.gmail.com> <1e2af89e0802281203x32c3e2f8v5367bd1383dfbdcc@mail.gmail.com> <3d375d730802281210l65138b4fi8f2c51915e56093c@mail.gmail.com> <1e2af89e0802281215n54ec1711ue7881257cb4159fe@mail.gmail.com> <1e2af89e0802281227t34edb062ub440c898604cb66f@mail.gmail.com> <47C7AB02.1010107@enthought.com> Message-ID: <20080301191709.GA6408@giton> Oh man this is good for me. I have had real problems convincing some users to try python because it is 'free'. Having a support version that I can let them use that is compatible with my open source version, is a life saver. Does it allow for f2py to compile without setting up the compilers myself in windows. I often use this to make callback functions, and it will help make me convince some people. Gabriel On Fri, Feb 29, 2008 at 12:49:38AM -0600, Travis E. Oliphant wrote: > Matthew Brett wrote: > >> With the bundle - yes of course. I mean, if the bundle becomes the > >> defacto standard for deploying python, numpy etc, and the charges > >> worry people, then they may be less likely to try it. I've no idea > >> whether that will be a big factor - I guess it's difficult to say. > >> > > > > For example - imagine the following scenario. Someone moderately > > junior has just joined a company or government institution. They > > become keen on python, start playing with python, VTK and so on, and > > try and persuade others in the company to try it. If they find they > > have to ask the accounts people for money before they install the > > bundle, or they have to build it all from scratch, they may not, which > > means lost users. > > > > We try to advertise and make very clear that there is no charge at all > even for commercial users to just try EPD. That same moderately junior > person could get EPD from the internet, install it and show all their > friends. > > The only thing we ask is that if a company starts making substantial use > of EPD, they help defer the costs of producing it. We are really very > keen on having as many people as possible use the tools. > > We have had requests for a supported binary installation for a long > time. So, we are responding. > > Best regards, > > > -Travis O. > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev From ggellner at uoguelph.ca Sat Mar 1 15:20:55 2008 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sat, 01 Mar 2008 13:20:55 -0700 Subject: [SciPy-dev] ctypes callbacks, changing signature Message-ID: <20080301202055.GA14001@giton> I am wrapping some ode solvers using ctypes. And I have run into, what I imagine is, a common problem. The expected C callback is of the form void func(int n, double t, double* y, double* yprime); which I can wrap fine using ctypes. But I would like to be able to use a python (or f2py generated) function of the normal form def func(t, y): # some callculations return yprime # instead of def func(n, t, y, prime): return None My question is what do people usually do in this case? Any tips suggestions, examples. Thanks a bunch. Gabriel From robert.kern at gmail.com Sat Mar 1 15:26:52 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 1 Mar 2008 14:26:52 -0600 Subject: [SciPy-dev] ctypes callbacks, changing signature In-Reply-To: <20080301202055.GA14001@giton> References: <20080301202055.GA14001@giton> Message-ID: <3d375d730803011226j6f8e7cb9q9f25e129de0d966d@mail.gmail.com> On Sat, Mar 1, 2008 at 2:20 PM, Gabriel Gellner wrote: > I am wrapping some ode solvers using ctypes. And I have run into, what I > imagine is, a common problem. > > The expected C callback is of the form > > void func(int n, double t, double* y, double* yprime); > > which I can wrap fine using ctypes. But I would like to be able to use a > python (or f2py generated) function of the normal form > > def func(t, y): > # some callculations > return yprime > > # instead of > def func(n, t, y, prime): > return None > > My question is what do people usually do in this case? Any tips suggestions, > examples. Wrap it with a function of the required signature. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Mar 1 17:21:10 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 1 Mar 2008 16:21:10 -0600 Subject: [SciPy-dev] ANN: Enthought Python Distribution - Beta In-Reply-To: <20080301191709.GA6408@giton> References: <733AFCE4-50FA-4A79-B032-B0965F75E434@enthought.com> <3d375d730802272118t26ea170cr8a63023d4a5243cb@mail.gmail.com> <5b9ba9310802281021v2478dd59ycaabe21bb347c5ed@mail.gmail.com> <3d375d730802281124v4246fff7o7e6d02dd83d770f5@mail.gmail.com> <1e2af89e0802281203x32c3e2f8v5367bd1383dfbdcc@mail.gmail.com> <3d375d730802281210l65138b4fi8f2c51915e56093c@mail.gmail.com> <1e2af89e0802281215n54ec1711ue7881257cb4159fe@mail.gmail.com> <1e2af89e0802281227t34edb062ub440c898604cb66f@mail.gmail.com> <47C7AB02.1010107@enthought.com> <20080301191709.GA6408@giton> Message-ID: <3d375d730803011421l520c547eyc218157b00278840@mail.gmail.com> On Sat, Mar 1, 2008 at 1:17 PM, Gabriel Gellner wrote: > Oh man this is good for me. I have had real problems convincing some users to > try python because it is 'free'. Having a support version that I can let them > use that is compatible with my open source version, is a life saver. > > Does it allow for f2py to compile without setting up the compilers myself in > windows. I often use this to make callback functions, and it will help make me > convince some people. g77 is provided along with mingw. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lpc at cmu.edu Sun Mar 2 15:12:15 2008 From: lpc at cmu.edu (Luis Pedro Coelho) Date: Sun, 2 Mar 2008 15:12:15 -0500 Subject: [SciPy-dev] [PATCH] scipy.stats.entropy documentation should mention log base Message-ID: <200803021512.15665.lpc@cmu.edu> Hello, It took me a while to figure out that stats.entropy does it logarithms base e. I expected it to work base 2 (from where I'm coming from, that's the traditional definition of entropy, but maybe other communities have other expectations). The attached patch adds a sentence to the documentation to that effect. It also adds an entropy2() function which works base 2. regards, -- Lu?s Pedro Coelho PhD Student in Computational Biology, CMU http://www.luispedro.org -------------- next part -------------- A non-text attachment was scrubbed... Name: entropy.patch Type: text/x-diff Size: 1464 bytes Desc: not available URL: From pearu at cens.ioc.ee Mon Mar 3 05:55:56 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon, 03 Mar 2008 11:55:56 +0100 Subject: [SciPy-dev] f2py + f90 file In-Reply-To: <47C7F659.1040804@scipy.org> References: <47BDC491.8010501@scipy.org> <62589.85.166.22.159.1203619400.squirrel@cens.ioc.ee> <47C3DE2B.8000201@scipy.org> <56269.85.166.22.159.1204019574.squirrel@cens.ioc.ee> <47C3E386.2060209@scipy.org> <56697.85.166.22.159.1204021019.squirrel@cens.ioc.ee> <47C3ED6A.7000200@scipy.org> <57377.85.166.22.159.1204024030.squirrel@cens.ioc.ee> <47C7F659.1040804@scipy.org> Message-ID: <47CBD93C.1030900@cens.ioc.ee> dmitrey wrote: > Pearu, could you take a look at the toms/587 routine (another one > constrained LLS)? > I guess it could be connected to scipy. > f2py yields lots of bugs when I try to build .so-file. > (I have downloaded the one with option "include (unoptimized) BLAS > routines" turned on, tgz file ~48 kb with a single file inside 587 that > I have renamed to toms587.f). > > http://netlib.org/cgi-bin/netlibfiles.pl?filename=/toms/587 > Here's full output below. (Unfortunately I'm not skilled in Fortran) The file has some non-Fortran footer. I cleaned it up and then it compiles with g77. You can grab the file from http://cens.ioc.ee/~pearu/toms587.f HTH, Pearu From mhaligowski at googlemail.com Mon Mar 3 11:19:10 2008 From: mhaligowski at googlemail.com (halish) Date: Mon, 3 Mar 2008 17:19:10 +0100 Subject: [SciPy-dev] Google Summer of Code 2008 and SciPy Message-ID: Hello, my name is Mateusz Haligowski and I am a student of the third (and the last) year of Baccalaureate studies at the University of Gda?sk. I study Econometrics and Statistics. Last year I participated in GSoC, writing a plugin for Scribus (DTP software). It was a great fun, but this year I'm longing to write something connected with my studies. I would like to ask, whether SciPy is planning to participate as a mentoring organisation in GSoC this year? I've received a message on python-gsoc mailing list, that you would need some econometric functions, and that would be a perfect thing for me. The registration process for mentoring organisations started today, ie. Monday the 3rd of March. It will last until 12th March. The details are available on http://code.google.com/soc/faqs.html. I'm really encouraging SciPy to participate in GSoC. Regards, Mateusz Haligowski (I try to stay at #scipy recently as halish, if you would like to talk to me) From russel at appliedminds.com Wed Mar 5 00:51:02 2008 From: russel at appliedminds.com (Russel Howe) Date: Tue, 04 Mar 2008 21:51:02 -0800 Subject: [SciPy-dev] Help with f2py Message-ID: <47CE34C6.5050304@appliedminds.com> I am attempting to add support for lsodi from odepack to scipy.integrate.ode. I have almost succeeded, but to be complete I would like to be able to recover values of ydot during integration. The documentation for lsodi recommends a call to intdy to return these values. How do I write a pyf file for this function? From the lsodi documentation: c call intdy (t, k, rwork(21), nyh, dky, iflag) c c the input parameters are.. c c t = value of independent variable where answers are desired c (normally the same as the t last returned by lsodi). c for valid results, t must lie between tcur - hu and tcur. c (see optional outputs for tcur and hu.) c k = integer order of the derivative desired. k must satisfy c 0 .le. k .le. nqcur, where nqcur is the current order c (see optional outputs). the capability corresponding c to k = 0, i.e. computing y(t), is already provided c by lsodi directly. since nqcur .ge. 1, the first c derivative dy/dt is always available with intdy. c rwork(21) = the base address of the history array yh. c nyh = column length of yh, equal to the initial value of neq. t, k, and nyh are simple but how do I pass the base address of an array? Further digging shows the the yh array has dimensions (neq, nqcur) both of which are accessible at call time but unknown at compile time. I use this information to construct an apropriate 2-d array to pass to intdy, but I don't see how to pass the full dimensions to f2py. Attached is my current attempt, plus a small test program. Russel -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: lsodi.patch URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: test.py URL: From pearu at cens.ioc.ee Wed Mar 5 09:28:55 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Wed, 5 Mar 2008 16:28:55 +0200 (EET) Subject: [SciPy-dev] Help with f2py In-Reply-To: <47CE34C6.5050304@appliedminds.com> References: <47CE34C6.5050304@appliedminds.com> Message-ID: <55474.129.240.228.26.1204727335.squirrel@cens.ioc.ee> On Wed, March 5, 2008 7:51 am, Russel Howe wrote: > I am attempting to add support for lsodi from odepack to > scipy.integrate.ode. I have almost succeeded, but to be complete I > would like to be able to recover values of ydot during integration. The > documentation for lsodi recommends a call to intdy to return these > values. How do I write a pyf file for this function? > > From the lsodi documentation: > > c call intdy (t, k, rwork(21), nyh, dky, iflag) > c > c the input parameters are.. > c > c t = value of independent variable where answers are desired > c (normally the same as the t last returned by lsodi). > c for valid results, t must lie between tcur - hu and tcur. > c (see optional outputs for tcur and hu.) > c k = integer order of the derivative desired. k must satisfy > c 0 .le. k .le. nqcur, where nqcur is the current order > c (see optional outputs). the capability corresponding > c to k = 0, i.e. computing y(t), is already provided > c by lsodi directly. since nqcur .ge. 1, the first > c derivative dy/dt is always available with intdy. > c rwork(21) = the base address of the history array yh. > c nyh = column length of yh, equal to the initial value of neq. > > > t, k, and nyh are simple but how do I pass the base address of an array? For example, use rwork[21-1:] where rwork is the array passed on to lsodi rwork argument. Make sure that no copies are made to rwork when calling lsodi (use contiguous array with proper type). > Further digging shows the the yh array has dimensions (neq, nqcur) > both of which are accessible at call time but unknown at compile time. Then in the wrapper of intdy use `*` for yh dimensions. > I use this information to construct an apropriate 2-d array to pass to > intdy, but I don't see how to pass the full dimensions to f2py. What do you mean? HTH, Pearu From russel at appliedminds.com Wed Mar 5 10:02:18 2008 From: russel at appliedminds.com (Russel Howe) Date: Wed, 05 Mar 2008 07:02:18 -0800 Subject: [SciPy-dev] Help with f2py In-Reply-To: <55474.129.240.228.26.1204727335.squirrel@cens.ioc.ee> References: <47CE34C6.5050304@appliedminds.com> <55474.129.240.228.26.1204727335.squirrel@cens.ioc.ee> Message-ID: <47CEB5FA.1060203@appliedminds.com> > > Then in the wrapper of intdy use `*` for yh dimensions. > What does * mean? Use the dimensions of the passed array? >> I use this information to construct an apropriate 2-d array to pass to >> intdy, but I don't see how to pass the full dimensions to f2py. > > What do you mean? > I mean I missed the easy way to solve the problem. Your email managed to straighten me out. I created a ticket here http://www.scipy.org/scipy/scipy/ticket/615 with the resulting patch, thanks! Russel From aisaac at american.edu Wed Mar 5 11:00:24 2008 From: aisaac at american.edu (Alan Isaac) Date: Wed, 5 Mar 2008 11:00:24 -0500 Subject: [SciPy-dev] bvls license In-Reply-To: <47C52773.2000602@scipy.org> References: <47C52773.2000602@scipy.org> Message-ID: On Wed, 27 Feb 2008, dmitrey wrote: > http://lib.stat.cmu.edu/general/bvls OK, here is the situation. It turns out that BVLS was previously released under GPL v.2 to the R project. The authors are happy to have it available under that license. However, they also appear willing to consider LGPL and possibly BSD if that would be helpful. I would need to be able to tell them how it would be helpful. Could someone better versed in SciPy/licenses interactions help here? This is what I think. GPL code can be used in SciKits but not in SciPy proper. If so, is it desirable/important that this code be available in SciPy proper? Finally, one author added the following statement: "I would be happy to license it under anything that keeps attribution to Bob and me and that keeps the code intact." I do not really know what this means. The GPL itself makes no commitment to keep the code "intact", right? Can I address this concern in any way? Thank you, Alan Isaac From robert.kern at gmail.com Wed Mar 5 14:00:04 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 5 Mar 2008 13:00:04 -0600 Subject: [SciPy-dev] bvls license In-Reply-To: References: <47C52773.2000602@scipy.org> Message-ID: <3d375d730803051100i74160914y51b39e6b27ec5833@mail.gmail.com> On Wed, Mar 5, 2008 at 10:00 AM, Alan Isaac wrote: > On Wed, 27 Feb 2008, dmitrey wrote: > > http://lib.stat.cmu.edu/general/bvls > > OK, here is the situation. It turns out that > BVLS was previously released under GPL v.2 to the R project. > The authors are happy to have it available under that license. > > However, they also appear willing to consider LGPL and > possibly BSD if that would be helpful. I would need to be > able to tell them how it would be helpful. Could someone > better versed in SciPy/licenses interactions help here? > > This is what I think. > > GPL code can be used in SciKits but not in SciPy proper. > If so, is it desirable/important that this code be available in SciPy proper? Sure. Even if it is just used in OpenOpt, OpenOpt would also benefit from it being BSD-licensed. > Finally, one author added the following statement: > > "I would be happy to license it under anything that > keeps attribution to Bob and me and that keeps the code > intact." > > I do not really know what this means. The GPL itself makes > no commitment to keep the code "intact", right? Can > I address this concern in any way? You will have to ask them exactly what they mean by that. Inform them that both the GPL and BSD licenses require attribution, but both also allow licensees to distribute modified code. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From russel at appliedminds.com Wed Mar 5 16:23:22 2008 From: russel at appliedminds.com (Russel Howe) Date: Wed, 05 Mar 2008 13:23:22 -0800 Subject: [SciPy-dev] lsodi - ticket 615 In-Reply-To: <47CEB5FA.1060203@appliedminds.com> References: <47CE34C6.5050304@appliedminds.com> <55474.129.240.228.26.1204727335.squirrel@cens.ioc.ee> <47CEB5FA.1060203@appliedminds.com> Message-ID: <47CF0F4A.10500@appliedminds.com> I have an updated patch. I found a small problem with my first patch and added the ability to integrate to critical points of the function (itask=4). Unfortunately, the trac wiki will not allow me to append a new file to the ticket as I do not have permissions. I have attached the new patch here in hopes that somebody with appropriate permissions can add it to the ticket. Russel -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: lsodi.patch URL: From lopmart at gmail.com Wed Mar 5 17:01:19 2008 From: lopmart at gmail.com (jose luis Lopez Martinez) Date: Wed, 5 Mar 2008 14:01:19 -0800 Subject: [SciPy-dev] solution sparse matrix Message-ID: <4eeef9d40803051401i19ec36b6n8b5bab2467e3ec8@mail.gmail.com> hi i am working in my thesis of master and i will work with sparse matrix ( i.e 128^4), so my question is if exist a command in scipy that solve sparse matrix using at the Gauss-Seidel iteration method? thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Wed Mar 5 17:15:47 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 5 Mar 2008 19:15:47 -0300 Subject: [SciPy-dev] solution sparse matrix In-Reply-To: <4eeef9d40803051401i19ec36b6n8b5bab2467e3ec8@mail.gmail.com> References: <4eeef9d40803051401i19ec36b6n8b5bab2467e3ec8@mail.gmail.com> Message-ID: On 3/5/08, jose luis Lopez Martinez wrote: > i am working in my thesis of master and i will work with sparse matrix > (i.e 128^4), What is 128^4 (=268,435,456) ? The row and column size? Or the number of nonzero entries? > so my question is > if exist a command in scipy that solve sparse matrix using at the > Gauss-Seidel iteration method? That is usually very slow and may even not converge. Have you considered using any Krylov iterative method?. > > thanks > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From aisaac at american.edu Wed Mar 5 22:06:34 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 5 Mar 2008 22:06:34 -0500 Subject: [SciPy-dev] Fwd: BVLS license Message-ID: The authors have agreed to release BVLS as BSD to match the SciPy license. (Email attached as documentation.) Cheers, Alan Isaac -------------- next part -------------- An embedded message was scrubbed... From: "Philip B. Stark" Subject: Re: Re[4]: BVLS license Date: Wed, 5 Mar 2008 12:58:41 -0800 Size: 4301 URL: -------------- next part -------------- I've heard from Bob. We are happy to release the code under BSD. Thanks again for your interest! Best wishes, Philip On Wed, Mar 5, 2008 at 12:28 PM, Alan G Isaac wrote: > On Wed, 5 Mar 2008, "Philip B. Stark" apparently wrote: > > I would be happy to license it under anything that keeps > > attribution to Bob and me and that keeps the code intact. > > Both the GPL and BSD licenses require attribution, > but both also allow licensees to distribute modified code. > So this part does not change relative to the GPL. > > > > But we need to wait for Bob to reply if you want something > > less restrictive than GPL v2. > > OK. Yes, I'd like the code to have the potential for being > distributed with SciPy proper, which would mean a BSD > license (to match the SciPy license > ). > Thanks for your willingness to consider a less restrictive license! > > Cheers, > Alan > > > > > -- Philip B. Stark | Professor of Statistics | University of California Berkeley, CA 94720-3860 | 510-642-1430 | statistics.berkeley.edu/~stark From dmitrey.kroshko at scipy.org Thu Mar 6 03:02:47 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Thu, 06 Mar 2008 10:02:47 +0200 Subject: [SciPy-dev] some Fortran code changes are required for bvls (Was: Fwd: BVLS license) In-Reply-To: References: Message-ID: <47CFA527.2040700@scipy.org> BSD for bvls is a good news, on the other hand, toms587->Python connection will be ready soon, and AFAIK it has BSD license as well. Also, toms587 routine seems to be more powerful. Unfortunately, bvls is not capable of solving sparse problems (I don't know about toms587 yet), and regularization parameters should be implemented by user directly, via modifying C and d matrices. I had sent a letter with some questions to bvls authors (some days ago), unfortunately, they had no answered. One of issues is a small modification to bvls (fortran) code, that is certainly required before connecting bvls to openopt and/or scipy. Line 464 reports print *,' BVLS fails to converge! ' We should handle the situation automatically (to be able to take profits by scikits.openopt solver(s)), i.e. implement one more return value, for example, istop<0 if the routine have reached the line or one of other "print" lines; and istop=+1 if all ok (i.e. not reached the lines). Also, the error messages could be assigned to one more string output parameter, for example "msg", and msg = 'converged' if all ok. It would be better if bvls authors have implemented the changes in their own code, but since they have no willing to answer, it should be done by ourselves. Is here anyone capable of the issue? I would done it by myself but I'm not skilled in Fortran. Regards, D. Alan G Isaac wrote: > The authors have agreed to release BVLS as BSD to match the > SciPy license. (Email attached as documentation.) > > Cheers, > Alan Isaac > > > > ------------------------------------------------------------------------ > > Subject: > Re: Re[4]: BVLS license > From: > "Philip B. Stark" > Date: > Wed, 5 Mar 2008 12:58:41 -0800 > To: > aisaac at american.edu > > To: > aisaac at american.edu > CC: > "Robert L. Parker" > > > I've heard from Bob. We are happy to release the code under BSD. > > Thanks again for your interest! > > Best wishes, > Philip > > > On Wed, Mar 5, 2008 at 12:28 PM, Alan G Isaac wrote: > >> On Wed, 5 Mar 2008, "Philip B. Stark" apparently wrote: >> > I would be happy to license it under anything that keeps >> > attribution to Bob and me and that keeps the code intact. >> >> Both the GPL and BSD licenses require attribution, >> but both also allow licensees to distribute modified code. >> So this part does not change relative to the GPL. >> >> >> > But we need to wait for Bob to reply if you want something >> > less restrictive than GPL v2. >> >> OK. Yes, I'd like the code to have the potential for being >> distributed with SciPy proper, which would mean a BSD >> license (to match the SciPy license >> ). >> Thanks for your willingness to consider a less restrictive license! >> >> Cheers, >> Alan >> >> >> >> >> >> > > > > > ------------------------------------------------------------------------ > > I've heard from Bob. We are happy to release the code under BSD. > > Thanks again for your interest! > > Best wishes, > Philip > > > On Wed, Mar 5, 2008 at 12:28 PM, Alan G Isaac wrote: > >> On Wed, 5 Mar 2008, "Philip B. Stark" apparently wrote: >> > I would be happy to license it under anything that keeps >> > attribution to Bob and me and that keeps the code intact. >> >> Both the GPL and BSD licenses require attribution, >> but both also allow licensees to distribute modified code. >> So this part does not change relative to the GPL. >> >> >> > But we need to wait for Bob to reply if you want something >> > less restrictive than GPL v2. >> >> OK. Yes, I'd like the code to have the potential for being >> distributed with SciPy proper, which would mean a BSD >> license (to match the SciPy license >> ). >> Thanks for your willingness to consider a less restrictive license! >> >> Cheers, >> Alan >> >> >> >> >> >> > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > From fperez.net at gmail.com Thu Mar 6 13:15:51 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 6 Mar 2008 10:15:51 -0800 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea Message-ID: Hi all, after the Scipy/Sage Days 8 meeting, we were all very impressed by the progress made by Cython. For those not familiar with it, Cython: http://www.cython.org/ is an evolved version of Pyrex (which is used by numpy and scipy) with lots of improvements. We'd like to position Cython as the preferred way of writing most, if not all, new extension code written for numpy and scipy, as it is easier to write, get right, debug (when you still get it wrong) and maintain than writing to the raw Python-C API. A specific project along these lines, that would be very beneficial for numpy could be: - Creating new matrix types in cython that match the cvxopt matrices. The creation of new numpy array types with efficient code would be very useful. - Rewriting the existing ndarray subclasses that ship with numpy, such as record arrays, in cython. In doing this, benchmarks of the relative performance of the new code should be obtained. Another possible project would be the addition to Cython of syntactic support for array expressions, multidimensional indexing, and other features of numpy. This is probably more difficult than the above, as it would require fairly detailed knowledge of both the numpy C API and the Cython internals, but would ultimately be extremely useful. Any student interested in this should quickly respond on the list; such a project would likely be co-mentored by people on the Numpy and Cython teams, since it is likely to require expertise from both ends. Cheers, f From lopmart at gmail.com Thu Mar 6 14:03:32 2008 From: lopmart at gmail.com (jose luis Lopez Martinez) Date: Thu, 6 Mar 2008 11:03:32 -0800 Subject: [SciPy-dev] solution sparse matrix Message-ID: <4eeef9d40803061103s60f18a66u90dbee3031a4d8e7@mail.gmail.com> hi the size of matrix sparse is 128^4(268,435,456 rows)x 128^4(268,435,456 colums). the matrix have diagonald dominancies that is a sufficient condition for convergence of Jacobi iteration , gauss seidel and xor(relaxion method), but is slow for converge. it exist a command in scipy that solve sparse matrix using at the Gauss-Seidel iteration method o krylov iterative methods? thanks for your advisor -------------- next part -------------- An HTML attachment was scrubbed... URL: From dominique.orban at gmail.com Thu Mar 6 15:23:20 2008 From: dominique.orban at gmail.com (Dominique Orban) Date: Thu, 6 Mar 2008 15:23:20 -0500 Subject: [SciPy-dev] solution sparse matrix In-Reply-To: <4eeef9d40803061103s60f18a66u90dbee3031a4d8e7@mail.gmail.com> References: <4eeef9d40803061103s60f18a66u90dbee3031a4d8e7@mail.gmail.com> Message-ID: <8793ae6e0803061223k2fbe4eccm635361055b398e29@mail.gmail.com> On Thu, Mar 6, 2008 at 2:03 PM, jose luis Lopez Martinez wrote: > hi > > the size of matrix sparse is 128^4(268,435,456 rows)x 128^4(268,435,456 > colums). > > the matrix have diagonald dominancies that is a sufficient condition > for convergence of Jacobi iteration , gauss seidel and xor(relaxion method), > but is slow for converge. > > it exist a command in scipy that solve sparse matrix using at the > Gauss-Seidel iteration method o krylov iterative methods? Is your matrix symmetric? If so, try the conjugate gradient (CG) method. It will only require matrix-vector products. You can even get around storing your sparse matrix by writing a function that evaluates the result of a matrix-vector product. This can be interesting if the density is high. If your matrix is not symmetric, try Bi-Cgstab. It also only requires matrix-vector products with the matrix (and not with its transpose). Same comment as above regarding storage. You can try Bi-Cgstab in either case, but if your matrix is symmetric, it will come out as more expensive than CG, although equivalent on paper. Specifically, it will perform twice as many matrix-vector products. For both, you need to specify a max number of iteration that is *at least* the order of the matrix. If it isn't too ill conditioned, chances are you'll need much less iterations than that. See scipy.linalg.cg and scipy.linalg.bicgstab Hope this helps Dominique From ollinger at wisc.edu Thu Mar 6 20:29:15 2008 From: ollinger at wisc.edu (John Ollinger) Date: Thu, 6 Mar 2008 19:29:15 -0600 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea In-Reply-To: References: Message-ID: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> Fernando, When I wrap a C package I spend 95% of my time modifying the package to run as a library and 5% of the time hand-wrapping it for python. I find that hand-wrapping makes me more aware of the interface than using swig, so I can recode portions of the library to make the overall package work better. A quick look at the cyrex documentation makes me wonder if it is primarily written for those who haven't written much C-code.. Is this an accurate impression? I don't want to spend time learning another set of syntax to do something I already know how to do. John On Thu, Mar 6, 2008 at 12:15 PM, Fernando Perez wrote: > Hi all, > > after the Scipy/Sage Days 8 meeting, we were all very impressed by the > progress made by Cython. For those not familiar with it, Cython: > > http://www.cython.org/ > > is an evolved version of Pyrex (which is used by numpy and scipy) with > lots of improvements. We'd like to position Cython as the preferred > way of writing most, if not all, new extension code written for numpy > and scipy, as it is easier to write, get right, debug (when you still > get it wrong) and maintain than writing to the raw Python-C API. > > A specific project along these lines, that would be very beneficial > for numpy could be: > > - Creating new matrix types in cython that match the cvxopt matrices. > The creation of new numpy array types with efficient code would be > very useful. > > - Rewriting the existing ndarray subclasses that ship with numpy, such > as record arrays, in cython. In doing this, benchmarks of the > relative performance of the new code should be obtained. > > > Another possible project would be the addition to Cython of syntactic > support for array expressions, multidimensional indexing, and other > features of numpy. This is probably more difficult than the above, as > it would require fairly detailed knowledge of both the numpy C API and > the Cython internals, but would ultimately be extremely useful. > > > Any student interested in this should quickly respond on the list; > such a project would likely be co-mentored by people on the Numpy and > Cython teams, since it is likely to require expertise from both ends. > > Cheers, > > f > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > -- John Ollinger University of Wisconsin Waisman Center, T233 1500 Highland Ave Madison, WI 53711 http://brainimaging.waisman.wisc.edu/~jjo/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Mar 6 20:52:06 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 6 Mar 2008 19:52:06 -0600 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea In-Reply-To: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> References: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> Message-ID: <3d375d730803061752x50e12118u75aa3a8dc3e7244a@mail.gmail.com> On Thu, Mar 6, 2008 at 7:29 PM, John Ollinger wrote: > Fernando, > > When I wrap a C package I spend 95% of my time modifying the package to run > as a library and 5% of the time hand-wrapping it for python. I find that > hand-wrapping makes me more aware of the interface than using swig, so I can > recode portions of the library to make the overall package work better. A > quick look at the cyrex documentation makes me wonder if it is primarily > written for those who haven't written much C-code.. Is this an accurate > impression? No. Using Cython or Pyrex eliminates an enormous amount of the boilerplate involved in writing a C extension especially when it comes to implementing types. They are definitely not the same kind of tool that SWIG is. Instead, they are tools to make hand-wrapping less painful, perhaps even pleasant. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From seefeld at sympatico.ca Thu Mar 6 20:57:09 2008 From: seefeld at sympatico.ca (Stefan Seefeld) Date: Thu, 06 Mar 2008 20:57:09 -0500 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea In-Reply-To: <3d375d730803061752x50e12118u75aa3a8dc3e7244a@mail.gmail.com> References: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> <3d375d730803061752x50e12118u75aa3a8dc3e7244a@mail.gmail.com> Message-ID: <47D0A0F5.4000903@sympatico.ca> Robert Kern wrote: > On Thu, Mar 6, 2008 at 7:29 PM, John Ollinger wrote: >> Fernando, >> >> When I wrap a C package I spend 95% of my time modifying the package to run >> as a library and 5% of the time hand-wrapping it for python. I find that >> hand-wrapping makes me more aware of the interface than using swig, so I can >> recode portions of the library to make the overall package work better. A >> quick look at the cyrex documentation makes me wonder if it is primarily >> written for those who haven't written much C-code.. Is this an accurate >> impression? > > No. Using Cython or Pyrex eliminates an enormous amount of the > boilerplate involved in writing a C extension especially when it comes > to implementing types. They are definitely not the same kind of tool > that SWIG is. Instead, they are tools to make hand-wrapping less > painful, perhaps even pleasant. Has any thought been spent on using boost.python for this ? If so, what were / are the arguments against it ? (http://boost.org/libs/python/doc/tutorial/doc/html/index.html) Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin... From robert.kern at gmail.com Thu Mar 6 21:05:34 2008 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 6 Mar 2008 20:05:34 -0600 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea In-Reply-To: <47D0A0F5.4000903@sympatico.ca> References: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> <3d375d730803061752x50e12118u75aa3a8dc3e7244a@mail.gmail.com> <47D0A0F5.4000903@sympatico.ca> Message-ID: <3d375d730803061805h3631e355y1a34ec60c6cb45b8@mail.gmail.com> On Thu, Mar 6, 2008 at 7:57 PM, Stefan Seefeld wrote: > Robert Kern wrote: > > On Thu, Mar 6, 2008 at 7:29 PM, John Ollinger wrote: > >> Fernando, > >> > >> When I wrap a C package I spend 95% of my time modifying the package to run > >> as a library and 5% of the time hand-wrapping it for python. I find that > >> hand-wrapping makes me more aware of the interface than using swig, so I can > >> recode portions of the library to make the overall package work better. A > >> quick look at the cyrex documentation makes me wonder if it is primarily > >> written for those who haven't written much C-code.. Is this an accurate > >> impression? > > > > No. Using Cython or Pyrex eliminates an enormous amount of the > > boilerplate involved in writing a C extension especially when it comes > > to implementing types. They are definitely not the same kind of tool > > that SWIG is. Instead, they are tools to make hand-wrapping less > > painful, perhaps even pleasant. > > Has any thought been spent on using boost.python for this ? If so, what > were / are the arguments against it ? Well, boost.python certainly had the goal to make hand-wrapping less painful, but in my experience, it just made it more C++y. If you're heavily invested in C++ and the advanced features of C++ that the Boost project uses, then this might be a win for you. For me, it was just an exchange of unpleasantness. The excessive compile time was the killer for me; I just couldn't develop quickly enough using it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Thu Mar 6 23:22:07 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Thu, 06 Mar 2008 22:22:07 -0600 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea In-Reply-To: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> References: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> Message-ID: <47D0C2EF.5020506@enthought.com> John Ollinger wrote: > Fernando, > > When I wrap a C package I spend 95% of my time modifying the package > to run as a library and 5% of the time hand-wrapping it for python. I > find that hand-wrapping makes me more aware of the interface than > using swig, so I can recode portions of the library to make the > overall package work better. A quick look at the cyrex documentation > makes me wonder if it is primarily written for those who haven't > written much C-code.. Is this an accurate impression? Yes, I think this is accurate. The big win, however with using cython is maintainability. Many more people can improve the code. Complicated things can be a bit of a pain to do in Cython yet, and there is a desperate need for multi-dimensional syntax, but I've been impressed with how many people have contributed to the Sage project and wonder if some of that is the fact that much of it is written in Cython. (There are other things the SciPy project can learn from the Sage project, as well). Best regards, -Travis Oliphant From ondrej at certik.cz Thu Mar 6 23:56:57 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 7 Mar 2008 05:56:57 +0100 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea In-Reply-To: <47D0C2EF.5020506@enthought.com> References: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> <47D0C2EF.5020506@enthought.com> Message-ID: <85b5c3130803062056x7839e7cctce6aff81f6108ef8@mail.gmail.com> On Fri, Mar 7, 2008 at 5:22 AM, Travis E. Oliphant wrote: > John Ollinger wrote: > > Fernando, > > > > When I wrap a C package I spend 95% of my time modifying the package > > to run as a library and 5% of the time hand-wrapping it for python. I > > find that hand-wrapping makes me more aware of the interface than > > using swig, so I can recode portions of the library to make the > > overall package work better. A quick look at the cyrex documentation > > makes me wonder if it is primarily written for those who haven't > > written much C-code.. Is this an accurate impression? > Yes, I think this is accurate. The big win, however with using cython > is maintainability. Many more people can improve the code. > Complicated things can be a bit of a pain to do in Cython yet, and there > is a desperate need for multi-dimensional syntax, but I've been > impressed with how many people have contributed to the Sage project and > wonder if some of that is the fact that much of it is written in > Cython. (There are other things the SciPy project can learn from the > Sage project, as well). Like release early, release often? :) Ondrej From fperez.net at gmail.com Fri Mar 7 03:54:04 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 7 Mar 2008 00:54:04 -0800 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea In-Reply-To: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> References: <5b9ba9310803061729sc503c3di4cac6c55c1157894@mail.gmail.com> Message-ID: Hi John, On Thu, Mar 6, 2008 at 5:29 PM, John Ollinger wrote: > Fernando, > > When I wrap a C package I spend 95% of my time modifying the package to run > as a library and 5% of the time hand-wrapping it for python. I find that > hand-wrapping makes me more aware of the interface than using swig, so I can > recode portions of the library to make the overall package work better. A > quick look at the cyrex documentation makes me wonder if it is primarily > written for those who haven't written much C-code.. Is this an accurate > impression? I don't want to spend time learning another set of syntax to do > something I already know how to do. Others have given you their perspective on this, but for me a huge issue is reducing the bus factor of our code. The python C API isn't particularly easy to use, and code written to it (even assuming an expert who does a perfect job of testing, documenting and describing the code) will inevitably be hard to maintain and extend in the future. Cython is close enough to Python to be fairly easy to pick up, and if anything, the Sage project has proved that such an infrastructure (among other things) encourages contributions and eases the maintainability problem. Cheers f From david at ar.media.kyoto-u.ac.jp Fri Mar 7 07:34:12 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 07 Mar 2008 21:34:12 +0900 Subject: [SciPy-dev] Merging branch to build scipy with scons Message-ID: <47D13644.2060209@ar.media.kyoto-u.ac.jp> Hi, I would like to discuss about the possibility to merge the build_with_scons branch of scipy into the trunk. It would enable building scipy with numscons. The merge should be much easier than for numpy: it only add setup and scons script files, it does not touch any source file, and by default, the new code should not even be seen by the build system, so it is unlikely to cause any unattended consequences (that is, if you do not build scipy with numscons, because it still have some rather subtle issues with fortran linking). cheers, David From dalcinl at gmail.com Fri Mar 7 10:38:24 2008 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 7 Mar 2008 12:38:24 -0300 Subject: [SciPy-dev] solution sparse matrix In-Reply-To: <4eeef9d40803061103s60f18a66u90dbee3031a4d8e7@mail.gmail.com> References: <4eeef9d40803061103s60f18a66u90dbee3031a4d8e7@mail.gmail.com> Message-ID: OK, but your matrix is really huge! Or has it very few nonzeros at each row? Could you describe my your application, that is, how do to obtain that matrix? On 3/6/08, jose luis Lopez Martinez wrote: > hi > > the size of matrix sparse is 128^4(268,435,456 rows)x 128^4(268,435,456 > colums). > > the matrix have diagonald dominancies that is a sufficient condition > for convergence of Jacobi iteration , gauss seidel and xor(relaxion method), > but is slow for converge. > > it exist a command in scipy that solve sparse matrix using at the > Gauss-Seidel iteration method o krylov iterative methods? > > thanks for your advisor > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From doutriaux1 at llnl.gov Fri Mar 7 14:01:16 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Fri, 07 Mar 2008 11:01:16 -0800 Subject: [SciPy-dev] changes with Python 2.5.2 ? In-Reply-To: References: <4eeef9d40803061103s60f18a66u90dbee3031a4d8e7@mail.gmail.com> Message-ID: <47D190FC.2090105@llnl.gov> HI, I'm running into a problem i didn't have before, i'm trying to do: import scipy.linalg and i get error about deprecate needing 3 args but only 1 passed Is that something triggered by python 2.5.2 ? C. From robert.kern at gmail.com Sat Mar 8 00:05:58 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 7 Mar 2008 23:05:58 -0600 Subject: [SciPy-dev] Merging branch to build scipy with scons In-Reply-To: <47D13644.2060209@ar.media.kyoto-u.ac.jp> References: <47D13644.2060209@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730803072105g91b76a3s7c7e9854f5675e01@mail.gmail.com> On Fri, Mar 7, 2008 at 6:34 AM, David Cournapeau wrote: > Hi, > > I would like to discuss about the possibility to merge the > build_with_scons branch of scipy into the trunk. It would enable > building scipy with numscons. > The merge should be much easier than for numpy: it only add setup > and scons script files, it does not touch any source file, and by > default, the new code should not even be seen by the build system, so it > is unlikely to cause any unattended consequences (that is, if you do not > build scipy with numscons, because it still have some rather subtle > issues with fortran linking). I'm in favor. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Mar 8 00:11:07 2008 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 7 Mar 2008 23:11:07 -0600 Subject: [SciPy-dev] Google Summer of Code 2008 and SciPy In-Reply-To: References: Message-ID: <3d375d730803072111s24ffd5abn70d739eb3fabc102@mail.gmail.com> On Mon, Mar 3, 2008 at 10:19 AM, halish wrote: > Hello, > my name is Mateusz Haligowski and I am a student of the third (and the > last) year of Baccalaureate studies at the University of Gda?sk. I > study Econometrics and Statistics. > > Last year I participated in GSoC, writing a plugin for Scribus (DTP > software). It was a great fun, but this year I'm longing to write > something connected with my studies. > > I would like to ask, whether SciPy is planning to participate as a > mentoring organisation in GSoC this year? I've received a message on > python-gsoc mailing list, that you would need some econometric > functions, and that would be a perfect thing for me. Most likely, the SciPy project will participate under the umbrella of the PSF. Feel free to talk about ideas here and find a mentor on this list. I'm guessing that it was Alan Isaac who requested econometric functions. He certainly would be a good mentor for such a project in any case. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From mhaligowski at googlemail.com Sat Mar 8 01:33:28 2008 From: mhaligowski at googlemail.com (halish) Date: Sat, 8 Mar 2008 07:33:28 +0100 Subject: [SciPy-dev] [Google Summer of Code] Econometric functions Message-ID: Hello, I would like to participate in Google Summer of Code 2008 for SciPy. As I study Econometrics and Statistics, I would be glad to add econometric functions to SciPy. This would include: * Ordinary Least Squares estimation * Generalized Least Squares estimation * Multi-equation estimation * Time series analysis (ie. stationary processes) Along with these elements will obviously come the verification tests as well. I'm looking forward for hearing from any potential mentor and for further suggestions. Regards, Mateusz Haligowski From david at ar.media.kyoto-u.ac.jp Sat Mar 8 06:26:41 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 08 Mar 2008 20:26:41 +0900 Subject: [SciPy-dev] [ANN] numscons 0.5.1: building scipy Message-ID: <47D277F1.4030302@ar.media.kyoto-u.ac.jp> Hi, Mumscons 0.5.1 is available through pypi (eggs and tarballs). This is the first version which can build the whole scipy source tree. To build scipy with numscons, you should first get the code in the branch: svn co http://svn.scipy.org/svn/scipy/branches/build_with_scons And then build it like numpy: python setupscons.py install Technically speaking, you can build scipy with numscons above a numpy build the standard way, but that's not a good idea (because of potential libraries and compilers mismatches between distutils and numscons). See http://projects.scipy.org/scipy/numpy/wiki/NumScons for more details. The only tested platform for now are: - linux + gcc; other compilers on linux should work as well. - solaris + sunstudio with sunperf. On both those platforms, only a few tests do not pass. I don't expect windows or mac OS X to work yet, but I can not test those platforms ATM. I am releasing the current state of numscons because I won't have much time to work on numscons the next few weeks unfortunately. PLEASE DO NOT USE IT FOR PRODUCTION USE ! There are still some serious issues: - I painfully discovered that at least g77 is extremely sensitive to different orders of linker flags (can cause crashes). I don't have any problem anymore on my workstation (Ubuntu 32 bits, atlas + gcc/g77), but this needs more testing. - there are some race conditions with f2py which I do not fully understand yet, and which prevents parallel build to work (so do not use the scons command --jobs option) - optimization flags of proprietary compilers: they are a PITA. They often break IEEE conformance in quite a hard way, and this causes crashes or wrong results (for example, the -fast option of sun compilers breaks the argsort function of numpy). So again, this is really just a release for people to test things if they want, but nothing else. cheers, David From aisaac at american.edu Sat Mar 8 12:07:29 2008 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 8 Mar 2008 12:07:29 -0500 Subject: [SciPy-dev] Google Summer of Code 2008 and SciPy In-Reply-To: <3d375d730803072111s24ffd5abn70d739eb3fabc102@mail.gmail.com> References: <3d375d730803072111s24ffd5abn70d739eb3fabc102@mail.gmail.com> Message-ID: On Fri, 7 Mar 2008, Robert Kern apparently wrote: > I'm guessing that it was Alan Isaac who requested > econometric functions. Just to elaborate a bit on Robert's comment ... I volunteered to be a mentor *if* the SciPy project decides to fund a student to expand the econometric capabilities of SciPy. Availability of a mentor does NOT ensure that this will be a SciPy priority. (It does not hurt, of course.) My personal interest is in seeing more time-series related econometric functionality in SciPy. (For examples of desirable functionality, I suggested looking at the functionality of the commercial EViews software, which is very popular for econometric time-series analysis.) However the priorities of the SciPy project may prove to be different. Note that SciPy contains substantial basic functionality already. For example, see Cheers, Alan Isaac From robert.kern at gmail.com Sat Mar 8 14:13:40 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 8 Mar 2008 13:13:40 -0600 Subject: [SciPy-dev] Google Summer of Code 2008 and SciPy In-Reply-To: References: <3d375d730803072111s24ffd5abn70d739eb3fabc102@mail.gmail.com> Message-ID: <3d375d730803081113p607f69d9lbcb1ce3154e2896b@mail.gmail.com> On Sat, Mar 8, 2008 at 11:07 AM, Alan G Isaac wrote: > On Fri, 7 Mar 2008, Robert Kern apparently wrote: > > I'm guessing that it was Alan Isaac who requested > > econometric functions. > > Just to elaborate a bit on Robert's comment ... > > I volunteered to be a mentor *if* the SciPy project decides > to fund a student to expand the econometric capabilities of > SciPy. Availability of a mentor does NOT ensure that this > will be a SciPy priority. (It does not hurt, of course.) I am more than happy to have them. I only have two concerns: 1. Due care should be taken to enhance the current functionality where it exists rather than creating duplicate functionality. 2. For SciPy, I prefer more general code than domain-specific code. If there is a choice between general statistical practice and specifically econometrics, I'd prefer that we go with general statistical practice. If there is some doubt, ask on the list, and I'm sure that you'll get a variety of opinions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From millman at berkeley.edu Sat Mar 8 15:42:07 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Sat, 8 Mar 2008 12:42:07 -0800 Subject: [SciPy-dev] Merging branch to build scipy with scons In-Reply-To: <47D13644.2060209@ar.media.kyoto-u.ac.jp> References: <47D13644.2060209@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Mar 7, 2008 at 4:34 AM, David Cournapeau wrote: > I would like to discuss about the possibility to merge the > build_with_scons branch of scipy into the trunk. It would enable > building scipy with numscons. > The merge should be much easier than for numpy: it only add setup > and scons script files, it does not touch any source file, and by > default, the new code should not even be seen by the build system, so it > is unlikely to cause any unattended consequences (that is, if you do not > build scipy with numscons, because it still have some rather subtle > issues with fortran linking). +1. The sooner you merge your work, the better as far as I am concerned. Thanks again for all your work on numscons. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david at ar.media.kyoto-u.ac.jp Sat Mar 8 23:14:39 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 09 Mar 2008 13:14:39 +0900 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? Message-ID: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> Hi, I think we already had a discussion about that a few months ago, and I would like to reiterate my wish of splitting the sparsetools_wrap source file. C++ compilers are already dead slow, and with such a big source file, compiling this file alone makes for a big part of the whole scipy build time. Is is possible to split source files ? Or having something smaller ? cheers, David From robert.kern at gmail.com Sat Mar 8 23:38:21 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 8 Mar 2008 22:38:21 -0600 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> On Sat, Mar 8, 2008 at 10:14 PM, David Cournapeau wrote: > Hi, > > I think we already had a discussion about that a few months ago, and > I would like to reiterate my wish of splitting the sparsetools_wrap > source file. C++ compilers are already dead slow, and with such a big > source file, compiling this file alone makes for a big part of the whole > scipy build time. Is is possible to split source files ? Or having > something smaller ? Bascially, no. You would have to split up the extension module itself, and that could be difficult. sparsetools_wrap.cxx is generated by SWIG. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Sat Mar 8 23:52:21 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 09 Mar 2008 13:52:21 +0900 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> Message-ID: <47D36D05.90600@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > > Bascially, no. You would have to split up the extension module itself, > and that could be difficult. sparsetools_wrap.cxx is generated by > SWIG. > > I know it is generated by swig. But from a quick look to the swig interface file, it looks like sparsetools is just a bunch of utilities functions, which could be split up, no ? For example, one swig generated file per function should be possible, without too much trouble ? Sorry if this is stupid, I have not been using swig for a while, so I do not remember how it works. cheers, David From robert.kern at gmail.com Sun Mar 9 00:36:32 2008 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 8 Mar 2008 23:36:32 -0600 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: <47D36D05.90600@ar.media.kyoto-u.ac.jp> References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> <47D36D05.90600@ar.media.kyoto-u.ac.jp> Message-ID: <3d375d730803082136j518debb3q30018e0ea1570504@mail.gmail.com> On Sat, Mar 8, 2008 at 10:52 PM, David Cournapeau wrote: > Robert Kern wrote: > > > > Bascially, no. You would have to split up the extension module itself, > > and that could be difficult. sparsetools_wrap.cxx is generated by > > SWIG. > > > > > I know it is generated by swig. But from a quick look to the swig > interface file, it looks like sparsetools is just a bunch of utilities > functions, which could be split up, no ? For example, one swig generated > file per function should be possible, without too much trouble ? Sorry > if this is stupid, I have not been using swig for a while, so I do not > remember how it works. If anything that would just exacerbate the problem. All of the SWIG utility functions would be repeated for each module. In any case, I split the file into 2 and the total time for compiling both is about the same as as compiling the current sparsetools_wrap.cxx. This would be a win for parallel builds, but until numscons is the official build system, I'm against making changes as invasive as splitting up an extension module just to optimize for that. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dmitrey.kroshko at scipy.org Sun Mar 9 14:34:10 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Sun, 09 Mar 2008 20:34:10 +0200 Subject: [SciPy-dev] could anyone check nlp_3.py from updated scikits.openopt? Message-ID: <47D42DA2.5000302@scipy.org> Hi all, Nils Wagner informs me of a bug that I don't observe, so I can't fix the one. Could anyone update openopt from svn and run (after "sudo python setup.py install") examples/nlp_3.py ? (It works well for me) Thank you in advance, D. From dave.hirschfeld at gmail.com Sun Mar 9 16:41:34 2008 From: dave.hirschfeld at gmail.com (Dave) Date: Sun, 9 Mar 2008 20:41:34 +0000 (UTC) Subject: [SciPy-dev] =?utf-8?q?could_anyone_check_nlp=5F3=2Epy_from_update?= =?utf-8?q?d=09scikits=2Eopenopt=3F?= References: <47D42DA2.5000302@scipy.org> Message-ID: dmitrey scipy.org> writes: > Could anyone update openopt from svn and run > (after "sudo python setup.py install") > examples/nlp_3.py ? I updated openopt from svn to rev. 896 and installed it with python setup.py install. Running the nlp_3.py script raises an exception suggesting that the solver ralg may not be installed. I'm not sure if I needed to do anything else with the install although I don't believe I had to previously. HTH, Dave Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] IPython 0.8.2.svn.r2848 -- An enhanced Interactive Python. In [1]: run -i nlp_3.py OO Error:incorrect solver is called, maybe the solver "ralg" is not installed --------------------------------------------------------------------------- OpenOptException Traceback (most recent call last) C:\src\openopt\scikits\openopt\examples\nlp_3.py in () 36 p.connectIterFcn('df') 37 ---> 38 r = p.solve(solver) 39 results[solver] = (r.ff, p.getMaxResidual(r.xf), r.elapsed['solver_t ime'], r.elapsed['solver_cputime'], r.evals['f'], r.evals['c'], r.evals['h']) 40 subplot(2,1,1) c:\python25\lib\site-packages\scikits\openopt\Kernel\BaseProblem.py in solve(sel f, solvers, *args, **kwargs) 229 230 def solve(self, solvers, *args, **kwargs): --> 231 return runProbSolver(self, solvers, *args, **kwargs) 232 233 def auxFunc(self, x, *args, **kwargs): c:\python25\lib\site-packages\scikits\openopt\Kernel\runProbSolver.py in runProb Solver(p_, solver_str, *args, **kwargs) 23 solverClass = getattr(__import__(solver_str), solver_str) 24 except: ---> 25 p.err('incorrect solver is called, maybe the solver "' + solver_ str +'" is not installed') 26 27 p.solver = solverClass() c:\python25\lib\site-packages\scikits\openopt\Kernel\oologfcn.py in ooerr(msg) 14 def ooerr(msg): 15 print 'OO Error:' + msg ---> 16 raise OpenOptException(msg) 17 18 def ooPWarn(msg): OpenOptException: incorrect solver is called, maybe the solver "ralg" is not ins talled WARNING: Failure executing file: In [2]: import numpy; numpy.__version__ Out[2]: '1.0.3.1' In [3]: import scipy; scipy.__version__ Out[3]: '0.7.0.dev3498' In [4]: from scikits import openopt; openopt.__version__ Out[4]: '0.15' From cournape at gmail.com Sun Mar 9 23:12:38 2008 From: cournape at gmail.com (David Cournapeau) Date: Mon, 10 Mar 2008 03:12:38 +0000 (UTC) Subject: [SciPy-dev] =?utf-8?q?Splitting_sparsetools=5Fwrap_source_file_?= =?utf-8?q?=3F?= References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> <47D36D05.90600@ar.media.kyoto-u.ac.jp> <3d375d730803082136j518debb3q30018e0ea1570504@mail.gmail.com> Message-ID: Robert Kern gmail.com> writes: > If anything that would just exacerbate the problem. All of the SWIG > utility functions would be repeated for each module. In any case, I > split the file into 2 and the total time for compiling both is about > the same as as compiling the current sparsetools_wrap.cxx. This would > be a win for parallel builds, but until numscons is the official build > system, I'm against making changes as invasive as splitting up an > extension module just to optimize for that. > It is not so much the build time that the memory consumption which I found problematic. It is becoming difficult to build scipy on my laptop with 512 Mb (compiling the sparsetools_wrap.cxx with gcc -O2 takes around 400 Mb, and some compilers are even worse), or on virtual machines. Is the patch really that invasive ? Splitting in 2 or 3 would already be good if the memory consumption reduction is decreased by the same order. cheers, David From dmitrey.kroshko at scipy.org Mon Mar 10 07:48:00 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Mon, 10 Mar 2008 13:48:00 +0200 Subject: [SciPy-dev] could anyone check nlp_3.py from updated scikits.openopt? In-Reply-To: References: <47D42DA2.5000302@scipy.org> Message-ID: <47D51FF0.3000307@scipy.org> Dave, could you update openopt from svn, run [sudo] python setup.py install and run nlp_3.py once again? (I made some changes so the bug will be described more precisely) Also, what's your Python version? Regards, D. Dave wrote: > dmitrey scipy.org> writes: > > >> Could anyone update openopt from svn and run >> (after "sudo python setup.py install") >> examples/nlp_3.py ? >> > > I updated openopt from svn to rev. 896 and installed it with python setup.py > install. > > Running the nlp_3.py script raises an exception suggesting that the solver ralg > may not be installed. I'm not sure if I needed to do anything else with the > install although I don't believe I had to previously. > > HTH, > Dave > > > > Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] > IPython 0.8.2.svn.r2848 -- An enhanced Interactive Python. > > In [1]: run -i nlp_3.py > OO Error:incorrect solver is called, maybe the solver "ralg" is not installed > --------------------------------------------------------------------------- > OpenOptException Traceback (most recent call last) > > C:\src\openopt\scikits\openopt\examples\nlp_3.py in () > 36 p.connectIterFcn('df') > 37 > ---> 38 r = p.solve(solver) > 39 results[solver] = (r.ff, p.getMaxResidual(r.xf), r.elapsed['solver_t > ime'], r.elapsed['solver_cputime'], r.evals['f'], r.evals['c'], r.evals['h']) > 40 subplot(2,1,1) > > c:\python25\lib\site-packages\scikits\openopt\Kernel\BaseProblem.py in solve(sel > f, solvers, *args, **kwargs) > 229 > 230 def solve(self, solvers, *args, **kwargs): > --> 231 return runProbSolver(self, solvers, *args, **kwargs) > 232 > 233 def auxFunc(self, x, *args, **kwargs): > > c:\python25\lib\site-packages\scikits\openopt\Kernel\runProbSolver.py in runProb > Solver(p_, solver_str, *args, **kwargs) > 23 solverClass = getattr(__import__(solver_str), solver_str) > 24 except: > ---> 25 p.err('incorrect solver is called, maybe the solver "' + solver_ > str +'" is not installed') > 26 > 27 p.solver = solverClass() > > c:\python25\lib\site-packages\scikits\openopt\Kernel\oologfcn.py in ooerr(msg) > 14 def ooerr(msg): > 15 print 'OO Error:' + msg > ---> 16 raise OpenOptException(msg) > 17 > 18 def ooPWarn(msg): > > OpenOptException: incorrect solver is called, maybe the solver "ralg" is not ins > talled > WARNING: Failure executing file: > > > In [2]: import numpy; numpy.__version__ > Out[2]: '1.0.3.1' > > In [3]: import scipy; scipy.__version__ > Out[3]: '0.7.0.dev3498' > > In [4]: from scikits import openopt; openopt.__version__ > Out[4]: '0.15' > > > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > From dave.hirschfeld at gmail.com Mon Mar 10 08:24:33 2008 From: dave.hirschfeld at gmail.com (Dave) Date: Mon, 10 Mar 2008 12:24:33 +0000 (UTC) Subject: [SciPy-dev] =?utf-8?q?could_anyone_check_nlp=5F3=2Epy_from_update?= =?utf-8?q?d=09scikits=2Eopenopt=3F?= References: <47D42DA2.5000302@scipy.org> <47D51FF0.3000307@scipy.org> Message-ID: dmitrey scipy.org> writes: > > Dave, could you update openopt from svn, run [sudo] python setup.py > install and run nlp_3.py once again? > (I made some changes so the bug will be described more precisely) > Also, what's your Python version? > Regards, D. System Info: > > Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] > > IPython 0.8.2.svn.r2848 -- An enhanced Interactive Python. > > In [2]: import numpy; numpy.__version__ > > Out[2]: '1.0.3.1' > > > > In [3]: import scipy; scipy.__version__ > > Out[3]: '0.7.0.dev3498' > > > > In [4]: from scikits import openopt; openopt.__version__ > > Out[4]: '0.15' (rev. 897) In [4]: run -i nlp_3.py --------------------------------------------------------------------------- ImportError Traceback (most recent call last) C:\src\openopt\scikits\openopt\examples\nlp_3.py in () 37 p.debug = 1 38 ---> 39 r = p.solve(solver) 40 results[solver] = (r.ff, p.getMaxResidual(r.xf), r.elapsed['solver_t ime'], r.elapsed['solver_cputime'], r.evals['f'], r.evals['c'], r.evals['h']) 41 subplot(2,1,1) c:\python25\lib\site-packages\scikits\openopt\Kernel\BaseProblem.py in solve(sel f, solvers, *args, **kwargs) 229 230 def solve(self, solvers, *args, **kwargs): --> 231 return runProbSolver(self, solvers, *args, **kwargs) 232 233 def auxFunc(self, x, *args, **kwargs): c:\python25\lib\site-packages\scikits\openopt\Kernel\runProbSolver.py in runProb Solver(p_, solver_str, *args, **kwargs) 20 21 if p.debug: ---> 22 solverClass = getattr(__import__(solver_str), solver_str) 23 else: 24 try: c:\python25\lib\site-packages\scikits\openopt\solvers\UkrOpt\ralg.py in () ----> 1 2 3 from numpy import diag, ones, inf, any, copy, zeros, dot, where, all, ti le, sum, nan, isfinite, float128, float64 4 from numpy.linalg import norm 5 from BaseAlg import * 6 from setDefaultIterFuncs import SMALL_DELTA_X, SMALL_DELTA_F, SMALL_DF , IS_LINE_SEARCH_FAILED 7 from UkrOptMisc import getConstrDirection ImportError: cannot import name float128 WARNING: Failure executing file: Also, the only floats available in my numpy version are: In [7]: numpy.float numpy.float numpy.float64 numpy.float_ numpy.float32 numpy.float96 numpy.floating HTH, Dave From dmitrey.kroshko at scipy.org Mon Mar 10 08:44:29 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Mon, 10 Mar 2008 14:44:29 +0200 Subject: [SciPy-dev] numpy.float128 absence yield bugs Message-ID: <47D52D2D.8040409@scipy.org> hi all, if numpy.float128 type is absent (BTW does it related to OS or something else?), it yields bugs even if the type is never used (while importing from a py-file). For example: from numpy import ..., float64, float128 if (some_cond): else: So numpy turns out not being 100% cross-platform. Maybe it would be better to raise a warning like "float128 is absent, float64 will be used instead"? Or, better, implement float128 being always present at all? (I.e. connect a library that will take care of float128 type provided native machine support is absent)? Regars, D. From matthieu.brucher at gmail.com Mon Mar 10 08:49:18 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 10 Mar 2008 13:49:18 +0100 Subject: [SciPy-dev] numpy.float128 absence yield bugs In-Reply-To: <47D52D2D.8040409@scipy.org> References: <47D52D2D.8040409@scipy.org> Message-ID: 2008/3/10, dmitrey : > > hi all, > if numpy.float128 type is absent (BTW does it related to OS or something > else?), it yields bugs even if the type is never used (while importing > from a py-file). For example: > > from numpy import ..., float64, float128 > if (some_cond): > > else: If you do not need a direct call to float128 in your condition, you can easily make it compatible on all platforms. Do not import numpy with "from numpy import ...", but only "import numpy", make your decision based on cond and the presence of float128 ('float128' in dir(numpy)) and everything will be fine ;) Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrey.kroshko at scipy.org Mon Mar 10 08:57:18 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Mon, 10 Mar 2008 14:57:18 +0200 Subject: [SciPy-dev] numpy.float128 absence yield bugs In-Reply-To: References: <47D52D2D.8040409@scipy.org> Message-ID: <47D5302E.7020000@scipy.org> Matthieu Brucher wrote: > > > 2008/3/10, dmitrey >: > > hi all, > if numpy.float128 type is absent (BTW does it related to OS or > something > else?), it yields bugs even if the type is never used (while importing > from a py-file). For example: > > from numpy import ..., float64, float128 > if (some_cond): > > else: > > > If you do not need a direct call to float128 in your condition, you > can easily make it compatible on all platforms. > Do not import numpy with "from numpy import ...", but only "import > numpy", make your decision based on cond and the presence of float128 > ('float128' in dir(numpy)) and everything will be fine ;) > > Matthieu I still think if something is wrong with from numpy import ... than it should be found and fixed instead of avoiding the one and proposing the solution to other numpy users/programmers. I guess putting the checks ... in dir(numpy) or hasattr(numpy, float128) everywhere isn't a best solution. Regards, D. From matthieu.brucher at gmail.com Mon Mar 10 09:02:37 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 10 Mar 2008 14:02:37 +0100 Subject: [SciPy-dev] numpy.float128 absence yield bugs In-Reply-To: <47D5302E.7020000@scipy.org> References: <47D52D2D.8040409@scipy.org> <47D5302E.7020000@scipy.org> Message-ID: > > > > If you do not need a direct call to float128 in your condition, you > > can easily make it compatible on all platforms. > > Do not import numpy with "from numpy import ...", but only "import > > numpy", make your decision based on cond and the presence of float128 > > ('float128' in dir(numpy)) and everything will be fine ;) > > > > Matthieu > > I still think if something is wrong with > from numpy import ... > than it should be found and fixed instead of avoiding the one and > proposing the solution to other numpy users/programmers. > I guess putting the checks ... in dir(numpy) or hasattr(numpy, float128) > everywhere isn't a best solution. > The issue is that not every platform has a float128 library, so either you make it available to noone so that such error cannot be raised, either you add some code in case you really need it and it is available. If float128 is implemented in a bogus way and your condition is true, there will be another issue and you will ask for a complete implementation, which will never be the case. I think that your best shot is with making with what is available, with its advantages and drawbacks, and that means testing for the existence of float128 if you need it. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From wnbell at gmail.com Mon Mar 10 09:05:50 2008 From: wnbell at gmail.com (Nathan Bell) Date: Mon, 10 Mar 2008 08:05:50 -0500 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> <47D36D05.90600@ar.media.kyoto-u.ac.jp> <3d375d730803082136j518debb3q30018e0ea1570504@mail.gmail.com> Message-ID: On Sun, Mar 9, 2008 at 10:12 PM, David Cournapeau wrote: > It is not so much the build time that the memory consumption which I found > problematic. It is becoming difficult to build scipy on my laptop with 512 Mb > (compiling the sparsetools_wrap.cxx with gcc -O2 takes around 400 Mb, and some > compilers are even worse), or on virtual machines. > > Is the patch really that invasive ? Splitting in 2 or 3 would already be good if > the memory consumption reduction is decreased by the same order. Splitting the file into multiple parts does reduce the memory usage, but not by the expected fraction. Aside from manually splitting the SWIG output into multiple files (which would be tedious, time consuming, and error-prone), I'm not sure how to remedy the situation. In the era of $25/GB RAM, is it not more expedient to simply increase your memory capacity? Using SWIG and C++ templates is a major convenience in sparsetools since adding new dtypes becomes trivial. However this implies that each function is instantiated ~15 times (once for each dtype) which results in high memory usage. If there's a simple solution that addresses your concerns, I'd be happy to make the necessary changes. Otherwise, I don't think the problem merits complicating sparsetools. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From dmitrey.kroshko at scipy.org Mon Mar 10 09:08:22 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Mon, 10 Mar 2008 15:08:22 +0200 Subject: [SciPy-dev] could anyone check nlp_3.py from updated scikits.openopt? In-Reply-To: References: <47D42DA2.5000302@scipy.org> <47D51FF0.3000307@scipy.org> Message-ID: <47D532C6.208@scipy.org> Dave, could you repeat (svn update -> nlp_3) once again? I think I have fixed the issue. Regards, D. Dave wrote: > ImportError: cannot import name float128 > WARNING: Failure executing file: > > > Also, the only floats available in my numpy version are: > > In [7]: numpy.float > numpy.float numpy.float64 numpy.float_ > numpy.float32 numpy.float96 numpy.floating > > > HTH, > Dave > From david at ar.media.kyoto-u.ac.jp Mon Mar 10 09:14:59 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 10 Mar 2008 22:14:59 +0900 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> <47D36D05.90600@ar.media.kyoto-u.ac.jp> <3d375d730803082136j518debb3q30018e0ea1570504@mail.gmail.com> Message-ID: <47D53453.3000806@ar.media.kyoto-u.ac.jp> Nathan Bell wrote: > > Splitting the file into multiple parts does reduce the memory usage, > but not by the expected fraction. Aside from manually splitting the > SWIG output into multiple files (which would be tedious, time > consuming, and error-prone), I'm not sure how to remedy the situation. > This is obviously a bad solution (splitting the generated swig file), and a nightmare to get right. I was more thinking about splitting the interface file, so that only a couple of functions are generated by each: this should be doable, no ? I can do it, if there is a chance for a patch to be included. There would be say N swig interface files (one for _diagonal, one for _scale, etc...), and sparsetools.py itself would be written manually, but would just import python functions from each generated python modules, that is would be a few lines only (I bet this python module could easily be generated, too, if wanted). > In the era of $25/GB RAM, is it not more expedient to simply increase > your memory capacity? Using SWIG and C++ templates is a major > convenience in sparsetools since adding new dtypes becomes trivial. > I am not suggesting giving up swig or C++ templates. But the problem is not the cost of memory: when virtual machines came into the game, you hit really quickly the 32 bits limits (or more exactly, the fact that most computers cannot physically handle more than 4 Gb of Memory). For example, when I test numscons on solaris, I use indiana, which is a binary distribution of open solaris available for free, and the VM takes more than 1 Gb of ram when compiling sparsetools. Even on my recent macbook with 2 Gb of Ram, I am at the limit. And virtual machines are the only way for me to test many platforms (and build bots too often run on vmware). cheers, David From dave.hirschfeld at gmail.com Mon Mar 10 10:00:45 2008 From: dave.hirschfeld at gmail.com (Dave) Date: Mon, 10 Mar 2008 14:00:45 +0000 (UTC) Subject: [SciPy-dev] =?utf-8?q?could_anyone_check_nlp=5F3=2Epy_from_update?= =?utf-8?q?d=09scikits=2Eopenopt=3F?= References: <47D42DA2.5000302@scipy.org> <47D51FF0.3000307@scipy.org> <47D532C6.208@scipy.org> Message-ID: dmitrey scipy.org> writes: > > Dave, could you repeat (svn update -> nlp_3) once again? > I think I have fixed the issue. > Regards, D. That seem to have fixed the initial problem. It now complains about not finding solver cvxopt_qp but I think that's expected as I haven't installed cvxopt. -Dave In [2]: run -i nlp_3.py starting solver ralg (license: BSD) with problem unnamed itn 0 : Fk= 111.281224049 MaxResidual: 1.33e+002 itn 86 Fk: 30.7036061227 MaxResidual: 9.98e-007(+) ls: 2 ralg has finished solving the problem unnamed istop: 4 (|| F[k] - F[k-1] || < ftol) Solver: Time Elapsed = 0.28 CPU Time Elapsed = 0.287123922174 Plotting: Time Elapsed = 3.64 CPU Time Elapsed = 3.62924114657 objFunValue: 30.7036061227 (feasible, max constraint = 9.98099e-007) starting solver scipy_cobyla (license: BSD) with problem unnamed scipy_cobyla has finished solving the problem unnamed istop: 1000 Solver: Time Elapsed = 0.36 CPU Time Elapsed = 0.352573581279 Plotting: Time Elapsed = 2.23 CPU Time Elapsed = 2.23213229614 objFunValue: 31.0165763677 (feasible, max constraint = 1.42109e-013) starting solver lincher (license: BSD) with problem unnamed itn 0: Fk= 111.281224049 maxResidual= 133.401636596 OO Error:incorrect solver is called, maybe the solver "cvxopt_qp" is not install ed. Maybe setting p.debug=1 could specify the matter more precisely --------------------------------------------------------------------------- OpenOptException Traceback (most recent call last) C:\src\openopt\scikits\openopt\examples\nlp_3.py in () 37 p.debug = 1 38 ---> 39 r = p.solve(solver) 40 results[solver] = (r.ff, p.getMaxResidual(r.xf), r.elapsed['solver_t ime'], r.elapsed['solver_cputime'], r.evals['f'], r.evals['c'], r.evals['h']) 41 subplot(2,1,1) c:\python25\lib\site-packages\scikits\openopt\Kernel\BaseProblem.py in solve(sel f, solvers, *args, **kwargs) 229 230 def solve(self, solvers, *args, **kwargs): --> 231 return runProbSolver(self, solvers, *args, **kwargs) 232 233 def auxFunc(self, x, *args, **kwargs): c:\python25\lib\site-packages\scikits\openopt\Kernel\runProbSolver.py in runProb Solver(p_, solver_str, *args, **kwargs) 155 if p.iprint >= 0: print 'starting solver', p.solverName, '(license : ' + p.solverLicense + ')', ' with problem ', p.name 156 try: --> 157 solver(p) 158 except isSolved: 159 p.xf = p.xk c:\python25\lib\site-packages\scikits\openopt\solvers\UkrOpt\lincher.py in __sol ver__(self, p) 180 qp = QP(H=H,f=df, A=A, Aeq=Aeq, b=b, beq=beq, lb=lb, ub = ub) 181 qp.iprint = -1 --> 182 r2 = qp.solve('cvxopt_qp') # TODO: replace solver by autoselect 183 #r2 = qp.solve('qld') # TODO: replace solver by auto select 184 if r2.istop <= 0: c:\python25\lib\site-packages\scikits\openopt\Kernel\BaseProblem.py in solve(sel f, solvers, *args, **kwargs) 229 230 def solve(self, solvers, *args, **kwargs): --> 231 return runProbSolver(self, solvers, *args, **kwargs) 232 233 def auxFunc(self, x, *args, **kwargs): c:\python25\lib\site-packages\scikits\openopt\Kernel\runProbSolver.py in runProb Solver(p_, solver_str, *args, **kwargs) 25 solverClass = getattr(__import__(solver_str), solver_str) 26 except: ---> 27 p.err('incorrect solver is called, maybe the solver "' + sol ver_str +'" is not installed. Maybe setting p.debug=1 could specify the matter m ore precisely') 28 29 p.solver = solverClass() c:\python25\lib\site-packages\scikits\openopt\Kernel\oologfcn.py in ooerr(msg) 14 def ooerr(msg): 15 print 'OO Error:' + msg ---> 16 raise OpenOptException(msg) 17 18 def ooPWarn(msg): OpenOptException: incorrect solver is called, maybe the solver "cvxopt_qp" is no t installed. Maybe setting p.debug=1 could specify the matter more precisely WARNING: Failure executing file: In [3]: From oliphant at enthought.com Mon Mar 10 13:10:02 2008 From: oliphant at enthought.com (Travis E. Oliphant) Date: Mon, 10 Mar 2008 12:10:02 -0500 Subject: [SciPy-dev] numpy.float128 absence yield bugs In-Reply-To: <47D52D2D.8040409@scipy.org> References: <47D52D2D.8040409@scipy.org> Message-ID: <47D56B6A.80604@enthought.com> dmitrey wrote: > hi all, > if numpy.float128 type is absent (BTW does it related to OS or something > else?), it yields bugs even if the type is never used (while importing > from a py-file). For example: > I suggest using numpy.longdouble if you are interested in cross-platform use of float128. There will always be a longdouble, but on some platforms it is 128-bit, on others 96-bit, and on others 64-bit (i.e. the same as double). -Travis O. From dmitrey.kroshko at scipy.org Mon Mar 10 13:37:42 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Mon, 10 Mar 2008 19:37:42 +0200 Subject: [SciPy-dev] numpy.float128 absence yield bugs In-Reply-To: <47D56B6A.80604@enthought.com> References: <47D52D2D.8040409@scipy.org> <47D56B6A.80604@enthought.com> Message-ID: <47D571E6.3090107@scipy.org> As for numpy.longdouble (and related types if they are intended to be implemented), I suppose numpy also should have a function to return number of bites used (128, 96, 64 etc). Ok. Another one question: does numpy have function(s), that accepts a numpy type (float32, float64, uint16 etc) and yields: max value (like 1e300); min value (like -1e300); min positive value (for floats) like 1e-300? Thank you in advance, D. Travis E. Oliphant wrote: > dmitrey wrote: > >> hi all, >> if numpy.float128 type is absent (BTW does it related to OS or something >> else?), it yields bugs even if the type is never used (while importing >> from a py-file). For example: >> >> > > I suggest using numpy.longdouble if you are interested in cross-platform > use of float128. There will always be a longdouble, but on some > platforms it is 128-bit, on others 96-bit, and on others 64-bit (i.e. > the same as double). > > -Travis O. > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > From robert.kern at gmail.com Mon Mar 10 13:44:52 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Mar 2008 12:44:52 -0500 Subject: [SciPy-dev] numpy.float128 absence yield bugs In-Reply-To: <47D571E6.3090107@scipy.org> References: <47D52D2D.8040409@scipy.org> <47D56B6A.80604@enthought.com> <47D571E6.3090107@scipy.org> Message-ID: <3d375d730803101044we56df29qf907255fe3781d0f@mail.gmail.com> On Mon, Mar 10, 2008 at 12:37 PM, dmitrey wrote: > As for numpy.longdouble (and related types if they are intended to be > implemented), I suppose numpy also should have a function to return > number of bites used (128, 96, 64 etc). In [6]: numpy.dtype(numpy.longdouble).itemsize Out[6]: 16 > Ok. Another one question: does numpy have function(s), that accepts a > numpy type (float32, float64, uint16 etc) and yields: max value (like > 1e300); min value (like -1e300); min positive value (for floats) like > 1e-300? In [7]: i = numpy.finfo(numpy.longdouble) In [8]: i. i.__class__ i.__module__ i.__weakref__ i._str_tiny i.max i.precision i.__delattr__ i.__new__ i._finfo_cache i.dtype i.maxexp i.resolution i.__dict__ i.__reduce__ i._init i.eps i.min i.tiny i.__doc__ i.__reduce_ex__ i._str_eps i.epsneg i.minexp i.__getattribute__ i.__repr__ i._str_epsneg i.iexp i.negep i.__hash__ i.__setattr__ i._str_max i.machar i.nexp i.__init__ i.__str__ i._str_resolution i.machep i.nmant In [8]: i.min Out[8]: -1.189731495357231765021e+4932 In [9]: i.max Out[9]: 1.189731495357231765021e+4932 In [10]: i.tiny Out[10]: array(3.362103143112093506263e-4932, dtype=float128) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthieu.brucher at gmail.com Mon Mar 10 13:44:32 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 10 Mar 2008 18:44:32 +0100 Subject: [SciPy-dev] numpy.float128 absence yield bugs In-Reply-To: <47D571E6.3090107@scipy.org> References: <47D52D2D.8040409@scipy.org> <47D56B6A.80604@enthought.com> <47D571E6.3090107@scipy.org> Message-ID: You mean like : numpy.float64.nbytes and : numpy.finfo(numpy.float64) ? Matthieu 2008/3/10, dmitrey : > > As for numpy.longdouble (and related types if they are intended to be > implemented), I suppose numpy also should have a function to return > number of bites used (128, 96, 64 etc). > > Ok. Another one question: does numpy have function(s), that accepts a > numpy type (float32, float64, uint16 etc) and yields: max value (like > 1e300); min value (like -1e300); min positive value (for floats) like > 1e-300? > > Thank you in advance, D. > > > > Travis E. Oliphant wrote: > > dmitrey wrote: > > > >> hi all, > >> if numpy.float128 type is absent (BTW does it related to OS or > something > >> else?), it yields bugs even if the type is never used (while importing > >> from a py-file). For example: > >> > >> > > > > I suggest using numpy.longdouble if you are interested in cross-platform > > use of float128. There will always be a longdouble, but on some > > platforms it is 128-bit, on others 96-bit, and on others 64-bit (i.e. > > the same as double). > > > > -Travis O. > > > > _______________________________________________ > > Scipy-dev mailing list > > Scipy-dev at scipy.org > > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > > > > > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanandel at ucar.edu Mon Mar 10 17:40:36 2008 From: vanandel at ucar.edu (Joseph VanAndel) Date: Mon, 10 Mar 2008 15:40:36 -0600 Subject: [SciPy-dev] building SciPy for CentOS 5 Message-ID: <47D5AAD4.6020102@ucar.edu> I'm trying to build SciPy for Python 2.5.1 on CentOS 5, and am having quite a few problems. First, I uninstalled any existing blas and lapack RPMs, since the existing RPMs are pretty old. I've built and installed refblas3, lapack3, and atlas, using the source RPMS from http://download.opensuse.org/repositories/home:/ashigabou/Fedora_7/src/ When I install the RPMS, I use the '--relocate' option to install the files in /usr/local, rather than /usr. (Our system admins prefer we not install 'unoffical' RPMS into /usr) I've installed numpy, using easy_install. I've edited site-packages/numpy/distutils/site.cfg to match where the blas and lapack libraries got installed. When I try to build scipy, it complains about various missing packages, but still builds, and installs files into /site-packages. (I've attached the log file). However, scipy doesn't work: import scipy Traceback (most recent call last): File "", line 1, in File "scipy/__init__.py", line 54, in from __config__ import show as show_config ImportError: No module named __config__ I'm an experienced programmer, and I've already invested 10+ hours in trying to install scipy. I'm a bit frustrated with the documentation and the build/install procedures - it just shouldn't be this hard to install scipy! I'd be happy to contribute detailed documentation (and RPMS) for build with CentOS 5, once I get a working system. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: setup.log URL: From robert.kern at gmail.com Mon Mar 10 19:07:14 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 10 Mar 2008 18:07:14 -0500 Subject: [SciPy-dev] building SciPy for CentOS 5 In-Reply-To: <47D5AAD4.6020102@ucar.edu> References: <47D5AAD4.6020102@ucar.edu> Message-ID: <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> On Mon, Mar 10, 2008 at 4:40 PM, Joseph VanAndel wrote: > I'm trying to build SciPy for Python 2.5.1 on CentOS 5, and am having > quite a few problems. > > First, I uninstalled any existing blas and lapack RPMs, since the > existing RPMs are pretty old. > > I've built and installed refblas3, lapack3, and atlas, using the source > RPMS from > > http://download.opensuse.org/repositories/home:/ashigabou/Fedora_7/src/ > > When I install the RPMS, I use the '--relocate' option to install the > files in /usr/local, rather than /usr. (Our system admins prefer we not > install 'unoffical' RPMS into /usr) > > I've installed numpy, using easy_install. > > I've edited site-packages/numpy/distutils/site.cfg to match where the > blas and lapack libraries got installed. > > When I try to build scipy, it complains about various missing packages, > but still builds, and installs files into /site-packages. > (I've attached the log file). > > However, scipy doesn't work: > > import scipy > Traceback (most recent call last): > File "", line 1, in > File "scipy/__init__.py", line 54, in > from __config__ import show as show_config > ImportError: No module named __config__ Change to another directory. Python looks in the current directory for modules and packages before it looks in site-packages, so you are seeing the incomplete source tree rather than the installed scipy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dmitrey.kroshko at scipy.org Tue Mar 11 04:50:56 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Tue, 11 Mar 2008 10:50:56 +0200 Subject: [SciPy-dev] returning string by f2py Message-ID: <47D647F0.7020401@scipy.org> hi all, I have modified bvls.f file to make it good enough to be connected to scipy (add istop and msg - integer & string description of stop case). However, I always get same msg value that I pass to the routine (via f2py). Could anyone check the modified bvls.f file? http://scipy.org/scipy/scikits/attachment/wiki/OOFiles/bvls.f (Or mb something should be modified in Python code?) Thank you in advance, D. From vanandel at ucar.edu Tue Mar 11 08:59:45 2008 From: vanandel at ucar.edu (Joseph VanAndel) Date: Tue, 11 Mar 2008 06:59:45 -0600 Subject: [SciPy-dev] building SciPy for CentOS 5 In-Reply-To: <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> Message-ID: <47D68241.4090509@ucar.edu> Robert, thanks for your help. Now, when I run python2.5 >> import scipy >> scipy.test(level=1, verbosity=2) ======================================= Error 1: I get the message: WARNING: cblas module is empty As I mentioned earlier, I build Atlas from the source RPMS found here: http://download.opensuse.org/repositories/home:/ashigabou/Fedora_7/src/ There is no mention of cblas in the atlas.spec file, so I don't see how the Atlas SRPM could supply cblas . ====================================== Error 2: python2.5.1 crashes : check_complex_dotc (scipy.linalg.tests.test_blas.test_fblas1_simple)Segmentation fault Robert Kern wrote: > Change to another directory. Python looks in the current directory for > modules and packages before it looks in site-packages, so you are > seeing the incomplete source tree rather than the installed scipy. > From pearu at cens.ioc.ee Tue Mar 11 09:14:37 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Tue, 11 Mar 2008 15:14:37 +0200 (EET) Subject: [SciPy-dev] returning string by f2py In-Reply-To: <47D647F0.7020401@scipy.org> References: <47D647F0.7020401@scipy.org> Message-ID: <51089.129.240.228.26.1205241277.squirrel@cens.ioc.ee> On Tue, March 11, 2008 10:50 am, dmitrey wrote: > hi all, > I have modified bvls.f file to make it good enough to be connected to > scipy (add istop and msg - integer & string description of stop case). > > However, I always get same msg value that I pass to the routine (via > f2py). > > Could anyone check the modified bvls.f file? > http://scipy.org/scipy/scikits/attachment/wiki/OOFiles/bvls.f > > (Or mb something should be modified in Python code?) Can you give an example how to create the wrapper and how to you use it from Python? Until that I can only suggest using intent(out) for msg argument. Pearu From david at ar.media.kyoto-u.ac.jp Tue Mar 11 09:10:18 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 11 Mar 2008 22:10:18 +0900 Subject: [SciPy-dev] building SciPy for CentOS 5 In-Reply-To: <47D68241.4090509@ucar.edu> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> Message-ID: <47D684BA.1070206@ar.media.kyoto-u.ac.jp> Joseph VanAndel wrote: > Robert, thanks for your help. > > Now, when I run > python2.5 > >> import scipy > >> scipy.test(level=1, verbosity=2) > > ======================================= > > Error 1: > I get the message: > WARNING: cblas module is empty > > > > > As I mentioned earlier, I build Atlas from the source RPMS found here: > http://download.opensuse.org/repositories/home:/ashigabou/Fedora_7/src/ > > > There is no mention of cblas in the atlas.spec file, so I don't see how > the Atlas SRPM could supply cblas . > > ====================================== > > Error 2: python2.5.1 crashes : > > check_complex_dotc > (scipy.linalg.tests.test_blas.test_fblas1_simple) What is the fortran compiler on CENTOS 5 ? I bet you built scipy with g77, but ATLAS for FC 7 are built with gfortran. Could you provide the scipy build log ? cheers, David From vanandel at ucar.edu Tue Mar 11 09:59:05 2008 From: vanandel at ucar.edu (Joseph VanAndel) Date: Tue, 11 Mar 2008 07:59:05 -0600 Subject: [SciPy-dev] building SciPy for CentOS 5 In-Reply-To: <47D684BA.1070206@ar.media.kyoto-u.ac.jp> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> Message-ID: <47D69029.4040204@ucar.edu> I built ATLAS from source (using the ashigabou FC 7 SRPMS), using gfortran 4.1.2 Is the ashigabou SRPM supposed to provide 'cblas'? Should I use http://math-atlas.sourceforge.net instead? How do I specify which fortran compiler is used to build scipy? I've attached the scipy log file. David Cournapeau wrote: > Joseph VanAndel wrote: >> Robert, thanks for your help. >> >> Now, when I run >> python2.5 >> >> import scipy >> >> scipy.test(level=1, verbosity=2) >> >> ======================================= >> >> Error 1: >> I get the message: >> WARNING: cblas module is empty >> >> >> >> >> As I mentioned earlier, I build Atlas from the source RPMS found here: >> http://download.opensuse.org/repositories/home:/ashigabou/Fedora_7/src/ >> >> >> There is no mention of cblas in the atlas.spec file, so I don't see how >> the Atlas SRPM could supply cblas . >> >> ====================================== >> >> Error 2: python2.5.1 crashes : >> >> check_complex_dotc >> (scipy.linalg.tests.test_blas.test_fblas1_simple) > > What is the fortran compiler on CENTOS 5 ? I bet you built scipy with > g77, but ATLAS for FC 7 are built with gfortran. Could you provide the > scipy build log ? > > cheers, > > David > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: scipy.build.log URL: From david at ar.media.kyoto-u.ac.jp Tue Mar 11 09:58:33 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 11 Mar 2008 22:58:33 +0900 Subject: [SciPy-dev] building SciPy for CentOS 5 In-Reply-To: <47D69029.4040204@ucar.edu> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> <47D69029.4040204@ucar.edu> Message-ID: <47D69009.2040501@ar.media.kyoto-u.ac.jp> Joseph VanAndel wrote: > I built ATLAS from source (using the ashigabou FC 7 SRPMS), using > gfortran 4.1.2 > > If I remember well, I forced using gfortran for FC 7. > Is the ashigabou SRPM supposed to provide 'cblas'? Not really, because it is not needed. The problem is that ATLAS is basically impossible to build in a reproducible way, so I do not provide binary rpm. The idea was that you install RPM for scipy, and then build ATLAS from the SRPM on your machine, which can be used as a drop-in by scipy (ATLAS is picked up instead of BLAS/LAPACK at runtime). > > Should I use http://math-atlas.sourceforge.net instead? It is your choice, but using the rpms is way easier. I have not tested on CENTOS, what is the difference between FC and CENTOS ? I built those rpms, but I do not use those distributions, so I do not necessarily know how to do thing the right way. > > How do I specify which fortran compiler is used to build scipy? This will force using gfortran: python setup.py build --fcompiler=gnu95 But this is taken care of by the scipy RPMS on ashigabou.... > > I've attached the scipy log file. The log is incomplete: you should give us a complete log (that is, with the build directory deleted first), because otherwise, most info are not there. cheers, David From dmitrey.kroshko at scipy.org Tue Mar 11 10:45:04 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Tue, 11 Mar 2008 16:45:04 +0200 Subject: [SciPy-dev] returning string by f2py In-Reply-To: <51089.129.240.228.26.1205241277.squirrel@cens.ioc.ee> References: <47D647F0.7020401@scipy.org> <51089.129.240.228.26.1205241277.squirrel@cens.ioc.ee> Message-ID: <47D69AF0.80304@scipy.org> Pearu Peterson wrote: > On Tue, March 11, 2008 10:50 am, dmitrey wrote: > >> hi all, >> I have modified bvls.f file to make it good enough to be connected to >> scipy (add istop and msg - integer & string description of stop case). >> >> However, I always get same msg value that I pass to the routine (via >> f2py). >> >> Could anyone check the modified bvls.f file? >> http://scipy.org/scipy/scikits/attachment/wiki/OOFiles/bvls.f >> >> (Or mb something should be modified in Python code?) >> > > Can you give an example how to create the wrapper > and how to you use it from Python? Until that I can only > suggest using intent(out) for msg argument. > > Pearu > > here's (same) test file attached; as for f2py I just use f2py -c -m bvls bvls.f I think it's better to have default f2py using (w/o any additional wrap files like .pyf), and make all required changes in py-file that will call compiled f-file. I intend to make it available immediately from scikits.openopt LLSP webpage (for download and using from OO) and then connect it to scipy.optimize (also w/o auxiliary wrapper files). Currently it returns same values as have been passed: istop=0 and same msg. Regards, D. -------------- next part -------------- A non-text attachment was scrubbed... Name: test_bvls.py Type: text/x-python Size: 1284 bytes Desc: not available URL: From vanandel at ucar.edu Tue Mar 11 11:02:50 2008 From: vanandel at ucar.edu (Joseph VanAndel) Date: Tue, 11 Mar 2008 09:02:50 -0600 Subject: [SciPy-dev] building SciPy for CentOS 5 In-Reply-To: <47D69009.2040501@ar.media.kyoto-u.ac.jp> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> <47D69029.4040204@ucar.edu> <47D69009.2040501@ar.media.kyoto-u.ac.jp> Message-ID: <47D69F1A.9030304@ucar.edu> I rebuilt scipy, using g95 (thanks David!). Now, the scipy test doesn't crash, but has two failures. Any ideas on why these Orthogonal Distance Regression (ODRPACK) tests fail? ====================================================================== FAIL: test_explicit (scipy.tests.test_odr.test_odr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/net/opt_lnx/local_el5_64/lib/python2.5/site-packages/scipy/odr/tests/test_odr.py", line 50, in test_explicit -8.7849712165253724e-02]), File "/opt/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 230, in assert_array_almost_equal header='Arrays are not almost equal') File "/opt/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 215, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 1.26462971e+03, -5.42545890e+01, -8.64250389e-02]) y: array([ 1.26465481e+03, -5.40184100e+01, -8.78497122e-02]) ====================================================================== FAIL: test_multi (scipy.tests.test_odr.test_odr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/net/opt_lnx/local_el5_64/lib/python2.5/site-packages/scipy/odr/tests/test_odr.py", line 191, in test_multi 0.5101147161764654, 0.5173902330489161]), File "/opt/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 230, in assert_array_almost_equal header='Arrays are not almost equal') File "/opt/local/lib/python2.5/site-packages/numpy/testing/utils.py", line 215, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 4.31272063, 2.44289312, 7.76215871, 0.55995622, 0.46423343]) y: array([ 4.37998803, 2.43330576, 8.00288459, 0.51011472, 0.51739023]) ---------------------------------------------------------------------- Ran 1716 tests in 3.270s FAILED (failures=2) David Cournapeau wrote: > Joseph VanAndel wrote: >> I built ATLAS from source (using the ashigabou FC 7 SRPMS), using >> gfortran 4.1.2 >> >> > > If I remember well, I forced using gfortran for FC 7. > >> Is the ashigabou SRPM supposed to provide 'cblas'? > Not really, because it is not needed. The problem is that ATLAS is > basically impossible to build in a reproducible way, so I do not provide > binary rpm. The idea was that you install RPM for scipy, and then build > ATLAS from the SRPM on your machine, which can be used as a drop-in by > scipy (ATLAS is picked up instead of BLAS/LAPACK at runtime). > >> Should I use http://math-atlas.sourceforge.net instead? > It is your choice, but using the rpms is way easier. I have not tested > on CENTOS, what is the difference between FC and CENTOS ? I built those > rpms, but I do not use those distributions, so I do not necessarily know > how to do thing the right way. >> How do I specify which fortran compiler is used to build scipy? > This will force using gfortran: > > python setup.py build --fcompiler=gnu95 > > But this is taken care of by the scipy RPMS on ashigabou.... >> I've attached the scipy log file. > The log is incomplete: you should give us a complete log (that is, with > the build directory deleted first), because otherwise, most info are not > there. > > cheers, > > David > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev From phaustin at gmail.com Tue Mar 11 12:14:24 2008 From: phaustin at gmail.com (Phil Austin) Date: Tue, 11 Mar 2008 09:14:24 -0700 Subject: [SciPy-dev] building SciPy for CentOS 5 In-Reply-To: <47D69F1A.9030304@ucar.edu> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> <47D69029.4040204@ucar.edu> <47D69009.2040501@ar.media.kyoto-u.ac.jp> <47D69F1A.9030304@ucar.edu> Message-ID: <47D6AFE0.8070201@gmail.com> Joseph VanAndel wrote: > I rebuilt scipy, using g95 (thanks David!). > > Any ideas on why these Orthogonal Distance Regression (ODRPACK) tests fail? All these pass for me with Centos 5.1 test_explicit (test_odr.TestODR) ... ok test_implicit (test_odr.TestODR) ... ok test_lorentz (test_odr.TestODR) ... ok test_multi (test_odr.TestODR) ... ok test_pearson (test_odr.TestODR) ... ok with Python 2.5.1 (all code compiled from source) svn.version.version '0.7.0.dev3893' on x86_64 and Python 2.5.2 '0.7.0.dev4004' on i686 From slesarev.anton at gmail.com Tue Mar 11 14:50:32 2008 From: slesarev.anton at gmail.com (Anton Slesarev) Date: Tue, 11 Mar 2008 21:50:32 +0300 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> Message-ID: <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> Hi all, it might be a good idea to have a machine learning(ML) package in SciPy. As I understand there are some ML code in SciKits, but it is in raw state? There are a lot of machine learning projects, with its own data format, number of classifiers, feature selection algorithms and benchmarks. But if you want to compare your own algorithm with some others, you should convert your data format to input format of every tool you want to use and after training, you should convert output format of each tools to the single format to have facility to compare results(for example you want to see common which features was selected by different tools). Now I'm analyzing different ML approaches for the special case of text classification problem. I couldn't find ML framework appropriate for my task. I've got two simple requirements for this framework. It should support sparse data format and has at least svm classifier. For example, Orange [1] is a vary good data mining project but has poor sparse format support. PyML [2] has all needed features, but there are problems with installation on different platforms and code design is not perfect. I believe that creation framework, which will be convenient for scientist to integrate their algorithms to it, is a vary useful challenge. Scientists often talk about standard machine learning software[3] and may be SciPy will be appropriate platform for developing such software. I can write detailed proposal, but I want to see is it interesting for someone? Any wishes and recommendations? 1. Orange http://magix.fri.uni-lj.si/orange/ 2. PyML http://pyml.sourceforge.net/ 3. The Need for Open Source Software in Machine Learning http://www.jmlr.org/papers/volume8/sonnenburg07a/sonnenburg07a.pdf -- Anton Slesarev -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Tue Mar 11 15:05:24 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 11 Mar 2008 20:05:24 +0100 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> Message-ID: Hi, David Cournapeau is maintaining the learn scikit. This is the main place where machine learning code will be put. For instance, there are classifiers (SVMs with libsvm) and there will be in the near future the more used manifold learning techniques. I didn't understand what you meant by "you want to see common which features was selected by different tools". Sparse matrix support must be made at the C level for libsvm, you would have to ask Albert who wrapped libsvm. For the manifold learning code, techniques that can support sparse matrices support them (for instance Laplacian Eigenmaps). Matthieu 2008/3/11, Anton Slesarev : > > Hi all, > > it might be a good idea to have a machine learning(ML) package in SciPy. > As I understand there are some ML code in SciKits, but it is in raw state? > > There are a lot of machine learning projects, with its own data format, > number of classifiers, feature selection algorithms and benchmarks. But if > you want to compare your own algorithm with some others, you should convert > your data format to input format of every tool you want to use and after > training, you should convert output format of each tools to the single > format to have facility to compare results(for example you want to see > common which features was selected by different tools). > > Now I'm analyzing different ML approaches for the special case of text > classification problem. I couldn't find ML framework appropriate for my > task. I've got two simple requirements for this framework. It should support > sparse data format and has at least svm classifier. For example, Orange [1] > is a vary good data mining project but has poor sparse format support. PyML > [2] has all needed features, but there are problems with installation on > different platforms and code design is not perfect. > > I believe that creation framework, which will be convenient for scientist > to integrate their algorithms to it, is a vary useful challenge. Scientists > often talk about standard machine learning software[3] and may be SciPy will > be appropriate platform for developing such software. > > I can write detailed proposal, but I want to see is it interesting for > someone? Any wishes and recommendations? > > 1. Orange http://magix.fri.uni-lj.si/orange/ > 2. PyML http://pyml.sourceforge.net/ > 3. The Need for Open Source Software in Machine Learning > http://www.jmlr.org/papers/volume8/sonnenburg07a/sonnenburg07a.pdf > > -- > Anton Slesarev > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanandel at ucar.edu Tue Mar 11 15:08:50 2008 From: vanandel at ucar.edu (Joseph VanAndel) Date: Tue, 11 Mar 2008 13:08:50 -0600 Subject: [SciPy-dev] time to import scipy? In-Reply-To: <47D6AFE0.8070201@gmail.com> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> <47D69029.4040204@ucar.edu> <47D69009.2040501@ar.media.kyoto-u.ac.jp> <47D69F1A.9030304@ucar.edu> <47D6AFE0.8070201@gmail.com> Message-ID: <47D6D8C2.8000603@ucar.edu> When I import scipy (or just scipy.io) on a CentOS 5 64 bit computer, 4 processor, 3.00 Ghz server, with Python 2.5.1, it takes ~10 seconds of elapsed time. When I'm developing and testing a Python script, I typically start the python interpreter dozens of times, so I concerned by how much time I spend waiting for the Python to startup. 1) Is there anything I can do to speed up the import of scipy? 2) What debugging techniques do you suggest that avoid restarting the interpreter? For example, do you routinely use ipython and reload(my_module) From slesarev.anton at gmail.com Tue Mar 11 15:19:19 2008 From: slesarev.anton at gmail.com (Anton Slesarev) Date: Tue, 11 Mar 2008 22:19:19 +0300 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> Message-ID: <11abf2720803111219k8064254h67dd9002dcf55199@mail.gmail.com> > > David Cournapeau is maintaining the learn scikit. This is the main place > where machine learning code will be put. > For instance, there are classifiers (SVMs with libsvm) and there will be > in the near future the more used manifold learning techniques. > > I didn't understand what you meant by "you want to see common which > features was selected by different tools". I mean that if we have standard format for different classifiers we can compare their results, we can see intersection of features that have been selected. If we use different tools, it is need to make exhausting conversions between different formats. > > Sparse matrix support must be made at the C level for libsvm, you would > have to ask Albert who wrapped libsvm. I see. I say that it is good idea to write parsers for different data formats. > > For the manifold learning code, techniques that can support sparse > matrices support them (for instance Laplacian Eigenmaps). > > Matthieu > > 2008/3/11, Anton Slesarev : > > > > Hi all, > > > > it might be a good idea to have a machine learning(ML) package in SciPy. > > As I understand there are some ML code in SciKits, but it is in raw state? > > > > There are a lot of machine learning projects, with its own data format, > > number of classifiers, feature selection algorithms and benchmarks. But if > > you want to compare your own algorithm with some others, you should convert > > your data format to input format of every tool you want to use and after > > training, you should convert output format of each tools to the single > > format to have facility to compare results(for example you want to see > > common which features was selected by different tools). > > > > Now I'm analyzing different ML approaches for the special case of text > > classification problem. I couldn't find ML framework appropriate for my > > task. I've got two simple requirements for this framework. It should support > > sparse data format and has at least svm classifier. For example, Orange [1] > > is a vary good data mining project but has poor sparse format support. PyML > > [2] has all needed features, but there are problems with installation on > > different platforms and code design is not perfect. > > > > I believe that creation framework, which will be convenient for > > scientist to integrate their algorithms to it, is a vary useful challenge. > > Scientists often talk about standard machine learning software[3] and may be > > SciPy will be appropriate platform for developing such software. > > > > I can write detailed proposal, but I want to see is it interesting for > > someone? Any wishes and recommendations? > > > > 1. Orange http://magix.fri.uni-lj.si/orange/ > > 2. PyML http://pyml.sourceforge.net/ > > 3. The Need for Open Source Software in Machine Learning > > http://www.jmlr.org/papers/volume8/sonnenburg07a/sonnenburg07a.pdf > > > > -- > > Anton Slesarev > > _______________________________________________ > > Scipy-dev mailing list > > Scipy-dev at scipy.org > > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > > > > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > -- Anton Slesarev -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Tue Mar 11 15:30:53 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Tue, 11 Mar 2008 20:30:53 +0100 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: <11abf2720803111219k8064254h67dd9002dcf55199@mail.gmail.com> References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <11abf2720803111219k8064254h67dd9002dcf55199@mail.gmail.com> Message-ID: 2008/3/11, Anton Slesarev : > > David Cournapeau is maintaining the learn scikit. This is the main place > > where machine learning code will be put. > > For instance, there are classifiers (SVMs with libsvm) and there will be > > in the near future the more used manifold learning techniques. > > > > I didn't understand what you meant by "you want to see common which > > features was selected by different tools". > > > I mean that if we have standard format for different classifiers we can > compare their results, we can see intersection of features that have been > selected. If we use different tools, it is need to make exhausting > conversions between different formats. > Well, a classifier gives back the class to which a point belongs to, doesn't it ? Do you mean adding probabilities ? It would depend on the kind of the classifier (strong or fuzzy). For the time being, David proposed a data format, at least for inputs. The same format could be extended to output values (when it has sense). Sparse matrix support must be made at the C level for libsvm, you should > > have to ask Albert who wrapped libsvm. > > I see. I say that it is good idea to write parsers for different data > formats. > I agree, I hope that David will be able to finish the Scons build of numpy and scipy so that he will be able to enhance his data format proposal :) Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From ondrej at certik.cz Tue Mar 11 20:27:48 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 12 Mar 2008 01:27:48 +0100 Subject: [SciPy-dev] time to import scipy? In-Reply-To: <47D6D8C2.8000603@ucar.edu> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> <47D69029.4040204@ucar.edu> <47D69009.2040501@ar.media.kyoto-u.ac.jp> <47D69F1A.9030304@ucar.edu> <47D6AFE0.8070201@gmail.com> <47D6D8C2.8000603@ucar.edu> Message-ID: <85b5c3130803111727q70d607e0j2b0af500d1550518@mail.gmail.com> On Tue, Mar 11, 2008 at 8:08 PM, Joseph VanAndel wrote: > When I import scipy (or just scipy.io) on a CentOS 5 64 bit computer, 4 > processor, 3.00 Ghz server, with Python 2.5.1, it takes ~10 seconds of > elapsed time. > > When I'm developing and testing a Python script, I typically start the > python interpreter dozens of times, so I concerned by how much time I > spend waiting for the Python to startup. > > 1) Is there anything I can do to speed up the import of scipy? > > 2) What debugging techniques do you suggest that avoid restarting the > interpreter? For example, do you routinely use ipython and > reload(my_module) Which scipy version are you using? It takes 0.33s on my laptop and scipy in Debian: $ ipython Python 2.4.4 (#2, Mar 3 2008, 01:16:37) Type "copyright", "credits" or "license" for more information. IPython 0.8.2 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [1]: time import scipy CPU times: user 0.24 s, sys: 0.07 s, total: 0.31 s Wall time: 0.33 s Ondrej From vanandel at ucar.edu Tue Mar 11 21:07:57 2008 From: vanandel at ucar.edu (Joseph VanAndel) Date: Tue, 11 Mar 2008 19:07:57 -0600 Subject: [SciPy-dev] time to import scipy? In-Reply-To: <85b5c3130803111727q70d607e0j2b0af500d1550518@mail.gmail.com> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> <47D69029.4040204@ucar.edu> <47D69009.2040501@ar.media.kyoto-u.ac.jp> <47D69F1A.9030304@ucar.edu> <47D6AFE0.8070201@gmail.com> <47D6D8C2.8000603@ucar.edu> <85b5c3130803111727q70d607e0j2b0af500d1550518@mail.gmail.com> Message-ID: <47D72CED.8090903@ucar.edu> I'm using scipy 0.6.0 Ondrej Certik wrote: > On Tue, Mar 11, 2008 at 8:08 PM, Joseph VanAndel wrote: >> When I import scipy (or just scipy.io) on a CentOS 5 64 bit computer, 4 >> processor, 3.00 Ghz server, with Python 2.5.1, it takes ~10 seconds of >> elapsed time. >> >> When I'm developing and testing a Python script, I typically start the >> python interpreter dozens of times, so I concerned by how much time I >> spend waiting for the Python to startup. >> >> 1) Is there anything I can do to speed up the import of scipy? >> >> 2) What debugging techniques do you suggest that avoid restarting the >> interpreter? For example, do you routinely use ipython and >> reload(my_module) > > Which scipy version are you using? It takes 0.33s on my laptop and > scipy in Debian: > > $ ipython > Python 2.4.4 (#2, Mar 3 2008, 01:16:37) > Type "copyright", "credits" or "license" for more information. > > IPython 0.8.2 -- An enhanced Interactive Python. > ? -> Introduction and overview of IPython's features. > %quickref -> Quick reference. > help -> Python's own help system. > object? -> Details about 'object'. ?object also works, ?? prints more. > > In [1]: time import scipy > CPU times: user 0.24 s, sys: 0.07 s, total: 0.31 s > Wall time: 0.33 s > > Ondrej > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev From david at ar.media.kyoto-u.ac.jp Tue Mar 11 22:30:45 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 12 Mar 2008 11:30:45 +0900 Subject: [SciPy-dev] time to import scipy? In-Reply-To: <47D6D8C2.8000603@ucar.edu> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> <47D69029.4040204@ucar.edu> <47D69009.2040501@ar.media.kyoto-u.ac.jp> <47D69F1A.9030304@ucar.edu> <47D6AFE0.8070201@gmail.com> <47D6D8C2.8000603@ucar.edu> Message-ID: <47D74055.4040205@ar.media.kyoto-u.ac.jp> Joseph VanAndel wrote: > When I import scipy (or just scipy.io) on a CentOS 5 64 bit computer, 4 > processor, 3.00 Ghz server, with Python 2.5.1, it takes ~10 seconds of > elapsed time. > > When I'm developing and testing a Python script, I typically start the > python interpreter dozens of times, so I concerned by how much time I > spend waiting for the Python to startup. > > 1) Is there anything I can do to speed up the import of scipy? > Where is scipy (and its dependencies) installed ? Is it on NFS (or slow filesystem, usb, etc...) ? If so, putting it locally could be a huge win. 10 second for scipy the first time (cold start) is not surprising: scipy is a big package, and most time is spent on IO (reading files from the fs). But the 2d time, a lot of this is in memory (IO buffer), so it will be very fast. Numbers given by Ondrej suggest that it is a hot start in his case (or that he has an extremely powerful workstation:) ). In my case (and scipy is on NFS): (cold start) time python -c "import scipy" real 0m4.007s user 0m0.440s sys 0m0.236s (hot start) time python -c "import scipy" real 0m0.867s user 0m0.388s sys 0m0.212s > 2) What debugging techniques do you suggest that avoid restarting the > interpreter? For example, do you routinely use ipython and > reload(my_module) It depends on the situation. If you are debugging one script, ipython has the %run method for that. But if you need to modify several modules which import each other, it becomes tricky. cheers, David From peridot.faceted at gmail.com Tue Mar 11 22:52:05 2008 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 11 Mar 2008 22:52:05 -0400 Subject: [SciPy-dev] time to import scipy? In-Reply-To: <47D74055.4040205@ar.media.kyoto-u.ac.jp> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> <47D69029.4040204@ucar.edu> <47D69009.2040501@ar.media.kyoto-u.ac.jp> <47D69F1A.9030304@ucar.edu> <47D6AFE0.8070201@gmail.com> <47D6D8C2.8000603@ucar.edu> <47D74055.4040205@ar.media.kyoto-u.ac.jp> Message-ID: On 11/03/2008, David Cournapeau wrote: > > 1) Is there anything I can do to speed up the import of scipy? > > > > > Where is scipy (and its dependencies) installed ? Is it on NFS (or slow > filesystem, usb, etc...) ? If so, putting it locally could be a huge win. Linking time may be an issue as well. Do the "prelink" tools for C libraries help at all with scipy? Or, how much of the load time is loading compiled python code, and how much is loading shared libraries? For the OP: is python perhaps recompiling the .py files every time? Normally python produces a .pyc to accompany every .py file, but if the installation directory is not writeable, this may not be possible. Normally installation of python packages premakes all these files, but if you have installed it in some unusual fashion, they could be missing. Anne From stefan at sun.ac.za Tue Mar 11 23:01:44 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 11 Mar 2008 20:01:44 -0700 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com> References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com> Message-ID: <9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com> On Tue, Mar 11, 2008 at 11:50 AM, Anton Slesarev wrote: > Hi all, > > it might be a good idea to have a machine learning(ML) package in SciPy. As > I understand there are some ML code in SciKits, but it is in raw state? > > There are a lot of machine learning projects, with its own data format, > number of classifiers, feature selection algorithms and benchmarks. But if > you want to compare your own algorithm with some others, you should convert > your data format to input format of every tool you want to use and after > training, you should convert output format of each tools to the single > format to have facility to compare results(for example you want to see > common which features was selected by different tools). > > Now I'm analyzing different ML approaches for the special case of text > classification problem. I couldn't find ML framework appropriate for my > task. I've got two simple requirements for this framework. It should support > sparse data format and has at least svm classifier. For example, Orange [1] > is a vary good data mining project but has poor sparse format support. PyML > [2] has all needed features, but there are problems with installation on > different platforms and code design is not perfect. > > I believe that creation framework, which will be convenient for scientist to > integrate their algorithms to it, is a vary useful challenge. Scientists > often talk about standard machine learning software[3] and may be SciPy will > be appropriate platform for developing such software. > > I can write detailed proposal, but I want to see is it interesting for > someone? Any wishes and recommendations? > > 1. Orange http://magix.fri.uni-lj.si/orange/ > 2. PyML http://pyml.sourceforge.net/ > 3. The Need for Open Source Software in Machine Learning > http://www.jmlr.org/papers/volume8/sonnenburg07a/sonnenburg07a.pdf I also recently learned of Elefant, http://elefant.developer.nicta.com.au/ but haven't had a chance to investigate in more detail. Regards St?fan From david at ar.media.kyoto-u.ac.jp Wed Mar 12 00:17:45 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 12 Mar 2008 13:17:45 +0900 Subject: [SciPy-dev] RHEL 5 and CENTOS 5 rpms for blas/lapack/numpy/scipy available on ashigabou Message-ID: <47D75969.2040204@ar.media.kyoto-u.ac.jp> Hi, Since some people had problems with RHEL/CENTOS, and since the opensuse build system does provide facilities to build rpms for RHEL and CENTOS for some time, I quickly updated the ashigabou repository to handle those distributions. I also added opensuse 10.3 and FC 8, but those did not require any changes: http://download.opensuse.org/repositories/home:/ashigabou/ (note that it may take time for the rpms to appear there from the time they successfully build on the compiler farm, which they just did). cheers, David From ondrej at certik.cz Wed Mar 12 06:32:45 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 12 Mar 2008 11:32:45 +0100 Subject: [SciPy-dev] time to import scipy? In-Reply-To: <47D74055.4040205@ar.media.kyoto-u.ac.jp> References: <47D5AAD4.6020102@ucar.edu> <3d375d730803101607k7d74ef70nc00294714e6b9d72@mail.gmail.com> <47D68241.4090509@ucar.edu> <47D684BA.1070206@ar.media.kyoto-u.ac.jp> <47D69029.4040204@ucar.edu> <47D69009.2040501@ar.media.kyoto-u.ac.jp> <47D69F1A.9030304@ucar.edu> <47D6AFE0.8070201@gmail.com> <47D6D8C2.8000603@ucar.edu> <47D74055.4040205@ar.media.kyoto-u.ac.jp> Message-ID: <85b5c3130803120332m6b859f55v68fed52395ae79a7@mail.gmail.com> On Wed, Mar 12, 2008 at 3:30 AM, David Cournapeau wrote: > Joseph VanAndel wrote: > > When I import scipy (or just scipy.io) on a CentOS 5 64 bit computer, 4 > > processor, 3.00 Ghz server, with Python 2.5.1, it takes ~10 seconds of > > elapsed time. > > > > When I'm developing and testing a Python script, I typically start the > > python interpreter dozens of times, so I concerned by how much time I > > spend waiting for the Python to startup. > > > > 1) Is there anything I can do to speed up the import of scipy? > > > > Where is scipy (and its dependencies) installed ? Is it on NFS (or slow > filesystem, usb, etc...) ? If so, putting it locally could be a huge win. > > 10 second for scipy the first time (cold start) is not surprising: scipy > is a big package, and most time is spent on IO (reading files from the > fs). But the 2d time, a lot of this is in memory (IO buffer), so it will > be very fast. Numbers given by Ondrej suggest that it is a hot start in > his case (or that he has an extremely powerful workstation:) ). > > In my case (and scipy is on NFS): > > (cold start) > time python -c "import scipy" > > real 0m4.007s > user 0m0.440s > sys 0m0.236s > > (hot start) > time python -c "import scipy" > > real 0m0.867s > user 0m0.388s > sys 0m0.212s Yes, my numbers are for "hot start", because that is what is used when developing with scipy. Ondrej From travis at enthought.com Wed Mar 12 11:36:20 2008 From: travis at enthought.com (Travis Vaught) Date: Wed, 12 Mar 2008 10:36:20 -0500 Subject: [SciPy-dev] ANN: EuroSciPy 2008 Conference - Leipzig, Germany Message-ID: <1FA8105E-095B-4610-ABE8-57EE9D711AE4@enthought.com> Greetings, We're pleased to announce the EuroSciPy 2008 Conference to be held in Leipzig, Germany on July 26-27, 2008. http://www.scipy.org/EuroSciPy2008 We are very excited to create a venue for the European community of users of the Python programming language in science. This conference will bring the presentations and collaboration that we've enjoyed at Caltech each year closer to home for many users of SciPy, NumPy and Python generally--with a similar focus and schedule. Call for Participation: ---------------------- If you are a scientist using Python for your computational work, we'd love to have you formally present your results, methods or experiences. To apply to present a talk at this year's EuroSciPy, please submit an abstract of your talk as a PDF, MS Word or plain text file to euroabstracts at scipy.org. The deadline for abstract submission is April 30, 2008. Papers and/or presentation slides are acceptable and are due by June 15, 2008. Presentations will be allotted 30 minutes. Registration: ------------ Registration will open April 1, 2008. The registration fee will be 100.00? for early registrants and will increase to 150.00? for late registration. Registration will include breakfast, snacks and lunch for Saturday and Sunday. Volunteers Welcome: ------------------ If you're interested in volunteering to help organize things, please email us at info at scipy.org. From doutriaux1 at llnl.gov Wed Mar 12 13:47:54 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 12 Mar 2008 10:47:54 -0700 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: <9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com> References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com> <9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com> Message-ID: <47D8174A.1080309@llnl.gov> Hi there It seems that ever since I switched to Python 2.5.2 I get (no matter which version of scipy) Any idea? C. >>> import scipy.linalg Traceback (most recent call last): File "", line 1, in File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/linalg/__init__.py", line 13, in from iterative import * File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/linalg/iterative.py", line 5, in from scipy.splinalg import isolve File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/__init__.py", line 5, in from isolve import * File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/isolve/__init__.py", line 4, in from iterative import * File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/isolve/iterative.py", line 18, in from scipy.splinalg.interface import LinearOperator File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/interface.py", line 3, in from scipy.sparse.sputils import isshape File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/__init__.py", line 5, in from base import * File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", line 45, in class spmatrix(object): File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", line 139, in spmatrix @deprecate TypeError: deprecate() takes exactly 3 arguments (1 given) From aisaac at american.edu Wed Mar 12 13:56:58 2008 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 12 Mar 2008 13:56:58 -0400 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: <47D8174A.1080309@llnl.gov> References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com><9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com><47D8174A.1080309@llnl.gov> Message-ID: On Wed, 12 Mar 2008, Charles Doutriaux apparently wrote: > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", > line 139, in spmatrix > @deprecate > TypeError: deprecate() takes exactly 3 arguments (1 given) Looks like a name clash? You should be able to just comment out the decorator with no ill effects, I would think. Cheers, Alan Isaac From matthieu.brucher at gmail.com Wed Mar 12 13:56:53 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 12 Mar 2008 18:56:53 +0100 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: <47D8174A.1080309@llnl.gov> References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com> <9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com> <47D8174A.1080309@llnl.gov> Message-ID: Hi, I had the same error (but with Python 2.5.1). This is numpy related, update your version and everything will be fine ;) Matthieu 2008/3/12, Charles Doutriaux : > > Hi there > > It seems that ever since I switched to Python 2.5.2 I get (no matter > which version of scipy) > > Any idea? > > C. > > >>> import scipy.linalg > Traceback (most recent call last): > File "", line 1, in > File > > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/linalg/__init__.py", > line 13, in > from iterative import * > File > > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/linalg/iterative.py", > line 5, in > from scipy.splinalg import isolve > File > > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/__init__.py", > line 5, in > from isolve import * > File > > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/isolve/__init__.py", > line 4, in > from iterative import * > File > > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/isolve/iterative.py", > line 18, in > from scipy.splinalg.interface import LinearOperator > File > > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/interface.py", > line 3, in > from scipy.sparse.sputils import isshape > File > > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/__init__.py", > line 5, in > from base import * > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", > line 45, in > class spmatrix(object): > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", > line 139, in spmatrix > @deprecate > TypeError: deprecate() takes exactly 3 arguments (1 given) > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From doutriaux1 at llnl.gov Wed Mar 12 14:07:22 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 12 Mar 2008 11:07:22 -0700 Subject: [SciPy-dev] depracte name clash ? In-Reply-To: References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com><9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com><47D8174A.1080309@llnl.gov> Message-ID: <47D81BDA.3070607@llnl.gov> Hi, I'm resending this with an appropriate subject in the email. Thansk Alan for your answer. It seems that ever since I switched to Python 2.5.2 I get (no matter which version of scipy) Any idea? Thanks Alan for your suggestion. I'm not sure of what's happening. numpy's deprecate function does take 3 args. So i don't know what's happening. This used to work fine in 2.5.1. What is disturbing about it, is that the error message does ask for the right number of arguments as numpy requires. I could see the message was reverted "expects 1 got 3", then i would suspect an overload. Here i'm not sure what's going on. I don't really want to start editing scipy's sources either, it's genreally the best way to run into problems and not being able to get support from the list "because it's your own hacked version" C. Alan G Isaac wrote: > On Wed, 12 Mar 2008, Charles Doutriaux apparently wrote: > >> File >> "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", >> line 139, in spmatrix >> @deprecate >> TypeError: deprecate() takes exactly 3 arguments (1 given) >> > > Looks like a name clash? > You should be able to just comment out the decorator > with no ill effects, I would think. > > Cheers, > Alan Isaac > > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > From doutriaux1 at llnl.gov Wed Mar 12 14:37:06 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 12 Mar 2008 11:37:06 -0700 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com> <9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com> <47D8174A.1080309@llnl.gov> Message-ID: <47D822D2.3060107@llnl.gov> Hum... except the latest numpy (from svn) gives me: import scipy Traceback (most recent call last): File "", line 1, in File "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/__init__.py", line 19, in from numpy import show_config as show_numpy_config File "/export/svn/Numpy/trunk/numpy/__init__.py", line 27, in ImportError: No module named __config__ I'm stuck... I guess i'll report that to numpy people.... C. Matthieu Brucher wrote: > Hi, > > I had the same error (but with Python 2.5.1). This is numpy related, > update your version and everything will be fine ;) > > Matthieu > > 2008/3/12, Charles Doutriaux >: > > Hi there > > It seems that ever since I switched to Python 2.5.2 I get (no matter > which version of scipy) > > Any idea? > > C. > > >>> import scipy.linalg > Traceback (most recent call last): > File "", line 1, in > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/linalg/__init__.py", > line 13, in > from iterative import * > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/linalg/iterative.py", > line 5, in > from scipy.splinalg import isolve > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/__init__.py", > line 5, in > from isolve import * > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/isolve/__init__.py", > line 4, in > from iterative import * > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/isolve/iterative.py", > line 18, in > from scipy.splinalg.interface import LinearOperator > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/interface.py", > line 3, in > from scipy.sparse.sputils import isshape > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/__init__.py", > line 5, in > from base import * > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", > line 45, in > class spmatrix(object): > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", > line 139, in spmatrix > @deprecate > TypeError: deprecate() takes exactly 3 arguments (1 given) > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > ------------------------------------------------------------------------ > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > From hoytak at gmail.com Wed Mar 12 14:40:33 2008 From: hoytak at gmail.com (Hoyt Koepke) Date: Wed, 12 Mar 2008 11:40:33 -0700 Subject: [SciPy-dev] depracte name clash ? In-Reply-To: <47D81BDA.3070607@llnl.gov> References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com> <9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com> <47D8174A.1080309@llnl.gov> <47D81BDA.3070607@llnl.gov> Message-ID: <4db580fd0803121140h171ade74y91bcdf9218f67a6a@mail.gmail.com> Have you tried doing a complete rebuild? I.e. rm -Rf the build/ directories in both numpy and scipy, then reinstall. --Hoyt On Wed, Mar 12, 2008 at 11:07 AM, Charles Doutriaux wrote: > Hi, > > I'm resending this with an appropriate subject in the email. Thansk > Alan for your answer. > > It seems that ever since I switched to Python 2.5.2 I get (no matter > which version of scipy) > > Any idea? > Thanks Alan for your suggestion. I'm not sure of what's happening. numpy's deprecate function does take 3 args. > > So i don't know what's happening. This used to work fine in 2.5.1. > > What is disturbing about it, is that the error message does ask for the right number of arguments as numpy requires. I could see the message was reverted "expects 1 got 3", then i would suspect an overload. > > Here i'm not sure what's going on. > > I don't really want to start editing scipy's sources either, it's genreally the best way to run into problems and not being able to get support from the list "because it's your own hacked version" > > > C. > > > Alan G Isaac wrote: > > On Wed, 12 Mar 2008, Charles Doutriaux apparently wrote: > > > >> File > >> "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", > >> line 139, in spmatrix > >> @deprecate > >> TypeError: deprecate() takes exactly 3 arguments (1 given) > >> > > > > Looks like a name clash? > > You should be able to just comment out the decorator > > with no ill effects, I would think. > > > > Cheers, > > Alan Isaac > > > > > > > > _______________________________________________ > > Scipy-dev mailing list > > Scipy-dev at scipy.org > > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > From doutriaux1 at llnl.gov Wed Mar 12 14:42:06 2008 From: doutriaux1 at llnl.gov (Charles Doutriaux) Date: Wed, 12 Mar 2008 11:42:06 -0700 Subject: [SciPy-dev] [GSoC 2008]Machine learning package in SciPy In-Reply-To: References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com> <9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com> <47D8174A.1080309@llnl.gov> Message-ID: <47D823FE.8040309@llnl.gov> Ok my last error was because i was in the trunk directory, seems to create issues. The latest versino indeed fixes the deprecate issue. BUT it breaks the oldnumeric.ma which i need too... Thanks for the suggestion, for now knowing it'll be fix in the next release satisfies me :) (as long it is not too far away) C. Matthieu Brucher wrote: > Hi, > > I had the same error (but with Python 2.5.1). This is numpy related, > update your version and everything will be fine ;) > > Matthieu > > 2008/3/12, Charles Doutriaux >: > > Hi there > > It seems that ever since I switched to Python 2.5.2 I get (no matter > which version of scipy) > > Any idea? > > C. > > >>> import scipy.linalg > Traceback (most recent call last): > File "", line 1, in > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/linalg/__init__.py", > line 13, in > from iterative import * > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/linalg/iterative.py", > line 5, in > from scipy.splinalg import isolve > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/__init__.py", > line 5, in > from isolve import * > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/isolve/__init__.py", > line 4, in > from iterative import * > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/isolve/iterative.py", > line 18, in > from scipy.splinalg.interface import LinearOperator > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/splinalg/interface.py", > line 3, in > from scipy.sparse.sputils import isshape > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/__init__.py", > line 5, in > from base import * > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", > line 45, in > class spmatrix(object): > File > "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/base.py", > line 139, in spmatrix > @deprecate > TypeError: deprecate() takes exactly 3 arguments (1 given) > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > ------------------------------------------------------------------------ > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > From justus.schwabedal at gmx.de Wed Mar 12 18:57:48 2008 From: justus.schwabedal at gmx.de (Justus Schwabedal) Date: Wed, 12 Mar 2008 23:57:48 +0100 Subject: [SciPy-dev] import scipy.linalg problems In-Reply-To: References: <11abf2720803090442m37665f9dx40dc27fe5251b656@mail.gmail.com> <11abf2720803111150o5b51ce43o913a1f93eb686313@mail.gmail.com> <9457e7c80803111959n277fcca3u75a3e5086ea203@mail.gmail.com><9457e7c80803112001t5e406c32o3815fdd2b480ba8e@mail.gmail.com><47D8174A.1080309@llnl.gov> Message-ID: <186A2A4A-3FD9-4C3D-AFD2-492FC3973369@gmx.de> I think so, too. I had similar problems. The reason was, that I had written some python scripts having there names in common with some scipy module. Trivial example: > ls -a . .. my_test_script.py scipy.py > less test_script.py ... import scipy ... This will lead to unpredictable events. Have you miss-set your python path-variable? ...watch the subject... On Mar 12, 2008, at 6:56 PM, Alan G Isaac wrote: > On Wed, 12 Mar 2008, Charles Doutriaux apparently wrote: >> File >> "/lgm/cdat/5.0.0.alpha7/lib/python2.5/site-packages/scipy/sparse/ >> base.py", >> line 139, in spmatrix >> @deprecate >> TypeError: deprecate() takes exactly 3 arguments (1 given) > > Looks like a name clash? > You should be able to just comment out the decorator > with no ill effects, I would think. > > Cheers, > Alan Isaac > > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev From lesommer at hmg.inpg.fr Wed Mar 12 23:58:39 2008 From: lesommer at hmg.inpg.fr (Julien Le Sommer) Date: Thu, 13 Mar 2008 14:58:39 +1100 Subject: [SciPy-dev] pbm in numpy.linalg.polyfit and nupy.linalg polyfit Message-ID: Hi, I just want to mention a possible bug in numpy/lib/polynomial.py with numpy version > 1.0.4 (with python2.4). the modification introduced with revision 3338 of numpy by charris, (see http://www.scipy.org/scipy/numpy/log/trunk/numpy/lib/polynomial.py) has broken the possibility to use polyfit with two-dimensional arrays. IPython gives the following message : ------------------------------------------------------------------------------------------ 308 # scale returned coefficients 309 if scale != 0 : --> 310 c /= vander([scale], order)[0] 311 312 if full : ValueError: shape mismatch: objects cannot be broadcast to a single shape -------------------------------------------------------------------------------------------- with the following script : -------------------------------------------------------------------------------------------- import numpy from numpy.random import * from numpy import linalg #- create some arrays : n = 100 r = 0.5 a = 1.4 # 1D vector T = numpy.array(xrange(n),dtype=numpy.float32)*2/(n-1) -1. X = a*T +r*(rand(n)-0.5) # (n * 2) array aa = numpy.array([[0.7,1.2]]) rr = numpy.array([[0.4,0.6]]) TT = T.reshape((n,1))*numpy.ones((n,2)) XX = aa*TT +rr*(randn(n,2)-0.5) # (n * 3) array aaa = numpy.array([[0.7,1.2,1.]]) rrr = numpy.array([[0.4,0.6,0.2]]) TTT = T.reshape((n,1))*numpy.ones((n,3)) XXX = aaa*TTT +rrr*(randn(n,3)-0.5) #- try numpy.polyfit # works ok (ar,br) = numpy.polyfit(T,X,1) # works ok (ar,br) = numpy.polyfit(T,XX,1) # fails (ar,br) = numpy.polyfit(T,XXX,1) -- Julien Le Sommer, LEGI/MEOM, BP53, 38041 Grenoble cedex 9, France Tel : +33 4 76 82 50 65 Fax : +33 4 76 82 52 71 http://www.legi.hmg.inpg.fr/~lesommer/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Thu Mar 13 01:56:23 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 12 Mar 2008 22:56:23 -0700 Subject: [SciPy-dev] Google Summer of Code Ideas Message-ID: Hello, I have started a Google Summer of Code Ideas page: http://scipy.org/scipy/scipy/wiki/SummerofCodeIdeas Please feel free to add any ideas you have for a summer project especially if you would be interested in mentoring it. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Thu Mar 13 02:22:46 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Wed, 12 Mar 2008 23:22:46 -0700 Subject: [SciPy-dev] Merging branch to build scipy with scons In-Reply-To: <47D13644.2060209@ar.media.kyoto-u.ac.jp> References: <47D13644.2060209@ar.media.kyoto-u.ac.jp> Message-ID: On Fri, Mar 7, 2008 at 5:34 AM, David Cournapeau wrote: > I would like to discuss about the possibility to merge the > build_with_scons branch of scipy into the trunk. It would enable > building scipy with numscons. Hey David, Any progress on the merge? I would really like to see this happen soon. It seems like the only responses I saw were positive, so I would recommend that you go ahead a merge as soon as you get the time. Thanks for all your work, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david at ar.media.kyoto-u.ac.jp Thu Mar 13 02:13:59 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 13 Mar 2008 15:13:59 +0900 Subject: [SciPy-dev] Merging branch to build scipy with scons In-Reply-To: References: <47D13644.2060209@ar.media.kyoto-u.ac.jp> Message-ID: <47D8C627.2060100@ar.media.kyoto-u.ac.jp> Jarrod Millman wrote: > > Hey David, > > Any progress on the merge? I would really like to see this happen > soon. It seems like the only responses I saw were positive, so I > would recommend that you go ahead a merge as soon as you get the time. > Ok, will do it, then, hopefully within today, cheers, David From stefan at sun.ac.za Thu Mar 13 04:09:43 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 13 Mar 2008 01:09:43 -0700 Subject: [SciPy-dev] pbm in numpy.linalg.polyfit and nupy.linalg polyfit In-Reply-To: References: Message-ID: <9457e7c80803130109n76730f3dlf27f511ae416026d@mail.gmail.com> Hi Julien Would you mind filing a bug at http://projects.scipy.org/scipy/numpy so that we don't lose track of this issue? Thanks St?fan On Wed, Mar 12, 2008 at 8:58 PM, Julien Le Sommer wrote: > > Hi, > > I just want to mention a possible bug in numpy/lib/polynomial.py > with numpy version > 1.0.4 (with python2.4). > > the modification introduced with revision 3338 of numpy by charris, > (see http://www.scipy.org/scipy/numpy/log/trunk/numpy/lib/polynomial.py) > has broken the possibility to use polyfit with two-dimensional arrays. > > IPython gives the following message : > ------------------------------------------------------------------------------------------ > 308 # scale returned coefficients > 309 if scale != 0 : > --> 310 c /= vander([scale], order)[0] > 311 > 312 if full : > > ValueError: shape mismatch: objects cannot be broadcast to a single shape > -------------------------------------------------------------------------------------------- > > with the following script : > -------------------------------------------------------------------------------------------- > import numpy > from numpy.random import * > from numpy import linalg > > #- create some arrays : > n = 100 > r = 0.5 > a = 1.4 > > # 1D vector > T = numpy.array(xrange(n),dtype=numpy.float32)*2/(n-1) -1. > X = a*T +r*(rand(n)-0.5) > > # (n * 2) array > aa = numpy.array([[0.7,1.2]]) > rr = numpy.array([[0.4,0.6]]) > TT = T.reshape((n,1))*numpy.ones((n,2)) > XX = aa*TT +rr*(randn(n,2)-0.5) > > # (n * 3) array > aaa = numpy.array([[0.7,1.2,1.]]) > rrr = numpy.array([[0.4,0.6,0.2]]) > TTT = T.reshape((n,1))*numpy.ones((n,3)) > XXX = aaa*TTT +rrr*(randn(n,3)-0.5) > > > #- try numpy.polyfit > > > # works ok > (ar,br) = numpy.polyfit(T,X,1) > > # works ok > (ar,br) = numpy.polyfit(T,XX,1) > > # fails > (ar,br) = numpy.polyfit(T,XXX,1) > > > -- > Julien Le Sommer, LEGI/MEOM, > BP53, 38041 Grenoble cedex 9, France > Tel : +33 4 76 82 50 65 > Fax : +33 4 76 82 52 71 > http://www.legi.hmg.inpg.fr/~lesommer/ > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > From millman at berkeley.edu Thu Mar 13 04:32:33 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 13 Mar 2008 01:32:33 -0700 Subject: [SciPy-dev] Merging branch to build scipy with scons In-Reply-To: <47D8C627.2060100@ar.media.kyoto-u.ac.jp> References: <47D13644.2060209@ar.media.kyoto-u.ac.jp> <47D8C627.2060100@ar.media.kyoto-u.ac.jp> Message-ID: On Wed, Mar 12, 2008 at 11:13 PM, David Cournapeau wrote: > Ok, will do it, then, hopefully within today, You rock! Just to let you know, a lot of people are watching your work adding scons support. For instance, the Sage developers were very keen to see numscons support added to both NumPy and SciPy. They were asking about it during the Austin meeting. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From millman at berkeley.edu Thu Mar 13 05:10:14 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 13 Mar 2008 02:10:14 -0700 Subject: [SciPy-dev] lsodi - ticket 615 In-Reply-To: <47CF0F4A.10500@appliedminds.com> References: <47CE34C6.5050304@appliedminds.com> <55474.129.240.228.26.1204727335.squirrel@cens.ioc.ee> <47CEB5FA.1060203@appliedminds.com> <47CF0F4A.10500@appliedminds.com> Message-ID: On Wed, Mar 5, 2008 at 2:23 PM, Russel Howe wrote: > I have an updated patch. I found a small problem with my first patch > and added the ability to integrate to critical points of the function > (itask=4). > > Unfortunately, the trac wiki will not allow me to append a new file to > the ticket as I do not have permissions. I have attached the new patch > here in hopes that somebody with appropriate permissions can add it to > the ticket. Hey Russel, I have updated your trac permissions, so you should be able to upload the new patch. Thanks for taking the time to work on this. Sincerely, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From david at ar.media.kyoto-u.ac.jp Thu Mar 13 05:36:00 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 13 Mar 2008 18:36:00 +0900 Subject: [SciPy-dev] Merging branch to build scipy with scons In-Reply-To: References: <47D13644.2060209@ar.media.kyoto-u.ac.jp> <47D8C627.2060100@ar.media.kyoto-u.ac.jp> Message-ID: <47D8F580.2050403@ar.media.kyoto-u.ac.jp> Jarrod Millman wrote: > > You rock! Just to let you know, a lot of people are watching your > work adding scons support. For instance, the Sage developers were > very keen to see numscons support added to both NumPy and SciPy. They > were asking about it during the Austin meeting. > Hm, I guess I am cursed, svnmerge failed again. Since I only add non existing files, maybe I can just put new files in the trunk ? Also, people interested in numscons really should discuss with me first, because right now, I consider it to be my playground, and I break various things when I need it. And also, there are some key things missing in scons, which I am working on now (that's most of the time I spent on numscons recently), to make the changes upstream, since I don't want to keep internal changes to scons for too long. cheers, David From dmitrey.kroshko at scipy.org Thu Mar 13 07:56:56 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Thu, 13 Mar 2008 13:56:56 +0200 Subject: [SciPy-dev] returning string by f2py In-Reply-To: <47D69AF0.80304@scipy.org> References: <47D647F0.7020401@scipy.org> <51089.129.240.228.26.1205241277.squirrel@cens.ioc.ee> <47D69AF0.80304@scipy.org> Message-ID: <47D91688.6070406@scipy.org> Hi Pearu, have you encountered any success with the problem (returning string from fortran)? If no, I could try to modify and connect to openopt and scipy bvls.f90 version. (However, I think connecting f77 code is more preferred than f90, because some computers can have f77 but have no f90 compiler installed, am I right)? Regards, D. BTW, I had noticed some parameters return incorrect value as well, for example loopa (number of iterations elapsed) always has same value that I pass to the fortran routine (even w/o any my modifications to code, simple f2py -c -m ...). dmitrey wrote: > Pearu Peterson wrote: >> On Tue, March 11, 2008 10:50 am, dmitrey wrote: >> >>> hi all, >>> I have modified bvls.f file to make it good enough to be connected to >>> scipy (add istop and msg - integer & string description of stop case). >>> >>> However, I always get same msg value that I pass to the routine (via >>> f2py). >>> >>> Could anyone check the modified bvls.f file? >>> http://scipy.org/scipy/scikits/attachment/wiki/OOFiles/bvls.f >>> >>> (Or mb something should be modified in Python code?) >>> >> >> Can you give an example how to create the wrapper >> and how to you use it from Python? Until that I can only >> suggest using intent(out) for msg argument. >> >> Pearu >> >> > here's (same) test file attached; as for f2py I just use > f2py -c -m bvls bvls.f > I think it's better to have default f2py using (w/o any additional > wrap files like .pyf), and make all required changes in py-file that > will call compiled f-file. I intend to make it available immediately > from scikits.openopt LLSP webpage (for download and using from OO) and > then connect it to scipy.optimize (also w/o auxiliary wrapper files). > Currently it returns same values as have been passed: istop=0 and same > msg. > Regards, D. > > ------------------------------------------------------------------------ > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev From pearu at cens.ioc.ee Thu Mar 13 09:57:09 2008 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Thu, 13 Mar 2008 15:57:09 +0200 (EET) Subject: [SciPy-dev] returning string by f2py In-Reply-To: <47D91688.6070406@scipy.org> References: <47D647F0.7020401@scipy.org> <51089.129.240.228.26.1205241277.squirrel@cens.ioc.ee> <47D69AF0.80304@scipy.org> <47D91688.6070406@scipy.org> Message-ID: <60477.88.89.195.179.1205416629.squirrel@cens.ioc.ee> On Thu, March 13, 2008 1:56 pm, dmitrey wrote: > Hi Pearu, > > have you encountered any success with the problem (returning string from > fortran)? $ cat foo.f subroutine foo(msg) character *(10) msg cf2py intent(out) msg msg = "abc" end $ f2py -m m foo.f -c >>> import m >>> m.foo() 'abc ' >>> Is that what you need? > (However, I think connecting f77 code is more preferred than f90, > because some computers can have f77 but have no f90 compiler installed, > am I right)? f2py is designed to wrap any F77 code, and only a subset of F90. > BTW, I had noticed some parameters return incorrect value as well, for > example loopa (number of iterations elapsed) always has same value that > I pass to the fortran routine (even w/o any my modifications to code, > simple f2py -c -m ...). Somehow it seems to me that you have not been trying to use intent(out).. HTH, Pearu From wnbell at gmail.com Thu Mar 13 11:15:29 2008 From: wnbell at gmail.com (Nathan Bell) Date: Thu, 13 Mar 2008 10:15:29 -0500 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: <47D53453.3000806@ar.media.kyoto-u.ac.jp> References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> <47D36D05.90600@ar.media.kyoto-u.ac.jp> <3d375d730803082136j518debb3q30018e0ea1570504@mail.gmail.com> <47D53453.3000806@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Mar 10, 2008 at 8:14 AM, David Cournapeau wrote: > and a nightmare to get right. I was more thinking about splitting the > interface file, so that only a couple of functions are generated by > each: this should be doable, no ? I can do it, if there is a chance for > a patch to be included. There would be say N swig interface files (one > for _diagonal, one for _scale, etc...), and sparsetools.py itself would > be written manually, but would just import python functions from each > generated python modules, that is would be a few lines only (I bet this > python module could easily be generated, too, if wanted). While better than manually splitting the _wrap file, this approach is still cumbersome. There are ~35 functions in sparsetools, so a 1 function : 1 file policy is not really scalable. I tried lumping all the CSR functions together and found only modest savings. Disabling the templates that unroll dense loops for the BSR matrix (see bsr_matvec) produced a measureable improvement in memory usage so I've committed this version of sparsetools.h to SVN. > I am not suggesting giving up swig or C++ templates. But the problem is > not the cost of memory: when virtual machines came into the game, you > hit really quickly the 32 bits limits (or more exactly, the fact that > most computers cannot physically handle more than 4 Gb of Memory). For > example, when I test numscons on solaris, I use indiana, which is a > binary distribution of open solaris available for free, and the VM takes > more than 1 Gb of ram when compiling sparsetools. Even on my recent > macbook with 2 Gb of Ram, I am at the limit. And virtual machines are > the only way for me to test many platforms (and build bots too often run > on vmware). Are you saying that g++ fails to compile on the VM, or that it starts swapping to disk? -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From nwagner at iam.uni-stuttgart.de Thu Mar 13 13:11:20 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 13 Mar 2008 18:11:20 +0100 Subject: [SciPy-dev] pbm in numpy.linalg.polyfit and nupy.linalg polyfit In-Reply-To: <9457e7c80803130109n76730f3dlf27f511ae416026d@mail.gmail.com> References: <9457e7c80803130109n76730f3dlf27f511ae416026d@mail.gmail.com> Message-ID: On Thu, 13 Mar 2008 01:09:43 -0700 "St?fan van der Walt" wrote: > Hi Julien > > Would you mind filing a bug at > > http://projects.scipy.org/scipy/numpy > > so that we don't lose track of this issue? > > Thanks > St?fan Done. See http://scipy.org/scipy/numpy/ticket/697 Cheers, Nils From wnbell at gmail.com Fri Mar 14 08:51:05 2008 From: wnbell at gmail.com (Nathan Bell) Date: Fri, 14 Mar 2008 07:51:05 -0500 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> <47D36D05.90600@ar.media.kyoto-u.ac.jp> <3d375d730803082136j518debb3q30018e0ea1570504@mail.gmail.com> <47D53453.3000806@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Mar 13, 2008 at 10:15 AM, Nathan Bell wrote: > > I am not suggesting giving up swig or C++ templates. But the problem is > > not the cost of memory: when virtual machines came into the game, you > > hit really quickly the 32 bits limits (or more exactly, the fact that > > most computers cannot physically handle more than 4 Gb of Memory). For > > example, when I test numscons on solaris, I use indiana, which is a > > binary distribution of open solaris available for free, and the VM takes > > more than 1 Gb of ram when compiling sparsetools. Even on my recent > > macbook with 2 Gb of Ram, I am at the limit. And virtual machines are > > the only way for me to test many platforms (and build bots too often run > > on vmware). > > Are you saying that g++ fails to compile on the VM, or that it starts > swapping to disk? David, I came across this blog post which should address your problem: http://hostingfu.com/article/compiling-with-gcc-on-low-memory-vps I compiled sparsetools_wrap.cxx on with "g++ --param ggc-min-expand=10 --param ggc-min-heapsize=8192" and memory usage peaked at 225MB (on a 32-bit machine). Setting ggc-min-expand=0 takes ages (I threw in the towel at 20 minutes), so I don't recommend that setting :) Please let me know if the problem persists. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From david at ar.media.kyoto-u.ac.jp Sat Mar 15 05:22:53 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 15 Mar 2008 18:22:53 +0900 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> <47D36D05.90600@ar.media.kyoto-u.ac.jp> <3d375d730803082136j518debb3q30018e0ea1570504@mail.gmail.com> <47D53453.3000806@ar.media.kyoto-u.ac.jp> Message-ID: <47DB956D.9000002@ar.media.kyoto-u.ac.jp> Nathan Bell wrote: > > While better than manually splitting the _wrap file, this approach is > still cumbersome. There are ~35 functions in sparsetools, so a 1 > function : 1 file policy is not really scalable. Well, having one file/function is overkill. I just think that 500-600 Mb of Ram to build one source file is really not good for various reasons; several MB source files is fundementaly a bad thing. Splitting it in half would already be pretty good, and that can be done easily: you just have to be careful about what functions to put in which file to get the memory reduction, though. > > I tried lumping all the CSR functions together and found only modest savings. With two files, I could get from 500 Mb to around 260 Mb, which is already pretty good I think. > > Disabling the templates that unroll dense loops for the BSR matrix > (see bsr_matvec) produced a measureable improvement in memory usage so > I've committed this version of sparsetools.h to SVN. the *matvec functions are indeed the ones which take a lot of memory for compilation, which is not surprising if they use expression template. Is it really useful ? I certainly do not want this to cause any performance penalty or anything. Putting those functions in a different file is what gives the most advantage in my experiment. > > Are you saying that g++ fails to compile on the VM, or that it starts > swapping to disk? Yes, that's what I am saying. I need to allocate 1 Gb of memory for solaris when compiling sparsetools with sun compilers right now. Even by today standards, that's an awful lot of memory. cheers, David From david at ar.media.kyoto-u.ac.jp Sat Mar 15 05:24:56 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 15 Mar 2008 18:24:56 +0900 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> <47D36D05.90600@ar.media.kyoto-u.ac.jp> <3d375d730803082136j518debb3q30018e0ea1570504@mail.gmail.com> <47D53453.3000806@ar.media.kyoto-u.ac.jp> Message-ID: <47DB95E8.3080104@ar.media.kyoto-u.ac.jp> Nathan Bell wrote: > > David, I came across this blog post which should address your problem: > http://hostingfu.com/article/compiling-with-gcc-on-low-memory-vps > > I compiled sparsetools_wrap.cxx on with "g++ --param ggc-min-expand=10 > --param ggc-min-heapsize=8192" and memory usage peaked at 225MB (on a > 32-bit machine). Setting ggc-min-expand=0 takes ages (I threw in the > towel at 20 minutes), so I don't recommend that setting :) > > Please let me know if the problem persists. > > Thanks for those info, I was not aware it was possible to control the memory manager of gcc, that's pretty interesting. Unfortunately, it won't work with other C++ compilers, so I am afraid it cannot be used in our case. cheers, David From david at ar.media.kyoto-u.ac.jp Sat Mar 15 06:59:24 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 15 Mar 2008 19:59:24 +0900 Subject: [SciPy-dev] Dealing with stupid compilers: ifort on itanium cannot compile empty source files Message-ID: <47DBAC0C.3040305@ar.media.kyoto-u.ac.jp> Hi, While helping someone to get scipy compiled with intel compilers on itanium, I encountered a serious problem: ifort cannot compile empty source files on itanium !! The files I got problems with are in ARPACK (two .f files are empty), but there may be other too, I don't know. Putting comments do not seem to work, there must be some statements... What would be the best solution in this case ? Is it ok to remove the files (since they are empty, this cannot be too bad, I guess, except if there is a need to track upstream library) cheers, David From david at ar.media.kyoto-u.ac.jp Sat Mar 15 07:03:27 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 15 Mar 2008 20:03:27 +0900 Subject: [SciPy-dev] scons build scripts in scipy trunk: merge is done Message-ID: <47DBACFF.4060500@ar.media.kyoto-u.ac.jp> Hi, Since I could not make svnmerge work, I manually merged my branch to build scipy with scons in the trunk, and the merge is done. Since I did it manually, I may have missed something; I quickly try it on linux, and it built, though. Again, do not use scons to build scipy for anything but testing. thanks, David From hagberg at lanl.gov Sat Mar 15 09:24:53 2008 From: hagberg at lanl.gov (Aric Hagberg) Date: Sat, 15 Mar 2008 07:24:53 -0600 Subject: [SciPy-dev] Dealing with stupid compilers: ifort on itanium cannot compile empty source files In-Reply-To: <47DBAC0C.3040305@ar.media.kyoto-u.ac.jp> References: <47DBAC0C.3040305@ar.media.kyoto-u.ac.jp> Message-ID: <20080315132453.GH13057@frappa.lanl.gov> On Sat, Mar 15, 2008 at 07:59:24PM +0900, David Cournapeau wrote: > While helping someone to get scipy compiled with intel compilers on > itanium, I encountered a serious problem: ifort cannot compile empty > source files on itanium !! The files I got problems with are in ARPACK > (two .f files are empty), but there may be other too, I don't know. > Putting comments do not seem to work, there must be some statements... > What would be the best solution in this case ? Is it ok to remove > the files (since they are empty, this cannot be too bad, I guess, except > if there is a need to track upstream library) I removed the files. The upstream library seems to be changing very little, if at all, so that shouldn't be too much trouble to track. Aric From nwagner at iam.uni-stuttgart.de Sun Mar 16 07:00:21 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Sun, 16 Mar 2008 12:00:21 +0100 Subject: [SciPy-dev] scikits learn Message-ID: Hi all, I found a syntax error in scikit learn. This is related to python version 2.4 Compiling /usr/lib/python2.4/site-packages/scikits/learn/utils/arffread.py ... File "/usr/lib/python2.4/site-packages/scikits/learn/utils/arffread.py", line 401 finally: SyntaxError: invalid syntax Nils From matthieu.brucher at gmail.com Sun Mar 16 07:58:20 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sun, 16 Mar 2008 12:58:20 +0100 Subject: [SciPy-dev] scikits learn In-Reply-To: References: Message-ID: Hi, This is a syntax introducted in Python 2.5, so Python 2.4 is not supported by the scikit. Matthieu 2008/3/16, Nils Wagner : > > Hi all, > > I found a syntax error in scikit learn. This is related to > python version 2.4 > > Compiling > /usr/lib/python2.4/site-packages/scikits/learn/utils/arffread.py > ... > File > "/usr/lib/python2.4/site-packages/scikits/learn/utils/arffread.py", > line 401 > finally: > > SyntaxError: invalid syntax > > Nils > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From slesarev.anton at gmail.com Sun Mar 16 08:41:29 2008 From: slesarev.anton at gmail.com (Anton Slesarev) Date: Sun, 16 Mar 2008 15:41:29 +0300 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) Message-ID: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> Hi. I'm going to describe what problems I see in current version of scikits.learn. After that I'll write what I want to improve during Google Summer of Code. In my last letter I tried to numerate some limitations in other open-source frameworks such as PyML and Orange. Let's start about Scikits.learn. First of all is a lack of documentation. I can find nothing beside David Cournapeau proposal on google Summer of Code. Nothing in wiki and nothing in maillist. There are few examples in svm, of course. But it is very hard use only examples. I can't find parser of different data formats. Only for datasets. As I understand datasets don't support sparse data format. There is no common structure in ML package. It has scattered modules such as svm, em, ann, but no main idea. If I mistake in understanding current state of affair you can correct me. Well, now about what I want to change. I am going to make learn package appropriate for text classification. Also I want to copy most of PyML (pyml.sourceforge.net/) functionality. First of all we need sparse data format. I want to write parsers for a number of common data formats. We need some preprocessing utilities, such as normalization, feature selection algorithms. This part should be common for all of machine learning package. Also package is need a number of classifiers. There are at least 2 state-of-art approaches in text classification and categorization:svm and Bayesian logistic regression. Svm has already been implemented in Scikits. There are a lot of implementations of logistic regression. I am going to integrate one of them (http://www.stat.rutgers.edu/~madigan/BBR/) into scikits. It is need interpretation module, which consists of processing results(different metrics of quality), visualization, feature comparison. There are common text collection (for instance http://trec.nist.gov/data/reuters/reuters.html). I'll try to make work with them absolutely simple. After all, it is very important to write(or generate) reference documentation and tutorial. OK, that's all. I expect to hear your opinions. Particularly I want to see answer of David Cournapeau, who is ,as I understand, maintainer of the learn package. -- Anton Slesarev -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrey.kroshko at scipy.org Sun Mar 16 08:55:59 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Sun, 16 Mar 2008 14:55:59 +0200 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> Message-ID: <47DD18DF.1060507@scipy.org> Anton Slesarev wrote: > > I am going to make learn package appropriate for text classification. > Also I want to copy most of PyML (pyml.sourceforge.net/ > ) functionality. > > -- > Anton Slesarev Maybe, you will be also interested in some other ML packages from here (they should be checked to prevent waste of time and efforts for code duplicating) http://mloss.org/software/ lots of them have Python API implemented or, moreover, are Python-written Regards, D. From matthieu.brucher at gmail.com Sun Mar 16 08:58:28 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sun, 16 Mar 2008 13:58:28 +0100 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> Message-ID: Hi, I completely agree with you, there should be more documentation, but I still don't see your point with sparse data format. Scipy proposes this, doesn't it ? BBR will be welcomed by a lot of people, when you will implement it, code everythign as generic as possible. For instance, feature selection (perhaps extraction as well) could be used by other algorithms (even SVMs), so it should be as generic as possible (feature comparison should be explained, is it in terms of classification results ?). I'd like to add manifold learning tools (this can be thought as some feature extraction tools, visualization, ...) which could benefit from your approach and vice-versa. Matthieu 2008/3/16, Anton Slesarev : > > Hi. > > I'm going to describe what problems I see in current version of > scikits.learn. After that I'll write what I want to improve during Google > Summer of Code. In my last letter I tried to numerate some limitations in > other open-source frameworks such as PyML and Orange. > > Let's start about Scikits.learn. > > First of all is a lack of documentation. I can find nothing beside David > Cournapeau proposal on google Summer of Code. Nothing in wiki and nothing in > maillist. There are few examples in svm, of course. But it is very hard use > only examples. I can't find parser of different data formats. Only for > datasets. As I understand datasets don't support sparse data format. There > is no common structure in ML package. It has scattered modules such as svm, > em, ann, but no main idea. > > If I mistake in understanding current state of affair you can correct me. > > Well, now about what I want to change. > > I am going to make learn package appropriate for text classification. Also > I want to copy most of PyML (pyml.sourceforge.net/) functionality. > > First of all we need sparse data format. I want to write parsers for a > number of common data formats. > > We need some preprocessing utilities, such as normalization, feature > selection algorithms. > This part should be common for all of machine learning package. > > Also package is need a number of classifiers. There are at least 2 > state-of-art approaches in text classification and categorization:svm and > Bayesian logistic regression. Svm has already been implemented in Scikits. > There are a lot of implementations of logistic regression. I am going to > integrate one of them (http://www.stat.rutgers.edu/~madigan/BBR/) > into scikits. > > It is need interpretation module, which consists of processing > results(different metrics of quality), visualization, feature comparison. > > There are common text collection (for instance > http://trec.nist.gov/data/reuters/reuters.html). I'll try to make work > with them absolutely simple. > > After all, it is very important to write(or generate) reference > documentation and tutorial. > > OK, that's all. I expect to hear your opinions. Particularly I want to see > answer of David Cournapeau, who is ,as I understand, maintainer of the learn > package. > > > > -- > Anton Slesarev > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From millman at berkeley.edu Mon Mar 17 09:46:22 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Mon, 17 Mar 2008 06:46:22 -0700 Subject: [SciPy-dev] scons build scripts in scipy trunk: merge is done In-Reply-To: <47DBACFF.4060500@ar.media.kyoto-u.ac.jp> References: <47DBACFF.4060500@ar.media.kyoto-u.ac.jp> Message-ID: On Sat, Mar 15, 2008 at 4:03 AM, David Cournapeau wrote: > Since I could not make svnmerge work, I manually merged my branch to > build scipy with scons in the trunk, and the merge is done. Since I did > it manually, I may have missed something; I quickly try it on linux, and > it built, though. Thanks! -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From bsouthey at gmail.com Mon Mar 17 10:33:27 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 17 Mar 2008 09:33:27 -0500 Subject: [SciPy-dev] scikits learn In-Reply-To: References: Message-ID: <47DE8137.7090207@gmail.com> Hi, This should be reported as a bug as should any Python 2.4 version code because neither NumPy nor SciPy require only a 2.5 series of Python: The SciPy FAQ (http://www.scipy.org/FAQ) page states that NumPy requires Python 2.3.x or 2.4.x or 2.5.x SciPy General building instructions: http://www.scipy.org/Installing_SciPy/BuildingGeneral - To build SciPy, Python version 2.3 or newer is required. Sure, I know that the Python 2.3 is about 4 years old (2.3.4 released May 2004) but some long terms stability distros (like Centos 4) may not updated Python to even Python 2.4 let alone Python 2.5. I understand that scikits are somewhat special so the requirements of a scikit need to be clearly stated and the scikit should exit if an inappropriate version of Python is used. Note this type of check is going to be essential for Python 2.6 (Python 2.6a1 Released: February 29th 2008) and Python 3.0 (Python 3.0a3 Release: February 29th 2008) because Python 3.0 will break backwards compatibility. Regards Bruce Matthieu Brucher wrote: > Hi, > > This is a syntax introducted in Python 2.5, so Python 2.4 is not > supported by the scikit. > > Matthieu > > 2008/3/16, Nils Wagner >: > > Hi all, > > I found a syntax error in scikit learn. This is related to > python version 2.4 > > Compiling > /usr/lib/python2.4/site-packages/scikits/learn/utils/arffread.py > ... > File > "/usr/lib/python2.4/site-packages/scikits/learn/utils/arffread.py", > line 401 > finally: > > SyntaxError: invalid syntax > > Nils > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > ------------------------------------------------------------------------ > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > From stefan at sun.ac.za Mon Mar 17 11:53:17 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 17 Mar 2008 16:53:17 +0100 Subject: [SciPy-dev] scikits learn In-Reply-To: <47DE8137.7090207@gmail.com> References: <47DE8137.7090207@gmail.com> Message-ID: <9457e7c80803170853h2d5ab62btf23953b56d659ba5@mail.gmail.com> On Mon, Mar 17, 2008 at 3:33 PM, Bruce Southey wrote: > This should be reported as a bug as should any Python 2.4 version code > because neither NumPy nor SciPy require only a 2.5 series of Python: > > The SciPy FAQ (http://www.scipy.org/FAQ) page states that NumPy requires > Python 2.3.x or 2.4.x or 2.5.x > SciPy General building instructions: > http://www.scipy.org/Installing_SciPy/BuildingGeneral - To build SciPy, > Python version 2.3 or newer is required. NumPy requires >= 2.3 and SciPy >= 2.4. There is currently no guideline for the scikits, and their authors decide for themselves. Interestingly, ipython will soon move to 2.5 and make use of the 'with' statement to grab code blocks. Cheers St?fan From nwagner at iam.uni-stuttgart.de Mon Mar 17 15:37:57 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 17 Mar 2008 20:37:57 +0100 Subject: [SciPy-dev] Python and self-validating algorithms Message-ID: Hi all, Is someone aware of a python package for self-validating algorithms (similar to INTLAB for Matlab) ? Cheers, Nils References: http://www.ti3.tu-harburg.de/rump/intlab/ http://www.cs.cas.cz/rohn/matlab/ From slesarev.anton at gmail.com Tue Mar 18 08:05:02 2008 From: slesarev.anton at gmail.com (Anton Slesarev) Date: Tue, 18 Mar 2008 15:05:02 +0300 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> Message-ID: <11abf2720803180505n4c9717a6u2b7e405d2378bf44@mail.gmail.com> On Sun, Mar 16, 2008 at 3:58 PM, Matthieu Brucher < matthieu.brucher at gmail.com> wrote: > Hi, > > I completely agree with you, there should be more documentation, but I > still don't see your point with sparse data format. Scipy proposes this, > doesn't it ? Scipy support work with sparse matrix. It is true. But I mean that learn should support sparse format in parsers. To learn svm it should be enough to use syntax like: data = LoadSparseData('data.file') s = svm() s.train(data) I hope to implement such kind of syntax. User needn't know what exactly data is. He should just use it. > > BBR will be welcomed by a lot of people, when you will implement it, code > everythign as generic as possible. For instance, feature selection (perhaps > extraction as well) could be used by other algorithms (even SVMs), so it > should be as generic as possible (feature comparison should be explained, is > it in terms of classification results ?). Yes, it's a good wish. But I think that main part of bbr is classifier, feature selection can be implemented in separate module. It is not need to integrate to scikits their implementation of feature selection algorithm. > > > I'd like to add manifold learning tools (this can be thought as some > feature extraction tools, visualization, ...) which could benefit from your > approach and vice-versa. > That's great! > > Matthieu > > 2008/3/16, Anton Slesarev : > > > > Hi. > > > > I'm going to describe what problems I see in current version of > > scikits.learn. After that I'll write what I want to improve during > > Google Summer of Code. In my last letter I tried to numerate some > > limitations in other open-source frameworks such as PyML and Orange. > > > > Let's start about Scikits.learn. > > > > First of all is a lack of documentation. I can find nothing beside David > > Cournapeau proposal on google Summer of Code. Nothing in wiki and nothing in > > maillist. There are few examples in svm, of course. But it is very hard use > > only examples. I can't find parser of different data formats. Only for > > datasets. As I understand datasets don't support sparse data format. There > > is no common structure in ML package. It has scattered modules such as svm, > > em, ann, but no main idea. > > > > If I mistake in understanding current state of affair you can correct > > me. > > > > Well, now about what I want to change. > > > > I am going to make learn package appropriate for text classification. > > Also I want to copy most of PyML (pyml.sourceforge.net/) functionality. > > > > First of all we need sparse data format. I want to write parsers for a > > number of common data formats. > > > > We need some preprocessing utilities, such as normalization, feature > > selection algorithms. > > This part should be common for all of machine learning package. > > > > Also package is need a number of classifiers. There are at least 2 > > state-of-art approaches in text classification and categorization:svm and > > Bayesian logistic regression. Svm has already been implemented in Scikits. > > There are a lot of implementations of logistic regression. I am going to > > integrate one of them (http://www.stat.rutgers.edu/~madigan/BBR/) > > into scikits. > > > > It is need interpretation module, which consists of processing > > results(different metrics of quality), visualization, feature comparison. > > > > There are common text collection (for instance > > http://trec.nist.gov/data/reuters/reuters.html). I'll try to make work > > with them absolutely simple. > > > > After all, it is very important to write(or generate) reference > > documentation and tutorial. > > > > OK, that's all. I expect to hear your opinions. Particularly I want to > > see answer of David Cournapeau, who is ,as I understand, maintainer of the > > learn package. > > > > > > > > -- > > Anton Slesarev > > _______________________________________________ > > Scipy-dev mailing list > > Scipy-dev at scipy.org > > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > > > > -- > French PhD student > Website : http://matthieu-brucher.developpez.com/ > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > LinkedIn : http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > -- Anton Slesarev -------------- next part -------------- An HTML attachment was scrubbed... URL: From wnbell at gmail.com Tue Mar 18 10:03:57 2008 From: wnbell at gmail.com (Nathan Bell) Date: Tue, 18 Mar 2008 09:03:57 -0500 Subject: [SciPy-dev] Splitting sparsetools_wrap source file ? In-Reply-To: <47DB95E8.3080104@ar.media.kyoto-u.ac.jp> References: <47D3642F.3070401@ar.media.kyoto-u.ac.jp> <3d375d730803082038u1164fda6q1129b09ba69e498b@mail.gmail.com> <47D36D05.90600@ar.media.kyoto-u.ac.jp> <3d375d730803082136j518debb3q30018e0ea1570504@mail.gmail.com> <47D53453.3000806@ar.media.kyoto-u.ac.jp> <47DB95E8.3080104@ar.media.kyoto-u.ac.jp> Message-ID: On Sat, Mar 15, 2008 at 4:24 AM, David Cournapeau wrote: > Thanks for those info, I was not aware it was possible to control the > memory manager of gcc, that's pretty interesting. Unfortunately, it > won't work with other C++ compilers, so I am afraid it cannot be used in > our case. David, I've split sparsetools into five separate modules, one for each matrix type. Previously sparsetools compiled to sparse/_sparsetools.so, whereas now there are sparse/sparsetools/_csr.so and _csc.so etc. I was able to preserve the old interface by combining the new modules in sparse/sparsetools/__init__py (which just imports * from each of the submodules). It may be necessary to do a clean build to test the new code since the old sparsetools.py will be competing with the new sparsetools directory. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From stefan at sun.ac.za Tue Mar 18 10:30:37 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 18 Mar 2008 15:30:37 +0100 Subject: [SciPy-dev] DEFAULT section in site.cfg Message-ID: <9457e7c80803180730w5cbac729pe904c193e5f3742a@mail.gmail.com> Hi all, ConfigParser no longer allows a 'DEFAULT' section under Python 2.6, so I'd like to change that part of the site.cfg to something like 'general' or 'all'. Any suggestions, comments? Regards St?fan From robert.kern at gmail.com Tue Mar 18 14:19:43 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Mar 2008 13:19:43 -0500 Subject: [SciPy-dev] DEFAULT section in site.cfg In-Reply-To: <9457e7c80803180730w5cbac729pe904c193e5f3742a@mail.gmail.com> References: <9457e7c80803180730w5cbac729pe904c193e5f3742a@mail.gmail.com> Message-ID: <3d375d730803181119s3a022a2al58fd54d291076c07@mail.gmail.com> On Tue, Mar 18, 2008 at 9:30 AM, St?fan van der Walt wrote: > Hi all, > > ConfigParser no longer allows a 'DEFAULT' section under Python 2.6, so > I'd like to change that part of the site.cfg to something like > 'general' or 'all'. Any suggestions, comments? I leave it to your judgment. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From argriffi at ncsu.edu Tue Mar 18 19:56:13 2008 From: argriffi at ncsu.edu (argriffi) Date: Tue, 18 Mar 2008 19:56:13 -0400 Subject: [SciPy-dev] scipy.optimize.nonlin.broyden2 Message-ID: <47E0569D.8030502@ncsu.edu> This function solved my problem when I used 20 iterations, but when I used 40 iterations I got (nan, nan, nan). I logged the successive points for which my function was evaluated, and the process seemed to have converged to the correct answer before suddenly deciding to give nans. My current solution is to just use 20 iterations, but I would be interested to hear a safer solution. Alex From millman at berkeley.edu Tue Mar 18 21:43:19 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Tue, 18 Mar 2008 18:43:19 -0700 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> Message-ID: Hey Anton, Sorry I haven't responded sooner; I am at a coding sprint. Anyway, I am meeting with the developers of Multivariate Pattern Analysis in Python: http://pkg-exppsy.alioth.debian.org/pymvpa/ I think that this package looks very good. David Cournapeau will be visiting Berkeley in early April and I plan to discuss whether it would make sense to merge this into scikits.learn. So it would make sense for you to take a look at the manual here: http://pkg-exppsy.alioth.debian.org/pymvpa/manual.html I am really excited that you are planning to submit a proposal for the SoC. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From wnbell at gmail.com Wed Mar 19 03:22:25 2008 From: wnbell at gmail.com (Nathan Bell) Date: Wed, 19 Mar 2008 02:22:25 -0500 Subject: [SciPy-dev] unexpectedly slow norm() Message-ID: I use norm() frequently in my own codes and I was recently surprised to see it near the top of my profiling results. It seems that both SciPy and NumPy incur a large amount of overhead in this operation. Consider the following script ================================= from time import clock from numpy import inner, sqrt, ones import numpy.linalg import scipy.linalg from scipy.lib.blas import get_blas_funcs x = ones(5*1000*1000) scipy_norm = scipy.linalg.norm numpy_norm = numpy.linalg.norm def adhoc_norm(x): return sqrt(inner(x,x)) blas_norm = get_blas_funcs(('nrm2',),(x,))[0] for fn in [scipy_norm, numpy_norm, adhoc_norm, blas_norm ]: start = clock() for i in range(10): n = fn(x) end = clock() print "%f seconds per call" % ((end - start)/10.0) ================================= Which outputs the following on my laptop: 0.407000 seconds per call 0.089000 seconds per call 0.013000 seconds per call 0.012000 seconds per call -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From carlos.s.santos at gmail.com Wed Mar 19 07:39:27 2008 From: carlos.s.santos at gmail.com (Carlos da Silva Santos) Date: Wed, 19 Mar 2008 11:39:27 +0000 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <11abf2720803180505n4c9717a6u2b7e405d2378bf44@mail.gmail.com> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <11abf2720803180505n4c9717a6u2b7e405d2378bf44@mail.gmail.com> Message-ID: <1dc6ddb60803190439k7c09693q145dae7f60585ce0@mail.gmail.com> On Tue, Mar 18, 2008 at 12:05 PM, Anton Slesarev wrote: > > Scipy support work with sparse matrix. It is true. But I mean that learn > should support sparse format in parsers. To learn svm it should be enough to > use syntax like: > > data = LoadSparseData('data.file') > s = svm() > s.train(data) > > I hope to implement such kind of syntax. User needn't know what exactly data > is. He should just use it. I have no first hand experience with it, but apparently the Shogun toolbox provides some support for sparse data. From their homepage: http://www.shogun-toolbox.org/ "The machine learning toolbox's focus is on large scale kernel methods and especially on Support Vector Machines (SVM) [1]. It provides a generic SVM object interfacing to several different SVM implementations, among them the state of the art LibSVM [2], SVMLight, [3] SVMLin [4] and GPDT [5]. Each of the SVMs can be combined with a variety of kernels. ... The input feature-objects can be dense, sparse or strings and of type int/short/double/char and can be converted into different feature types. Chains of preprocessors (e.g. substracting the mean) can be attached to each feature object allowing for on-the-fly pre-processing. SHOGUN is implemented in C++ and interfaces to Matlab(tm), R, Octave and Python." I hope this helps. Carlos > > > > > > > > BBR will be welcomed by a lot of people, when you will implement it, code > everythign as generic as possible. For instance, feature selection (perhaps > extraction as well) could be used by other algorithms (even SVMs), so it > should be as generic as possible (feature comparison should be explained, is > it in terms of classification results ?). > > Yes, it's a good wish. But I think that main part of bbr is classifier, > feature selection can be implemented in separate module. It is not need to > integrate to scikits their implementation of feature selection algorithm. > > > > > > > I'd like to add manifold learning tools (this can be thought as some > feature extraction tools, visualization, ...) which could benefit from your > approach and vice-versa. > > > That's great! > > > > > > Matthieu > > > > > > 2008/3/16, Anton Slesarev : > > > > > > > > > > > > Hi. > > > > > > I'm going to describe what problems I see in current version of > scikits.learn. After that I'll write what I want to improve during Google > Summer of Code. In my last letter I tried to numerate some limitations in > other open-source frameworks such as PyML and Orange. > > > > > > Let's start about Scikits.learn. > > > > > > First of all is a lack of documentation. I can find nothing beside David > Cournapeau proposal on google Summer of Code. Nothing in wiki and nothing in > maillist. There are few examples in svm, of course. But it is very hard use > only examples. I can't find parser of different data formats. Only for > datasets. As I understand datasets don't support sparse data format. There > is no common structure in ML package. It has scattered modules such as svm, > em, ann, but no main idea. > > > > > > If I mistake in understanding current state of affair you can correct > me. > > > > > > Well, now about what I want to change. > > > > > > I am going to make learn package appropriate for text classification. > Also I want to copy most of PyML (pyml.sourceforge.net/) functionality. > > > > > > First of all we need sparse data format. I want to write parsers for a > number of common data formats. > > > > > > We need some preprocessing utilities, such as normalization, feature > selection algorithms. > > > This part should be common for all of machine learning package. > > > > > > Also package is need a number of classifiers. There are at least 2 > state-of-art approaches in text classification and categorization:svm and > Bayesian logistic regression. Svm has already been implemented in Scikits. > There are a lot of implementations of logistic regression. I am going to > integrate one of them (http://www.stat.rutgers.edu/~madigan/BBR/) into > scikits. > > > > > > It is need interpretation module, which consists of processing > results(different metrics of quality), visualization, feature comparison. > > > > > > There are common text collection (for instance > http://trec.nist.gov/data/reuters/reuters.html). I'll try to make work with > them absolutely simple. > > > > > > After all, it is very important to write(or generate) reference > documentation and tutorial. > > > > > > OK, that's all. I expect to hear your opinions. Particularly I want to > see answer of David Cournapeau, who is ,as I understand, maintainer of the > learn package. > > > > > > > > > > > > -- > > > Anton Slesarev > > > > > > _______________________________________________ > > > Scipy-dev mailing list > > > Scipy-dev at scipy.org > > > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > > > > > > > > > > > -- > > French PhD student > > Website : http://matthieu-brucher.developpez.com/ > > Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 > > LinkedIn : http://www.linkedin.com/in/matthieubrucher > > _______________________________________________ > > Scipy-dev mailing list > > Scipy-dev at scipy.org > > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > > > > > -- > Anton Slesarev > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > From ctw at cogsci.info Wed Mar 19 09:50:56 2008 From: ctw at cogsci.info (Christoph T. Weidemann) Date: Wed, 19 Mar 2008 09:50:56 -0400 Subject: [SciPy-dev] bug in scipy.signal.wavelets.morlet()? Message-ID: Hi All! I came across what seems to me like an obvious bug in scipy.signal.wavelets.morlet(): If complete==True (the default setting) the output does not correspond to the formula in the docstring. I believe the bug would be fixed if the if statement were changed from: if complete: x -= exp(-0.5*(w**2)) to if complete: output -= exp(-0.5*(w**2)) Best, Christoph Weidemann From stefan at sun.ac.za Wed Mar 19 10:17:11 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Wed, 19 Mar 2008 15:17:11 +0100 Subject: [SciPy-dev] bug in scipy.signal.wavelets.morlet()? In-Reply-To: References: Message-ID: <9457e7c80803190717w77846e75s4e4b44f6874bbe66@mail.gmail.com> Thanks, Christoph. This is now fixed in SVN. Do you have a simple test case we can use to ensure this doesn't happen again? Regards St?fan On Wed, Mar 19, 2008 at 2:50 PM, Christoph T. Weidemann wrote: > Hi All! > > I came across what seems to me like an obvious bug in > scipy.signal.wavelets.morlet(): > > If complete==True (the default setting) the output does not correspond > to the formula in the docstring. > I believe the bug would be fixed if the if statement were changed from: > > if complete: > x -= exp(-0.5*(w**2)) > > to > > if complete: > output -= exp(-0.5*(w**2)) > > Best, > Christoph Weidemann From ondrej at certik.cz Wed Mar 19 12:25:39 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 19 Mar 2008 17:25:39 +0100 Subject: [SciPy-dev] scipy.optimize.nonlin.broyden2 In-Reply-To: <47E0569D.8030502@ncsu.edu> References: <47E0569D.8030502@ncsu.edu> Message-ID: <85b5c3130803190925j477c1a2ayc2246d131eaeff78@mail.gmail.com> Hi Alex, On Wed, Mar 19, 2008 at 12:56 AM, argriffi wrote: > This function solved my problem when I used 20 iterations, but when I > used 40 iterations I got (nan, nan, nan). I logged the successive > points for which my function was evaluated, and the process seemed to > have converged to the correct answer before suddenly deciding to give > nans. My current solution is to just use 20 iterations, but I would be > interested to hear a safer solution. I wrote this code. Could you please send here your code with instructions how to run it to get the above results? Maybe I'll find why it doesn't work for you. Ondrej From ctw at cogsci.info Wed Mar 19 13:48:06 2008 From: ctw at cogsci.info (Christoph T. Weidemann) Date: Wed, 19 Mar 2008 13:48:06 -0400 Subject: [SciPy-dev] bug in scipy.signal.wavelets.morlet()? Message-ID: > Thanks, Christoph. This is now fixed in SVN. Do you have a simple > test case we can use to ensure this doesn't happen again? I'll try to write a test case. Once I have the code, what's the best way for me to submit it? It would probably be easiest for me if I could get write access to the SVN repository (I'd be happy to make more contributions in the future ... I often notice small things that could be improved such as docstrings, but don't always take the time to submit a bug report), but I understand if you have more stringent criteria than a bug report and stated desire for granting people write access. On a slightly off-topic note, the following bug report (with fix!) seems to have been ignored since it was reported 2 months ago: http://scipy.org/scipy/scipy/ticket/595 Generally, what's the best way to report bugs? the wiki, this mailing list, both, ...? Best, Christoph From robert.kern at gmail.com Wed Mar 19 14:44:41 2008 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 19 Mar 2008 13:44:41 -0500 Subject: [SciPy-dev] unexpectedly slow norm() In-Reply-To: References: Message-ID: <3d375d730803191144o479957fl1c7af4491508277c@mail.gmail.com> On Wed, Mar 19, 2008 at 2:22 AM, Nathan Bell wrote: > I use norm() frequently in my own codes and I was recently surprised > to see it near the top of my profiling results. It seems that both > SciPy and NumPy incur a large amount of overhead in this operation. They are general functions for computing a variety of norms, not just the L2 norm. The numpy implementation of the L2 norm avoids some unnecessary function calls and could replace the scipy one. Both do a general formulation for both complex and real arrays which is probably costing us. If you would like to code up the extra test for realness to evaluate the L2 norm faster, go ahead. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ondrej at certik.cz Wed Mar 19 18:18:43 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 19 Mar 2008 23:18:43 +0100 Subject: [SciPy-dev] scipy.optimize.nonlin.broyden2 In-Reply-To: <47E177E2.30804@ncsu.edu> References: <47E0569D.8030502@ncsu.edu> <85b5c3130803190925j477c1a2ayc2246d131eaeff78@mail.gmail.com> <47E177E2.30804@ncsu.edu> Message-ID: <85b5c3130803191518g32f5cabl964dee3a3fe03981@mail.gmail.com> On Wed, Mar 19, 2008 at 9:30 PM, argriffi wrote: > Hi Ondrej, > > Running the attached script should demonstrate the problem I mentioned. > I removed dependencies on other modules I had written, but let me know > if you want it reduced even further. Thanks, that is enough. I can reproduce the problem now: ------------------ test_solver_20 (__main__.TestCodonInversion) ... ok test_solver_40 (__main__.TestCodonInversion) ... ERROR ====================================================================== ERROR: test_solver_40 (__main__.TestCodonInversion) ---------------------------------------------------------------------- Traceback (most recent call last): File "CodonInversion.py", line 181, in test_solver_40 self._test_solver_helper(40) File "CodonInversion.py", line 173, in _test_solver_helper raise RuntimeError('failed on iteration %d of %d: %s' % (len(objective_function.guesses), iterations, str(e))) RuntimeError: failed on iteration 22 of 40: detected possibly invalid input to the objective function: (nan, nan, nan) ---------------------------------------------------------------------- Ran 2 tests in 0.029s FAILED (errors=1) ------------------------- And the reason is simple. If you look at this code: for n in range(iter): deltaxm=-Gm*Fxm xm=xm+deltaxm Fxm1=myF(F,xm) deltaFxm=Fxm1-Fxm Fxm=Fxm1 Gm=Gm+(deltaxm-Gm*deltaFxm)*deltaFxm.T/norm(deltaFxm)**2 And if you debug it, as I just did (for example using winpdb), you'll notice, that deltaFxm gets zero, after a lot of iterations, because you converged to a solution already. Then norm(deltaFxm) is 0 and after dividing by 0, you get a NaN in Gm, thus you get NaN in deltaxm and xm and you get NaN as an input into your function and here we are. The solution is to check that deltaFxm is near zero and stop iterating. Do you think you could please provide such a patch? :) Also with a test case. Thanks, Ondrej P.S. I CCed scipy-dev as I think you forgot to forward your email to it. I hope it's ok. From stefan at sun.ac.za Wed Mar 19 19:49:36 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Thu, 20 Mar 2008 00:49:36 +0100 Subject: [SciPy-dev] bug in scipy.signal.wavelets.morlet()? In-Reply-To: References: Message-ID: <9457e7c80803191649re924dddj4f11a52fe95db9fa@mail.gmail.com> Hi Christoph On Wed, Mar 19, 2008 at 6:48 PM, Christoph T. Weidemann wrote: > I'll try to write a test case. Once I have the code, what's the best > way for me to submit it? For now, please attach it as a ticket. > It would probably be easiest for me if I > could get write access to the SVN repository (I'd be happy to make > more contributions in the future ... I often notice small things that > could be improved such as docstrings, but don't always take the time > to submit a bug report), but I understand if you have more stringent > criteria than a bug report and stated desire for granting people write > access. Your contributions are very welcome. Attach a couple of patches to tickets, and Jarrod Millman will contact you with your SVN credentials. > On a slightly off-topic note, the following bug report (with fix!) > seems to have been ignored since it was reported 2 months ago: > http://scipy.org/scipy/scipy/ticket/595 Thanks for the reminder -- fixed in r4043. > Generally, what's the best way to report bugs? the wiki, this mailing > list, both, ...? For enhancements and minor issues, file a ticket. For more serious problems also send a message to the list so we don't miss it. Thanks again for your input! Regards St?fan From argriffi at ncsu.edu Wed Mar 19 23:11:38 2008 From: argriffi at ncsu.edu (argriffi) Date: Wed, 19 Mar 2008 23:11:38 -0400 Subject: [SciPy-dev] scipy.optimize.nonlin.broyden2 In-Reply-To: <85b5c3130803191518g32f5cabl964dee3a3fe03981@mail.gmail.com> References: <47E0569D.8030502@ncsu.edu> <85b5c3130803190925j477c1a2ayc2246d131eaeff78@mail.gmail.com> <47E177E2.30804@ncsu.edu> <85b5c3130803191518g32f5cabl964dee3a3fe03981@mail.gmail.com> Message-ID: <47E1D5EA.8070902@ncsu.edu> Hi Ondrej, As requested I have attached a modification and a test case. The modification checks a value for equality to zero, but maybe it should more appropriately handle values near zero. Also, other functions in nonlin.py may benefit from similar modifications. Alex Ondrej Certik wrote: > On Wed, Mar 19, 2008 at 9:30 PM, argriffi wrote: > >> Hi Ondrej, >> >> Running the attached script should demonstrate the problem I mentioned. >> I removed dependencies on other modules I had written, but let me know >> if you want it reduced even further. >> > > Thanks, that is enough. I can reproduce the problem now: > > ------------------ > > test_solver_20 (__main__.TestCodonInversion) ... ok > test_solver_40 (__main__.TestCodonInversion) ... ERROR > > ====================================================================== > ERROR: test_solver_40 (__main__.TestCodonInversion) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "CodonInversion.py", line 181, in test_solver_40 > self._test_solver_helper(40) > File "CodonInversion.py", line 173, in _test_solver_helper > raise RuntimeError('failed on iteration %d of %d: %s' % > (len(objective_function.guesses), iterations, str(e))) > RuntimeError: failed on iteration 22 of 40: detected possibly invalid > input to the objective function: (nan, nan, nan) > > ---------------------------------------------------------------------- > Ran 2 tests in 0.029s > > FAILED (errors=1) > > ------------------------- > > > And the reason is simple. If you look at this code: > > > for n in range(iter): > deltaxm=-Gm*Fxm > xm=xm+deltaxm > Fxm1=myF(F,xm) > deltaFxm=Fxm1-Fxm > Fxm=Fxm1 > Gm=Gm+(deltaxm-Gm*deltaFxm)*deltaFxm.T/norm(deltaFxm)**2 > > > And if you debug it, as I just did (for example using winpdb), you'll > notice, that deltaFxm gets zero, after a lot of iterations, because > you converged > to a solution already. Then norm(deltaFxm) is 0 and after dividing by > 0, you get a NaN in Gm, thus you get NaN in deltaxm and xm and you get > NaN as an input into your function and here we are. > > The solution is to check that deltaFxm is near zero and stop > iterating. Do you think you could please provide such a patch? :) > > Also with a test case. > > Thanks, > Ondrej > > P.S. I CCed scipy-dev as I think you forgot to forward your email to > it. I hope it's ok. > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Broyden.py URL: From david at ar.media.kyoto-u.ac.jp Wed Mar 19 23:13:38 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 20 Mar 2008 12:13:38 +0900 Subject: [SciPy-dev] scikits learn In-Reply-To: <47DE8137.7090207@gmail.com> References: <47DE8137.7090207@gmail.com> Message-ID: <47E1D662.1010509@ar.media.kyoto-u.ac.jp> Bruce Southey wrote: > I understand that scikits are somewhat special so the requirements of a > scikit need to be clearly stated and the scikit should exit if an > inappropriate version of Python is used. Note this type of check is > going to be essential for Python 2.6 (Python 2.6a1 Released: February > 29th 2008) and Python 3.0 (Python 3.0a3 Release: February 29th 2008) > because Python 3.0 will break backwards compatibility. > I don't know about the general policy for scikits, but in the present case, it was just because I was not aware the try/except/finally construction was python 2.5 specific. Generally, I did not work on scikits.learn for the last 6 months because of my work on numscons (there is a limit on what I can do besides my PhD :) ), sorry about that. Since numscons starts being in good shape, I hope to be back on scikits soon (there will be a scipy/numpy meeting with other scipy/numpy people in a few weeks, and scikits.learn will be one of the main topic for sure) cheers, David From david at ar.media.kyoto-u.ac.jp Wed Mar 19 23:57:39 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 20 Mar 2008 12:57:39 +0900 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> Message-ID: <47E1E0B3.9010505@ar.media.kyoto-u.ac.jp> Anton Slesarev wrote: > Hi. > > I'm going to describe what problems I see in current version of > scikits.learn. After that I'll write what I want to improve during > Google Summer of Code. In my last letter I tried to numerate some > limitations in other open-source frameworks such as PyML and Orange. > Hi Anton, Sorry for the late answer, but I have been busy the last few days at a conference. Most of your points are valid, that is there are some useful bits in there now, but disparate. The main reason is that I started working on a new build system for numpy/scipy, which took all my free time. I intend to work again on scikits.learn soon, though. Here are some comments: > Let's start about Scikits.learn. > > First of all is a lack of documentation. I can find nothing beside > David Cournapeau proposal on google Summer of Code. Nothing in wiki > and nothing in maillist. There are few examples in svm, of course. But > it is very hard use only examples. scikits.machine.em and scikits.machine.svm are usable, but they do not have common usage. They do have docs, though (particularly scikits.machine.em: there is 15 pages pdf tutorial). Most examples do not work, but that's only because of some changes in the dataset format (see below), and that should be easy to change. I think examples are really important. The problem with docs is the format and tools to generate them; other people will know better on this particular point. I know I don't like the current tools myself (epydoc, etc...), but that's just my opinion. > I can't find parser of different data formats. Only for datasets. As I > understand datasets don't support sparse data format. On dataset, there is something: http://projects.scipy.org/scipy/scikits/browser/trunk/learn/scikits/learn/datasets/DATASET_PROPOSAL.txt The basic idea is to NOT care about format. Just provides basic conventions, such as any tool who need dataset can use what they want through introspection. In particular, I do not see any problem wrt sparse data, but I have not thought a lot about it, so maybe I missed something. So there is no format, and no parsing needs :) I think it is impossible to agree on something which is useful and usable by everybody (the proposal does not focus exclusively on machine learning), hence the lack of format specification. I worked on a parser for arff (weka format), which could read most arff files I could throw at it. But recently, someone wrote a grammar for the arff format: http://permalink.gmane.org/gmane.comp.ai.weka/11742 Unfortunately, I do not know anything about parsing and co, but I think anybody who studied computer science in some way would know how to get a python module to parse files following this grammar in no time, I guess. > There is no common structure in ML package. It has scattered modules > such as svm, em, ann, but no main idea. Here is how I see thing: different people have different usages. Some people like to try many different learning algorithms (need common structure, general tools, etc...), some just want to focus on one algorithm. It should be important to keep the different "learning" algorithms independant: scikits.machine.em should be usable independently, same for other scikits.machine modules; that is, there should be one level at which you can just use the algorithms with straight numpy arrays (no dataset class or anything). Ideally, I want them to have a 100 % python implementation, so that they can be used for education purpose. Of course, some C/whatever implementation can be possible too, but that should be a complement. So the main idea is to have pre-processing tools, and other things common to most algorithms on one side, and the actual learning algorithms on another side. There is already some code to scale data (handling nan if necessary) in scikits/utils with some basic tests. That's really basic, though. > > If I mistake in understanding current state of affair you can correct me. > > Well, now about what I want to change. > > I am going to make learn package appropriate for text classification. > Also I want to copy most of PyML (pyml.sourceforge.net/ > ) functionality. > > First of all we need sparse data format. I want to write parsers for a > number of common data formats. What do you mean exactly by sparse data ? At the implementation level, ideally, the algorithms should use scipy.sparse, I think. At the "high" level, something like spider should be used: http://www.kyb.tuebingen.mpg.de/bs/people/spider/main.html Spider is quite good, and has a good interface, and this with matlab, which is quite an achievement. Of course, it is your proposal, hence your choice, but I don't think focusing on many format to be the right thing at first. One thing I had in mind was to implement a "proxy" to communicate between the high level representation in scikits.learn and other packages such as weka, orange, etc... This would give a practical way use weka with some python tools, until we get something on our own for visualization. Spider does have something to communicate with weka, for example (it is easier in matlab I guess since matlab has a jvm and weka is in java). cheers, David From millman at berkeley.edu Thu Mar 20 07:15:08 2008 From: millman at berkeley.edu (Jarrod Millman) Date: Thu, 20 Mar 2008 04:15:08 -0700 Subject: [SciPy-dev] NumPy (1.0.5) DocDay (Fri, Mar. 21) Message-ID: Hello, As I mentioned yesterday, I am holding a NumPy DocDay on Friday, March 21st. I am in Paris near the RER B or C Saint-Michel station (with Stefan van der Walt, Matthieu Brucher, and Gael Varoquaux). If you are in the area and want to join us just send me an email by the end of tonight and I will let you know where we are meeting. If you can't stop by, but are still willing to help out we will convene on IRC during the day on Friday (9:30am-?? GMT+1). Come join us at irc.freenode.net (channel scipy). We may update the list of priorities which is still located on the NumPy Trac Wiki: http://projects.scipy.org/scipy/numpy/wiki/DocDays While I am hoping to have everyone focus on NumPy, I would be happy if anyone wants to work on SciPy documentation as well: http://projects.scipy.org/scipy/scipy/wiki/DocDays Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From ondrej at certik.cz Thu Mar 20 08:45:10 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 20 Mar 2008 13:45:10 +0100 Subject: [SciPy-dev] scipy.optimize.nonlin.broyden2 In-Reply-To: <47E1D5EA.8070902@ncsu.edu> References: <47E0569D.8030502@ncsu.edu> <85b5c3130803190925j477c1a2ayc2246d131eaeff78@mail.gmail.com> <47E177E2.30804@ncsu.edu> <85b5c3130803191518g32f5cabl964dee3a3fe03981@mail.gmail.com> <47E1D5EA.8070902@ncsu.edu> Message-ID: <85b5c3130803200545g25072664hf78463df15238cbf@mail.gmail.com> On Thu, Mar 20, 2008 at 4:11 AM, argriffi wrote: > Hi Ondrej, > > As requested I have attached a modification and a test case. The > modification checks a value for equality to zero, but maybe it should > more appropriately handle values near zero. Also, other functions in > nonlin.py may benefit from similar modifications. Excellent, thanks a lot for the work. Here are my comments: denominator = norm(deltaFxm)**2 if not denominator: break use abs(denominator) < eps and define eps as a default parameter of the function, for example eps=1e-10. Also could you please apply this into the scipy svn, and send use the output of "svn di"? So that someone with a svn access can just take it and apply it. As to the tests, please add them to scipy/optimize/tests/test_nonlin.py. Simply do all of this in your local svn copy and send use the result of "svn di". Thanks very much, Ondrej From ondrej at certik.cz Thu Mar 20 08:46:09 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 20 Mar 2008 13:46:09 +0100 Subject: [SciPy-dev] scipy.optimize.nonlin.broyden2 In-Reply-To: <85b5c3130803200545g25072664hf78463df15238cbf@mail.gmail.com> References: <47E0569D.8030502@ncsu.edu> <85b5c3130803190925j477c1a2ayc2246d131eaeff78@mail.gmail.com> <47E177E2.30804@ncsu.edu> <85b5c3130803191518g32f5cabl964dee3a3fe03981@mail.gmail.com> <47E1D5EA.8070902@ncsu.edu> <85b5c3130803200545g25072664hf78463df15238cbf@mail.gmail.com> Message-ID: <85b5c3130803200546h68b6b44x7f7905870df17bd4@mail.gmail.com> On Thu, Mar 20, 2008 at 1:45 PM, Ondrej Certik wrote: > On Thu, Mar 20, 2008 at 4:11 AM, argriffi wrote: > > Hi Ondrej, > > > > > As requested I have attached a modification and a test case. The > > modification checks a value for equality to zero, but maybe it should > > more appropriately handle values near zero. Also, other functions in > > nonlin.py may benefit from similar modifications. > > Excellent, thanks a lot for the work. Here are my comments: > > > denominator = norm(deltaFxm)**2 > if not denominator: > break > > use abs(denominator) < eps > > and define eps as a default parameter of the function, for example eps=1e-10. > > Also could you please apply this into the scipy svn, and send use the use -> us > output of "svn di"? So that someone with a svn access can just take it > and apply it. > > As to the tests, please add them to > scipy/optimize/tests/test_nonlin.py. Simply do all of this in your > local svn copy and send use the result of "svn di". use -> us. O. From david at ar.media.kyoto-u.ac.jp Thu Mar 20 09:28:54 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 20 Mar 2008 22:28:54 +0900 Subject: [SciPy-dev] Splitting sparsetools: tahnk you Message-ID: <47E26696.3090001@ar.media.kyoto-u.ac.jp> Hi Nathan, I saw yesterday that you split the sparsetools module: the max memory usage during build is now half of what used to be necessary, thank you very much for your effort, cheers, David From argriffi at ncsu.edu Thu Mar 20 11:12:26 2008 From: argriffi at ncsu.edu (argriffi) Date: Thu, 20 Mar 2008 11:12:26 -0400 Subject: [SciPy-dev] scipy.optimize.nonlin.broyden2 In-Reply-To: <85b5c3130803200545g25072664hf78463df15238cbf@mail.gmail.com> References: <47E0569D.8030502@ncsu.edu> <85b5c3130803190925j477c1a2ayc2246d131eaeff78@mail.gmail.com> <47E177E2.30804@ncsu.edu> <85b5c3130803191518g32f5cabl964dee3a3fe03981@mail.gmail.com> <47E1D5EA.8070902@ncsu.edu> <85b5c3130803200545g25072664hf78463df15238cbf@mail.gmail.com> Message-ID: <47E27EDA.8000009@ncsu.edu> I've been using a packaged scipy release, and I just checked out the svn scipy trunk but haven't used or modified it yet. What is the best way to manage the two installations so that by default my scripts use the official release, but so that I can alternatively use the svn version by specifying it somehow? I also have a question about the proposed eps argument for broyden2. Could this be named more descriptively? For example, I have seen small values named rel_tol and abs_tol for relative and absolute tolerance in other contexts. If eps has a meaningful interpretation in broyden2 then maybe it should be named more appropriately. Alex Ondrej Certik wrote: > On Thu, Mar 20, 2008 at 4:11 AM, argriffi wrote: > >> Hi Ondrej, >> >> As requested I have attached a modification and a test case. The >> modification checks a value for equality to zero, but maybe it should >> more appropriately handle values near zero. Also, other functions in >> nonlin.py may benefit from similar modifications. >> > > Excellent, thanks a lot for the work. Here are my comments: > > > denominator = norm(deltaFxm)**2 > if not denominator: > break > > use abs(denominator) < eps > > and define eps as a default parameter of the function, for example eps=1e-10. > > Also could you please apply this into the scipy svn, and send use the > output of "svn di"? So that someone with a svn access can just take it > and apply it. > > As to the tests, please add them to > scipy/optimize/tests/test_nonlin.py. Simply do all of this in your > local svn copy and send use the result of "svn di". > > Thanks very much, > Ondrej > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > From nwagner at iam.uni-stuttgart.de Thu Mar 20 12:38:16 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 20 Mar 2008 17:38:16 +0100 Subject: [SciPy-dev] openopt and algencan Message-ID: Hi all, I encountered a problem related to the python interface of algencan. See http://www.ime.usp.br/~egbirgin/tango/py.php The backtrace of my small test (comparison.py) is as follows: starting solver ALGENCAN (license: GPL) with problem unnamed *** glibc detected *** double free or corruption (out): 0x0000000000e17a40 *** PYTHON INTERFACE ERROR: in inip, PyEval_CallFunction returned NULL Program received signal SIGABRT, Aborted. [Switching to Thread 182894186368 (LWP 5735)] 0x0000003390e2e21d in raise () from /lib64/tls/libc.so.6 (gdb) bt #0 0x0000003390e2e21d in raise () from /lib64/tls/libc.so.6 #1 0x0000003390e2fa1e in abort () from /lib64/tls/libc.so.6 #2 0x0000003390e63291 in __libc_message () from /lib64/tls/libc.so.6 #3 0x0000003390e68eae in _int_free () from /lib64/tls/libc.so.6 #4 0x0000003390e691f6 in free () from /lib64/tls/libc.so.6 #5 0x0000002a9df3bd71 in memFree () from /data/home/nwagner/TANGO/pywrapper.so #6 0x0000002a9df3dc6b in pywrapper_solver () from /data/home/nwagner/TANGO/pywrapper.so #7 0x000000000048994b in PyEval_EvalFrameEx (f=0x1234c30, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3564 #8 0x000000000048a205 in PyEval_EvalFrameEx (f=0x124f6b0, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3650 #9 0x000000000048b791 in PyEval_EvalCodeEx (co=0x2a9d6ff288, globals=Variable "globals" is not available. ) at Python/ceval.c:2831 #10 0x00000000004894d1 in PyEval_EvalFrameEx (f=0x120f9e0, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3659 #11 0x000000000048b791 in PyEval_EvalCodeEx (co=0x2a9c911918, globals=Variable "globals" is not available. ) at Python/ceval.c:2831 #12 0x00000000004d8ab8 in function_call (func=0x2a9c911aa0, arg=0x2a9cc91200, kw=0x12393f0) at Objects/funcobject.c:517 #13 0x00000000004194c3 in PyObject_Call (func=Variable "func" is not available. ) at Objects/abstract.c:1860 #14 0x0000000000487f3b in PyEval_EvalFrameEx (f=0x1210980, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3844 #15 0x000000000048b791 in PyEval_EvalCodeEx (co=0x2a9b6b93f0, globals=Variable "globals" is not available. ) at Python/ceval.c:2831 #16 0x00000000004894d1 in PyEval_EvalFrameEx (f=0x6d2690, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3659 #17 0x000000000048b791 in PyEval_EvalCodeEx (co=0x2a955fec60, globals=Variable "globals" is not available. ) at Python/ceval.c:2831 #18 0x000000000048b902 in PyEval_EvalCode (co=Variable "co" is not available. ) at Python/ceval.c:494 #19 0x00000000004adc9e in PyRun_FileExFlags (fp=0x65c010, filename=0x7fbffffa2c "comparison.py", start=Variable "start" is not available. ) at Python/pythonrun.c:1271 #20 0x00000000004aeb1f in PyRun_SimpleFileExFlags (fp=0x65c010, filename=0x7fbffffa2c "comparison.py", closeit=1, flags=0x7fbffff5dc) at Python/pythonrun.c:877 #21 0x0000000000411f77 in Py_Main (argc=-1073743316, argv=Variable "argv" is not available. ) at Modules/main.c:523 #22 0x0000003390e1c3fb in __libc_start_main () from /lib64/tls/libc.so.6 #23 0x000000000041163a in _start () How can I resolve the problem ? Any pointer would be appreciated. Thanks in advance Nils -------------- next part -------------- A non-text attachment was scrubbed... Name: comparison.py Type: application/x-python Size: 2724 bytes Desc: not available URL: From dmitrey.kroshko at scipy.org Thu Mar 20 13:03:25 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Thu, 20 Mar 2008 19:03:25 +0200 Subject: [SciPy-dev] openopt and algencan In-Reply-To: References: Message-ID: <47E298DD.1050203@scipy.org> Hi Nils, I have made some changes to svn, try using ALGENCAN now. Regards, D. Nils Wagner wrote: > Hi all, > > I encountered a problem related to the python interface of algencan. > See http://www.ime.usp.br/~egbirgin/tango/py.php > > The backtrace of my small test (comparison.py) is > as follows: > > > starting solver ALGENCAN (license: GPL) with problem unnamed > *** glibc detected *** double free or corruption (out): > 0x0000000000e17a40 *** > > PYTHON INTERFACE ERROR: in inip, PyEval_CallFunction returned NULL > > > Program received signal SIGABRT, Aborted. > [Switching to Thread 182894186368 (LWP 5735)] > 0x0000003390e2e21d in raise () from /lib64/tls/libc.so.6 > (gdb) bt > #0 0x0000003390e2e21d in raise () from /lib64/tls/libc.so.6 > #1 0x0000003390e2fa1e in abort () from /lib64/tls/libc.so.6 > #2 0x0000003390e63291 in __libc_message () from /lib64/tls/libc.so.6 > #3 0x0000003390e68eae in _int_free () from /lib64/tls/libc.so.6 > #4 0x0000003390e691f6 in free () from /lib64/tls/libc.so.6 > #5 0x0000002a9df3bd71 in memFree () from > /data/home/nwagner/TANGO/pywrapper.so > #6 0x0000002a9df3dc6b in pywrapper_solver () from > /data/home/nwagner/TANGO/pywrapper.so > #7 0x000000000048994b in PyEval_EvalFrameEx (f=0x1234c30, > throwflag=Variable "throwflag" is not available. > ) at Python/ceval.c:3564 > #8 0x000000000048a205 in PyEval_EvalFrameEx (f=0x124f6b0, > throwflag=Variable "throwflag" is not available. > ) at Python/ceval.c:3650 > #9 0x000000000048b791 in PyEval_EvalCodeEx (co=0x2a9d6ff288, > globals=Variable "globals" is not available. > ) at Python/ceval.c:2831 > #10 0x00000000004894d1 in PyEval_EvalFrameEx (f=0x120f9e0, > throwflag=Variable "throwflag" is not available. > ) at Python/ceval.c:3659 > #11 0x000000000048b791 in PyEval_EvalCodeEx (co=0x2a9c911918, > globals=Variable "globals" is not available. > ) at Python/ceval.c:2831 > #12 0x00000000004d8ab8 in function_call (func=0x2a9c911aa0, > arg=0x2a9cc91200, kw=0x12393f0) at Objects/funcobject.c:517 > #13 0x00000000004194c3 in PyObject_Call (func=Variable "func" is not > available. > ) at Objects/abstract.c:1860 > #14 0x0000000000487f3b in PyEval_EvalFrameEx (f=0x1210980, > throwflag=Variable "throwflag" is not available. > ) at Python/ceval.c:3844 > #15 0x000000000048b791 in PyEval_EvalCodeEx (co=0x2a9b6b93f0, > globals=Variable "globals" is not available. > ) at Python/ceval.c:2831 > #16 0x00000000004894d1 in PyEval_EvalFrameEx (f=0x6d2690, > throwflag=Variable "throwflag" is not available. > ) at Python/ceval.c:3659 > #17 0x000000000048b791 in PyEval_EvalCodeEx (co=0x2a955fec60, > globals=Variable "globals" is not available. > ) at Python/ceval.c:2831 > #18 0x000000000048b902 in PyEval_EvalCode (co=Variable "co" is not > available. > ) at Python/ceval.c:494 > #19 0x00000000004adc9e in PyRun_FileExFlags (fp=0x65c010, > filename=0x7fbffffa2c "comparison.py", start=Variable "start" is not > available. > ) > at Python/pythonrun.c:1271 > #20 0x00000000004aeb1f in PyRun_SimpleFileExFlags (fp=0x65c010, > filename=0x7fbffffa2c "comparison.py", closeit=1, > flags=0x7fbffff5dc) at Python/pythonrun.c:877 > #21 0x0000000000411f77 in Py_Main (argc=-1073743316, argv=Variable > "argv" is not available. > ) at Modules/main.c:523 > #22 0x0000003390e1c3fb in __libc_start_main () from /lib64/tls/libc.so.6 > #23 0x000000000041163a in _start () > > How can I resolve the problem ? > > Any pointer would be appreciated. > > Thanks in advance > > Nils > > ------------------------------------------------------------------------ > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > From nwagner at iam.uni-stuttgart.de Thu Mar 20 13:58:17 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 20 Mar 2008 18:58:17 +0100 Subject: [SciPy-dev] openopt and algencan In-Reply-To: <47E298DD.1050203@scipy.org> References: <47E298DD.1050203@scipy.org> Message-ID: On Thu, 20 Mar 2008 19:03:25 +0200 dmitrey wrote: > Hi Nils, > I have made some changes to svn, try using ALGENCAN now. Great ! It works for me. Thank you very much ! Nils From Adam.Raczkowski at tufts.edu Thu Mar 20 14:21:08 2008 From: Adam.Raczkowski at tufts.edu (AJ Raczkowski) Date: Thu, 20 Mar 2008 14:21:08 -0400 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea Message-ID: Hi, My name is Adam Raczkowski, and I'm interested in working on the proposed GSoC project for this summer, 2008. I am currently in my junior year as an undergrad at Tufts University. My majors are computer science and mathematics. I discovered SciPy/NumPy during a class I took this past fall in numerical linear algebra, and I remain very much interested in that field. I would like to discuss this project (conversion of code to cython) with a possible mentor so that I can ensure this would be a good fit for me. Regards, Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitrey.kroshko at scipy.org Thu Mar 20 14:32:53 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Thu, 20 Mar 2008 20:32:53 +0200 Subject: [SciPy-dev] openopt and algencan In-Reply-To: References: <47E298DD.1050203@scipy.org> Message-ID: <47E2ADD5.4060209@scipy.org> Nils, try using p.xlabel = 'nf' this is temporary name for "number of function evaluations", I will think what is better name for the case, and then commit it into OO doc. This is more appropriate vs your "iteration" that you have chosen for your x label. Your "iteration" is just a single objfunc evaluation, while users will suppose solvers have different iterations with different number of objFunc evaluations in each one. Also, take into account, that ralg can yield infeasible iteration points, so your single-subplot picture doesn't provide all convergence info. Ralg is intended to handle lb-ub problems more bette in future. Regards, D. Nils Wagner wrote: > On Thu, 20 Mar 2008 19:03:25 +0200 > dmitrey wrote: > >> Hi Nils, >> I have made some changes to svn, try using ALGENCAN now. >> > > Great ! It works for me. Thank you very much ! > > Nils > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > From nwagner at iam.uni-stuttgart.de Thu Mar 20 15:49:32 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 20 Mar 2008 20:49:32 +0100 Subject: [SciPy-dev] openopt and algencan In-Reply-To: <47E2ADD5.4060209@scipy.org> References: <47E298DD.1050203@scipy.org> <47E2ADD5.4060209@scipy.org> Message-ID: On Thu, 20 Mar 2008 20:32:53 +0200 dmitrey wrote: > Nils, try using p.xlabel = 'nf' > > this is temporary name for "number of function >evaluations", I will > think what is better name for the case, and then commit >it into OO doc. > > This is more appropriate vs your "iteration" that you >have chosen for > your x label. > Your "iteration" is just a single objfunc evaluation, >while users will > suppose solvers have different iterations with different >number of > objFunc evaluations in each one. > > Also, take into account, that ralg can yield infeasible >iteration > points, so your single-subplot picture doesn't provide >all convergence info. > Ralg is intended to handle lb-ub problems more bette in >future. > Regards, D. > > > > Nils Wagner wrote: >> On Thu, 20 Mar 2008 19:03:25 +0200 >> dmitrey wrote: >> >>> Hi Nils, >>> I have made some changes to svn, try using ALGENCAN now. >>> >> >> Great ! It works for me. Thank you very much ! >> >> Nils >> _______________________________________________ >> Scipy-dev mailing list >> Scipy-dev at scipy.org >> http://projects.scipy.org/mailman/listinfo/scipy-dev >> >> >> >> > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev Hi Dmitrey, Thank you for your comments. BTW, do you have an optional specification file algencan.dat that you can send me ? I just noticed the message The optional specification file algencan.dat was not found in the current directory (this is not a problem nor an error). The default values for the ALGENCAN parameters will be used. ALGENCAN PARAMETERS: rhoauto = T rhotype = 1 rhomult = 1.0000E+01 rhofrac = 5.0000E-01 gtype = 0 hptype = 6 intype = 0 precond = QNCGNA checkder = F epsfeas = 1.0000E-06 epsopt = 1.0000E-06 maxoutit = 50 maxtotit = 1000 maxtotfc = 10000 iprint = 0 ncomp = 5 ALGENCAN has finished solving the problem unnamed istop: 2 (|| gradient F(X[k]) || < gradtol) Solver: Time Elapsed = 0.21 CPU Time Elapsed = 0.2 objFunValue: 443.752171145 (feasible, max constraint = 0) Nils From dmitrey.kroshko at scipy.org Thu Mar 20 16:05:33 2008 From: dmitrey.kroshko at scipy.org (dmitrey) Date: Thu, 20 Mar 2008 22:05:33 +0200 Subject: [SciPy-dev] openopt and algencan In-Reply-To: References: <47E298DD.1050203@scipy.org> <47E2ADD5.4060209@scipy.org> Message-ID: <47E2C38D.6000908@scipy.org> Nils Wagner wrote: > On Thu, 20 Mar 2008 20:32:53 +0200 > dmitrey wrote: > >> Nils, try using p.xlabel = 'nf' >> >> this is temporary name for "number of function >> evaluations", I will >> think what is better name for the case, and then commit >> it into OO doc. >> >> This is more appropriate vs your "iteration" that you >> have chosen for >> your x label. >> Your "iteration" is just a single objfunc evaluation, >> while users will >> suppose solvers have different iterations with different >> number of >> objFunc evaluations in each one. >> >> Also, take into account, that ralg can yield infeasible >> iteration >> points, so your single-subplot picture doesn't provide >> all convergence info. >> Ralg is intended to handle lb-ub problems more bette in >> future. >> Regards, D. >> >> >> >> Nils Wagner wrote: >> >>> On Thu, 20 Mar 2008 19:03:25 +0200 >>> dmitrey wrote: >>> >>> >>>> Hi Nils, >>>> I have made some changes to svn, try using ALGENCAN now. >>>> >>>> >>> >>> Great ! It works for me. Thank you very much ! >>> >>> Nils >>> _______________________________________________ >>> Scipy-dev mailing list >>> Scipy-dev at scipy.org >>> http://projects.scipy.org/mailman/listinfo/scipy-dev >>> >>> >>> >>> >>> >> _______________________________________________ >> Scipy-dev mailing list >> Scipy-dev at scipy.org >> http://projects.scipy.org/mailman/listinfo/scipy-dev >> > > Hi Dmitrey, > > Thank you for your comments. > > BTW, do you have an > optional specification file algencan.dat > that you can send me ? > > I just noticed the message > > The optional specification file algencan.dat was not > found in the current > directory (this is not a problem nor an error). The > default values for the > ALGENCAN parameters will be used. > > ALGENCAN PARAMETERS: > rhoauto = T > rhotype = 1 > rhomult = 1.0000E+01 > rhofrac = 5.0000E-01 > gtype = 0 > hptype = 6 > intype = 0 > precond = QNCGNA > checkder = F > epsfeas = 1.0000E-06 > epsopt = 1.0000E-06 > maxoutit = 50 > maxtotit = 1000 > maxtotfc = 10000 > iprint = 0 > ncomp = 5 > This output is generated by ALGENCAN, not OO. You'd better contact ALGENCAN developers about the file. Regards, D. > ALGENCAN has finished solving the problem unnamed > istop: 2 (|| gradient F(X[k]) || < gradtol) > Solver: Time Elapsed = 0.21 CPU Time Elapsed = 0.2 > objFunValue: 443.752171145 (feasible, max constraint = 0) > > Nils > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > From ondrej at certik.cz Thu Mar 20 17:47:58 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 20 Mar 2008 22:47:58 +0100 Subject: [SciPy-dev] scipy.optimize.nonlin.broyden2 In-Reply-To: <47E27EDA.8000009@ncsu.edu> References: <47E0569D.8030502@ncsu.edu> <85b5c3130803190925j477c1a2ayc2246d131eaeff78@mail.gmail.com> <47E177E2.30804@ncsu.edu> <85b5c3130803191518g32f5cabl964dee3a3fe03981@mail.gmail.com> <47E1D5EA.8070902@ncsu.edu> <85b5c3130803200545g25072664hf78463df15238cbf@mail.gmail.com> <47E27EDA.8000009@ncsu.edu> Message-ID: <85b5c3130803201447q4fa54298q5acd9f195132a890@mail.gmail.com> On Thu, Mar 20, 2008 at 4:12 PM, argriffi wrote: > I've been using a packaged scipy release, and I just checked out the svn > scipy trunk but haven't used or modified it yet. What is the best way > to manage the two installations so that by default my scripts use the > official release, but so that I can alternatively use the svn version by > specifying it somehow? Maybe there are other ways how to deal with this, but I do: ./setup install --home=/some/path everytime I change something - it takes just a while to do it if you only modify python files. And then you add /some/path into your PYTHONPATH - just temporarily. > > I also have a question about the proposed eps argument for broyden2. > Could this be named more descriptively? For example, I have seen small > values named rel_tol and abs_tol for relative and absolute tolerance in > other contexts. If eps has a meaningful interpretation in broyden2 then > maybe it should be named more appropriately. Absolutely, name it more appropriate. That was just a general suggestion from me how to do things. Ondrej From wnbell at gmail.com Fri Mar 21 13:33:49 2008 From: wnbell at gmail.com (Nathan Bell) Date: Fri, 21 Mar 2008 12:33:49 -0500 Subject: [SciPy-dev] Splitting sparsetools: tahnk you In-Reply-To: <47E26696.3090001@ar.media.kyoto-u.ac.jp> References: <47E26696.3090001@ar.media.kyoto-u.ac.jp> Message-ID: On Thu, Mar 20, 2008 at 8:28 AM, David Cournapeau wrote: > Hi Nathan, > > I saw yesterday that you split the sparsetools module: the max > memory usage during build is now half of what used to be necessary, > thank you very much for your effort, I'm glad it to hear that. In addition to the concerns you raised, I realized that sparsetools.h was a bit too large itself. At some point I will want to reintroduce the unrolled variants of bsr_matvec and bsr_matmat. If these template-heavy functions cause similar problems for you, let me know and we'll figure something out. Ideally we would generate the SWIG interface files and wrappers dynamically. We could then automatically create interface files on a per function basis (if need be) without polluting the repository. E.g. csr_foo() -> csr_foo.i -> csr_foo_wrap.cxx In this case csr_foo.i consists of only a few lines, much like csr.i or dia.i do now. I wouldn't recommend this approach yet, since few people have SWIG installed and even fewer have the most recent SWIG installed (which is necessary on some platforms). Perhaps in a year or so. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From stefan at sun.ac.za Sun Mar 23 21:15:14 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 24 Mar 2008 02:15:14 +0100 Subject: [SciPy-dev] Numpy/Cython Google Summer of Code project idea In-Reply-To: References: Message-ID: <9457e7c80803231815p14e16c7dsefe96f10cab2c5dc@mail.gmail.com> Hi Adam On Thu, Mar 20, 2008 at 7:21 PM, AJ Raczkowski wrote: > My name is Adam Raczkowski, and I'm interested in working on the proposed > GSoC project for this summer, 2008. > > I am currently in my junior year as an undergrad at Tufts University. My > majors are computer science and mathematics. I discovered SciPy/NumPy during > a class I took this past fall in numerical linear algebra, and I remain very > much interested in that field. > > I would like to discuss this project (conversion of code to cython) with a > possible mentor so that I can ensure this would be a good fit for me. I see there hasn't been any response yet, so let me chip in. There are many advantages to having our routines in cython, so that is certainly a good suggestion for a project. Unfortunately, the numpy support in cython is the topic of another GSOC 2008 project IIRC, and as I see it your project would require the work done there to continue. Let's hear what the other people on the list have to say. Regards St?fan From slesarev.anton at gmail.com Mon Mar 24 05:41:21 2008 From: slesarev.anton at gmail.com (Anton Slesarev) Date: Mon, 24 Mar 2008 12:41:21 +0300 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> Message-ID: <11abf2720803240241j69b5ec0au1e5dc132949e6be8@mail.gmail.com> It really looks pretty good. I think that such functionality should be in scipy(or scikits). I'll try to take it into account in my proposal. On Wed, Mar 19, 2008 at 4:43 AM, Jarrod Millman wrote: > Hey Anton, > > Sorry I haven't responded sooner; I am at a coding sprint. Anyway, I > am meeting with the developers of Multivariate Pattern Analysis in > Python: http://pkg-exppsy.alioth.debian.org/pymvpa/ > I think that this package looks very good. David Cournapeau will be > visiting Berkeley in early April and I plan to discuss whether it > would make sense to merge this into scikits.learn. So it would make > sense for you to take a look at the manual here: > http://pkg-exppsy.alioth.debian.org/pymvpa/manual.html > > I am really excited that you are planning to submit a proposal for the > SoC. > > Thanks, > > -- > Jarrod Millman > Computational Infrastructure for Research Labs > 10 Giannini Hall, UC Berkeley > phone: 510.643.4014 > http://cirl.berkeley.edu/ > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > -- Anton Slesarev -------------- next part -------------- An HTML attachment was scrubbed... URL: From slesarev.anton at gmail.com Mon Mar 24 06:09:44 2008 From: slesarev.anton at gmail.com (Anton Slesarev) Date: Mon, 24 Mar 2008 13:09:44 +0300 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <47E1E0B3.9010505@ar.media.kyoto-u.ac.jp> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <47E1E0B3.9010505@ar.media.kyoto-u.ac.jp> Message-ID: <11abf2720803240309k1b29d968jc2980a7dfd746fa@mail.gmail.com> > > > The basic idea is to NOT care about format. Just provides basic > conventions, such as any tool who need dataset can use what they want > through introspection. In particular, I do not see any problem wrt > sparse data, but I have not thought a lot about it, so maybe I missed > something. > If you don't care about format users will have to do it. And users don't like to do it. I don't want parsers for all formats in the world. Tell the truth, I don't exactly know what I want, but in nearly future I'll try to learn this question and describe it in derails in proposal. > > > There is no common structure in ML package. It has scattered modules > > such as svm, em, ann, but no main idea. > > Here is how I see thing: different people have different usages. Some > people like to try many different learning algorithms (need common > structure, general tools, etc...), some just want to focus on one > algorithm. It should be important to keep the different "learning" > algorithms independant: scikits.machine.em should be usable > independently, same for other scikits.machine modules; that is, there > should be one level at which you can just use the algorithms with > straight numpy arrays (no dataset class or anything). Ideally, I want > them to have a 100 % python implementation, so that they can be used for > education purpose. Of course, some C/whatever implementation can be > possible too, but that should be a complement. > Svm implementation in scikits has a few strange dataset classes(train dataset, test dataset, etc). My idea is each classifier should use common dataset classes. It make possible common preprocessing tools. > > So the main idea is to have pre-processing tools, and other things > common to most algorithms on one side, and the actual learning > algorithms on another side. > > There is already some code to scale data (handling nan if necessary) in > scikits/utils with some basic tests. That's really basic, though.> > > If I mistake in understanding current state of affair you can correct me. > > > > Well, now about what I want to change. > > > > I am going to make learn package appropriate for text classification. > > Also I want to copy most of PyML (pyml.sourceforge.net/ > > ) functionality. > > > > First of all we need sparse data format. I want to write parsers for a > > number of common data formats. > > What do you mean exactly by sparse data ? At the implementation level, > ideally, the algorithms should use scipy.sparse, I think. At the "high" > level, something like spider should be used: > I mean that I have 1 million text pages with 150 thousands different words(features), but each document has only small part of all 150 thousands world. And if I use simple matrix it will be huge. But if i use sparse format such as libsvm data format than input file will be much smaller. I don't know how to do it with scikits now. I know how to do it with libsvm and many other tools, but I want to make scipy appropriate for this task. And I want make a tutorial in which one paragraph will be about "Sparse data". > > http://www.kyb.tuebingen.mpg.de/bs/people/spider/main.html > > Spider is quite good, and has a good interface, and this with matlab, > which is quite an achievement. > > Of course, it is your proposal, hence your choice, but I don't think > focusing on many format to be the right thing at first. One thing I had > in mind was to implement a "proxy" to communicate between the high level > representation in scikits.learn and other packages such as weka, orange, > etc... This would give a practical way use weka with some python tools, > until we get something on our own for visualization. Spider does have > something to communicate with weka, for example (it is easier in matlab > I guess since matlab has a jvm and weka is in java). > I get it, but I don't like java:) And I think a lot of weka functionality is not very difficult to implement in scipy. > > cheers, > > David > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > -- Anton Slesarev -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Mon Mar 24 06:41:24 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 24 Mar 2008 19:41:24 +0900 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <11abf2720803240309k1b29d968jc2980a7dfd746fa@mail.gmail.com> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <47E1E0B3.9010505@ar.media.kyoto-u.ac.jp> <11abf2720803240309k1b29d968jc2980a7dfd746fa@mail.gmail.com> Message-ID: <47E78554.2090901@ar.media.kyoto-u.ac.jp> Anton Slesarev wrote: > If you don't care about format users will have to do it. Not really. I think having one format of dataset usable by any learning algorithm is impossible. The common format internally is the array given by numpy array, sparse, etc... What people do care about is the interface to the dataset, not the format of the dataset itself: they want to pre process them, select some attributes, etc... Nothing prevents anyone from implementing some "proxies" to convert one common dataset format into a bunch of numpy arrays. Did you look at the dataset proposal ? It does provide some examples. Also, there is a proof of concept that you can implement data attributes selection, whitening and normalization on them. Again, the current state is quite messy, but I really hope to clean the mess in a few weeks when I will go to the US. Right now, I have one conference talk to prepare, and two articles to finish for in a few days, so I can't do it now, I am sorry for that, I think it would make clearer what I had in mind. > > Svm implementation in scikits has a few strange dataset classes(train > dataset, test dataset, etc). My idea is each classifier should use > common dataset classes. It make possible common preprocessing tools. svm has the "strange" dataset classes imposed by libsvm. That's typically one example why I don't think you will be able to design a format usable by everything. Depending on the existing implementation, there are already incompatible dataset classes. I agree that common pre-processing tools should be available. The way I saw it was to do all this with pure numpy arrays, and then use some proxies to convert back and forth between different packages. Each learning packages does it own thing, should have a way to convert between its own format and a common interface, and the common tools deal with the common interface. I think you should not focus on format, but on interfaces. Almost any "learning" algorithm will need the features, the labels, and maybe a representation of the labels. This cover supervised learning, clustering and regression. That's the only thing defined by the dataset proposal. > > I mean that I have 1 million text pages with 150 thousands different > words(features), but each document has only small part of all 150 > thousands world. And if I use simple matrix it will be huge. But if > i use sparse format such as libsvm data format than input file will be > much smaller. I don't know how to do it with scikits now. I know how > to do it with libsvm and many other tools, but I want to make scipy > appropriate for this task. And I want make a tutorial in which one > paragraph will be about "Sparse data". I understand sparse, I don't understand why you cannot use existing scipy implementation :) > > I get it, but I don't like java:) I don't like it either, but it was an example of how the interface of spider could be used with foreign packages, weka here. > And I think a lot of weka functionality is not very difficult to > implement in scipy. Well, nothing is really difficult, in a sense, it just needs time. But before we can provide everything weka can provide (and weka here is really one example), I think a simple workflow where you can go back and forth between between environments is nice. It means more people use scikits.learn, which mean potentially more developers, etc... cheers, David From mcoletti at gmail.com Mon Mar 24 06:50:18 2008 From: mcoletti at gmail.com (Mark Coletti) Date: Mon, 24 Mar 2008 10:50:18 +0000 (UTC) Subject: [SciPy-dev] building SciPy for CentOS 5 References: <47D5AAD4.6020102@ucar.edu> Message-ID: Joseph VanAndel ucar.edu> writes: > However, scipy doesn't work: > > import scipy > Traceback (most recent call last): > File "", line 1, in > File "scipy/__init__.py", line 54, in > from __config__ import show as show_config > ImportError: No module named __config__ > I'm having the same problem with my SuSE 10.1 box. I've had to install python 2.5 in /usr/local/; 2.4 is still in /usr/bin. This *shouldn't* be a problem, but I worry there might be some sort of conflict between the two. I hope there's not some compatibility problem between the latest python and scipy. Have you heard any solutions from others? Cheers, Mark Coletti mcoletti at gmail.com From david at ar.media.kyoto-u.ac.jp Mon Mar 24 07:18:18 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 24 Mar 2008 20:18:18 +0900 Subject: [SciPy-dev] building SciPy for CentOS 5 In-Reply-To: References: <47D5AAD4.6020102@ucar.edu> Message-ID: <47E78DFA.1030101@ar.media.kyoto-u.ac.jp> Mark Coletti wrote: > > I'm having the same problem with my SuSE 10.1 box. I've had to install python > 2.5 in /usr/local/; 2.4 is still in /usr/bin. This *shouldn't* be a problem, > but I worry there might be some sort of conflict between the two. I hope > there's not some compatibility problem between the latest python and scipy. > This is because you are trying to import numpy while being in the numpy source tree, which does not work. The solution is to simply relaunch your python interpreter from another directory. The last svn has a better error message, because the above error is indeed quite obscure. Also, note that I put some CENTOS 5.0 RPM (well, RHEL 5, but they are supposed to be the same if I understand correctly) for scipy and numpy on ashigabou: http://download.opensuse.org/repositories/home:/ashigabou/ cheers, David From wnbell at gmail.com Mon Mar 24 08:55:15 2008 From: wnbell at gmail.com (Nathan Bell) Date: Mon, 24 Mar 2008 07:55:15 -0500 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <47E78554.2090901@ar.media.kyoto-u.ac.jp> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <47E1E0B3.9010505@ar.media.kyoto-u.ac.jp> <11abf2720803240309k1b29d968jc2980a7dfd746fa@mail.gmail.com> <47E78554.2090901@ar.media.kyoto-u.ac.jp> Message-ID: On Mon, Mar 24, 2008 at 5:41 AM, David Cournapeau wrote: > > I mean that I have 1 million text pages with 150 thousands different > > words(features), but each document has only small part of all 150 > > thousands world. And if I use simple matrix it will be huge. But if > > i use sparse format such as libsvm data format than input file will be > > much smaller. I don't know how to do it with scikits now. I know how > > to do it with libsvm and many other tools, but I want to make scipy > > appropriate for this task. And I want make a tutorial in which one > > paragraph will be about "Sparse data". > > I understand sparse, I don't understand why you cannot use existing > scipy implementation :) Anton, can you describe libsvm's sparse format? I think it's highly likely that scipy.sparse supports the functionality you need. Currently you can load a sparse matrix from disk using MatrixMarket format (scipy.io.mmread) or MATLAB format (scipy.io.loadmat). Both of these functions should be fast enough for your 150K by 1M example. FWIW the MATLAB files will generally be smaller and load faster. -- Nathan Bell wnbell at gmail.com http://graphics.cs.uiuc.edu/~wnbell/ From slesarev.anton at gmail.com Mon Mar 24 09:30:25 2008 From: slesarev.anton at gmail.com (Anton Slesarev) Date: Mon, 24 Mar 2008 16:30:25 +0300 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <47E78554.2090901@ar.media.kyoto-u.ac.jp> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <47E1E0B3.9010505@ar.media.kyoto-u.ac.jp> <11abf2720803240309k1b29d968jc2980a7dfd746fa@mail.gmail.com> <47E78554.2090901@ar.media.kyoto-u.ac.jp> Message-ID: <11abf2720803240630r4ce2640dkc9a4fc8a219338fc@mail.gmail.com> On Mon, Mar 24, 2008 at 1:41 PM, David Cournapeau < david at ar.media.kyoto-u.ac.jp> wrote: > Anton Slesarev wrote: > > If you don't care about format users will have to do it. > > Not really. I think having one format of dataset usable by any learning > algorithm is impossible. The common format internally is the array given > by numpy array, sparse, etc... What people do care about is the > interface to the dataset, not the format of the dataset itself: they > want to pre process them, select some attributes, etc... Yes! We need common (for all scikit.learn) dataset class, which can load different formats of data. I mean approximately this. > > > Nothing prevents anyone from implementing some "proxies" to convert one > common dataset format into a bunch of numpy arrays. Did you look at the > dataset proposal ? It does provide some examples. Also, there is a proof > of concept that you can implement data attributes selection, whitening > and normalization on them. I've read it. And it is not support sparse format. >From proposal: - selecting only a subset of all samples. - selecting only a subset of the attributes (only sepal length and width, for example). - selecting only the samples of a given class. - small summary of the dataset. I need different attributes for different samples. > > > > > Svm implementation in scikits has a few strange dataset classes(train > > dataset, test dataset, etc). My idea is each classifier should use > > common dataset classes. It make possible common preprocessing tools. > > svm has the "strange" dataset classes imposed by libsvm. That's > typically one example why I don't think you will be able to design a > format usable by everything. Depending on the existing implementation, > there are already incompatible dataset classes. > > I agree that common pre-processing tools should be available. The way I > saw it was to do all this with pure numpy arrays, and then use some > proxies to convert back and forth between different packages. Each > learning packages does it own thing, should have a way to convert > between its own format and a common interface, and the common tools deal > with the common interface. > > I think you should not focus on format, but on interfaces. Almost any > "learning" algorithm will need the features, the labels, and maybe a > representation of the labels. This cover supervised learning, clustering > and regression. That's the only thing defined by the dataset proposal. > Yes > > > > > I mean that I have 1 million text pages with 150 thousands different > > words(features), but each document has only small part of all 150 > > thousands world. And if I use simple matrix it will be huge. But if > > i use sparse format such as libsvm data format than input file will be > > much smaller. I don't know how to do it with scikits now. I know how > > to do it with libsvm and many other tools, but I want to make scipy > > appropriate for this task. And I want make a tutorial in which one > > paragraph will be about "Sparse data". > > I understand sparse, I don't understand why you cannot use existing > scipy implementation :) > Becouse you write about learn.datasets, but you want I use scipy implementation without feature selection, common normalization and other features you just describe. I say that it should be in wrapper which is call dataset(maybe sparsedataset) and user don't want what in it. Yes it can be implemented by .sparse. But it should be implemented, thats all I want to say. > > > > > I get it, but I don't like java:) > > I don't like it either, but it was an example of how the interface of > spider could be used with foreign packages, weka here. > > > And I think a lot of weka functionality is not very difficult to > > implement in scipy. > > Well, nothing is really difficult, in a sense, it just needs time. But > before we can provide everything weka can provide (and weka here is > really one example), I think a simple workflow where you can go back and > forth between between environments is nice. It means more people use > scikits.learn, which mean potentially more developers, etc... > May be. > > cheers, > > David > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > We speak about the same thing. I just want to improve current dataset, one of the improving to make it possible to work with sparse data. A format of sparse text files is really no matter. I understand your position about integrating weka or something else in scipy. May be it is a good idea. But tell the truth it is hard to imagine for me in details. I want to integrate some separate libraries such as libsvm, bbr and something else or rewrite it in python if it is possible. -- Anton Slesarev -------------- next part -------------- An HTML attachment was scrubbed... URL: From slesarev.anton at gmail.com Mon Mar 24 09:35:02 2008 From: slesarev.anton at gmail.com (Anton Slesarev) Date: Mon, 24 Mar 2008 16:35:02 +0300 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <47E1E0B3.9010505@ar.media.kyoto-u.ac.jp> <11abf2720803240309k1b29d968jc2980a7dfd746fa@mail.gmail.com> <47E78554.2090901@ar.media.kyoto-u.ac.jp> Message-ID: <11abf2720803240635gf07d044k5519aab00851b445@mail.gmail.com> On Mon, Mar 24, 2008 at 3:55 PM, Nathan Bell wrote: > > On Mon, Mar 24, 2008 at 5:41 AM, David Cournapeau > wrote: > > > I mean that I have 1 million text pages with 150 thousands different > > > words(features), but each document has only small part of all 150 > > > thousands world. And if I use simple matrix it will be huge. But if > > > i use sparse format such as libsvm data format than input file will be > > > much smaller. I don't know how to do it with scikits now. I know how > > > to do it with libsvm and many other tools, but I want to make scipy > > > appropriate for this task. And I want make a tutorial in which one > > > paragraph will be about "Sparse data". > > > > I understand sparse, I don't understand why you cannot use existing > > scipy implementation :) > > Anton, can you describe libsvm's sparse format? I think it's highly > likely that scipy.sparse supports the functionality you need. > > Currently you can load a sparse matrix from disk using MatrixMarket > format (scipy.io.mmread) or MATLAB format (scipy.io.loadmat). Both > of these functions should be fast enough for your 150K by 1M example. > > FWIW the MATLAB files will generally be smaller and load faster. > > -- > Nathan Bell wnbell at gmail.com > http://graphics.cs.uiuc.edu/~wnbell/ > > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > libsvm format: "libsvm uses the so called "sparse" format where zero values do not need to be stored. Hence a data with attributes 1 0 2 0 is represented as 1:1 3:2" I understand that it is possible to use scipy.sparse and something else but what about if I need to make feature selection or some specific normalization? I think that we can integrate this procedure(with scipy.sparse and reading huge files) to dataset class in scikits.learn. -- Anton Slesarev -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at onerussian.com Mon Mar 24 10:32:31 2008 From: lists at onerussian.com (Yaroslav Halchenko) Date: Mon, 24 Mar 2008 10:32:31 -0400 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <11abf2720803240241j69b5ec0au1e5dc132949e6be8@mail.gmail.com> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <11abf2720803240241j69b5ec0au1e5dc132949e6be8@mail.gmail.com> Message-ID: <20080324143230.GF26722@washoe.rutgers.edu> Hi Anton, Thank you for the positive feedback. Let me introduce myself briefly: I am one of the authors of PyMVPA and met Jarrod at NiPy coding sprint at Paris. There we met with another group (of Jean B. Poline) which develops analogous (yet closed source but due to our persuasion hopefully open-source soon) toolbox. Although we had a lot of ideas in common, some aspects were conceptually different (unfortunately I've forgotten exact name of their toolbox: mindmine, or smth like that) In the case of our PyMVPA we tried to build up a framework where it is possible to combine various ML boxes together via parametrization of the classes in the constructors. That could serve an equivalent role to those GUI-based frameworks where you build your analysis from blocks by connecting them with "lines". And the whole PyMVPA computation pipeline is initiated whenever resultant object sees some data (like within train() of a classifier). It is also somewhat similar to approach taken by MDP guys (http://mdp-toolkit.sourceforge.net/). Also we have similar approach to existing scikits.learn to abstract all relevant data within a Dataset class. The primary user-base we target originally with PyMVPA is brainimaging research community, thus we had to provide not only simple blocks but somewhat obvious and straightforward to use software and a reasonable documentation. That could cost us little loss of generality, although I don't see it happening yet ;-) In that other toolbox, they took the approach of creating a collection of independent blocks which take just basic data structures (ie numpy arrays) as their parameters -- 1 for data, 1 for labels if that is a fit() of a classifier, and I think it might be more appropriate for the scikits.learn, so that it simply provides all necessary building blocks for any other higher level workflow creation and reuse, and there is no explicit need to decide should we store sparse or dense arrays, etc. It is just that all blocks should have unified interface within the classes of the same kind (and appropriate modularization/hierarchy of cause). Another aspect of scikits.learn to think about is either should it reimplement existing ML tools (e.g. SVM, etc) or may be just provide unified interface to their implementation elsewhere -- like SVMs from shogun. Probably it should do both since some classifiers might be not yet freely available from the libraries (like SMLR which we have within our PyMVPA). That would make it possible to utilize many ML tools within a single point of entry (ie scikits.learn). And few words about PyMVPA: I think we have few cute implementation ideas within our PyMVPA which could be borrowed/adapted within scikits.learn (if PyMVPA is not to become an integral part of .learn ;-)) 1. State variables: any classifier internally might store more than just nuisance variables of a classifier, but some times that additional data is costly either to store or to compute. State variable, which looks outside just like a property of an instance, can be enabled and then it actually can be charged with the data. That is why PyMVPA Classifiers can store and expose lots of additional information about internal processing which has already happened. For instance, in recursive feature elimination, it might be interesting not only to get an optimal feature set, but to see/analyze all feature sensitivities per each step of the elimination. In PyMVPA's RFE, if state 'sensitivities' enabled, it just appends a new sensitivity each time, so after RFE is done, it can be easily extracted from that object for further analysis. But that is not a desired behavior by default since it might exhaust the memory ;-) A little more about statefull objects: http://pkg-exppsy.alioth.debian.org/pymvpa/manual.html#stateful-objects 2. Debug (or simply progress) information: for the quick assessment of internal computational workflow, and seeing what takes the longest, etc, it is critical imho to provide a mechanism to expose what is happening inside of a classifier/feature_selection/etc while it being 'processed'. In the case of PyMVPA we provide debug target (ID) for each interesting class, thus debugging output can be spit out for anything of interest. For example: *$> MVPA_DEBUG=CLF,SVM MVPA_DEBUG_METRICS=asctime,reltime,vmem ./clfs_examples.py 2>&1 Haxby 8-cat subject 1: [CLF] DBG{ 'Mon Mar 24 10:16:05 2008' '0.000 sec' 'VmSize:\t 121164 kB'}: Training classifier SVMBase(kernel_type=0, C=-1.0, degree=3, weight_label=[], probability=0, shrinking=1, weight=[], eps=1e-05, svm_type=0, p=0.1, cache_size=100, nr_weight=0, coef0=0.0, nu=0.5, gamma=0.0, enable_states=['training_time', 'predicting_time', 'predictions', 'trained_labels']) on dataset [SVM] DBG{ 'Mon Mar 24 10:16:07 2008' '1.113 sec' 'VmSize:\t 131092 kB'}: Default C computed to be 0.001509 [CLF] DBG{ 'Mon Mar 24 10:16:12 2008' '5.240 sec' 'VmSize:\t 128032 kB'}: Predicting classifier SVMBase(kernel_type=0, C=0.00150852096501, degree=3, weight_label=[], probability=0, shrinking=1, weight=[], eps=1e-05, svm_type=0, p=0.1, cache_size=100, nr_weight=0, coef0=0.0, nu=0.5, gamma=0.00188679245283, enable_states=['training_time', 'predicting_time', 'predictions', 'trained_labels']) on data (72, 530) [CLF] DBG{ 'Mon Mar 24 10:16:12 2008' '0.581 sec' 'VmSize:\t 127880 kB'}: Training classifier SVMBase(kernel_type=0, C=0.00150852096501, degree=3, weight_label=[], probability=0, shrinking=1, weight=[], eps=1e-05, svm_type=0, p=0.1, cache_size=100, nr_weight=0, coef0=0.0, nu=0.5, gamma=0.00188679245283, enable_states=['training_time', 'predicting_time', 'predictions', 'trained_labels']) on dataset [SVM] DBG{ 'Mon Mar 24 10:16:13 2008' '1.112 sec' 'VmSize:\t 137680 kB'}: Default C computed to be 0.001478 So we selected to output anything about classifiers, and SVM in particular. As well as few useful metrics (so we could see how long it took and is there any memory leak happening). And MVPA_DEBUG takes python regexps, so =.* would enable *all* debugging output if that is necessary. More about debug (as well as verbose and warning) http://pkg-exppsy.alioth.debian.org/pymvpa/manual.html#progress-tracking I hope that this could be interesting/useful for someone ;-) Cheers and keep in touch! Yarik On Mon, 24 Mar 2008, Anton Slesarev wrote: > It really looks pretty good. I think that such functionality should be > in scipy(or scikits). I'll try to take it into account in my proposal. > On Wed, Mar 19, 2008 at 4:43 AM, Jarrod Millman > <[1]millman at berkeley.edu> wrote: > Hey Anton, > Sorry I haven't responded sooner; I am at a coding sprint. Anyway, > I > am meeting with the developers of Multivariate Pattern Analysis in > Python: [2]http://pkg-exppsy.alioth.debian.org/pymvpa/ > I think that this package looks very good. David Cournapeau will be > visiting Berkeley in early April and I plan to discuss whether it > would make sense to merge this into scikits.learn. So it would make > sense for you to take a look at the manual here: > [3]http://pkg-exppsy.alioth.debian.org/pymvpa/manual.html > I am really excited that you are planning to submit a proposal for > the SoC. > Thanks, -- .-. =------------------------------ /v\ ----------------------------= Keep in touch // \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User ^^-^^ [175555] From bsouthey at gmail.com Mon Mar 24 10:49:14 2008 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 24 Mar 2008 09:49:14 -0500 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <11abf2720803240635gf07d044k5519aab00851b445@mail.gmail.com> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <47E1E0B3.9010505@ar.media.kyoto-u.ac.jp> <11abf2720803240309k1b29d968jc2980a7dfd746fa@mail.gmail.com> <47E78554.2090901@ar.media.kyoto-u.ac.jp> <11abf2720803240635gf07d044k5519aab00851b445@mail.gmail.com> Message-ID: <47E7BF6A.8060103@gmail.com> Hi, The main problem that I have with using sparse input formats is that it tends to ignore the complete picture. Typically the algorithms are typically not implemented to utilize sparse matrices and associated techniques so the internals and outputs are not stored as a sparse format. This means that the only gain is the apparent ease of input because any storage advantage is lost if the input needs to be converted to a dense format (especially if both copies are. Using record arrays with both masked and sparse arrays would provide probably address many concerns. Record arrays would allow labels like 'target' without forcing any order in the data storage, masked arrays would allow for missing values in the input and sparse arrays would potentially provide storage and algorithmic advantages. Regards Bruce Anton Slesarev wrote: > > > On Mon, Mar 24, 2008 at 3:55 PM, Nathan Bell > wrote: > > > > On Mon, Mar 24, 2008 at 5:41 AM, David Cournapeau > > > > wrote: > > > > I mean that I have 1 million text pages with 150 thousands > different > > > > words(features), but each document has only small part of > all 150 > > > > thousands world. And if I use simple matrix it will be huge. > But if > > > > i use sparse format such as libsvm data format than input file > will be > > > > much smaller. I don't know how to do it with scikits now. I > know how > > > > to do it with libsvm and many other tools, but I want to make scipy > > > > appropriate for this task. And I want make a tutorial in which one > > > > paragraph will be about "Sparse data". > > > > > > I understand sparse, I don't understand why you cannot use existing > > > scipy implementation :) > > > > Anton, can you describe libsvm's sparse format? I think it's highly > > likely that scipy.sparse supports the functionality you need. > > > > Currently you can load a sparse matrix from disk using MatrixMarket > > format (scipy.io.mmread) or MATLAB format (scipy.io.loadmat). Both > > of these functions should be fast enough for your 150K by 1M example. > > > > FWIW the MATLAB files will generally be smaller and load faster. > > > > -- > > Nathan Bell wnbell at gmail.com > > http://graphics.cs.uiuc.edu/~wnbell/ > > > > > > > > > _______________________________________________ > > Scipy-dev mailing list > > Scipy-dev at scipy.org > > http://projects.scipy.org/mailman/listinfo/scipy-dev > > > > libsvm format: > > > > "libsvm uses the so called "sparse" format where zero values do not > need to be stored. Hence a data with attributes 1 0 2 0 > > is represented as 1:1 3:2" > > > I understand that it is possible to use scipy.sparse and something > else but what about if I need to make feature selection or some > specific normalization? I think that we can integrate this > procedure(with scipy.sparse and reading huge files) to dataset class > in scikits.learn. > > > > > > > -- > Anton Slesarev > ------------------------------------------------------------------------ > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-dev > From matthieu.brucher at gmail.com Mon Mar 24 12:05:53 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 24 Mar 2008 17:05:53 +0100 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <20080324143230.GF26722@washoe.rutgers.edu> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <11abf2720803240241j69b5ec0au1e5dc132949e6be8@mail.gmail.com> <20080324143230.GF26722@washoe.rutgers.edu> Message-ID: Hi ;) 2008/3/24, Yaroslav Halchenko : > > Hi Anton, > > Thank you for the positive feedback. Let me introduce myself briefly: I > am one of the authors of PyMVPA and met Jarrod at NiPy coding sprint at > Paris. There we met with another group (of Jean B. Poline) which > develops analogous (yet closed source but due to our persuasion > hopefully open-source soon) toolbox. Although we had a lot of ideas in > common, some aspects were conceptually different (unfortunately I've > forgotten exact name of their toolbox: mindmine, or smth like that) > > In the case of our PyMVPA we tried to build up a framework where it is > possible > to combine various ML boxes together via parametrization of the classes in > the > constructors. That could serve an equivalent role to those GUI-based > frameworks > where you build your analysis from blocks by connecting them with "lines". > And > the whole PyMVPA computation pipeline is initiated whenever resultant > object > sees some data (like within train() of a classifier). It is also somewhat > similar to approach taken by MDP guys (http://mdp-toolkit.sourceforge.net/ > ). > Also we have similar approach to existing scikits.learn to abstract all > relevant data within a Dataset class. The primary user-base we target > originally with PyMVPA is brainimaging research community, thus we had to > provide not only simple blocks but somewhat obvious and straightforward to > use > software and a reasonable documentation. That could cost us little loss > of generality, although I don't see it happening yet ;-) Indeed, the framework is great and is easy to use. I have an additional question I forgot to ask during the sprint. You didn't show an exemple of a classification training and then a single test. For instance, train a SVM and then test one image (for SVMs, it is now needed to be able to get the underlying cost function, more and more publications use it to show interesting results, and it is what could be thought as a state in your framework ?). This is expecting when testing one individual versus two groups (it is done in anatomic brainimaging). Could you add an exemple with this ? I also know that Windows is not very tested for the moment, I hope the mix with scikits.learn will help promoting your tool and fix the different bugs that will be found (or not if there are no bugs :D) with other platforms. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at onerussian.com Mon Mar 24 13:43:33 2008 From: lists at onerussian.com (Yaroslav Halchenko) Date: Mon, 24 Mar 2008 13:43:33 -0400 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <11abf2720803240241j69b5ec0au1e5dc132949e6be8@mail.gmail.com> <20080324143230.GF26722@washoe.rutgers.edu> Message-ID: <20080324174332.GG26722@washoe.rutgers.edu> > Indeed, the framework is great and is easy to use. that is the hope ;-) it is easy to be used by us but we will see how neurosci people be able to adopt it for their needs ;-) > didn't show an exemple of a classification training and then a > single test. For instance, train a SVM and then test one image (for > SVMs, it if I got it right, you just want to train on 1 dataset (name it training) and transfer (predict) another (name it validation) (which might have just a single sample/image) As always there are multiple ways to accomplish the mission. Explicit (and "low-level") one would be clf = SomeClassifier() clf.train(training) predictions = clf.predict(validation) or if the main goal also to get an error measure out of the box, then just use terr = TransferError(clf, enabled_states=['confusion']) error = terr(validation, training) what good about terr is that you could use it as an error measure for feature selection (e.g. RFE). Also I enabled confusion state variable, so it stores confusion matrix, which also contains all 'validated' samples' target and predicted labels, or in your case it will be just 1 pair of such. You can easily access target and predicted labels through terr.confusion.sets[0] NB [0] is solely since you had just 1 training/validation pair. > is now needed to be able to get the underlying cost function, more and > more publications use it to show interesting results, and it is what > could be thought as a state in your framework ?). ah, so you want to see the actual value which was used to decide on the label. Eg for linear SVM it would be w*x+b value, right? For those we actually have already state variable "values" (as opposed to labels), but we are yet to "connect" it to the real computed value I believe since we haven't used it yet (actually now it is linked to prediction probability as it is implemented within libsvm)... I guess we should add cleaner interface to both cost_function and prediction probability results produced by classifier for any given sample. But if you are interested in w on itself we call it LinearSVMSensitivity ;-) Also please have a look at doc/examples/pylab_2d.py which plots probability maps for few classifiers ;-) > This is expecting > when testing one individual versus two groups (it is done in anatomic > brainimaging). Could you add an exemple with this ? shouldn't you simply have a binary classifier to classify between two groups? > I also know that Windows is not very tested for the moment, I hope the > mix with scikits.learn will help promoting your tool and fix the > different bugs that will be found (or not if there are no bugs :D) with > other platforms. Yeah -- eventually we should try it on some proprietary OS like Windows but we aren't brave enough :-) > Matthieu -- .-. =------------------------------ /v\ ----------------------------= Keep in touch // \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User ^^-^^ [175555] From cimrman3 at ntc.zcu.cz Wed Mar 26 11:58:08 2008 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 26 Mar 2008 16:58:08 +0100 Subject: [SciPy-dev] ANN: SfePy 00.41.03 Message-ID: <47EA7290.7030009@ntc.zcu.cz> Greetings, I'm pleased to announce the release 00.41.03 of SfePy (formerly SFE) SfePy is a finite element analysis software in Python, based primarily on Numpy and SciPy. Mailing lists, issue tracking, mercurial repository: http://code.google.com/p/sfepy/ Home page: http://sfepy.kme.zcu.cz Major improvements: - works on 64 bits - support for various mesh formats - Schroedinger equation solver - see http://code.google.com/p/sfepy/wiki/Examples - new solvers: - generic time-dependent problem solver - pysparse, symeig, scipy-based eigenproblem solvers - scipy-based iterative solvers - many new terms For information on this release, see http://sfepy.googlecode.com/svn/web/releases/004103_RELEASE_NOTES.txt Best regards, r. From rhys.ulerich at gmail.com Wed Mar 26 13:08:30 2008 From: rhys.ulerich at gmail.com (Rhys Ulerich) Date: Wed, 26 Mar 2008 12:08:30 -0500 Subject: [SciPy-dev] Import problem in scipy.interpolate.lagrange Message-ID: <4a00655d0803261008m7fd0ff9fhdc7b956adf5018a6@mail.gmail.com> Hi all, scipy/interpolate/interpolate.py is missing an import for numpy.poly1d. Python 2.5 (r25:51908, Nov 6 2007, 15:55:44) [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy >>> import scipy.interpolate >>> scipy.interpolate.lagrange([0.0,1.0],[1.0,0.0]) Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python2.5/site-packages/scipy/interpolate/interpolate.py", line 30, in lagrange p = poly1d(0.0) NameError: global name 'poly1d' is not defined The problem still appears in svn trunk: http://svn.scipy.org/svn/scipy/trunk/scipy/interpolate/interpolate.py It can (I think) be fixed by adding 'poly1d' into the 'from numpy import...' list. It looks like the problem has existed since changeset 3068. How can I get a SciPy Trac login so I can file this ticket? - Rhys -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Wed Mar 26 15:45:38 2008 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Wed, 26 Mar 2008 20:45:38 +0100 Subject: [SciPy-dev] Import problem in scipy.interpolate.lagrange In-Reply-To: <4a00655d0803261008m7fd0ff9fhdc7b956adf5018a6@mail.gmail.com> References: <4a00655d0803261008m7fd0ff9fhdc7b956adf5018a6@mail.gmail.com> Message-ID: On Wed, 26 Mar 2008 12:08:30 -0500 "Rhys Ulerich" wrote: > Hi all, > > scipy/interpolate/interpolate.py is missing an import >for numpy.poly1d. > > Python 2.5 (r25:51908, Nov 6 2007, 15:55:44) > [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2 > Type "help", "copyright", "credits" or "license" for >more information. >>>> import scipy >>>> import scipy.interpolate >>>> scipy.interpolate.lagrange([0.0,1.0],[1.0,0.0]) > Traceback (most recent call last): > File "", line 1, in > File > "/usr/lib64/python2.5/site-packages/scipy/interpolate/interpolate.py", >line > 30, in lagrange > p = poly1d(0.0) > NameError: global name 'poly1d' is not defined > > The problem still appears in svn trunk: > http://svn.scipy.org/svn/scipy/trunk/scipy/interpolate/interpolate.py > It can (I think) be fixed by adding 'poly1d' into the >'from numpy import...' > list. It looks like the problem > has existed since changeset 3068. > > How can I get a SciPy Trac login so I can file this >ticket? > > - Rhys See http://www.scipy.org/BugReport for details. BTW, I can confirm the bug >>> import scipy >>> import scipy.interpolate >>> scipy.interpolate.lagrange([0.0,1.0],[1.0,0.0]) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/site-packages/scipy/interpolate/interpolate.py", line 30, in lagrange p = poly1d(0.0) NameError: global name 'poly1d' is not defined >>> scipy.__version__ '0.7.0.dev4050' HTH, Nils From matthieu.brucher at gmail.com Fri Mar 28 10:04:55 2008 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Fri, 28 Mar 2008 15:04:55 +0100 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <20080324174332.GG26722@washoe.rutgers.edu> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> <11abf2720803240241j69b5ec0au1e5dc132949e6be8@mail.gmail.com> <20080324143230.GF26722@washoe.rutgers.edu> <20080324174332.GG26722@washoe.rutgers.edu> Message-ID: > > if I got it right, you just want to train on 1 dataset (name it > training) and transfer (predict) another (name it validation) (which > might have just a single sample/image) Exactly. (...) thanks for the example, I'll try that :) > is now needed to be able to get the underlying cost function, more and > > more publications use it to show interesting results, and it is what > > could be thought as a state in your framework ?). > > ah, so you want to see the actual value which was used to decide on the > label. Eg for linear SVM it would be w*x+b value, right? > For those we actually have already state variable "values" (as opposed > to labels), but we are yet to "connect" it to the real computed value I > believe since we haven't used it yet (actually now it is linked to > prediction probability as it is implemented within libsvm)... That's what I want, yes. What is the value it contains if not the wx+b one ? Also please have a look at doc/examples/pylab_2d.py which plots > probability maps for few classifiers ;-) Thanks for the tip, this may be of interest as well :) > This is expecting > > when testing one individual versus two groups (it is done in anatomic > > brainimaging). Could you add an exemple with this ? > > shouldn't you simply have a binary classifier to classify between two > groups? Yes, that's what you usually have, but in some case you want to have the difference between the sample and the other class. > I also know that Windows is not very tested for the moment, I hope the > > mix with scikits.learn will help promoting your tool and fix the > > different bugs that will be found (or not if there are no bugs :D) > with > > other platforms. > > Yeah -- eventually we should try it on some proprietary OS like Windows > but we aren't brave enough :-) :) Thanks for the intel Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From slesarev.anton at gmail.com Mon Mar 31 06:13:17 2008 From: slesarev.anton at gmail.com (Anton Slesarev) Date: Mon, 31 Mar 2008 14:13:17 +0400 Subject: [SciPy-dev] Google Summer of Code and scipy.learn (another trying) In-Reply-To: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> References: <11abf2720803160541i7ce7d5ebq7c3e1b762d11ff20@mail.gmail.com> Message-ID: <11abf2720803310313w63f5922dl2dfe972ea2ec863b@mail.gmail.com> Thank you all for comments. I tried to take into account all your wishes. I wrote the proposal: http://slesarev.anton.googlepages.com/proposal and post it to google summer of code. I am going to make two main things: -Make scikits.learn convenient to integrate text classificators. -Use first and integrate BBR. That's all. Main problem how to connect it with your idea to use PyMVPA. I don't think that it is a good idea to duplicate its functionality. But I hope we can find approach to cooperate without writing useless code. I hope you find mentor who want to take this application. -- Anton Slesarev -------------- next part -------------- An HTML attachment was scrubbed... URL: