From midel at sfr.fr Fri Apr 1 08:00:54 2011 From: midel at sfr.fr (midel) Date: Fri, 1 Apr 2011 14:00:54 +0200 (CEST) Subject: [SciPy-User] (pas de sujet) Message-ID: <3981665.414641301659254202.JavaMail.www@wsfrf1126> Hi everybody, I began using scipy a few days ago and I think a long story between us is beginning ;) I come from the Matlab/Scilab world and I have no big problem using SciPy. Well, in fact, I have one problem, that is why I come here ! I will try to explain. I have a main program in one file "verif.py" and I want it to import functions that i have defined in another file "fonctions.py". I import Scipy in the main program but I also need the scipy functions to be available in the functions and do not manage to do so. The main verif.py program is : ????? from scipy import * ????? from fonction import truc ? ? ????? a=2; ????? b=truc(a); ????? print b fonction.py is : ????? from scipy import * ????? def truc(machin): ????? ????? plouc=machin*2; ????? ????? A=array([[1,2],[3,4]]); ????? ????? return A Under iPython, when I do : ????? run verif.py I get an error pointing to truc(machin): ????? NameError: global name 'array' is not defined ????? WARNING: failure executing file: I have no error running ????? run fonction.py Which seems normal as I explicitely import scipy. Of course I have no error if I put the function in the same file as the main code : ????? from scipy import * ????? ????? def truc(machin): ????? ????? plouc=machin*2; ????? ????? A=array([[1,2],[3,4]]); ????? ????? return A ????? a=2; ????? b=truc(a); ????? print b runs errorless. The most mysterious thing is that sometimes I have no error even using two separate files, but I don't see why... Well, I am convinced that I miss something important and basic... I am lost. midel From eckjoh2 at web.de Fri Apr 1 08:11:51 2011 From: eckjoh2 at web.de (Johannes Eckstein) Date: Fri, 01 Apr 2011 14:11:51 +0200 Subject: [SciPy-User] (pas de sujet) In-Reply-To: <3981665.414641301659254202.JavaMail.www@wsfrf1126> References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> Message-ID: <4D95C107.4060104@web.de> Hi there, i also began using scipy some days ago and I think you simply forgot to do something like: from numpy import array in your fonction.py before you actually use it in your line: A=array([[1,2],[3,4]]); I send you attached an example of a collection of two functions I wrote yesterday ;-) Cheers! Johannes > Hi everybody, > > I began using scipy a few days ago and I think a long story between us is beginning ;) I come from the Matlab/Scilab world and I have no big problem using SciPy. > > Well, in fact, I have one problem, that is why I come here ! I will try to explain. > > I have a main program in one file "verif.py" and I want it to import functions that i have defined in another file "fonctions.py". I import Scipy in the main program but I also need the scipy functions to be available in the functions and do not manage to do so. > > The main verif.py program is : > > from scipy import * > from fonction import truc > > a=2; > b=truc(a); > print b > > > > fonction.py is : > > from scipy import * > > def truc(machin): > plouc=machin*2; > A=array([[1,2],[3,4]]); > return A > > > Under iPython, when I do : > > run verif.py > > I get an error pointing to truc(machin): > > NameError: global name 'array' is not defined > WARNING: failure executing file: > > I have no error running > > run fonction.py > > Which seems normal as I explicitely import scipy. > > > > Of course I have no error if I put the function in the same file as the main code : > > from scipy import * > > def truc(machin): > plouc=machin*2; > A=array([[1,2],[3,4]]); > return A > > a=2; > b=truc(a); > print b > > runs errorless. > > The most mysterious thing is that sometimes I have no error even using two separate files, but I don't see why... > > Well, I am convinced that I miss something important and basic... I am lost. > > midel > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Dipl.-Ing. Johannes Eckstein Automatisierungstechnik i. d. Produktion Universit?t Stuttgart Apfelbl?tenweg 9 70569 Stuttgart Mob: +491711939513 Tel: +4971150427990 Mail: eckjoh2 at web.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: importascii.py URL: From cweisiger at msg.ucsf.edu Fri Apr 1 10:58:36 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Fri, 1 Apr 2011 07:58:36 -0700 Subject: [SciPy-User] (pas de sujet) In-Reply-To: <3981665.414641301659254202.JavaMail.www@wsfrf1126> References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> Message-ID: As Johannes noted, the "array" function is in numpy, not scipy. Also, you are able to run fonction.py directly because the code inside truc() never runs, so Python never sees the invalid function. You could do something like this if you wanted to: def foo(): a = variableThatNeverExists and Python would not complain until you called foo() and it tried to figure out where variableThatNeverExists came from. If you want to test that a module works correctly, you can add this: if __name__ == '__main__': truc(2) That will cause the module to run truc() by itself, but only if it is the "main" module being run by Python (i.e. you did "python fonctions.py"). If it is imported from another module, then the code inside that if statement is not run. -Chris On Fri, Apr 1, 2011 at 5:00 AM, midel wrote: > Hi everybody, > > I began using scipy a few days ago and I think a long story between us is > beginning ;) I come from the Matlab/Scilab world and I have no big problem > using SciPy. > > Well, in fact, I have one problem, that is why I come here ! I will try to > explain. > > I have a main program in one file "verif.py" and I want it to import > functions that i have defined in another file "fonctions.py". I import Scipy > in the main program but I also need the scipy functions to be available in > the functions and do not manage to do so. > > The main verif.py program is : > > from scipy import * > from fonction import truc > > a=2; > b=truc(a); > print b > > > > fonction.py is : > > from scipy import * > > def truc(machin): > plouc=machin*2; > A=array([[1,2],[3,4]]); > return A > > > Under iPython, when I do : > > run verif.py > > I get an error pointing to truc(machin): > > NameError: global name 'array' is not defined > WARNING: failure executing file: > > I have no error running > > run fonction.py > > Which seems normal as I explicitely import scipy. > > > > Of course I have no error if I put the function in the same file as the > main code : > > from scipy import * > > def truc(machin): > plouc=machin*2; > A=array([[1,2],[3,4]]); > return A > > a=2; > b=truc(a); > print b > > runs errorless. > > The most mysterious thing is that sometimes I have no error even using two > separate files, but I don't see why... > > Well, I am convinced that I miss something important and basic... I am > lost. > > midel > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From evanmason at gmail.com Fri Apr 1 12:06:33 2011 From: evanmason at gmail.com (Evan Mason) Date: Fri, 1 Apr 2011 16:06:33 +0000 (UTC) Subject: [SciPy-User] interpolation with CloughTocher2DInterpolator Message-ID: Hi, first I'd like to say thank you to the developers for the new interpolation options in scipy 0.9.0 (griddata, CloughTocher2DInterpolator). I have a question however about functionality. Using matplotlib's delaunay I can do: tri = delaunay.Triangulation(x, y) zi = tri.nn_interpolator(z) Once I have computed 'tri', I can then reuse it for successive interpolations of a changing z. This saves a lot of time as I have sequences of z in the hundreds. But this approach doesn't seem to be possible with CloughTocher2DInterpolator, as every call needs x, y, z: CloughTocher2DInterpolator(points, values, tol=1e-6) My question is whether this functionality could be built into future releases of CloughTocher2DInterpolator? Thanks a lot, Evan From Chris.Barker at noaa.gov Fri Apr 1 13:06:20 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 01 Apr 2011 10:06:20 -0700 Subject: [SciPy-User] (pas de sujet) In-Reply-To: <3981665.414641301659254202.JavaMail.www@wsfrf1126> References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> Message-ID: <4D96060C.7020409@noaa.gov> On 4/1/11 5:00 AM, midel wrote: > I began using scipy a few days ago and I think a long story between us is beginning ;) I come from the Matlab/Scilab world and I have no big problem using SciPy. > > Well, in fact, I have one problem, that is why I come here ! I will try to explain. > > I have a main program in one file "verif.py" and I want it to import functions that i have defined in another file "fonctions.py". I import Scipy in the main program but I also need the scipy functions to be available in the functions and do not manage to do so. > > The main verif.py program is : > > from scipy import * > from fonction import truc I think your immediate question was answered, but a suggestion: From "The Zen of Python": "Namespaces are one honking great idea -- let's do more of those!" Matlab puts eveyting in one namespace -- this is convenient for interactive use at the command line, but gets really painful after a while for larger project. I HIGHLY recommend that you get out of that habit, and use python namespaces. What that means is that you don't use "import *" EVER. Some module names are a bit long to type if you are calling a lot of functions, so you can give them shorter names when you import. For instance, most folks now use: import numpy as np and often import scipy as sp So your code would look like: The main verif.py program is : import fonction a=2; b=fonction.truc(a); print b fonction.py is : import numpy as np def truc(machin): plouc=machin*2; A=np.array([[1,2],[3,4]]); return A Note that "from fonction import truc" is often fine, too -- you are being explicit about exactly where "truc" came from. Another one from "the Zen of Python" "Explicit is better than implicit" This may seem difficult and like too much typing at first, but believe me, it really is a better way to write code. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From seb.haase at gmail.com Fri Apr 1 13:18:52 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Fri, 1 Apr 2011 19:18:52 +0200 Subject: [SciPy-User] (pas de sujet) In-Reply-To: <4D96060C.7020409@noaa.gov> References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> Message-ID: On Fri, Apr 1, 2011 at 7:06 PM, Christopher Barker wrote: > On 4/1/11 5:00 AM, midel wrote: >> I began using scipy a few days ago and I think a long story between us is beginning ;) I come from the Matlab/Scilab world and I have no big problem using SciPy. >> >> Well, in fact, I have one problem, that is why I come here ! I will try to explain. >> >> I have a main program in one file "verif.py" and I want it to import functions that i have defined in another file "fonctions.py". I import Scipy in the main program but I also need the scipy functions to be available in the functions and do not manage to do so. >> >> The main verif.py program is : >> >> ? ? ? ?from scipy import * >> ? ? ? ?from fonction import truc > > I think your immediate question was answered, but a suggestion: > > ?From "The Zen of Python": > > "Namespaces are one honking great idea -- let's do more of those!" > > Matlab puts eveyting in one namespace -- this is convenient for > interactive use at the command line, but gets really painful after a > while for larger project. I HIGHLY recommend that you get out of that > habit, and use python namespaces. What that means is that you don't use > "import *" EVER. > > Some module names are a bit long to type if you are calling a lot of > functions, so you can give them shorter names when you import. For > instance, most folks now use: > > import numpy as np > > and often > > import scipy as sp > > > So your code would look like: > > The main verif.py program is : > > ? ? ? import fonction > > ? ? ? a=2; > ? ? ? b=fonction.truc(a); > ? ? ? print b > > > > fonction.py is : > > ? ? ? import numpy as np > > ? ? ? def truc(machin): > ? ? ? ? ? ? plouc=machin*2; > ? ? ? ? ? ? A=np.array([[1,2],[3,4]]); > ? ? ? ? ? ? return A > > > Note that "from fonction import truc" is often fine, too -- you are > being explicit about exactly where "truc" came from. > > Another one from "the Zen of Python" > > "Explicit is better than implicit" > > This may seem difficult and like too much typing at first, but believe > me, it really is a better way to write code. > > -Chris try >>> import this ;-) - Sebastian Haase From pav at iki.fi Fri Apr 1 13:31:29 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 1 Apr 2011 17:31:29 +0000 (UTC) Subject: [SciPy-User] interpolation with CloughTocher2DInterpolator References: Message-ID: Fri, 01 Apr 2011 16:06:33 +0000, Evan Mason wrote: [clip] > But this approach doesn't seem to be possible with > CloughTocher2DInterpolator, as every call needs x, y, z: > > CloughTocher2DInterpolator(points, values, tol=1e-6) You can in the meantime do this: import numpy as np import scipy.interpolate.interpnd as interpnd class MyInterpolator(interpnd.CloughTocher2DInterpolator): def __init__(self, tri, values, fill_value=np.nan, tol=1e-6, maxiter=400): interpnd.NDInterpolatorBase.__init__(self, tri.points, values, ndim=2, fill_value=fill_value) self.tri = tri self.grad = interpnd.estimate_gradients_2d_global(self.tri, self.values, tol=tol, maxiter=maxiter) tri = Delaunay(points) ip = MyInterpolator(tri, values) > My question is whether this functionality could be built into future > releases of CloughTocher2DInterpolator? Certainly. If you want to make sure about this, file an enhancement request at http://projects.scipy.org/scipy/ Pauli From renato.fabbri at gmail.com Fri Apr 1 14:52:41 2011 From: renato.fabbri at gmail.com (Renato Fabbri) Date: Fri, 1 Apr 2011 15:52:41 -0300 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> Message-ID: for a long time i imported all libraries with a capital letter, so a typical first line would be: import numpy as N, pylab as P, scikits.audiolab as A, pywt as W, scipy as S now i find that ugly and unconfortable (as subjective as that may sound..) and i do: import numpy as n, pylab as p, scikits.audiolab as a, pywt as w, scipy as s which leads me to nice looking code and confortable writing: avar=n.arange(100) p.plot(avar,avar**2) p.show() i get 10x more girls now. 2011/4/1 Sebastian Haase : > On Fri, Apr 1, 2011 at 7:06 PM, Christopher Barker > wrote: >> On 4/1/11 5:00 AM, midel wrote: >>> I began using scipy a few days ago and I think a long story between us is beginning ;) I come from the Matlab/Scilab world and I have no big problem using SciPy. >>> >>> Well, in fact, I have one problem, that is why I come here ! I will try to explain. >>> >>> I have a main program in one file "verif.py" and I want it to import functions that i have defined in another file "fonctions.py". I import Scipy in the main program but I also need the scipy functions to be available in the functions and do not manage to do so. >>> >>> The main verif.py program is : >>> >>> ? ? ? ?from scipy import * >>> ? ? ? ?from fonction import truc >> >> I think your immediate question was answered, but a suggestion: >> >> ?From "The Zen of Python": >> >> "Namespaces are one honking great idea -- let's do more of those!" >> >> Matlab puts eveyting in one namespace -- this is convenient for >> interactive use at the command line, but gets really painful after a >> while for larger project. I HIGHLY recommend that you get out of that >> habit, and use python namespaces. What that means is that you don't use >> "import *" EVER. >> >> Some module names are a bit long to type if you are calling a lot of >> functions, so you can give them shorter names when you import. For >> instance, most folks now use: >> >> import numpy as np >> >> and often >> >> import scipy as sp >> >> >> So your code would look like: >> >> The main verif.py program is : >> >> ? ? ? import fonction >> >> ? ? ? a=2; >> ? ? ? b=fonction.truc(a); >> ? ? ? print b >> >> >> >> fonction.py is : >> >> ? ? ? import numpy as np >> >> ? ? ? def truc(machin): >> ? ? ? ? ? ? plouc=machin*2; >> ? ? ? ? ? ? A=np.array([[1,2],[3,4]]); >> ? ? ? ? ? ? return A >> >> >> Note that "from fonction import truc" is often fine, too -- you are >> being explicit about exactly where "truc" came from. >> >> Another one from "the Zen of Python" >> >> "Explicit is better than implicit" >> >> This may seem difficult and like too much typing at first, but believe >> me, it really is a better way to write code. >> >> -Chris > > try > >>>> import this > > ;-) > > - Sebastian Haase > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- GNU/Linux User #479299 skype: fabbri.renato From Agile.Aspect at GMail.COM Fri Apr 1 15:13:21 2011 From: Agile.Aspect at GMail.COM (Agile Aspect) Date: Fri, 01 Apr 2011 12:13:21 -0700 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> Message-ID: <4D9623D1.7040006@GMail.COM> Renato Fabbri wrote: > for a long time i imported all libraries with a capital letter, so a > typical first line would be: > > import numpy as N, pylab as P, scikits.audiolab as A, pywt as W, scipy as S > > now i find that ugly and unconfortable (as subjective as that may > sound..) and i do: > > import numpy as n, pylab as p, scikits.audiolab as a, pywt as w, scipy as s > > which leads me to nice looking code and confortable writing: > > avar=n.arange(100) > p.plot(avar,avar**2) > p.show() > > i get 10x more girls now. Works for me too! > > > 2011/4/1 Sebastian Haase : >> On Fri, Apr 1, 2011 at 7:06 PM, Christopher Barker >> wrote: >>> On 4/1/11 5:00 AM, midel wrote: >>>> I began using scipy a few days ago and I think a long story between us is beginning ;) I come from the Matlab/Scilab world and I have no big problem using SciPy. >>>> >>>> Well, in fact, I have one problem, that is why I come here ! I will try to explain. >>>> >>>> I have a main program in one file "verif.py" and I want it to import functions that i have defined in another file "fonctions.py". I import Scipy in the main program but I also need the scipy functions to be available in the functions and do not manage to do so. >>>> >>>> The main verif.py program is : >>>> >>>> from scipy import * >>>> from fonction import truc >>> I think your immediate question was answered, but a suggestion: >>> >>> From "The Zen of Python": >>> >>> "Namespaces are one honking great idea -- let's do more of those!" >>> >>> Matlab puts eveyting in one namespace -- this is convenient for >>> interactive use at the command line, but gets really painful after a >>> while for larger project. I HIGHLY recommend that you get out of that >>> habit, and use python namespaces. What that means is that you don't use >>> "import *" EVER. >>> >>> Some module names are a bit long to type if you are calling a lot of >>> functions, so you can give them shorter names when you import. For >>> instance, most folks now use: >>> >>> import numpy as np >>> >>> and often >>> >>> import scipy as sp >>> >>> >>> So your code would look like: >>> >>> The main verif.py program is : >>> >>> import fonction >>> >>> a=2; >>> b=fonction.truc(a); >>> print b >>> >>> >>> >>> fonction.py is : >>> >>> import numpy as np >>> >>> def truc(machin): >>> plouc=machin*2; >>> A=np.array([[1,2],[3,4]]); >>> return A >>> >>> >>> Note that "from fonction import truc" is often fine, too -- you are >>> being explicit about exactly where "truc" came from. >>> >>> Another one from "the Zen of Python" >>> >>> "Explicit is better than implicit" >>> >>> This may seem difficult and like too much typing at first, but believe >>> me, it really is a better way to write code. >>> >>> -Chris >> try >> >>>>> import this >> ;-) >> >> - Sebastian Haase >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > From cweisiger at msg.ucsf.edu Fri Apr 1 15:22:48 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Fri, 1 Apr 2011 12:22:48 -0700 Subject: [SciPy-User] (pas de sujet) In-Reply-To: <4D9623D1.7040006@GMail.COM> References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> Message-ID: Keep in mind that if you ever anticipate showing your code to someone else, it pays off big-time to be verbose. Terse code is harder to search, harder to understand at a glance, harder to trace, and generally harder to understand. Also keep in mind that the you one month down the line counts as "someone else". :) On the assumption that any code you write will be used for more than a month, then, you should err on the side of verbosity. Invariably I find that the cost to me of typing out something like "scipy.ndimage.map_coordinates" instead of just e.g. "nd.map_coordinates" more than pays off by the time saved down the road when I have to come back and modify the code. Likewise the cost of naming a variable "costOfPath" (or even just "cost") instead of "c". Typing is not the roadblock to efficient coding; thinking is. The less time you have to waste figuring out what code you've already written does, the better. -Chris On Fri, Apr 1, 2011 at 12:13 PM, Agile Aspect wrote: > Renato Fabbri wrote: > > for a long time i imported all libraries with a capital letter, so a > > typical first line would be: > > > > import numpy as N, pylab as P, scikits.audiolab as A, pywt as W, scipy as > S > > > > now i find that ugly and unconfortable (as subjective as that may > > sound..) and i do: > > > > import numpy as n, pylab as p, scikits.audiolab as a, pywt as w, scipy as > s > > > > which leads me to nice looking code and confortable writing: > > > > avar=n.arange(100) > > p.plot(avar,avar**2) > > p.show() > > > > i get 10x more girls now. > > Works for me too! > > > > > > 2011/4/1 Sebastian Haase : > >> On Fri, Apr 1, 2011 at 7:06 PM, Christopher Barker > >> wrote: > >>> On 4/1/11 5:00 AM, midel wrote: > >>>> I began using scipy a few days ago and I think a long story between us > is beginning ;) I come from the Matlab/Scilab world and I have no big > problem using SciPy. > >>>> > >>>> Well, in fact, I have one problem, that is why I come here ! I will > try to explain. > >>>> > >>>> I have a main program in one file "verif.py" and I want it to import > functions that i have defined in another file "fonctions.py". I import Scipy > in the main program but I also need the scipy functions to be available in > the functions and do not manage to do so. > >>>> > >>>> The main verif.py program is : > >>>> > >>>> from scipy import * > >>>> from fonction import truc > >>> I think your immediate question was answered, but a suggestion: > >>> > >>> From "The Zen of Python": > >>> > >>> "Namespaces are one honking great idea -- let's do more of those!" > >>> > >>> Matlab puts eveyting in one namespace -- this is convenient for > >>> interactive use at the command line, but gets really painful after a > >>> while for larger project. I HIGHLY recommend that you get out of that > >>> habit, and use python namespaces. What that means is that you don't use > >>> "import *" EVER. > >>> > >>> Some module names are a bit long to type if you are calling a lot of > >>> functions, so you can give them shorter names when you import. For > >>> instance, most folks now use: > >>> > >>> import numpy as np > >>> > >>> and often > >>> > >>> import scipy as sp > >>> > >>> > >>> So your code would look like: > >>> > >>> The main verif.py program is : > >>> > >>> import fonction > >>> > >>> a=2; > >>> b=fonction.truc(a); > >>> print b > >>> > >>> > >>> > >>> fonction.py is : > >>> > >>> import numpy as np > >>> > >>> def truc(machin): > >>> plouc=machin*2; > >>> A=np.array([[1,2],[3,4]]); > >>> return A > >>> > >>> > >>> Note that "from fonction import truc" is often fine, too -- you are > >>> being explicit about exactly where "truc" came from. > >>> > >>> Another one from "the Zen of Python" > >>> > >>> "Explicit is better than implicit" > >>> > >>> This may seem difficult and like too much typing at first, but believe > >>> me, it really is a better way to write code. > >>> > >>> -Chris > >> try > >> > >>>>> import this > >> ;-) > >> > >> - Sebastian Haase > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > > > > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Apr 1 15:33:56 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 01 Apr 2011 12:33:56 -0700 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> Message-ID: <4D9628A4.8050906@noaa.gov> On 4/1/11 12:22 PM, Chris Weisiger wrote: > Keep in mind that if you ever anticipate showing your code to someone > else, it pays off big-time to be verbose. Terse code is harder to > search, harder to understand at a glance, harder to trace, and generally > harder to understand. yup -- so standards are a good idea now. I _think_ : import numpy as np has been more-or-less accepted as the standard. I'm not sure if: import scipy as sp is as common, but it seems reasonable to me. On 4/1/11 11:52 AM, Renato Fabbri wrote: > for a long time i imported all libraries with a capital letter, so a > typical first line would be: > > import numpy as N, pylab as P, scikits.audiolab as A, pywt as W, scipy as S me too, and I liked it fine, but a couple years ago, a consensus of sorts was arrived at on the numpy list that "import numpy as np" was the best way to go, so that's what I do in all my code now. I think having this more or less standardized is a really good idea. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Fri Apr 1 15:59:00 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 1 Apr 2011 14:59:00 -0500 Subject: [SciPy-User] (pas de sujet) In-Reply-To: <4D9628A4.8050906@noaa.gov> References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: On Fri, Apr 1, 2011 at 14:33, Christopher Barker wrote: > I'm not sure if: > > import scipy as sp > > is as common, but it seems reasonable to me. It's utterly useless since the scipy subpackages aren't imported along with it. Don't do it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Fri Apr 1 16:02:55 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 1 Apr 2011 22:02:55 +0200 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: On Fri, Apr 1, 2011 at 9:59 PM, Robert Kern wrote: > On Fri, Apr 1, 2011 at 14:33, Christopher Barker wrote: > >> I'm not sure if: >> >> import scipy as sp >> >> is as common, but it seems reasonable to me. > > It's utterly useless since the scipy subpackages aren't imported along > with it. Don't do it. Our docs disagree: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt It still saves a few characters writing "sp.linalg.xxx" instead of "scipy.linalg.xxx". Ralf From renato.fabbri at gmail.com Fri Apr 1 16:03:17 2011 From: renato.fabbri at gmail.com (Renato Fabbri) Date: Fri, 1 Apr 2011 17:03:17 -0300 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: 2011/4/1 Robert Kern : > On Fri, Apr 1, 2011 at 14:33, Christopher Barker wrote: > >> I'm not sure if: >> >> import scipy as sp >> >> is as common, but it seems reasonable to me. > > It's utterly useless since the scipy subpackages aren't imported along > with it. Don't do it. yeah. why does that happen? i mean, when I do import scipy ("as s" if preferred) I can't reach a bunch of subpackages. I need to do: from scipy import * or from scipy import subpackage to reach the subpackage. Why is that? best, rf > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- GNU/Linux User #479299 skype: fabbri.renato From cweisiger at msg.ucsf.edu Fri Apr 1 16:25:50 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Fri, 1 Apr 2011 13:25:50 -0700 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: On Fri, Apr 1, 2011 at 1:02 PM, Ralf Gommers wrote: > On Fri, Apr 1, 2011 at 9:59 PM, Robert Kern wrote: > > On Fri, Apr 1, 2011 at 14:33, Christopher Barker > wrote: > > > >> I'm not sure if: > >> > >> import scipy as sp > >> > >> is as common, but it seems reasonable to me. > > > > It's utterly useless since the scipy subpackages aren't imported along > > with it. Don't do it. > > Our docs disagree: > https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt > > It still saves a few characters writing "sp.linalg.xxx" instead of > "scipy.linalg.xxx". > How valuable is saving a couple of characters, though? The quality of code is not in how terse it is but in how effectively it communicates what it wants to do, both to computers and to humans. Otherwise we'd always say "y" instead of "why". To respond cross-thread, as for why importing scipy doesn't grab all of the sub-modules, but importing numpy does, check out these examples. % denotes a Linux/OSX/Unix/etc. command line prompt, >>> denotes a Python REPL prompt. % cat > foo.py import bar ^D % cat > bar.py a = 10 ^D % python >>> import foo >>> foo.bar.a 10 In other words, because foo imports bar, bar is in foo's namespace; thus it is valid to say "foo.bar" and to access contents of bar through foo. Compare to this: % mkdir foo % touch foo/__init__.py % cat > foo/bar.py a = 10 ^D % python >>> import foo >>> foo.bar Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'bar' In this case, "foo" is a package that contains many modules, but importing the package does not actually import any of the modules in that package. That __init__.py file created using the "touch" command is what allows you to import foo in the first place, despite it not actually being a Python module. The package is merely a container for other modules; if you want to access those modules you have to specifically import them. You could modify the contents of the __init__.py module to make it insert things into the package namespace when you import the package, but for whatever reason that's not done when you import scipy. My guess is that scipy has so many diverse modules that any given program only uses a small proportion of its capabilities. Importing the entire thing could seriously slow down your program initialization; there's no need to pay that extra cost. Meanwhile, numpy has a comparatively narrow scope, so it's not a huge deal to just grab everything numpy can do when you do "import numpy". -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Fri Apr 1 16:44:38 2011 From: e.antero.tammi at gmail.com (eat) Date: Fri, 1 Apr 2011 23:44:38 +0300 Subject: [SciPy-User] (pas de sujet) In-Reply-To: <4D9628A4.8050906@noaa.gov> References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: Hi, On Fri, Apr 1, 2011 at 10:33 PM, Christopher Barker wrote: > On 4/1/11 12:22 PM, Chris Weisiger wrote: > > Keep in mind that if you ever anticipate showing your code to someone > > else, it pays off big-time to be verbose. Terse code is harder to > > search, harder to understand at a glance, harder to trace, and generally > > harder to understand. > > yup -- so standards are a good idea now. > > I _think_ : > > import numpy as np > > has been more-or-less accepted as the standard. > > I'm not sure if: > > import scipy as sp > > is as common, but it seems reasonable to me. > > On 4/1/11 11:52 AM, Renato Fabbri wrote: > > for a long time i imported all libraries with a capital letter, so a > > typical first line would be: > > > > import numpy as N, pylab as P, scikits.audiolab as A, pywt as W, scipy as > S > > me too, and I liked it fine, but a couple years ago, a consensus of > sorts was arrived at on the numpy list that "import numpy as np" was the > best way to go, so that's what I do in all my code now. > > I think having this more or less standardized is a really good idea I just like to ask yours and others matured ones opinion about systematically using import pattern like: from numpy import x, y, z, longname as ln from scipy import u, v, x as sx from scipy.abc import e, f, g With this scheme I think that: - programmer takes (still) the whole responsibility of avoiding name conflicts - reader (user) avoids the possible confusion between modules and classes - reader (user) still needs to be aware where the names originated - it's kind of against of the philosophy of import this; explicit is better than implicit (but that depends actually on how familiar you are with the imported names) - it still feels somehow right to express that these (and only these) are my external dependencies - it seems kind of to unify the way how you import external dependencies Regards, eat . > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hg+scipy at schaathun.net Fri Apr 1 16:55:07 2011 From: hg+scipy at schaathun.net (Hans Georg Schaathun) Date: Fri, 1 Apr 2011 22:55:07 +0200 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: <20110401205507.GA1105@macwilliams.local> On Fri, Apr 01, 2011 at 01:25:50PM -0700, Chris Weisiger wrote: > How valuable is saving a couple of characters, though? The quality of code > is not in how terse it is but in how effectively it communicates what it > wants to do, both to computers and to humans. Not much for a few characters, but your idea of typing out third-level subpackages with long names in full might be just a little over the top. Long names also means that lines have to wrap where they otherwise would not, and the reader may need to look thrice to see if two names are actually the same, or differing in the second part of the name. Verbosity does not always increase legibility. Sometimes it is just information overload. There is a time for everything. If I use the long name only once, writing it out in full is fine. If I need it a handful of times during a module, I would import it under a shorter, but yet intutive name. YMMV I have also found that using a * import interactively is somewhat counterintuitive. Very quickly I start forgetting which package each function comes from. Interactively, importing numpy/scipy as np/sp is a serious saving, and then, using the same conventions interactively and non-interactively has merits, which is probably why these abbreviations have become so whidespread. :-- George From robert.kern at gmail.com Fri Apr 1 16:56:08 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 1 Apr 2011 15:56:08 -0500 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: On Fri, Apr 1, 2011 at 15:02, Ralf Gommers wrote: > On Fri, Apr 1, 2011 at 9:59 PM, Robert Kern wrote: >> On Fri, Apr 1, 2011 at 14:33, Christopher Barker wrote: >> >>> I'm not sure if: >>> >>> import scipy as sp >>> >>> is as common, but it seems reasonable to me. >> >> It's utterly useless since the scipy subpackages aren't imported along >> with it. Don't do it. > > Our docs disagree: > https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt That part was written by someone plagued by hobgoblins and who didn't think it through. > It still saves a few characters writing "sp.linalg.xxx" instead of > "scipy.linalg.xxx". False comparison. "scipy.linalg.xxx" is not thing to compare to. # Right way. from scipy import linalg linalg.xxx vs. # Silly, unnecessarily complicated way. import scipy as sp import scipy.linalg sp.linalg.xxx -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ralf.gommers at googlemail.com Fri Apr 1 17:05:13 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 1 Apr 2011 23:05:13 +0200 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: On Fri, Apr 1, 2011 at 10:56 PM, Robert Kern wrote: > On Fri, Apr 1, 2011 at 15:02, Ralf Gommers wrote: >> On Fri, Apr 1, 2011 at 9:59 PM, Robert Kern wrote: >>> On Fri, Apr 1, 2011 at 14:33, Christopher Barker wrote: >>> >>>> I'm not sure if: >>>> >>>> import scipy as sp >>>> >>>> is as common, but it seems reasonable to me. >>> >>> It's utterly useless since the scipy subpackages aren't imported along >>> with it. Don't do it. >> >> Our docs disagree: >> https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt > > That part was written by someone plagued by hobgoblins and who didn't > think it through. :) >> It still saves a few characters writing "sp.linalg.xxx" instead of >> "scipy.linalg.xxx". > > False comparison. "scipy.linalg.xxx" is not thing to compare to. > > ?# Right way. > ?from scipy import linalg > ?linalg.xxx That is clearer, I agree. But this is bad: >>> import io >>> io.__file__ '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/io.pyc' >>> from scipy import io >>> io.__file__ '/Users/rgommers/Code/scipy/scipy/io/__init__.pyc' So there you do want sp.io. Perhaps the thinking was to make it consistent in the docs by always prefixing with sp (I'm not saying that's the best solution though). Ralf From cweisiger at msg.ucsf.edu Fri Apr 1 17:07:21 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Fri, 1 Apr 2011 14:07:21 -0700 Subject: [SciPy-User] (pas de sujet) In-Reply-To: <20110401205507.GA1105@macwilliams.local> References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> <20110401205507.GA1105@macwilliams.local> Message-ID: On Fri, Apr 1, 2011 at 1:55 PM, Hans Georg Schaathun wrote: > On Fri, Apr 01, 2011 at 01:25:50PM -0700, Chris Weisiger wrote: > > How valuable is saving a couple of characters, though? The quality of > code > > is not in how terse it is but in how effectively it communicates what it > > wants to do, both to computers and to humans. > > Not much for a few characters, but your idea of typing out third-level > subpackages with long names in full might be just a little over the > top. Long names also means that lines have to wrap where they otherwise > would not, and the reader may need to look thrice to see if two names > are actually the same, or differing in the second part of the name. > Verbosity does not always increase legibility. Sometimes it is just > information overload. > > Certainly there's an exception to every rule, and I don't always type everything out myself either. I just prefer to err on the side of verbosity, because erring on the side of terseness has burnt me several times in the past. I have also found that using a * import interactively is somewhat > counterintuitive. Very quickly I start forgetting which package > each function comes from. Interactively, importing numpy/scipy > as np/sp is a serious saving, and then, using the same conventions > interactively and non-interactively has merits, which is probably > why these abbreviations have become so whidespread. > > This is why I added the "code will be reused inside of a month" loophole -- using the REPL (read-eval-print loop, a.k.a. interactive mode) you're almost certainly going to code differently than when you're writing something more permanent. I use short, obtuse variable names and import-as in the REPL all the time, because I only need things to persist for as long as the session does. If I end up writing code in the REPL that I keep for later, then the first thing I do is give it a quick style overhaul. -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From hg+scipy at schaathun.net Fri Apr 1 17:16:46 2011 From: hg+scipy at schaathun.net (Hans Georg Schaathun) Date: Fri, 1 Apr 2011 23:16:46 +0200 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> <20110401205507.GA1105@macwilliams.local> Message-ID: <20110401211646.GA1228@macwilliams.local> On Fri, Apr 01, 2011 at 02:07:21PM -0700, Chris Weisiger wrote: > This is why I added the "code will be reused inside of a month" loophole -- > using the REPL (read-eval-print loop, a.k.a. interactive mode) you're almost > certainly going to code differently than when you're writing something more > permanent. I use short, obtuse variable names and import-as in the REPL all > the time, because I only need things to persist for as long as the session > does. If I end up writing code in the REPL that I keep for later, then the > first thing I do is give it a quick style overhaul. Sure, but I also think that there is a strong case, when abbreviations are used so frequently that they get into your fingers (like np/sp), that they should be used everywhere, also for the persistent code. Admittedly, there is a huge gray area of abbreviations which sits in the fingers for the next 6/9/12 months, but who knows about the year after :-) -- :-- Hans Georg From robert.kern at gmail.com Fri Apr 1 17:17:37 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 1 Apr 2011 16:17:37 -0500 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: On Fri, Apr 1, 2011 at 16:05, Ralf Gommers wrote: >>>> import io >>>> io.__file__ > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/io.pyc' >>>> from scipy import io >>>> io.__file__ > '/Users/rgommers/Code/scipy/scipy/io/__init__.pyc' > > So there you do want sp.io. Perhaps the thinking was to make it > consistent in the docs by always prefixing with sp (I'm not saying > that's the best solution though). I think import io from scipy import io as spio # Name collision! spio.whatever() is better than import io import scipy as ip import scipy.io sp.io.whatever() Honestly, that's just confusing. The only non-confusing use of "import scipy as sp" is to use it for all of the numpy symbols it imports, e.g. "sp.array()", but that's a practice I would like to squash down on hard. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From hg+scipy at schaathun.net Fri Apr 1 17:28:01 2011 From: hg+scipy at schaathun.net (Hans Georg Schaathun) Date: Fri, 1 Apr 2011 23:28:01 +0200 Subject: [SciPy-User] (pas de sujet) Message-ID: <20110401212801.GA1281@macwilliams.local> On Fri, Apr 01, 2011 at 11:44:38PM +0300, eat wrote: > I just like to ask yours and others matured ones opinion about > systematically using import pattern like: > from numpy import x, y, z, longname as ln > from scipy import u, v, x as sx > from scipy.abc import e, f, g > With this scheme I think that: > - programmer takes (still) the whole responsibility of avoiding name > conflicts Good. > - reader (user) avoids the possible confusion between modules and classes Does he? How? > - reader (user) still needs to be aware where the names originated No. He will have to look it up in the import line, which may or may not be easy to find, but only the reader with photographic memory will be `aware' of it. > - it's kind of against of the philosophy of import this; explicit is better > than implicit (but that depends actually on how familiar you are with the > imported names) I am not sure I understand you here. > - it still feels somehow right to express that these (and only these) are > my external dependencies I certainly find that this is useful. When I need to restructure modules it does help to see the dependencies to this level of detail. > - it seems kind of to unify the way how you import external dependencies And that's probably its bane as well. You very quickly end up using the same name for different objects and different names for the same object in different files of your project, because you need to improvise aliases to avoid name clashes as you go along. (I cannot imagine you being able to plan a consistent alias scheme for your entire project, but maybe that's only my imagination.) It gets nasty very quickly. The case for spelling out /rarely/ used names in full is very strong. Your scheme works very well for important objects which you may need to replace with a different implementation at a later stage. For instance, I always use import numpy.random as rnd for the very reason that I am likely to reconsider its use, and may want to just redefine rnd as a different PRNG, without any desire to do search and replace throughout the code. In short, being systematic about this is probably a bad thing. :-- George From ralf.gommers at googlemail.com Fri Apr 1 17:33:43 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 1 Apr 2011 23:33:43 +0200 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: On Fri, Apr 1, 2011 at 11:17 PM, Robert Kern wrote: > On Fri, Apr 1, 2011 at 16:05, Ralf Gommers wrote: > >>>>> import io >>>>> io.__file__ >> '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/io.pyc' >>>>> from scipy import io >>>>> io.__file__ >> '/Users/rgommers/Code/scipy/scipy/io/__init__.pyc' >> >> So there you do want sp.io. Perhaps the thinking was to make it >> consistent in the docs by always prefixing with sp (I'm not saying >> that's the best solution though). > > I think > > ?import io > ?from scipy import io as spio # Name collision! > ?spio.whatever() > > is better than > > ?import io > ?import scipy as ip > ?import scipy.io > ?sp.io.whatever() > > Honestly, that's just confusing. The only non-confusing use of "import > scipy as sp" is to use it for all of the numpy symbols it imports, > e.g. "sp.array()", but that's a practice I would like to squash down > on hard. True, that's useless. Do you feel like writing a patch for the howto_document file? Ralf From robert.kern at gmail.com Fri Apr 1 18:49:46 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 1 Apr 2011 17:49:46 -0500 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: On Fri, Apr 1, 2011 at 16:33, Ralf Gommers wrote: > Do you feel like writing a patch for the > howto_document file? https://github.com/numpy/numpy/pull/67 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From david_baddeley at yahoo.com.au Sat Apr 2 02:04:15 2011 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Fri, 1 Apr 2011 23:04:15 -0700 (PDT) Subject: [SciPy-User] Optimisation of a rough function Message-ID: <615281.34961.qm@web113410.mail.gq1.yahoo.com> Hi all, there are enough optimisation gurus here that hopefully someone might have some ideas. I'm trying to optimise a goal function that has a well defined minimum, but also a certain amount of roughness (see attached figure for an example). Most of the time (80%) optimize.fmin seems to work, but it sometimes gets stuck in the roughness. optimise.anneal works, but needs many more function evaluations, making it a bit impractical (the goal is to be semi-interactive & the fmin implementation already struggles). The goal function is quite complex and already written in c. I'm really after some form of solver which will skip over all the little bumps, but still benefits from using the gradient information. Should probably also mention that it will usually have ~ 10 parameters, rather than the 2 parameter case shown, but that the same features should be present. Would welcome any ideas. thanks, David -------------- next part -------------- A non-text attachment was scrubbed... Name: optimisation_surface.png Type: application/octet-stream Size: 108306 bytes Desc: not available URL: From ralf.gommers at googlemail.com Sat Apr 2 05:32:14 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 2 Apr 2011 11:32:14 +0200 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: On Sat, Apr 2, 2011 at 12:49 AM, Robert Kern wrote: > On Fri, Apr 1, 2011 at 16:33, Ralf Gommers wrote: >> Do you feel like writing a patch for the >> howto_document file? > > https://github.com/numpy/numpy/pull/67 Thanks Robert! From gael.varoquaux at normalesup.org Sat Apr 2 05:41:17 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 2 Apr 2011 11:41:17 +0200 Subject: [SciPy-User] Optimisation of a rough function In-Reply-To: <615281.34961.qm@web113410.mail.gq1.yahoo.com> References: <615281.34961.qm@web113410.mail.gq1.yahoo.com> Message-ID: <20110402094117.GB10793@phare.normalesup.org> On Fri, Apr 01, 2011 at 11:04:15PM -0700, David Baddeley wrote: > there are enough optimisation gurus here that hopefully someone might > have some ideas. I'm trying to optimise a goal function that has a well > defined minimum, but also a certain amount of roughness (see attached > figure for an example). Most of the time (80%) optimize.fmin seems to > work, but it sometimes gets stuck in the roughness. optimise.anneal > works, but needs many more function evaluations, making it a bit > impractical (the goal is to be semi-interactive & the fmin > implementation already struggles). The goal function is quite complex > and already written in c. I'm really after some form of solver which > will skip over all the little bumps, but still benefits from using the > gradient information. Should probably also mention that it will usually > have ~ 10 parameters, rather than the 2 parameter case shown, but that > the same features should be present. So your function is rough, but bell shaped at a large scale? I don't have a silver bullet here (I'd love one), but I'd suggest looking at a simplex algorithm, such as scipy.optimize.fmin, which uses the Nelder-Mead. Simplex optimization does not rely on gradients, and is thus more robust to such problems. However, this will lead to other problems. Namely that errors from the bell shape will induce problems in the Nelder Mead wich will converge to the wrong position. I think that Matthieu Brucher suggested that a clever scheme alternating Nelder-Mead and some kind of grid-search in the simplex defined by the algorithm would be a good option. If someone has a reference on such an approach, I'd be very much interested. Gael From seb.haase at gmail.com Sat Apr 2 07:21:27 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Sat, 2 Apr 2011 13:21:27 +0200 Subject: [SciPy-User] Optimisation of a rough function In-Reply-To: <20110402094117.GB10793@phare.normalesup.org> References: <615281.34961.qm@web113410.mail.gq1.yahoo.com> <20110402094117.GB10793@phare.normalesup.org> Message-ID: On Sat, Apr 2, 2011 at 11:41 AM, Gael Varoquaux wrote: > On Fri, Apr 01, 2011 at 11:04:15PM -0700, David Baddeley wrote: >> there are enough optimisation gurus here that hopefully someone might >> have some ideas. I'm trying to optimise a goal function that has a well >> defined minimum, but also a certain amount of roughness (see attached >> figure for an example). Most of the time (80%) optimize.fmin seems to >> work, but it sometimes gets stuck in the roughness. optimise.anneal >> works, but needs many more function evaluations, making it a bit >> impractical (the goal is to be semi-interactive & the fmin >> implementation already struggles). The goal function is quite complex >> and already written in c. I'm really after some form of solver which >> will skip over all the little bumps, but still benefits from using the >> gradient information. Should probably also mention that it will usually >> have ~ 10 parameters, rather than the 2 parameter case shown, but that >> the same features should be present. > > So your function is rough, but bell shaped at a large scale? > > I don't have a silver bullet here (I'd love one), but I'd suggest looking > at a simplex algorithm, such as scipy.optimize.fmin, which uses the > Nelder-Mead. Simplex optimization does not rely on gradients, and is thus > more robust to such problems. > > However, this will lead to other problems. Namely that errors from the > bell shape will induce problems in the Nelder Mead wich will converge to > the wrong position. I think that Matthieu Brucher suggested that a clever > scheme alternating Nelder-Mead and some kind of grid-search in the > simplex defined by the algorithm would be a good option. If someone has a > reference on such an approach, I'd be very much interested. > > Gael How do you get your initial parameters ? How about doing perpendicular projections and a Gauss fit on those ? Those fits could just use the global extremum as initial values. For the example you show that should work really well. (Not sure if sum projections or max projections (maybe min projections) would be best.) - Sebastian From paul.anton.letnes at gmail.com Sat Apr 2 09:49:01 2011 From: paul.anton.letnes at gmail.com (Paul Anton Letnes) Date: Sat, 2 Apr 2011 15:49:01 +0200 Subject: [SciPy-User] Optimisation of a rough function In-Reply-To: <615281.34961.qm@web113410.mail.gq1.yahoo.com> References: <615281.34961.qm@web113410.mail.gq1.yahoo.com> Message-ID: On 2. apr. 2011, at 08.04, David Baddeley wrote: > Hi all, > > there are enough optimisation gurus here that hopefully someone might have some > ideas. I'm trying to optimise a goal function that has a well defined minimum, > but also a certain amount of roughness (see attached figure for an example). > Most of the time (80%) optimize.fmin seems to work, but it sometimes gets stuck > in the roughness. optimise.anneal works, but needs many more function > evaluations, making it a bit impractical (the goal is to be semi-interactive & > the fmin implementation already struggles). The goal function is quite complex > and already written in c. I'm really after some form of solver which will skip > over all the little bumps, but still benefits from using the gradient > information. Should probably also mention that it will usually have ~ 10 > parameters, rather than the 2 parameter case shown, but that the same features > should be present. Would welcome any ideas. > > thanks, > David I am no guru, but I have done some optimization work. 1) Genetic algorithms are good at not getting stuck in local minima. 2) Same can be said for the simulated annealing group of algorithms (simulated tempering et al.), at least if you get your parameters (temperature etc) right. 3) A simpler way can be what some people call "bad derivatives". Simply put, smoothen your objective function, or take long steps in the beginning. As you get closer to convergence, adjust your smoothing or step length. I do not remember any references off-hand, but perhaps a sketch of the algorithm is found in Numerical Recipes? (Disclaimer: I have never used this particular method myself.) 4) Simplest yet: since computation is quite cheap for such a simple (correct me if I am wrong here) objective function, simply perform N optimization runs with randomized starting points. Pick the one which converges to the smallest/biggest value. Good luck, Paul. From ralf.gommers at googlemail.com Sat Apr 2 09:51:39 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 2 Apr 2011 15:51:39 +0200 Subject: [SciPy-User] Normalization of window functions (documentation suggestion) In-Reply-To: <1301595702.6965.71.camel@mypride> References: <1301595702.6965.71.camel@mypride> Message-ID: On Thu, Mar 31, 2011 at 8:21 PM, Yury V. Zaytsev wrote: > Hi! > > In the documentation of the return values of the window functions in > NumPy, such as: > > bartlett(M) > blackman(M) > hamming(M) > hanning(M) > kaiser(M, beta) > > it always says something like "The window, normalized to one". I find > this slightly confusing, because I would actually read that as if the > values were normalized such that the area under the curve is one. > > I think it would make sense to change it to something like "The window, > with the maximum value normalized to one" to state it more explicitly. Done in commit 250245d. Thanks for the suggestion. Cheers, Ralf From matthieu.brucher at gmail.com Sat Apr 2 10:58:35 2011 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 2 Apr 2011 16:58:35 +0200 Subject: [SciPy-User] Optimisation of a rough function In-Reply-To: References: <615281.34961.qm@web113410.mail.gq1.yahoo.com> Message-ID: This seems to be indeed a global optimization issue. A local optimization (like Nelden-Mead or conjugated-gradient) is only successful if the function is convex, or if you start near the global optimium and if in this area, the function is convex. First, if you have access to the gradient, you should use it if its computation is not too slow. Then, if your function is not convex (which seems to be the case), you may start with simulated annealing or genetic algorithms to get inside a locally convex region. Next step would be retaining several sets of parameters and try a local optimization method on each set. I have several gradient-based optimizers available in scikits.optimization (I have some tutorials on my blog: http://matt.eifelle.com/category/python/generic-optimizers/) that you may try, but you can also use fmin by specifying the gradient. Matthieu 2011/4/2 Paul Anton Letnes > > On 2. apr. 2011, at 08.04, David Baddeley wrote: > > > Hi all, > > > > there are enough optimisation gurus here that hopefully someone might > have some > > ideas. I'm trying to optimise a goal function that has a well defined > minimum, > > but also a certain amount of roughness (see attached figure for an > example). > > Most of the time (80%) optimize.fmin seems to work, but it sometimes gets > stuck > > in the roughness. optimise.anneal works, but needs many more function > > evaluations, making it a bit impractical (the goal is to be > semi-interactive & > > the fmin implementation already struggles). The goal function is quite > complex > > and already written in c. I'm really after some form of solver which will > skip > > over all the little bumps, but still benefits from using the gradient > > information. Should probably also mention that it will usually have ~ 10 > > parameters, rather than the 2 parameter case shown, but that the same > features > > should be present. Would welcome any ideas. > > > > thanks, > > David > > I am no guru, but I have done some optimization work. > > 1) Genetic algorithms are good at not getting stuck in local minima. > 2) Same can be said for the simulated annealing group of algorithms > (simulated tempering et al.), at least if you get your parameters > (temperature etc) right. > 3) A simpler way can be what some people call "bad derivatives". Simply > put, smoothen your objective function, or take long steps in the beginning. > As you get closer to convergence, adjust your smoothing or step length. I do > not remember any references off-hand, but perhaps a sketch of the algorithm > is found in Numerical Recipes? (Disclaimer: I have never used this > particular method myself.) > 4) Simplest yet: since computation is quite cheap for such a simple > (correct me if I am wrong here) objective function, simply perform N > optimization runs with randomized starting points. Pick the one which > converges to the smallest/biggest value. > > Good luck, > Paul. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher -------------- next part -------------- An HTML attachment was scrubbed... URL: From silva at lma.cnrs-mrs.fr Sat Apr 2 11:31:45 2011 From: silva at lma.cnrs-mrs.fr (Fabrice Silva) Date: Sat, 02 Apr 2011 12:31:45 -0300 Subject: [SciPy-User] Normalization of window functions (documentation suggestion) In-Reply-To: References: <1301595702.6965.71.camel@mypride> Message-ID: <1301758305.23962.5.camel@Portable-s2m.cnrs-mrs.fr> Le samedi 02 avril 2011 ? 15:51 +0200, Ralf Gommers a ?crit : > On Thu, Mar 31, 2011 at 8:21 PM, Yury V. Zaytsev wrote: > > Hi! > > > > In the documentation of the return values of the window functions in > > NumPy, such as: > > > > bartlett(M) > > blackman(M) > > hamming(M) > > hanning(M) > > kaiser(M, beta) > > > > it always says something like "The window, normalized to one". I find > > this slightly confusing, because I would actually read that as if the > > values were normalized such that the area under the curve is one. > > > > I think it would make sense to change it to something like "The window, > > with the maximum value normalized to one" to state it more explicitly. > > Done in commit 250245d. Thanks for the suggestion. The maximum coefficient of the array returned is not 1 if an even number of values is required: >>> import numpy as np >>> np.hanning(5) array([ 0. , 0.5, 1. , 0.5, 0. ]) >>> np.hanning(4) array([ 0. , 0.75, 0.75, 0. ]) (same behaviour for others windows) For even number, the mean value (zero for centered window) is not sampled for symmetry reason. It is the underlying analytical (time continuous) function that has a maximum value normalised to one. -- Fabrice Silva From ralf.gommers at googlemail.com Sat Apr 2 11:37:43 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 2 Apr 2011 17:37:43 +0200 Subject: [SciPy-User] Normalization of window functions (documentation suggestion) In-Reply-To: <1301758305.23962.5.camel@Portable-s2m.cnrs-mrs.fr> References: <1301595702.6965.71.camel@mypride> <1301758305.23962.5.camel@Portable-s2m.cnrs-mrs.fr> Message-ID: On Sat, Apr 2, 2011 at 5:31 PM, Fabrice Silva wrote: > Le samedi 02 avril 2011 ? 15:51 +0200, Ralf Gommers a ?crit : >> On Thu, Mar 31, 2011 at 8:21 PM, Yury V. Zaytsev wrote: >> > Hi! >> > >> > In the documentation of the return values of the window functions in >> > NumPy, such as: >> > >> > bartlett(M) >> > blackman(M) >> > hamming(M) >> > hanning(M) >> > kaiser(M, beta) >> > >> > it always says something like "The window, normalized to one". I find >> > this slightly confusing, because I would actually read that as if the >> > values were normalized such that the area under the curve is one. >> > >> > I think it would make sense to change it to something like "The window, >> > with the maximum value normalized to one" to state it more explicitly. >> >> Done in commit 250245d. Thanks for the suggestion. > > The maximum coefficient of the array returned is not 1 if an even number > of values is required: Yes, the docs already said that. Current text is: The window, with the maximum value normalized to one (the value one appears only if the number of samples is odd). More improvements always welcome, but Yury's suggestion definitely made it clearer. Cheers, Ralf From dominique.orban at gmail.com Sat Apr 2 12:25:43 2011 From: dominique.orban at gmail.com (Dominique Orban) Date: Sat, 2 Apr 2011 12:25:43 -0400 Subject: [SciPy-User] Optimisation of a rough function Message-ID: > Date:?Sat, 2 Apr 2011 11:41:17 +0200 > Subject:?Re: [SciPy-User] Optimisation of a rough function > On Fri, Apr 01, 2011 at 11:04:15PM -0700, David Baddeley wrote: >> there are enough optimisation gurus here that hopefully someone might >> have some ideas. I'm trying to optimise a goal function that has a well >> defined minimum, but also a certain amount of roughness (see attached >> figure for an example). Most of the time (80%) optimize.fmin seems to >> work, but it sometimes gets stuck in the roughness. optimise.anneal >> works, but needs many more function evaluations, making it a bit >> impractical (the goal is to be semi-interactive & the fmin >> implementation already struggles). The goal function is quite complex >> and already written in c. I'm really after some form of solver which >> will skip over all the little bumps, but still benefits from using the >> gradient information. Should probably also mention that it will usually >> have ~ 10 parameters, rather than the 2 parameter case shown, but that >> the same features should be present. You have a noisy optimization problem. Not sure if it will solve your problem but you want to look into implicit filtering: http://www4.ncsu.edu/~ctk/imfil.html (don't know if there's a Python version of this). The method is described in Tim Kelley's book: http://www.siam.org/books/kelley/fr18/index.php -- Dominique From jsseabold at gmail.com Sat Apr 2 13:33:42 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Sat, 2 Apr 2011 13:33:42 -0400 Subject: [SciPy-User] scikits.timeseries on windows: invalid object used in slice Message-ID: Any idea what's going on here? Code works fine on Linux with the same version. But on Windows 7, 64-bit with the binaries from Chrisoph , I get the following In [1]: import scikits.timeseries as ts In [2]: s = """1947-01-01,21.700 ...: 1947-04-01,22.010 ...: 1947-07-01,22.490""" In [3]: from StringIO import StringIO In [4]: cpi = ts.tsfromtxt(StringIO(s), delimiter=",", freq="Q", datecols=(0), ...: dtype=float) In [5]: cpi Out[5]: timeseries([21.7 22.01 22.49], dates = [1947Q1 ... 1947Q3], freq = Q-DEC) In [6]: cpi[1:] --------------------------------------------------------------------------- ValueError Traceback (most recent call last) C:\school\seaboldsvn\projects\greatinfl\ in () C:\Python27\lib\site-packages\numpy\ma\core.pyc in __getslice__(self, i, j) 3060 3061 """ -> 3062 return self.__getitem__(slice(i, j)) 3063 3064 def __setslice__(self, i, j, value): C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in __getitem__(self , indx) 650 Returns the item described by i. Not a copy. 651 """ --> 652 (sindx, dindx, recheck) = self._index_checker(indx) 653 _data = ndarray.__getattribute__(self, '_data') 654 _mask = ndarray.__getattribute__(self, '_mask') C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in _index_checker(s elf, indx) 613 if isinstance(indx, slice): 614 indx = slice(self._slicebound_checker(indx.start), --> 615 self._slicebound_checker(indx.stop), 616 indx.step) 617 return (indx, indx, False) C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in _slicebound_chec ker(self, bound) 636 if not isinstance(bound, Date): 637 raise ValueError( --> 638 "invalid object used in slice: %s" % repr(bound)) 639 if bound.freq != _dates.freq: 640 raise TimeSeriesCompatibilityError('freq', ValueError: invalid object used in slice: 9223372036854775807L Cheers, Skipper From jsseabold at gmail.com Sat Apr 2 13:48:22 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Sat, 2 Apr 2011 13:48:22 -0400 Subject: [SciPy-User] scikits.timeseries on windows: invalid object used in slice In-Reply-To: References: Message-ID: On Sat, Apr 2, 2011 at 1:33 PM, Skipper Seabold wrote: > Any idea what's going on here? Code works fine on Linux with the same > version. But on Windows 7, 64-bit with the binaries from Chrisoph > , I get the following > > In [1]: import scikits.timeseries as ts > > In [2]: s = """1947-01-01,21.700 > ? ...: 1947-04-01,22.010 > ? ...: 1947-07-01,22.490""" > > In [3]: from StringIO import StringIO > > In [4]: cpi = ts.tsfromtxt(StringIO(s), delimiter=",", freq="Q", datecols=(0), > ? ...: ? ? ? ? ? ? ? ? ? ? ? ? dtype=float) > > In [5]: cpi > Out[5]: > timeseries([21.7 22.01 22.49], > ? dates = [1947Q1 ... 1947Q3], > ? freq ?= Q-DEC) > > > In [6]: cpi[1:] > --------------------------------------------------------------------------- > ValueError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call last) > > C:\school\seaboldsvn\projects\greatinfl\ in () > > C:\Python27\lib\site-packages\numpy\ma\core.pyc in __getslice__(self, i, j) > ? 3060 > ? 3061 ? ? ? ? """ > -> 3062 ? ? ? ? return self.__getitem__(slice(i, j)) > ? 3063 > ? 3064 ? ? def __setslice__(self, i, j, value): > > C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in __getitem__(self > , indx) > ? ?650 ? ? Returns the item described by i. Not a copy. > ? ?651 ? ? ? ? """ > --> 652 ? ? ? ? (sindx, dindx, recheck) = self._index_checker(indx) > ? ?653 ? ? ? ? _data = ndarray.__getattribute__(self, '_data') > ? ?654 ? ? ? ? _mask = ndarray.__getattribute__(self, '_mask') > > C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in _index_checker(s > elf, indx) > ? ?613 ? ? ? ? if isinstance(indx, slice): > ? ?614 ? ? ? ? ? ? indx = slice(self._slicebound_checker(indx.start), > --> 615 ? ? ? ? ? ? ? ? ? ? ? ? ?self._slicebound_checker(indx.stop), > ? ?616 ? ? ? ? ? ? ? ? ? ? ? ? ?indx.step) > ? ?617 ? ? ? ? ? ? return (indx, indx, False) > > C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in _slicebound_chec > ker(self, bound) > ? ?636 ? ? ? ? if not isinstance(bound, Date): > ? ?637 ? ? ? ? ? ? raise ValueError( > --> 638 ? ? ? ? ? ? ? ? "invalid object used in slice: %s" % repr(bound)) > ? ?639 ? ? ? ? if bound.freq != _dates.freq: > ? ?640 ? ? ? ? ? ? raise TimeSeriesCompatibilityError('freq', > > ValueError: invalid object used in slice: 9223372036854775807L > > Cheers, > > Skipper > And surprisingly, this works In [20]: cpi[:-1] Out[20]: timeseries([21.7 22.01], dates = [1947Q1 1947Q2], freq = Q-DEC) Skipper From cgohlke at uci.edu Sat Apr 2 14:36:00 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 02 Apr 2011 11:36:00 -0700 Subject: [SciPy-User] scikits.timeseries on windows: invalid object used in slice In-Reply-To: References: Message-ID: <4D976C90.2030105@uci.edu> On 4/2/2011 10:33 AM, Skipper Seabold wrote: > Any idea what's going on here? Code works fine on Linux with the same > version. But on Windows 7, 64-bit with the binaries from Chrisoph > , I get the following > > In [1]: import scikits.timeseries as ts > > In [2]: s = """1947-01-01,21.700 > ...: 1947-04-01,22.010 > ...: 1947-07-01,22.490""" > > In [3]: from StringIO import StringIO > > In [4]: cpi = ts.tsfromtxt(StringIO(s), delimiter=",", freq="Q", datecols=(0), > ...: dtype=float) > > In [5]: cpi > Out[5]: > timeseries([21.7 22.01 22.49], > dates = [1947Q1 ... 1947Q3], > freq = Q-DEC) > > > In [6]: cpi[1:] > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > > C:\school\seaboldsvn\projects\greatinfl\ in() > > C:\Python27\lib\site-packages\numpy\ma\core.pyc in __getslice__(self, i, j) > 3060 > 3061 """ > -> 3062 return self.__getitem__(slice(i, j)) > 3063 > 3064 def __setslice__(self, i, j, value): > > C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in __getitem__(self > , indx) > 650 Returns the item described by i. Not a copy. > 651 """ > --> 652 (sindx, dindx, recheck) = self._index_checker(indx) > 653 _data = ndarray.__getattribute__(self, '_data') > 654 _mask = ndarray.__getattribute__(self, '_mask') > > C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in _index_checker(s > elf, indx) > 613 if isinstance(indx, slice): > 614 indx = slice(self._slicebound_checker(indx.start), > --> 615 self._slicebound_checker(indx.stop), > 616 indx.step) > 617 return (indx, indx, False) > > C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in _slicebound_chec > ker(self, bound) > 636 if not isinstance(bound, Date): > 637 raise ValueError( > --> 638 "invalid object used in slice: %s" % repr(bound)) > 639 if bound.freq != _dates.freq: > 640 raise TimeSeriesCompatibilityError('freq', > > ValueError: invalid object used in slice: 9223372036854775807L > > Cheers, > > Skipper Your code works for me on win-amd64-py2.x but several timeseries selftests fail with this error. Maybe it has to do with timeseries internally storing 'absdates' as C type long, not ssize_t. Christoph From cgohlke at uci.edu Sat Apr 2 15:40:35 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sat, 02 Apr 2011 12:40:35 -0700 Subject: [SciPy-User] scikits.timeseries on windows: invalid object used in slice In-Reply-To: <4D976C90.2030105@uci.edu> References: <4D976C90.2030105@uci.edu> Message-ID: <4D977BB3.9020405@uci.edu> On 4/2/2011 11:36 AM, Christoph Gohlke wrote: > > > On 4/2/2011 10:33 AM, Skipper Seabold wrote: >> Any idea what's going on here? Code works fine on Linux with the same >> version. But on Windows 7, 64-bit with the binaries from Chrisoph >> , I get the following >> >> In [1]: import scikits.timeseries as ts >> >> In [2]: s = """1947-01-01,21.700 >> ...: 1947-04-01,22.010 >> ...: 1947-07-01,22.490""" >> >> In [3]: from StringIO import StringIO >> >> In [4]: cpi = ts.tsfromtxt(StringIO(s), delimiter=",", freq="Q", datecols=(0), >> ...: dtype=float) >> >> In [5]: cpi >> Out[5]: >> timeseries([21.7 22.01 22.49], >> dates = [1947Q1 ... 1947Q3], >> freq = Q-DEC) >> >> >> In [6]: cpi[1:] >> --------------------------------------------------------------------------- >> ValueError Traceback (most recent call last) >> >> C:\school\seaboldsvn\projects\greatinfl\ in() >> >> C:\Python27\lib\site-packages\numpy\ma\core.pyc in __getslice__(self, i, j) >> 3060 >> 3061 """ >> -> 3062 return self.__getitem__(slice(i, j)) >> 3063 >> 3064 def __setslice__(self, i, j, value): >> >> C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in __getitem__(self >> , indx) >> 650 Returns the item described by i. Not a copy. >> 651 """ >> --> 652 (sindx, dindx, recheck) = self._index_checker(indx) >> 653 _data = ndarray.__getattribute__(self, '_data') >> 654 _mask = ndarray.__getattribute__(self, '_mask') >> >> C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in _index_checker(s >> elf, indx) >> 613 if isinstance(indx, slice): >> 614 indx = slice(self._slicebound_checker(indx.start), >> --> 615 self._slicebound_checker(indx.stop), >> 616 indx.step) >> 617 return (indx, indx, False) >> >> C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in _slicebound_chec >> ker(self, bound) >> 636 if not isinstance(bound, Date): >> 637 raise ValueError( >> --> 638 "invalid object used in slice: %s" % repr(bound)) >> 639 if bound.freq != _dates.freq: >> 640 raise TimeSeriesCompatibilityError('freq', >> >> ValueError: invalid object used in slice: 9223372036854775807L >> >> Cheers, >> >> Skipper > > > Your code works for me on win-amd64-py2.x but several timeseries > selftests fail with this error. Maybe it has to do with timeseries > internally storing 'absdates' as C type long, not ssize_t. > > Christoph > A quick fix is attached. Christoph -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: _index_checker.diff URL: From pierre.raybaut at gmail.com Sat Apr 2 17:01:07 2011 From: pierre.raybaut at gmail.com (Pierre Raybaut) Date: Sat, 2 Apr 2011 23:01:07 +0200 Subject: [SciPy-User] ANN: Spyder v2.0.9 Message-ID: Hi all, I am pleased to announced that Spyder v2.0.9 has just been released. As this is mostly a maintenance release, a lot of bugs were fixed (see below) and some minor features were added. One of the most significative change since v2.0.8 is a major performance improvement for the editor's code introspection features (i.e. code completion, calltips and get definition location -- all three powered by the "rope" library) provided by an exclusive patch for "rope". Embedding an interactive Python console into your GUI-based application (Qt): This version includes an example of application using the Spyder's internal shell as a debugging console which also demonstrates the py2exe deployment procedure. Spyder (previously known as Pydee) is a free open-source Python development environment providing MATLAB-like features in a simple and light-weighted software, available for Windows XP/Vista/7, GNU/Linux and MacOS X: http://spyderlib.googlecode.com/ Spyder is part of spyderlib, a Python module based on PyQt4, pyflakes and rope (QScintilla's dependency has been removed in version 2.0 and rope features have been integrated since this version as well). Some of Spyder basic features: * Python, C/C++, Fortran source editor with class/function browser, code completion and calltips * consoles: o open as many Python interpreters, IPython consoles or command windows as you need o code completion and calltips o variable explorer with GUI-based editors for a lot of data types (numbers, strings, lists, arrays, dictionaries, ...) * object inspector: provide documentation or source code on any Python object (class, function, module, ...) * online documentation: automatically generated html documentation on installed Python modules * find in files * file explorer * project manager * MATLAB-like PYTHONPATH management dialog box (works with all consoles) * Windows only: current user environment variables editor * direct links to documentation (Python, Qt, Matplotlib, NumPy, Scipy, etc.) * direct link to Python(x,y) launcher * direct links to QtDesigner, QtLinguist and QtAssistant (Qt documentation) Bug fixes (since v2.0.8): Console: added option to ignore PyQt/sip errors when trying to set sip API (fixed Enthought Tool Suite 3.6.0 compatibility issue) utils.dochelpers.getargtxt/bugfix: retrieving builtin function arguments was no longer working (Fixes Issue 499) Editor-related keyboard shortcuts were not applied after opening files (Fixes Issue 575) Tab scroll buttons were not shown on OS X resulting in clamped/changing window sizes (Fixes Issue 574) Debugging: Spyder only synced at debugger breakpoints (Fixes Issue 576) "Source / Remove trailing spaces" was removing newline at the end of file (+ added support for "undo") (Fixes Issue 582) Console: changing font preferences was requiring a restart to be fully taken into account (Fixes Issue 562) Spyder was unable to restore editor's outline explorer tree when mixed ' and " characters were found in tree entries (Fixes Issue 590) Shell/"Clear line" shortcut was not correct: this is actually "Shift+Escape" (not "Escape") (Fixes Issue 591) History log was systematically erased when updating Spyder version Outline explorer/bugfix: when opening file, the 'show/hide all files' option was not applied (this was then applied when switching from a file to another) (Fixes Issue 602) Backported from v2.1 a couple of bugfixes related to Editor and multiple panels Object inspector: when raised automatically above other dockwidgets, plugin refresh was unnecessarily triggered Editor/code completion-bugfix: some key events (e.g. Ctrl+V) were lost during code completion-related hang-up (Fixes Issue 599) Multiline text pasting was not working in a newly opened console (i.e. first prompt) Enhancements (since v2.0.8): Major change/Translations: moved from 'QtLinguist' to 'gettext' (localizing Spyder should now be easier) Console: increased default maximum line count (buffer depth) up to 10,000 lines (instead of only 300 lines) Editor's rope-based introspection features (code completion, calltips, go to definition): new rope monkey-patch providing major performance improvements File explorer/Project explorer - opening file with associated application: now supported on all platforms Added action "Reset window layout" in "View" menu to reset main window layout to default Documentation: added page on debugging Editor: added syntax highlighters for diff/patch files (.diff, .patch, .rej) and gettext files (.po, .pot) (Fixes Issue 537) Global working directory toolbar: removed label considering the fact that the toolbar widgets are quite explicit on its role (and the combo box tooltip is explaining it in detail) (Fixes Issue 598) Added a .desktop file in source package (Fixes Issue 87) Editor plugin's title now show the current script filename Cheers, Pierre From david_baddeley at yahoo.com.au Sat Apr 2 17:25:27 2011 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Sat, 2 Apr 2011 14:25:27 -0700 (PDT) Subject: [SciPy-User] Optimisation of a rough function In-Reply-To: References: Message-ID: <464893.73571.qm@web113404.mail.gq1.yahoo.com> Thanks everyone! Implicit filtering definitely sounds like it might be a step in the right direction. To address some of the previous comments, the goal function is normally quite costly, making the brute force type of method somewhat impractical. To give you a bit more insight, I'm transforming a set of (x,y) points, calculating a triangularisation of the transformed point set, and calculating the product of all the edge lengths in that triangularisation. I'm trying to find parameters for the transformation which minimise this product. The goal surface I plotted was for a simplified case - 100 points and a simple linear transformation (ie 2 parameters). In normal applications I'll be looking at ~100k points, and a non-linear transformation (~10 parameters). cheers, David ----- Original Message ---- From: Dominique Orban To: scipy-user at scipy.org Sent: Sun, 3 April, 2011 4:25:43 AM Subject: Re: [SciPy-User] Optimisation of a rough function > Date: Sat, 2 Apr 2011 11:41:17 +0200 > Subject: Re: [SciPy-User] Optimisation of a rough function > On Fri, Apr 01, 2011 at 11:04:15PM -0700, David Baddeley wrote: >> there are enough optimisation gurus here that hopefully someone might >> have some ideas. I'm trying to optimise a goal function that has a well >> defined minimum, but also a certain amount of roughness (see attached >> figure for an example). Most of the time (80%) optimize.fmin seems to >> work, but it sometimes gets stuck in the roughness. optimise.anneal >> works, but needs many more function evaluations, making it a bit >> impractical (the goal is to be semi-interactive & the fmin >> implementation already struggles). The goal function is quite complex >> and already written in c. I'm really after some form of solver which >> will skip over all the little bumps, but still benefits from using the >> gradient information. Should probably also mention that it will usually >> have ~ 10 parameters, rather than the 2 parameter case shown, but that >> the same features should be present. You have a noisy optimization problem. Not sure if it will solve your problem but you want to look into implicit filtering: http://www4.ncsu.edu/~ctk/imfil.html (don't know if there's a Python version of this). The method is described in Tim Kelley's book: http://www.siam.org/books/kelley/fr18/index.php -- Dominique _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From charlesr.harris at gmail.com Sat Apr 2 17:30:01 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 2 Apr 2011 15:30:01 -0600 Subject: [SciPy-User] Optimisation of a rough function In-Reply-To: <464893.73571.qm@web113404.mail.gq1.yahoo.com> References: <464893.73571.qm@web113404.mail.gq1.yahoo.com> Message-ID: On Sat, Apr 2, 2011 at 3:25 PM, David Baddeley wrote: > Thanks everyone! Implicit filtering definitely sounds like it might be a > step in > the right direction. > > To address some of the previous comments, the goal function is normally > quite > costly, making the brute force type of method somewhat impractical. To give > you > a bit more insight, I'm transforming a set of (x,y) points, calculating a > triangularisation of the transformed point set, and calculating the product > of > all the edge lengths in that triangularisation. I'm trying to find > parameters > for the transformation which minimise this product. The goal surface I > plotted > was for a simplified case - 100 points and a simple linear transformation > (ie 2 > parameters). In normal applications I'll be looking at ~100k points, and a > non-linear transformation (~10 parameters). > > Map all the points to zero. Problem solved ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnl at cs.wisc.edu Sat Apr 2 22:35:55 2011 From: johnl at cs.wisc.edu (J. David Lee) Date: Sat, 02 Apr 2011 21:35:55 -0500 Subject: [SciPy-User] weave.inline alternative Message-ID: <4D97DD0B.1000905@cs.wisc.edu> Hello. In the last couple of days I've worked up a simple alternative to weave.inline to use in some of my own work, and on the chance that someone else would be interested in it, I've posted it here, with some documentation: http://pages.cs.wisc.edu/~johnl/np_inline/ My goal was to produce something that was as simple and maintainable as possible while still having the features of weave.inline that I'm interested in: passing floats, ints, and numpy arrays into C functions, manipulate arrays, and returning numeric values. An additional feature is the ability to compile the extension module properly when being accessed from multiple processes. Notably I'm not interested in creating or manipulating primitive python objects in C. If you're at all interested in something like this, please take a look and let me know what you think. Thanks, David From gennadiy.rishkin at gmail.com Sun Apr 3 07:42:32 2011 From: gennadiy.rishkin at gmail.com (Gennadiy Rishkin) Date: Sun, 3 Apr 2011 12:42:32 +0100 Subject: [SciPy-User] Calling odeint from C++ Message-ID: Hi, Is it possible to call odeint from C++. It solves my stiff system of ODEs excellently. I tried CVODE but that failed. I've also tried a C++ implementation of RADAU5, which also failed. Many thanks, Gennadiy -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sun Apr 3 15:06:20 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 3 Apr 2011 12:06:20 -0700 Subject: [SciPy-User] weave.inline alternative In-Reply-To: <4D97DD0B.1000905@cs.wisc.edu> References: <4D97DD0B.1000905@cs.wisc.edu> Message-ID: On Sat, Apr 2, 2011 at 7:35 PM, J. David Lee wrote: > Hello. > > In the last couple of days I've worked up a simple alternative to > weave.inline to use in some of my own work, and on the chance that > someone else would be interested in it, I've posted it here, with some > documentation: > > http://pages.cs.wisc.edu/~johnl/np_inline/ FYI, cython recently added this: http://wiki.cython.org/enhancements/inline I don't know if yours overlaps/extends cython's, I just mention it for reference since it's fairly new. Cheers, f From jeremy at jeremysanders.net Sun Apr 3 17:18:58 2011 From: jeremy at jeremysanders.net (Jeremy Sanders) Date: Sun, 03 Apr 2011 22:18:58 +0100 Subject: [SciPy-User] ANN: Veusz 1.11 - a Python scientific plotting package and module Message-ID: Veusz 1.11 ---------- Velvet Ember Under Sky Zenith ----------------------------- http://home.gna.org/veusz/ Copyright (C) 2003-2011 Jeremy Sanders and contributors. Licenced under the GPL (version 2 or greater). Veusz is a Qt4 based scientific plotting package. It is written in Python, using PyQt4 for display and user-interfaces, and numpy for handling the numeric data. Veusz is designed to produce publication-ready Postscript/PDF/SVG output. The user interface aims to be simple, consistent and powerful. Veusz provides a GUI, command line, embedding and scripting interface (based on Python) to its plotting facilities. It also allows for manipulation and editing of datasets. Data can be captured from external sources such as internet sockets or other programs. Changes in 1.11: * New data point picker for finding coordinates of points on plot (contributed by B.K. Stuhl) * New data navigator window for filtering, sorting and examining dataset statistics * ".." button next to dataset settings pops up data navigator for choosing datasets * Data fitting can now use PyMinuit, giving error estimates (B.K. Stuhl) * Console history now uses currently entered characters to select lines from history (B.K. Stuhl) * New self test script, comparing graph output with expected output * Put superscripts and subscripts above each other when formatting (B.K. Stuhl) * Key entries can have multiple lines (using \\) (B.K. Stuhl) * Option to treat blanks as data items in CSV files * Locale support added for number formatting - Can use current locale or US/English in documents - Can use US/English or current local in user interface * Linux binaries are now created on a more modern system * Windows binaries now use MSVC for compilation Bug fixes: * CSV import with blank columns fixed * Embedding module now working when using binary * Remember current directory with unicode characters * Extension module now compiles under MSVC in Windows * Output is always appended to console (B.K. Stuhl) * \r characters sometimes break data import in Windows * If using --export option, add directory of script to import path Minor bug fixes: * Zero sized dataset contour plot fix * Fix problem on context menu for axis match setting * Small values on log axis fix * Disable data edit dialog context menu when no datasets * Loading files with unicode filenames on command line * Do not allow non finite float settings Features of package: * X-Y plots (with errorbars) * Line and function plots * Contour plots * Images (with colour mappings and colorbars) * Stepped plots (for histograms) * Bar graphs * Vector field plots * Box plots * Polar plots * Plotting dates * Fitting functions to data * Stacked plots and arrays of plots * Plot keys * Plot labels * Shapes and arrows on plots * LaTeX-like formatting for text * EPS/PDF/PNG/SVG/EMF export * Scripting interface * Dataset creation/manipulation * Embed Veusz within other programs * Text, CSV, FITS and user-plugin importing * Data can be captured from external sources * User defined functions, constants and can import external Python functions * Plugin interface to allow user to write or load code to - import data using new formats - make new datasets, optionally linked to existing datasets - arbitrarily manipulate the document * Data picker Requirements for source install: Python (2.4 or greater required) http://www.python.org/ Qt >= 4.3 (free edition) http://www.trolltech.com/products/qt/ PyQt >= 4.3 (SIP is required to be installed first) http://www.riverbankcomputing.co.uk/pyqt/ http://www.riverbankcomputing.co.uk/sip/ numpy >= 1.0 http://numpy.scipy.org/ Optional: Microsoft Core Fonts (recommended for nice output) http://corefonts.sourceforge.net/ PyFITS >= 1.1 (optional for FITS import) http://www.stsci.edu/resources/software_hardware/pyfits pyemf >= 2.0.0 (optional for EMF export) http://pyemf.sourceforge.net/ PyMinuit >= 1.1.2 (optional improved fitting) http://code.google.com/p/pyminuit/ For EMF and better SVG export, PyQt >= 4.6 or better is required, to fix a bug in the C++ wrapping For documentation on using Veusz, see the "Documents" directory. The manual is in PDF, HTML and text format (generated from docbook). The examples are also useful documentation. Please also see and contribute to the Veusz wiki: http://barmag.net/veusz-wiki/ Issues with the current version: * Some recent versions of PyQt/SIP will causes crashes when exporting SVG files. Update to 4.7.4 (if released) or a recent snapshot to solve this problem. If you enjoy using Veusz, we would love to hear from you. Please join the mailing lists at https://gna.org/mail/?group=veusz to discuss new features or if you'd like to contribute code. The latest code can always be found in the Git repository at https://github.com/jeremysanders/veusz.git. From dtlussier at gmail.com Sun Apr 3 17:37:13 2011 From: dtlussier at gmail.com (Dan Lussier) Date: Sun, 3 Apr 2011 16:37:13 -0500 Subject: [SciPy-User] benchmark timeseries for correlated data Message-ID: <216A8349-396A-4772-80DF-F16701ED0BFF@gmail.com> Hello, I am doing some basic statistical manipulation on time series data (reading scalar from simulation code at regular interval). However, adjacent samples (or even nearby samples) in my data is likely to be somewhat correlated, due to the dynamics of the system under study. As a pragmatic approach to estimating a windowed mean and variance, I am using an interative block-averaging apporach (Flyvbjerg and Petersen. Error-Estimates on Averages of Correlated Data. Journal of Chemical Physics (1989) vol. 91 (1) pp. 461-466). This helps avoid the need to explicitly calculate the correlation time. I was wondering if anyone might have a good benchmark time series, with known properties, that I might use to test my code? or alternatively, point me in the direction of a source of this type of benchmark data for timeseries/statistical work. Many thanks, Dan From midel at sfr.fr Mon Apr 4 03:00:19 2011 From: midel at sfr.fr (midel) Date: Mon, 4 Apr 2011 09:00:19 +0200 (CEST) Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: <8017176.19901301900419924.JavaMail.www@wsfrf1125> Well, thank you very much for all your answers. In fact you gave me more answers than I had questions ;) I cannot try your solutions right now but I'll do it asap. It was clear to me that the way I coded was not rigorous and that was the reason why I had these errors. I'm a young scipy user so I will try to get the good habits from the beginning, in particularly with the separate namespaces. I've allready heard of it as I tried a few days ago to replace matlab with c++... Well, harder than scipy/matplotlib ! But I keep trying. In fact Scipy was so easy compared to c++ that I may have underestimated the "acclimatation" time :) Thanks ! ======================================== Message du : 02/04/2011 De : "Ralf Gommers " A : "SciPy Users List" Copie ? : Sujet : Re: [SciPy-User] (pas de sujet) On Sat, Apr 2, 2011 at 12:49 AM, Robert Kern wrote: > On Fri, Apr 1, 2011 at 16:33, Ralf Gommers wrote: >> Do you feel like writing a patch for the >> howto_document file? > > https://github.com/numpy/numpy/pull/67 Thanks Robert! _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From yury at shurup.com Mon Apr 4 08:39:31 2011 From: yury at shurup.com (Yury V. Zaytsev) Date: Mon, 04 Apr 2011 14:39:31 +0200 Subject: [SciPy-User] Normalization of window functions (documentation suggestion) In-Reply-To: References: <1301595702.6965.71.camel@mypride> Message-ID: <1301920771.13863.25.camel@mypride> On Sat, 2011-04-02 at 15:51 +0200, Ralf Gommers wrote: > Done in commit 250245d. Thanks for the suggestion. Many thanks! -- Sincerely yours, Yury V. Zaytsev From aisaac at american.edu Fri Apr 1 14:48:01 2011 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 01 Apr 2011 14:48:01 -0400 Subject: [SciPy-User] GARCH estimation Message-ID: <4D961DE1.3020702@american.edu> I see Enthought covers "estimating volatility using GARCH" in their Python for Quants course: http://www.enthought.com/training/python_for_financial_analysis.php Are the GARCH estimation routines available / part of SciPy? Thanks, Alan Isaac From wesmckinn at gmail.com Mon Apr 4 09:59:35 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 4 Apr 2011 09:59:35 -0400 Subject: [SciPy-User] GARCH estimation In-Reply-To: <4D961DE1.3020702@american.edu> References: <4D961DE1.3020702@american.edu> Message-ID: On Fri, Apr 1, 2011 at 2:48 PM, Alan G Isaac wrote: > I see Enthought covers "estimating volatility using GARCH" > in their Python for Quants course: > http://www.enthought.com/training/python_for_financial_analysis.php > > Are the GARCH estimation routines available / part of SciPy? > > Thanks, > Alan Isaac > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > I am not aware of anything in SciPy. Josef has put in some work on it in statsmodels: http://bazaar.launchpad.net/~scipystats/statsmodels/devel/view/head:/scikits/statsmodels/sandbox/tsa/garch.py From josef.pktd at gmail.com Mon Apr 4 10:46:35 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 4 Apr 2011 10:46:35 -0400 Subject: [SciPy-User] GARCH estimation In-Reply-To: References: <4D961DE1.3020702@american.edu> Message-ID: On Mon, Apr 4, 2011 at 9:59 AM, Wes McKinney wrote: > On Fri, Apr 1, 2011 at 2:48 PM, Alan G Isaac wrote: >> I see Enthought covers "estimating volatility using GARCH" >> in their Python for Quants course: >> http://www.enthought.com/training/python_for_financial_analysis.php >> >> Are the GARCH estimation routines available / part of SciPy? >> >> Thanks, >> Alan Isaac >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > I am not aware of anything in SciPy. Josef has put in some work on it > in statsmodels: > > http://bazaar.launchpad.net/~scipystats/statsmodels/devel/view/head:/scikits/statsmodels/sandbox/tsa/garch.py This needs considerable updating. It was written as the first MLE models for time series analysis and I was more interested in the general framework than producing a GARCH model, and a year ago we didn't have AR and ARMA yet. (AR and similar in that module are obsolete by now. It also still depends on numdifftools.) It works reasonably well in "nice" examples, but the two main things missing are parameter restrictions and analytical derivatives. A start for generic derivatives for the main distributions, normal and t distribution, are in http://bazaar.launchpad.net/~scipystats/statsmodels/devel/view/head:/scikits/statsmodels/sandbox/regression/tools.py Josef > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From johnl at cs.wisc.edu Mon Apr 4 12:01:28 2011 From: johnl at cs.wisc.edu (J. David Lee) Date: Mon, 04 Apr 2011 11:01:28 -0500 Subject: [SciPy-User] weave.inline alternative In-Reply-To: <4D97DD0B.1000905@cs.wisc.edu> References: <4D97DD0B.1000905@cs.wisc.edu> Message-ID: <4D99EB58.7040808@cs.wisc.edu> On 04/02/2011 09:35 PM, J. David Lee wrote: > Hello. > > In the last couple of days I've worked up a simple alternative to > weave.inline to use in some of my own work, and on the chance that > someone else would be interested in it, I've posted it here, with some > documentation: > > http://pages.cs.wisc.edu/~johnl/np_inline/ > > My goal was to produce something that was as simple and maintainable as > possible while still having the features of weave.inline that I'm > interested in: passing floats, ints, and numpy arrays into C functions, > manipulate arrays, and returning numeric values. An additional feature > is the ability to compile the extension module properly when being > accessed from multiple processes. Notably I'm not interested in creating > or manipulating primitive python objects in C. > > If you're at all interested in something like this, please take a look > and let me know what you think. I've created github repository for the code for those interested: https://github.com/johnnylee/np_inline Thanks, J. David Lee From Chris.Barker at noaa.gov Mon Apr 4 14:45:05 2011 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 04 Apr 2011 11:45:05 -0700 Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: <4D9A11B1.2080204@noaa.gov> On 4/1/11 3:49 PM, Robert Kern wrote: > On Fri, Apr 1, 2011 at 16:33, Ralf Gommers wrote: >> Do you feel like writing a patch for the >> howto_document file? > > https://github.com/numpy/numpy/pull/67 OK -- but I'm a bit confused as to what we do recommend as an import style. I know I feel strongly that: 1) "import *" is a Bad Idea It's a Good Idea to have a community standard. I think the community as more or less settled on: import numpy as np but I have no idea for scipy. Personally I generally find a use only a few specific things form scipy in an given script, so: from scipy.some_module import some_class works great for me, but I don't know if others have broad enough use of lots of scipy modules in a single script such that that would be onerous. So: what do folks use/recommend? On 4/1/11 1:44 PM, eat wrote: > I just like to ask yours and others matured ones opinion about > systematically using import pattern like: > from numpy import x, y, z, longname as ln > from scipy import u, v, x as sx > from scipy.abc import e, f, g I think that gets almost as bad as "import *", once you use a lot -- not as bad, but still not good. While I don't want "import *", but I also don't want really long method names like: "scipy.special.erf(...)" So I think there is a bit of discretion in every choice -- If I'm using only one or two functions, I'll do: from scipy.special import erf But If I'm using a lot from a given module, I tend to strip off all but the last module name: from scipy import special and maybe even give it a shorter name: from scipy import special as spc But this still keeps a qualification in place in the code. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cweisiger at msg.ucsf.edu Mon Apr 4 14:53:08 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Mon, 4 Apr 2011 11:53:08 -0700 Subject: [SciPy-User] (pas de sujet) In-Reply-To: <4D9A11B1.2080204@noaa.gov> References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> <4D9A11B1.2080204@noaa.gov> Message-ID: On Mon, Apr 4, 2011 at 11:45 AM, Christopher Barker wrote: > On 4/1/11 3:49 PM, Robert Kern wrote: > > On Fri, Apr 1, 2011 at 16:33, Ralf Gommers > wrote: > >> Do you feel like writing a patch for the > >> howto_document file? > > > > https://github.com/numpy/numpy/pull/67 > > OK -- but I'm a bit confused as to what we do recommend as an import > style. I know I feel strongly that: > > 1) "import *" is a Bad Idea > > It's a Good Idea to have a community standard. I think the community as > more or less settled on: > > import numpy as np > > but I have no idea for scipy. > > Personally I generally find a use only a few specific things form scipy > in an given script, so: > > from scipy.some_module import some_class > > works great for me, but I don't know if others have broad enough use of > lots of scipy modules in a single script such that that would be onerous. > > So: what do folks use/recommend? > > I think by now my personal stance is pretty clear. :) I accept most people aren't willing to have as long of invocations as I am. However, I do still strongly feel that the only time it's reasonable to have a bare (namespace-free) name in a piece of code is if the name is defined in that module. In other words, names without attached namespaces are implicitly locally-defined. Thus I don't like "from module.submodule import function" because when you use function later on, it's not clear that it came from outside. "from module import submodule" is fine though: when you see "submodule.function" you know where to start looking for information on the function. -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From jgomezdans at gmail.com Mon Apr 4 14:59:22 2011 From: jgomezdans at gmail.com (Jose Gomez-Dans) Date: Mon, 4 Apr 2011 19:59:22 +0100 Subject: [SciPy-User] Quick interpolation question Message-ID: Hi, I just want to interpolate, in the simplest possible terms, a 3D dataset. Linear interpolation, nearest neighbour, all that would suffice (this is to start off some algorithm, so no accurate estimate is required). In new scipy versions, things like griddata would be useful, but currently I only have scipy 0.8. So I have a "cube" (data[:,:,:], (NixNjxNk)) array, and an array of flags (flags[:,:,:,], True or False) of the same size. I want to interpolate my data for the elements of data where the corresponding element of flag is False, using eg the nearest valid datapoint in data, or some linear combination of "close by" points. I sort of think that map_coordinates would work, but I would need to filter data, and recalculate all the locations, so not efficient. Other ideas are to use np.interp1d and do it dimension by dimension. Would work, but clumsy. In essence, what I need is some "gapfilling" simple multi-dimensional interpolation that works fast (Ni, Nj and Nk are pretty large, >500 each). Any suggestions? Thanks! J -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Apr 4 16:04:25 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 4 Apr 2011 22:04:25 +0200 Subject: [SciPy-User] ANN: Numpy 1.6.0 beta 2 Message-ID: Hi, I am pleased to announce the availability of the second beta of NumPy 1.6.0. Due to the extensive changes in the Numpy core for this release, the beta testing phase will last at least one month. Please test this beta and report any problems on the Numpy mailing list. Many bug fixes went in since beta 1, among which: - fix installation of source release with python 3 - f2py fixes for assumed shape arrays - several loadtxt bug fixes and enhancements - change default floating point error handling from "print" to "warn" - much more I quickly counted in the timeline, and in the last few weeks the number of open tickets has been decreased by over 100. Thanks to everyone who contributed to this spring cleaning! Sources and binaries can be found at http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b2/ For (preliminary) release notes see below. Enjoy, Ralf Note: NumPy 1.6.0 is not yet released. ========================= NumPy 1.6.0 Release Notes ========================= This release includes several new features as well as numerous bug fixes and improved documentation. It is backward compatible with the 1.5.0 release, and supports Python 2.4 - 2.7 and 3.1 - 3.2. Highlights ========== * Re-introduction of datetime dtype support to deal with dates in arrays. * A new 16-bit floating point type. * A new iterator, which improves performance of many functions. New features ============ New 16-bit floating point type ------------------------------ This release adds support for the IEEE 754-2008 binary16 format, available as the data type ``numpy.half``. Within Python, the type behaves similarly to `float` or `double`, and C extensions can add support for it with the exposed half-float API. New iterator ------------ A new iterator has been added, replacing the functionality of the existing iterator and multi-iterator with a single object and API. This iterator works well with general memory layouts different from C or Fortran contiguous, and handles both standard NumPy and customized broadcasting. The buffering, automatic data type conversion, and optional output parameters, offered by ufuncs but difficult to replicate elsewhere, are now exposed by this iterator. Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` ------------------------------------------------------------------------- Extend the number of polynomials available in the polynomial package. In addition, a new ``window`` attribute has been added to the classes in order to specify the range the ``domain`` maps to. This is mostly useful for the Laguerre, Hermite, and HermiteE polynomials whose natural domains are infinite and provides a more intuitive way to get the correct mapping of values without playing unnatural tricks with the domain. Fortran assumed shape array and size function support in ``numpy.f2py`` ----------------------------------------------------------------------- F2py now supports wrapping Fortran 90 routines that use assumed shape arrays. Before such routines could be called from Python but the corresponding Fortran routines received assumed shape arrays as zero length arrays which caused unpredicted results. Thanks to Lorenz H?depohl for pointing out the correct way to interface routines with assumed shape arrays. In addition, f2py interprets Fortran expression ``size(array, dim)`` as ``shape(array, dim-1)`` which makes it possible to automatically wrap Fortran routines that use two argument ``size`` function in dimension specifications. Before users were forced to apply this mapping manually. Other new functions ------------------- ``numpy.ravel_multi_index`` : Converts a multi-index tuple into an array of flat indices, applying boundary modes to the indices. ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the Einstein summation convention, many common multi-dimensional array operations can be represented in a simple fashion. This function provides a way compute such summations. ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose the underlying type promotion used by the ufuncs and other operations to determine the types of outputs. These improve upon the ``numpy.common_type`` and ``numpy.mintypecode`` which provide similar functionality but do not match the ufunc implementation. Changes ======= Changes and improvements in the numpy core ------------------------------------------ ``default error handling`` -------------------------- The default error handling has been change from ``print`` to ``warn`` for all except for ``underflow``, which remains as ``ignore``. ``numpy.distutils`` ------------------- Several new compilers are supported for building Numpy: the Portland Group Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C compiler on Linux. ``numpy.testing`` ----------------- The testing framework gained ``numpy.testing.assert_allclose``, which provides a more convenient way to compare floating point arrays than `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`. ``C API`` --------- In addition to the APIs for the new iterator and half data type, a number of other additions have been made to the C API. The type promotion mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration ``NPY_CASTING`` has been added which controls what types of casts are permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` and ``PyArray_CanCastTypeTo``. A more flexible way to handle conversion of arbitrary python objects into arrays is exposed by ``PyArray_GetArrayParamsFromObject``. Deprecated features =================== The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality will be replaced by the new "density" keyword. Removed features ================ ``numpy.fft`` ------------- The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, which were aliases for the same functions without the 'e' in the name, were removed. ``numpy.memmap`` ---------------- The `sync()` and `close()` methods of memmap were removed. Use `flush()` and "del memmap" instead. ``numpy.lib`` ------------- The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. ``numpy.ma`` ------------ Several deprecated items were removed from the ``numpy.ma`` module:: * ``numpy.ma.MaskedArray`` "raw_data" method * ``numpy.ma.MaskedArray`` constructor "flag" keyword * ``numpy.ma.make_mask`` "flag" keyword * ``numpy.ma.allclose`` "fill_value" keyword ``numpy.distutils`` ------------------- The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include`` instead. From coomteng at gmail.com Mon Apr 4 17:12:24 2011 From: coomteng at gmail.com (Ting Zhou) Date: Mon, 04 Apr 2011 23:12:24 +0200 Subject: [SciPy-User] compiling problem of scipy 0.9 by intel icc Message-ID: <4D9A3438.4040808@gmail.com> hi, here is the error message: scipy/spatial/qhull/src/qhull_a.h(106): error: expected a ";" template ^ my compiler version is Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 12.0.0.084 Build 20101006. Thank you for help. Best wishes, Ting From cweisiger at msg.ucsf.edu Mon Apr 4 17:18:07 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Mon, 4 Apr 2011 14:18:07 -0700 Subject: [SciPy-User] fmin and step size Message-ID: Right, now my data slicing is working, I need to bend scipy.optimize.fmin to my will. My goal here is to take two images of the same sample taken from slightly different perspectives, and find the transformation that causes one to overlap the other. The cost function takes in four parameters (X translate, Y translate, rotation, zoom), applies them to a 2D slice of data, and gets the correlation coefficient between that transformed slice and a reference slice. By maximizing this (minimizing 1 - correlation coefficient) I should be able to use fmin to align the two slices precisely. I'm not worried about Z alignment at this time. There are also other potential transformations (e.g. skew) but their influence for our data is small enough that it can be ignored. The problem I'm seeing is that I don't have any influence over fmin's initial step size. Its first step with rotation is .00025, when the actual needed rotation is more than 1. Thus it always gets stuck in a local minimum that is far from the actual minimum. Similarly, the zoom parameter should never stray more than maybe .1 away from 1, but I frequently end up with absurd zoom amounts (even negative zoom). Of course I can eliminate that by making infinite cost for anything with zoom greater than some cutoff, but I suspect then I'd just end up with a local minimum along the cutoff edge. Some Googling suggested that I try fmin_powell and use the "direc" parameter, but it's not clear how that's supposed to be used. If I pass in e.g. [5, 5, 2, .01] then I would naively expect it to try only small zoom changes but relatively large X/Y/rotate changes, but the first time it calls the cost function with parameters different than the initial guess of [52.0, 27.0, 0.0, 1.0], I get [57.0, 32.0, 5.0, 6.0], so it's stepping 5 (the largest value in direc) in all four parameters. Any advice would be lovely. -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_baddeley at yahoo.com.au Mon Apr 4 17:18:10 2011 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Mon, 4 Apr 2011 14:18:10 -0700 (PDT) Subject: [SciPy-User] Quick interpolation question In-Reply-To: References: Message-ID: <721287.74166.qm@web113417.mail.gq1.yahoo.com> Hi Jose, am I correct in assuming that you are trying to recover missing data points on an existing regular grid? Depending on what fraction of your data is missing, you might be able to do something based on convolution - for example (2D, but easily generalisable): #set up a convolution kernel kernel = np.array([[1,1,1],[1,0,1],[1,1,1]], 'f') # could also use a less connected version, eg [[0,1,0],[1,0,1], [0,1,0]] #normalise so integral is 1 kernel = kernel/kernel.sum() #start with 0 for all missing values f = data*flags #convolve with the kernel - will replace each point with the mean of it's neighbours f_smooth = scipy.ndimage.convolve(f, kernel) #obtain a new estimate for the data by replacing the missing values with those from the smoothed version f = data*flags + f_smooth*(1-flags) this will effectively fix any isolated missing values. If you have larger missing regions, you should be able to reach a solution by iterating these last two steps - stop when the solution doesn't change. As convolution with such a small kernel is fast, the method should be fast for small missing regions, but will tend to blow up if the regions get too large & the iteration number goes up. cheers, David ________________________________ From: Jose Gomez-Dans To: SciPy Users List Sent: Tue, 5 April, 2011 6:59:22 AM Subject: [SciPy-User] Quick interpolation question Hi, I just want to interpolate, in the simplest possible terms, a 3D dataset. Linear interpolation, nearest neighbour, all that would suffice (this is to start off some algorithm, so no accurate estimate is required). In new scipy versions, things like griddata would be useful, but currently I only have scipy 0.8. So I have a "cube" (data[:,:,:], (NixNjxNk)) array, and an array of flags (flags[:,:,:,], True or False) of the same size. I want to interpolate my data for the elements of data where the corresponding element of flag is False, using eg the nearest valid datapoint in data, or some linear combination of "close by" points. I sort of think that map_coordinates would work, but I would need to filter data, and recalculate all the locations, so not efficient. Other ideas are to use np.interp1d and do it dimension by dimension. Would work, but clumsy. In essence, what I need is some "gapfilling" simple multi-dimensional interpolation that works fast (Ni, Nj and Nk are pretty large, >500 each). Any suggestions? Thanks! J -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_baddeley at yahoo.com.au Mon Apr 4 17:46:25 2011 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Mon, 4 Apr 2011 14:46:25 -0700 (PDT) Subject: [SciPy-User] fmin and step size In-Reply-To: References: Message-ID: <132348.87768.qm@web113405.mail.gq1.yahoo.com> Silly suggestion, but can you rescale the parameters so that they're all in the same ballpark (eg express translation as a fraction of the total image width, and rotation as a fraction of 2pi)? cheers, David ________________________________ From: Chris Weisiger To: SciPy Users List Sent: Tue, 5 April, 2011 9:18:07 AM Subject: [SciPy-User] fmin and step size Right, now my data slicing is working, I need to bend scipy.optimize.fmin to my will. My goal here is to take two images of the same sample taken from slightly different perspectives, and find the transformation that causes one to overlap the other. The cost function takes in four parameters (X translate, Y translate, rotation, zoom), applies them to a 2D slice of data, and gets the correlation coefficient between that transformed slice and a reference slice. By maximizing this (minimizing 1 - correlation coefficient) I should be able to use fmin to align the two slices precisely. I'm not worried about Z alignment at this time. There are also other potential transformations (e.g. skew) but their influence for our data is small enough that it can be ignored. The problem I'm seeing is that I don't have any influence over fmin's initial step size. Its first step with rotation is .00025, when the actual needed rotation is more than 1. Thus it always gets stuck in a local minimum that is far from the actual minimum. Similarly, the zoom parameter should never stray more than maybe .1 away from 1, but I frequently end up with absurd zoom amounts (even negative zoom). Of course I can eliminate that by making infinite cost for anything with zoom greater than some cutoff, but I suspect then I'd just end up with a local minimum along the cutoff edge. Some Googling suggested that I try fmin_powell and use the "direc" parameter, but it's not clear how that's supposed to be used. If I pass in e.g. [5, 5, 2, .01] then I would naively expect it to try only small zoom changes but relatively large X/Y/rotate changes, but the first time it calls the cost function with parameters different than the initial guess of [52.0, 27.0, 0.0, 1.0], I get [57.0, 32.0, 5.0, 6.0], so it's stepping 5 (the largest value in direc) in all four parameters. Any advice would be lovely. -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Mon Apr 4 17:54:45 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 04 Apr 2011 14:54:45 -0700 Subject: [SciPy-User] fmin and step size In-Reply-To: References: Message-ID: <4D9A3E25.7090608@uci.edu> On 4/4/2011 2:18 PM, Chris Weisiger wrote: > Right, now my data slicing is working, I need to bend > scipy.optimize.fmin to my will. My goal here is to take two images of > the same sample taken from slightly different perspectives, and find the > transformation that causes one to overlap the other. The cost function > takes in four parameters (X translate, Y translate, rotation, zoom), > applies them to a 2D slice of data, and gets the correlation coefficient > between that transformed slice and a reference slice. By maximizing this > (minimizing 1 - correlation coefficient) I should be able to use fmin to > align the two slices precisely. I'm not worried about Z alignment at > this time. There are also other potential transformations (e.g. skew) > but their influence for our data is small enough that it can be ignored. > > The problem I'm seeing is that I don't have any influence over fmin's > initial step size. Its first step with rotation is .00025, when the > actual needed rotation is more than 1. Thus it always gets stuck in a > local minimum that is far from the actual minimum. Similarly, the zoom > parameter should never stray more than maybe .1 away from 1, but I > frequently end up with absurd zoom amounts (even negative zoom). Of > course I can eliminate that by making infinite cost for anything with > zoom greater than some cutoff, but I suspect then I'd just end up with a > local minimum along the cutoff edge. > > Some Googling suggested that I try fmin_powell and use the "direc" > parameter, but it's not clear how that's supposed to be used. If I pass > in e.g. [5, 5, 2, .01] then I would naively expect it to try only small > zoom changes but relatively large X/Y/rotate changes, but the first time > it calls the cost function with parameters different than the initial > guess of [52.0, 27.0, 0.0, 1.0], I get [57.0, 32.0, 5.0, 6.0], so it's > stepping 5 (the largest value in direc) in all four parameters. > > Any advice would be lovely. > > -Chris > > If skew is not relevant you could use "An FFT-based technique for translation, rotation, and scale-invariant image registration" . Christoph From cweisiger at msg.ucsf.edu Mon Apr 4 18:23:30 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Mon, 4 Apr 2011 15:23:30 -0700 Subject: [SciPy-User] fmin and step size In-Reply-To: <132348.87768.qm@web113405.mail.gq1.yahoo.com> References: <132348.87768.qm@web113405.mail.gq1.yahoo.com> Message-ID: I should have thought of this. Obvious in hindsight. I'll give it a shot. The FT-based alignment also looks interesting, Christoph; I'll have to take a look at it. Thanks! -Chris On Mon, Apr 4, 2011 at 2:46 PM, David Baddeley wrote: > Silly suggestion, but can you rescale the parameters so that they're all in > the same ballpark (eg express translation as a fraction of the total image > width, and rotation as a fraction of 2pi)? > > cheers, > David > > ------------------------------ > *From:* Chris Weisiger > *To:* SciPy Users List > *Sent:* Tue, 5 April, 2011 9:18:07 AM > *Subject:* [SciPy-User] fmin and step size > > Right, now my data slicing is working, I need to bend scipy.optimize.fmin > to my will. My goal here is to take two images of the same sample taken from > slightly different perspectives, and find the transformation that causes one > to overlap the other. The cost function takes in four parameters (X > translate, Y translate, rotation, zoom), applies them to a 2D slice of data, > and gets the correlation coefficient between that transformed slice and a > reference slice. By maximizing this (minimizing 1 - correlation coefficient) > I should be able to use fmin to align the two slices precisely. I'm not > worried about Z alignment at this time. There are also other potential > transformations (e.g. skew) but their influence for our data is small enough > that it can be ignored. > > The problem I'm seeing is that I don't have any influence over fmin's > initial step size. Its first step with rotation is .00025, when the actual > needed rotation is more than 1. Thus it always gets stuck in a local minimum > that is far from the actual minimum. Similarly, the zoom parameter should > never stray more than maybe .1 away from 1, but I frequently end up with > absurd zoom amounts (even negative zoom). Of course I can eliminate that by > making infinite cost for anything with zoom greater than some cutoff, but I > suspect then I'd just end up with a local minimum along the cutoff edge. > > Some Googling suggested that I try fmin_powell and use the "direc" > parameter, but it's not clear how that's supposed to be used. If I pass in > e.g. [5, 5, 2, .01] then I would naively expect it to try only small zoom > changes but relatively large X/Y/rotate changes, but the first time it calls > the cost function with parameters different than the initial guess of [52.0, > 27.0, 0.0, 1.0], I get [57.0, 32.0, 5.0, 6.0], so it's stepping 5 (the > largest value in direc) in all four parameters. > > Any advice would be lovely. > > -Chris > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoytak at stat.washington.edu Mon Apr 4 19:33:52 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Mon, 4 Apr 2011 16:33:52 -0700 Subject: [SciPy-User] compiling problem of scipy 0.9 by intel icc In-Reply-To: <4D9A3438.4040808@gmail.com> References: <4D9A3438.4040808@gmail.com> Message-ID: That line is a bug and needs to be commented out. It's a line that is special cased for intel, but it shouldn't be -- the line for other compilers works fine. You can see my complete notes on compiling numpy/scipy with icc here: http://old.nabble.com/Compiling-numpy-using-icc-gets-missing-library-error-td31213447.html I'm trying to get that up on the wiki, but I've been swamped with travel and grad school lately :-/. --Hoyt On Mon, Apr 4, 2011 at 2:12 PM, Ting Zhou wrote: > hi, > > here is the error message: > > scipy/spatial/qhull/src/qhull_a.h(106): error: expected a ";" > ? template > ? ? ? ? ? ?^ > > my compiler version is Intel(R) C Intel(R) 64 Compiler XE for > applications running on Intel(R) 64, Version 12.0.0.084 Build 20101006. > > Thank you for help. > Best wishes, > Ting > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dplepage at gmail.com Mon Apr 4 20:17:18 2011 From: dplepage at gmail.com (Daniel Lepage) Date: Mon, 4 Apr 2011 20:17:18 -0400 Subject: [SciPy-User] fmin and step size In-Reply-To: <4D9A3E25.7090608@uci.edu> References: <4D9A3E25.7090608@uci.edu> Message-ID: FWIW, I've usually seen bundle adjustment done with the Levenberg-Marquardt algorithm. If your data has any structure to it (hard edges, corners, etc.) your optimization will be much faster if you first detect these features and then solve for the transformation using just these features (SIFT features are often used for this, but may be overkill if your images are all roughly the same scale). OpenCV provides some tools for this; see if you want to call it from python (scikits.image also wraps some of OpenCV). HTH, Dan Lepage On Mon, Apr 4, 2011 at 5:54 PM, Christoph Gohlke wrote: > > > On 4/4/2011 2:18 PM, Chris Weisiger wrote: >> Right, now my data slicing is working, I need to bend >> scipy.optimize.fmin to my will. My goal here is to take two images of >> the same sample taken from slightly different perspectives, and find the >> transformation that causes one to overlap the other. The cost function >> takes in four parameters (X translate, Y translate, rotation, zoom), >> applies them to a 2D slice of data, and gets the correlation coefficient >> between that transformed slice and a reference slice. By maximizing this >> (minimizing 1 - correlation coefficient) I should be able to use fmin to >> align the two slices precisely. I'm not worried about Z alignment at >> this time. There are also other potential transformations (e.g. skew) >> but their influence for our data is small enough that it can be ignored. >> >> The problem I'm seeing is that I don't have any influence over fmin's >> initial step size. Its first step with rotation is .00025, when the >> actual needed rotation is more than 1. Thus it always gets stuck in a >> local minimum that is far from the actual minimum. Similarly, the zoom >> parameter should never stray more than maybe .1 away from 1, but I >> frequently end up with absurd zoom amounts (even negative zoom). Of >> course I can eliminate that by making infinite cost for anything with >> zoom greater than some cutoff, but I suspect then I'd just end up with a >> local minimum along the cutoff edge. >> >> Some Googling suggested that I try fmin_powell and use the "direc" >> parameter, but it's not clear how that's supposed to be used. If I pass >> in e.g. [5, 5, 2, .01] then I would naively expect it to try only small >> zoom changes but relatively large X/Y/rotate changes, but the first time >> it calls the cost function with parameters different than the initial >> guess of [52.0, 27.0, 0.0, 1.0], I get [57.0, 32.0, 5.0, 6.0], so it's >> stepping 5 (the largest value in direc) in all four parameters. >> >> Any advice would be lovely. >> >> -Chris >> >> > > If skew is not relevant you could use "An FFT-based technique for > translation, rotation, and scale-invariant image registration" > . > > Christoph > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From coolhead.pranay at gmail.com Mon Apr 4 22:38:00 2011 From: coolhead.pranay at gmail.com (coolhead.pranay at gmail.com) Date: Mon, 4 Apr 2011 22:38:00 -0400 Subject: [SciPy-User] Working with sparse matrices Message-ID: Hi, I am working with huge sparse matrices(20K * 20K) using the scipy.sparse package. In each iteration of the algorithm I need to compute matrix product of this matrix with itself, store the result in the same matrix and then raise the value of each non zero entry in the matrix by a constant number. What would be the ideal format of the sparse matrix that I need to use in this scenario. I am currently using coo_matrix to construct the sparse matrix and then converted it to "csr" format for working in the loop. But I am getting ScipyEfficiencyWarning. #Function is something like this def algo(inpfile): mat = read_sparse_from_file(inpfile) #inpfile contains the actual matrix..reads line by line and returns a csc_matrix for i in range(10): mat = mat.dot(mat) mat = my_exponent(mat,exp) #my_exponent is a function that computes raises each element to "exp" and does some other stuff...returns csc matrix Thanks, Pranay Anchuri -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Mon Apr 4 22:41:17 2011 From: david at silveregg.co.jp (David) Date: Tue, 05 Apr 2011 11:41:17 +0900 Subject: [SciPy-User] Working with sparse matrices In-Reply-To: References: Message-ID: <4D9A814D.1070409@silveregg.co.jp> On 04/05/2011 11:38 AM, coolhead.pranay at gmail.com wrote: > Hi, > > I am working with huge sparse matrices(20K * 20K) using the scipy.sparse > package. In each iteration of the algorithm I need to compute matrix > product of this matrix with itself, store the result in the same matrix > and then raise the value of each non zero entry in the matrix by a > constant number. Why do you need to do this ? Depending on the sparse structure of you matrix, raising it as some integer power will very quickly fill it to near-full matrix. Generally, if you need to multiply sparse matrices between them, it should raise a warning and make you wonder whether you can avoid it. cheers, David From coomteng at gmail.com Tue Apr 5 02:37:19 2011 From: coomteng at gmail.com (Ting Zhou) Date: Tue, 05 Apr 2011 08:37:19 +0200 Subject: [SciPy-User] compiling problem of scipy 0.9 by intel icc In-Reply-To: References: <4D9A3438.4040808@gmail.com> Message-ID: <4D9AB89F.9010903@gmail.com> ? 2011-4-5 1:33, Hoyt Koepke ??: > That line is a bug and needs to be commented out. It's a line that is > special cased for intel, but it shouldn't be -- the line for other compilers > works fine. > > You can see my complete notes on compiling numpy/scipy with icc here: > > http://old.nabble.com/Compiling-numpy-using-icc-gets-missing-library-error-td31213447.html > > I'm trying to get that up on the wiki, but I've been swamped with > travel and grad school lately :-/. > > --Hoyt > > > On Mon, Apr 4, 2011 at 2:12 PM, Ting Zhou wrote: >> hi, >> >> here is the error message: >> >> scipy/spatial/qhull/src/qhull_a.h(106): error: expected a ";" >> template >> ^ >> >> my compiler version is Intel(R) C Intel(R) 64 Compiler XE for >> applications running on Intel(R) 64, Version 12.0.0.084 Build 20101006. >> >> Thank you for help. >> Best wishes, >> Ting >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > Thank you Hoyt. I just Replace #if defined(__INTEL_COMPILER) && !defined(QHULL_OS_WIN) template inline void qhullUnused(T &x) { (void)x; } # define QHULL_UNUSED(x) qhullUnused(x); #else # define QHULL_UNUSED(x) (void)x; #endif with #define QHULL_UNUSED(x) as you suggest. best wishes, ting From seb.haase at gmail.com Tue Apr 5 03:24:07 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 5 Apr 2011 09:24:07 +0200 Subject: [SciPy-User] fmin and step size In-Reply-To: <4D9A3E25.7090608@uci.edu> References: <4D9A3E25.7090608@uci.edu> Message-ID: Hi Christoph, we don't seem to have access to that journal over here. Is there any chance you could upload that paper somewhere for us to grab it - or maybe you could sent it to me off-list. I have been trying to remember this FT-based rotation approach for a long time; I had heard about such an approach many years ago, and could never manage to find it. Thank you for pointing it out. - Sebastian Haase On Mon, Apr 4, 2011 at 11:54 PM, Christoph Gohlke wrote: > > > On 4/4/2011 2:18 PM, Chris Weisiger wrote: >> Right, now my data slicing is working, I need to bend >> scipy.optimize.fmin to my will. My goal here is to take two images of >> the same sample taken from slightly different perspectives, and find the >> transformation that causes one to overlap the other. The cost function >> takes in four parameters (X translate, Y translate, rotation, zoom), >> applies them to a 2D slice of data, and gets the correlation coefficient >> between that transformed slice and a reference slice. By maximizing this >> (minimizing 1 - correlation coefficient) I should be able to use fmin to >> align the two slices precisely. I'm not worried about Z alignment at >> this time. There are also other potential transformations (e.g. skew) >> but their influence for our data is small enough that it can be ignored. >> >> The problem I'm seeing is that I don't have any influence over fmin's >> initial step size. Its first step with rotation is .00025, when the >> actual needed rotation is more than 1. Thus it always gets stuck in a >> local minimum that is far from the actual minimum. Similarly, the zoom >> parameter should never stray more than maybe .1 away from 1, but I >> frequently end up with absurd zoom amounts (even negative zoom). Of >> course I can eliminate that by making infinite cost for anything with >> zoom greater than some cutoff, but I suspect then I'd just end up with a >> local minimum along the cutoff edge. >> >> Some Googling suggested that I try fmin_powell and use the "direc" >> parameter, but it's not clear how that's supposed to be used. If I pass >> in e.g. [5, 5, 2, .01] then I would naively expect it to try only small >> zoom changes but relatively large X/Y/rotate changes, but the first time >> it calls the cost function with parameters different than the initial >> guess of [52.0, 27.0, 0.0, 1.0], I get [57.0, 32.0, 5.0, 6.0], so it's >> stepping 5 (the largest value in direc) in all four parameters. >> >> Any advice would be lovely. >> >> -Chris >> >> > > If skew is not relevant you could use "An FFT-based technique for > translation, rotation, and scale-invariant image registration" > . > > Christoph > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From jgomezdans at gmail.com Tue Apr 5 07:02:45 2011 From: jgomezdans at gmail.com (Jose Gomez-Dans) Date: Tue, 5 Apr 2011 12:02:45 +0100 Subject: [SciPy-User] Quick interpolation question In-Reply-To: <721287.74166.qm@web113417.mail.gq1.yahoo.com> References: <721287.74166.qm@web113417.mail.gq1.yahoo.com> Message-ID: Hi David, Thanks for your comments On 4 April 2011 22:18, David Baddeley wrote: > am I correct in assuming that you are trying to recover missing data points > on an existing regular grid? Depending on what fraction of your data is > missing, you might be able to do something based on convolution - for > example (2D, but easily generalisable): > Yes, you got that right. However, when I don't have data, I have large gaps in two of the dimensions. Maybe all I need is a larger kernel for these two dimensions. The other thing that bothers me is you would convolve with missing observations, giving them a value of 0. I don't want to do this, as they are missing observations, and they would bias the estimate. I was thinking of coding a nearest neighbour estimator, but I'm pretty sure that there has to be something off the shelf! Cheers, Jose -------------- next part -------------- An HTML attachment was scrubbed... URL: From hg+scipy at schaathun.net Tue Apr 5 07:10:54 2011 From: hg+scipy at schaathun.net (Hans Georg Schaathun) Date: Tue, 5 Apr 2011 13:10:54 +0200 Subject: [SciPy-User] Probability Density Estimation Message-ID: <20110405111054.GA2196@macwilliams.local> Hi, does anyone here have any experience with probability density estimation? I have found the scipy.stats.gaussian_kde, but I find that it is extremely sensitive to outliers. I have datasets which tend to have outliers at up to about 35 standard deviations in a sample of 500-1000. The PDF estimate then turns out to be close to uniform, and not at all useful. While I would be grateful for any pointers and advice on the numerical and algorithmic sides of my problem, this is a scipy list after all. The scipy question is this: Are there other, non-Gaussian KDE-s available, that I have missed? Or even KDE-s which allows the bandwidth to be specified precisely? TIA -- :-- Hans Georg From jgomezdans at gmail.com Tue Apr 5 08:36:49 2011 From: jgomezdans at gmail.com (Jose Gomez-Dans) Date: Tue, 5 Apr 2011 13:36:49 +0100 Subject: [SciPy-User] Quick interpolation question In-Reply-To: <043ef25c-6d90-4ae9-ad53-52dbc88f2763@r13g2000yqk.googlegroups.com> References: <043ef25c-6d90-4ae9-ad53-52dbc88f2763@r13g2000yqk.googlegroups.com> Message-ID: Denis, On 5 April 2011 13:15, denis wrote: > Jose, > see also the combination of scipy.spatial.cKDTree and inverse > distance weighting under > > http://stackoverflow.com/questions/3104781/inverse-distance-weighted-idw-interpolation-with-python Yes, that seems to do what I need. Thanks! Jose -------------- next part -------------- An HTML attachment was scrubbed... URL: From evanmason at gmail.com Tue Apr 5 09:17:37 2011 From: evanmason at gmail.com (Evan Mason) Date: Tue, 5 Apr 2011 13:17:37 +0000 (UTC) Subject: [SciPy-User] interpolation with CloughTocher2DInterpolator References: Message-ID: > class MyInterpolator(interpnd.CloughTocher2DInterpolator): > def __init__(self, tri, values, fill_value=np.nan, > tol=1e-6, maxiter=400): > interpnd.NDInterpolatorBase.__init__(self, tri.points, values, ndim=2, > fill_value=fill_value) .... > > tri = Delaunay(points) > ip = MyInterpolator(tri, values) Thanks a lot for your reply. For 'Delaunay' I tried to use the matplotlib function: tri = delaunay.Triangulation(points[:,0],points[:,1]) However this does not have a tri.points method; I tried using tri.triangle_nodes and several other alternatives but none are suitable. Can you point me to the Delaunay that you were intending? Thanks, Evan From pav at iki.fi Tue Apr 5 09:20:28 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 5 Apr 2011 13:20:28 +0000 (UTC) Subject: [SciPy-User] interpolation with CloughTocher2DInterpolator References: Message-ID: Tue, 05 Apr 2011 13:17:37 +0000, Evan Mason wrote: [clip] > Thanks a lot for your reply. For 'Delaunay' I tried to use the > matplotlib function: > > tri = delaunay.Triangulation(points[:,0],points[:,1]) > > However this does not have a tri.points method; I tried using > tri.triangle_nodes and several other alternatives but none are suitable. > Can you point me to the Delaunay that you were intending? Try: from scipy.spatial import Delaunay Cheers, Pauli From kiyo at cita.utoronto.ca Tue Apr 5 12:10:29 2011 From: kiyo at cita.utoronto.ca (Kiyoshi Masui) Date: Tue, 05 Apr 2011 12:10:29 -0400 Subject: [SciPy-User] Telling the difference between new from template and view casting in __array_finalize__ In-Reply-To: <4D90B73A.3050904@cita.utoronto.ca> References: <4D90B73A.3050904@cita.utoronto.ca> Message-ID: <4D9B3EF5.4000206@cita.utoronto.ca> Hi all, When sub classing an array, how do you tell the difference between 'view casting' and 'new from template' in `__array_finalize__`? The document here: http://docs.scipy.org/doc/numpy/user/basics.subclassing.html states that the second argument to `__array_finalize__` is always an instance of my subclass for 'new from template' but for operations like `A = NewArrayClass(sp.zeros(5)).view(NewArrayClass)`, this second argument will also be an instance of my subclass for view casting. Is there a way to tell these apart? Thanks, Kiyo From gnurser at gmail.com Tue Apr 5 12:31:44 2011 From: gnurser at gmail.com (George Nurser) Date: Tue, 5 Apr 2011 17:31:44 +0100 Subject: [SciPy-User] [Numpy-discussion] ANN: Numpy 1.6.0 beta 2 In-Reply-To: References: Message-ID: SUSE 11.3, python 2.7, gcc 4.3.4, gfortran from gcc 4.6.0, I get two failures on commit 1439a8ddcb2eda20fa102aa44e846783f29c0af3 (head of 1.6.x maintenance branch). --George. ====================================================================== FAIL: Test basic arithmetic function errors ---------------------------------------------------------------------- Traceback (most recent call last): File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/testing/decorators.py", line 215, in knownfailer return f(*args, **kwargs) File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 321, in test_floating_exceptions lambda a,b:a/b, ft_tiny, ft_max) File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 271, in assert_raises_fpe "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: Type did not raise fpe error 'underflow'. ====================================================================== FAIL: test_kind.TestKind.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/nose/case.py", line 187, in runTest self.test(*self.arg) File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/f2py/tests/test_kind.py", line 30, in test_all 'selectedrealkind(%s): expected %r but got %r' % (i, selected_real_kind(i), selectedrealkind(i))) File "/noc/users/agn/ext/SUSE_11/lib/python2.7/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: selectedrealkind(19): expected -1 but got 16 ---------------------------------------------------------------------- Ran 3544 tests in 48.039s FAILED (KNOWNFAIL=4, failures=2) On 4 April 2011 21:04, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the second beta of NumPy > 1.6.0. Due to the extensive changes in the Numpy core for this > release, the beta testing phase will last at least one month. Please > test this beta and report any problems on the Numpy mailing list. > > Many bug fixes went in since beta 1, among which: > - fix installation of source release with python 3 > - f2py fixes for assumed shape arrays > - several loadtxt bug fixes and enhancements > - change default floating point error handling from "print" to "warn" > - much more > > I quickly counted in the timeline, and in the last few weeks the > number of open tickets has been decreased by over 100. Thanks to > everyone who contributed to this spring cleaning! > > Sources and binaries can be found at > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0b2/ > For (preliminary) release notes see below. > > Enjoy, > Ralf > > > > Note: NumPy 1.6.0 is not yet released. > > > ========================= > NumPy 1.6.0 Release Notes > ========================= > > This release includes several new features as well as numerous bug fixes and > improved documentation. ?It is backward compatible with the 1.5.0 release, and > supports Python 2.4 - 2.7 and 3.1 - 3.2. > > > Highlights > ========== > > * Re-introduction of datetime dtype support to deal with dates in arrays. > > * A new 16-bit floating point type. > > * A new iterator, which improves performance of many functions. > > > New features > ============ > > New 16-bit floating point type > ------------------------------ > > This release adds support for the IEEE 754-2008 binary16 format, available as > the data type ``numpy.half``. ?Within Python, the type behaves similarly to > `float` or `double`, and C extensions can add support for it with the exposed > half-float API. > > > New iterator > ------------ > > A new iterator has been added, replacing the functionality of the > existing iterator and multi-iterator with a single object and API. > This iterator works well with general memory layouts different from > C or Fortran contiguous, and handles both standard NumPy and > customized broadcasting. The buffering, automatic data type > conversion, and optional output parameters, offered by > ufuncs but difficult to replicate elsewhere, are now exposed by this > iterator. > > > Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` > ------------------------------------------------------------------------- > > Extend the number of polynomials available in the polynomial package. In > addition, a new ``window`` attribute has been added to the classes in > order to specify the range the ``domain`` maps to. This is mostly useful > for the Laguerre, Hermite, and HermiteE polynomials whose natural domains > are infinite and provides a more intuitive way to get the correct mapping > of values without playing unnatural tricks with the domain. > > > Fortran assumed shape array and size function support in ``numpy.f2py`` > ----------------------------------------------------------------------- > > F2py now supports wrapping Fortran 90 routines that use assumed shape > arrays. ?Before such routines could be called from Python but the > corresponding Fortran routines received assumed shape arrays as zero > length arrays which caused unpredicted results. Thanks to Lorenz > H?depohl for pointing out the correct way to interface routines with > assumed shape arrays. > > In addition, f2py interprets Fortran expression ``size(array, dim)`` > as ``shape(array, dim-1)`` which makes it possible to automatically > wrap Fortran routines that use two argument ``size`` function in > dimension specifications. Before users were forced to apply this > mapping manually. > > > Other new functions > ------------------- > > ``numpy.ravel_multi_index`` : Converts a multi-index tuple into > an array of flat indices, applying boundary modes to the indices. > > ``numpy.einsum`` : Evaluate the Einstein summation convention. ?Using the > Einstein summation convention, many common multi-dimensional array operations > can be represented in a simple fashion. ?This function provides a way compute > such summations. > > ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. > > ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose > the underlying type promotion used by the ufuncs and other operations to > determine the types of outputs. These improve upon the ``numpy.common_type`` > and ``numpy.mintypecode`` which provide similar functionality but do > not match the ufunc implementation. > > > Changes > ======= > > Changes and improvements in the numpy core > ------------------------------------------ > > ``default error handling`` > -------------------------- > > The default error handling has been change from ``print`` to ``warn`` for > all except for ``underflow``, which remains as ``ignore``. > > > ``numpy.distutils`` > ------------------- > > Several new compilers are supported for building Numpy: the Portland Group > Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C > compiler on Linux. > > > ``numpy.testing`` > ----------------- > > The testing framework gained ``numpy.testing.assert_allclose``, which provides > a more convenient way to compare floating point arrays than > `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`. > > > ``C API`` > --------- > > In addition to the APIs for the new iterator and half data type, a number > of other additions have been made to the C API. The type promotion > mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, > ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration > ``NPY_CASTING`` has been added which controls what types of casts are > permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` > and ``PyArray_CanCastTypeTo``. ?A more flexible way to handle > conversion of arbitrary python objects into arrays is exposed by > ``PyArray_GetArrayParamsFromObject``. > > > Deprecated features > =================== > > The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality > will be replaced by the new "density" keyword. > > > Removed features > ================ > > ``numpy.fft`` > ------------- > > The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, > which were aliases for the same functions without the 'e' in the name, were > removed. > > > ``numpy.memmap`` > ---------------- > > The `sync()` and `close()` methods of memmap were removed. ?Use `flush()` and > "del memmap" instead. > > > ``numpy.lib`` > ------------- > > The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, > ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. > > > ``numpy.ma`` > ------------ > > Several deprecated items were removed from the ``numpy.ma`` module:: > > ?* ``numpy.ma.MaskedArray`` "raw_data" method > ?* ``numpy.ma.MaskedArray`` constructor "flag" keyword > ?* ``numpy.ma.make_mask`` "flag" keyword > ?* ``numpy.ma.allclose`` "fill_value" keyword > > > ``numpy.distutils`` > ------------------- > > The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include`` > instead. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Tue Apr 5 16:12:59 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 5 Apr 2011 16:12:59 -0400 Subject: [SciPy-User] Probability Density Estimation In-Reply-To: <20110405111054.GA2196@macwilliams.local> References: <20110405111054.GA2196@macwilliams.local> Message-ID: On Tue, Apr 5, 2011 at 7:10 AM, Hans Georg Schaathun wrote: > Hi, > > does anyone here have any experience with probability density > estimation? > > I have found the scipy.stats.gaussian_kde, but I find that > it is extremely sensitive to outliers. ?I have datasets which > tend to have outliers at up to about 35 standard deviations > in a sample of 500-1000. ?The PDF estimate then turns out to > be close to uniform, and not at all useful. > > While I would be grateful for any pointers and advice on the > numerical and algorithmic sides of my problem, this is a scipy > list after all. ?The scipy question is this: Are there other, > non-Gaussian KDE-s available, that I have missed? ?Or even > KDE-s which allows the bandwidth to be specified precisely? There is a univariate kde in statsmodels that allows for a larger choice of kernels, it's still work in progress, http://bazaar.launchpad.net/~m-j-a-crowe/statsmodels/mike-branch/files/head:/scikits/statsmodels/sandbox/nonparametric/ Here is a recipe how to subclass scipy.stats.gaussian_kde to set the bandwidth manually: http://mail.scipy.org/pipermail/scipy-user/2010-January/023877.html (I have misplaced the file right now, which happens twice a year.) As alternative, if they are really outliers, then you could try to identify them and remove them from the dataset before running the kde. But maybe they are not outliers and you have a distribution with heavy tails. Another alternative would be to try to estimate a mixture model, where one of the mixture might capture the outliers. Josef > > TIA > -- > :-- Hans Georg > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From zachary.pincus at yale.edu Tue Apr 5 16:18:26 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Tue, 5 Apr 2011 16:18:26 -0400 Subject: [SciPy-User] Probability Density Estimation In-Reply-To: References: <20110405111054.GA2196@macwilliams.local> Message-ID: <0BAEA311-2673-4BCC-A39E-0657536BE90E@yale.edu> On Apr 5, 2011, at 4:12 PM, josef.pktd at gmail.com wrote: > Here is a recipe how to subclass scipy.stats.gaussian_kde to set the > bandwidth manually: > > http://mail.scipy.org/pipermail/scipy-user/2010-January/023877.html > (I have misplaced the file right now, which happens twice a year.) > > As alternative, if they are really outliers, then you could try to > identify them and remove them from the dataset before running the kde. > But maybe they are not outliers and you have a distribution with heavy > tails. Or, better, remove them from the dataset before calculating the bandwidth, but add them back for the actual density estimation. Or (effectively the same procedure), substitute in a robust covariance estimator for the calls to numpy.cov (or whatever it is in there) -- look e.g. at the MCD method. (Very easy in 1D -- I have code for that special case but not the general case.) Zach From fperez.net at gmail.com Wed Apr 6 03:26:07 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 6 Apr 2011 00:26:07 -0700 Subject: [SciPy-User] Scientific Python at SIAM CSE 2011 conference Message-ID: Hi all, sorry for the massive cross-post, but since all these projects were highlighted with talks at this event, I figured there would be interest... Hans-Petter Langtangen, Randy LeVeque and I organized a set of Python-focused sessions at the recent SIAM Computational Science and Engineering conference, with talks on numpy/scipy, cython, matplotlib, ipython, sympy, as well as application-oriented talks on astronomy and femhub. For those interested: - The slides: http://fperez.org/events/2011_siam_cse/ - A blog post: http://blog.fperez.org/2011/04/python-goes-to-reno-siam-cse-2011.html - Some pictures: https://picasaweb.google.com/fdo.perez/SIAMCSE2011InReno# Back to being quiet... f From Pierre.RAYBAUT at CEA.FR Wed Apr 6 05:34:57 2011 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Wed, 6 Apr 2011 11:34:57 +0200 Subject: [SciPy-User] [ANN] guidata v1.3.0 Message-ID: Hi all, I am pleased to announce that `guidata` v1.3.0 has been released. Note that the project has recently been moved to GoogleCode: http://guidata.googlecode.com The `guidata` documentation with examples, API reference, etc. is available here: http://packages.python.org/guidata/ Based on the Qt Python binding module PyQt4, guidata is a Python library generating graphical user interfaces for easy dataset editing and display. It also provides helpers and application development tools for PyQt4. guidata also provides the following features: * guidata.qthelpers: PyQt4 helpers * guidata.disthelpers: py2exe helpers * guidata.userconfig: .ini configuration management helpers (based on Python standard module ConfigParser) * guidata.configtools: library/application data management * guidata.gettext_helpers: translation helpers (based on the GNU tool gettext) * guidata.guitest: automatic GUI-based test launcher * guidata.utils: miscelleneous utilities guidata has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guidata/ Documentation, screenshots: http://packages.python.org/guidata/ Downloads (source + Python(x,y) plugin): http://guidata.googlecode.com Cheers, Pierre --- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From Pierre.RAYBAUT at CEA.FR Wed Apr 6 05:38:59 2011 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Wed, 6 Apr 2011 11:38:59 +0200 Subject: [SciPy-User] [ANN] guiqwt v2.1.0 Message-ID: Hi all, I am pleased to announce that `guiqwt` v2.1.0 has been released. Note that the project has recently been moved to GoogleCode: http://guiqwt.googlecode.com This version of `guiqwt` includes a demo software, Sift (for Signal and Image Filtering Tool), based on `guidata` and `guiqwt`: http://packages.python.org/guiqwt/sift.html Windows users may even download the portable version of Sift 0.22 to test it without having to install anything: http://code.google.com/p/guiqwt/downloads/detail?name=sift022_portable.zip The `guiqwt` documentation with examples, API reference, etc. is available here: http://packages.python.org/guiqwt/ Based on PyQwt (plotting widgets for PyQt4 graphical user interfaces) and on the scientific modules NumPy and SciPy, guiqwt is a Python library providing efficient 2D data-plotting features (curve/image visualization and related tools) for interactive computing and signal/image processing application development. When compared to the excellent module `matplotlib`, the main advantage of `guiqwt` is performance: see http://packages.python.org/guiqwt/overview.html#performances. But `guiqwt` is more than a plotting library; it also provides: * Helper functions for data processing: see the example http://packages.python.org/guiqwt/examples.html#curve-fitting * Framework for signal/image processing application development: see http://packages.python.org/guiqwt/examples.html * And many other features like making executable Windows programs easily (py2exe helpers): see http://packages.python.org/guiqwt/disthelpers.html guiqwt plotting features are the following: guiqwt.pyplot: equivalent to matplotlib's pyplot module (pylab) supported plot items: * curves, error bar curves and 1-D histograms * images (RGB images are not supported), images with non-linear x/y scales, images with specified pixel size (e.g. loaded from DICOM files), 2-D histograms, pseudo-color images (pcolor) * labels, curve plot legends * shapes: polygon, polylines, rectangle, circle, ellipse and segment * annotated shapes (shapes with labels showing position and dimensions): rectangle with center position and size, circle with center position and diameter, ellipse with center position and diameters (these items are very useful to measure things directly on displayed images) curves, images and shapes: * multiple object selection for moving objects or editing their properties through automatically generated dialog boxes (guidata) * item list panel: move objects from foreground to background, show/hide objects, remove objects, ... * customizable aspect ratio * a lot of ready-to-use tools: plot canvas export to image file, image snapshot, image rectangular filter, etc. curves: * interval selection tools with labels showing results of computing on selected area * curve fitting tool with automatic fit, manual fit with sliders, ... images: * contrast adjustment panel: select the LUT by moving a range selection object on the image levels histogram, eliminate outliers, ... * X-axis and Y-axis cross-sections: support for multiple images, average cross-section tool on a rectangular area, ... * apply any affine transform to displayed images in real-time (rotation, magnification, translation, horizontal/vertical flip, ...) application development helpers: * ready-to-use curve and image plot widgets and dialog boxes * load/save graphical objects (curves, images, shapes) * a lot of test scripts which demonstrate guiqwt features guiqwt has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guiqwt/ Documentation, screenshots: http://packages.python.org/guiqwt/ Downloads (source + Python(x,y) plugin): http://guiqwt.googlecode.com Cheers, Pierre --- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From lists at hilboll.de Wed Apr 6 06:41:24 2011 From: lists at hilboll.de (Andreas) Date: Wed, 06 Apr 2011 12:41:24 +0200 Subject: [SciPy-User] how to use kendalltau_seasonal Message-ID: <4D9C4354.5080100@hilboll.de> Hi there, I'm a bit at a loss how to use scipy.stats.mstats.kendalltau_seasonal. The docstring and the online documentation aren't really that helpful about what input and output mean ... Thanks for any insight! Cheers, Andreas. From hg+scipy at schaathun.net Wed Apr 6 07:29:47 2011 From: hg+scipy at schaathun.net (Hans Georg Schaathun) Date: Wed, 6 Apr 2011 13:29:47 +0200 Subject: [SciPy-User] Probability Density Estimation Message-ID: <20110406112947.GA1994@macwilliams.local> [josef.pktd at gmail.com] > There is a univariate kde in statsmodels that allows for a larger > choice of kernels, it's still work in progress, Thanks. Unfortunately, I will need multivariate. [Zachary Pincus] > Or, better, remove them from the dataset before calculating the > bandwidth, but add them back for the actual density estimation. Or > (effectively the same procedure), substitute in a robust covariance > estimator for the calls to numpy.cov (or whatever it is in there) -- > look e.g. at the MCD method. (Very easy in 1D -- I have code for that > special case but not the general case.) That sounds like a very good idea. It is a pity that the gaussian_kde is not made with a good API to do that sort of things, but with a bit of reverse engineering and the pointer below, I am sure I shall manage. Thank you very much, both of you. There is little doubt that the distribution has heavy (or at least long) tails combined with an extreme peak around the mean. What I am actually trying to do is to estimate differential mutual information (between a multivariate continuous distribution (feature space) and a boolean classification label for machine learning). This means that I have two samples which should have rather similar distributions. The outlier at 35 standard deviations will likely shows up only in one of the two, but it is caused by actual heavy tails which should exist in both distributions. The result is that the two estimated distributions differ more than they should. [josef.pktd at gmail.com] > Here is a recipe how to subclass scipy.stats.gaussian_kde to set the > bandwidth manually: > > http://mail.scipy.org/pipermail/scipy-user/2010-January/023877.html > (I have misplaced the file right now, which happens twice a year.) -- :-- Hans Georg From pgmdevlist at gmail.com Wed Apr 6 08:01:27 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 6 Apr 2011 14:01:27 +0200 Subject: [SciPy-User] how to use kendalltau_seasonal In-Reply-To: <4D9C4354.5080100@hilboll.de> References: <4D9C4354.5080100@hilboll.de> Message-ID: <83F2D73D-2775-47A6-A369-D07E6F2EE45C@gmail.com> On Apr 6, 2011, at 12:41 PM, Andreas wrote: > Hi there, > I'm a bit at a loss how to use scipy.stats.mstats.kendalltau_seasonal. > The docstring and the online documentation aren't really that helpful > about what input and output mean ... Somebody was too busy coding to pay attention to doc writing... Imagine you need to check whether there's a trend in a time series X, but want to take a potential seasonal dependence into account. Reshape your data so that it's 2D, with seasons as columns, and feed it to the function. You'll get a dictionary giving you the value of Kendall's tau (w/ and w/o the seasonal effects) with the corresponding p-values that should help you figure out without there's a trend in your data and whether seasonality plays a role. Example: You have a 60-element array corresponding to 5 years of consecutive monthly data, and want to check the trend, taking potential monthly variations into account. In that case, your "season" is a month. Reshape it as a (5,12), and use the result as input to the function. More info: http://pubs.usgs.gov/sir/2005/5275/pdf/sir2005-5275.pdf Hirsch R.M., Slack J.R. and Smith R.A. (1982), Techniques of Trend Analysis for Monthly Water Quality Data, Water Resources Research 18(1), 107-12 I'm sure there must be a R equivalent somewhere. In any case, please do help to document the function as needed. Don't hesitate to contact me if you need more info. Cheers P. From robert.kern at gmail.com Wed Apr 6 09:49:52 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 6 Apr 2011 08:49:52 -0500 Subject: [SciPy-User] Probability Density Estimation In-Reply-To: <20110406112947.GA1994@macwilliams.local> References: <20110406112947.GA1994@macwilliams.local> Message-ID: On Wed, Apr 6, 2011 at 06:29, Hans Georg Schaathun wrote: > That sounds like a very good idea. ?It is a pity that the gaussian_kde > is not made with a good API to do that sort of things, but with a bit > of reverse engineering and the pointer below, I am sure I shall manage. We've invited people to fix it, but no one has ever bothered. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Wed Apr 6 11:47:26 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 6 Apr 2011 11:47:26 -0400 Subject: [SciPy-User] Probability Density Estimation In-Reply-To: <20110406112947.GA1994@macwilliams.local> References: <20110406112947.GA1994@macwilliams.local> Message-ID: On Wed, Apr 6, 2011 at 7:29 AM, Hans Georg Schaathun wrote: > [josef.pktd at gmail.com] >> There is a univariate kde in statsmodels that allows for a larger >> choice of kernels, it's still work in progress, > > Thanks. ?Unfortunately, I will need multivariate. > > [Zachary Pincus] >> Or, better, remove them from the dataset before calculating the >> bandwidth, but add them back for the actual density estimation. Or >> (effectively the same procedure), substitute in a robust covariance >> estimator for the calls to numpy.cov (or whatever it is in there) -- >> look e.g. at the MCD method. (Very easy in 1D -- I have code for that >> special case but not the general case.) > > That sounds like a very good idea. ?It is a pity that the gaussian_kde > is not made with a good API to do that sort of things, but with a bit > of reverse engineering and the pointer below, I am sure I shall manage. > Thank you very much, both of you. > > There is little doubt that the distribution has heavy (or at least long) > tails combined with an extreme peak around the mean. > > What I am actually trying to do is to estimate differential mutual > information (between a multivariate continuous distribution (feature space) > and a boolean classification label for machine learning). Sounds a bit familiar, but with different usage in mind http://bazaar.launchpad.net/~scipystats/statsmodels/devel/view/head:/scikits/statsmodels/sandbox/distributions/mv_measures.py and the bug report against myself https://bugs.launchpad.net/statsmodels/+bug/717511 (see description) because I also needed more flexible kde. If you have or find some results and are willing to share, then I will be very interested. My objective was measures for general (non-linear) dependence between two random variables, and tests for independence. Josef > This means that I have two samples which should have rather similar > distributions. ?The outlier at 35 standard deviations will likely > shows up only in one of the two, but it is caused by actual heavy > tails which should exist in both distributions. ?The result is that the > two estimated distributions differ more than they should. > > [josef.pktd at gmail.com] >> Here is a recipe how to subclass scipy.stats.gaussian_kde to set the >> bandwidth manually: >> >> http://mail.scipy.org/pipermail/scipy-user/2010-January/023877.html >> (I have misplaced the file right now, which happens twice a year.) > > -- > :-- Hans Georg > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From jason.roberts at duke.edu Wed Apr 6 15:41:04 2011 From: jason.roberts at duke.edu (Jason Roberts) Date: Wed, 6 Apr 2011 15:41:04 -0400 Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image Message-ID: <02cd01cbf492$919b2ea0$b4d18be0$@roberts@duke.edu> I have a 2D image labeled with scipy.ndimage.label and a list of integers that identify features I would like to eliminate (set to 0), or keep (set the others to 0). I can think of several ways to do this (nested loops, use generic_filter, etc.). Some techniques might provide better performance than others, depending on the size of the image, the number of features to eliminate/keep, etc. Is there a technique that provides good performance across a wide range of scenarios? This seems like one of those problems where the answer should be obvious-there should be some function specifically for it-but I have not come across it yet. Thanks for any advice you can provide, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.pincus at yale.edu Wed Apr 6 17:40:03 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 6 Apr 2011 17:40:03 -0400 Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image In-Reply-To: <02cd01cbf492$919b2ea0$b4d18be0$@roberts@duke.edu> References: <02cd01cbf492$919b2ea0$b4d18be0$@roberts@duke.edu> Message-ID: > I have a 2D image labeled with scipy.ndimage.label and a list of > integers that identify features I would like to eliminate (set to > 0), or keep (set the others to 0). I can think of several ways to do > this (nested loops, use generic_filter, etc.). Some techniques might > provide better performance than others, depending on the size of the > image, the number of features to eliminate/keep, etc. Is there a > technique that provides good performance across a wide range of > scenarios? > > This seems like one of those problems where the answer should be > obvious?there should be some function specifically for it?but I have > not come across it yet. What I usually do in this situation is to make and then or (or and, as required) together multiple boolean masks. Say I want to pick out the regions of arr that are either 1- or 2-valued: mask = (arr == 1) | (arr == 2) arr *= mask This obviously scales badly when there are many values one wants to pick out. I don't know of any numpy function that compiles a boolean mask with "values-in" sort of logic, (e.g. mask = values_in(arr, [1,2]) ) but there might be some way to achieve this functionality. Alternately, that would be very simple to implement in cython as an ad- hoc solution. (And it could be extremely fast if the dtype of the input array were limited to 16-bit or something so a lookup-table approach to doing the "in" test would suffice, instead of having to iterate through the list at each array element.) Zach From zachary.pincus at yale.edu Wed Apr 6 17:52:09 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 6 Apr 2011 17:52:09 -0400 Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image In-Reply-To: References: <02cd01cbf492$919b2ea0$b4d18be0$@roberts@duke.edu> Message-ID: <2EE5BAED-9DA2-45FC-80AB-83C83A25D110@yale.edu> On Apr 6, 2011, at 5:40 PM, Zachary Pincus wrote: >> I have a 2D image labeled with scipy.ndimage.label and a list of >> integers that identify features I would like to eliminate (set to >> 0), or keep (set the others to 0). I can think of several ways to do >> this (nested loops, use generic_filter, etc.). Some techniques might >> provide better performance than others, depending on the size of the >> image, the number of features to eliminate/keep, etc. Is there a >> technique that provides good performance across a wide range of >> scenarios? >> >> This seems like one of those problems where the answer should be >> obvious?there should be some function specifically for it?but I have >> not come across it yet. > > What I usually do in this situation is to make and then or (or and, as > required) together multiple boolean masks. Say I want to pick out the > regions of arr that are either 1- or 2-valued: > > mask = (arr == 1) | (arr == 2) > arr *= mask > > This obviously scales badly when there are many values one wants to > pick out. I don't know of any numpy function that compiles a boolean > mask with "values-in" sort of logic, (e.g. mask = values_in(arr, > [1,2]) ) but there might be some way to achieve this functionality. > Alternately, that would be very simple to implement in cython as an > ad- > hoc solution. (And it could be extremely fast if the dtype of the > input array were limited to 16-bit or something so a lookup-table > approach to doing the "in" test would suffice, instead of having to > iterate through the list at each array element.) A second's thought suggests that numpy.choose is perfect here! a = numpy.arange(16).reshape((4,4)) In [71]: a Out[71]: array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15]]) In [72]: a.choose([0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1]) Out[72]: array([[0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1]]) Basically, as input to choose, you pass an array that's as long as the maximum label, and has ones at the indices of the labels you want to keep, and zeros elsewhwere. Given a list of the labels you want, it's easy to make such an array with fancy indexing: In [76]: z = numpy.zeros(16, dtype=int) In [77]: z[[4,6,2]] = 1 In [78]: z Out[78]: array([0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]) In [79]: a.choose(z) Out[79]: array([[0, 0, 1, 0], [1, 0, 1, 0], [0, 0, 0, 0], [0, 0, 0, 0]]) so to keep labels with values from a list: keep_labels = [1,2,5] choices = numpy.zeros(array.max(), dtype=int) choices[keep_labels] = 1 array *= array.choose(choices) There might be even more compact ways of generating the choices array... Zach From jason.roberts at duke.edu Wed Apr 6 19:37:42 2011 From: jason.roberts at duke.edu (Jason Roberts) Date: Wed, 6 Apr 2011 19:37:42 -0400 Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image In-Reply-To: <2EE5BAED-9DA2-45FC-80AB-83C83A25D110@yale.edu> References: <02cd01cbf492$919b2ea0$b4d18be0$%roberts@duke.edu> <2EE5BAED-9DA2-45FC-80AB-83C83A25D110@yale.edu> Message-ID: <02ea01cbf4b3$a00003f0$e0000bd0$@roberts@duke.edu> > Basically, as input to choose, you pass an array that's as long as the > maximum label, and has ones at the indices of the labels you want to > keep, and zeros elsewhwere. Zach, Thanks for your suggestions! I did not realize that numpy.choose could be used that way, and that approach works when there are 32 or less choices (i.e. features to keep). Unfortunately numpy.choose is limited to 32 or less and I often have hundreds. Any other ideas? If not, I guess I will roll my own C or Cython function for this. This "element-wise a in b" functionality seems like it would be common enough that numpy would include it, but I guess not... Best, Jason From david_baddeley at yahoo.com.au Wed Apr 6 19:50:42 2011 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Wed, 6 Apr 2011 16:50:42 -0700 (PDT) Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image In-Reply-To: <02ea01cbf4b3$a00003f0$e0000bd0$@roberts@duke.edu> References: <02cd01cbf492$919b2ea0$b4d18be0$%roberts@duke.edu> <2EE5BAED-9DA2-45FC-80AB-83C83A25D110@yale.edu> <02ea01cbf4b3$a00003f0$e0000bd0$@roberts@duke.edu> Message-ID: <819255.40588.qm@web113420.mail.gq1.yahoo.com> I've just tried np.choose with 1000 choices, and it seemed to work ... the documentation mentions nothing about a limit of 32. cheers, David ----- Original Message ---- From: Jason Roberts To: SciPy Users List Sent: Thu, 7 April, 2011 11:37:42 AM Subject: Re: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image > Basically, as input to choose, you pass an array that's as long as the > maximum label, and has ones at the indices of the labels you want to > keep, and zeros elsewhwere. Zach, Thanks for your suggestions! I did not realize that numpy.choose could be used that way, and that approach works when there are 32 or less choices (i.e. features to keep). Unfortunately numpy.choose is limited to 32 or less and I often have hundreds. Any other ideas? If not, I guess I will roll my own C or Cython function for this. This "element-wise a in b" functionality seems like it would be common enough that numpy would include it, but I guess not... Best, Jason _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From zachary.pincus at yale.edu Wed Apr 6 19:54:34 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 6 Apr 2011 19:54:34 -0400 Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image In-Reply-To: <02ea01cbf4b3$a00003f0$e0000bd0$@roberts@duke.edu> References: <02cd01cbf492$919b2ea0$b4d18be0$%roberts@duke.edu> <2EE5BAED-9DA2-45FC-80AB-83C83A25D110@yale.edu> <02ea01cbf4b3$a00003f0$e0000bd0$@roberts@duke.edu> Message-ID: <2F7DE870-D19D-4C4A-AE7C-6D79963C247D@yale.edu> >> Basically, as input to choose, you pass an array that's as long as >> the >> maximum label, and has ones at the indices of the labels you want to >> keep, and zeros elsewhwere. > > Zach, > > Thanks for your suggestions! I did not realize that numpy.choose > could be > used that way, and that approach works when there are 32 or less > choices > (i.e. features to keep). Unfortunately numpy.choose is limited to 32 > or less > and I often have hundreds. Ugh! Not documented and I sadly did not test the upper limits of this case. However, this reminded me that in simple cases, fancy indexing works just as choose does. So another second's puzzling gives this solution, which I think is pretty clever: In [93]: a = numpy.arange(400).reshape((20,20)) In [94]: keep = numpy.zeros(400, dtype=int) In [95]: keep[128+numpy.arange(64)] = 1 In [96]: mask = keep[a] Fancy indexing never ceases to amaze. Perfect for this sort of look-up table task. From zachary.pincus at yale.edu Wed Apr 6 19:56:55 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 6 Apr 2011 19:56:55 -0400 Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image In-Reply-To: <819255.40588.qm@web113420.mail.gq1.yahoo.com> References: <02cd01cbf492$919b2ea0$b4d18be0$%roberts@duke.edu> <2EE5BAED-9DA2-45FC-80AB-83C83A25D110@yale.edu> <02ea01cbf4b3$a00003f0$e0000bd0$@roberts@duke.edu> <819255.40588.qm@web113420.mail.gq1.yahoo.com> Message-ID: > I've just tried np.choose with 1000 choices, and it seemed to > work ... the > documentation mentions nothing about a limit of 32. > > cheers, > David > Huh... In [82]: a = numpy.arange(400).reshape((20,20)) In [83]: keep = numpy.zeros(400, dtype=int) In [84]: keep[numpy.arange(64)] = 1 In [86]: a.choose(keep) ValueError: Need between 2 and (32) array objects (inclusive). In [99]: numpy.version.version Out[99]: '2.0.0.dev8611' Possibly the problem is fixed in newer numpys? Nevertheless, the fancy-indexing solution I sent a second ago is even better for this particular problem. Zach From jason.roberts at duke.edu Wed Apr 6 20:07:09 2011 From: jason.roberts at duke.edu (Jason Roberts) Date: Wed, 6 Apr 2011 20:07:09 -0400 Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image In-Reply-To: <819255.40588.qm@web113420.mail.gq1.yahoo.com> References: <02cd01cbf492$919b2ea0$b4d18be0$%roberts@duke.edu> <2EE5BAED-9DA2-45FC-80AB-83C83A25D110@yale.edu> <02ea01cbf4b3$a00003f0$e0000bd0$%roberts@duke.edu> <819255.40588.qm@web113420.mail.gq1.yahoo.com> Message-ID: <02ee01cbf4b7$bd3327f0$379977d0$@roberts@duke.edu> Hi David, We must be trying something different. Here's an example that works for me, with feature IDs ranging up to 9: >>> a = numpy.array([[ 1, 0, 0, 0, 0], ... [ 1, 0, 2, 2, 2], ... [ 0, 0, 2, 2, 0], ... [ 0, 0, 0, 0, 0], ... [ 9, 9, 9, 0, 0]]) >>> choices = [0, 1, 1, 0, 0, 0, 0, 0, 0, 0] >>> a.choose(choices) array([[1, 0, 0, 0, 0], [1, 0, 1, 1, 1], [0, 0, 1, 1, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]]) Here's an example with feature IDs ranging up to 99: >>> a = numpy.array([[ 1, 0, 0, 0, 0], ... [ 1, 0, 2, 2, 2], ... [ 0, 0, 2, 2, 0], ... [ 0, 0, 0, 0, 0], ... [99, 99, 99, 0, 0]]) >>> choices = [0, 1, 1] + [0] * 97 >>> a.choose(choices) Traceback (most recent call last): File "", line 1, in ValueError: Need between 2 and (32) array objects (inclusive). Is it a numpy version problem? I'm using 1.4.1 on this particular machine. I know that's kind of dated, but the choice function has been in numpy forever so I'm guessing it hasn't changed... Jason -----Original Message----- From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of David Baddeley Sent: Wednesday, April 06, 2011 7:51 PM To: SciPy Users List Subject: Re: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image I've just tried np.choose with 1000 choices, and it seemed to work ... the documentation mentions nothing about a limit of 32. cheers, David ----- Original Message ---- From: Jason Roberts To: SciPy Users List Sent: Thu, 7 April, 2011 11:37:42 AM Subject: Re: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image > Basically, as input to choose, you pass an array that's as long as the > maximum label, and has ones at the indices of the labels you want to > keep, and zeros elsewhwere. Zach, Thanks for your suggestions! I did not realize that numpy.choose could be used that way, and that approach works when there are 32 or less choices (i.e. features to keep). Unfortunately numpy.choose is limited to 32 or less and I often have hundreds. Any other ideas? If not, I guess I will roll my own C or Cython function for this. This "element-wise a in b" functionality seems like it would be common enough that numpy would include it, but I guess not... Best, Jason _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From david_baddeley at yahoo.com.au Wed Apr 6 20:14:03 2011 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Wed, 6 Apr 2011 17:14:03 -0700 (PDT) Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image In-Reply-To: References: <02cd01cbf492$919b2ea0$b4d18be0$%roberts@duke.edu> <2EE5BAED-9DA2-45FC-80AB-83C83A25D110@yale.edu> <02ea01cbf4b3$a00003f0$e0000bd0$@roberts@duke.edu> <819255.40588.qm@web113420.mail.gq1.yahoo.com> Message-ID: <961679.52391.qm@web113416.mail.gq1.yahoo.com> might be a regression ... In [52]: i = arange(4000).reshape((20,200)) In [53]: b = zeros(4000) In [54]: b[numpy.arange(300)] = 1 In [55]: i.choose(b) Out[55]: array([[ 1., 1., 1., ..., 1., 1., 1.], [ 1., 1., 1., ..., 0., 0., 0.], [ 0., 0., 0., ..., 0., 0., 0.], ..., [ 0., 0., 0., ..., 0., 0., 0.], [ 0., 0., 0., ..., 0., 0., 0.], [ 0., 0., 0., ..., 0., 0., 0.]]) In [56]: np.version.version Out[56]: '1.2.1' anyone know if there's any rationale behind this? cheers, David ----- Original Message ---- From: Zachary Pincus To: SciPy Users List Sent: Thu, 7 April, 2011 11:56:55 AM Subject: Re: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image > I've just tried np.choose with 1000 choices, and it seemed to > work ... the > documentation mentions nothing about a limit of 32. > > cheers, > David > Huh... In [82]: a = numpy.arange(400).reshape((20,20)) In [83]: keep = numpy.zeros(400, dtype=int) In [84]: keep[numpy.arange(64)] = 1 In [86]: a.choose(keep) ValueError: Need between 2 and (32) array objects (inclusive). In [99]: numpy.version.version Out[99]: '2.0.0.dev8611' Possibly the problem is fixed in newer numpys? Nevertheless, the fancy-indexing solution I sent a second ago is even better for this particular problem. Zach _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From jason.roberts at duke.edu Wed Apr 6 20:23:01 2011 From: jason.roberts at duke.edu (Jason Roberts) Date: Wed, 6 Apr 2011 20:23:01 -0400 Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image In-Reply-To: <2F7DE870-D19D-4C4A-AE7C-6D79963C247D@yale.edu> References: <02cd01cbf492$919b2ea0$b4d18be0$%roberts@duke.edu> <2EE5BAED-9DA2-45FC-80AB-83C83A25D110@yale.edu> <02ea01cbf4b3$a00003f0$e0000bd0$%roberts@duke.edu> <2F7DE870-D19D-4C4A-AE7C-6D79963C247D@yale.edu> Message-ID: <02ef01cbf4b9$f4c68890$de5399b0$@roberts@duke.edu> Zach, Nicely done! I obviously did not think if doing it that way, but now that I've seen what you did, I understand how it would work. Jason -----Original Message----- From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of Zachary Pincus Sent: Wednesday, April 06, 2011 7:55 PM To: SciPy Users List Subject: Re: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image >> Basically, as input to choose, you pass an array that's as long as >> the >> maximum label, and has ones at the indices of the labels you want to >> keep, and zeros elsewhwere. > > Zach, > > Thanks for your suggestions! I did not realize that numpy.choose > could be > used that way, and that approach works when there are 32 or less > choices > (i.e. features to keep). Unfortunately numpy.choose is limited to 32 > or less > and I often have hundreds. Ugh! Not documented and I sadly did not test the upper limits of this case. However, this reminded me that in simple cases, fancy indexing works just as choose does. So another second's puzzling gives this solution, which I think is pretty clever: In [93]: a = numpy.arange(400).reshape((20,20)) In [94]: keep = numpy.zeros(400, dtype=int) In [95]: keep[128+numpy.arange(64)] = 1 In [96]: mask = keep[a] Fancy indexing never ceases to amaze. Perfect for this sort of look-up table task. _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From jturner at gemini.edu Thu Apr 7 12:00:35 2011 From: jturner at gemini.edu (James Turner) Date: Thu, 7 Apr 2011 13:00:35 -0300 Subject: [SciPy-User] Seg. fault from scipy.interpolate or numpy In-Reply-To: References: <4D8CF861.9030302@gemini.edu> Message-ID: <4D9DDFA3.8070400@gemini.edu> Sorry, I didn't see this right away. Thanks for the reply. On 28/03/11 11:23, Bruce Southey wrote: > [snip] > [I did see the ticket comments and it doesn't crash for me on Linux > and Python2.7.] > > Why do you use 'input' for as the variable name? > This could be a problem because input is a builtin function. If the > error goes away then it is most likely a Python bug. Oh, I hadn't noticed that. That's unfortunate. Changing it doesn't make any difference though. > Second, is there a reason for using copy() for 'objfit' instead of one of these? > zeros_like : Return an array of zeros with shape and type of input. > ones_like : Return an array of ones with shape and type of input. > empty_like : Return an empty array with shape and type of input. > ones : Return a new array setting values to one. > empty : Return a new uninitialized array. Not really. I was partly inheriting what someone else had done. Thanks for the tip. Are these faster? I have changed it to zeros_like, without affecting the segmentation fault. > So can you determine exactly which line it crashes on (especially the > value yc) and the shapes of the arrays? Yes, it crashes on the last line, beginning "noise=", not in the loop. Good question about yc; it's value is 127, which is correct, as the test array has a shape of (128, 2048). I just tried splitting up the last line into separate operations and the one that crashes is objfit.clip(min=0.0). If I replace the last line with just that call, I still get the crash. I thought I'd try printing objfit.min() and max() just for fun and those work without crashing, returning -185.079 and 711.543. Thanks, James. From frank.astier at gmail.com Thu Apr 7 12:08:11 2011 From: frank.astier at gmail.com (Astier Frank) Date: Thu, 7 Apr 2011 09:08:11 -0700 Subject: [SciPy-User] Error when trying to use scipy.optimize.fmin Message-ID: <9DA5F99B-7FE7-4960-A3AF-6D3D66D7CBF7@gmail.com> Hi - I'm trying to use fmin from scipy.optimize, and I get the following error. My machine is a MBP with MacOSX 10.6.7. I'm running the example below with the system's Python 2.6.1, which is a universal binary supporting 3 architectures (see below). I got scipy from the superpack maintained by Cristopher Fonnesbeck, who couldn't reproduce and has never seen that for any other user... Any idea?? frank-2|506|/Library/Python/2.6/site-packages>file /usr/bin/python /usr/bin/python: Mach-O universal binary with 3 architectures /usr/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64 /usr/bin/python (for architecture i386): Mach-O executable i386 /usr/bin/python (for architecture ppc7400): Mach-O executable ppc frank-2|505|/Library/Python/2.6/site-packages>/usr/bin/python Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from scipy.optimize import fmin Traceback (most recent call last): File "", line 1, in File "/Library/Python/2.6/site-packages/scipy-0.10.0.dev_20110407-py2.6-macosx-10.6-universal.egg/scipy/optimize/__init__.py", line 7, in from optimize import * File "/Library/Python/2.6/site-packages/scipy-0.10.0.dev_20110407-py2.6-macosx-10.6-universal.egg/scipy/optimize/optimize.py", line 28, in from linesearch import \ File "/Library/Python/2.6/site-packages/scipy-0.10.0.dev_20110407-py2.6-macosx-10.6-universal.egg/scipy/optimize/linesearch.py", line 1, in from scipy.optimize import minpack2 ImportError: dlopen(/Library/Python/2.6/site-packages/scipy-0.10.0.dev_20110407-py2.6-macosx-10.6-universal.egg/scipy/optimize/minpack2.so, 2): Symbol not found: _dcsrch_ Referenced from: /Library/Python/2.6/site-packages/scipy-0.10.0.dev_20110407-py2.6-macosx-10.6-universal.egg/scipy/optimize/minpack2.so Expected in: flat namespace in /Library/Python/2.6/site-packages/scipy-0.10.0.dev_20110407-py2.6-macosx-10.6-universal.egg/scipy/optimize/minpack2.so >>> Frank From bsouthey at gmail.com Thu Apr 7 12:33:35 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 07 Apr 2011 11:33:35 -0500 Subject: [SciPy-User] Seg. fault from scipy.interpolate or numpy In-Reply-To: <4D9DDFA3.8070400@gemini.edu> References: <4D8CF861.9030302@gemini.edu> <4D9DDFA3.8070400@gemini.edu> Message-ID: <4D9DE75F.9030203@gmail.com> On 04/07/2011 11:00 AM, James Turner wrote: > Sorry, I didn't see this right away. Thanks for the reply. > > On 28/03/11 11:23, Bruce Southey wrote: >> [snip] >> [I did see the ticket comments and it doesn't crash for me on Linux >> and Python2.7.] >> >> Why do you use 'input' for as the variable name? >> This could be a problem because input is a builtin function. If the >> error goes away then it is most likely a Python bug. > > Oh, I hadn't noticed that. That's unfortunate. Changing it doesn't > make any difference though. > >> Second, is there a reason for using copy() for 'objfit' instead of >> one of these? >> zeros_like : Return an array of zeros with shape and type of input. >> ones_like : Return an array of ones with shape and type of input. >> empty_like : Return an empty array with shape and type of input. >> ones : Return a new array setting values to one. >> empty : Return a new uninitialized array. > > Not really. I was partly inheriting what someone else had done. > Thanks for the tip. Are these faster? I have changed it to zeros_like, > without affecting the segmentation fault. > >> So can you determine exactly which line it crashes on (especially the >> value yc) and the shapes of the arrays? > > Yes, it crashes on the last line, beginning "noise=", not in the loop. > > Good question about yc; it's value is 127, which is correct, as the > test array has a shape of (128, 2048). > > I just tried splitting up the last line into separate operations and > the one that crashes is objfit.clip(min=0.0). If I replace the last > line with just that call, I still get the crash. > > I thought I'd try printing objfit.min() and max() just for fun and > those work without crashing, returning -185.079 and 711.543. > > Thanks, > > James. Good! I think that you should be using 'a_min' not 'min' an the argument to an ndarray: clip(...) a.clip(a_min, a_max, out=None) Also you could try using np.clip(objfit, 0.0) as it should be the same as objfit.clip(0.0). But I am still curious is why it is crashing on your system. The ticket says Python 2.5 on a 64-bit Linux but your backtrace includes 'i686' which suggests that it is 32-bit. Can you be more specific of the versions of the software regarding 32 vs 64 bit? Bruce From pierre.puiseux at univ-pau.fr Thu Apr 7 16:22:08 2011 From: pierre.puiseux at univ-pau.fr (Puiseux) Date: Thu, 07 Apr 2011 22:22:08 +0200 Subject: [SciPy-User] Write binary file from python and read from C++ Message-ID: <4D9E1CF0.9050306@univ-pau.fr> Hello, I have a simple problem and could not find the solution in scipy/python doc. Maybe i did not search in the good place... I have some numpy.array (float32), and i want to write it in a binary file with my own format. This file has to be readable by a C++ program. Thats all. Thanks for your answer From emmanuelle.gouillart at normalesup.org Thu Apr 7 16:27:42 2011 From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart) Date: Thu, 7 Apr 2011 22:27:42 +0200 Subject: [SciPy-User] Write binary file from python and read from C++ In-Reply-To: <4D9E1CF0.9050306@univ-pau.fr> References: <4D9E1CF0.9050306@univ-pau.fr> Message-ID: <20110407202742.GB19385@phare.normalesup.org> Hello Pierre, would the ndarray.tofile method suit your needs? >>> import numpy as np >>> a = np.arange(10.) >>> a.tofile('data.raw') Cheers, Emmanuelle On Thu, Apr 07, 2011 at 10:22:08PM +0200, Puiseux wrote: > Hello, > I have a simple problem and could not find the solution in scipy/python doc. > Maybe i did not search in the good place... > I have some numpy.array (float32), and i want to write it in a binary > file with my own format. > This file has to be readable by a C++ program. > Thats all. > Thanks for your answer > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From charlesr.harris at gmail.com Thu Apr 7 20:46:26 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 7 Apr 2011 18:46:26 -0600 Subject: [SciPy-User] Write binary file from python and read from C++ In-Reply-To: <4D9E1CF0.9050306@univ-pau.fr> References: <4D9E1CF0.9050306@univ-pau.fr> Message-ID: 2011/4/7 Puiseux > Hello, > > > I have a simple problem and could not find the solution in scipy/python > doc. > Maybe i did not search in the good place... > > I have some numpy.array (float32), and i want to write it in a binary > file with my own format. > What is your own format? > This file has to be readable by a C++ program. > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From 2huggie at gmail.com Fri Apr 8 06:14:12 2011 From: 2huggie at gmail.com (Timothy Wu) Date: Fri, 8 Apr 2011 18:14:12 +0800 Subject: [SciPy-User] What "Array" means Message-ID: Hi I am trying to run Scipy's D'Agostino's normality test as documented here http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.normaltest.html For the array argument I tried something like this scipy.array([1,2,3]) or numpy.array([1,2,3]) and axis ignored. But with both method the test fails: File "/usr/lib/python2.6/dist-packages/scipy/stats/mstats_basic.py", line 1546, in kurtosistest n = a.count(axis=axis).astype(float) AttributeError: 'int' object has no attribute 'astype' I'm not familiar with numpy nor scipy. What exactly should I put in there? Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Apr 8 06:45:14 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 8 Apr 2011 06:45:14 -0400 Subject: [SciPy-User] What "Array" means In-Reply-To: References: Message-ID: On Fri, Apr 8, 2011 at 6:14 AM, Timothy Wu <2huggie at gmail.com> wrote: > Hi I am trying to run Scipy's D'Agostino's normality test as documented here > http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.normaltest.html > > For the array argument I tried something like this > scipy.array([1,2,3]) > or > numpy.array([1,2,3]) > > and axis ignored. > > But with both method the test fails: > > File "/usr/lib/python2.6/dist-packages/scipy/stats/mstats_basic.py", line > 1546, in kurtosistest > ??? n = a.count(axis=axis).astype(float) > AttributeError: 'int' object has no attribute 'astype' > > I'm not familiar with numpy nor scipy. What exactly should I put in there? It looks like mstats.normaltest only works with 2-dimensional arrays, stats.normaltest works with 1-dimensional arrays. rvs[:,None] in the example below adds an additional axis, so that it is a column array with shape (20,1) If you don't need the masked array version, then you can use stats.normaltest I haven't looked at the source yet, but this looks like a bug to me. >>> rvs = np.random.randn(20) >>> rvs array([ 0.02724005, -0.17836266, 0.40530377, 1.313246 , 0.74069068, -0.69010129, -0.24958557, -2.28311759, 0.10525733, 0.07986322, -0.87282545, -1.41364294, 1.16027037, 0.23541801, -0.06663458, 0.39173207, 0.06979893, 0.4400277 , -1.29361117, -1.71524228]) >>> stats.normaltest(rvs) (1.7052869564079727, 0.42628656195988301) >>> stats.mstats.normaltest(rvs[:,None]) (masked_array(data = [1.70528695641], mask = [False], fill_value = 1e+20) , masked_array(data = [ 0.42628656], mask = False, fill_value = 1e+20) ) >>> stats.mstats.normaltest(rvs) Traceback (most recent call last): File "", line 1, in stats.mstats.normaltest(rvs) File "C:\Programs\Python27\lib\site-packages\scipy\stats\mstats_basic.py", line 1642, in normaltest k,_ = kurtosistest(a,axis) File "C:\Programs\Python27\lib\site-packages\scipy\stats\mstats_basic.py", line 1618, in kurtosistest n = a.count(axis=axis).astype(float) AttributeError: 'int' object has no attribute 'astype' Josef > > Timothy > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From JRadinger at gmx.at Fri Apr 8 09:32:51 2011 From: JRadinger at gmx.at (Johannes Radinger) Date: Fri, 08 Apr 2011 15:32:51 +0200 Subject: [SciPy-User] Integration from x1 to x2 of normal distribution Message-ID: <20110408133251.12040@gmx.net> Hej Scipy-users, it some time ago when I asked you about normal distributions and integrations... ...it's so long ago that I forgot how it worked, so maybe you can help me again: I have roughly following function of two superimposed normaldistributions: function = p * Normaldistribution1 + (1-p) * Normaldistribution2 for each normaldistribution I've got the scale (standard deviation) and the mean as a variable (s1, s2, m1, m2). Now I want the integral of this function from x1 to x2. How can I do that? Can that be solved with the scipy.stats package? thank you /johannes -- GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit gratis Handy-Flat! http://portal.gmx.net/de/go/dsl From josef.pktd at gmail.com Fri Apr 8 09:41:19 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 8 Apr 2011 09:41:19 -0400 Subject: [SciPy-User] Integration from x1 to x2 of normal distribution In-Reply-To: <20110408133251.12040@gmx.net> References: <20110408133251.12040@gmx.net> Message-ID: On Fri, Apr 8, 2011 at 9:32 AM, Johannes Radinger wrote: > Hej Scipy-users, > > > it some time ago when I asked you about normal distributions and integrations... > ...it's so long ago that I forgot how it worked, so maybe you can help me again: > > I have roughly following function of two superimposed normaldistributions: > > function = p * Normaldistribution1 + (1-p) * Normaldistribution2 > > for each normaldistribution I've got the scale (standard deviation) and the mean as a variable (s1, s2, m1, m2). > > Now I want the integral of this function from x1 to x2. How can I do that? Can that be solved > with the scipy.stats package? If your two distributions are independent, then it should be (mixture of two normal distributions): cdf(x) = p*stats.norm.cdf(x, **args1) + (1-p)*stats.norm.cdf(x, **args2) prob(x1, x2) = cdf(x2) - cdf(x1) where args are loc and scale for the two distributions Josef > > > thank you > /johannes > > -- > GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit > gratis Handy-Flat! http://portal.gmx.net/de/go/dsl > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Fri Apr 8 09:42:44 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 8 Apr 2011 09:42:44 -0400 Subject: [SciPy-User] Integration from x1 to x2 of normal distribution In-Reply-To: References: <20110408133251.12040@gmx.net> Message-ID: On Fri, Apr 8, 2011 at 9:41 AM, wrote: > On Fri, Apr 8, 2011 at 9:32 AM, Johannes Radinger wrote: >> Hej Scipy-users, >> >> >> it some time ago when I asked you about normal distributions and integrations... >> ...it's so long ago that I forgot how it worked, so maybe you can help me again: >> >> I have roughly following function of two superimposed normaldistributions: >> >> function = p * Normaldistribution1 + (1-p) * Normaldistribution2 >> >> for each normaldistribution I've got the scale (standard deviation) and the mean as a variable (s1, s2, m1, m2). >> >> Now I want the integral of this function from x1 to x2. How can I do that? Can that be solved >> with the scipy.stats package? > > If your two distributions are independent, then it should be (mixture > of two normal distributions): > > cdf(x) ?= p*stats.norm.cdf(x, **args1) + (1-p)*stats.norm.cdf(x, **args2) > > prob(x1, x2) = cdf(x2) - cdf(x1) > > where args are loc and scale for the two distributions assuming univariate distributions, multivariate normal would also be possible with scipy > > Josef > >> >> >> thank you >> /johannes >> >> -- >> GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit >> gratis Handy-Flat! http://portal.gmx.net/de/go/dsl >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > From JRadinger at gmx.at Fri Apr 8 10:58:26 2011 From: JRadinger at gmx.at (Johannes Radinger) Date: Fri, 08 Apr 2011 16:58:26 +0200 Subject: [SciPy-User] Integration from x1 to x2 of normal distribution In-Reply-To: References: <20110408133251.12040@gmx.net> Message-ID: <20110408145826.281760@gmx.net> -------- Original-Nachricht -------- > Datum: Fri, 8 Apr 2011 09:42:44 -0400 > Von: josef.pktd at gmail.com > An: SciPy Users List > Betreff: Re: [SciPy-User] Integration from x1 to x2 of normal distribution > On Fri, Apr 8, 2011 at 9:41 AM, wrote: > > On Fri, Apr 8, 2011 at 9:32 AM, Johannes Radinger > wrote: > >> Hej Scipy-users, > >> > >> > >> it some time ago when I asked you about normal distributions and > integrations... > >> ...it's so long ago that I forgot how it worked, so maybe you can help > me again: > >> > >> I have roughly following function of two superimposed > normaldistributions: > >> > >> function = p * Normaldistribution1 + (1-p) * Normaldistribution2 > >> > >> for each normaldistribution I've got the scale (standard deviation) and > the mean as a variable (s1, s2, m1, m2). > >> > >> Now I want the integral of this function from x1 to x2. How can I do > that? Can that be solved > >> with the scipy.stats package? > > > > If your two distributions are independent, then it should be (mixture > > of two normal distributions): > > > > cdf(x) ?= p*stats.norm.cdf(x, **args1) + (1-p)*stats.norm.cdf(x, > **args2) > > > > prob(x1, x2) = cdf(x2) - cdf(x1) > > > > where args are loc and scale for the two distributions > > assuming univariate distributions, multivariate normal would also be > possible with scipy Thank you very much for that quick response.. /johannes > > > > > Josef > > > >> > >> > >> thank you > >> /johannes > >> > >> -- > >> GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit > >> gratis Handy-Flat! http://portal.gmx.net/de/go/dsl > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit gratis Handy-Flat! http://portal.gmx.net/de/go/dsl From esko.lehtonen at gmail.com Fri Apr 8 17:16:56 2011 From: esko.lehtonen at gmail.com (Esko Lehtonen) Date: Sat, 09 Apr 2011 00:16:56 +0300 Subject: [SciPy-User] Write binary file from python and read from C++ In-Reply-To: References: Message-ID: <4D9F7B48.8000108@gmail.com> > From: Puiseux > I have a simple problem and could not find the solution in scipy/python doc. > Maybe i did not search in the good place... > > I have some numpy.array (float32), and i want to write it in a binary > file with my own format. > This file has to be readable by a C++ program. With struct module (from Python standard libray) you can "pack" any combination of values to strings containing the binary representation (you can choose bit-endianess etc.). Then you can write these strings to a file. Myself, I use it otherway round to parse stored UDP packets from. Such binary formats are not very user friendly, so I prefer standardized formats if possible (like HDF5, see pytables). - Esko From fperez.net at gmail.com Sat Apr 9 04:05:26 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 9 Apr 2011 01:05:26 -0700 Subject: [SciPy-User] [ANN] IPython 0.10.2 is out Message-ID: Hi all, we've just released IPython 0.10.2, full release notes are below. Downloads in source and windows binary form are available in the usual location: http://ipython.scipy.org/dist/ as well as on github: http://github.com/ipython/ipython/archives/rel-0.10.2 and at the Python Package Index (which easy_install will use by default): http://pypi.python.org/pypi/ipython so at any time you should find a location with good download speeds. You can find the full documentation at: http://ipython.github.com/ipython-doc/rel-0.10.2/html/ http://ipython.github.com/ipython-doc/rel-0.10.2/ipython.pdf Enjoy! Fernando (on behalf of the whole IPython team) Release 0.10.2 ============== IPython 0.10.2 was released April 9, 2011. This is a minor bugfix release that preserves backward compatibility. At this point, all IPython development resources are focused on the 0.11 series that includes a complete architectural restructuring of the project as well as many new capabilities, so this is likely to be the last release of the 0.10.x series. We have tried to fix all major bugs in this series so that it remains a viable platform for those not ready yet to transition to the 0.11 and newer codebase (since that will require some porting effort, as a number of APIs have changed). Thus, we are not opening a 0.10.3 active development branch yet, but if the user community requires new patches and is willing to maintain/release such a branch, we'll be happy to host it on the IPython github repositories. Highlights of this release: - The main one is the closing of github ticket #185, a major regression we had in 0.10.1 where pylab mode with GTK (or gthread) was not working correctly, hence plots were blocking with GTK. Since this is the default matplotlib backend on Unix systems, this was a major annoyance for many users. Many thanks to Paul Ivanov for helping resolve this issue. - Fix IOError bug on Windows when used with -gthread. - Work robustly if $HOME is missing from environment. - Better POSIX support in ssh scripts (remove bash-specific idioms). - Improved support for non-ascii characters in log files. - Work correctly in environments where GTK can be imported but not started (such as a linux text console without X11). For this release we merged 24 commits, contributed by the following people (please let us know if we ommitted your name and we'll gladly fix this in the notes for the future): * Fernando Perez * MinRK * Paul Ivanov * Pieter Cristiaan de Groot * TvrtkoM From sturla at molden.no Sun Apr 10 19:20:39 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 11 Apr 2011 01:20:39 +0200 Subject: [SciPy-User] Shared memory ndarrays (update) Message-ID: <4DA23B47.10301@molden.no> Here is an update for the shared memory arrays that Ga?l and I wrote two years ago. They are NumPy arrays referencing shared memory, and IPC using multiprocessing.Queue is possible by monkey patching how ndarrays are pickled. Usage: import numpy as np import sharedmem as sm shared_array = sm.zeros(n) I.e. the only difference from ndarrays is that pickle.dumps and multiprocessing.Queue do not make a copy of the buffer, and that allocated memory is shared between professes (e.g. created with os.fork, subprocess or multiprocessing.) A named memory map of the paging file is used on Windows. Unix System V IPC is used on Linux/Unix (thanks to Philip Semanchuk for assistance). Changes: - 64-bit support. - Memory leak on Linux/Unix should be gone (monkey patch for os._exit). - Added a global lock as there are callbacks to Python (the GIL is not sufficient serialization). I need help with testing, particularly on Linux / Apple and with the most recent NumPy. I'm an idiot with build tools, hence no setup.py. Invoke Cython and then cc. Compile sharedmemory_sysv.pyx for Linux/Unix or sharedmemory_sysv.pyx and ntqueryobject.c for Windows. Regards, Sturla -------------- next part -------------- A non-text attachment was scrubbed... Name: sharedmem.zip Type: application/x-zip-compressed Size: 8794 bytes Desc: not available URL: From sturla at molden.no Sun Apr 10 19:24:16 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 11 Apr 2011 01:24:16 +0200 Subject: [SciPy-User] [Numpy-discussion] Shared memory ndarrays (update) In-Reply-To: <4DA23B47.10301@molden.no> References: <4DA23B47.10301@molden.no> Message-ID: <4DA23C20.6060605@molden.no> Den 11.04.2011 01:20, skrev Sturla Molden: > > I'm an idiot with build tools, hence no setup.py. Invoke Cython and > then cc. Compile sharedmemory_sysv.pyx for Linux/Unix or > sharedmemory_sysv.pyx and ntqueryobject.c for Windows. Eh, that is sharedmemory_win.pyx and ntqueryobject.c for Windows :-) Sturla From sturla at molden.no Sun Apr 10 21:36:11 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 11 Apr 2011 03:36:11 +0200 Subject: [SciPy-User] What "Array" means In-Reply-To: References: Message-ID: <4DA25B0B.40901@molden.no> Den 08.04.2011 12:14, skrev Timothy Wu: > > File "/usr/lib/python2.6/dist-packages/scipy/stats/mstats_basic.py", > line 1546, in kurtosistest > n = a.count(axis=axis).astype(float) > AttributeError: 'int' object has no attribute 'astype' > > I'm not familiar with numpy nor scipy. What exactly should I put in there? I find multiple cases of code like this: a, axis = _chk_asarray(a, axis) n = a.count(axis) a, axis = _chk_asarray(a, axis) n = a.count(axis=axis) a, axis = _chk_asarray(a, axis) n = a.count(axis) a, axis = _chk_asarray(a, axis) n = a.count(axis=axis).astype(float) All of them are bugs: _chk_asarray sometimes returns an ndarray (due to np.ma.ravel), which does not have a count attribute. Sometimes count returns an int, which does not have an asfloat attribute. Sturla From gael.varoquaux at normalesup.org Mon Apr 11 01:05:23 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 11 Apr 2011 07:05:23 +0200 Subject: [SciPy-User] [Numpy-discussion] Shared memory ndarrays (update) In-Reply-To: <4DA23B47.10301@molden.no> References: <4DA23B47.10301@molden.no> Message-ID: <20110411050523.GA30988@phare.normalesup.org> Hey Sturla, It's really great that you are still working on that. I'll test the code under Linux. The scipy community has moved to github. If I create a repository under github and put the code on it, would you use it? If I find time, I'll add a setup.py. Ga?l From seb.haase at gmail.com Mon Apr 11 03:21:23 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Mon, 11 Apr 2011 09:21:23 +0200 Subject: [SciPy-User] [Numpy-discussion] Shared memory ndarrays (update) In-Reply-To: <20110411050523.GA30988@phare.normalesup.org> References: <4DA23B47.10301@molden.no> <20110411050523.GA30988@phare.normalesup.org> Message-ID: On Mon, Apr 11, 2011 at 7:05 AM, Gael Varoquaux wrote: > Hey Sturla, > > It's really great that you are still working on that. I'll test the code > under Linux. > > The scipy community has moved to github. If I create a repository under > github and put the code on it, would you use it? If I find time, I'll add > a setup.py. > > Ga?l > Hi, just wanted to say that I find this module, and shared memory in general very interesting, because I work with very large image data sets. I have a non python question: for Java there seems to exist a module/package/class called nio http://download.oracle.com/javase/1.4.2/docs/api/java/nio/MappedByteBuffer.html public abstract class MappedByteBuffer extends ByteBuffer A direct byte buffer whose content is a memory-mapped region of a file. Could this be used to share memory between Java applications and numpy ? Just hoping that someone here knows Java much better than me. - Sebastian Haase From hg+scipy at schaathun.net Mon Apr 11 05:52:50 2011 From: hg+scipy at schaathun.net (Hans Georg Schaathun) Date: Mon, 11 Apr 2011 11:52:50 +0200 Subject: [SciPy-User] Probability Density Estimation In-Reply-To: References: <20110406112947.GA1994@macwilliams.local> Message-ID: <20110411095250.GA1592@macwilliams.local> On Wed, Apr 06, 2011 at 11:47:26AM -0400, josef.pktd at gmail.com wrote: > My objective was measures for general (non-linear) dependence between > two random variables, and tests for independence. OK. The measure I need to estimate is I(X;Y) = H(X) - 0.5( H(X|Y=0) + H(X|Y=1) ) Where X is continuous and normally multivariate, and Y is boolean. The entropy H is estimated as H(X) ~= sum log f(x) where the sum is taken over all the observations x of X, and f is the KDE. This is following Ahmad and Lin (IEEE Trans IT 1989). I initially estimated the KDE independently for the full set of X observations, and for each class (Y=0 and Y=1). This gave some silly results. A much better approach is to calculate the bandwidth once; I calculate it from the full set of X observations, and then use the same bandwidth matrix for all the three KDE-s. Thanks to all of you for getting me on track. I'll see if I can clean up my annotated and augmented version of gaussian_kde for publications. I'll share when it is readable and consistent :-) If you have any ideas or are curious about further details, I shall be happy to listen and discuss it further. -- :-- Hans Georg From bsouthey at gmail.com Mon Apr 11 09:29:11 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 11 Apr 2011 08:29:11 -0500 Subject: [SciPy-User] What "Array" means In-Reply-To: <4DA25B0B.40901@molden.no> References: <4DA25B0B.40901@molden.no> Message-ID: <4DA30227.302@gmail.com> On 04/10/2011 08:36 PM, Sturla Molden wrote: > Den 08.04.2011 12:14, skrev Timothy Wu: >> File "/usr/lib/python2.6/dist-packages/scipy/stats/mstats_basic.py", >> line 1546, in kurtosistest >> n = a.count(axis=axis).astype(float) >> AttributeError: 'int' object has no attribute 'astype' >> >> I'm not familiar with numpy nor scipy. What exactly should I put in there? > I find multiple cases of code like this: > > a, axis = _chk_asarray(a, axis) > n = a.count(axis) > > a, axis = _chk_asarray(a, axis) > n = a.count(axis=axis) > > a, axis = _chk_asarray(a, axis) > n = a.count(axis) > > a, axis = _chk_asarray(a, axis) > n = a.count(axis=axis).astype(float) > > > All of them are bugs: > > _chk_asarray sometimes returns an ndarray (due to np.ma.ravel), which > does not have a count attribute. > > Sometimes count returns an int, which does not have an asfloat attribute. > > > Sturla > > Where did you get your 'count' function from? That is, what type of array is 'a'? Consequently the outcome is obvious and one that has been said many times before because _chk_asarray either uses np.ravel() or np.asarray(). Unfortunately the fix requires people with long term time and dedication because it is not an easy task (as I naively first thought). Bruce PS Only use 'standard ndarrays' (not matrix, structured or masked arrays) with scipy stats functions unless you you know that the function can handle your ndarray type. From henrylindsaysmith at gmail.com Mon Apr 11 13:41:52 2011 From: henrylindsaysmith at gmail.com (henry lindsay smith) Date: Mon, 11 Apr 2011 18:41:52 +0100 Subject: [SciPy-User] interpolating an impulse response Message-ID: ok, my filter knowledge is a bit rusty it seems. I have a simple filter - for a loudness model. It is specified at 48khz. it is a 2nd order IIR filter. To have the same frequency response at 44.1khz I think I need to interpolate the impulse response. however thats hard when there are only 3 coefficient in a and b. any ideas? or perhaps there is a different way. I don't have the specs of the filter wrt pass band frequency and transition band. thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Apr 11 14:56:53 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 11 Apr 2011 20:56:53 +0200 Subject: [SciPy-User] What "Array" means In-Reply-To: <4DA30227.302@gmail.com> References: <4DA25B0B.40901@molden.no> <4DA30227.302@gmail.com> Message-ID: <4DA34EF5.7020301@molden.no> Den 11.04.2011 15:29, skrev Bruce Southey: > Where did you get your 'count' function from? That is, what type of > array is 'a'? These are excepts from scipy/stats/mstats_basic.py The count attribute belongs to masked arrays. _chk_asarray returns a masked array (which has a count method) or an ndarray (which does not), hence one of the bugs. > Consequently the outcome is obvious and one that has been said many > times before because _chk_asarray either uses np.ravel() or > np.asarray(). It uses neither. Sturla From bsouthey at gmail.com Mon Apr 11 15:17:48 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 11 Apr 2011 14:17:48 -0500 Subject: [SciPy-User] What "Array" means In-Reply-To: <4DA34EF5.7020301@molden.no> References: <4DA25B0B.40901@molden.no> <4DA30227.302@gmail.com> <4DA34EF5.7020301@molden.no> Message-ID: <4DA353DC.7080203@gmail.com> On 04/11/2011 01:56 PM, Sturla Molden wrote: > Den 11.04.2011 15:29, skrev Bruce Southey: >> Where did you get your 'count' function from? That is, what type of >> array is 'a'? > These are excepts from scipy/stats/mstats_basic.py Thanks for the clarification of where you are looking. > The count attribute belongs to masked arrays. _chk_asarray returns a > masked array (which has a count method) or an ndarray (which does not), > hence one of the bugs. You can not say that without providing any examples. >> Consequently the outcome is obvious and one that has been said many >> times before because _chk_asarray either uses np.ravel() or >> np.asarray(). > It uses neither. > > _chk_asarray is defined differently in two places in scipy.stats. Consequently, without examples and error messages it is likely that the wrong definition is being used. So if you provide the examples in a different thread others can verify these apparent bugs. Bruce From sturla at molden.no Mon Apr 11 15:39:17 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 11 Apr 2011 21:39:17 +0200 Subject: [SciPy-User] What "Array" means In-Reply-To: <4DA353DC.7080203@gmail.com> References: <4DA25B0B.40901@molden.no> <4DA30227.302@gmail.com> <4DA34EF5.7020301@molden.no> <4DA353DC.7080203@gmail.com> Message-ID: <4DA358E5.6060903@molden.no> Den 11.04.2011 21:17, skrev Bruce Southey: > > _chk_asarray is defined differently in two places in scipy.stats. > Consequently, without examples and error messages it is likely that the > wrong definition is being used. So if you provide the examples in a > different thread others can verify these apparent bugs. I think the author of _chk_asarray in scipy/stats/mstats_basic.py have thought that np.ma.ravel returns a masked array, as the return value is used as such. But at least I my computer it does not :-) Sturla From bsouthey at gmail.com Mon Apr 11 16:08:16 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 11 Apr 2011 15:08:16 -0500 Subject: [SciPy-User] What "Array" means In-Reply-To: References: Message-ID: On Fri, Apr 8, 2011 at 5:45 AM, wrote: > On Fri, Apr 8, 2011 at 6:14 AM, Timothy Wu <2huggie at gmail.com> wrote: >> Hi I am trying to run Scipy's D'Agostino's normality test as documented here >> http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.normaltest.html >> >> For the array argument I tried something like this >> scipy.array([1,2,3]) >> or >> numpy.array([1,2,3]) >> >> and axis ignored. >> >> But with both method the test fails: >> >> File "/usr/lib/python2.6/dist-packages/scipy/stats/mstats_basic.py", line >> 1546, in kurtosistest >> ??? n = a.count(axis=axis).astype(float) >> AttributeError: 'int' object has no attribute 'astype' >> >> I'm not familiar with numpy nor scipy. What exactly should I put in there? > > > It looks like mstats.normaltest only works with 2-dimensional arrays, > stats.normaltest works with 1-dimensional arrays. > > rvs[:,None] ?in the example below adds an additional axis, so that it > is a column array with shape (20,1) > If you don't need the masked array version, then you can use stats.normaltest > > I haven't looked at the source yet, but this looks like a bug to me. > >>>> rvs = np.random.randn(20) >>>> rvs > array([ 0.02724005, -0.17836266, ?0.40530377, ?1.313246 ?, ?0.74069068, > ? ? ? -0.69010129, -0.24958557, -2.28311759, ?0.10525733, ?0.07986322, > ? ? ? -0.87282545, -1.41364294, ?1.16027037, ?0.23541801, -0.06663458, > ? ? ? ?0.39173207, ?0.06979893, ?0.4400277 , -1.29361117, -1.71524228]) >>>> stats.normaltest(rvs) > (1.7052869564079727, 0.42628656195988301) >>>> stats.mstats.normaltest(rvs[:,None]) > (masked_array(data = [1.70528695641], > ? ? ? ? ? ? mask = [False], > ? ? ? fill_value = 1e+20) > , masked_array(data = [ 0.42628656], > ? ? ? ? ? ? mask = False, > ? ? ? fill_value = 1e+20) > ) >>>> stats.mstats.normaltest(rvs) > > Traceback (most recent call last): > ?File "", line 1, in > ? ?stats.mstats.normaltest(rvs) > ?File "C:\Programs\Python27\lib\site-packages\scipy\stats\mstats_basic.py", > line 1642, in normaltest > ? ?k,_ = kurtosistest(a,axis) > ?File "C:\Programs\Python27\lib\site-packages\scipy\stats\mstats_basic.py", > line 1618, in kurtosistest > ? ?n = a.count(axis=axis).astype(float) > AttributeError: 'int' object has no attribute 'astype' > > Josef > Yes that is a bug so can someone create a ticket? (I don't have time today.) That occurs because ma.count() returns either an int (which causes the bug) or a ndarray. Actually that '.astype(float)' is probably not needed because as far as I can determine that every usage of an 'n' as an integer should still results in a float. There is also a second 'bug' because n must be greater than 3. I was looking for that because estimating kurtosis needs more than 3 observations: "This bias-corrected formula requires that X contain at least four elements." http://www.mathworks.com/help/toolbox/stats/kurtosis.html This a different ticket because we need to catch the cases when only one particular 'column' has less than 4 but the other are fine. >>> rvs = np.random.randn(20,10) >>> stats.mstats.normaltest(rvs, axis=0) (masked_array(data = [0.713606808604 0.132722315345 7.78660833457 5.38597554393 0.725711290319 0.172342343314 4.02320908322 1.46363950653 3.79550214574 0.293759931912], mask = [False False False False False False False False False False], fill_value = 1e+20) , masked_array(data = [ 0.69991008 0.93579283 0.0203779 0.06767843 0.69568685 0.91743718 0.13377386 0.48103283 0.14990537 0.86339761], mask = False, fill_value = 1e+20) ) >>> stats.mstats.normaltest(rvs, axis=1) (masked_array(data = [0.314582042621 0.4436261479 2.98149400163 2.02242070422 3.46138431999 9.94304440942 0.026055683609 5.7060731383 1.03808026381 0.169589515995 10.5681767508 1.28212296678 3.7013014714 0.43713740004 3.62659584833 0.289410600885 1.46353531025 0.745198884215 1.51022347547 0.00707268228071], mask = [False False False False False False False False False False False False False False False False False False False False], fill_value = 1e+20) , masked_array(data = [ 0.85445536 0.80106509 0.22520436 0.36377841 0.17716174 0.00693259 0.98705665 0.05766894 0.59509148 0.91870082 0.00507165 0.52673301 0.15713488 0.80366827 0.16311531 0.86527725 0.48105789 0.68894114 0.4699581 0.9964699 ], mask = False, fill_value = 1e+20) ) >>> stats.mstats.normaltest(rvs, axis=None) Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python2.7/site-packages/scipy/stats/mstats_basic.py", line 1649, in normaltest k,_ = kurtosistest(a,axis) File "/usr/lib64/python2.7/site-packages/scipy/stats/mstats_basic.py", line 1625, in kurtosistest n = a.count(axis=axis).astype(float) AttributeError: 'int' object has no attribute 'astype' >>> That is because: >>> mrvs=rvs.view(ma.MaskedArray) >>> type(mrvs) >>> type(mrvs.count(axis=0)) >>> type(mrvs.count(axis=1)) >>> type(mrvs.count(axis=None)) Bruce From jsseabold at gmail.com Mon Apr 11 20:13:37 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Mon, 11 Apr 2011 20:13:37 -0400 Subject: [SciPy-User] scikits.timeseries on windows: invalid object used in slice In-Reply-To: <4D977BB3.9020405@uci.edu> References: <4D976C90.2030105@uci.edu> <4D977BB3.9020405@uci.edu> Message-ID: On Sat, Apr 2, 2011 at 3:40 PM, Christoph Gohlke wrote: > > > On 4/2/2011 11:36 AM, Christoph Gohlke wrote: >> >> >> On 4/2/2011 10:33 AM, Skipper Seabold wrote: >>> >>> Any idea what's going on here? Code works fine on Linux with the same >>> version. But on Windows 7, 64-bit with the binaries from Chrisoph >>> , I get the following >>> >>> In [1]: import scikits.timeseries as ts >>> >>> In [2]: s = """1947-01-01,21.700 >>> ? ? ...: 1947-04-01,22.010 >>> ? ? ...: 1947-07-01,22.490""" >>> >>> In [3]: from StringIO import StringIO >>> >>> In [4]: cpi = ts.tsfromtxt(StringIO(s), delimiter=",", freq="Q", >>> datecols=(0), >>> ? ? ...: ? ? ? ? ? ? ? ? ? ? ? ? dtype=float) >>> >>> In [5]: cpi >>> Out[5]: >>> timeseries([21.7 22.01 22.49], >>> ? ? dates = [1947Q1 ... 1947Q3], >>> ? ? freq ?= Q-DEC) >>> >>> >>> In [6]: cpi[1:] >>> >>> --------------------------------------------------------------------------- >>> ValueError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call >>> last) >>> >>> C:\school\seaboldsvn\projects\greatinfl\ ? in() >>> >>> C:\Python27\lib\site-packages\numpy\ma\core.pyc in __getslice__(self, i, >>> j) >>> ? ? 3060 >>> ? ? 3061 ? ? ? ? """ >>> -> ? 3062 ? ? ? ? return self.__getitem__(slice(i, j)) >>> ? ? 3063 >>> ? ? 3064 ? ? def __setslice__(self, i, j, value): >>> >>> C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in >>> __getitem__(self >>> , indx) >>> ? ? ?650 ? ? Returns the item described by i. Not a copy. >>> ? ? ?651 ? ? ? ? """ >>> --> ? 652 ? ? ? ? (sindx, dindx, recheck) = self._index_checker(indx) >>> ? ? ?653 ? ? ? ? _data = ndarray.__getattribute__(self, '_data') >>> ? ? ?654 ? ? ? ? _mask = ndarray.__getattribute__(self, '_mask') >>> >>> C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in >>> _index_checker(s >>> elf, indx) >>> ? ? ?613 ? ? ? ? if isinstance(indx, slice): >>> ? ? ?614 ? ? ? ? ? ? indx = slice(self._slicebound_checker(indx.start), >>> --> ? 615 ? ? ? ? ? ? ? ? ? ? ? ? ?self._slicebound_checker(indx.stop), >>> ? ? ?616 ? ? ? ? ? ? ? ? ? ? ? ? ?indx.step) >>> ? ? ?617 ? ? ? ? ? ? return (indx, indx, False) >>> >>> C:\Python27\lib\site-packages\scikits\timeseries\tseries.pyc in >>> _slicebound_chec >>> ker(self, bound) >>> ? ? ?636 ? ? ? ? if not isinstance(bound, Date): >>> ? ? ?637 ? ? ? ? ? ? raise ValueError( >>> --> ? 638 ? ? ? ? ? ? ? ? "invalid object used in slice: %s" % >>> repr(bound)) >>> ? ? ?639 ? ? ? ? if bound.freq != _dates.freq: >>> ? ? ?640 ? ? ? ? ? ? raise TimeSeriesCompatibilityError('freq', >>> >>> ValueError: invalid object used in slice: 9223372036854775807L >>> >>> Cheers, >>> >>> Skipper >> >> >> Your code works for me on win-amd64-py2.x but several timeseries >> selftests fail with this error. Maybe it has to do with timeseries >> internally storing 'absdates' as C type long, not ssize_t. >> >> Christoph >> > > A quick fix is attached. > > Christoph Thanks. I filed a bug report [1] and attached your patch. Is this still the appropriate place to report bugs in the timeseries code? Where is the timeseries code hosted now? Skipper [1] http://projects.scipy.org/scikits/ticket/118 From pgmdevlist at gmail.com Mon Apr 11 20:33:45 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 12 Apr 2011 02:33:45 +0200 Subject: [SciPy-User] scikits.timeseries on windows: invalid object used in slice In-Reply-To: References: <4D976C90.2030105@uci.edu> <4D977BB3.9020405@uci.edu> Message-ID: On Apr 12, 2011, at 2:13 AM, Skipper Seabold wrote: > On Sat, Apr 2, 2011 at 3:40 PM, Christoph Gohlke wrote: > > Thanks. I filed a bug report [1] and attached your patch. > Skipper, Christoph, thanks a million. Don't hesitate to apply the patch yourself. Email me if you need more info > Is this still the appropriate place to report bugs in the timeseries > code? Where is the timeseries code hosted now? Still on the SVN, officially. There's a github spot that I keep on intending to use, but I'm a tad swamped those days... From ptittmann at gmail.com Mon Apr 11 22:52:56 2011 From: ptittmann at gmail.com (Peter) Date: Mon, 11 Apr 2011 19:52:56 -0700 Subject: [SciPy-User] stats repeated measures ANOVA Message-ID: <4DA3BE88.4060003@gmail.com> All, I would like to determine the effect of two independent variables (tree size class, and canopy percent cover) which are factor vectors (non-continuous) on the variance between two methods of measuring tree height. I believe this is referred to as two-way repeated measures ANOVA. Can anyone refer me to existing scipy (or other) methods for conducting this type of analysis? Best, Peter From josef.pktd at gmail.com Mon Apr 11 23:44:10 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 11 Apr 2011 23:44:10 -0400 Subject: [SciPy-User] stats repeated measures ANOVA In-Reply-To: <4DA3BE88.4060003@gmail.com> References: <4DA3BE88.4060003@gmail.com> Message-ID: On Mon, Apr 11, 2011 at 10:52 PM, Peter wrote: > All, > > I would like to determine the effect of ?two independent variables (tree > size class, and canopy percent cover) which are factor vectors > (non-continuous) on the variance between two methods of measuring tree > height. > > I believe this is referred to as two-way repeated measures ANOVA. As far as I understand ANOVA only tests for differences in means, or mean effects, not differences in variances, equality of variances is usually a standard assumption for ANOVA. But it's not clear to me what you want to test or estimate, with size, cover and method you have actually 3 factors. Are the two measuring methods for height used for every tree? Do you want the effect of size and cover on the measurement variance for each method separately? ....? I don't know much about two-way repeated measures ANOVA directly, but my guess is that it can be translated into estimating with a linear model and doing some heteroscedasticity tests on it. statsmodels might have the basis for this, but without more details on what exactly you want to estimate or test it's difficult to tell how much work this is. Cheers, Josef > > Can anyone refer me to existing scipy (or other) methods for conducting > this type of analysis? > > Best, > Peter > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From charlesr.harris at gmail.com Tue Apr 12 00:23:15 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 11 Apr 2011 22:23:15 -0600 Subject: [SciPy-User] interpolating an impulse response In-Reply-To: References: Message-ID: On Mon, Apr 11, 2011 at 11:41 AM, henry lindsay smith < henrylindsaysmith at gmail.com> wrote: > ok, my filter knowledge is a bit rusty it seems. > > I have a simple filter - for a loudness model. It is specified at 48khz. > it is a 2nd order IIR filter. To have the same frequency response at > 44.1khz I think I need to interpolate the impulse response. however thats > hard when there are only 3 coefficient in a and b. any ideas? > I don't think you can do it exactly. If you have O[n] + a_0 * O[n-1] + a_1 * O[n-2] + ... = I[n], the frequency response of the left hand side is 1 + a_0/z + a_1/z^2 + ..., where z = exp(2*pi*j*f/48e3). Note the discrete set of frequencies. Trying to match a sum of such such discrete frequencies with a sum of differently spaced discrete frequencies isn't going to work, the results will have different fundamental periods. However, presuming that the input has gone through an anti-aliasing filter up front, you should be able to make an approximation over the range of frequencies you are interested in that is probably good enough by resampling in the frequency domain, doing an inverse fft, and discarding the smaller coefficients in the result. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Apr 12 00:54:02 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 11 Apr 2011 22:54:02 -0600 Subject: [SciPy-User] interpolating an impulse response In-Reply-To: References: Message-ID: On Mon, Apr 11, 2011 at 10:23 PM, Charles R Harris < charlesr.harris at gmail.com> wrote: > > > On Mon, Apr 11, 2011 at 11:41 AM, henry lindsay smith < > henrylindsaysmith at gmail.com> wrote: > >> ok, my filter knowledge is a bit rusty it seems. >> >> I have a simple filter - for a loudness model. It is specified at 48khz. >> it is a 2nd order IIR filter. To have the same frequency response at >> 44.1khz I think I need to interpolate the impulse response. however thats >> hard when there are only 3 coefficient in a and b. any ideas? >> > > I don't think you can do it exactly. If you have O[n] + a_0 * O[n-1] + a_1 > * O[n-2] + ... = I[n], the frequency response of the left hand side is 1 + > a_0/z + a_1/z^2 + ..., where z = exp(2*pi*j*f/48e3). Note the discrete set > of frequencies. Trying to match a sum of such such discrete frequencies with > a sum of differently spaced discrete frequencies isn't going to work, the > results will have different fundamental periods. However, presuming that the > input has gone through an anti-aliasing filter up front, you should be able > to make an approximation over the range of frequencies you are interested in > that is probably good enough by resampling in the frequency domain, doing an > inverse fft, and discarding the smaller coefficients in the result. > > This changes a bit if you don't care about matching the phase, which I suspect is the case here. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ptittmann at gmail.com Tue Apr 12 01:31:18 2011 From: ptittmann at gmail.com (Peter Tittmann) Date: Mon, 11 Apr 2011 22:31:18 -0700 Subject: [SciPy-User] stats repeated measures ANOVA In-Reply-To: References: <4DA3BE88.4060003@gmail.com> Message-ID: Thanks for your reply Josef, sorry for the obscurity, I've third to clarify below.... On Apr 11, 2011 8:44 PM, wrote: > > On Mon, Apr 11, 2011 at 10:52 PM, Peter wrote: > > All, > > > > I would like to determine the effect of two independent variables (tree > > size class, and canopy percent cover) which are factor vectors > > (non-continuous) on the variance between two methods of measuring tree > > height. > > > > I believe this is referred to as two-way repeated measures ANOVA. > > As far as I understand ANOVA only tests for differences in means, or > mean effects, not differences in variances, equality of variances is > usually a standard assumption for ANOVA. There's a tutorial on doing it in R here http://www.r-bloggers.com/r-tutorial-series-two-way-repeated-measures-anova/ but it has a lot of array operations involved which I would preferr to do in python. > > But it's not clear to me what you want to test or estimate, with size, > cover and method you have actually 3 factors. > Are the two measuring methods for height used for every tree? > Do you want the effect of size and cover on the measurement variance > for each method separately? > ....? One method for measurement is field based and the other uses aerial data. The objective is to determine if the two factors (canopy density and tree diameter class) explain variance between of the measurement methods (ie accuracy of the aerial method) > > I don't know much about two-way repeated measures ANOVA directly, but > my guess is that it can be translated into estimating with a linear > model and doing some heteroscedasticity tests on it. > statsmodels might have the basis for this, but without more details on > what exactly you want to estimate or test it's difficult to tell how > much work this is. > > Cheers, > > Josef > > > > > > > Can anyone refer me to existing scipy (or other) methods for conducting > > this type of analysis? > > > > Best, > > Peter > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Apr 12 01:51:26 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 12 Apr 2011 01:51:26 -0400 Subject: [SciPy-User] stats repeated measures ANOVA In-Reply-To: References: <4DA3BE88.4060003@gmail.com> Message-ID: On Tue, Apr 12, 2011 at 1:31 AM, Peter Tittmann wrote: > Thanks for your reply Josef, sorry for the obscurity, I've third to clarify > below.... > > On Apr 11, 2011 8:44 PM, wrote: >> >> On Mon, Apr 11, 2011 at 10:52 PM, Peter wrote: >> > All, >> > >> > I would like to determine the effect of ?two independent variables (tree >> > size class, and canopy percent cover) which are factor vectors >> > (non-continuous) on the variance between two methods of measuring tree >> > height. >> > >> > I believe this is referred to as two-way repeated measures ANOVA. >> >> As far as I understand ANOVA only tests for differences in means, or >> mean effects, not differences in variances, equality of variances is >> usually a standard assumption for ANOVA. > There's a tutorial on doing it in R here > http://www.r-bloggers.com/r-tutorial-series-two-way-repeated-measures-anova/ > but it has a lot of array operations involved which I would preferr to do in > python. >> >> But it's not clear to me what you want to test or estimate, with size, >> cover and method you have actually 3 factors. >> Are the two measuring methods for height used for every tree? >> Do you want the effect of size and cover on the measurement variance >> for each method separately? >> ....? > One method for measurement is field based and the other uses aerial data. > The objective is to determine if the two factors (canopy density and tree > diameter class) explain <"variance"> between of the measurement methods (ie do you actually mean "variance" here, or just "difference" (in the sense of differences in mean)? Josef > accuracy of the aerial method) >> >> I don't know much about two-way repeated measures ANOVA directly, but >> my guess is that it can be translated into estimating with a linear >> model and doing some heteroscedasticity tests on it. >> statsmodels might have the basis for this, but without more details on >> what exactly you want to estimate or test it's difficult to tell how >> much work this is. >> >> Cheers, >> >> Josef >> >> >> >> > >> > Can anyone refer me to existing scipy (or other) methods for conducting >> > this type of analysis? >> > >> > Best, >> > Peter >> > >> > >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> > >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From c-b at asu.edu Tue Apr 12 01:58:49 2011 From: c-b at asu.edu (Christopher Brown) Date: Mon, 11 Apr 2011 22:58:49 -0700 Subject: [SciPy-User] stats repeated measures ANOVA In-Reply-To: References: <4DA3BE88.4060003@gmail.com> Message-ID: <201104112258.50158.c-b@asu.edu> On Monday, April 11, 2011 10:31:18 PM Peter Tittmann wrote: > Thanks for your reply Josef, sorry for the obscurity, I've third to clarify > below.... > > On Apr 11, 2011 8:44 PM, wrote: > > On Mon, Apr 11, 2011 at 10:52 PM, Peter wrote: > > > All, > > > > > > I would like to determine the effect of two independent variables > > > (tree size class, and canopy percent cover) which are factor vectors > > > (non-continuous) on the variance between two methods of measuring tree > > > height. > > > > > > I believe this is referred to as two-way repeated measures ANOVA. > > > > As far as I understand ANOVA only tests for differences in means, or > > mean effects, not differences in variances, equality of variances is > > usually a standard assumption for ANOVA. > > There's a tutorial on doing it in R here > http://www.r-bloggers.com/r-tutorial-series-two-way-repeated-measures-anova > / but it has a lot of array operations involved which I would preferr to do > in python. > > > But it's not clear to me what you want to test or estimate, with size, > > cover and method you have actually 3 factors. > > Are the two measuring methods for height used for every tree? > > Do you want the effect of size and cover on the measurement variance > > for each method separately? > > ....? > > One method for measurement is field based and the other uses aerial data. > The objective is to determine if the two factors (canopy density and tree > diameter class) explain variance between of the measurement methods (ie > accuracy of the aerial method) From what you describe, this does not sound like repeated measures anova to me. I am not sure exactly what you want, but it sounds like ancova might be close: http://en.wikipedia.org/wiki/Analysis_of_covariance -- Christopher Brown, Ph.D. Associate Research Professor Department of Speech and Hearing Science Arizona State University http://pal.asu.edu From sturla at molden.no Tue Apr 12 03:32:11 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 12 Apr 2011 09:32:11 +0200 Subject: [SciPy-User] stats repeated measures ANOVA In-Reply-To: References: <4DA3BE88.4060003@gmail.com> Message-ID: Den 12. apr. 2011 kl. 05.44 skrev josef.pktd at gmail.com: >> >> > As far as I understand ANOVA only tests for differences in means, or > mean effects, not differences in variances, equality of variances is > usually a standard assumption for ANOVA. > Otherwise he will get a Behrens-Fisher problem, only more complex because there are more than two groups. > > I don't know much about two-way repeated measures ANOVA directly, but > my guess is that it can be translated into estimating with a linear > model and doing some heteroscedasticity tests >> >> >> >> "Repeated measures" ANOVA is just a misnomer for using the "randomized block design" as a substitute for not knowing MANOVA or Hotelling's T- square test, and as such leads to conclusions that are very hard to interpret. The real value of repeated measures ANOVA in medical litterature is often to inform the reader that the authors don't understand the statistics they use ;-) Sturla From henrylindsaysmith at gmail.com Tue Apr 12 08:43:01 2011 From: henrylindsaysmith at gmail.com (henry lindsay smith) Date: Tue, 12 Apr 2011 13:43:01 +0100 Subject: [SciPy-User] interpolating an impulse response In-Reply-To: References: Message-ID: yes I don't care about matching phase. It's for loudness estimation so I will be summing the power afterwards I'll have a go at the fft/ifft method you suggested. On Tue, Apr 12, 2011 at 5:54 AM, Charles R Harris wrote: > > > On Mon, Apr 11, 2011 at 10:23 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Mon, Apr 11, 2011 at 11:41 AM, henry lindsay smith < >> henrylindsaysmith at gmail.com> wrote: >> >>> ok, my filter knowledge is a bit rusty it seems. >>> >>> I have a simple filter - for a loudness model. It is specified at >>> 48khz. it is a 2nd order IIR filter. To have the same frequency response >>> at 44.1khz I think I need to interpolate the impulse response. however >>> thats hard when there are only 3 coefficient in a and b. any ideas? >>> >> >> I don't think you can do it exactly. If you have O[n] + a_0 * O[n-1] + a_1 >> * O[n-2] + ... = I[n], the frequency response of the left hand side is 1 + >> a_0/z + a_1/z^2 + ..., where z = exp(2*pi*j*f/48e3). Note the discrete set >> of frequencies. Trying to match a sum of such such discrete frequencies with >> a sum of differently spaced discrete frequencies isn't going to work, the >> results will have different fundamental periods. However, presuming that the >> input has gone through an anti-aliasing filter up front, you should be able >> to make an approximation over the range of frequencies you are interested in >> that is probably good enough by resampling in the frequency domain, doing an >> inverse fft, and discarding the smaller coefficients in the result. >> >> > This changes a bit if you don't care about matching the phase, which I > suspect is the case here. > > Chuck > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.roberts at duke.edu Wed Apr 6 14:42:17 2011 From: jason.roberts at duke.edu (Jason Roberts) Date: Wed, 6 Apr 2011 14:42:17 -0400 Subject: [SciPy-User] Most efficient way to eliminate or keep specified features from labeled image Message-ID: <02b501cbf48a$5b57dc60$12079520$@roberts@duke.edu> I have a 2D image labeled with scipy.ndimage.label and a list of integers that identify features I would like to eliminate (set to 0), or keep (set the others to 0). I can think of several ways to do this (nested loops, use generic_filter, etc.). Some techniques might provide better performance than others, depending on the size of the image, the number of features to eliminate/keep, etc. Is there a technique that provides good performance across a wide range of scenarios? This seems like one of those problems where the answer should be obvious-there should be some function specifically for it-but I have not come across it yet. Thanks for any advice you can provide, Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.delque at sfr.fr Fri Apr 1 08:06:39 2011 From: michael.delque at sfr.fr (michael delquee) Date: Fri, 1 Apr 2011 14:06:39 +0200 (CEST) Subject: [SciPy-User] using separate files for functions with scipy Message-ID: <22626044.417021301659599828.JavaMail.www@wsfrf1126> Hi everybody, I began using scipy a few days ago and I think a long story between us is beginning ;) I come from the Matlab/Scilab world and I have no big problem using SciPy. Well, in fact, I have one problem, that is why I come here ! I will try to explain. I have a main program in one file "verif.py" and I want it to import functions that i have defined in another file "fonctions.py". I import Scipy in the main program but I also need the scipy functions to be available in the functions and do not manage to do so. The main verif.py program is : ????? from scipy import * ????? from fonction import truc ? ? ????? a=2; ????? b=truc(a); ????? print b fonction.py is : ????? from scipy import * ????? def truc(machin): ????? ????? plouc=machin*2; ????? ????? A=array([[1,2],[3,4]]); ????? ????? return A Under iPython, when I do : ????? run verif.py I get an error pointing to truc(machin): ????? NameError: global name 'array' is not defined ????? WARNING: failure executing file: I have no error running ????? run fonction.py Which seems normal as I explicitely import scipy. Of course I have no error if I put the function in the same file as the main code : ????? from scipy import * ????? ????? def truc(machin): ????? ????? plouc=machin*2; ????? ????? A=array([[1,2],[3,4]]); ????? ????? return A ????? a=2; ????? b=truc(a); ????? print b runs errorless. The most mysterious thing is that sometimes I have no error even using two separate files, but I don't see why... Well, I am convinced that I miss something important and basic... I am lost. midel -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.delque at sfr.fr Sat Apr 2 08:14:52 2011 From: michael.delque at sfr.fr (michael delquee) Date: Sat, 2 Apr 2011 14:14:52 +0200 (CEST) Subject: [SciPy-User] (pas de sujet) In-Reply-To: References: <3981665.414641301659254202.JavaMail.www@wsfrf1126> <4D96060C.7020409@noaa.gov> <4D9623D1.7040006@GMail.COM> <4D9628A4.8050906@noaa.gov> Message-ID: <9648882.193491301746492041.JavaMail.www@wsfrf1214> Well, thank you very much for all your answers. In fact you gave me more answers than I had questions ;) I cannot try your solutions right now but I'll do it asap. It was clear to me that the way I coded was not rigorous and that was the reason why I had these errors. I'm a young scipy user so I will try to get the good habits from the beginning, in particularly with the separate namespaces. I've allready heard of it as I tried a few days ago to replace matlab with c++... Well, harder than scipy/matplotlib ! But I keep trying. In fact Scipy was so easy compared to c++ that I may have underestimated the "acclimatation" time :) Thanks ! ======================================== Message du : 02/04/2011 De : "Ralf Gommers " A : "SciPy Users List" Copie ? : Sujet : Re: [SciPy-User] (pas de sujet) On Sat, Apr 2, 2011 at 12:49 AM, Robert Kern wrote: > On Fri, Apr 1, 2011 at 16:33, Ralf Gommers wrote: >> Do you feel like writing a patch for the >> howto_document file? > > https://github.com/numpy/numpy/pull/67 Thanks Robert! _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas at hilboll.de Wed Apr 6 10:36:15 2011 From: andreas at hilboll.de (Andreas) Date: Wed, 06 Apr 2011 16:36:15 +0200 Subject: [SciPy-User] how to use kendalltau_seasonal In-Reply-To: <83F2D73D-2775-47A6-A369-D07E6F2EE45C@gmail.com> References: <4D9C4354.5080100@hilboll.de> <83F2D73D-2775-47A6-A369-D07E6F2EE45C@gmail.com> Message-ID: <4D9C7A5F.7010903@hilboll.de> Hi, thanks for the info! > Imagine you need to check whether there's a trend in a time series X, but want to take a potential seasonal dependence into account. > Reshape your data so that it's 2D, with seasons as columns, and feed it to the function. Ah, good to know. > You'll get a dictionary giving you the value of Kendall's tau (w/ and w/o the seasonal effects) with the corresponding p-values that should help you figure out without there's a trend in your data and whether seasonality plays a role. Mhh. I think I do understand the statistics. However, I'm still not totally sure about all the elements of the returned dictionary: scipy.stats.mstats.kendalltau_seasonal(TT.reshape((15,12))).keys() Out[39]: ['chi2 total', 'global tau (alt)', 'chi2 trend', 'seasonal tau', 'global p-value (dep)', 'global tau', 'global p-value (indep)', 'seasonal p-value'] Can you please shed some more light on this? Once I understand what the function does, I'll be more than happy to document it ... Cheers, Andreas. > Example: > You have a 60-element array corresponding to 5 years of consecutive monthly data, and want to check the trend, taking potential monthly variations into account. In that case, your "season" is a month. Reshape it as a (5,12), and use the result as input to the function. > > More info: > http://pubs.usgs.gov/sir/2005/5275/pdf/sir2005-5275.pdf > Hirsch R.M., Slack J.R. and Smith R.A. (1982), Techniques of Trend Analysis > for Monthly Water Quality Data, Water Resources Research 18(1), 107-12 > > I'm sure there must be a R equivalent somewhere. > In any case, please do help to document the function as needed. Don't hesitate to contact me if you need more info. > Cheers > P. > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From pholvey at gmail.com Fri Apr 8 14:08:12 2011 From: pholvey at gmail.com (Patrick Holvey) Date: Fri, 8 Apr 2011 14:08:12 -0400 Subject: [SciPy-User] fmin_cg - what should fprime return so fmin_cg has the gradient? Message-ID: Hi everybody, I'm using fmin_cg to minimize atom positions using a Keating potential (one term for the bond stretching and one for the bond angle bending). It's been great but now I'm trying to speed up the minimization by giving fmin_cg the optional argument of fprime. I'm coding up the gradient function now, but I'm not sure what form the vector field is supposed to be when it returns the vector field to the optimization for fmin_cg to be happy. Does anyone have any experience with this? Thanks, Patrick -- Patrick Holvey Graduate Student Dept. of Materials Science and Engineering Johns Hopkins University pholvey1 at jhu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From jallikattu at googlemail.com Mon Apr 11 05:23:15 2011 From: jallikattu at googlemail.com (morovia morovia) Date: Mon, 11 Apr 2011 14:53:15 +0530 Subject: [SciPy-User] Shifting peak to origin in fourier space Message-ID: Hello I am trying to demodulate a signal. Here I need to translate (shift) the peak at the desired frequency in the fourier space to the origin. Can anyone suggest me how to do it. Thanks Best regards Morovia. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.harrison at liquidmesh.com Mon Apr 11 09:52:22 2011 From: ben.harrison at liquidmesh.com (Ben Harrison) Date: Mon, 11 Apr 2011 23:52:22 +1000 Subject: [SciPy-User] genfromtxt but I need to search file first Message-ID: Hi, I'm processing a bunch of text files with headers. The header is many lines long, may be 43 lines, may be 42, maybe something else. The last line of the header is always : ~ASCII section After this there are columns of data that I want to load via genfromtxt (mainly to strip leading spaces, and to load directly into an array). Thing is, genfromtxt already loops twice through each file, so adding a bit of code like this: infile = open("my_text.txt","r") text = infile.read() infile.close() means the file is read 3 times. Is there another way? I'm new to python. Ben. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pholvey at gmail.com Mon Apr 11 20:52:29 2011 From: pholvey at gmail.com (Patrick Holvey) Date: Mon, 11 Apr 2011 20:52:29 -0400 Subject: [SciPy-User] Passing fmin_cg a gradient Message-ID: Hi everybody, I'm using fmin_cg to minimize atom positions (xyz coordinates) using a Keating potential (one term for the bond stretching and one for the bond angle bending). It's been great but now I'm trying to speed up the minimization by giving fmin_cg the optional argument of fprime. I'm coding up the gradient function now, but I'm not sure what form the vector field is supposed to be when it returns the vector field to the optimization for fmin_cg to be happy. Should the gradient be returned as a list of arrays (since there would be an xyz magnitude associated with each atom position) or should it be in some other form that I'm not thinking of? Does anyone have any experience with this? I've tried to figure out what I should be passing from online examples of fmin_cg usage, but I'm not really seeing examples that are able to be extended to this problem in a straightforward manner. I'm also open to alternate optimization calls if anyone has recommendations. I've got both an analytic potential and gradient so I do have an fprime term if the optimization call needs it. However, I'd appreciate guidance on what form the gradient function should return since I don't want to be dealing with a new optimization method and be caught in the same spot again. Thanks for any help! Patrick -- Patrick Holvey Graduate Student Dept. of Materials Science and Engineering Johns Hopkins University pholvey1 at jhu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From ptittmann at gmail.com Tue Apr 12 09:32:24 2011 From: ptittmann at gmail.com (Peter Tittmann) Date: Tue, 12 Apr 2011 06:32:24 -0700 Subject: [SciPy-User] stats repeated measures ANOVA In-Reply-To: References: <4DA3BE88.4060003@gmail.com> Message-ID: I mean the difference between the means of the two measurement methods. Again, apologies for the confusing use of terms. I'm obviously not a statistician... On Apr 11, 2011 10:51 PM, wrote: > On Tue, Apr 12, 2011 at 1:31 AM, Peter Tittmann wrote: >> Thanks for your reply Josef, sorry for the obscurity, I've third to clarify >> below.... >> >> On Apr 11, 2011 8:44 PM, wrote: >>> >>> On Mon, Apr 11, 2011 at 10:52 PM, Peter wrote: >>> > All, >>> > >>> > I would like to determine the effect of two independent variables (tree >>> > size class, and canopy percent cover) which are factor vectors >>> > (non-continuous) on the variance between two methods of measuring tree >>> > height. >>> > >>> > I believe this is referred to as two-way repeated measures ANOVA. >>> >>> As far as I understand ANOVA only tests for differences in means, or >>> mean effects, not differences in variances, equality of variances is >>> usually a standard assumption for ANOVA. >> There's a tutorial on doing it in R here >> http://www.r-bloggers.com/r-tutorial-series-two-way-repeated-measures-anova/ >> but it has a lot of array operations involved which I would preferr to do in >> python. >>> >>> But it's not clear to me what you want to test or estimate, with size, >>> cover and method you have actually 3 factors. >>> Are the two measuring methods for height used for every tree? >>> Do you want the effect of size and cover on the measurement variance >>> for each method separately? >>> ....? >> One method for measurement is field based and the other uses aerial data. >> The objective is to determine if the two factors (canopy density and tree >> diameter class) explain <"variance"> between of the measurement methods (ie > > do you actually mean "variance" here, or just "difference" (in the > sense of differences in mean)? > > Josef > >> accuracy of the aerial method) >>> >>> I don't know much about two-way repeated measures ANOVA directly, but >>> my guess is that it can be translated into estimating with a linear >>> model and doing some heteroscedasticity tests on it. >>> statsmodels might have the basis for this, but without more details on >>> what exactly you want to estimate or test it's difficult to tell how >>> much work this is. >>> >>> Cheers, >>> >>> Josef >>> >>> >>> >>> > >>> > Can anyone refer me to existing scipy (or other) methods for conducting >>> > this type of analysis? >>> > >>> > Best, >>> > Peter >>> > >>> > >>> > _______________________________________________ >>> > SciPy-User mailing list >>> > SciPy-User at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/scipy-user >>> > >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Tue Apr 12 10:10:31 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 12 Apr 2011 09:10:31 -0500 Subject: [SciPy-User] stats repeated measures ANOVA In-Reply-To: References: <4DA3BE88.4060003@gmail.com> Message-ID: <4DA45D57.3060007@gmail.com> On 04/12/2011 08:32 AM, Peter Tittmann wrote: > > I mean the difference between the means of the two measurement > methods. Again, apologies for the confusing use of terms. I'm > obviously not a statistician... > > On Apr 11, 2011 10:51 PM, > wrote: > > On Tue, Apr 12, 2011 at 1:31 AM, Peter Tittmann > wrote: > >> Thanks for your reply Josef, sorry for the obscurity, I've third to > clarify > >> below.... > >> > >> On Apr 11, 2011 8:44 PM, > wrote: > >>> > >>> On Mon, Apr 11, 2011 at 10:52 PM, Peter > wrote: > >>> > All, > >>> > > >>> > I would like to determine the effect of two independent > variables (tree > >>> > size class, and canopy percent cover) which are factor vectors > >>> > (non-continuous) on the variance between two methods of > measuring tree > >>> > height. > >>> > > >>> > I believe this is referred to as two-way repeated measures ANOVA. > >>> > >>> As far as I understand ANOVA only tests for differences in means, or > >>> mean effects, not differences in variances, equality of variances is > >>> usually a standard assumption for ANOVA. > >> There's a tutorial on doing it in R here > >> > http://www.r-bloggers.com/r-tutorial-series-two-way-repeated-measures-anova/ > >> but it has a lot of array operations involved which I would preferr > to do in > >> python. > >>> > >>> But it's not clear to me what you want to test or estimate, with size, > >>> cover and method you have actually 3 factors. > >>> Are the two measuring methods for height used for every tree? > >>> Do you want the effect of size and cover on the measurement variance > >>> for each method separately? > >>> ....? > >> One method for measurement is field based and the other uses aerial > data. > >> The objective is to determine if the two factors (canopy density > and tree > >> diameter class) explain <"variance"> between of the measurement > methods (ie > > > > do you actually mean "variance" here, or just "difference" (in the > > sense of differences in mean)? > > > > Josef > > > >> accuracy of the aerial method) > >>> > >>> I don't know much about two-way repeated measures ANOVA directly, but > >>> my guess is that it can be translated into estimating with a linear > >>> model and doing some heteroscedasticity tests on it. > >>> statsmodels might have the basis for this, but without more details on > >>> what exactly you want to estimate or test it's difficult to tell how > >>> much work this is. > >>> > >>> Cheers, > >>> > >>> Josef > >>> > >>> > >>> > >>> > > >>> > Can anyone refer me to existing scipy (or other) methods for > conducting > >>> > this type of analysis? > >>> > > >>> > Best, > >>> > Peter > >>> > > >>> > > >>> > _______________________________________________ > >>> > SciPy-User mailing list > >>> > SciPy-User at scipy.org > >>> > http://mail.scipy.org/mailman/listinfo/scipy-user > >>> > > >>> _______________________________________________ > >>> SciPy-User mailing list > >>> SciPy-User at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > >> _______________________________________________ > >> SciPy-User mailing list > >> SciPy-User at scipy.org > >> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Do you mean 'multivariate analysis of variance (MANOVA)'? http://www.epa.gov/bioiweb1/statprimer/manova.html This is what I thought you meant because you have two models where tree size class and canopy percent cover are factors: Tree height by method 1 = mu + tree size class + canopy percent cover + e1 Tree height by method 2 = mu + tree size class + canopy percent cover + e2 This leads to the manova test gives an overall test for tree size class and canopy percent cover across both methods. However, if your measures are highly correlated (say >0.9), then you really don't gain anything at all. Do not know if this is in pystatsmodels bu you can do a very similar thing with a mixed model. However, I think you are better off with say R or SAS as you can find worked examples until you understand the problem more. By the way, two different measure of the same thing is not called repeated measures. Under repeated measures, you are measuring exactly the same thing on the exact same subject (tree) most commonly over time - such as measuring a tree's height over many months/years. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Tue Apr 12 10:13:57 2011 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 12 Apr 2011 16:13:57 +0200 Subject: [SciPy-User] fmin_bfgs In-Reply-To: References: Message-ID: On Thu, 31 Mar 2011 05:54:15 -0700 (PDT) Dirk Nachbar wrote: > I am using fmin_bfgs to minimise a function, I put a >print in the > function so I can see the steps. It does show me that >the value goes > down but then goes back to the starting value and shows >that it is > optimised. But from the print I know it's not the true >min, how can I > make the function stop at the right min. I am not >specifying a fprime > function. > > Dirk Can you provide a self-consistent example ? Nils From josef.pktd at gmail.com Tue Apr 12 10:24:21 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 12 Apr 2011 10:24:21 -0400 Subject: [SciPy-User] stats repeated measures ANOVA In-Reply-To: <4DA45D57.3060007@gmail.com> References: <4DA3BE88.4060003@gmail.com> <4DA45D57.3060007@gmail.com> Message-ID: On Tue, Apr 12, 2011 at 10:10 AM, Bruce Southey wrote: > On 04/12/2011 08:32 AM, Peter Tittmann wrote: > > I mean the difference between the means of the two measurement methods. > Again, apologies for the confusing use of terms. I'm obviously not a > statistician... > > On Apr 11, 2011 10:51 PM, wrote: >> On Tue, Apr 12, 2011 at 1:31 AM, Peter Tittmann >> wrote: >>> Thanks for your reply Josef, sorry for the obscurity, I've third to >>> clarify >>> below.... >>> >>> On Apr 11, 2011 8:44 PM, wrote: >>>> >>>> On Mon, Apr 11, 2011 at 10:52 PM, Peter wrote: >>>> > All, >>>> > >>>> > I would like to determine the effect of ?two independent variables >>>> > (tree >>>> > size class, and canopy percent cover) which are factor vectors >>>> > (non-continuous) on the variance between two methods of measuring tree >>>> > height. >>>> > >>>> > I believe this is referred to as two-way repeated measures ANOVA. >>>> >>>> As far as I understand ANOVA only tests for differences in means, or >>>> mean effects, not differences in variances, equality of variances is >>>> usually a standard assumption for ANOVA. >>> There's a tutorial on doing it in R here >>> >>> http://www.r-bloggers.com/r-tutorial-series-two-way-repeated-measures-anova/ >>> but it has a lot of array operations involved which I would preferr to do >>> in >>> python. >>>> >>>> But it's not clear to me what you want to test or estimate, with size, >>>> cover and method you have actually 3 factors. >>>> Are the two measuring methods for height used for every tree? >>>> Do you want the effect of size and cover on the measurement variance >>>> for each method separately? >>>> ....? >>> One method for measurement is field based and the other uses aerial data. >>> The objective is to determine if the two factors (canopy density and tree >>> diameter class) explain <"variance"> between of the measurement methods >>> (ie >> >> do you actually mean "variance" here, or just "difference" (in the >> sense of differences in mean)? >> >> Josef >> >>> accuracy of the aerial method) >>>> >>>> I don't know much about two-way repeated measures ANOVA directly, but >>>> my guess is that it can be translated into estimating with a linear >>>> model and doing some heteroscedasticity tests on it. >>>> statsmodels might have the basis for this, but without more details on >>>> what exactly you want to estimate or test it's difficult to tell how >>>> much work this is. >>>> >>>> Cheers, >>>> >>>> Josef >>>> >>>> >>>> >>>> > >>>> > Can anyone refer me to existing scipy (or other) methods for >>>> > conducting >>>> > this type of analysis? >>>> > >>>> > Best, >>>> > Peter >>>> > >>>> > >>>> > _______________________________________________ >>>> > SciPy-User mailing list >>>> > SciPy-User at scipy.org >>>> > http://mail.scipy.org/mailman/listinfo/scipy-user >>>> > >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > Do you mean 'multivariate analysis of variance (MANOVA)'? > http://www.epa.gov/bioiweb1/statprimer/manova.html > > This is what I thought you meant because you have two models where tree size > class and canopy percent cover are factors: > Tree height by method 1 = mu + tree size class + canopy percent cover + e1 > Tree height by method 2 = mu + tree size class + canopy percent cover + e2 or maybe Tree height by method 1 - Tree height by method 2 = mu + tree size class + canopy percent cover + e or maybe Tree height = mu + method + tree size class + canopy percent cover + interaction terms + e plus indivdual effect for repeated measures (?) > > This leads to the manova test gives an overall test for tree size class and > canopy percent cover across both methods. However, if your measures are > highly correlated (say >0.9), then you really don't gain anything at all. > > Do not know if this is in pystatsmodels bu you can do a very similar thing > with a mixed model. However, I think you are better off with say R or SAS as > you can find worked examples until you understand the problem more. I agree. In statsmodels this might require some work. Josef > > By the way, two different measure of the same thing is not called repeated > measures. Under repeated measures, you are measuring exactly the same thing > on the exact same subject (tree) most commonly over time - such as measuring > a tree's height over many months/years. > > Bruce > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From warren.weckesser at enthought.com Tue Apr 12 10:25:44 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Tue, 12 Apr 2011 09:25:44 -0500 Subject: [SciPy-User] genfromtxt but I need to search file first In-Reply-To: References: Message-ID: Hi Ben, On Mon, Apr 11, 2011 at 8:52 AM, Ben Harrison wrote: > Hi, I'm processing a bunch of text files with headers. The header is many > lines long, may be 43 lines, may be 42, maybe something else. The last line > of the header is always : > > ~ASCII section > > By any changes is this a LAS file? Take a look here: http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055563.html and at my response. I included a "quick and dirty" implementation of a LAS reader. (I have a nicer version in progress, but it will be a few days before I can finish it.) In the version attached to that email, I just use loadtxt() to read the data after the ~A line. Warren > After this there are columns of data that I want to load via genfromtxt > (mainly to strip leading spaces, and to load directly into an array). > Thing is, genfromtxt already loops twice through each file, so adding a bit > of code like this: > > infile = open("my_text.txt","r") > text = infile.read() > infile.close() > > means the file is read 3 times. > > Is there another way? I'm new to python. > > Ben. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattknox.ca at gmail.com Tue Apr 12 11:49:25 2011 From: mattknox.ca at gmail.com (Matt Knox) Date: Tue, 12 Apr 2011 15:49:25 +0000 (UTC) Subject: [SciPy-User] =?utf-8?q?scikits=2Etimeseries_on_windows=3A_invalid?= =?utf-8?q?_object_used=09in_slice?= References: <4D976C90.2030105@uci.edu> <4D977BB3.9020405@uci.edu> Message-ID: > > Is this still the appropriate place to report bugs in the timeseries > > code? Where is the timeseries code hosted now? > > Still on the SVN, officially. There's a github spot that I keep on intending > to use, but I'm a tad swamped those days... > Hi Pierre. Given that we have both been inactive on this project for a while, perhaps it is best that we expedite the move to github and grant commit / merge privileges to any interested maintainers and retire the svn repo. Then at least interested parties can pick up the slack if they want. I still use the module, but don't rely on it as heavily as I once did so my motivation for fixing / improving it has diminished (I'm selfish, I know :P ). If you are ok with this, I can look after coordinating the details of the transition. - Matt From pgmdevlist at gmail.com Tue Apr 12 12:03:29 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 12 Apr 2011 18:03:29 +0200 Subject: [SciPy-User] scikits.timeseries on windows: invalid object used in slice In-Reply-To: References: <4D976C90.2030105@uci.edu> <4D977BB3.9020405@uci.edu> Message-ID: <0D673964-13B4-43B2-834D-5E5CBFCDEA06@gmail.com> On Apr 12, 2011, at 5:49 PM, Matt Knox wrote: >>> Is this still the appropriate place to report bugs in the timeseries >>> code? Where is the timeseries code hosted now? >> >> Still on the SVN, officially. There's a github spot that I keep on intending >> to use, but I'm a tad swamped those days... >> > > Hi Pierre. Given that we have both been inactive on this project for a while, > perhaps it is best that we expedite the move to github and grant commit / merge > privileges to any interested maintainers and retire the svn repo. Then at least > interested parties can pick up the slack if they want. I created two different github projects. One is just the SVN ported to git last September, accessible here: https://github.com/pierregm/scikits.timeseries I have a second one in parallel, timeseries-sandbox, which contains some cool additions (irregular frequencies) and some brilliant failures (DateArray as a pure subclass of ndarray in C. Nope, I can't speak good C enough...). My idea was to get something as close as the new dtype as possible, but, well. Fail. > If you are ok with this, I can look after coordinating the details of the > transition. Yay! A game of kick-the-baby! Hop, your turn. From jrennie at gmail.com Tue Apr 12 12:45:20 2011 From: jrennie at gmail.com (Jason Rennie) Date: Tue, 12 Apr 2011 12:45:20 -0400 Subject: [SciPy-User] Passing fmin_cg a gradient In-Reply-To: References: Message-ID: On Mon, Apr 11, 2011 at 8:52 PM, Patrick Holvey wrote: > Should the gradient be returned as a list of arrays (since there would be > an xyz magnitude associated with each atom position) or should it be in some > other form that I'm not thinking of? Does anyone have any experience with > this? I've tried to figure out what I should be passing from online > examples of fmin_cg usage, but I'm not really seeing examples that are able > to be extended to this problem in a straightforward manner. > IIUC, it should be the same form as your "f" argument. I usually flatten my parameters and use a 1-d array. I wouldn't be surprised if a 1-d ndarray is required, but I'm not sure. Cheers, Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Tue Apr 12 13:12:55 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 12 Apr 2011 12:12:55 -0500 Subject: [SciPy-User] What "Array" means In-Reply-To: References: Message-ID: On Mon, Apr 11, 2011 at 3:08 PM, Bruce Southey wrote: > On Fri, Apr 8, 2011 at 5:45 AM, ? wrote: >> On Fri, Apr 8, 2011 at 6:14 AM, Timothy Wu <2huggie at gmail.com> wrote: >>> Hi I am trying to run Scipy's D'Agostino's normality test as documented here >>> http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.normaltest.html >>> >>> For the array argument I tried something like this >>> scipy.array([1,2,3]) >>> or >>> numpy.array([1,2,3]) >>> >>> and axis ignored. >>> >>> But with both method the test fails: >>> >>> File "/usr/lib/python2.6/dist-packages/scipy/stats/mstats_basic.py", line >>> 1546, in kurtosistest >>> ??? n = a.count(axis=axis).astype(float) >>> AttributeError: 'int' object has no attribute 'astype' >>> >>> I'm not familiar with numpy nor scipy. What exactly should I put in there? >> >> >> It looks like mstats.normaltest only works with 2-dimensional arrays, >> stats.normaltest works with 1-dimensional arrays. >> >> rvs[:,None] ?in the example below adds an additional axis, so that it >> is a column array with shape (20,1) >> If you don't need the masked array version, then you can use stats.normaltest >> >> I haven't looked at the source yet, but this looks like a bug to me. >> >>>>> rvs = np.random.randn(20) >>>>> rvs >> array([ 0.02724005, -0.17836266, ?0.40530377, ?1.313246 ?, ?0.74069068, >> ? ? ? -0.69010129, -0.24958557, -2.28311759, ?0.10525733, ?0.07986322, >> ? ? ? -0.87282545, -1.41364294, ?1.16027037, ?0.23541801, -0.06663458, >> ? ? ? ?0.39173207, ?0.06979893, ?0.4400277 , -1.29361117, -1.71524228]) >>>>> stats.normaltest(rvs) >> (1.7052869564079727, 0.42628656195988301) >>>>> stats.mstats.normaltest(rvs[:,None]) >> (masked_array(data = [1.70528695641], >> ? ? ? ? ? ? mask = [False], >> ? ? ? fill_value = 1e+20) >> , masked_array(data = [ 0.42628656], >> ? ? ? ? ? ? mask = False, >> ? ? ? fill_value = 1e+20) >> ) >>>>> stats.mstats.normaltest(rvs) >> >> Traceback (most recent call last): >> ?File "", line 1, in >> ? ?stats.mstats.normaltest(rvs) >> ?File "C:\Programs\Python27\lib\site-packages\scipy\stats\mstats_basic.py", >> line 1642, in normaltest >> ? ?k,_ = kurtosistest(a,axis) >> ?File "C:\Programs\Python27\lib\site-packages\scipy\stats\mstats_basic.py", >> line 1618, in kurtosistest >> ? ?n = a.count(axis=axis).astype(float) >> AttributeError: 'int' object has no attribute 'astype' >> >> Josef >> > > Yes that is a bug so can someone create a ticket? (I don't have time today.) > That occurs because ma.count() returns either an int (which causes the > bug) or a ndarray. Actually that '.astype(float)' is probably not > needed because as far as I can determine that every usage of an 'n' as > an integer should still results in a float. This is now ticket 1424 with a patch: http://projects.scipy.org/scipy/ticket/1424 It did require a second change that I commented because the code needs to index an array. > > There is also a second 'bug' because n must be greater than 3. I was > looking for that because estimating kurtosis needs more than 3 > observations: > "This bias-corrected formula requires that X contain at least four elements." > http://www.mathworks.com/help/toolbox/stats/kurtosis.html > > This a different ticket because we need to catch the cases when only > one particular 'column' has less than 4 but the other are fine. > > >>>> rvs = np.random.randn(20,10) >>>> stats.mstats.normaltest(rvs, axis=0) > (masked_array(data = [0.713606808604 0.132722315345 7.78660833457 > 5.38597554393 0.725711290319 > ?0.172342343314 4.02320908322 1.46363950653 3.79550214574 0.293759931912], > ? ? ? ? ? ? mask = [False False False False False False False False > False False], > ? ? ? fill_value = 1e+20) > , masked_array(data = [ 0.69991008 ?0.93579283 ?0.0203779 ? 0.06767843 > ?0.69568685 ?0.91743718 > ?0.13377386 ?0.48103283 ?0.14990537 ?0.86339761], > ? ? ? ? ? ? mask = False, > ? ? ? fill_value = 1e+20) > ) >>>> stats.mstats.normaltest(rvs, axis=1) > (masked_array(data = [0.314582042621 0.4436261479 2.98149400163 > 2.02242070422 3.46138431999 > ?9.94304440942 0.026055683609 5.7060731383 1.03808026381 0.169589515995 > ?10.5681767508 1.28212296678 3.7013014714 0.43713740004 3.62659584833 > ?0.289410600885 1.46353531025 0.745198884215 1.51022347547 0.00707268228071], > ? ? ? ? ? ? mask = [False False False False False False False False > False False False False > ?False False False False False False False False], > ? ? ? fill_value = 1e+20) > , masked_array(data = [ 0.85445536 ?0.80106509 ?0.22520436 ?0.36377841 > ?0.17716174 ?0.00693259 > ?0.98705665 ?0.05766894 ?0.59509148 ?0.91870082 ?0.00507165 ?0.52673301 > ?0.15713488 ?0.80366827 ?0.16311531 ?0.86527725 ?0.48105789 ?0.68894114 > ?0.4699581 ? 0.9964699 ], > ? ? ? ? ? ? mask = False, > ? ? ? fill_value = 1e+20) > ) >>>> stats.mstats.normaltest(rvs, axis=None) > Traceback (most recent call last): > ?File "", line 1, in > ?File "/usr/lib64/python2.7/site-packages/scipy/stats/mstats_basic.py", > line 1649, in normaltest > ? ?k,_ = kurtosistest(a,axis) > ?File "/usr/lib64/python2.7/site-packages/scipy/stats/mstats_basic.py", > line 1625, in kurtosistest > ? ?n = a.count(axis=axis).astype(float) > AttributeError: 'int' object has no attribute 'astype' >>>> > > That is because: >>>> mrvs=rvs.view(ma.MaskedArray) >>>> type(mrvs) > >>>> type(mrvs.count(axis=0)) > >>>> type(mrvs.count(axis=1)) > >>>> type(mrvs.count(axis=None)) > > > > Bruce > This is now ticket 1425 with patches: http://projects.scipy.org/scipy/ticket/1425 However the patch for mstats_basic.py does need some work. Basically only those specific cases with less than 4 observations should be 0 not all cases. Bruce From ralf.gommers at googlemail.com Tue Apr 12 14:28:53 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 12 Apr 2011 20:28:53 +0200 Subject: [SciPy-User] [SciPy-user] Using Pyinstaller or cx_freeze with scipy In-Reply-To: <31227146.post@talk.nabble.com> References: <31227146.post@talk.nabble.com> Message-ID: On Thu, Mar 24, 2011 at 10:03 AM, rafiki38 wrote: > > Hi Craig, > I am wondering whether you were able to solve your problem, since I have > exactly the same one! (under linux platform) > ----------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib/pymodules/python2.6/cx_Freeze/initscripts/Console.py", line > 29, in > ? ?exec code in m.__dict__ > ?File "at.py", line 12, in > ? ?from scipy.signal import butter > ?File "/usr/lib/python2.6/dist-packages/scipy/signal/__init__.py", line 11, > in > ? ?from ltisys import * > ?File "/usr/lib/python2.6/dist-packages/scipy/signal/ltisys.py", line 9, in > > ? ?import scipy.interpolate as interpolate > ?File "/usr/lib/python2.6/dist-packages/scipy/interpolate/__init__.py", > line 15, in > ? ?from polyint import * > ?File "/usr/lib/python2.6/dist-packages/scipy/interpolate/polyint.py", line > 2, in > ? ?from scipy import factorial > ImportError: cannot import name factorial > ----------------------------------------------------- The fix is to change line 2 of polyint.py to "from scipy.misc import factorial". But this is already the case in 0.9.0 and current master, so just upgrading will also fix this. Ralf > > Craig Howard-2 wrote: >> >> Hello: >> >> I was attempting to use cx_freeze to create a stand-alone application >> that uses scipy, but ran into a problem. When I attempt to run the >> frozen application, I get the error: >> scipy\interpolate\polyint.py >> Cannot import name factorial >> >> I looked at the scipy package and the factorial function is in >> scipy.misc.common.py but the file scipy.interpolate.polyint.py has the >> line: >> from scipy import factorial >> >> So the polyint script is looking for the factorial function in some >> other place than its actual physical location. I guess scipy somehow >> inserts the factorial function in the correct namespace so that it >> runs correctly. Is there a way to get scipy and cx_freeze to get >> along? Thanks for your help in advance. >> >> Regards, >> Craig Howard >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > -- > View this message in context: http://old.nabble.com/Using-Pyinstaller-or-cx_freeze-with-scipy-tp27786679p31227146.html > Sent from the Scipy-User mailing list archive at Nabble.com. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From mttate at usgs.gov Tue Apr 12 14:35:13 2011 From: mttate at usgs.gov (Michael T Tate) Date: Tue, 12 Apr 2011 13:35:13 -0500 Subject: [SciPy-User] sequence verification Message-ID: I am working a QC method that checks to make sure a sequence is complete. The sequence is 12 - 0s followed by 3 - 1s, 4 - 2s, 3 - 3s and finally 3 - 1s. The sequence looks something like this: [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 2. 2. 2. 2. 3. 3. 3. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 2. 2. 2. 2. 3. 3. 3. 1. 1.] Generally, this sequence is followed except if the instrument is stopped for service or there is a power failure. I need the program to check if there are 12 zeros preceding the sequence beginning with 3 - 1s. Then need to check to make sure there are 3 - 3s followed by 4 - 2s, 3 - 3s and 2 - 1s. If this series breaks down, I want to delete the incomplete series. Any guidance on how to do this would be greatly appreciated. Thanks in advance, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From oguzyarimtepe at gmail.com Tue Apr 12 15:07:46 2011 From: oguzyarimtepe at gmail.com (Oguz Yarimtepe) Date: Tue, 12 Apr 2011 22:07:46 +0300 Subject: [SciPy-User] pylab plotting problem Message-ID: <20110412220746.9d5eb5fc.oguzyarimtepe@gmail.com> Hi, At my application i am using pylab for plotting float values. I have a for loop and inside i have a line for plotting: P.plot(x,y,'s', alpha = 0.7, ms = 3) Everytime the loop iterates x and y value sqeuncer are changing each time. How can i plot all the values created? When i just use plot function for x and y i can see that some range values are not at the graph. Will be happy if someone gives a guidance. -- Oguz Yarimtepe From e.antero.tammi at gmail.com Tue Apr 12 15:34:26 2011 From: e.antero.tammi at gmail.com (eat) Date: Tue, 12 Apr 2011 22:34:26 +0300 Subject: [SciPy-User] sequence verification In-Reply-To: References: Message-ID: Hi Mike, On Tue, Apr 12, 2011 at 9:35 PM, Michael T Tate wrote: > > > I am working a QC method that checks to make sure a sequence is complete. > > The sequence is 12 - 0s followed by 3 - 1s, 4 - 2s, 3 - 3s and finally 3 - > 1s. The sequence looks something like this: > > [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 2. 2. 2. 2. > 3. 3. 3. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. > 1. 2. 2. 2. 2. 3. 3. 3. 1. 1.] > > Generally, this sequence is followed except if the instrument is stopped > for service or there is a power failure. > > I need the program to check if there are 12 zeros preceding the sequence > beginning with 3 - 1s. Then need to check to make sure there are 3 - 3s > followed by 4 - 2s, 3 - 3s and 2 - 1s. If this series breaks down, I want to > delete the incomplete series. > > Any guidance on how to do this would be greatly appreciated. Here is a slightly simplified example (using integers) to demonstrate how to find out if your sequence is following some (strict) pattern. In []: seq Out[]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1]) In []: pat Out[]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1]) In []: n= len(seq)/ len(pat) # obviously len(pat) must integer multiple of len(seq) In []: not any(seq- pat[None, :].repeat(n, axis= 0).ravel()) Out[]: True Regards, eat > > > Thanks in advance, > > Mike > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Tue Apr 12 16:23:30 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Tue, 12 Apr 2011 16:23:30 -0400 Subject: [SciPy-User] sequence verification In-Reply-To: References: Message-ID: On Tue, Apr 12, 2011 at 3:34 PM, eat wrote: > Hi Mike, > > On Tue, Apr 12, 2011 at 9:35 PM, Michael T Tate wrote: >> >> >> I am working a QC method that checks to make sure a sequence is complete. >> >> The sequence is 12 - 0s followed by 3 - 1s, 4 - 2s, 3 - 3s and finally 3 - >> 1s. The sequence looks something like this: >> >> [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 2. 2. 2. >> 2. 3. 3. 3. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. >> 1. 1. 2. 2. 2. 2. 3. 3. 3. 1. 1.] >> >> Generally, this sequence is followed except if the instrument is stopped >> for service or there is a power failure. >> >> I need the program to check if there are 12 zeros preceding the sequence >> beginning with 3 - 1s. Then need to check to make sure there are 3 - 3s >> followed by 4 - 2s, 3 - 3s and 2 - 1s. If this series breaks down, I want to >> delete the incomplete series. >> >> Any guidance on how to do this would be greatly appreciated. > > Here is a slightly simplified example (using integers) to demonstrate how to > find out if your sequence is following some (strict) pattern. > In []: seq > Out[]: > array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, > 1, 1, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, > 1, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, > 1, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, > 1, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, > 1]) > In []: pat > Out[]: > array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, > 1, 1]) > In []: n= len(seq)/ len(pat) # obviously len(pat) must integer multiple of > len(seq) > In []: not any(seq- pat[None, :].repeat(n, axis= 0).ravel()) > Out[]: True > Regards, > eat >> I've been playing around with similar problems lately. Would something like this work? Fair warning, there is probably an easier way using the signal processing tools. I'm still learning my way around. In [130]: seq Out[130]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1]) In [131]: pat Out[131]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 1, 1]) Get the number of sequences In [132]: np.sum(np.correlate(pat,seq,mode='valid')==np.sum(pat**2)) Out[132]: 5 Are they continguous? In [133]: np.all(np.diff(np.where(np.correlate(pat,seq,mode='valid')==np.sum(pat**2))) == len(pat)) Out[133]: True Make a new sequence with some noise / bad values In [134]: seq2 = np.hstack((seq, np.random.randint(0,10,size=35), seq)) How many patterns? In [135]: np.sum(np.correlate(pat,seq2,mode='valid')==np.sum(pat**2)) Out[135]: 10 Are they contiguous? In [136]: np.all(np.diff(np.where(np.correlate(pat,seq2,mode='valid')==np.sum(pat**2))) == len(pat)) Out[136]: False What's the index of the contiguous ones? First get where they start In [137]: idx = np.where(np.correlate(pat,seq2,mode='valid')==np.sum(pat**2))[0] Then make index sequences from the start of each series of len(pat) In [138]: idx = np.hstack([np.arange(i,i+len(pat)) for i in idx]) Get only the series you want In [139]: goodseq = seq2[idx] In [140]: np.all(goodseq[:len(seq)] == seq) Out[140]: True In [141]: np.all(goodseq[len(seq):] == seq) Out[141]: True Skipper -------------- next part -------------- An HTML attachment was scrubbed... URL: From gruben at bigpond.net.au Tue Apr 12 22:26:09 2011 From: gruben at bigpond.net.au (gary ruben) Date: Wed, 13 Apr 2011 12:26:09 +1000 Subject: [SciPy-User] Shifting peak to origin in fourier space In-Reply-To: References: Message-ID: How about the numpy.roll function? On Mon, Apr 11, 2011 at 7:23 PM, morovia morovia wrote: > Hello > > ??? I am trying to demodulate a signal. > > ??? Here I need to translate (shift) the peak at the desired frequency > in the fourier space to the origin. > > Can anyone suggest me how to do it. > > Thanks > Best regards > Morovia. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From david_baddeley at yahoo.com.au Tue Apr 12 23:20:52 2011 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Tue, 12 Apr 2011 20:20:52 -0700 (PDT) Subject: [SciPy-User] Shifting peak to origin in fourier space In-Reply-To: References: Message-ID: <6133.57054.qm@web113417.mail.gq1.yahoo.com> normal way of demodulating would be to multiply with a sine/cosine/complex exponential at the carrier frequency. Your exact choice will probably depend on what type of modulation it is and whether you're interested in the phase information or not. cheers, David ----- Original Message ---- From: gary ruben To: SciPy Users List Cc: morovia morovia Sent: Wed, 13 April, 2011 2:26:09 PM Subject: Re: [SciPy-User] Shifting peak to origin in fourier space How about the numpy.roll function? On Mon, Apr 11, 2011 at 7:23 PM, morovia morovia wrote: > Hello > > I am trying to demodulate a signal. > > Here I need to translate (shift) the peak at the desired frequency > in the fourier space to the origin. > > Can anyone suggest me how to do it. > > Thanks > Best regards > Morovia. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From ben.harrison at liquidmesh.com Tue Apr 12 23:54:54 2011 From: ben.harrison at liquidmesh.com (Ben Harrison) Date: Wed, 13 Apr 2011 13:54:54 +1000 Subject: [SciPy-User] genfromtxt but I need to search file first In-Reply-To: References: Message-ID: On 13/04/11 00:25, Warren Weckesser wrote: > By any changes is this a LAS file? Take a look here: > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055563.html > and at my response. I included a "quick and dirty" implementation of a > LAS reader. (I have a nicer version in progress, but it will be a few > days before I can finish it.) In the version attached to that email, I > just use loadtxt() to read the data after the ~A line. > > Warren > Yes it is a LAS file, they are pesky things. I'll try your script, or steal methods from it! Will you be publishing your nicer script anywhere in particular? Could you let me know somehow when you have? Thanks, Ben. From yosefmel at post.tau.ac.il Wed Apr 13 02:36:10 2011 From: yosefmel at post.tau.ac.il (Yosef Meller) Date: Wed, 13 Apr 2011 09:36:10 +0300 Subject: [SciPy-User] fmin_bfgs In-Reply-To: References: Message-ID: <201104130936.11245.yosefmel@post.tau.ac.il> On ??? ????? 31 ??? 2011 14:54:15 Dirk Nachbar wrote: > I am using fmin_bfgs to minimise a function, I put a print in the > function so I can see the steps. It does show me that the value goes > down but then goes back to the starting value and shows that it is > optimised. But from the print I know it's not the true min, how can I > make the function stop at the right min. I am not specifying a fprime > function. The function may be called several times around a point to see that no improvement is possible (even more times if you're using it with no gradient function). Check which value is returned, not which one is printed last. Yosef. From midel at sfr.fr Wed Apr 13 03:59:19 2011 From: midel at sfr.fr (midel) Date: Wed, 13 Apr 2011 09:59:19 +0200 (CEST) Subject: [SciPy-User] Linux and windows scipy code compatible or not ? Message-ID: <837333.424001302681559475.JavaMail.www@wsfrf1236> Hello every body, I keep on discovering Scipy and trying to use it the proper way ! I have met a new problem which is, for me, really a big one... I switch from windws to linux all week long, and I need my codes to run "OS independantly". In one of my programs written under Win7, I import data from a text file using numpy.genfromtxt (or scipy.genfromtxt which looks identical) : ???? data=np.genfromtxt("blabla.txt",skip_header=17,skip_footer=2) And it works perfectly. When I used this program under linux, I got an error telling me that the options skip_header and skip_footer do not exist ! Indeed, the definition of genfromtxt seems different : Under windows Python 2.7.1 / numpy 1.5.1 numpy.genfromtxt = genfromtxt(fname, dtype=, comments='#', delimit er=None, skiprows=0, skip_header=0, skip_footer=0, converters=None, missing='', missing_values=None, filling_values=None, usecols=None, names=None, excludelist= None, deletechars=None, replace_space='_', autostrip=False, case_sensitive=True, ?defaultfmt='f%i', unpack=None, usemask=False, loose=True, invalid_raise=True) ??? Load data from a text file, with missing values handled as specified. Under linux Python 2.6.6 / numpy 1.3.0 numpy.genfromtxt = genfromtxt(fname, dtype=, comments='#', delimit er=None, skiprows=0, converters=None, missing='', missing_values=None, usecols=None, names=None, excludelist=None, deletechars=None, case_sensitive=True, unpack=None, usemask=False, loose=True) ??? Load data from a text file. Neither skip_header nor skip_footer. What's the reason for this difference ? Numpy version ? midel -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Wed Apr 13 04:13:53 2011 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 13 Apr 2011 10:13:53 +0200 Subject: [SciPy-User] Linux and windows scipy code compatible or not ? In-Reply-To: <837333.424001302681559475.JavaMail.www@wsfrf1236> References: <837333.424001302681559475.JavaMail.www@wsfrf1236> Message-ID: yes, you need numpy 1.5.1 on both. -Sebastian On Wed, Apr 13, 2011 at 9:59 AM, midel wrote: > Hello every body, > > I keep on discovering Scipy and trying to use it the proper way ! > > I have met a new problem which is, for me, really a big one... > > I switch from windws to linux all week long, and I need my codes to run "OS > independantly". In one of my programs written under Win7, I import data from > a text file using numpy.genfromtxt (or scipy.genfromtxt which looks > identical) : > > ???? data=np.genfromtxt("blabla.txt",skip_header=17,skip_footer=2) > > And it works perfectly. > > When I used this program under linux, I got an error telling me that the > options skip_header and skip_footer do not exist ! Indeed, the definition of > genfromtxt seems different : > > > Under windows Python 2.7.1 / numpy 1.5.1 > > numpy.genfromtxt = genfromtxt(fname, dtype=, comments='#', > delimit > er=None, skiprows=0, skip_header=0, skip_footer=0, converters=None, > missing='', > missing_values=None, filling_values=None, usecols=None, names=None, > excludelist= > None, deletechars=None, replace_space='_', autostrip=False, > case_sensitive=True, > ?defaultfmt='f%i', unpack=None, usemask=False, loose=True, > invalid_raise=True) > ??? Load data from a text file, with missing values handled as specified. > > > Under linux Python 2.6.6 / numpy 1.3.0 > > numpy.genfromtxt = genfromtxt(fname, dtype=, comments='#', > delimit > er=None, skiprows=0, converters=None, missing='', missing_values=None, > usecols=None, names=None, excludelist=None, deletechars=None, > case_sensitive=True, unpack=None, usemask=False, loose=True) > ??? Load data from a text file. > > Neither skip_header nor skip_footer. > > What's the reason for this difference ? Numpy version ? > > midel > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From pierre.raybaut at gmail.com Wed Apr 13 07:43:43 2011 From: pierre.raybaut at gmail.com (Pierre Raybaut) Date: Wed, 13 Apr 2011 13:43:43 +0200 Subject: [SciPy-User] ANN: Python(x,y) 2.6.6.0 Message-ID: Hi all, Gabi Davar and I are pleased to announce that Python(x,y) 2.6.6.0 has been released. This is the last major release based on Python 2.6. We will now concentrate on building Python-2.7-based plugins for the forthcoming Python(x,y) 2.7.X.0 (X >= 1) release. Python(x,y) is a free Python distribution providing a ready-to-use scientific development software for numerical computations, data analysis and data visualization based on Python programming language, Qt graphical user interfaces (and development framework), Eclipse integrated development environment and Spyder interactive development environment. Its purpose is to help scientific programmers used to interpreted languages (such as MATLAB or IDL) or compiled languages (C/C++ or Fortran) to switch to Python. Here are the plugins updated in version 2.6.6.0: * Python 2.6.6 (Upgrade available only with a complete install of Python(x,y)) * Spyder 2.0.10 * guidata 1.3.0 * guiqwt 2.1.0 * xy 1.2.5 * xydoc 1.0.4 * PyQt 4.8.3 * QtHelp 4.7.1 * PyQwt 5.2.1 * wxPython 2.8.11.0 * numexpr 1.4.2 * SciPy 0.9.0 * Matplotlib 1.0.1 * PIL 1.1.7.1 * Pyreadline 1.6 * Enthought Tool Suite 3.6.0 * VTK 5.6.1.1 * PyOpenGL 3.0.1.1 * VPython 5.41 * NetworkX 1.4 * MDP 3.1 * h5py 1.3.1 * pyhdf 0.8.3 * netcdf4 0.9.3 * GDAL 1.8.0 * PP 1.6.1 * Pywin32 2.15 * Cython 0.14.1 * Sphinx 1.0.7 * jinja2 2.5.5 * pygments 1.4.0 * simplejson 2.1.3 * nose 1.0.0 * pylint 0.23.0 * Veusz 1.10 * SciTE 2.25 * MinGW 4.5.2 * SWIG 2.0.1 * vitables 2.1 * pyparallel 0.2.0.1 * Console 2.0.147.1 -Gabi Davar and Pierre Raybaut From ghostility at gmail.com Tue Apr 12 09:39:41 2011 From: ghostility at gmail.com (Ghostly) Date: Tue, 12 Apr 2011 15:39:41 +0200 Subject: [SciPy-User] Connecting Windows PCs for parallel computing with iPython? In-Reply-To: <20110330023112.043A.5FE01CF9@gmail.com> References: <20110330023112.043A.5FE01CF9@gmail.com> Message-ID: <20110412153937.DB21.5FE01CF9@gmail.com> Hi, as anyone can see from the message date, I posted this mail two weeks ago. 3 days ago this message also appeared on IPython mailing list where it is discussed. I didn't know that messages travel so long ;) and anyone can track the positive answer there, together with problems trying to connect Windows PCs with latest IPyton dev version based on 0MQ instead Twisted Cheers From pholvey at gmail.com Tue Apr 12 12:49:06 2011 From: pholvey at gmail.com (Patrick Holvey) Date: Tue, 12 Apr 2011 12:49:06 -0400 Subject: [SciPy-User] Passing fmin_cg a gradient In-Reply-To: References: Message-ID: Hey Jason, thanks for getting back to me. You're correct. I was not thinking clearly until I walked out the door this morning and realized that I should just be returning a 1D array of the same size as the position array with each gradient element corresponding to a position element. So simple, yet it took me so long to grasp it. Thanks for helping out! Patrick On Tue, Apr 12, 2011 at 12:45 PM, Jason Rennie wrote: > On Mon, Apr 11, 2011 at 8:52 PM, Patrick Holvey wrote: > >> Should the gradient be returned as a list of arrays (since there would be >> an xyz magnitude associated with each atom position) or should it be in some >> other form that I'm not thinking of? Does anyone have any experience with >> this? I've tried to figure out what I should be passing from online >> examples of fmin_cg usage, but I'm not really seeing examples that are able >> to be extended to this problem in a straightforward manner. >> > > IIUC, it should be the same form as your "f" argument. I usually flatten > my parameters and use a 1-d array. I wouldn't be surprised if a 1-d ndarray > is required, but I'm not sure. > > Cheers, > > Jason > > -- > Jason Rennie > Research Scientist, ITA Software > 617-714-2645 > http://www.itasoftware.com/ > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Patrick Holvey Graduate Student Dept. of Materials Science and Engineering Johns Hopkins University pholvey1 at jhu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From jallikattu at googlemail.com Wed Apr 13 00:13:07 2011 From: jallikattu at googlemail.com (morovia morovia) Date: Wed, 13 Apr 2011 09:43:07 +0530 Subject: [SciPy-User] Shifting peak to origin in fourier space In-Reply-To: References: Message-ID: Gary : > How about the numpy.roll function? > Thanks for this info. I will try this. Thanks Best regards Morovia. > > On Mon, Apr 11, 2011 at 7:23 PM, morovia morovia > wrote: > > Hello > > > > I am trying to demodulate a signal. > > > > Here I need to translate (shift) the peak at the desired frequency > > in the fourier space to the origin. > > > > Can anyone suggest me how to do it. > > > > Thanks > > Best regards > > Morovia. > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.harrison at liquidmesh.com Wed Apr 13 00:19:58 2011 From: ben.harrison at liquidmesh.com (Ben Harrison) Date: Wed, 13 Apr 2011 14:19:58 +1000 Subject: [SciPy-User] optional args to genfromtxt unavailable? Message-ID: <4DA5246E.7060903@liquidmesh.com> hi, i've had errors returned when trying to use some of the optional arguments to genfromtxt in ipython (with pylab). Here's a small sample: In [17]: import numpy as n In [18]: In [19]: directory = '/work/ben/' In [20]: filename = directory + 'log.csv' In [21]: In [22]: tdata = n.genfromtxt(filename, skiprows=2, delimiter=',', autostrip=True) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /work/ben/ in () TypeError: genfromtxt() got an unexpected keyword argument 'autostrip' I also get similar errors when using skip_header, filling_values (but not missing_values), and possibly others. Why is this? harb at joan:~$ python --version Python 2.6.5 In [27]: np.__version__ Out[27]: '1.3.0' Ben. From pgmdevlist at gmail.com Wed Apr 13 09:38:55 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 13 Apr 2011 15:38:55 +0200 Subject: [SciPy-User] optional args to genfromtxt unavailable? In-Reply-To: <4DA5246E.7060903@liquidmesh.com> References: <4DA5246E.7060903@liquidmesh.com> Message-ID: On Apr 13, 2011, at 6:19 AM, Ben Harrison wrote: > > TypeError: genfromtxt() got an unexpected keyword argument 'autostrip' > > I also get similar errors when using skip_header, filling_values (but > not missing_values), and possibly others. Why is this? > > harb at joan:~$ python --version > Python 2.6.5 > > In [27]: np.__version__ > Out[27]: '1.3.0' Because very likely, the arguments you're trying to use were introduced in more recent versions of numpy: 1.3 is already a bit old. Get the list of arguments recognized by your version through the help (type `np.genfromtxt?` in ipython). From oguzyarimtepe at gmail.com Wed Apr 13 09:49:34 2011 From: oguzyarimtepe at gmail.com (=?UTF-8?B?T8SfdXogWWFyxLFtdGVwZQ==?=) Date: Wed, 13 Apr 2011 16:49:34 +0300 Subject: [SciPy-User] pyplot problem Message-ID: I didn't understand why i can not see 20 dots with the below command, but i just see 3 dots at the graph. import pylab as P import numpy as N a1 = N.ndarray(shape=(10,2)) x = a1[:,0] y = a1[:,1] a2 = N.ndarray(shape=(10,2))+10 xx = a2[:,0] yy = a2[:,1] n_x = N.append(x,xx) n_y = N.append(y,yy) P.scatter(n_x, n_y, alpha=0.5) P.xlim(xmin=n_x.min()*10, xmax=n_x.max()*10) P.ylim(ymin=n_y.min()*10, ymax=n_y.max()*10) P.plot(n_x, n_y, 's', alpha=0.7, ms=3) P.show() Normally i have 20 different points to be plotted but because they are so close values it just plots three of them. Any idea how i can fix it? From robert.kern at gmail.com Wed Apr 13 11:04:49 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 13 Apr 2011 10:04:49 -0500 Subject: [SciPy-User] pyplot problem In-Reply-To: References: Message-ID: On Wed, Apr 13, 2011 at 08:49, O?uz Yar?mtepe wrote: > I didn't understand why i can not see 20 dots with the below command, > but i just see 3 dots at the graph. > > import pylab as P The matplotlib list is over here: https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From ckkart at hoc.net Wed Apr 13 15:41:07 2011 From: ckkart at hoc.net (Christian K.) Date: Wed, 13 Apr 2011 21:41:07 +0200 Subject: [SciPy-User] Passing fmin_cg a gradient In-Reply-To: References: Message-ID: Hi Patrick, Am 12.04.11 02:52, schrieb Patrick Holvey: > Hi everybody, > > I'm using fmin_cg to minimize atom positions (xyz coordinates) using a > Keating potential (one term for the bond stretching and one for the bond > angle bending). It's been great but now I'm trying to speed up the > I'm also open to alternate optimization calls if anyone has > recommendations. I've got both an analytic potential and gradient so I > do have an fprime term if the optimization call needs it. However, I'd > appreciate guidance on what form the gradient function should return > since I don't want to be dealing with a new optimization method and be > caught in the same spot again. I have been working on the same thing a couple of years ago during my PhD. In the end I found, that fmin_tnc is performing best among all minimizers available in scipy. I implemented the potential in C using scipy.weave and could manage up to 2 millions atoms in reasonable time. Just curious, what are you working on exactly and what do you want to find out? Regards, Christian From cpeters at edisonmission.com Wed Apr 13 16:03:49 2011 From: cpeters at edisonmission.com (Christopher Peters) Date: Wed, 13 Apr 2011 16:03:49 -0400 Subject: [SciPy-User] Separate scales for left and right axis broken Message-ID: All, I am trying to run the code at http://pytseries.sourceforge.net/lib.plotting.examples.html#separate-scales-for-left-and-right-axis to generate separate scales for left and right axes for tsplot in scikits.timeseries.lib.plotlib, but I am getting an error. Package versions: Numpy: 1.5.1 Matplotlib: 1.0.1 Scikits Timeseries: 0.91.3 Code to test: import numpy as np print np.__version__ import numpy.ma as ma import matplotlib.pyplot as plt print matplotlib.__version__ import scikits.timeseries as ts print ts.__version__ import scikits.timeseries.lib.plotlib as tpl # generate some random data data1 = np.cumprod(1 + np.random.normal(0, 1, 300)/100) data2 = np.cumprod(1 + np.random.normal(0, 1, 300)/100)*100 start_date = ts.Date(freq='M', year=1982, month=1) series1 = ts.time_series(data1, start_date=start_date-50) series2 = ts.time_series(data2, start_date=start_date) fig = tpl.tsfigure() fsp = fig.add_tsplot(111) # plot series on left axis fsp.tsplot(series1, 'b-', label='<- left series') fsp.set_ylim(ma.min(series1.series), ma.max(series1.series)) # create right axis fsp_right = fsp.add_yaxis(position='right', yscale='log') # plot series on right axis fsp_right.tsplot(series2, 'r-', label='-> right series') fsp_right.set_ylim(ma.min(series2.series), ma.max(series2.series)) # setup legend fsp.legend( (fsp.lines[-1], fsp_right.lines[-1]), (fsp.lines[-1].get_label(), fsp_right.lines[-1].get_label()), ) plt.show() Traceback (most recent call last): File "", line 23, in fsp_right = fsp.add_yaxis(position='right', yscale='log') File "C:\Python27\lib\site-packages\scikits\timeseries\lib\plotlib.py", line 1196, in add_yaxis fsp_alt_args = (fsp._rows, fsp._cols, fsp._num + 1) AttributeError: 'TimeSeriesPlot' object has no attribute '_rows' My guess is that the newest version of matplotlib (1.0.1) removed the _rows (_cols, and _num as well) attribute for the Subplot base class, but this was not propagated to plotlib.py. Looking at the revision log at http://projects.scipy.org/scikits/log/trunk/timeseries/scikits/timeseries/lib/plotlib.py , it appears that 20 months ago there were some fixes for matplotlib version 0.99.0. It seems that this question was already posed by Dave Hirschfeld on 2010 Nov 8 (a post entitled "TimeSeries Plotting Broken?" at http://permalink.gmane.org/gmane.comp.python.scientific.devel/14645), but there were no replies. Any help on this would be greatly appreciated! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjstickel at vcn.com Wed Apr 13 16:55:09 2011 From: jjstickel at vcn.com (Jonathan Stickel) Date: Wed, 13 Apr 2011 14:55:09 -0600 Subject: [SciPy-User] ANN: scikit.datasmooth 0.6.1 Message-ID: <4DA60DAD.80104@vcn.com> The scikit "datasmooth" has been updated to version 0.6.1. The primary purpose of this release is the addition of html documentation (generated by pydoc) and examples. Other notable changes can be found in the NEWS file. A source distribution can be found at: http://pypi.python.org/pypi/scikits.datasmooth and the development tree is on github: https://github.com/jjstickel/scikit-datasmooth This scikit is intended to include numerical methods for smoothing data that are not yet in scipy. Currently, only a regularization method for smoothing 1-D (y vs. x) data is included, for which cross-validation for determining the optimization parameter has been implemented. Constrained smoothing is also available but requires the cvxopt package. Comments and questions are welcome, as well as contributions of other smoothing methods. Regards, Jonathan From nwagner at iam.uni-stuttgart.de Thu Apr 14 03:10:44 2011 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 14 Apr 2011 09:10:44 +0200 Subject: [SciPy-User] ANN: scikit.datasmooth 0.6.1 In-Reply-To: <4DA60DAD.80104@vcn.com> References: <4DA60DAD.80104@vcn.com> Message-ID: On Wed, 13 Apr 2011 14:55:09 -0600 Jonathan Stickel wrote: > The scikit "datasmooth" has been updated to version >0.6.1. The primary > purpose of this release is the addition of html >documentation (generated > by pydoc) and examples. Other notable changes can be >found in the NEWS > file. > > A source distribution can be found at: > > http://pypi.python.org/pypi/scikits.datasmooth > > and the development tree is on github: > > https://github.com/jjstickel/scikit-datasmooth > > > This scikit is intended to include numerical methods for >smoothing data > that are not yet in scipy. Currently, only a >regularization method for > smoothing 1-D (y vs. x) data is included, for which >cross-validation for > determining the optimization parameter has been >implemented. > Constrained smoothing is also available but requires the >cvxopt package. > > Comments and questions are welcome, as well as >contributions of other > smoothing methods. > > Regards, > Jonathan Hi Jonathan, Just now I installed your scikit and got a syntax error. See below for details. Nils python setup.py install --prefix=$HOME/local running install running bdist_egg running egg_info running build_src build_src creating scikits.datasmooth.egg-info writing scikits.datasmooth.egg-info/PKG-INFO writing namespace_packages to scikits.datasmooth.egg-info/namespace_packages.txt writing top-level names to scikits.datasmooth.egg-info/top_level.txt writing dependency_links to scikits.datasmooth.egg-info/dependency_links.txt writing manifest file 'scikits.datasmooth.egg-info/SOURCES.txt' reading manifest file 'scikits.datasmooth.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'scikits.datasmooth.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib creating build/lib/scikits copying scikits/__init__.py -> build/lib/scikits creating build/lib/scikits/datasmooth copying scikits/datasmooth/regularsmooth.py -> build/lib/scikits/datasmooth copying scikits/datasmooth/__init__.py -> build/lib/scikits/datasmooth creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/scikits creating build/bdist.linux-x86_64/egg/scikits/datasmooth copying build/lib/scikits/datasmooth/regularsmooth.py -> build/bdist.linux-x86_64/egg/scikits/datasmooth copying build/lib/scikits/datasmooth/__init__.py -> build/bdist.linux-x86_64/egg/scikits/datasmooth copying build/lib/scikits/__init__.py -> build/bdist.linux-x86_64/egg/scikits byte-compiling build/bdist.linux-x86_64/egg/scikits/datasmooth/regularsmooth.py to regularsmooth.pyc byte-compiling build/bdist.linux-x86_64/egg/scikits/datasmooth/__init__.py to __init__.pyc File "build/bdist.linux-x86_64/egg/scikits/datasmooth/__init__.py", line 1 __version__ = 0.6.1 ^ SyntaxError: invalid syntax byte-compiling build/bdist.linux-x86_64/egg/scikits/__init__.py to __init__.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying scikits.datasmooth.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying scikits.datasmooth.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying scikits.datasmooth.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying scikits.datasmooth.egg-info/namespace_packages.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying scikits.datasmooth.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying scikits.datasmooth.egg-info/zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO creating dist creating 'dist/scikits.datasmooth-0.6.1-py2.5.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing scikits.datasmooth-0.6.1-py2.5.egg Copying scikits.datasmooth-0.6.1-py2.5.egg to /data/home/nwagner/local/lib/python2.5/site-packages Adding scikits.datasmooth 0.6.1 to easy-install.pth file Installed /data/home/nwagner/local/lib/python2.5/site-packages/scikits.datasmooth-0.6.1-py2.5.egg Processing dependencies for scikits.datasmooth==0.6.1 Finished processing dependencies for scikits.datasmooth==0.6.1 From joscha.schmiedt at googlemail.com Thu Apr 14 06:20:22 2011 From: joscha.schmiedt at googlemail.com (Joscha Schmiedt) Date: Thu, 14 Apr 2011 11:20:22 +0100 Subject: [SciPy-User] Why not filtfilt in decimate function? Message-ID: Dear list, I just noticed that the decimate function from scipy.signal does not use filtfilt to filter the signal but lfilter and thereby introduces nasty phase shifts. Is there a reason why this is done? Kind regards, Joscha -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwlodarczak at uni-bielefeld.de Thu Apr 14 09:09:54 2011 From: mwlodarczak at uni-bielefeld.de (Marcin Wlodarczak) Date: Thu, 14 Apr 2011 15:09:54 +0200 Subject: [SciPy-User] Setting time step for moving windows Message-ID: <3759_1302786587_ZZh0o3p2Mt60w.00_4DA6F222.8060901@uni-bielefeld.de> Hi, Is there any way of modifying the time step for the moving window functions? That would be most useful but I cannot see it documented anywhere. Best wishes, Marcin From mwlodarczak at uni-bielefeld.de Thu Apr 14 09:12:53 2011 From: mwlodarczak at uni-bielefeld.de (Marcin Wlodarczak) Date: Thu, 14 Apr 2011 15:12:53 +0200 Subject: [SciPy-User] Setting time step for moving windows In-Reply-To: <4DA6F222.8060901@uni-bielefeld.de> References: <4DA6F222.8060901@uni-bielefeld.de> Message-ID: <17168_1302786773_ZZh0o3p4MtN0G.00_4DA6F2D5.90203@uni-bielefeld.de> On 14.04.2011 15:09, Marcin Wlodarczak wrote: > Is there any way of modifying the time step for the moving window > functions? That would be most useful but I cannot see it documented > anywhere. I meant the moving window functions from the scikits.timeseries module of course. Sorry if that wasn't clear. M. From pgmdevlist at gmail.com Thu Apr 14 09:46:03 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 14 Apr 2011 15:46:03 +0200 Subject: [SciPy-User] Setting time step for moving windows In-Reply-To: <17168_1302786773_ZZh0o3p4MtN0G.00_4DA6F2D5.90203@uni-bielefeld.de> References: <4DA6F222.8060901@uni-bielefeld.de> <17168_1302786773_ZZh0o3p4MtN0G.00_4DA6F2D5.90203@uni-bielefeld.de> Message-ID: <3F9AA360-1117-4CE3-BE6D-BA4264FEA251@gmail.com> On Apr 14, 2011, at 3:12 PM, Marcin Wlodarczak wrote: > On 14.04.2011 15:09, Marcin Wlodarczak wrote: > >> Is there any way of modifying the time step for the moving window >> functions? That would be most useful but I cannot see it documented >> anywhere. > > I meant the moving window functions from the scikits.timeseries module > of course. Sorry if that wasn't clear. What do you mean, modifying the time step? Could you give us an example of what you have in mind? From mwlodarczak at uni-bielefeld.de Thu Apr 14 10:00:52 2011 From: mwlodarczak at uni-bielefeld.de (Marcin Wlodarczak) Date: Thu, 14 Apr 2011 16:00:52 +0200 Subject: [SciPy-User] Setting time step for moving windows In-Reply-To: <3F9AA360-1117-4CE3-BE6D-BA4264FEA251@gmail.com> References: <4DA6F222.8060901@uni-bielefeld.de> <17168_1302786773_ZZh0o3p4MtN0G.00_4DA6F2D5.90203@uni-bielefeld.de> <3F9AA360-1117-4CE3-BE6D-BA4264FEA251@gmail.com> Message-ID: <3950_1302789646_ZZh0o5pcYs3pY.00_4DA6FE14.1060501@uni-bielefeld.de> On 14.04.2011 15:46, Pierre GM wrote: >>> Is there any way of modifying the time step for the moving window >>> functions? That would be most useful but I cannot see it documented >>> anywhere. >> >> I meant the moving window functions from the scikits.timeseries module >> of course. Sorry if that wasn't clear. > > What do you mean, modifying the time step? Could you give us an example of what you have in mind? By the time step I mean by how much the window each shifted each time. Say you have an array x = np.array(10) and you want a window with the width (span) of 3. With time step of 1 you get the following windows: [0, 1, 2] [1, 2, 3] [2, 3, 4] [3, 4, 5] and so on. With time step of 2 you would get [0, 1, 2] [2, 3, 4] [4, 5, 6] and so on. In other words, time step would control how much consecutive windows overlap. Perhaps there is another (better?) name for it but `time step' is used commonly in applications such as pitch extraction (see e.g. http://www.fon.hum.uva.nl/praat/manual/Time_step_settings___.html) Best, Marcin From jjstickel at vcn.com Thu Apr 14 10:02:32 2011 From: jjstickel at vcn.com (Jonathan Stickel) Date: Thu, 14 Apr 2011 08:02:32 -0600 Subject: [SciPy-User] ANN: scikit.datasmooth 0.6.1 In-Reply-To: References: Message-ID: <4DA6FE78.4010108@vcn.com> On 4/14/11 07:12 , scipy-user-request at scipy.org wrote: > Date: Thu, 14 Apr 2011 09:10:44 +0200 > From: "Nils Wagner" > Subject: Re: [SciPy-User] ANN: scikit.datasmooth 0.6.1 > > On Wed, 13 Apr 2011 14:55:09 -0600 > Jonathan Stickel wrote: >> > The scikit "datasmooth" has been updated to version >> >0.6.1. The primary >> > purpose of this release is the addition of html >> >documentation (generated >> > by pydoc) and examples. Other notable changes can be >> >found in the NEWS >> > file. >> > >> > A source distribution can be found at: >> > >> > http://pypi.python.org/pypi/scikits.datasmooth >> > >> > and the development tree is on github: >> > >> > https://github.com/jjstickel/scikit-datasmooth >> > >> > >> > This scikit is intended to include numerical methods for >> >smoothing data >> > that are not yet in scipy. Currently, only a >> >regularization method for >> > smoothing 1-D (y vs. x) data is included, for which >> >cross-validation for >> > determining the optimization parameter has been >> >implemented. >> > Constrained smoothing is also available but requires the >> >cvxopt package. >> > >> > Comments and questions are welcome, as well as >> >contributions of other >> > smoothing methods. >> > >> > Regards, >> > Jonathan > Hi Jonathan, > > Just now I installed your scikit and got a syntax error. > See below for details. > > Nils > > "build/bdist.linux-x86_64/egg/scikits/datasmooth/__init__.py", > line 1 > __version__ = 0.6.1 > ^ > SyntaxError: invalid syntax > Oops! Sorry about that. I was having some trouble with pypi and had to arbitrarily increment the version number. Apparently two periods are not allowed. Thank you very much for reporting the problem. A new source distribution (0.61) has now been uploaded and properly tested. Regards, Jonathan From pgmdevlist at gmail.com Thu Apr 14 10:23:50 2011 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 14 Apr 2011 16:23:50 +0200 Subject: [SciPy-User] Setting time step for moving windows In-Reply-To: <3950_1302789646_ZZh0o5pcYs3pY.00_4DA6FE14.1060501@uni-bielefeld.de> References: <4DA6F222.8060901@uni-bielefeld.de> <17168_1302786773_ZZh0o3p4MtN0G.00_4DA6F2D5.90203@uni-bielefeld.de> <3F9AA360-1117-4CE3-BE6D-BA4264FEA251@gmail.com> <3950_1302789646_ZZh0o5pcYs3pY.00_4DA6FE14.1060501@uni-bielefeld.de> Message-ID: <7F12C781-73C2-4DBF-98E2-E553242DCC9F@gmail.com> On Apr 14, 2011, at 4:00 PM, Marcin Wlodarczak wrote: > On 14.04.2011 15:46, Pierre GM wrote: >>>> Is there any way of modifying the time step for the moving window >>>> functions? That would be most useful but I cannot see it documented >>>> anywhere. >>> >>> I meant the moving window functions from the scikits.timeseries module >>> of course. Sorry if that wasn't clear. >> >> What do you mean, modifying the time step? Could you give us an example of what you have in mind? > > By the time step I mean by how much the window each shifted each time. Ah OK, I thnk I'm getting it. Well, the moving_funcs move your window by 1, and only 1, sorry. If you need a larger time step, you can sample the result of the moving_func every `timestep`: >>> mov_func(yourdata, yourwindowsize)[::yourtimestep] OK, that's a bit sad because you end up computing a lot of data for nothing, but that's the only workaround I can think of right now. That would be a nice addition, though, so don't hesitate to open a ticket for that. From mwlodarczak at uni-bielefeld.de Thu Apr 14 11:30:16 2011 From: mwlodarczak at uni-bielefeld.de (=?UTF-8?B?TWFyY2luIFfFgm9kYXJjemFr?=) Date: Thu, 14 Apr 2011 17:30:16 +0200 Subject: [SciPy-User] Setting time step for moving windows In-Reply-To: <7F12C781-73C2-4DBF-98E2-E553242DCC9F@gmail.com> References: <4DA6F222.8060901@uni-bielefeld.de> <17168_1302786773_ZZh0o3p4MtN0G.00_4DA6F2D5.90203@uni-bielefeld.de> <3F9AA360-1117-4CE3-BE6D-BA4264FEA251@gmail.com> <3950_1302789646_ZZh0o5pcYs3pY.00_4DA6FE14.1060501@uni-bielefeld.de> <7F12C781-73C2-4DBF-98E2-E553242DCC9F@gmail.com> Message-ID: <19107_1302795017_ZZh0o1qbhv2Vl.00_4DA71308.8080806@uni-bielefeld.de> On 04/14/2011 04:23 PM, Pierre GM wrote: > Well, the moving_funcs move your window by 1, and only 1, sorry. If > you need a larger time step, you can sample the result of the > moving_func every `timestep`: >>>> mov_func(yourdata, yourwindowsize)[::yourtimestep] > OK, that's a bit sad because you end up computing a lot of data for > nothing, but that's the only workaround I can think of right now. I guess that's better than nothing. > That would be a nice addition, though, so don't hesitate to open a > ticket for that. I'll do that. Thanks for help. Best, Marcin From ralf.gommers at googlemail.com Thu Apr 14 15:06:25 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 14 Apr 2011 21:06:25 +0200 Subject: [SciPy-User] ANN: scikit.datasmooth 0.6.1 In-Reply-To: <4DA6FE78.4010108@vcn.com> References: <4DA6FE78.4010108@vcn.com> Message-ID: On Thu, Apr 14, 2011 at 4:02 PM, Jonathan Stickel wrote: > On 4/14/11 07:12 , scipy-user-request at scipy.org wrote: >> Date: Thu, 14 Apr 2011 09:10:44 +0200 >> From: "Nils Wagner" >> Subject: Re: [SciPy-User] ANN: scikit.datasmooth 0.6.1 >> >> On Wed, 13 Apr 2011 14:55:09 -0600 >> ? ?Jonathan Stickel ?wrote: >>> > ?The scikit "datasmooth" has been updated to version >>> >0.6.1. ?The primary >>> > ?purpose of this release is the addition of html >>> >documentation (generated >>> > ?by pydoc) and examples. ?Other notable changes can be >>> >found in the NEWS >>> > ?file. >>> > >>> > ?A source distribution can be found at: >>> > >>> > ?http://pypi.python.org/pypi/scikits.datasmooth >>> > >>> > ?and the development tree is on github: >>> > >>> > ?https://github.com/jjstickel/scikit-datasmooth >>> > >>> > >>> > ?This scikit is intended to include numerical methods for >>> >smoothing data >>> > ?that are not yet in scipy. ?Currently, only a >>> >regularization method for >>> > ?smoothing 1-D (y vs. x) data is included, for which >>> >cross-validation for >>> > ?determining the optimization parameter has been >>> >implemented. >>> > ?Constrained smoothing is also available but requires the >>> >cvxopt package. >>> > >>> > ?Comments and questions are welcome, as well as >>> >contributions of other >>> > ?smoothing methods. >>> > >>> > ?Regards, >>> > ?Jonathan >> Hi Jonathan, >> >> Just now I installed your scikit and got a syntax error. >> See below for details. >> >> Nils >> >> "build/bdist.linux-x86_64/egg/scikits/datasmooth/__init__.py", >> line 1 >> ? ? ? __version__ = 0.6.1 >> ? ? ? ? ? ? ? ? ? ? ? ? ^ >> SyntaxError: invalid syntax >> > > Oops! ?Sorry about that. ?I was having some trouble with pypi and had to > arbitrarily increment the version number. ?Apparently two periods are > not allowed. ?Thank you very much for reporting the problem. Versions with two dots are certainly allowed (they're very common). Your __version__ should be a string: In [1]: __version__ = '0.6.1' In [2]: __version__ = 0.6.1 ------------------------------------------------------------ File "", line 1 __version__ = 0.6.1 ^ SyntaxError: invalid syntax Cheers, Ralf From warren.weckesser at enthought.com Thu Apr 14 18:00:28 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Thu, 14 Apr 2011 17:00:28 -0500 Subject: [SciPy-User] genfromtxt but I need to search file first In-Reply-To: References: Message-ID: On Tue, Apr 12, 2011 at 10:54 PM, Ben Harrison wrote: > On 13/04/11 00:25, Warren Weckesser wrote: > > By any changes is this a LAS file? Take a look here: > > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055563.html > > and at my response. I included a "quick and dirty" implementation of a > > LAS reader. (I have a nicer version in progress, but it will be a few > > days before I can finish it.) In the version attached to that email, I > > just use loadtxt() to read the data after the ~A line. > > > > Warren > > > > Yes it is a LAS file, they are pesky things. > I'll try your script, or steal methods from it! > > Will you be publishing your nicer script anywhere in particular? Could > you let me know somehow when you have? > The "Input Output" section of the SciPy Cookbook ( http://www.scipy.org/Cookbook) looks like as good a place as any. I'll create a page for it there once I've clean up a few things. Warren > Thanks, > Ben. > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From odysseus9672 at gmail.com Fri Apr 15 04:10:59 2011 From: odysseus9672 at gmail.com (Sean Lake) Date: Fri, 15 Apr 2011 01:10:59 -0700 Subject: [SciPy-User] Special Function Question Message-ID: <051ABF87-5BFB-49F5-BBB8-D54ECE97C0DB@gmail.com> Hello all, Is there a reason that gammainc in scipy.special is the normalized version? I would tend to think that the denormalized version, and it's ability to handle negative arguments (in either but not both) would be preferred. Just curious, Sean From roban.kramer at phys.ethz.ch Fri Apr 15 05:10:32 2011 From: roban.kramer at phys.ethz.ch (Roban Hultman Kramer) Date: Fri, 15 Apr 2011 11:10:32 +0200 Subject: [SciPy-User] Special Function Question In-Reply-To: <051ABF87-5BFB-49F5-BBB8-D54ECE97C0DB@gmail.com> References: <051ABF87-5BFB-49F5-BBB8-D54ECE97C0DB@gmail.com> Message-ID: I just want to echo the sentiment that it would be useful to have the unnormalized version also available. I've used the gammainc from the mpmath package instead for this purpose. On Fri, Apr 15, 2011 at 10:10 AM, Sean Lake wrote: > Hello all, > > Is there a reason that gammainc in scipy.special is the normalized version? I would tend to think that the denormalized version, and it's ability to handle negative arguments (in either but not both) would be preferred. > > Just curious, > Sean > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From mickael.paris at gmail.com Fri Apr 15 08:20:21 2011 From: mickael.paris at gmail.com (Mickael) Date: Fri, 15 Apr 2011 14:20:21 +0200 Subject: [SciPy-User] Special Function Question In-Reply-To: References: <051ABF87-5BFB-49F5-BBB8-D54ECE97C0DB@gmail.com> Message-ID: Hi Sean, >> Is there a reason that gammainc in scipy.special is the normalized version? this function is not really normelized, since the gamma function is completely defined ==> on |R_(+)^(*) see (Bohr?Mollerup theorem) ==> and for complexe where, Re(z)>0 so all values must be positives... ??? Mickael. 2011/4/15 Roban Hultman Kramer : > I just want to echo the sentiment that it would be useful to have the > unnormalized version also available. I've used the gammainc from the > mpmath package instead for this purpose. > > On Fri, Apr 15, 2011 at 10:10 AM, Sean Lake wrote: >> Hello all, >> >> Is there a reason that gammainc in scipy.special is the normalized version? I would tend to think that the denormalized version, and it's ability to handle negative arguments (in either but not both) would be preferred. >> >> Just curious, >> Sean >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From robert.kern at gmail.com Fri Apr 15 11:16:30 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Apr 2011 10:16:30 -0500 Subject: [SciPy-User] Special Function Question In-Reply-To: <051ABF87-5BFB-49F5-BBB8-D54ECE97C0DB@gmail.com> References: <051ABF87-5BFB-49F5-BBB8-D54ECE97C0DB@gmail.com> Message-ID: On Fri, Apr 15, 2011 at 03:10, Sean Lake wrote: > Hello all, > > Is there a reason that gammainc in scipy.special is the normalized version? Because the library we wrapped happened to choose that version. ;-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From warren.weckesser at enthought.com Fri Apr 15 16:10:48 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Fri, 15 Apr 2011 15:10:48 -0500 Subject: [SciPy-User] Why not filtfilt in decimate function? In-Reply-To: References: Message-ID: On Thu, Apr 14, 2011 at 5:20 AM, Joscha Schmiedt < joscha.schmiedt at googlemail.com> wrote: > Dear list, > > I just noticed that the decimate function from scipy.signal does not use > filtfilt to filter the signal but lfilter and thereby introduces nasty phase > shifts. Is there a reason why this is done? > I suspect that when decimate was written, the filtfilt function was not yet in scipy.signal. You can avoid the phase shift in decimate by using the keyword argument ftype='fir', so an FIR filter is used instead of an IIR filter. Or you could take complete control and implement your own version of decimate. If you take a look at the source code for decimate, you'll see that it is fairly simple, and the IIR filter that it uses by default (a Chebyshev filter) has some hard-code parameters that you might want to tweak--or you could replace it completely with a different IIR filter or something like filtfilt. (But be aware that filtfilt has a bug that affects the values that it computes near the beginning and end of the signal: http://projects.scipy.org/scipy/ticket/1410) Warren > > Kind regards, > Joscha > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrennie at gmail.com Fri Apr 15 17:26:12 2011 From: jrennie at gmail.com (Jason Rennie) Date: Fri, 15 Apr 2011 17:26:12 -0400 Subject: [SciPy-User] Passing fmin_cg a gradient In-Reply-To: References: Message-ID: On Wed, Apr 13, 2011 at 3:41 PM, Christian K. wrote: > I have been working on the same thing a couple of years ago during my > PhD. In the end I found, that fmin_tnc is performing best among all > minimizers available in scipy. I'm actually on the look-out for a good open source CG or L-BFGS implementation. Thanks for the fmin_tnc mention---I'll check it out. Do you know what the "tnc" refers to and/or what are the differences with the other implementations? Thanks, Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 15 17:38:21 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 15 Apr 2011 16:38:21 -0500 Subject: [SciPy-User] Passing fmin_cg a gradient In-Reply-To: References: Message-ID: On Fri, Apr 15, 2011 at 16:26, Jason Rennie wrote: > On Wed, Apr 13, 2011 at 3:41 PM, Christian K. wrote: >> >> I have been working on the same thing a couple of years ago during my >> PhD. In the end I found, that fmin_tnc is performing best among all >> minimizers available in scipy. > > I'm actually on the look-out for a good open source CG or L-BFGS > implementation. ?Thanks for the fmin_tnc mention---I'll check it out. ?Do > you know what the "tnc" refers to and/or what are the differences with the > other implementations? http://js2007.free.fr/code/index.html#TNC """ TNC is a C implementation of TNBC, a truncated newton optimization package originally developed by Stephen G. Nash in Fortran. The original source code can be found at Stephen G. Nash Software Page. This software aims at minimizing the value of a nonlinear function whose variables are subject to bound constraints. It requires to be able to evaluate the function and its gradient and is especially useful for solving large scale problems. The C source code and an example, along with license and copyright information are included in the package. This software has been tested on the majority of UNIXes and on Windows. Since version 1.0.5, a Python interface module is provided. This interface can also be found in SciPy. """ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From gokhansever at gmail.com Sat Apr 16 14:14:04 2011 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Sat, 16 Apr 2011 12:14:04 -0600 Subject: [SciPy-User] ODE solving issues Message-ID: Hello, Considering my example code at http://code.google.com/p/ccnworks/source/browse/trunk/simple_parcel/python/parcel-ode.py Currently odeint solved version of the code yields very similar values obtained by forward differencing approach ( http://code.google.com/p/ccnworks/source/browse/trunk/simple_parcel/python/parcel.py) (For t_init=0, t_final=500) However the results differ in between these two scripts quite a lot when I ask a solution for a larger final t value (i.e. 5000) In myfunc function the parameters v, w... all depends on the previous integration of z, p... for the specified t values. For instance in this statement v = (u[4]) * gascon_air / (u[1]) I incorrectly assume that u[4] is the integration of "tk" variable for my initially specified time steps. Is there a way to obtain t-step integrals as returned from the function? Say that for t=np.arange(0, 500, 0.05) the odeint yields a 10000 element array for each of the equation. Then I can see the integration result for the specified time increments (e.g. at t=100 or 120.25 etc..) Again, Is there any way to obtain these results right within the function so I can calculate my v,w,... parameters correctly? My second question is regarding to the scipy.integrate.ode. In the parcel-ode.py file, uncommenting those relevant lines and running the code results: create_cb_arglist: Failed to build argument list (siz) with enough arguments (tot-opt) required by user-supplied function (siz,tot,opt=3,9,0). How to fix this problem? Thanks. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From masashi.kameda at gmail.com Sat Apr 16 16:48:39 2011 From: masashi.kameda at gmail.com (=?ISO-2022-JP?B?GyRCNTVFREdPO1YbKEI=?=) Date: Sun, 17 Apr 2011 05:48:39 +0900 Subject: [SciPy-User] How to tell a domain to fmin Message-ID: Hello, there. I'm a newbie to scypi, and have stuck around it for a week. Please give me some advice. Suppose I have a formula, geometric mean with logarithm. Let's say I have something like g(x, y) = 0.35*ln|1+4*x+2*y|+0.35*ln|1-x+2*y|+0.3*ln|1-x-y| and want to find its maximum. fmin seems estimated to calculate the minimum, so I just give this formula with negative sign to fmin. fmin told the minimum(actually maximum) of the function. It gives me the result: Out[4]: array([ -5.30653193e-09, 4.00000010e-01]) The first element is the negative one. But this is not what I want to get. To be honest, I want to know the maximum in a certain domain, for instance, x >= 0, y >= 0, x + y < 1 ; however I don't know how to tell the domain to fmin. I tested some similar geometric means given to fmin, and it sometimes gives "math domain error" or "maximum function evaluates exceeded" because fmin calculates over the area where I estimated, or domain. Please tell me how to tell domain to fmin. -- ???? Masahi Kameda -------------- next part -------------- An HTML attachment was scrubbed... URL: From cweisiger at msg.ucsf.edu Sat Apr 16 17:06:40 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Sat, 16 Apr 2011 14:06:40 -0700 Subject: [SciPy-User] How to tell a domain to fmin In-Reply-To: References: Message-ID: You are in control of the cost function passed to fmin. If it wants a value outside the domain, you can simply return a very high cost. That should effectively constrain the search space to only values inside the desired domain. E.g. def costFunc(coordinates): x, y = coordinates if x < 0 or x > 1 or x < 0 or x > 1: return 10 ** 10 return g(x, y) 2011/4/16 ???? > Hello, there. > I'm a newbie to scypi, and have stuck around it for a week. Please give me > some advice. > > Suppose I have a formula, geometric mean with logarithm. Let's say I have > something like > > g(x, y) = 0.35*ln|1+4*x+2*y|+0.35*ln|1-x+2*y|+0.3*ln|1-x-y| > > and want to find its maximum. fmin seems estimated to calculate the > minimum, so I just give > this formula with negative sign to fmin. > fmin told the minimum(actually maximum) of the function. It gives me the > result: > > Out[4]: array([ -5.30653193e-09, 4.00000010e-01]) > > The first element is the negative one. But this is not what I want to get. > To be honest, I want to know the maximum in a certain domain, for instance, > x >= 0, y >= 0, x + y < 1 > ; however I don't know how to tell the domain to fmin. > > I tested some similar geometric means given to fmin, and it sometimes gives > "math domain error" or > "maximum function evaluates exceeded" because fmin calculates over the area > where I estimated, or domain. > > Please tell me how to tell domain to fmin. > > > -- > ???? > Masahi Kameda > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cweisiger at msg.ucsf.edu Sat Apr 16 17:07:14 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Sat, 16 Apr 2011 14:07:14 -0700 Subject: [SciPy-User] How to tell a domain to fmin In-Reply-To: References: Message-ID: Er, except that it should have read "if x < 0 or x > 1 or y < 0 or y > 1", my apologies. 2011/4/16 Chris Weisiger > You are in control of the cost function passed to fmin. If it wants a value > outside the domain, you can simply return a very high cost. That should > effectively constrain the search space to only values inside the desired > domain. E.g. > > def costFunc(coordinates): > x, y = coordinates > if x < 0 or x > 1 or x < 0 or x > 1: > return 10 ** 10 > return g(x, y) > > 2011/4/16 ???? > >> Hello, there. >> I'm a newbie to scypi, and have stuck around it for a week. Please give me >> some advice. >> >> Suppose I have a formula, geometric mean with logarithm. Let's say I have >> something like >> >> g(x, y) = 0.35*ln|1+4*x+2*y|+0.35*ln|1-x+2*y|+0.3*ln|1-x-y| >> >> and want to find its maximum. fmin seems estimated to calculate the >> minimum, so I just give >> this formula with negative sign to fmin. >> fmin told the minimum(actually maximum) of the function. It gives me the >> result: >> >> Out[4]: array([ -5.30653193e-09, 4.00000010e-01]) >> >> The first element is the negative one. But this is not what I want to get. >> To be honest, I want to know the maximum in a certain domain, for >> instance, x >= 0, y >= 0, x + y < 1 >> ; however I don't know how to tell the domain to fmin. >> >> I tested some similar geometric means given to fmin, and it sometimes >> gives "math domain error" or >> "maximum function evaluates exceeded" because fmin calculates over the >> area where I estimated, or domain. >> >> Please tell me how to tell domain to fmin. >> >> >> -- >> ???? >> Masahi Kameda >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From masashi.kameda at gmail.com Sat Apr 16 17:38:46 2011 From: masashi.kameda at gmail.com (=?ISO-2022-JP?B?GyRCNTVFREdPO1YbKEI=?=) Date: Sun, 17 Apr 2011 06:38:46 +0900 Subject: [SciPy-User] How to tell a domain to fmin In-Reply-To: References: Message-ID: Mr. Chris Weisiger, thanks for the advice. It seems interesting. However, again, I am not interested in the maximum outside the domain. What I am interested in is the coordinate which maximize the function *in* the domain(that means it is not necessary to get true maximum). For instance, does this def costFunc(coordinates): x, y = coordinates if x < 0 or x > 1 or y < 0 or y > 1: # command to recalculate with fmin again? return g(x, y) work? It seems to be an infinite loop....... Uuh. I mean, 10 ** 10 is not the information which I want to need. 2011?4?17?6:07 Chris Weisiger : > Er, except that it should have read "if x < 0 or x > 1 or y < 0 or y > 1", > my apologies. > > > 2011/4/16 Chris Weisiger > >> You are in control of the cost function passed to fmin. If it wants a >> value outside the domain, you can simply return a very high cost. That >> should effectively constrain the search space to only values inside the >> desired domain. E.g. >> >> def costFunc(coordinates): >> x, y = coordinates >> if x < 0 or x > 1 or x < 0 or x > 1: >> return 10 ** 10 >> return g(x, y) >> >> 2011/4/16 ???? >> >>> Hello, there. >>> I'm a newbie to scypi, and have stuck around it for a week. Please give >>> me some advice. >>> >>> Suppose I have a formula, geometric mean with logarithm. Let's say I have >>> something like >>> >>> g(x, y) = 0.35*ln|1+4*x+2*y|+0.35*ln|1-x+2*y|+0.3*ln|1-x-y| >>> >>> and want to find its maximum. fmin seems estimated to calculate the >>> minimum, so I just give >>> this formula with negative sign to fmin. >>> fmin told the minimum(actually maximum) of the function. It gives me the >>> result: >>> >>> Out[4]: array([ -5.30653193e-09, 4.00000010e-01]) >>> >>> The first element is the negative one. But this is not what I want to >>> get. >>> To be honest, I want to know the maximum in a certain domain, for >>> instance, x >= 0, y >= 0, x + y < 1 >>> ; however I don't know how to tell the domain to fmin. >>> >>> I tested some similar geometric means given to fmin, and it sometimes >>> gives "math domain error" or >>> "maximum function evaluates exceeded" because fmin calculates over the >>> area where I estimated, or domain. >>> >>> Please tell me how to tell domain to fmin. >>> >>> >>> -- >>> ???? >>> Masahi Kameda >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- ???? Masahi Kameda -------------- next part -------------- An HTML attachment was scrubbed... URL: From cweisiger at msg.ucsf.edu Sat Apr 16 18:09:42 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Sat, 16 Apr 2011 15:09:42 -0700 Subject: [SciPy-User] How to tell a domain to fmin In-Reply-To: References: Message-ID: 2011/4/16 ???? > > > Mr. Chris Weisiger, thanks for the advice. It seems interesting. > > However, again, I am not interested in the maximum outside the domain. What > I am interested > in is the coordinate which maximize the function *in* the domain(that means > it is not necessary to get true maximum). > > For instance, does this > > def costFunc(coordinates): > x, y = coordinates > if x < 0 or x > 1 or y < 0 or y > 1: > # command to recalculate with fmin again? > return g(x, y) > > work? It seems to be an infinite loop....... Uuh. > > Sorry, I was writing for a function you want to minimize, since that is how fmin works. Of course you would need to negate your return values to maximize. What I'm doing here is supplying a wrapper function around your ordinary cost function g(x, y). This function forbids values outside the desired domain by giving them values which fmin would always consider to be "bad" because they are so large. Basically what I'm saying is "If (X, Y) is outside of the domain, then return a very large number, otherwise return g(x, y)". Since you say you're trying to maximize the value of g(x, y), of course you'd actually return -g(x, y). The important thing is that the return value when you're inside the domain is always smaller than the return value when you're outside of it. fmin should then consider any values outside of the domain to be suboptimal. For example, if g(x, y)'s range is [0, 100] in the desired domain, then the wrapper function will return values in the range [-100, 0] when in the domain, and 10 ** 10 when outside of it. fmin searches for the minimum result value, which should be where the function returns -100. There's probably a more elegant way to do this, but this should at least work. -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From masashi.kameda at gmail.com Sat Apr 16 18:58:01 2011 From: masashi.kameda at gmail.com (=?ISO-2022-JP?B?GyRCNTVFREdPO1YbKEI=?=) Date: Sun, 17 Apr 2011 07:58:01 +0900 Subject: [SciPy-User] How to tell a domain to fmin In-Reply-To: References: Message-ID: To Mr. Chris Weisiger * * *I see. Now I slowly understand what 10 ** 10 means. 10 ** 10 doesn't mean a concrete * *number (actually yes, it is, though), but means some error message to fmin. Now I * understand. Actually I'm not used to try except structure of Python (even Python itself, honestly speaking), then I'm very afraid of using raise ValueError to fmin. But this is a surprising to have this simple substitution. I try this wrapper. I will tell what is going on. Thank you. 2011?4?17?7:09 Chris Weisiger : > 2011/4/16 ???? > >> >> >> Mr. Chris Weisiger, thanks for the advice. It seems interesting. >> >> However, again, I am not interested in the maximum outside the domain. >> What I am interested >> in is the coordinate which maximize the function *in* the domain(that >> means it is not necessary to get true maximum). >> >> For instance, does this >> >> def costFunc(coordinates): >> x, y = coordinates >> if x < 0 or x > 1 or y < 0 or y > 1: >> # command to recalculate with fmin again? >> return g(x, y) >> >> work? It seems to be an infinite loop....... Uuh. >> >> > Sorry, I was writing for a function you want to minimize, since that is how > fmin works. Of course you would need to negate your return values to > maximize. > > What I'm doing here is supplying a wrapper function around your ordinary > cost function g(x, y). This function forbids values outside the desired > domain by giving them values which fmin would always consider to be "bad" > because they are so large. Basically what I'm saying is "If (X, Y) is > outside of the domain, then return a very large number, otherwise return > g(x, y)". Since you say you're trying to maximize the value of g(x, y), of > course you'd actually return -g(x, y). The important thing is that the > return value when you're inside the domain is always smaller than the return > value when you're outside of it. fmin should then consider any values > outside of the domain to be suboptimal. > > For example, if g(x, y)'s range is [0, 100] in the desired domain, then the > wrapper function will return values in the range [-100, 0] when in the > domain, and 10 ** 10 when outside of it. fmin searches for the minimum > result value, which should be where the function returns -100. > > There's probably a more elegant way to do this, but this should at least > work. > > -Chris > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- ???? Masahi Kameda -------------- next part -------------- An HTML attachment was scrubbed... URL: From masashi.kameda at gmail.com Sat Apr 16 22:30:53 2011 From: masashi.kameda at gmail.com (=?ISO-2022-JP?B?GyRCNTVFREdPO1YbKEI=?=) Date: Sun, 17 Apr 2011 11:30:53 +0900 Subject: [SciPy-User] How to tell a domain to fmin In-Reply-To: References: Message-ID: Yes, it works! Thank you With Mr.Chris Weisiger's idea, I try making costFunc. The domain I need is, for instance, x + y < 1, x > 0, y > 0. Then, at first, I got to make a helper function. def lesser_p(ls): for i in ls: if i < 0: return True return False This function checks out whether all elements of a list given are greater than 0 or not. If this finds a number less than 0, it stops the evaluation and return True immediately. Otherwise, it returns False. With this helper function, what I made is this. def costFunc(x): if lesser_p(x): return 10 ** 10 elif sum(x) >= 1: return 10 ** 10 else: return g(x) This shows the condition, x + y must be less than 1, and each x and y must be greater than 0. With Python, g(x) got to be like this: def g(x): return -(0.35*math.log(1+4*x[0]+2*x[1])+0.35*math.log(1-x[0]+2*x[1])+0.3*math.log(1-x[0]-x[1])) Then, with fmin, Optimization terminated successfully. Current function value: -0.082283 Iterations: 108 Function evaluations: 200 Out[28]: array([ 6.97177401e-09, 3.99999991e-01]) Yes, it gives me back a pair of positive numbers! Thank you very much, Mr.Chris Weisiger!!! 2011?4?17?7:58 ???? : > To Mr. Chris Weisiger > * > * > *I see. Now I slowly understand what 10 ** 10 means. 10 ** 10 doesn't mean > a concrete > * > *number (actually yes, it is, though), but means some error message to > fmin. Now I > * > understand. > Actually I'm not used to try except structure of Python (even Python > itself, honestly speaking), then I'm very afraid of using raise ValueError > to fmin. But this is a surprising to have this simple substitution. > > I try this wrapper. I will tell what is going on. > > Thank you. > > > 2011?4?17?7:09 Chris Weisiger : > >> 2011/4/16 ???? >> >>> >>> >>> Mr. Chris Weisiger, thanks for the advice. It seems interesting. >>> >>> However, again, I am not interested in the maximum outside the domain. >>> What I am interested >>> in is the coordinate which maximize the function *in* the domain(that >>> means it is not necessary to get true maximum). >>> >>> For instance, does this >>> >>> def costFunc(coordinates): >>> x, y = coordinates >>> if x < 0 or x > 1 or y < 0 or y > 1: >>> # command to recalculate with fmin again? >>> return g(x, y) >>> >>> work? It seems to be an infinite loop....... Uuh. >>> >>> >> Sorry, I was writing for a function you want to minimize, since that is >> how fmin works. Of course you would need to negate your return values to >> maximize. >> >> What I'm doing here is supplying a wrapper function around your ordinary >> cost function g(x, y). This function forbids values outside the desired >> domain by giving them values which fmin would always consider to be "bad" >> because they are so large. Basically what I'm saying is "If (X, Y) is >> outside of the domain, then return a very large number, otherwise return >> g(x, y)". Since you say you're trying to maximize the value of g(x, y), of >> course you'd actually return -g(x, y). The important thing is that the >> return value when you're inside the domain is always smaller than the return >> value when you're outside of it. fmin should then consider any values >> outside of the domain to be suboptimal. >> >> For example, if g(x, y)'s range is [0, 100] in the desired domain, then >> the wrapper function will return values in the range [-100, 0] when in the >> domain, and 10 ** 10 when outside of it. fmin searches for the minimum >> result value, which should be where the function returns -100. >> >> There's probably a more elegant way to do this, but this should at least >> work. >> >> -Chris >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > > -- > ???? > Masahi Kameda > -- ???? Masahi Kameda -------------- next part -------------- An HTML attachment was scrubbed... URL: From yury at shurup.com Sun Apr 17 10:11:31 2011 From: yury at shurup.com (Yury V. Zaytsev) Date: Sun, 17 Apr 2011 16:11:31 +0200 Subject: [SciPy-User] Passing fmin_cg a gradient In-Reply-To: References: Message-ID: <1303049491.6576.7.camel@mypride> On Fri, 2011-04-15 at 17:26 -0400, Jason Rennie wrote: > > I'm actually on the look-out for a good open source CG or L-BFGS > implementation. Thanks for the fmin_tnc mention---I'll check it out. On Mon, 2011-04-11 at 20:52 -0400, Patrick Holvey wrote: > I'm also open to alternate optimization calls if anyone has > recommendations. So far I have been very happy with NLopt which got to know thanks to the advice someone posted on this list. It features a nice pure C implementation of BFGS, which, I guess, for most problems would be a good starting point and a number of other gradient-based methods. Also, I have found through experimentation, that NLopt Downhill-Simplex implementation is much more stable for the functions that I have to optimize than fmin from SciPy. Do check it out. -- Sincerely yours, Yury V. Zaytsev From cwebster at enthought.com Sun Apr 17 11:44:00 2011 From: cwebster at enthought.com (Corran Webster) Date: Sun, 17 Apr 2011 10:44:00 -0500 Subject: [SciPy-User] How to tell a domain to fmin In-Reply-To: References: Message-ID: Hi Masahi, Chris, you might also try using fmin_cobyla, which allows you to explicitly pass constraints in as arguments to the optimizer. See: http://docs.scipy.org/doc/scipy-0.7.x/reference/generated/scipy.optimize.fmin_cobyla.html#scipy.optimize.fmin_cobyla Best Regards, Corran Webster 2011/4/16 ???? > Yes, it works! Thank you > > With Mr.Chris Weisiger's idea, I try making costFunc. The domain I need is, > for instance, > x + y < 1, x > 0, y > 0. Then, at first, I got to make a helper function. > > def lesser_p(ls): > for i in ls: > if i < 0: > return True > return False > > This function checks out whether all elements of a list given are greater > than 0 or not. If > this finds a number less than 0, it stops the evaluation and return True > immediately. > Otherwise, it returns False. > > With this helper function, what I made is this. > > def costFunc(x): > > if lesser_p(x): > return 10 ** 10 > elif sum(x) >= 1: > return 10 ** 10 > else: > return g(x) > > This shows the condition, x + y must be less than 1, and each x and y must > be greater than 0. > With Python, g(x) got to be like this: > > def g(x): return > -(0.35*math.log(1+4*x[0]+2*x[1])+0.35*math.log(1-x[0]+2*x[1])+0.3*math.log(1-x[0]-x[1])) > > Then, with fmin, > > Optimization terminated successfully. > Current function value: -0.082283 > Iterations: 108 > Function evaluations: 200 > Out[28]: array([ 6.97177401e-09, 3.99999991e-01]) > > Yes, it gives me back a pair of positive numbers! > > Thank you very much, Mr.Chris Weisiger!!! > > 2011?4?17?7:58 ???? : > > To Mr. Chris Weisiger >> * >> * >> *I see. Now I slowly understand what 10 ** 10 means. 10 ** 10 doesn't >> mean a concrete >> * >> *number (actually yes, it is, though), but means some error message to >> fmin. Now I >> * >> understand. >> Actually I'm not used to try except structure of Python (even Python >> itself, honestly speaking), then I'm very afraid of using raise ValueError >> to fmin. But this is a surprising to have this simple substitution. >> >> I try this wrapper. I will tell what is going on. >> >> Thank you. >> >> >> 2011?4?17?7:09 Chris Weisiger : >> >>> 2011/4/16 ???? >>> >>>> >>>> >>>> Mr. Chris Weisiger, thanks for the advice. It seems interesting. >>>> >>>> However, again, I am not interested in the maximum outside the domain. >>>> What I am interested >>>> in is the coordinate which maximize the function *in* the domain(that >>>> means it is not necessary to get true maximum). >>>> >>>> For instance, does this >>>> >>>> def costFunc(coordinates): >>>> x, y = coordinates >>>> if x < 0 or x > 1 or y < 0 or y > 1: >>>> # command to recalculate with fmin again? >>>> return g(x, y) >>>> >>>> work? It seems to be an infinite loop....... Uuh. >>>> >>>> >>> Sorry, I was writing for a function you want to minimize, since that is >>> how fmin works. Of course you would need to negate your return values to >>> maximize. >>> >>> What I'm doing here is supplying a wrapper function around your ordinary >>> cost function g(x, y). This function forbids values outside the desired >>> domain by giving them values which fmin would always consider to be "bad" >>> because they are so large. Basically what I'm saying is "If (X, Y) is >>> outside of the domain, then return a very large number, otherwise return >>> g(x, y)". Since you say you're trying to maximize the value of g(x, y), of >>> course you'd actually return -g(x, y). The important thing is that the >>> return value when you're inside the domain is always smaller than the return >>> value when you're outside of it. fmin should then consider any values >>> outside of the domain to be suboptimal. >>> >>> For example, if g(x, y)'s range is [0, 100] in the desired domain, then >>> the wrapper function will return values in the range [-100, 0] when in the >>> domain, and 10 ** 10 when outside of it. fmin searches for the minimum >>> result value, which should be where the function returns -100. >>> >>> There's probably a more elegant way to do this, but this should at least >>> work. >>> >>> -Chris >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> >> >> -- >> ???? >> Masahi Kameda >> > > > > -- > ???? > Masahi Kameda > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kortak at gmail.com Sat Apr 16 17:04:40 2011 From: kortak at gmail.com (Renaux Christian) Date: Sat, 16 Apr 2011 23:04:40 +0200 Subject: [SciPy-User] UnivariateSpline interpolation problem Message-ID: Hi, I'm having a weird problem trying to interpolate data using the UnivariateSpline function. Interpolating through all the points (s=0) and the spline function does not give a result on the entire set of data. The result for s>=1 is also very weird. As I think it is related to the data I'm using, I join them in attachement. I'm stuck, so if anyone have a good idea on a solution, I will really appreciate. Thanks, here part of the code: import numpy as np import matplotlib.pyplot as plt from scipy.interpolate import UnivariateSpline def openfile(infilename): ifile = open(infilename, 'r') # open file for reading lines = ifile.readlines() ifile.close() return lines def extractData(lines): data=[] CV=[] for i in range(len(lines)): item=lines[i].split() for j in range(len(item)): item[j]=float(item[j]) data.append(item[j]) CV=np.array(data) CV.shape = (len(CV)/3,3) return CV if __name__ == "__main__": lines=openfile("D:\capamos\LOCOS\cap15L1_rec_mod.csv") CV=extractData(lines) Vg1=CV[:,0] C1=CV[:,1] Cmax=C1.max() Cmin=C1.min() S=0.002 Cfb=compute(Cmax,Cmin,S) #compute the flat band capacitance print "Cfb=",Cfb splineCV= UnivariateSpline(Vg1,C1,s=0) x = linspace(-5, 5, 1000) # just to draw the spline function y=splineCV(x) Vfb=splineCV(Cfb) # find the flat band voltage at Cfb print "Vfb=",Vfb print y plt.figure(1) p1=plot(Vg1,C1,'b',label='edge') p2=plot(x,y,'g') plt.axis([-6,6,1e-11,80e-12]) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cap15L1_rec_mod.csv Type: text/csv Size: 5147 bytes Desc: not available URL: From josef.pktd at gmail.com Sun Apr 17 17:47:06 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 17 Apr 2011 17:47:06 -0400 Subject: [SciPy-User] UnivariateSpline interpolation problem In-Reply-To: References: Message-ID: On Sat, Apr 16, 2011 at 5:04 PM, Renaux Christian wrote: > Hi, > I'm having a weird problem trying to interpolate data using > the?UnivariateSpline function. ?Interpolating through all the points (s=0) > and the spline function does not give a result on the entire set of data. > ?The result for s>=1 is also very weird. ?As I think it is related to the > data I'm using, I join them in attachement. > I'm stuck, so if anyone have a good idea on a solution, I will really > appreciate. > Thanks, > here part of the code: > import numpy as np > import matplotlib.pyplot as plt > from scipy.interpolate import UnivariateSpline > > def openfile(infilename): > ?? ?ifile = open(infilename, 'r') # open file for reading > ?? ?lines = ifile.readlines() > ?? ?ifile.close() > ?? ?return lines > def extractData(lines): > ?? ?data=[] > ?? ?CV=[] > ?? ?for i in range(len(lines)): > ?? ? ? ?item=lines[i].split() > ?? ? ? ?for j in range(len(item)): > ?? ? ? ? ? ?item[j]=float(item[j]) > ?? ? ? ? ? ?data.append(item[j]) > ?? ?CV=np.array(data) > ?? ?CV.shape = (len(CV)/3,3) > ?? ?return CV > if __name__ == "__main__": > ?? ?lines=openfile("D:\capamos\LOCOS\cap15L1_rec_mod.csv") > ?? ?CV=extractData(lines) > ?? ?Vg1=CV[:,0] > ?? ?C1=CV[:,1] > ?? ?Cmax=C1.max() > ?? ?Cmin=C1.min() > ?? ?S=0.002 > ?? ?Cfb=compute(Cmax,Cmin,S) #compute the flat band capacitance > ?? ?print "Cfb=",Cfb > > ?? ?splineCV= UnivariateSpline(Vg1,C1,s=0) > ?? ?x = linspace(-5, 5, 1000) ? # just to draw the spline function > ?? ?y=splineCV(x) > ?? ?Vfb=splineCV(Cfb) ?# find the flat band voltage at Cfb > ?? ?print "Vfb=",Vfb > ?? ?print y > ?? ?plt.figure(1) > ?? ?p1=plot(Vg1,C1,'b',label='edge') > ?? ?p2=plot(x,y,'g') > ?? ?plt.axis([-6,6,1e-11,80e-12]) just a guess I don't see any place where you sort (reverse) your data from the docs class scipy.interpolate.UnivariateSpline Parameters : x : sequence input dimension of data points ? must be increasing Josef > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From masashi.kameda at gmail.com Mon Apr 18 03:03:48 2011 From: masashi.kameda at gmail.com (=?ISO-2022-JP?B?GyRCNTVFREdPO1YbKEI=?=) Date: Mon, 18 Apr 2011 16:03:48 +0900 Subject: [SciPy-User] How to tell a domain to fmin In-Reply-To: References: Message-ID: Oh, math domain error again...... orz Strange. 2011/4/18 Corran Webster > Hi Masahi, Chris, > > you might also try using fmin_cobyla, which allows you to explicitly pass > constraints in as arguments to the optimizer. See: > > > http://docs.scipy.org/doc/scipy-0.7.x/reference/generated/scipy.optimize.fmin_cobyla.html#scipy.optimize.fmin_cobyla > > Best Regards, > Corran Webster > > > 2011/4/16 ???? > >> Yes, it works! Thank you >> >> With Mr.Chris Weisiger's idea, I try making costFunc. The domain I need >> is, for instance, >> x + y < 1, x > 0, y > 0. Then, at first, I got to make a helper function. >> >> def lesser_p(ls): >> for i in ls: >> if i < 0: >> return True >> return False >> >> This function checks out whether all elements of a list given are greater >> than 0 or not. If >> this finds a number less than 0, it stops the evaluation and return True >> immediately. >> Otherwise, it returns False. >> >> With this helper function, what I made is this. >> >> def costFunc(x): >> >> if lesser_p(x): >> return 10 ** 10 >> elif sum(x) >= 1: >> return 10 ** 10 >> else: >> return g(x) >> >> This shows the condition, x + y must be less than 1, and each x and y must >> be greater than 0. >> With Python, g(x) got to be like this: >> >> def g(x): return >> -(0.35*math.log(1+4*x[0]+2*x[1])+0.35*math.log(1-x[0]+2*x[1])+0.3*math.log(1-x[0]-x[1])) >> >> Then, with fmin, >> >> Optimization terminated successfully. >> Current function value: -0.082283 >> Iterations: 108 >> Function evaluations: 200 >> Out[28]: array([ 6.97177401e-09, 3.99999991e-01]) >> >> Yes, it gives me back a pair of positive numbers! >> >> Thank you very much, Mr.Chris Weisiger!!! >> >> 2011?4?17?7:58 ???? : >> >> To Mr. Chris Weisiger >>> * >>> * >>> *I see. Now I slowly understand what 10 ** 10 means. 10 ** 10 doesn't >>> mean a concrete >>> * >>> *number (actually yes, it is, though), but means some error message to >>> fmin. Now I >>> * >>> understand. >>> Actually I'm not used to try except structure of Python (even Python >>> itself, honestly speaking), then I'm very afraid of using raise ValueError >>> to fmin. But this is a surprising to have this simple substitution. >>> >>> I try this wrapper. I will tell what is going on. >>> >>> Thank you. >>> >>> >>> 2011?4?17?7:09 Chris Weisiger : >>> >>>> 2011/4/16 ???? >>>> >>>>> >>>>> >>>>> Mr. Chris Weisiger, thanks for the advice. It seems interesting. >>>>> >>>>> However, again, I am not interested in the maximum outside the domain. >>>>> What I am interested >>>>> in is the coordinate which maximize the function *in* the domain(that >>>>> means it is not necessary to get true maximum). >>>>> >>>>> For instance, does this >>>>> >>>>> def costFunc(coordinates): >>>>> x, y = coordinates >>>>> if x < 0 or x > 1 or y < 0 or y > 1: >>>>> # command to recalculate with fmin again? >>>>> return g(x, y) >>>>> >>>>> work? It seems to be an infinite loop....... Uuh. >>>>> >>>>> >>>> Sorry, I was writing for a function you want to minimize, since that is >>>> how fmin works. Of course you would need to negate your return values to >>>> maximize. >>>> >>>> What I'm doing here is supplying a wrapper function around your ordinary >>>> cost function g(x, y). This function forbids values outside the desired >>>> domain by giving them values which fmin would always consider to be "bad" >>>> because they are so large. Basically what I'm saying is "If (X, Y) is >>>> outside of the domain, then return a very large number, otherwise return >>>> g(x, y)". Since you say you're trying to maximize the value of g(x, y), of >>>> course you'd actually return -g(x, y). The important thing is that the >>>> return value when you're inside the domain is always smaller than the return >>>> value when you're outside of it. fmin should then consider any values >>>> outside of the domain to be suboptimal. >>>> >>>> For example, if g(x, y)'s range is [0, 100] in the desired domain, then >>>> the wrapper function will return values in the range [-100, 0] when in the >>>> domain, and 10 ** 10 when outside of it. fmin searches for the minimum >>>> result value, which should be where the function returns -100. >>>> >>>> There's probably a more elegant way to do this, but this should at least >>>> work. >>>> >>>> -Chris >>>> >>>> _______________________________________________ >>>> SciPy-User mailing list >>>> SciPy-User at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-user >>>> >>>> >>> >>> >>> -- >>> ???? >>> Masahi Kameda >>> >> >> >> >> -- >> ???? >> Masahi Kameda >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- ???? Masahi Kameda -------------- next part -------------- An HTML attachment was scrubbed... URL: From cweisiger at msg.ucsf.edu Mon Apr 18 18:37:51 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Mon, 18 Apr 2011 15:37:51 -0700 Subject: [SciPy-User] Finding islands and centroids Message-ID: (As you may recall from previous threads, my current project involves aligning two arrays of pixel data that were taken from slightly different perspectives) In an attempt to quantify the accuracy of the alignment I've obtained, I want to do some centroid analysis. The images I'm working with are of "beads" (very small fluorescent blobs), thus each 512x512 image is of a number of more-or-less circular islands each on the order of 50 pixels or so. I figure that I can threshold each image, identify distinct islands, get their centroids, map those to centroids in the other wavelength, and thus get the distance between centroids, which should be a good absolute measure of alignment quality. I can hack something together to do this easily enough, where I find a pixel in one of the islands, flood-fill out to get all connected pixels, calculate the centroid, flip the pixels to 0, and repeat until all islands are gone. This isn't exactly very speedy though. What's the efficient way to do this? Is there one? Is there a better approach I should be taking? The image processing class I dimly remember taking years ago didn't cover this kind of thing, so I'm lacking even the basic vocabulary needed to search for algorithms. -Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmueller at python-academy.de Mon Apr 18 18:54:17 2011 From: mmueller at python-academy.de (=?ISO-8859-15?Q?Mike_M=FCller?=) Date: Tue, 19 Apr 2011 00:54:17 +0200 Subject: [SciPy-User] [ANN] Courses in Colorado: "Introduction to Python and Python for Scientists and Engineers" Message-ID: <4DACC119.9030206@python-academy.de> Python Course in Golden, CO, USA ================================ Introduction to Python and Python for Scientists and Engineers -------------------------------------------------------------- June 3 - 4, 2011 Introduction to Python June 5, 2011 Python for Scientists and Engineers Both courses can be booked individually or together. Venue: Colorado School of Mines, Golden, CO (20 minutes west of Denver) Trainer: Mike M?ller Target Audience --------------- The introductory course is designed for people with basic programming background. Since it is a general introduction to Python it is suitable for everybody interested in Python. The scientist's course assumes a working knowledge of Python. You will be fine if you take the two-day introduction before hand. The topics are of general interest for scientists and engineers. Even though some examples come from the groundwater modeling domain, they are easy to understand for people without prior knowledge in this field. About the Trainer ----------------- Mike M?ller, has been teaching Python since 2004. He is the founder of Python Academy and regularly gives open and in-house Python courses as well as tutorials at PyCon US, OSCON, EuroSciPy and PyCon Asia-Pacific. More Information and Course Registration ---------------------------------------- http://igwmc.mines.edu/short-course/intro_python.html -- Mike mmueller at python-academy.de From jason-sage at creativetrax.com Mon Apr 18 23:40:01 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Mon, 18 Apr 2011 22:40:01 -0500 Subject: [SciPy-User] Central File Exchange for Scipy Message-ID: <4DAD0411.6010405@creativetrax.com> I've been funded over the summer by an NSF grant to build a library of Sage "interacts" (basically small snippets of Sage/Python code). Fernando Perez strongly encouraged me at a recent Sage Days to adopt a version controlled snippet model (ala Gist). The other night I threw together a very rough proof-of-concept, just-barely-working initial start of something along these lines (I hope I put enough disclaimers in there!) My code is here: https://github.com/jasongrout/snippets It requires Flask and Mercurial to be installed. Yes, the irony of using a Mercurial backend in a project on github is not lost on me. Though I prefer git, the python API to use and create mercurial repositories was too nice to turn down in this prototyping stage. I tried to encapsulate the VCS commands in a wrapper class so that the backend could be easily switched to git if need be. This reminded me of the Central File Exchange thread from last November, and in particular, several people saying that they were working on a version-controlled snippet database [1]. William or Andrew, have you posted your work anywhere? I think we have very similar goals. I won't be able to work on this heavily for about a month (until after the semester), but I will be hitting it pretty hard during the summer. The hope is to have a good production version by July. If anyone else is working on a related project, please let me know, as we can probably collaborate. If anyone wants to fork the github repo above and work on it, feel free! To prevent license discussions from eating up too much energy/time, I have decided that the site that I set up will have all snippets be (modified) BSD licensed. That may change, of course, before it's actually implemented, but it's not up for debate now. To encourage collaboration you folks, I'm willing to put a BSD license on the codebase as well if possible, though the Mercurial docs page seems to indicate that I'm forced to use GPLv2 (+?) if I use their API, which I am at this point [2]. Thanks, Jason [1] http://mail.scipy.org/pipermail/scipy-user/2010-November/027690.html [2] http://mercurial.selenic.com/wiki/MercurialApi -- Jason Grout From josh.holbrook at gmail.com Tue Apr 19 00:54:22 2011 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Mon, 18 Apr 2011 20:54:22 -0800 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: <4DAD0411.6010405@creativetrax.com> References: <4DAD0411.6010405@creativetrax.com> Message-ID: On Mon, Apr 18, 2011 at 7:40 PM, Jason Grout wrote: > > I've been funded over the summer by an NSF grant to build a library of > Sage "interacts" (basically small snippets of Sage/Python code). > Fernando Perez strongly encouraged me at a recent Sage Days to adopt a > version controlled snippet model (ala Gist). ?The other night I threw > together a very rough proof-of-concept, just-barely-working initial > start of something along these lines (I hope I put enough disclaimers in > there!) ?My code is here: > > https://github.com/jasongrout/snippets This sounds awesome! *watched* > It requires Flask and Mercurial to be installed. Yes, the irony of using > a Mercurial backend in a project on github is not lost on me. ?Though I > prefer git, the python API to use and create mercurial repositories was > too nice to turn down in this prototyping stage. ?I tried to encapsulate > the VCS commands in a wrapper class so that the backend could be easily > switched to git if need be. I don't think there's anything wrong with using mercurial here. If you want to support git pulls (though a lot of pythonistas prefer hg it seems the scientific community's leaning towards git), you could always make a system to convert repos to git on-the-fly, maybe using something like http://offbytwo.com/git-hg/ . In fact, I think I would do something like that as long as it wasn't too slow. Or, y'know, rewrite the VCS commands. :) Either way. > This reminded me of the Central File Exchange thread from last November, > and in particular, several people saying that they were working on a > version-controlled snippet database [1]. ?William or Andrew, have you > posted your work anywhere? ?I think we have very similar goals. Yeah, it sounds like you're doing something really similar, though with a Sage focus. How well do these Sage interacts things work with more vanilla python? I for one don't really use Sage, so if it was ridiculously different I don't know how much utility the SagExchange would be for me. > To prevent license discussions from eating up too much energy/time, I > have decided that the site that I set up will have all snippets be > (modified) BSD licensed. Word. I approve. I hope good things come of this. I'm mostly working with javascript these days, but if I have time I might take a look or two. --Josh From ben.harrison at liquidmesh.com Tue Apr 19 01:04:11 2011 From: ben.harrison at liquidmesh.com (Ben Harrison) Date: Tue, 19 Apr 2011 15:04:11 +1000 Subject: [SciPy-User] optional args to genfromtxt unavailable? In-Reply-To: References: <4DA5246E.7060903@liquidmesh.com> Message-ID: On 13/04/11 23:38, Pierre GM wrote: > > Because very likely, the arguments you're trying to use were introduced in more recent versions of numpy: 1.3 is already a bit old. > Get the list of arguments recognized by your version through the help (type `np.genfromtxt?` in ipython). Thank you, I rtwfm (w == wrong). From jason-sage at creativetrax.com Tue Apr 19 01:14:37 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Tue, 19 Apr 2011 00:14:37 -0500 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> Message-ID: <4DAD1A3D.2090703@creativetrax.com> On 4/18/11 11:54 PM, Joshua Holbrook wrote: > > Yeah, it sounds like you're doing something really similar, though > with a Sage focus. How well do these Sage interacts things work with > more vanilla python? I for one don't really use Sage, so if it was > ridiculously different I don't know how much utility the SagExchange > would be for me. The site would be simply a snippet site, geared towards python. Additionally, for each snippet, it would be able to send the snippet to a server to execute the snippet and display the results, like the htmlnotebook project in the IPython trunk. Except we'd probably also use another project we're currently working on, a "single-cell Sage Notebook server": https://github.com/jasongrout/simple-python-db-compute (see http://wiki.sagemath.org/DrakeSageGroup for rough notes of our progress and todo list). The "interacts" part just is some client-side javascript that makes sliders, textboxes, etc. In a sense, the Sage interact site would be like a combination of Gist, Wolfram Demonstrations, Pastebin, Central File Exchange, etc. For your purposes, you could ignore the "interact" part and just think of it as a version-controlled snippet site that also lets you execute the snippets on a remote server. > >> To prevent license discussions from eating up too much energy/time, I >> have decided that the site that I set up will have all snippets be >> (modified) BSD licensed. > > Word. I approve. I hope good things come of this. I'm mostly working > with javascript these days, but if I have time I might take a look or > two. We could definitely use some Javascript expertise, especially when we start working on integrating the single-cell server mentioned above with the snippet site. Thanks, Jason From gael.varoquaux at normalesup.org Tue Apr 19 01:22:52 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 19 Apr 2011 07:22:52 +0200 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: <4DAD0411.6010405@creativetrax.com> References: <4DAD0411.6010405@creativetrax.com> Message-ID: <20110419052252.GB4613@phare.normalesup.org> On Mon, Apr 18, 2011 at 10:40:01PM -0500, Jason Grout wrote: > I've been funded over the summer by an NSF grant to build a library of > Sage "interacts" (basically small snippets of Sage/Python code). > Fernando Perez strongly encouraged me at a recent Sage Days to adopt a > version controlled snippet model (ala Gist). The other night I threw > together a very rough proof-of-concept, just-barely-working initial > start of something along these lines (I hope I put enough disclaimers in > there!) My code is here: > https://github.com/jasongrout/snippets This is great. Congratulations for doing this. And congratulations for doing it in such a way that it appeals to the scipy community in addition to the sage community! Gael From gruben at bigpond.net.au Tue Apr 19 02:10:42 2011 From: gruben at bigpond.net.au (gary ruben) Date: Tue, 19 Apr 2011 16:10:42 +1000 Subject: [SciPy-User] Finding islands and centroids In-Reply-To: References: Message-ID: You might start by browsing through the pymorph documentation: http://www.mmorph.com/pymorph/morph/index.html The mmdcells example might be a starting point: http://www.mmorph.com/pymorph/morph/mmdemos/mmdcells.html Gary R. On Tue, Apr 19, 2011 at 8:37 AM, Chris Weisiger wrote: > (As you may recall from previous threads, my current project involves > aligning two arrays of pixel data that were taken from slightly different > perspectives) > > In an attempt to quantify the accuracy of the alignment I've obtained, I > want to do some centroid analysis. The images I'm working with are of > "beads" (very small fluorescent blobs), thus each 512x512 image is of a > number of more-or-less circular islands each on the order of 50 pixels or > so. I figure that I can threshold each image, identify distinct islands, get > their centroids, map those to centroids in the other wavelength, and thus > get the distance between centroids, which should be a good absolute measure > of alignment quality. > > I can hack something together to do this easily enough, where I find a pixel > in one of the islands, flood-fill out to get all connected pixels, calculate > the centroid, flip the pixels to 0, and repeat until all islands are gone. > This isn't exactly very speedy though. What's the efficient way to do this? > Is there one? Is there a better approach I should be taking? The image > processing class I dimly remember taking years ago didn't cover this kind of > thing, so I'm lacking even the basic vocabulary needed to search for > algorithms. > > -Chris > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From josh.holbrook at gmail.com Tue Apr 19 02:30:48 2011 From: josh.holbrook at gmail.com (Joshua Holbrook) Date: Mon, 18 Apr 2011 22:30:48 -0800 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: <4DAD1A3D.2090703@creativetrax.com> References: <4DAD0411.6010405@creativetrax.com> <4DAD1A3D.2090703@creativetrax.com> Message-ID: On Mon, Apr 18, 2011 at 9:14 PM, Jason Grout wrote: > The site would be simply a snippet site, geared towards python. > Additionally, for each snippet, it would be able to send the snippet to > a server to execute the snippet and display the results, like the > htmlnotebook project in the IPython trunk. ?Except we'd probably also > use another project we're currently working on, a "single-cell Sage > Notebook server": https://github.com/jasongrout/simple-python-db-compute > (see http://wiki.sagemath.org/DrakeSageGroup for rough notes of our > progress and todo list). ?The "interacts" part just is some client-side > javascript that makes sliders, textboxes, etc. This sounds pretty sweet. What do you mean by "single-cell?" Looking forward to seeing how it goes! Either way, I'm watching the compute server project now too. > > We could definitely use some Javascript expertise, especially when we > start working on integrating the single-cell server mentioned above with > the snippet site. > I'd hardly call myself a javascript expert XD but feel free to hit me up if/when you're ready. You might find it easiest to get ahold of me in the #scipy irc channel (irc://irc.freenode.net/#scipy) as "jesusabdullah." Good luck! --Josh From pav at iki.fi Tue Apr 19 04:48:17 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 19 Apr 2011 08:48:17 +0000 (UTC) Subject: [SciPy-User] Central File Exchange for Scipy References: <4DAD0411.6010405@creativetrax.com> Message-ID: Mon, 18 Apr 2011 22:40:01 -0500, Jason Grout wrote: [clip] > I won't be able to work on this heavily for about a month (until after > the semester), but I will be hitting it pretty hard during the summer. > The hope is to have a good production version by July. > > If anyone else is working on a related project, please let me know, as > we can probably collaborate. If anyone wants to fork the github repo > above and work on it, feel free! I have something here: https://github.com/pv/scipyshare Version controlled, yes, but on the file system, and not only for snippets but also for slightly larger projects + reference links to projects on PyPi and elsewhere. There's no great UI yet, but the basic functionality is there. The community part functionality is not fully there yet, though. Pauli From pav at iki.fi Tue Apr 19 04:51:25 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 19 Apr 2011 08:51:25 +0000 (UTC) Subject: [SciPy-User] Central File Exchange for Scipy References: <4DAD0411.6010405@creativetrax.com> Message-ID: Tue, 19 Apr 2011 08:48:17 +0000, Pauli Virtanen wrote: [clip] > There's no great UI yet, but the basic functionality is there. Should have said that there's essentially *no* real UI yet, it's basically just dumping data and forms to HTML; but real UI should be easy to add once the rest of the things work. Pauli From nwerneck at gmail.com Tue Apr 19 08:00:18 2011 From: nwerneck at gmail.com (Nicolau Werneck) Date: Tue, 19 Apr 2011 09:00:18 -0300 Subject: [SciPy-User] Finding islands and centroids In-Reply-To: References: Message-ID: <20110419120018.GA2694@spirit> You should try OpenCV, which has bindings for python. It has functions to perform many tasks related to this, including thresholding an image, and finding and labeling blobs. I think it can even returns lists of blob centroids... I don't know if you have lots of points in each image, but you may also take a look at the kd-tree classes in scipy for finding the matching points in the images. I started using it the other day for something similar, and it's a breeze! ++nic On Mon, Apr 18, 2011 at 03:37:51PM -0700, Chris Weisiger wrote: > (As you may recall from previous threads, my current project involves > aligning two arrays of pixel data that were taken from slightly different > perspectives) > > In an attempt to quantify the accuracy of the alignment I've obtained, I > want to do some centroid analysis. The images I'm working with are of > "beads" (very small fluorescent blobs), thus each 512x512 image is of a > number of more-or-less circular islands each on the order of 50 pixels or > so. I figure that I can threshold each image, identify distinct islands, > get their centroids, map those to centroids in the other wavelength, and > thus get the distance between centroids, which should be a good absolute > measure of alignment quality. > > I can hack something together to do this easily enough, where I find a > pixel in one of the islands, flood-fill out to get all connected pixels, > calculate the centroid, flip the pixels to 0, and repeat until all islands > are gone. This isn't exactly very speedy though. What's the efficient way > to do this? Is there one? Is there a better approach I should be taking? > The image processing class I dimly remember taking years ago didn't cover > this kind of thing, so I'm lacking even the basic vocabulary needed to > search for algorithms. > > -Chris > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Nicolau Werneck C3CF E29F 5350 5DAA 3705 http://www.lti.pcs.usp.br/~nwerneck 7B9E D6C4 37BB DA64 6F15 Linux user #460716 "There is only one corner of the universe you can be certain of improving and that is your own self." -- Aldous Huxley From william.ratcliff at gmail.com Tue Apr 19 09:11:40 2011 From: william.ratcliff at gmail.com (william ratcliff) Date: Tue, 19 Apr 2011 09:11:40 -0400 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> Message-ID: Andrew Wilson and I were also playing with this--one thing I was looking into before getting waylaid by work was using S3 as a backing store for git/hg. William On Tue, Apr 19, 2011 at 4:51 AM, Pauli Virtanen wrote: > Tue, 19 Apr 2011 08:48:17 +0000, Pauli Virtanen wrote: > [clip] > > There's no great UI yet, but the basic functionality is there. > > Should have said that there's essentially *no* real UI yet, it's > basically just dumping data and forms to HTML; but real UI should be easy > to add once the rest of the things work. > > Pauli > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jh at physics.ucf.edu Tue Apr 19 09:29:52 2011 From: jh at physics.ucf.edu (Joe Harrington) Date: Tue, 19 Apr 2011 09:29:52 -0400 Subject: [SciPy-User] Finding islands and centroids In-Reply-To: (scipy-user-request@scipy.org) References: Message-ID: Finding multiple point-like (or at least peaked) sources in an image is a common task in astronomy. One approach is to threshold the image and then fit a point-spread function, or even just a simple Gaussian, to each local high point. Subtract all the models and do it again iteratively until you are left with noise. There are packages that do this in IRAF (DAOPHOT is one), which can be accessed through PyRAF. There may be something in astropy as well. On the accuracy and precision of centering routines, my group has published a preliminary assessment; see the online Supplementary Information to Stevenson, K. B., J. Harrington, S. Nymeyer, N. Madhusudhan, S. Seager, W. C. Bowman, R. Hardy, D. Deming, E. Rauscher, and N. Lust 2010. Possible thermochemical disequilibrium in the atmosphere of the exoplanet GJ 436b. Nature 464, 1161?1164. We have a more detailed article in draft that I can send to you, if centipixel accuracy is needed. --jh-- On Tue, Apr 19, 2011 at 8:37 AM, Chris Weisiger wrote: > (As you may recall from previous threads, my current project involves > aligning two arrays of pixel data that were taken from slightly different > perspectives) > > In an attempt to quantify the accuracy of the alignment I've obtained, I > want to do some centroid analysis. The images I'm working with are of > "beads" (very small fluorescent blobs), thus each 512x512 image is of a > number of more-or-less circular islands each on the order of 50 pixels or > so. I figure that I can threshold each image, identify distinct islands, get > their centroids, map those to centroids in the other wavelength, and thus > get the distance between centroids, which should be a good absolute measure > of alignment quality. > > I can hack something together to do this easily enough, where I find a pixel > in one of the islands, flood-fill out to get all connected pixels, calculate > the centroid, flip the pixels to 0, and repeat until all islands are gone. > This isn't exactly very speedy though. What's the efficient way to do this? > Is there one? Is there a better approach I should be taking? The image > processing class I dimly remember taking years ago didn't cover this kind of > thing, so I'm lacking even the basic vocabulary needed to search for > algorithms. > > -Chris > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From alan.isaac at gmail.com Tue Apr 19 09:35:16 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Tue, 19 Apr 2011 09:35:16 -0400 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: <4DAD0411.6010405@creativetrax.com> References: <4DAD0411.6010405@creativetrax.com> Message-ID: <4DAD8F94.1010403@gmail.com> On 4/18/2011 11:40 PM, Jason Grout wrote: > though the Mercurial docs page seems to > indicate that I'm forced to use GPLv2 (+?) if I use their API That's only if you use their *internal* API, which is discouraged. fwiw, Alan Isaac From waterbug at pangalactic.us Tue Apr 19 10:55:22 2011 From: waterbug at pangalactic.us (Stephen Waterbury) Date: Tue, 19 Apr 2011 10:55:22 -0400 Subject: [SciPy-User] How to create Interface classes at runtime? In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> Message-ID: <4DADA25A.3090906@pangalactic.us> I've been reading the traits modules has_traits.py and protocols.py and trying to figure out what I'd use to create Interface classes at runtime, but my brain is beginning to melt so thought I'd ask for some advice. I have code that creates zope.interface-style Interface classes at runtime by instantiating InterfaceClass, but PyProtocols, which is used by traits, has a significantly different set of apis and the code is somewhat more difficult to read than zope.interface code, so it's not obvious how to create traits-style Interface classes at runtime ... any hints would be appreciated! Thanks, Steve From robert.kern at gmail.com Tue Apr 19 11:07:24 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Apr 2011 10:07:24 -0500 Subject: [SciPy-User] How to create Interface classes at runtime? In-Reply-To: <4DADA25A.3090906@pangalactic.us> References: <4DAD0411.6010405@creativetrax.com> <4DADA25A.3090906@pangalactic.us> Message-ID: On Tue, Apr 19, 2011 at 09:55, Stephen Waterbury wrote: > I've been reading the traits modules has_traits.py and protocols.py > and trying to figure out what I'd use to create Interface classes > at runtime, but my brain is beginning to melt so thought I'd ask for > some advice. I can answer your question over on enthought-dev: https://mail.enthought.com/mailman/listinfo/enthought-dev -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From jason-sage at creativetrax.com Tue Apr 19 16:27:40 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Tue, 19 Apr 2011 15:27:40 -0500 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> Message-ID: <4DADF03C.3080804@creativetrax.com> On 4/19/11 2:00 PM, Pauli Virtanen wrote: > Since a DVCS would not be exposed to the outside of the application, I think it would be really nice if the DVCS was exposed to the extent that someone could git/hg clone their snippet/package repository and move it to github/bitbucket once it became a bigger project. On the other hand, I can imagine some people wanting to work locally with their package repository and then push to the server. I don't think we need to do a gitorious/github/bitbucket rewrite, but having a way to get your repository off the server or push to the server would make it much easier to transition projects that grow larger. I notice you have three categories of entries according to the DESIGN.rst. Here are some design questions/comments about each: 1. Hosted software-Code submitted by people directly to this site. Edited by the original submitters solely. - It would make sense to allow multiple people to commit to a package, if it's easy. If it's not, then I suppose we encourage projects that need more group-aware tools to use github or bitbucket or something and instead just link to their repository. Having a very simple "forking" feature would also be nice. 2. Pointers to externally hosted packages, hosted on PyPi, github, bitbucket, someone's homepage, etc. Editable by anyone. - Editable by anyone worries me---these are like the grown-up versions of (1), so it makes sense that the original submitter is the one to update these. 3. Short code snippets and code examples. Public domain and editable by anyone. - Sounds good, though I'm ambivalent about whether these should be wiki-style or gist-style (e.g., editable by anyone, or forkable by anyone). These are the sorts of things I envisioned having an "execute" button on that would send the code to a server, execute it, and post back the results. Finally: which pieces do you envision having a version history exposed to the user? Do you envision a "forking" or "branching" action for any of these? Jason From pav at iki.fi Tue Apr 19 17:25:13 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 19 Apr 2011 21:25:13 +0000 (UTC) Subject: [SciPy-User] Central File Exchange for Scipy References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> Message-ID: On Tue, 19 Apr 2011 15:27:40 -0500, Jason Grout wrote: > On 4/19/11 2:00 PM, Pauli Virtanen wrote: >> Since a DVCS would not be exposed to the outside of the application, > > I think it would be really nice if the DVCS was exposed to the extent > that someone could git/hg clone their snippet/package repository and > move it to github/bitbucket once it became a bigger project. On the > other hand, I can imagine some people wanting to work locally with their > package repository and then push to the server. I don't think we need > to do a gitorious/github/bitbucket rewrite, but having a way to get your > repository off the server or push to the server would make it much > easier to transition projects that grow larger. I think we have a slightly different ideas of what we are trying to make. Maybe it would be useful to try to clear up the aim before going to nitty-gritty :) My idea was to get something like Wikipedia or ohloh.net centered on scientific Python software. So the focus would be mainly on "released" software, rather than being a platform for software development, such as Sourceforce, Gitorius, or Gists. Allowing file and cookbook page hosting would then be just a side feature, just for convenience for the people who are too lazy to learn how to use bitbucket and other tools properly. In this view, use of DVCS is out of scope. Joint development would be diverted to the "traditional" channels when possible --- direct communication with the original authors, or whatever community site they use to share their work. Whether this is an useful target, is of course up to debate. Pros: + simple to implement + puts more weight to "finished" and (hopefully) more useful works Cons: - does not especially encourage collaboration *** > I notice you have three categories of entries according to the > DESIGN.rst. Here are some design questions/comments about each: > > 1. Hosted software-Code submitted by people directly to this site. > Edited by the original submitters solely. > > - It would make sense to allow multiple people to commit to a package, > if it's easy. If it's not, then I suppose we encourage projects that > need more group-aware tools to use github or bitbucket or something and > instead just link to their repository. > > Having a very simple "forking" feature would also be nice. My view would be that we only want more or less "finished" works to appear there. If so, the catalog does not really need features related to software development --- except maybe a link pointing to where the development is ongoing. > 2. Pointers to externally hosted packages, hosted on PyPi, github, > bitbucket, someone's homepage, etc. Editable by anyone. > > - Editable by anyone worries me---these are like the grown-up versions > of (1), so it makes sense that the original submitter is the one to > update these. Being editable by anyone makes sense if you think of this part as a Wikipedia/ohloh.net of scientific Python software. The main issue here is that it would be useful for other people to be able to also add links to projects you aren't an author of, improve incomplete or terse descriptions, etc. Note that these type-2 projects do not have any hosted code. The criticism on enabling vandalism and spam is a valid one. Dealing with it requires an adequate reversion & change monitoring UI, and enough manpower. As a middle way, the changes could of course be moderated manually, so that they appear only after approved by a group of editors (and/or the original submitter). But in any case, retaining the ability for anyone (registered) to submit corrections to these catalog entries seems quite useful to me. > 3. Short code snippets and code examples. Public domain and editable by > anyone. > > - Sounds good, though I'm ambivalent about whether these should be > wiki-style or gist-style (e.g., editable by anyone, or forkable by > anyone). The aim with these "snippets" was mainly to replace what's currently in scipy.org/Cookbook, rather than to build a branded competitor to Gist or similar services. Wiki-style seems like a useful choice here to me (and my calling them "snippets" was a bad choice of words --- they're not really the same). The aim that I had in mind would be more to get an organized collection of useful recipes with explanations how things work, rather than a collection of miscellaneous code snippets "owned" by different people. Wikipedia vs. Knol... > These are the sorts of things I envisioned having an "execute" > button on that would send the code to a server, execute it, and post > back the results. Needs some serious sandboxing on the server side. Sage seems to manage to do this, though. > Finally: which pieces do you envision having a version history exposed > to the user? Do you envision a "forking" or "branching" action for any > of these? The version history exposure would be essentially what you get in your average wiki. You don't normally look at it, unless you want to revert something. Forking and branching is out of scope, since the focus is on released "products" rather than their development. Pauli From william.ratcliff at gmail.com Tue Apr 19 17:28:55 2011 From: william.ratcliff at gmail.com (william ratcliff) Date: Tue, 19 Apr 2011 17:28:55 -0400 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> Message-ID: Check out apparmor.. On Apr 19, 2011 5:25 PM, "Pauli Virtanen" wrote: > On Tue, 19 Apr 2011 15:27:40 -0500, Jason Grout wrote: >> On 4/19/11 2:00 PM, Pauli Virtanen wrote: >>> Since a DVCS would not be exposed to the outside of the application, >> >> I think it would be really nice if the DVCS was exposed to the extent >> that someone could git/hg clone their snippet/package repository and >> move it to github/bitbucket once it became a bigger project. On the >> other hand, I can imagine some people wanting to work locally with their >> package repository and then push to the server. I don't think we need >> to do a gitorious/github/bitbucket rewrite, but having a way to get your >> repository off the server or push to the server would make it much >> easier to transition projects that grow larger. > > I think we have a slightly different ideas of what we are trying to make. > Maybe it would be useful to try to clear up the aim before going to > nitty-gritty :) > > My idea was to get something like Wikipedia or ohloh.net centered on > scientific Python software. So the focus would be mainly on "released" > software, rather than being a platform for software development, such as > Sourceforce, Gitorius, or Gists. > > Allowing file and cookbook page hosting would then be just a side > feature, just for convenience for the people who are too lazy to learn > how to use bitbucket and other tools properly. In this view, use of DVCS > is out of scope. Joint development would be diverted to the "traditional" > channels when possible --- direct communication with the original > authors, or whatever community site they use to share their work. > > Whether this is an useful target, is of course up to debate. > > > Pros: > > + simple to implement > > + puts more weight to "finished" and (hopefully) more useful works > > Cons: > > - does not especially encourage collaboration > > > *** > >> I notice you have three categories of entries according to the >> DESIGN.rst. Here are some design questions/comments about each: >> >> 1. Hosted software-Code submitted by people directly to this site. >> Edited by the original submitters solely. >> >> - It would make sense to allow multiple people to commit to a package, >> if it's easy. If it's not, then I suppose we encourage projects that >> need more group-aware tools to use github or bitbucket or something and >> instead just link to their repository. >> >> Having a very simple "forking" feature would also be nice. > > My view would be that we only want more or less "finished" works to > appear there. If so, the catalog does not really need features related to > software development --- except maybe a link pointing to where the > development is ongoing. > >> 2. Pointers to externally hosted packages, hosted on PyPi, github, >> bitbucket, someone's homepage, etc. Editable by anyone. >> >> - Editable by anyone worries me---these are like the grown-up versions >> of (1), so it makes sense that the original submitter is the one to >> update these. > > Being editable by anyone makes sense if you think of this part as a > Wikipedia/ohloh.net of scientific Python software. > > The main issue here is that it would be useful for other people to be > able to also add links to projects you aren't an author of, improve > incomplete or terse descriptions, etc. Note that these type-2 projects > do not have any hosted code. > > The criticism on enabling vandalism and spam is a valid one. Dealing with > it requires an adequate reversion & change monitoring UI, and enough > manpower. As a middle way, the changes could of course be moderated > manually, so that they appear only after approved by a group of editors > (and/or the original submitter). > > But in any case, retaining the ability for anyone (registered) to submit > corrections to these catalog entries seems quite useful to me. > >> 3. Short code snippets and code examples. Public domain and editable by >> anyone. >> >> - Sounds good, though I'm ambivalent about whether these should be >> wiki-style or gist-style (e.g., editable by anyone, or forkable by >> anyone). > > The aim with these "snippets" was mainly to replace what's currently in > scipy.org/Cookbook, rather than to build a branded competitor to Gist or > similar services. Wiki-style seems like a useful choice here to me (and > my calling them "snippets" was a bad choice of words --- they're not > really the same). > > The aim that I had in mind would be more to get an organized collection > of useful recipes with explanations how things work, rather than a > collection of miscellaneous code snippets "owned" by different people. > Wikipedia vs. Knol... > >> These are the sorts of things I envisioned having an "execute" >> button on that would send the code to a server, execute it, and post >> back the results. > > Needs some serious sandboxing on the server side. Sage seems to manage to > do this, though. > >> Finally: which pieces do you envision having a version history exposed >> to the user? Do you envision a "forking" or "branching" action for any >> of these? > > The version history exposure would be essentially what you get in your > average wiki. You don't normally look at it, unless you want to revert > something. > > Forking and branching is out of scope, since the focus is on released > "products" rather than their development. > > Pauli > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Tue Apr 19 17:38:41 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 19 Apr 2011 21:38:41 +0000 (UTC) Subject: [SciPy-User] Central File Exchange for Scipy References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> Message-ID: On Tue, 19 Apr 2011 17:28:55 -0400, william ratcliff wrote: > Check out apparmor.. Can you give a link? Google is fixated on the Linux security framework, which you probably don't mean here. From jason-sage at creativetrax.com Tue Apr 19 17:50:12 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Tue, 19 Apr 2011 16:50:12 -0500 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> Message-ID: <4DAE0394.2030203@creativetrax.com> On 4/19/11 4:28 PM, william ratcliff wrote: > Check out apparmor.. As well as packaginator: https://github.com/cartwheelweb/packaginator http://packaginator.readthedocs.org/en/latest/?redir Jason From robert.kern at gmail.com Tue Apr 19 17:49:58 2011 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 19 Apr 2011 16:49:58 -0500 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> Message-ID: On Tue, Apr 19, 2011 at 16:38, Pauli Virtanen wrote: > On Tue, 19 Apr 2011 17:28:55 -0400, william ratcliff wrote: >> Check out apparmor.. > > Can you give a link? Google is fixated on the Linux security framework, > which you probably don't mean here. I suspect he was only responding to the one comment on sandboxing, not the whole proposal. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pav at iki.fi Tue Apr 19 18:02:31 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 19 Apr 2011 22:02:31 +0000 (UTC) Subject: [SciPy-User] Central File Exchange for Scipy References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE0394.2030203@creativetrax.com> Message-ID: On Tue, 19 Apr 2011 16:50:12 -0500, Jason Grout wrote: > On 4/19/11 4:28 PM, william ratcliff wrote: >> Check out apparmor.. > > As well as packaginator: > > https://github.com/cartwheelweb/packaginator > > http://packaginator.readthedocs.org/en/latest/?redir Yes, I'm aware of djangopackages.com It however does not do all that is desired --- no hosting, no cookbook, no reviews. Those could, however, be possible to glue on it as separate Django apps. Pauli From pav at iki.fi Tue Apr 19 18:04:35 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 19 Apr 2011 22:04:35 +0000 (UTC) Subject: [SciPy-User] Central File Exchange for Scipy References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE0394.2030203@creativetrax.com> Message-ID: On Tue, 19 Apr 2011 16:50:12 -0500, Jason Grout wrote: > On 4/19/11 4:28 PM, william ratcliff wrote: >> Check out apparmor.. > > As well as packaginator: > > https://github.com/cartwheelweb/packaginator And djangosnippets: https://github.com/coleifer/djangosnippets.org/ From jason-sage at creativetrax.com Tue Apr 19 19:08:40 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Tue, 19 Apr 2011 18:08:40 -0500 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> Message-ID: <4DAE15F8.4090007@creativetrax.com> On 4/19/11 4:25 PM, Pauli Virtanen wrote: > On Tue, 19 Apr 2011 15:27:40 -0500, Jason Grout wrote: >> On 4/19/11 2:00 PM, Pauli Virtanen wrote: >>> Since a DVCS would not be exposed to the outside of the application, >> >> I think it would be really nice if the DVCS was exposed to the extent >> that someone could git/hg clone their snippet/package repository and >> move it to github/bitbucket once it became a bigger project. On the >> other hand, I can imagine some people wanting to work locally with their >> package repository and then push to the server. I don't think we need >> to do a gitorious/github/bitbucket rewrite, but having a way to get your >> repository off the server or push to the server would make it much >> easier to transition projects that grow larger. > > I think we have a slightly different ideas of what we are trying to make. > Maybe it would be useful to try to clear up the aim before going to > nitty-gritty :) > > My idea was to get something like Wikipedia or ohloh.net centered on > scientific Python software. So the focus would be mainly on "released" > software, rather than being a platform for software development, such as > Sourceforce, Gitorius, or Gists. > > Allowing file and cookbook page hosting would then be just a side > feature, just for convenience for the people who are too lazy to learn > how to use bitbucket and other tools properly. In this view, use of DVCS > is out of scope. Joint development would be diverted to the "traditional" > channels when possible --- direct communication with the original > authors, or whatever community site they use to share their work. > > Whether this is an useful target, is of course up to debate. Ah, you're right---I see now we may have different aims. My original idea was for a project somewhere in between pastebin/gist and github in functionality, but also having the tagging and comments features that you're proposing. Add to that a way to execute snippets, like http://python.codepad.org/ or the Sage notebook. With Sage interacts, executing the snippets of code would allow a user to play with sliders and buttons to interact with Sage/Python without having to log in to a web notebook. So I had envisioned much more of an active development site for small chunks of code, rather than a repository of pointers to packages developed on more heavyweight development sites. >> 1. Hosted software-Code submitted by people directly to this site. >> Edited by the original submitters solely. >> >> - It would make sense to allow multiple people to commit to a package, >> if it's easy. If it's not, then I suppose we encourage projects that >> need more group-aware tools to use github or bitbucket or something and >> instead just link to their repository. >> >> Having a very simple "forking" feature would also be nice. > > My view would be that we only want more or less "finished" works to > appear there. If so, the catalog does not really need features related to > software development --- except maybe a link pointing to where the > development is ongoing. So when you say "Hosted software", are you thinking of a PyPi type of site, where the release tarball might be hosted on the site, rather than the development repository? > >> 2. Pointers to externally hosted packages, hosted on PyPi, github, >> bitbucket, someone's homepage, etc. Editable by anyone. >> >> - Editable by anyone worries me---these are like the grown-up versions >> of (1), so it makes sense that the original submitter is the one to >> update these. > > Being editable by anyone makes sense if you think of this part as a > Wikipedia/ohloh.net of scientific Python software. > > The main issue here is that it would be useful for other people to be > able to also add links to projects you aren't an author of, improve > incomplete or terse descriptions, etc. Note that these type-2 projects > do not have any hosted code. > > The criticism on enabling vandalism and spam is a valid one. Dealing with > it requires an adequate reversion& change monitoring UI, and enough > manpower. As a middle way, the changes could of course be moderated > manually, so that they appear only after approved by a group of editors > (and/or the original submitter). > > But in any case, retaining the ability for anyone (registered) to submit > corrections to these catalog entries seems quite useful to me. I see. Your approach makes sense, especially with moderation and/or easily reverted edits. > >> 3. Short code snippets and code examples. Public domain and editable by >> anyone. >> >> - Sounds good, though I'm ambivalent about whether these should be >> wiki-style or gist-style (e.g., editable by anyone, or forkable by >> anyone). > > The aim with these "snippets" was mainly to replace what's currently in > scipy.org/Cookbook, rather than to build a branded competitor to Gist or > similar services. Wiki-style seems like a useful choice here to me (and > my calling them "snippets" was a bad choice of words --- they're not > really the same). > > The aim that I had in mind would be more to get an organized collection > of useful recipes with explanations how things work, rather than a > collection of miscellaneous code snippets "owned" by different people. > Wikipedia vs. Knol... Your explanation clears up a lot of confusion. Thanks. As I said, I'm not sure which style is best for the use-cases I have in mind (development/hosting of small educational snippets of code). It may be that the best approach for me is to point people to gist to do their development if they want to have forking, DVCS, etc., but then they should copy their code for the snippet website (or maybe even better, just give the gist and we can retrieve the code via the gist API). Thanks, Jason From stefan.schwarzburg at googlemail.com Wed Apr 20 02:30:50 2011 From: stefan.schwarzburg at googlemail.com (Stefan Schwarzburg) Date: Wed, 20 Apr 2011 08:30:50 +0200 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: <4DAE15F8.4090007@creativetrax.com> References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> Message-ID: Hi On Wed, Apr 20, 2011 at 01:08, Jason Grout wrote: > On 4/19/11 4:25 PM, Pauli Virtanen wrote: > > On Tue, 19 Apr 2011 15:27:40 -0500, Jason Grout wrote: > >> On 4/19/11 2:00 PM, Pauli Virtanen wrote: > >>> Since a DVCS would not be exposed to the outside of the application, > >> > >> I think it would be really nice if the DVCS was exposed to the extent > >> that someone could git/hg clone their snippet/package repository and > >> move it to github/bitbucket once it became a bigger project. On the > >> other hand, I can imagine some people wanting to work locally with their > >> package repository and then push to the server. I don't think we need > >> to do a gitorious/github/bitbucket rewrite, but having a way to get your > >> repository off the server or push to the server would make it much > >> easier to transition projects that grow larger. > > > > I think we have a slightly different ideas of what we are trying to make. > > Maybe it would be useful to try to clear up the aim before going to > > nitty-gritty :) > > > > My idea was to get something like Wikipedia or ohloh.net centered on > > scientific Python software. So the focus would be mainly on "released" > > software, rather than being a platform for software development, such as > > Sourceforce, Gitorius, or Gists. > > > > Allowing file and cookbook page hosting would then be just a side > > feature, just for convenience for the people who are too lazy to learn > > how to use bitbucket and other tools properly. In this view, use of DVCS > > is out of scope. Joint development would be diverted to the "traditional" > > channels when possible --- direct communication with the original > > authors, or whatever community site they use to share their work. > > > > Whether this is an useful target, is of course up to debate. > > > Ah, you're right---I see now we may have different aims. My original > idea was for a project somewhere in between pastebin/gist and github in > functionality, but also having the tagging and comments features that > you're proposing. Add to that a way to execute snippets, like > http://python.codepad.org/ or the Sage notebook. Did you have a look at http://aciresnippets.wordpress.com/? Although it does not do exactly what you want, it is very close. It's not a website for example, but a desktop program (acire) that interacts with a version controlled snippets repository. Maybe you could use the underlying DVCS controlled python-snippets ( http://aciresnippets.wordpress.com/contribute/) and just add a new category (Sage, scipy, ...) and build a website that interacts with these snippets. It might remove some of your work and additionally combine the work of a similar project with yours? > With Sage interacts, > executing the snippets of code would allow a user to play with sliders > and buttons to interact with Sage/Python without having to log in to a > web notebook. > > So I had envisioned much more of an active development site for small > chunks of code, rather than a repository of pointers to packages > developed on more heavyweight development sites. > > > >> 1. Hosted software-Code submitted by people directly to this site. > >> Edited by the original submitters solely. > >> > >> - It would make sense to allow multiple people to commit to a package, > >> if it's easy. If it's not, then I suppose we encourage projects that > >> need more group-aware tools to use github or bitbucket or something and > >> instead just link to their repository. > >> > >> Having a very simple "forking" feature would also be nice. > > > > My view would be that we only want more or less "finished" works to > > appear there. If so, the catalog does not really need features related to > > software development --- except maybe a link pointing to where the > > development is ongoing. > > So when you say "Hosted software", are you thinking of a PyPi type of > site, where the release tarball might be hosted on the site, rather than > the development repository? > > > > > >> 2. Pointers to externally hosted packages, hosted on PyPi, github, > >> bitbucket, someone's homepage, etc. Editable by anyone. > >> > >> - Editable by anyone worries me---these are like the grown-up versions > >> of (1), so it makes sense that the original submitter is the one to > >> update these. > > > > Being editable by anyone makes sense if you think of this part as a > > Wikipedia/ohloh.net of scientific Python software. > > > > The main issue here is that it would be useful for other people to be > > able to also add links to projects you aren't an author of, improve > > incomplete or terse descriptions, etc. Note that these type-2 projects > > do not have any hosted code. > > > > The criticism on enabling vandalism and spam is a valid one. Dealing with > > it requires an adequate reversion& change monitoring UI, and enough > > manpower. As a middle way, the changes could of course be moderated > > manually, so that they appear only after approved by a group of editors > > (and/or the original submitter). > > > > But in any case, retaining the ability for anyone (registered) to submit > > corrections to these catalog entries seems quite useful to me. > > > I see. Your approach makes sense, especially with moderation and/or > easily reverted edits. > > > > > >> 3. Short code snippets and code examples. Public domain and editable by > >> anyone. > >> > >> - Sounds good, though I'm ambivalent about whether these should be > >> wiki-style or gist-style (e.g., editable by anyone, or forkable by > >> anyone). > > > > The aim with these "snippets" was mainly to replace what's currently in > > scipy.org/Cookbook, rather than to build a branded competitor to Gist or > > similar services. Wiki-style seems like a useful choice here to me (and > > my calling them "snippets" was a bad choice of words --- they're not > > really the same). > > > > The aim that I had in mind would be more to get an organized collection > > of useful recipes with explanations how things work, rather than a > > collection of miscellaneous code snippets "owned" by different people. > > Wikipedia vs. Knol... > > > Your explanation clears up a lot of confusion. Thanks. As I said, I'm > not sure which style is best for the use-cases I have in mind > (development/hosting of small educational snippets of code). It may be > that the best approach for me is to point people to gist to do their > development if they want to have forking, DVCS, etc., but then they > should copy their code for the snippet website (or maybe even better, > just give the gist and we can retrieve the code via the gist API). > > Thanks, > > Jason > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Institut f?r Astronomie und Astrophysik Eberhard Karls Universit?t T?bingen Sand 1 - D-72076 T?bingen Tel.: 07071/29-78605 ----------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From Pierre.RAYBAUT at CEA.FR Wed Apr 20 04:00:14 2011 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Wed, 20 Apr 2011 10:00:14 +0200 Subject: [SciPy-User] [ANN] guiqwt v2.1.1 Message-ID: Hi all, I am pleased to announce that `guiqwt` v2.1.1 has been released. Main changes since `guiqwt` v2.1.0: * added support for NaNs in image plot items (default behaviour: NaN pixels are transparents) * added "oblique averaged cross section" feature * bugfixes This version of `guiqwt` includes a demo software, Sift (for Signal and Image Filtering Tool), based on `guidata` and `guiqwt`: http://packages.python.org/guiqwt/sift.html Windows users may even download the portable version of Sift 0.23 to test it without having to install anything: http://code.google.com/p/guiqwt/downloads/detail?name=sift023_portable.zip The `guiqwt` documentation with examples, API reference, etc. is available here: http://packages.python.org/guiqwt/ Based on PyQwt (plotting widgets for PyQt4 graphical user interfaces) and on the scientific modules NumPy and SciPy, guiqwt is a Python library providing efficient 2D data-plotting features (curve/image visualization and related tools) for interactive computing and signal/image processing application development. When compared to the excellent module `matplotlib`, the main advantage of `guiqwt` is performance: see http://packages.python.org/guiqwt/overview.html#performances. But `guiqwt` is more than a plotting library; it also provides: * Helper functions for data processing: see the example http://packages.python.org/guiqwt/examples.html#curve-fitting * Framework for signal/image processing application development: see http://packages.python.org/guiqwt/examples.html * And many other features like making executable Windows programs easily (py2exe helpers): see http://packages.python.org/guiqwt/disthelpers.html guiqwt plotting features are the following: guiqwt.pyplot: equivalent to matplotlib's pyplot module (pylab) supported plot items: * curves, error bar curves and 1-D histograms * images (RGB images are not supported), images with non-linear x/y scales, images with specified pixel size (e.g. loaded from DICOM files), 2-D histograms, pseudo-color images (pcolor) * labels, curve plot legends * shapes: polygon, polylines, rectangle, circle, ellipse and segment * annotated shapes (shapes with labels showing position and dimensions): rectangle with center position and size, circle with center position and diameter, ellipse with center position and diameters (these items are very useful to measure things directly on displayed images) curves, images and shapes: * multiple object selection for moving objects or editing their properties through automatically generated dialog boxes (guidata) * item list panel: move objects from foreground to background, show/hide objects, remove objects, ... * customizable aspect ratio * a lot of ready-to-use tools: plot canvas export to image file, image snapshot, image rectangular filter, etc. curves: * interval selection tools with labels showing results of computing on selected area * curve fitting tool with automatic fit, manual fit with sliders, ... images: * contrast adjustment panel: select the LUT by moving a range selection object on the image levels histogram, eliminate outliers, ... * X-axis and Y-axis cross-sections: support for multiple images, average cross-section tool on a rectangular area, ... * apply any affine transform to displayed images in real-time (rotation, magnification, translation, horizontal/vertical flip, ...) application development helpers: * ready-to-use curve and image plot widgets and dialog boxes * load/save graphical objects (curves, images, shapes) * a lot of test scripts which demonstrate guiqwt features guiqwt has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guiqwt/ Documentation, screenshots: http://packages.python.org/guiqwt/ Downloads (source + Python(x,y) plugin): http://guiqwt.googlecode.com Cheers, Pierre --- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From pav at iki.fi Wed Apr 20 08:45:50 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 20 Apr 2011 12:45:50 +0000 (UTC) Subject: [SciPy-User] Central File Exchange for Scipy References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> Message-ID: Tue, 19 Apr 2011 18:08:40 -0500, Jason Grout wrote: [clip] > Ah, you're right---I see now we may have different aims. My original > idea was for a project somewhere in between pastebin/gist and github in > functionality, but also having the tagging and comments features that > you're proposing. Add to that a way to execute snippets, like > http://python.codepad.org/ or the Sage notebook. With Sage interacts, > executing the snippets of code would allow a user to play with sliders > and buttons to interact with Sage/Python without having to log in to a > web notebook. > > So I had envisioned much more of an active development site for small > chunks of code, rather than a repository of pointers to packages > developed on more heavyweight development sites. Yes, I can see the value in having a more community-like collection of snippets. One thing to note is that there's no need to have a single backend or even the main parts of the UI shared by the snippets and the "catalog" (although this might some things simpler). The different components of the site can be loosely coupled. It should be possible to write a tagging/per-entry-comments/etc platform that can be used both for the catalog and for the snippets. (Django has support for 'generic' foreign keys, and rendering can be done via custom template tags.) So even if the end result is that the snippets need a different storage and UI approach, there would still be a non-negligible amount of code that can be shared (and perhaps some could be lifted from djangopackages.com), because a major part of the required "community" features are quite similar. Technical differences, I think, are not a reason to make two sites rather than one, as long as the "community" and "indexing" aspects are shared. I can imagine a multi-pronged approach with real python packages and snippets in different sections of the same site. Having the Sage community involved here would definitely be a big synergy advantage for this type of a site. [clip] > So when you say "Hosted software", are you thinking of a PyPi type of > site, where the release tarball might be hosted on the site, rather than > the development repository? Precisely so. The aim would be to make it less hassle to use than PyPi for a relative Python newbie. (Although uploading packages to PyPi is not hugely hassle-ful at the moment, as it is possible to do it using only the web interface.) So there would be a bit of an overlap with PyPi; one could however add some recommendations etc. to push people to use PyPi, if they are willing to jump through some extra hoops. *** At the moment, one thing seems clear: - Pointers to externally hosted projects (& semi-automatic import from PyPi) But the following are not so clear: - Hosted projects -- how much to overlap with PyPi? - Snippets -- the Wiki or the Knol? Or both? How much overlap with hosted projects? Pauli From dplepage at gmail.com Wed Apr 20 10:39:59 2011 From: dplepage at gmail.com (Daniel Lepage) Date: Wed, 20 Apr 2011 10:39:59 -0400 Subject: [SciPy-User] Bounded Linear Least-Squares Message-ID: Hi all, Does scipy have a function analogous to Matlab's lsqlin? I need to solve two problems of the form Ax = b, one subject to the constraint that 0 <= x, and one subject to 0 <= x <= 1. The first case is handled by scipy.optimize.nnls, but it doesn't support the second. I know that scipy.optimize includes several constrained optimization routines, but AFAICT they're all aimed at minimizing arbitrary functions, and as such I'd expect them to be far slower than an actual linear solver. Is there such a constrained linear solver in scipy (or numpy, or scikits.*, etc.)? Even better would be a constrained matrix factorization routine, i.e. that solves AX = B for X with A, X and B all being matrices, subject to 0 <= X <= 1, but obviously you can construct the latter from the former, so the former would suffice. Thanks, Dan Lepage From josef.pktd at gmail.com Wed Apr 20 10:54:18 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 20 Apr 2011 10:54:18 -0400 Subject: [SciPy-User] Bounded Linear Least-Squares In-Reply-To: References: Message-ID: On Wed, Apr 20, 2011 at 10:39 AM, Daniel Lepage wrote: > Hi all, > ? ?Does scipy have a function analogous to Matlab's lsqlin? I need to > solve two problems of the form Ax = b, one subject to the constraint > that 0 <= x, and one subject to 0 <= x <= 1. The first case is handled > by scipy.optimize.nnls, but it doesn't support the second. I know that > scipy.optimize includes several constrained optimization routines, but > AFAICT they're all aimed at minimizing arbitrary functions, and as > such I'd expect them to be far slower than an actual linear solver. Is > there such a constrained linear solver in scipy (or numpy, or > scikits.*, etc.)? > > Even better would be a constrained matrix factorization routine, i.e. > that solves AX = B for X with A, X and B all being matrices, subject > to 0 <= X <= 1, but obviously you can construct the latter from the > former, so the former would suffice. I don't know anything that would solve this directly, but I think that scipy.optimize.fmin_slsqp should work well in this case. Josef > > Thanks, > Dan Lepage > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From luis at luispedro.org Wed Apr 20 08:49:48 2011 From: luis at luispedro.org (Luis Pedro Coelho) Date: Wed, 20 Apr 2011 08:49:48 -0400 Subject: [SciPy-User] Finding islands and centroids In-Reply-To: References: Message-ID: <201104200849.56775.luis@luispedro.org> On Tuesday, April 19, 2011 02:10:42 am gary ruben wrote: > You might start by browsing through the pymorph documentation: > http://www.mmorph.com/pymorph/morph/index.html > > The mmdcells example might be a starting point: > http://www.mmorph.com/pymorph/morph/mmdemos/mmdcells.html By the way, that version of pymorph is very old. I adopted the project and it's available at: http://luispedro.org/software/pymorph HTH Luis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From luis at luispedro.org Wed Apr 20 08:52:43 2011 From: luis at luispedro.org (Luis Pedro Coelho) Date: Wed, 20 Apr 2011 08:52:43 -0400 Subject: [SciPy-User] Finding islands and centroids In-Reply-To: References: Message-ID: <201104200852.44275.luis@luispedro.org> On Monday, April 18, 2011 06:37:51 pm Chris Weisiger wrote: > I can hack something together to do this easily enough, where I find a > pixel in one of the islands, flood-fill out to get all connected pixels, > calculate the centroid, flip the pixels to 0, and repeat until all islands > are gone. This isn't exactly very speedy though. What's the efficient way > to do this? Is there one? Is there a better approach I should be taking? > The image processing class I dimly remember taking years ago didn't cover > this kind of thing, so I'm lacking even the basic vocabulary needed to > search for algorithms. You can probably do it in one go: 1) convolve with a Gaussian of roughly the size you expect the PFS of the beads to be. 2) threshold to get the beads. You can probably use a method like mahotas.threshold.otsu to do this automatically 3) scipy.ndimage.label to label the binary image 4, optional) remove noise by removing detections that are too small 5) find centroids of the remaining areas. HTH Luis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From cweisiger at msg.ucsf.edu Wed Apr 20 14:11:55 2011 From: cweisiger at msg.ucsf.edu (Chris Weisiger) Date: Wed, 20 Apr 2011 11:11:55 -0700 Subject: [SciPy-User] Finding islands and centroids In-Reply-To: <201104200852.44275.luis@luispedro.org> References: <201104200852.44275.luis@luispedro.org> Message-ID: Wow, for some reason I didn't get any of these emails until just now. You all have given me plenty of avenues to investigate. Thanks! -Chris On Wed, Apr 20, 2011 at 5:52 AM, Luis Pedro Coelho wrote: > On Monday, April 18, 2011 06:37:51 pm Chris Weisiger wrote: > > I can hack something together to do this easily enough, where I find a > > pixel in one of the islands, flood-fill out to get all connected pixels, > > calculate the centroid, flip the pixels to 0, and repeat until all > islands > > are gone. This isn't exactly very speedy though. What's the efficient way > > to do this? Is there one? Is there a better approach I should be taking? > > The image processing class I dimly remember taking years ago didn't cover > > this kind of thing, so I'm lacking even the basic vocabulary needed to > > search for algorithms. > > You can probably do it in one go: > > 1) convolve with a Gaussian of roughly the size you expect the PFS of the > beads to be. > > 2) threshold to get the beads. You can probably use a method like > mahotas.threshold.otsu to do this automatically > > 3) scipy.ndimage.label to label the binary image > > 4, optional) remove noise by removing detections that are too small > > 5) find centroids of the remaining areas. > > HTH > Luis > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dplepage at gmail.com Wed Apr 20 14:22:33 2011 From: dplepage at gmail.com (Daniel Lepage) Date: Wed, 20 Apr 2011 14:22:33 -0400 Subject: [SciPy-User] Bounded Linear Least-Squares In-Reply-To: References: Message-ID: On Wed, Apr 20, 2011 at 10:54 AM, wrote: > On Wed, Apr 20, 2011 at 10:39 AM, Daniel Lepage wrote: >> Hi all, >> ? ?Does scipy have a function analogous to Matlab's lsqlin? I need to >> solve two problems of the form Ax = b, one subject to the constraint >> that 0 <= x, and one subject to 0 <= x <= 1. The first case is handled >> by scipy.optimize.nnls, but it doesn't support the second. I know that >> scipy.optimize includes several constrained optimization routines, but >> AFAICT they're all aimed at minimizing arbitrary functions, and as >> such I'd expect them to be far slower than an actual linear solver. Is >> there such a constrained linear solver in scipy (or numpy, or >> scikits.*, etc.)? >> >> Even better would be a constrained matrix factorization routine, i.e. >> that solves AX = B for X with A, X and B all being matrices, subject >> to 0 <= X <= 1, but obviously you can construct the latter from the >> former, so the former would suffice. > > I don't know anything that would solve this directly, but I think that > > scipy.optimize.fmin_slsqp > > should work well in this case. It will work, but more slowly than a linear solver because it doesn't assume the function to be linear. To even hope to get comparable speeds, I'd need to enter the derivatives of my system by hand, which is possible but a pain, and even then I expect it will take longer. Fortunately, I figured out how to restructure my particular problem so that I only need nonnegativity, so I'll just use nnls. Thanks, Dan Lepage From josef.pktd at gmail.com Wed Apr 20 14:36:06 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 20 Apr 2011 14:36:06 -0400 Subject: [SciPy-User] Bounded Linear Least-Squares In-Reply-To: References: Message-ID: On Wed, Apr 20, 2011 at 2:22 PM, Daniel Lepage wrote: > On Wed, Apr 20, 2011 at 10:54 AM, ? wrote: >> On Wed, Apr 20, 2011 at 10:39 AM, Daniel Lepage wrote: >>> Hi all, >>> ? ?Does scipy have a function analogous to Matlab's lsqlin? I need to >>> solve two problems of the form Ax = b, one subject to the constraint >>> that 0 <= x, and one subject to 0 <= x <= 1. The first case is handled >>> by scipy.optimize.nnls, but it doesn't support the second. I know that >>> scipy.optimize includes several constrained optimization routines, but >>> AFAICT they're all aimed at minimizing arbitrary functions, and as >>> such I'd expect them to be far slower than an actual linear solver. Is >>> there such a constrained linear solver in scipy (or numpy, or >>> scikits.*, etc.)? >>> >>> Even better would be a constrained matrix factorization routine, i.e. >>> that solves AX = B for X with A, X and B all being matrices, subject >>> to 0 <= X <= 1, but obviously you can construct the latter from the >>> former, so the former would suffice. >> >> I don't know anything that would solve this directly, but I think that >> >> scipy.optimize.fmin_slsqp >> >> should work well in this case. > > It will work, but more slowly than a linear solver because it doesn't > assume the function to be linear. To even hope to get comparable > speeds, I'd need to enter the derivatives of my system by hand, which > is possible but a pain, and even then I expect it will take longer. > > Fortunately, I figured out how to restructure my particular problem so > that I only need nonnegativity, so I'll just use nnls. Since it's just a linear error function (quadratic objective function), the derivatives should be easy. I was just hoping you figure out a recipe example for this since I also have applications for this case and haven't looked at it yet. :) Josef > > Thanks, > Dan Lepage > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From dominique.orban at gmail.com Wed Apr 20 15:29:54 2011 From: dominique.orban at gmail.com (Dominique Orban) Date: Wed, 20 Apr 2011 15:29:54 -0400 Subject: [SciPy-User] Bounded Linear Least-Squares Message-ID: On Wed, Apr 20, 2011 at 1:00 PM, wrote: > From: Daniel Lepage > To: SciPy Users List > Date: Wed, 20 Apr 2011 10:39:59 -0400 > Subject: [SciPy-User] Bounded Linear Least-Squares > Hi all, > Does scipy have a function analogous to Matlab's lsqlin? I need to > solve two problems of the form Ax = b, one subject to the constraint > that 0 <= x, and one subject to 0 <= x <= 1. The first case is handled > by scipy.optimize.nnls, but it doesn't support the second. I know that > scipy.optimize includes several constrained optimization routines, but > AFAICT they're all aimed at minimizing arbitrary functions, and as > such I'd expect them to be far slower than an actual linear solver. Is > there such a constrained linear solver in scipy (or numpy, or > scikits.*, etc.)? > > Even better would be a constrained matrix factorization routine, i.e. > that solves AX = B for X with A, X and B all being matrices, subject > to 0 <= X <= 1, but obviously you can construct the latter from the > former, so the former would suffice. > Your subject line says "least squares", so are you trying to find an x that exactly satisfies Ax=b subject to 0 <= x <= 1, or are you trying to minimize the residual norm ||b - Ax|| subject to the constraint 0 <= x <= 1 ? If you want the former, you could formulate the feasibility problem minimize 0 subject to Ax=b and 0 <= x <= 1. This is simply a linear programming problem. If you want the latter, you'll want to look into bound-constrained least-squares which I'm not sure Scipy covers. There is a Matlab code here: http://www.cs.ubc.ca/~mpf/bcls There is a Fortran code here: http://people.sc.fsu.edu/~jburkardt/f_src/dqed/dqed.html Either problem can be solved with CVXMOD: http://cvxmod.net (unless it is very large perhaps). If A is very large, NLPy has a sparse LP solver, but it requires A to be available explicitly: https://github.com/dpo/nlpy It also has a sparse QP solver but it will not exploit the least-squares structure. Hope that helps. -- Dominique -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanleeuwen.martin at gmail.com Wed Apr 20 16:08:35 2011 From: vanleeuwen.martin at gmail.com (Martin van Leeuwen) Date: Wed, 20 Apr 2011 13:08:35 -0700 Subject: [SciPy-User] Finding islands and centroids In-Reply-To: References: <201104200852.44275.luis@luispedro.org> Message-ID: Hi Chris, You can also compute this from a stack of incrementally thresholded images. If a pixel is a local maximum it would have no blobs above it and be contained in a blob below. This is described in: Ying, H. , Zhou, F. , Shields, A. F. , Muzik, O. , Wu, D. and Heath, E. I. (2004) A novel computerized approach to enhancing lung tumor detection in whole-body PET images. Proceedings of the 26th Annual International Conference of the IEEE EMBS San Francisco, CA IEEE EMBS Electronic Resource Piscataway, NJ and I also used it (and had written it in IDL at the time): Van Leeuwen, M., Coops, N.C., Wulder, M.A. (2010) Canopy Surface Reconstruction from a LiDAR point cloud Using Hough Transform. Remote Sensing Letters 1: 125 ? 132, doi: 10.1080/01431161003649339 Hope that is useful enough, unfortunately I don't have any python code for this so that I could share. Martin. 2011/4/20 Chris Weisiger : > Wow, for some reason I didn't get any of these emails until just now. You > all have given me plenty of avenues to investigate. Thanks! > > -Chris > > On Wed, Apr 20, 2011 at 5:52 AM, Luis Pedro Coelho > wrote: >> >> On Monday, April 18, 2011 06:37:51 pm Chris Weisiger wrote: >> > I can hack something together to do this easily enough, where I find a >> > pixel in one of the islands, flood-fill out to get all connected pixels, >> > calculate the centroid, flip the pixels to 0, and repeat until all >> > islands >> > are gone. This isn't exactly very speedy though. What's the efficient >> > way >> > to do this? Is there one? Is there a better approach I should be taking? >> > The image processing class I dimly remember taking years ago didn't >> > cover >> > this kind of thing, so I'm lacking even the basic vocabulary needed to >> > search for algorithms. >> >> You can probably do it in one go: >> >> 1) convolve with a Gaussian of roughly the size you expect the PFS of the >> beads to be. >> >> 2) threshold to get the beads. You can probably use a method like >> mahotas.threshold.otsu to do this automatically >> >> 3) scipy.ndimage.label to label the binary image >> >> 4, optional) remove noise by removing detections that are too small >> >> 5) find centroids of the remaining areas. >> >> HTH >> Luis >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From wilson.andrew.j at gmail.com Wed Apr 20 14:11:33 2011 From: wilson.andrew.j at gmail.com (Andy Wilson) Date: Wed, 20 Apr 2011 13:11:33 -0500 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> Message-ID: Sorry for showing up late to the party... fwiw, I did some experimenting with flask+git. It is really rough but if you are interested you can find it here: https://github.com/wilsaj/gitsnippets I think using either git or hg for snippets is a good idea because it saves some trouble by giving you some nice things for free, namely: history of edits and the ability to rollback changes (in case of vandalism or errors). If you keep things linear and don't bother with forking or branching/merging then it isn't very different from writing to the filesystem, but revision history is automatically recorded. Repos are also self-contained and easy to manage. There are good reasons to NOT expose snippets via DVCS. In order for people to be able to push back a cloned repo, you have to have to deal with authorization and be ready to handle branches. That's a lot of complexity and there are already systems that are very good at hosting DVCS, so diminishing returns... The 'coming soon' parts of the github gist API look pretty enticing, no dice so far: http://develop.github.com/p/gist.html -andy -------------- next part -------------- An HTML attachment was scrubbed... URL: From guziy.sasha at gmail.com Thu Apr 21 01:30:47 2011 From: guziy.sasha at gmail.com (Oleksandr Huziy) Date: Thu, 21 Apr 2011 01:30:47 -0400 Subject: [SciPy-User] bug in scipy.optimize.slsqp.py Message-ID: Hello, I am not sure whether this is a place to write about it (but I did not manage to file a ticket to truc so here it is). I found a little bug in scipy.optimize.slsqp.py(line 265): for i in range(len(bounds)): if bounds[i][0] > bounds[i][1]: raise ValueError('SLSQP Error: lb > ub in bounds[' + str(i) + '] ' + str(bounds[4])) ------ but should be: for i in range(len(bounds)): if bounds[i][0] > bounds[i][1]: raise ValueError('SLSQP Error: lb > ub in bounds[' + str(i) + '] ' + str(bounds[i])) -- Oleksandr Huziy From jason-sage at creativetrax.com Thu Apr 21 02:18:35 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Thu, 21 Apr 2011 01:18:35 -0500 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> Message-ID: <4DAFCC3B.4070902@creativetrax.com> On 4/20/11 1:11 PM, Andy Wilson wrote: > > Sorry for showing up late to the party... > > > fwiw, I did some experimenting with flask+git. It is really rough but if > you are interested you can find it here: > https://github.com/wilsaj/gitsnippets > > > I think using either git or hg for snippets is a good idea because it > saves some trouble by giving you some nice things for free, namely: > history of edits and the ability to rollback changes (in case of > vandalism or errors). If you keep things linear and don't bother with > forking or branching/merging then it isn't very different from writing > to the filesystem, but revision history is automatically recorded. Repos > are also self-contained and easy to manage. > > There are good reasons to NOT expose snippets via DVCS. In order for > people to be able to push back a cloned repo, you have to have to deal > with authorization and be ready to handle branches. That's a lot of > complexity and there are already systems that are very good at hosting > DVCS, so diminishing returns... In playing with gist, it appears that it only shows the master branch in the web UI. Any other branches are impossible to create using the web UI (only the master HEAD can be edited), and if they are pushed to the underlying git repository, they are ignored. So that's one way of dealing with the branching problem with a DVCS. Jason From jason-sage at creativetrax.com Thu Apr 21 04:07:29 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Thu, 21 Apr 2011 03:07:29 -0500 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> Message-ID: <4DAFE5C1.2090705@creativetrax.com> On 4/20/11 7:45 AM, Pauli Virtanen wrote: > Tue, 19 Apr 2011 18:08:40 -0500, Jason Grout wrote: > [clip] >> Ah, you're right---I see now we may have different aims. My original >> idea was for a project somewhere in between pastebin/gist and github in >> functionality, but also having the tagging and comments features that >> you're proposing. Add to that a way to execute snippets, like >> http://python.codepad.org/ or the Sage notebook. With Sage interacts, >> executing the snippets of code would allow a user to play with sliders >> and buttons to interact with Sage/Python without having to log in to a >> web notebook. >> >> So I had envisioned much more of an active development site for small >> chunks of code, rather than a repository of pointers to packages >> developed on more heavyweight development sites. > > Yes, I can see the value in having a more community-like collection of > snippets. > > One thing to note is that there's no need to have a single backend or > even the main parts of the UI shared by the snippets and the > "catalog" (although this might some things simpler). The different > components of the site can be loosely coupled. > > It should be possible to write a tagging/per-entry-comments/etc platform > that can be used both for the catalog and for the snippets. (Django has > support for 'generic' foreign keys, and rendering can be done via custom > template tags.) So even if the end result is that the snippets need a > different storage and UI approach, there would still be a non-negligible > amount of code that can be shared (and perhaps some could be lifted from > djangopackages.com), because a major part of the required "community" > features are quite similar. > > Technical differences, I think, are not a reason to make two sites rather > than one, as long as the "community" and "indexing" aspects are shared. I > can imagine a multi-pronged approach with real python packages and > snippets in different sections of the same site. +1. Those goals are definitely shared between our two needs. > > Having the Sage community involved here would definitely be a big synergy > advantage for this type of a site. +1; and the same can be said from the Sage side for the scipy and wider scientific python community. > > [clip] >> So when you say "Hosted software", are you thinking of a PyPi type of >> site, where the release tarball might be hosted on the site, rather than >> the development repository? > > Precisely so. The aim would be to make it less hassle to use than PyPi > for a relative Python newbie. (Although uploading packages to PyPi is not > hugely hassle-ful at the moment, as it is possible to do it using only > the web interface.) So there would be a bit of an overlap with PyPi; one > could however add some recommendations etc. to push people to use PyPi, > if they are willing to jump through some extra hoops. What extra hoops? 1. Create an account on PyPi (make up username/password, click the email verification link) 2. Log in and go to their web form: http://pypi.python.org/pypi?%3Aaction=submit_form Fill out info and submit? (I didn't actually do this, but I'm assuming it's easy). Okay, there were some concerns about the process: 1. It seemed like the connection was *not* over SSL, which always concerns me when we have passwords going back and forth 2. It wouldn't let me use the web submission form when I logged in using OpenID. However, these seem like minor technical issues that could be solved. > > *** > > At the moment, one thing seems clear: > > - Pointers to externally hosted projects (& semi-automatic > import from PyPi) I just read up more on PyPi, and in particular, read up on their recent discussion which led to the disabling of the rating system [1]. I'm not convinced that this point is clear. How is our pointing to packages on PyPi and elsewhere improving on PyPi? Are there a number of other packages out there that are not cataloged on PyPi and which should not be cataloged there? I could see us adding value by having a better tagging system that was customized more for scientific software. On the other hand, maybe we could just improve the PyPi entries for such software so that a keyword search would pull up the packages. > > But the following are not so clear: > > - Hosted projects -- how much to overlap with PyPi? If it's easy to host on PyPi, it seems like we should point people over there. We have far fewer users (and infrastructure maintainers) than PyPi, and PyPi itself already has the authoritative blessing of Python. > > - Snippets -- the Wiki or the Knol? Or both? How much overlap with > hosted projects? The python snippet repository is: http://code.activestate.com/recipes/langs/python/ Advantages: 1. It already has a tagging and comment system 2. It is "blessed" in that http://wiki.python.org/moin/PublishingPythonModules points to it as the only active snippet repository 3. It is hosted and maintained by someone else (so no hassle for us) 4. It allows revisions and "forking" a snippet, so has some community development advantages. These don't seem as easy as gist, though. 5. You can select a license for a snippet: Apache, BSD, GPL 2/3, LGPL, MIT, PSF, and others are options 6. It already has thousands of recipes. Disadvantages 1. It is hosted by someone else, and that someone else seems to have commercial interests (for example, the sign-in form asks for your company, and you constantly have ads on the site). 2. There is no execution of snippets (needed for the Sage interact database we're building, so I'll have to do something more than it anyway) So: thoughts on the scope of this new project, and how it differentiates enough from the existing sites to be useful enough to build and maintain? Thanks, Jason [1] http://mail.python.org/pipermail/catalog-sig/2011-April/003542.html (and many, many replies to the thread) From pav at iki.fi Thu Apr 21 08:37:01 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 21 Apr 2011 12:37:01 +0000 (UTC) Subject: [SciPy-User] Central File Exchange for Scipy References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> <4DAFE5C1.2090705@creativetrax.com> Message-ID: Thu, 21 Apr 2011 03:07:29 -0500, Jason Grout wrote: [clip] >>> So when you say "Hosted software", are you thinking of a PyPi type of >>> site, where the release tarball might be hosted on the site, rather >>> than the development repository? >> >> Precisely so. The aim would be to make it less hassle to use than PyPi >> for a relative Python newbie. (Although uploading packages to PyPi is >> not hugely hassle-ful at the moment, as it is possible to do it using >> only the web interface.) So there would be a bit of an overlap with >> PyPi; one could however add some recommendations etc. to push people to >> use PyPi, if they are willing to jump through some extra hoops. > > What extra hoops? Mainly, for small pieces of code, you might not even want to create a named Python package. So some form of code hosting seems to be useful --- whether it is called "snippets" (multi-file) or "hosted projects". (The wiki-style Cookbook content is then the third category of items that could be useful to have.) Re: PyPi usability You cannot just upload a .py file onto PyPi. PyPi checks that the uploaded file (i) is a tarball, zip, or egg, and (ii) is named in a specific way, and, (iii) you need to go to a different site. So there are user experience issues with the web upload. Sure, the upload is manageable once you practice a bit, and many of the issues are probably fixable. [clip] >> At the moment, one thing seems clear: >> >> - Pointers to externally hosted projects (& semi-automatic >> import from PyPi) > > I just read up more on PyPi, and in particular, read up on their recent > discussion which led to the disabling of the rating system [1]. I'm not > convinced that this point is clear. How is our pointing to packages on > PyPi and elsewhere improving on PyPi? Are there a number of other > packages out there that are not cataloged on PyPi and which should not > be cataloged there? By the "clear" thing, I mean pointing to packages on PyPi (= almost all externally hosted projects), augmented with community tags etc. Pointing to packages outside PyPi is not essential --- but would be easy to add. One thing running for allowing "external" links is that adding packages to PyPi can only be done by their authors. There is currently a small number of relevant packages usable from Python that are not on PyPi (although they should); for example PyTrilinos. But I guess it should be possible to browbeat their authors to add a PyPi entry. > I could see us adding value by having a better tagging system that was > customized more for scientific software. On the other hand, maybe we > could just improve the PyPi entries for such software so that a keyword > search would pull up the packages. It seems clear to me that this feature would be useful. Also, combining the PyPi data with smaller code snippets would create a one-stop-shop. As you can surmise from the discussion you linked to, there is resistance in adding new community-oriented features to PyPi itself, as some people feel that such features are out-of-scope for it. Doing it externally also makes sense from the usability and branding point of view --- a site called "Python in science" with filtered package selection can be more convincing and convenient to navigate than browsing "Topic :: Scientific/ Engineering" on PyPi. The discussion on rating systems there is an useful read --- it's why I left out any star-based rating systems so far. Just adding "I use this" popularity measure probably works around most issues. (The PyPi download data is not a very reliable measure, as many of the bigger packages host their files externally.) *** On improving the entries on PyPi -- the keywords etc. there are editable only by the original submitter, and this will probably not change, so I don't think that will be a possible way to go. The PyPi package classifiers as they are now are assigned solely by the package authors, more or less at random and from a limited selection, and are not very reliable. There are several packages in the "Topic :: Scientific/Engineering" category that don't actually have much focus on either science or engineering. So, some filtering of PyPi entries would already be useful. > > But the following are not so clear: > > > > - Hosted projects -- how much to overlap with PyPi? > > If it's easy to host on PyPi, it seems like we should point people over > there. We have far fewer users (and infrastructure maintainers) than > PyPi, and PyPi itself already has the authoritative blessing of Python. For small contributions, you might not want to use PyPi. Aside from that, the hosted projects are not really required, provided PyPi is easy enough to use (which is not true for the setup.py way, but may be true for the web interace). > > - Snippets -- the Wiki or the Knol? Or both? How much overlap with > > hosted projects? > > The python snippet repository is: > http://code.activestate.com/recipes/langs/python/ [clip] The activestate snippet library seems not to be very actively used for Scipy et al. at the moment -- there are only ~15 recipes tagged with "scipy", "numpy", "scientific", "sage", or "science". Also: - Since it's not a focused site, relevant code snippets are mixed with non-relevant ones, including ones written in languages other than Python. - As tags are specified by the users, and free-form, stuff will be lost in the midst of non-relevant content. - The tagging feature could perhaps be improved --- it appears they can only be assigned by the author of the snippet. - There are some usability problems: e.g. clicking the "Tags" link on the top takes you away from Python-specific content. - The search feature is not especially good: it's just Google's site: search, so it does not explicitly know about tags or metadata. Other than that, it seems to do a reasonable work. [clip] > So: thoughts on the scope of this new project, and how it differentiates > enough from the existing sites to be useful enough to build and > maintain? Already focusing on "scientific" content is a differentiation big enough, IMHO. It's mostly a social question of creating a hub for exchanging this type of content; and also a question of branding. Technically, sure, there is not so much new under the sun. The first point would just to be to make the implementation slick and useful enough to attract people to a single place. The second point would be to provide a one-stop-shop for whatever you need related to Python in science --- which would have the extra benefit of showcasing that it is doing well, and is a credible tool for many purposes. At least based on earlier discussions on this list, it seems that at least the people who chimed in would prefer such a central hub over what is currently available. The current situation is, if I want to share a something science-related written in Python, it is not obvious where I should put it so that there would be some audience. For small contributions, in generic snippet sites your stuff gets easily lost in the middle of non-relevant content --- also, I'm not convinced many people (e.g. those on this mailing list) follow those. The scipy.org/Cookbook is also not very usable, as it's a generic wiki. For larger contributions, PyPi works (although it's mildly clumsy to use), but it does not offer much visibility. If I name my package as scikits.* it goes to scikits.appspot.com, but I guess that's not very widely used either. So that's the motivation. The snippet/hosted-projects part alone would address a part of what is missing, but I think one might as well go and make a one-stop-shop out of it. Best, Pauli From josef.pktd at gmail.com Thu Apr 21 09:46:05 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 21 Apr 2011 09:46:05 -0400 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> <4DAFE5C1.2090705@creativetrax.com> Message-ID: On Thu, Apr 21, 2011 at 8:37 AM, Pauli Virtanen wrote: > Thu, 21 Apr 2011 03:07:29 -0500, Jason Grout wrote: > [clip] > As you can surmise from the discussion you linked to, there is resistance > in adding new community-oriented features to PyPi itself, as some people > feel that such features are out-of-scope for it. Doing it externally also > makes sense from the usability and branding point of view --- a site > called "Python in science" with filtered package selection can be more > convincing and convenient to navigate than browsing "Topic :: Scientific/ > Engineering" on PyPi. > > The discussion on rating systems there is an useful read --- it's why I > left out any star-based rating systems so far. Just adding "I use this" > popularity measure probably works around most issues. (The PyPi download > data is not a very reliable measure, as many of the bigger packages host > their files externally.) Sorry to add a comment here. The thread on PyPi rating system was dominated by a few developers of big or famous packages. Ratings for Django or numpy or scipy might not be useful, but ratings (with required comments) are very useful for the huge number of smaller packages and will be for "snippets". matlab fileexchange is a good example for this. Josef From mail.till at gmx.de Thu Apr 21 10:20:34 2011 From: mail.till at gmx.de (Till Stensitzki) Date: Thu, 21 Apr 2011 14:20:34 +0000 (UTC) Subject: [SciPy-User] Bounded Linear Least-Squares References: Message-ID: Daniel Lepage gmail.com> writes: > > Hi all, > Does scipy have a function analogous to Matlab's lsqlin? I need to > solve two problems of the form Ax = b, one subject to the constraint > that 0 <= x, and one subject to 0 <= x <= 1. The first case is handled > by scipy.optimize.nnls, but it doesn't support the second. I know that > scipy.optimize includes several constrained optimization routines, but > AFAICT they're all aimed at minimizing arbitrary functions, and as > such I'd expect them to be far slower than an actual linear solver. Is > there such a constrained linear solver in scipy (or numpy, or > scikits.*, etc.)? > > Even better would be a constrained matrix factorization routine, i.e. > that solves AX = B for X with A, X and B all being matrices, subject > to 0 <= X <= 1, but obviously you can construct the latter from the > former, so the former would suffice. > > Thanks, > Dan Lepage > Openopt has a BVLS wrapper, which is bounded values linear square solver. You also could wrap bvls.f with fwrap. Is there any intrest to include bvls in scipy.optimize? I remember to saw a constrained matrix facorization routine for python, but i forget where :(. Till Stensitzki From robert.kern at gmail.com Thu Apr 21 10:31:31 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 21 Apr 2011 09:31:31 -0500 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> <4DAFE5C1.2090705@creativetrax.com> Message-ID: On Thu, Apr 21, 2011 at 08:46, wrote: > On Thu, Apr 21, 2011 at 8:37 AM, Pauli Virtanen wrote: >> Thu, 21 Apr 2011 03:07:29 -0500, Jason Grout wrote: >> [clip] > >> As you can surmise from the discussion you linked to, there is resistance >> in adding new community-oriented features to PyPi itself, as some people >> feel that such features are out-of-scope for it. Doing it externally also >> makes sense from the usability and branding point of view --- a site >> called "Python in science" with filtered package selection can be more >> convincing and convenient to navigate than browsing "Topic :: Scientific/ >> Engineering" on PyPi. >> >> The discussion on rating systems there is an useful read --- it's why I >> left out any star-based rating systems so far. Just adding "I use this" >> popularity measure probably works around most issues. (The PyPi download >> data is not a very reliable measure, as many of the bigger packages host >> their files externally.) > > Sorry to add a comment here. > > The thread on PyPi rating system was dominated by a few developers of > big or famous packages. Ratings for Django or numpy or scipy might not > be useful, but ratings (with required comments) are very useful for > the huge number of smaller packages and will be for "snippets". > > matlab fileexchange is a good example for this. Actually, there were a number of us who objected to the ratings as *users* of PyPI. I find them abhorrent and worse than useless. An "I use this" button works well enough, I think. Comments can be helpful if done properly (and doing it properly is a *lot* more involved than most people think). Provide comparison grids like those on Django Packages to help people compare features of different packages. Make it trivial to see the code, and most people will be able to come to their own, much better judgements. Ratings and comments are the best you can do for catalogs of items that you need to pay for to examine (e.g. Amazon), but for catalogs of open source software, you can do a lot better. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From jsseabold at gmail.com Thu Apr 21 10:33:59 2011 From: jsseabold at gmail.com (Skipper Seabold) Date: Thu, 21 Apr 2011 10:33:59 -0400 Subject: [SciPy-User] Bounded Linear Least-Squares In-Reply-To: References: Message-ID: On Thu, Apr 21, 2011 at 10:20 AM, Till Stensitzki wrote: > Is there any intrest to include bvls in scipy.optimize? I think this would be a useful addition. Skipper From william.ratcliff at gmail.com Thu Apr 21 10:37:20 2011 From: william.ratcliff at gmail.com (william ratcliff) Date: Thu, 21 Apr 2011 10:37:20 -0400 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> <4DAFE5C1.2090705@creativetrax.com> Message-ID: For code snippets, I think comments are useful--ala stack overflow. William On Thu, Apr 21, 2011 at 10:31 AM, Robert Kern wrote: > On Thu, Apr 21, 2011 at 08:46, wrote: > > On Thu, Apr 21, 2011 at 8:37 AM, Pauli Virtanen wrote: > >> Thu, 21 Apr 2011 03:07:29 -0500, Jason Grout wrote: > >> [clip] > > > >> As you can surmise from the discussion you linked to, there is > resistance > >> in adding new community-oriented features to PyPi itself, as some people > >> feel that such features are out-of-scope for it. Doing it externally > also > >> makes sense from the usability and branding point of view --- a site > >> called "Python in science" with filtered package selection can be more > >> convincing and convenient to navigate than browsing "Topic :: > Scientific/ > >> Engineering" on PyPi. > >> > >> The discussion on rating systems there is an useful read --- it's why I > >> left out any star-based rating systems so far. Just adding "I use this" > >> popularity measure probably works around most issues. (The PyPi download > >> data is not a very reliable measure, as many of the bigger packages host > >> their files externally.) > > > > Sorry to add a comment here. > > > > The thread on PyPi rating system was dominated by a few developers of > > big or famous packages. Ratings for Django or numpy or scipy might not > > be useful, but ratings (with required comments) are very useful for > > the huge number of smaller packages and will be for "snippets". > > > > matlab fileexchange is a good example for this. > > Actually, there were a number of us who objected to the ratings as > *users* of PyPI. I find them abhorrent and worse than useless. An "I > use this" button works well enough, I think. Comments can be helpful > if done properly (and doing it properly is a *lot* more involved than > most people think). Provide comparison grids like those on Django > Packages to help people compare features of different packages. Make > it trivial to see the code, and most people will be able to come to > their own, much better judgements. Ratings and comments are the best > you can do for catalogs of items that you need to pay for to examine > (e.g. Amazon), but for catalogs of open source software, you can do a > lot better. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_baddeley at yahoo.com.au Thu Apr 21 14:08:48 2011 From: david_baddeley at yahoo.com.au (David Baddeley) Date: Thu, 21 Apr 2011 11:08:48 -0700 (PDT) Subject: [SciPy-User] Finding islands and centroids In-Reply-To: References: <201104200852.44275.luis@luispedro.org> Message-ID: <818972.54477.qm@web113406.mail.gq1.yahoo.com> hmm, seems no one has suggested using scipy.ndimage.measure yet. When combined with ndimage.label, this should give you all you want without needing to dive in to any external image processing packages. That said ndimage.measure is not particularly fast & you can get a bit of a speed up if you calculate the centroids manually using pre-allocated coordinate arrays. If you want good accuracy you should think about subtracting the threshold value from the data before taking the centroid*. I do this 'bead' (or similar) localisation problem regularly and probably have some example python code I could dig up. *The centroid is nominally the maximum likelihood estimator of position if your noise model is Gaussian, but if you're only doing in in a small region (the labelled beads), you will introduce some quantisation bias. Subtracting the threshold value is a way of minimising this bias. cheers, David ----- Original Message ---- From: Martin van Leeuwen To: SciPy Users List Sent: Thu, 21 April, 2011 8:08:35 AM Subject: Re: [SciPy-User] Finding islands and centroids Hi Chris, You can also compute this from a stack of incrementally thresholded images. If a pixel is a local maximum it would have no blobs above it and be contained in a blob below. This is described in: Ying, H. , Zhou, F. , Shields, A. F. , Muzik, O. , Wu, D. and Heath, E. I. (2004) A novel computerized approach to enhancing lung tumor detection in whole-body PET images. Proceedings of the 26th Annual International Conference of the IEEE EMBS San Francisco, CA IEEE EMBS Electronic Resource Piscataway, NJ and I also used it (and had written it in IDL at the time): Van Leeuwen, M., Coops, N.C., Wulder, M.A. (2010) Canopy Surface Reconstruction from a LiDAR point cloud Using Hough Transform. Remote Sensing Letters 1: 125 ? 132, doi: 10.1080/01431161003649339 Hope that is useful enough, unfortunately I don't have any python code for this so that I could share. Martin. 2011/4/20 Chris Weisiger : > Wow, for some reason I didn't get any of these emails until just now. You > all have given me plenty of avenues to investigate. Thanks! > > -Chris > > On Wed, Apr 20, 2011 at 5:52 AM, Luis Pedro Coelho > wrote: >> >> On Monday, April 18, 2011 06:37:51 pm Chris Weisiger wrote: >> > I can hack something together to do this easily enough, where I find a >> > pixel in one of the islands, flood-fill out to get all connected pixels, >> > calculate the centroid, flip the pixels to 0, and repeat until all >> > islands >> > are gone. This isn't exactly very speedy though. What's the efficient >> > way >> > to do this? Is there one? Is there a better approach I should be taking? >> > The image processing class I dimly remember taking years ago didn't >> > cover >> > this kind of thing, so I'm lacking even the basic vocabulary needed to >> > search for algorithms. >> >> You can probably do it in one go: >> >> 1) convolve with a Gaussian of roughly the size you expect the PFS of the >> beads to be. >> >> 2) threshold to get the beads. You can probably use a method like >> mahotas.threshold.otsu to do this automatically >> >> 3) scipy.ndimage.label to label the binary image >> >> 4, optional) remove noise by removing detections that are too small >> >> 5) find centroids of the remaining areas. >> >> HTH >> Luis >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ SciPy-User mailing list SciPy-User at scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user From warren.weckesser at enthought.com Thu Apr 21 23:14:27 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Thu, 21 Apr 2011 22:14:27 -0500 Subject: [SciPy-User] genfromtxt but I need to search file first In-Reply-To: References: Message-ID: On Thu, Apr 14, 2011 at 5:00 PM, Warren Weckesser < warren.weckesser at enthought.com> wrote: > > > On Tue, Apr 12, 2011 at 10:54 PM, Ben Harrison < > ben.harrison at liquidmesh.com> wrote: > >> On 13/04/11 00:25, Warren Weckesser wrote: >> > By any changes is this a LAS file? Take a look here: >> > http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055563.html >> > and at my response. I included a "quick and dirty" implementation of a >> > LAS reader. (I have a nicer version in progress, but it will be a few >> > days before I can finish it.) In the version attached to that email, I >> > just use loadtxt() to read the data after the ~A line. >> > >> > Warren >> > >> >> Yes it is a LAS file, they are pesky things. >> I'll try your script, or steal methods from it! >> >> Will you be publishing your nicer script anywhere in particular? Could >> you let me know somehow when you have? >> > > > The "Input Output" section of the SciPy Cookbook ( > http://www.scipy.org/Cookbook) looks like as good a place as any. I'll > create a page for it there once I've clean up a few things. > > A LAS reader is now here: http://www.scipy.org/Cookbook/LASReader Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From lilyphysik at gmail.com Fri Apr 22 02:36:37 2011 From: lilyphysik at gmail.com (lilyphysik) Date: Fri, 22 Apr 2011 14:36:37 +0800 Subject: [SciPy-User] undefined symbol: dswap_ Message-ID: <4db121fa.a70f440a.40c1.ffffba84@mx.google.com> Hi everyone, I followed the instruction on the website and installed scipy from the source code. All the components are compiled and installed sucessfully by myself. But when I imported a library which is wrapped by f2py, I found an error: undefined symbol: dswap_ . I have added the compiler flags -fPIC and -m64. But it seems not working. I send the code here and hope anyone can help me. The OS and the software I used are: Red Hat Enterprise Linux Server release 5.4 (Tikanga) gcc version 4.1.2 20080704 (Red Hat 4.1.2-46) intel fortran compiler 11.1 Version 11.1 Lapack 3.3.0 Python 2.7.1 Numpy 1.5.1 scipy 0.9 Code: fortran90 part: MODULE spin CONTAINS subroutine sp(ax,bx,n) implicit none integer::n real*8::ax(1:n),bx(1:n) ax=1 bx=2 call dswap(n,ax,1,bx,1) write(*,*) ax end subroutine END MODULE spin python part: #!/duan/bin/python from numpy import * from scipy import * from edlib import * print spin.__doc__ I use the following commands to create the library named edlib: p2.7/bin/f2py test.f90 -m edlib -h test.pyf p2.7/bin/f2py -c test.f90 -m edlib --fcompiler=intelem -L/duan/p2.7/lib -llapack The error ocurrs when importing the edlib. Any suggestions? Thanks in advance. Chengbo duan 2011-04-22 lilyphysik -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Apr 22 08:47:33 2011 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 22 Apr 2011 12:47:33 +0000 (UTC) Subject: [SciPy-User] undefined symbol: dswap_ References: <4db121fa.a70f440a.40c1.ffffba84@mx.google.com> Message-ID: On Fri, 22 Apr 2011 14:36:37 +0800, lilyphysik wrote: [clip] > p2.7/bin/f2py test.f90 -m edlib -h test.pyf p2.7/bin/f2py -c test.f90 -m > edlib --fcompiler=intelem -L/duan/p2.7/lib -llapack > > The error ocurrs when importing the edlib. Any suggestions? Add -lblas after -llapack --- dswap is in it. From tmaiwald at web.de Thu Apr 21 14:08:44 2011 From: tmaiwald at web.de (Thomas Maiwald) Date: Thu, 21 Apr 2011 11:08:44 -0700 (PDT) Subject: [SciPy-User] [SciPy-user] Fitting a system of ODEs to data In-Reply-To: References: Message-ID: <31451555.post@talk.nabble.com> If you have access to Matlab, you could try the PottersWheel ODE modeling and fitting toolbox available at www.potterswheel.de. Greetings, Thomas -- View this message in context: http://old.nabble.com/Fitting-a-system-of-ODEs-to-data-tp25384199p31451555.html Sent from the Scipy-User mailing list archive at Nabble.com. From josef.pktd at gmail.com Fri Apr 22 17:22:43 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 22 Apr 2011 17:22:43 -0400 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> <4DAFE5C1.2090705@creativetrax.com> Message-ID: On Thu, Apr 21, 2011 at 10:37 AM, william ratcliff wrote: > For code snippets, I think comments are useful--ala stack overflow. > > William > > On Thu, Apr 21, 2011 at 10:31 AM, Robert Kern wrote: >> >> On Thu, Apr 21, 2011 at 08:46, ? wrote: >> > On Thu, Apr 21, 2011 at 8:37 AM, Pauli Virtanen wrote: >> >> Thu, 21 Apr 2011 03:07:29 -0500, Jason Grout wrote: >> >> [clip] >> > >> >> As you can surmise from the discussion you linked to, there is >> >> resistance >> >> in adding new community-oriented features to PyPi itself, as some >> >> people >> >> feel that such features are out-of-scope for it. Doing it externally >> >> also >> >> makes sense from the usability and branding point of view --- a site >> >> called "Python in science" with filtered package selection can be more >> >> convincing and convenient to navigate than browsing "Topic :: >> >> Scientific/ >> >> Engineering" on PyPi. >> >> >> >> The discussion on rating systems there is an useful read --- it's why I >> >> left out any star-based rating systems so far. Just adding "I use this" >> >> popularity measure probably works around most issues. (The PyPi >> >> download >> >> data is not a very reliable measure, as many of the bigger packages >> >> host >> >> their files externally.) >> > >> > Sorry to add a comment here. >> > >> > The thread on PyPi rating system was dominated by a few developers of >> > big or famous packages. Ratings for Django or numpy or scipy might not >> > be useful, but ratings (with required comments) are very useful for >> > the huge number of smaller packages and will be for "snippets". >> > >> > matlab fileexchange is a good example for this. >> >> Actually, there were a number of us who objected to the ratings as >> *users* of PyPI. I find them abhorrent and worse than useless. An "I >> use this" button works well enough, I think. Comments can be helpful >> if done properly (and doing it properly is a *lot* more involved than >> most people think). Provide comparison grids like those on Django >> Packages to help people compare features of different packages. Make >> it trivial to see the code, and most people will be able to come to >> their own, much better judgements. Ratings and comments are the best >> you can do for catalogs of items that you need to pay for to examine >> (e.g. Amazon), but for catalogs of open source software, you can do a >> lot better. after a facelift http://www.mathworks.com/matlabcentral/fileexchange/?sort=date_desc_updated&term= Josef >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma that is made terrible by our own mad attempt to interpret it as >> though it had an underlying truth." >> ? -- Umberto Eco >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From sturla at molden.no Fri Apr 22 17:40:55 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 22 Apr 2011 23:40:55 +0200 Subject: [SciPy-User] Central File Exchange for Scipy In-Reply-To: References: <4DAD0411.6010405@creativetrax.com> <4DADCADA.1010905@creativetrax.com> <4DADD381.60206@creativetrax.com> <4DADF03C.3080804@creativetrax.com> <4DAE15F8.4090007@creativetrax.com> <4DAFE5C1.2090705@creativetrax.com> Message-ID: <4DB1F5E7.2050502@molden.no> Den 22.04.2011 23:22, skrev josef.pktd at gmail.com: > > after a facelift > > http://www.mathworks.com/matlabcentral/fileexchange/?sort=date_desc_updated&term= > > Josef > This is indeed something I miss for scientific Python as well. I also miss a similar file exchange for Cython (not limited to scientific computing). Here is another site for comparison (yes I know about the SciPy cookbook): http://code.activestate.com/recipes/ Sturla From rnowling at gmail.com Fri Apr 22 18:09:19 2011 From: rnowling at gmail.com (RJ Nowling) Date: Fri, 22 Apr 2011 18:09:19 -0400 Subject: [SciPy-User] Why doesn't dot product transpose? Message-ID: Hi all, This is my first time posting to the scipy user mailing list. I found it rather frustrating to get the dot product working for (sparse) matrices. In particular, when trying to use dot product with two matrices of nx1, it doesn't perform the transpose of the first matrix. Thus, the dot product is really just an overload of the multiplication: >>> from numpy import ones >>> from scipy.sparse import csc_matrix >>> m = csc_matrix(ones((5, 4))) >>> m <5x4 sparse matrix of type '' with 20 stored elements in Compressed Sparse Column format> >>> m.getcol(1) <5x1 sparse matrix of type '' with 5 stored elements in Compressed Sparse Column format> Attempt to perform dot product: >>> m.getcol(1).dot(m.getcol(1)) ... ValueError: dimension mismatch Works when transposing the first matrix: >>> m.getcol(1).transpose().dot(m.getcol(1)) <1x1 sparse matrix of type '' with 1 stored elements in Compressed Sparse Row format> Dot product same as matrix multiplication? Shouldn't it transpose the first matrix for you? >>> m.getcol(1).transpose() * m.getcol(1) <1x1 sparse matrix of type '' with 1 stored elements in Compressed Sparse Row format> I understand that this is the same convention followed in Numpy, but it makes no sense to me. I thought that u dot v = uT * v ? Why even use the dot product if I still have to do the transpose? Thank you, RJ -- em rnowling at gmail.com c 954.496.2314 From jrennie at gmail.com Fri Apr 22 18:17:50 2011 From: jrennie at gmail.com (Jason Rennie) Date: Fri, 22 Apr 2011 18:17:50 -0400 Subject: [SciPy-User] Why doesn't dot product transpose? In-Reply-To: References: Message-ID: I think the documentation is pretty clear: For 2-D arrays it is equivalent to matrix multiplication, and for 1-D arrays to inner product of vectors (without complex conjugation). For N dimensions it is a sum product over the last axis of a and the second-to-last of b: dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m]) http://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html The example makes it clear that if a.shape == b.shape == (n,1) then dot(a,b) doesn't work. However, if a.shape == b.shape == (n,), I believe it does "the right thing" :-) Should you be using 1-d arrays instead of 2-d arrays? Cheers, Jason On Fri, Apr 22, 2011 at 6:09 PM, RJ Nowling wrote: > Hi all, > > This is my first time posting to the scipy user mailing list. I found > it rather frustrating to get the dot product working for (sparse) > matrices. In particular, when trying to use dot product with two > matrices of nx1, it doesn't perform the transpose of the first matrix. > Thus, the dot product is really just an overload of the > multiplication: > > >>> from numpy import ones > >>> from scipy.sparse import csc_matrix > >>> m = csc_matrix(ones((5, 4))) > >>> m > <5x4 sparse matrix of type '' > with 20 stored elements in Compressed Sparse Column format> > >>> m.getcol(1) > <5x1 sparse matrix of type '' > with 5 stored elements in Compressed Sparse Column format> > > > Attempt to perform dot product: > >>> m.getcol(1).dot(m.getcol(1)) > ... > ValueError: dimension mismatch > > Works when transposing the first matrix: > >>> m.getcol(1).transpose().dot(m.getcol(1)) > <1x1 sparse matrix of type '' > with 1 stored elements in Compressed Sparse Row format> > > Dot product same as matrix multiplication? Shouldn't it transpose the > first matrix for you? > >>> m.getcol(1).transpose() * m.getcol(1) > <1x1 sparse matrix of type '' > with 1 stored elements in Compressed Sparse Row format> > > I understand that this is the same convention followed in Numpy, but > it makes no sense to me. I thought that u dot v = uT * v ? Why even > use the dot product if I still have to do the transpose? > > Thank you, > RJ > > -- > em rnowling at gmail.com > c 954.496.2314 > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Fri Apr 22 18:40:30 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 22 Apr 2011 18:40:30 -0400 Subject: [SciPy-User] Why doesn't dot product transpose? In-Reply-To: References: Message-ID: On Fri, Apr 22, 2011 at 6:17 PM, Jason Rennie wrote: > I think the documentation is pretty clear: > > For 2-D arrays it is equivalent to matrix multiplication, and for 1-D arrays > to inner product of vectors (without complex conjugation). For N dimensions > it is a sum product over the last axis of a and the second-to-last of b: > dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m]) > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html > The example makes it clear that if a.shape == b.shape == (n,1) then dot(a,b) > doesn't work. ?However, if?a.shape == b.shape == (n,), I believe it does > "the right thing" :-) ?Should you be using 1-d arrays instead of 2-d arrays? > > Cheers, > Jason > > On Fri, Apr 22, 2011 at 6:09 PM, RJ Nowling wrote: >> >> Hi all, >> >> This is my first time posting to the scipy user mailing list. ?I found >> it rather frustrating to get the dot product working for (sparse) >> matrices. ?In particular, when trying to use dot product with two >> matrices of nx1, it doesn't perform the transpose of the first matrix. >> ?Thus, the dot product is really just an overload of the >> multiplication: >> >> >>> from numpy import ones >> >>> from scipy.sparse import csc_matrix >> >>> m = csc_matrix(ones((5, 4))) >> >>> m >> <5x4 sparse matrix of type '' >> ? ? ? ?with 20 stored elements in Compressed Sparse Column format> >> >>> m.getcol(1) >> <5x1 sparse matrix of type '' >> ? ? ? ?with 5 stored elements in Compressed Sparse Column format> >> >> >> Attempt to perform dot product: >> >>> m.getcol(1).dot(m.getcol(1)) >> ... >> ValueError: dimension mismatch >> >> Works when transposing the first matrix: >> >>> m.getcol(1).transpose().dot(m.getcol(1)) >> <1x1 sparse matrix of type '' >> ? ? ? ?with 1 stored elements in Compressed Sparse Row format> >> >> Dot product same as matrix multiplication? ?Shouldn't it transpose the >> first matrix for you? >> >>> m.getcol(1).transpose() * m.getcol(1) >> <1x1 sparse matrix of type '' >> ? ? ? ?with 1 stored elements in Compressed Sparse Row format> >> >> I understand that this is the same convention followed in Numpy, but >> it makes no sense to me. ?I thought that u dot v = uT * v ? ?Why even >> use the dot product if I still have to do the transpose? When we use 2d arrays or matrices, then I hope the rules for matrix multiplication applies. I rather get an error message than numpy or scipy guessing whether I actually meant an inner or outer product (u.T dot v) or (u dot v.T) ? if u and v are either both (n,1) or both (1,n) Josef >> >> Thank you, >> RJ >> >> -- >> em rnowling at gmail.com >> c 954.496.2314 >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > > -- > Jason Rennie > Research Scientist, ITA Software > 617-714-2645 > http://www.itasoftware.com/ > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From rnowling at gmail.com Sat Apr 23 13:19:18 2011 From: rnowling at gmail.com (RJ Nowling) Date: Sat, 23 Apr 2011 13:19:18 -0400 Subject: [SciPy-User] Why doesn't dot product transpose? In-Reply-To: References: Message-ID: Thank you for the response. As you pointed out, the simple answer is to use 1D arrays. We are using sparse matrices and want to perform dot products on the column vectors of those matrices. When I use csc_matrix.getcol() or matrix[:,1] syntax, a sparse 2D array is returned with shape (n, 1). It seems that my approach is getting the proper data structures for the dot product, as you pointed out. So, what's the correct way of getting a sparse shape (n,) column vector from a matrix? (E.g., we don't want to perform unnecessary operations by converting to dense representations.) Thank you, RJ > I think the documentation is pretty clear: > > For 2-D arrays it is equivalent to matrix multiplication, and for 1-D arrays > to inner product of vectors (without complex conjugation). For N dimensions > it is a sum product over the last axis of a and the second-to-last of b: > dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m]) > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html > > The example makes it clear that if a.shape == b.shape == (n,1) then dot(a,b) > doesn't work. ?However, if a.shape == b.shape == (n,), I believe it does > "the right thing" :-) ?Should you be using 1-d arrays instead of 2-d arrays? > > Cheers, > > Jason On Fri, Apr 22, 2011 at 6:09 PM, RJ Nowling wrote: > Hi all, > > This is my first time posting to the scipy user mailing list. ?I found > it rather frustrating to get the dot product working for (sparse) > matrices. ?In particular, when trying to use dot product with two > matrices of nx1, it doesn't perform the transpose of the first matrix. > ?Thus, the dot product is really just an overload of the > multiplication: > >>>> from numpy import ones >>>> from scipy.sparse import csc_matrix >>>> m = csc_matrix(ones((5, 4))) >>>> m > <5x4 sparse matrix of type '' > ? ? ? ?with 20 stored elements in Compressed Sparse Column format> >>>> m.getcol(1) > <5x1 sparse matrix of type '' > ? ? ? ?with 5 stored elements in Compressed Sparse Column format> > > > Attempt to perform dot product: >>>> m.getcol(1).dot(m.getcol(1)) > ... > ValueError: dimension mismatch > > Works when transposing the first matrix: >>>> m.getcol(1).transpose().dot(m.getcol(1)) > <1x1 sparse matrix of type '' > ? ? ? ?with 1 stored elements in Compressed Sparse Row format> > > Dot product same as matrix multiplication? ?Shouldn't it transpose the > first matrix for you? >>>> m.getcol(1).transpose() * m.getcol(1) > <1x1 sparse matrix of type '' > ? ? ? ?with 1 stored elements in Compressed Sparse Row format> > > I understand that this is the same convention followed in Numpy, but > it makes no sense to me. ?I thought that u dot v = uT * v ? ?Why even > use the dot product if I still have to do the transpose? > > Thank you, > RJ > > -- > em rnowling at gmail.com > c 954.496.2314 > -- em rnowling at gmail.com c 954.496.2314 From jrennie at gmail.com Sat Apr 23 14:25:30 2011 From: jrennie at gmail.com (Jason Rennie) Date: Sat, 23 Apr 2011 14:25:30 -0400 Subject: [SciPy-User] Why doesn't dot product transpose? In-Reply-To: References: Message-ID: On Sat, Apr 23, 2011 at 1:19 PM, RJ Nowling wrote: > As you pointed out, the simple answer is to use 1D arrays. We are > using sparse matrices and want to perform dot products on the column > vectors of those matrices. When I use csc_matrix.getcol() or > matrix[:,1] syntax, a sparse 2D array is returned with shape (n, 1). > It seems that my approach is getting the proper data structures for > the dot product, as you pointed out. > The problem is that sparse matrices are just that, *matrices*. AFAIK, there is no general sparse array structure. So, you may need to switch from sparse to dense if you want shape=(n,) arrays. You can easily change (dense) shapes with numpy.reshape(). > So, what's the correct way of getting a sparse shape (n,) column > vector from a matrix? (E.g., we don't want to perform unnecessary > operations by converting to dense representations.) > You can't AFAIK. What are you trying to do? Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From angvp at archlinux.org Sat Apr 23 19:27:22 2011 From: angvp at archlinux.org (=?ISO-8859-1?Q?=C1ngel_Vel=E1squez?=) Date: Sat, 23 Apr 2011 20:27:22 -0300 Subject: [SciPy-User] Unable to build scipy with python3.2 Message-ID: Hi, Let me make a short introduction, my name is Angel Vel?squez from Argentina, I am the maintainer of scipy on Arch Linux distribution. I am having this issue [1] I've tried several options (creating a site.cfg with [blas]) export the BLAS variable, etc, but I unfortunately can make it. I'm following these steps: $ python setup.py config_fc --fcompiler=gnu95 build The breakage comes in the first step, so I cannot do the following step: # python setup.py config_fc --fcompiler=gnu95 install --prefix=/usr --optimize=1 I was able to package scipy with python 2.7 in the same enviroment but not with python 3.2 Any help is appreciated, thanks in advance [1] http://dpaste.com/535077/ -- Angel Vel?squez angvp @ irc.freenode.net Arch Linux Developer / Trusted User Linux Counter: #359909 http://www.angvp.com From paulojamorim at gmail.com Sun Apr 24 22:22:33 2011 From: paulojamorim at gmail.com (Paulo Henrique Junqueira Amorim) Date: Sun, 24 Apr 2011 23:22:33 -0300 Subject: [SciPy-User] Image with black points Message-ID: Hi, I'm computing the 1D fft from image, and saving the image of frequency. But the saved image appears black point's (example image), it seems that do not belong. Is this correct? How could I remove it? Example image - http://img818.imageshack.us/i/fftthreshold.png/ Code: import scipy import numpy r = scipy.misc.pilutil.imread('face.jpg', 'F') fft = numpy.fft.fftshift(numpy.fft.fft(r).real) fft_2d = numpy.log10(numpy.abs(fft)) pil_array = scipy.misc.pilutil.toimage(fft_2d) scipy.misc.pilutil.imsave('fft.tif', pil_array) Regards, Paulo -------------- next part -------------- An HTML attachment was scrubbed... URL: From lasagnadavide at gmail.com Sun Apr 24 10:50:55 2011 From: lasagnadavide at gmail.com (Davide Lasagna) Date: Sun, 24 Apr 2011 16:50:55 +0200 Subject: [SciPy-User] [ANN]: OpenPiv. A python package for PIV image analysis Message-ID: <4DB438CF.3090602@gmail.com> Hi all, I am pleased to announce the re-birth of OpenPiv, a python package for the analysis of PIV images. PIV is a well established optical measurement technique for fluid flows. The package has the goal of provide an implementation of state-of-the-art processing algorithms, and compete with commercial closed-source software packages. OpenPiv is in alpha state and several thing has to be done. However, the code is already usable and a pure python implementation of the standard cross-correlation algorithm is already available. OpenPiv needs developers and contributors, helping in with code and documentation. If you can provide some, and if you are interested in becoming a core developer, please drop us an email at openpiv-develop at lists dot sourceforge dot net. A draft website can be found at www.openpiv.net/python. Thanks for your attention, Davide Lasagna From johnl at cs.wisc.edu Mon Apr 25 11:31:33 2011 From: johnl at cs.wisc.edu (J. David Lee) Date: Mon, 25 Apr 2011 10:31:33 -0500 Subject: [SciPy-User] Moving median code Message-ID: <4DB593D5.4010506@cs.wisc.edu> In working on a project recently, I wrote a moving median code that is about 10x faster than scipy.ndimage.median_filter. It utilizes a linked list structure to store values and track the median value. If anyone is interested, I've posted the code here (about 150 lines): http://pages.cs.wisc.edu/~johnl/median_code/median_code.c Thanks, J. David Lee From kwgoodman at gmail.com Mon Apr 25 12:01:13 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 25 Apr 2011 09:01:13 -0700 Subject: [SciPy-User] Moving median code In-Reply-To: <4DB593D5.4010506@cs.wisc.edu> References: <4DB593D5.4010506@cs.wisc.edu> Message-ID: On Mon, Apr 25, 2011 at 8:31 AM, J. David Lee wrote: > In working on a project recently, I wrote a moving median code that is > about 10x faster than scipy.ndimage.median_filter. It utilizes a linked > list structure to store values and track the median value. If anyone is > interested, I've posted the code here (about 150 lines): > > http://pages.cs.wisc.edu/~johnl/median_code/median_code.c I'm looking to add a moving window median function to the bottleneck package [1]. So your code takes 1d input where the window size is the length of the input and then you move a step forward by dropping the oldest data point and adding a new one and then update the median? Just making sure I understand. If that's a good fit for bottleneck I could add the ability to take nd input, unit tests, and docstrings---assuming I could figure out how to wrap it in Cython. Does anyone have a feel for how the speed of a linked list approach would compare to a double heap? Code for the double heap moving window median is in [2]. I also wonder which would be easier to wrap in Cython. [1] https://github.com/kwgoodman/bottleneck [2] http://home.tiac.net/~cri_a/source_code/winmedian From johnl at cs.wisc.edu Mon Apr 25 12:25:54 2011 From: johnl at cs.wisc.edu (J. David Lee) Date: Mon, 25 Apr 2011 11:25:54 -0500 Subject: [SciPy-User] Moving median code In-Reply-To: References: <4DB593D5.4010506@cs.wisc.edu> Message-ID: <4DB5A092.2000201@cs.wisc.edu> On 04/25/2011 11:01 AM, Keith Goodman wrote: > On Mon, Apr 25, 2011 at 8:31 AM, J. David Lee wrote: >> In working on a project recently, I wrote a moving median code that is >> about 10x faster than scipy.ndimage.median_filter. It utilizes a linked >> list structure to store values and track the median value. If anyone is >> interested, I've posted the code here (about 150 lines): >> >> http://pages.cs.wisc.edu/~johnl/median_code/median_code.c > I'm looking to add a moving window median function to the bottleneck > package [1]. > > So your code takes 1d input where the window size is the length of the > input and then you move a step forward by dropping the oldest data > point and adding a new one and then update the median? Just making > sure I understand. > > If that's a good fit for bottleneck I could add the ability to take nd > input, unit tests, and docstrings---assuming I could figure out how to > wrap it in Cython. > > Does anyone have a feel for how the speed of a linked list approach > would compare to a double heap? Code for the double heap moving window > median is in [2]. I also wonder which would be easier to wrap in > Cython. > > [1] https://github.com/kwgoodman/bottleneck > [2] http://home.tiac.net/~cri_a/source_code/winmedian > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Your understanding is correct, although I'm reusing the oldest node: moving it to the beginning of the list (by index), updating its value, and then moving it to the appropriate place in the value list using its new value. Any time two nodes are swapped I'm checking if one is the median value, and then swapping the median pointer if it is. Thanks for your interest, David From wesmckinn at gmail.com Mon Apr 25 12:27:48 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 25 Apr 2011 12:27:48 -0400 Subject: [SciPy-User] Moving median code In-Reply-To: <4DB5A092.2000201@cs.wisc.edu> References: <4DB593D5.4010506@cs.wisc.edu> <4DB5A092.2000201@cs.wisc.edu> Message-ID: On Mon, Apr 25, 2011 at 12:25 PM, J. David Lee wrote: > On 04/25/2011 11:01 AM, Keith Goodman wrote: >> On Mon, Apr 25, 2011 at 8:31 AM, J. David Lee ?wrote: >>> In working on a project recently, I wrote a moving median code that is >>> about 10x faster than scipy.ndimage.median_filter. It utilizes a linked >>> list structure to store values and track the median value. If anyone is >>> interested, I've posted the code here (about 150 lines): >>> >>> http://pages.cs.wisc.edu/~johnl/median_code/median_code.c >> I'm looking to add a moving window median function to the bottleneck >> package [1]. >> >> So your code takes 1d input where the window size is the length of the >> input and then you move a step forward by dropping the oldest data >> point and adding a new one and then update the median? Just making >> sure I understand. >> >> If that's a good fit for bottleneck I could add the ability to take nd >> input, unit tests, and docstrings---assuming I could figure out how to >> wrap it in Cython. >> >> Does anyone have a feel for how the speed of a linked list approach >> would compare to a double heap? Code for the double heap moving window >> median is in [2]. I also wonder which would be easier to wrap in >> Cython. >> >> [1] https://github.com/kwgoodman/bottleneck >> [2] http://home.tiac.net/~cri_a/source_code/winmedian >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > Your understanding is correct, although I'm reusing the oldest node: > moving it to the beginning of the list (by index), updating its value, > and then moving it to the appropriate place in the value list using its > new value. Any time two nodes are swapped I'm checking if one is the > median value, and then swapping the median pointer if it is. > > Thanks for your interest, > > David > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > I wonder how this compares speedwise with the indexable skiplist approach: http://rhettinger.wordpress.com/2010/02/06/lost-knowledge/ I converted to Cython and put in pandas here last year: https://github.com/wesm/pandas/blob/master/pandas/lib/src/skiplist.pyx https://github.com/wesm/pandas/blob/master/pandas/lib/src/moments.pyx#L442 This code definitely has "too much Python", I think, so a pure C version would be much faster. From wesmckinn at gmail.com Mon Apr 25 12:41:37 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 25 Apr 2011 12:41:37 -0400 Subject: [SciPy-User] Moving median code In-Reply-To: References: <4DB593D5.4010506@cs.wisc.edu> <4DB5A092.2000201@cs.wisc.edu> Message-ID: On Mon, Apr 25, 2011 at 12:27 PM, Wes McKinney wrote: > On Mon, Apr 25, 2011 at 12:25 PM, J. David Lee wrote: >> On 04/25/2011 11:01 AM, Keith Goodman wrote: >>> On Mon, Apr 25, 2011 at 8:31 AM, J. David Lee ?wrote: >>>> In working on a project recently, I wrote a moving median code that is >>>> about 10x faster than scipy.ndimage.median_filter. It utilizes a linked >>>> list structure to store values and track the median value. If anyone is >>>> interested, I've posted the code here (about 150 lines): >>>> >>>> http://pages.cs.wisc.edu/~johnl/median_code/median_code.c >>> I'm looking to add a moving window median function to the bottleneck >>> package [1]. >>> >>> So your code takes 1d input where the window size is the length of the >>> input and then you move a step forward by dropping the oldest data >>> point and adding a new one and then update the median? Just making >>> sure I understand. >>> >>> If that's a good fit for bottleneck I could add the ability to take nd >>> input, unit tests, and docstrings---assuming I could figure out how to >>> wrap it in Cython. >>> >>> Does anyone have a feel for how the speed of a linked list approach >>> would compare to a double heap? Code for the double heap moving window >>> median is in [2]. I also wonder which would be easier to wrap in >>> Cython. >>> >>> [1] https://github.com/kwgoodman/bottleneck >>> [2] http://home.tiac.net/~cri_a/source_code/winmedian >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >> Your understanding is correct, although I'm reusing the oldest node: >> moving it to the beginning of the list (by index), updating its value, >> and then moving it to the appropriate place in the value list using its >> new value. Any time two nodes are swapped I'm checking if one is the >> median value, and then swapping the median pointer if it is. >> >> Thanks for your interest, >> >> David >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > I wonder how this compares speedwise with the indexable skiplist approach: > > http://rhettinger.wordpress.com/2010/02/06/lost-knowledge/ > > I converted to Cython and put in pandas here last year: > > https://github.com/wesm/pandas/blob/master/pandas/lib/src/skiplist.pyx > https://github.com/wesm/pandas/blob/master/pandas/lib/src/moments.pyx#L442 > > This code definitely has "too much Python", I think, so a pure C > version would be much faster. > I took a peak at the C code-- looks like it's O(N * W). Maybe we should invest in a pure C O(N log W) approach using the skiplist? From kwgoodman at gmail.com Mon Apr 25 13:03:26 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 25 Apr 2011 10:03:26 -0700 Subject: [SciPy-User] Moving median code In-Reply-To: References: <4DB593D5.4010506@cs.wisc.edu> <4DB5A092.2000201@cs.wisc.edu> Message-ID: On Mon, Apr 25, 2011 at 9:41 AM, Wes McKinney wrote: > On Mon, Apr 25, 2011 at 12:27 PM, Wes McKinney wrote: >> On Mon, Apr 25, 2011 at 12:25 PM, J. David Lee wrote: >>> On 04/25/2011 11:01 AM, Keith Goodman wrote: >>>> On Mon, Apr 25, 2011 at 8:31 AM, J. David Lee ?wrote: >>>>> In working on a project recently, I wrote a moving median code that is >>>>> about 10x faster than scipy.ndimage.median_filter. It utilizes a linked >>>>> list structure to store values and track the median value. If anyone is >>>>> interested, I've posted the code here (about 150 lines): >>>>> >>>>> http://pages.cs.wisc.edu/~johnl/median_code/median_code.c >>>> >>>> So your code takes 1d input where the window size is the length of the >>>> input and then you move a step forward by dropping the oldest data >>>> point and adding a new one and then update the median? Just making >>>> sure I understand. >>> >>> Your understanding is correct, although I'm reusing the oldest node: >>> moving it to the beginning of the list (by index), updating its value, >>> and then moving it to the appropriate place in the value list using its >>> new value. Any time two nodes are swapped I'm checking if one is the >>> median value, and then swapping the median pointer if it is. >> >> I wonder how this compares speedwise with the indexable skiplist approach: >> >> http://rhettinger.wordpress.com/2010/02/06/lost-knowledge/ >> >> I converted to Cython and put in pandas here last year: >> >> https://github.com/wesm/pandas/blob/master/pandas/lib/src/skiplist.pyx >> https://github.com/wesm/pandas/blob/master/pandas/lib/src/moments.pyx#L442 >> >> This code definitely has "too much Python", I think, so a pure C >> version would be much faster. > > I took a peak at the C code-- looks like it's O(N * W). Maybe we > should invest in a pure C O(N log W) approach using the skiplist? Seems like there is a fair amount of interest in a moving median of numpy arrays. Due to my ignorance of the relative merits of each algorithm, could we wrap the C code of all three versions in cython and see which is fastest? Two algorithms are already coded in C. Making a cython function that only works on 1d, float64 input might serve as a good test bed. I can set up a temp github repo for a moving window median prototype if anyone is interested in working on it together. I've never cython wrapped C code. If I had, I'd already have a moving window median. From wesmckinn at gmail.com Mon Apr 25 13:08:12 2011 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 25 Apr 2011 13:08:12 -0400 Subject: [SciPy-User] Moving median code In-Reply-To: References: <4DB593D5.4010506@cs.wisc.edu> <4DB5A092.2000201@cs.wisc.edu> Message-ID: On Mon, Apr 25, 2011 at 1:03 PM, Keith Goodman wrote: > On Mon, Apr 25, 2011 at 9:41 AM, Wes McKinney wrote: >> On Mon, Apr 25, 2011 at 12:27 PM, Wes McKinney wrote: >>> On Mon, Apr 25, 2011 at 12:25 PM, J. David Lee wrote: >>>> On 04/25/2011 11:01 AM, Keith Goodman wrote: >>>>> On Mon, Apr 25, 2011 at 8:31 AM, J. David Lee ?wrote: >>>>>> In working on a project recently, I wrote a moving median code that is >>>>>> about 10x faster than scipy.ndimage.median_filter. It utilizes a linked >>>>>> list structure to store values and track the median value. If anyone is >>>>>> interested, I've posted the code here (about 150 lines): >>>>>> >>>>>> http://pages.cs.wisc.edu/~johnl/median_code/median_code.c >>>>> >>>>> So your code takes 1d input where the window size is the length of the >>>>> input and then you move a step forward by dropping the oldest data >>>>> point and adding a new one and then update the median? Just making >>>>> sure I understand. >>>> >>>> Your understanding is correct, although I'm reusing the oldest node: >>>> moving it to the beginning of the list (by index), updating its value, >>>> and then moving it to the appropriate place in the value list using its >>>> new value. Any time two nodes are swapped I'm checking if one is the >>>> median value, and then swapping the median pointer if it is. >>> >>> I wonder how this compares speedwise with the indexable skiplist approach: >>> >>> http://rhettinger.wordpress.com/2010/02/06/lost-knowledge/ >>> >>> I converted to Cython and put in pandas here last year: >>> >>> https://github.com/wesm/pandas/blob/master/pandas/lib/src/skiplist.pyx >>> https://github.com/wesm/pandas/blob/master/pandas/lib/src/moments.pyx#L442 >>> >>> This code definitely has "too much Python", I think, so a pure C >>> version would be much faster. >> >> I took a peak at the C code-- looks like it's O(N * W). Maybe we >> should invest in a pure C O(N log W) approach using the skiplist? > > Seems like there is a fair amount of interest in a moving median of > numpy arrays. > > Due to my ignorance of the relative merits of each algorithm, could we > wrap the C code of all three versions in cython and see which is > fastest? Two algorithms are already coded in C. Making a cython > function that only works on 1d, float64 input might serve as a good > test bed. > > I can set up a temp github repo for a moving window median prototype > if anyone is interested in working on it together. Go for it...not sure when I'll have time to help but probably in a few weeks. > I've never cython wrapped C code. If I had, I'd already have a moving > window median. It's very simple actually-- Cython docs have everything you need to know. > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From zachary.pincus at yale.edu Mon Apr 25 13:15:50 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Mon, 25 Apr 2011 13:15:50 -0400 Subject: [SciPy-User] Image with black points In-Reply-To: References: Message-ID: On Apr 24, 2011, at 10:22 PM, Paulo Henrique Junqueira Amorim wrote: > Hi, > > I'm computing the 1D fft from image, and saving the image of > frequency. > > But the saved image appears black point's (example image), it seems > that do not belong. > > Is this correct? How could I remove it? > > Example image - http://img818.imageshack.us/i/fftthreshold.png/ > > Code: > > import scipy > import numpy > > r = scipy.misc.pilutil.imread('face.jpg', 'F') > > fft = numpy.fft.fftshift(numpy.fft.fft(r).real) > fft_2d = numpy.log10(numpy.abs(fft)) > pil_array = scipy.misc.pilutil.toimage(fft_2d) > > scipy.misc.pilutil.imsave('fft.tif', pil_array) > If you could provide the input image that would be helpful for people trying to determine what the issue is. As a stab in the dark -- have you checked (from python) what the values of at the "black points" are? Are they NaNs? (Perhaps from the log10 operation?) Zach From paulojamorim at gmail.com Mon Apr 25 19:32:26 2011 From: paulojamorim at gmail.com (Paulo Henrique Junqueira Amorim) Date: Mon, 25 Apr 2011 20:32:26 -0300 Subject: [SciPy-User] Image with black points In-Reply-To: References: Message-ID: On 25 April 2011 14:15, Zachary Pincus wrote: > On Apr 24, 2011, at 10:22 PM, Paulo Henrique Junqueira Amorim wrote: > > > Hi, > > > > I'm computing the 1D fft from image, and saving the image of > > frequency. > > > > But the saved image appears black point's (example image), it seems > > that do not belong. > > > > Is this correct? How could I remove it? > > > > Example image - http://img818.imageshack.us/i/fftthreshold.png/ > > > > Code: > > > > import scipy > > import numpy > > > > r = scipy.misc.pilutil.imread('face.jpg', 'F') > > > > fft = numpy.fft.fftshift(numpy.fft.fft(r).real) > > fft_2d = numpy.log10(numpy.abs(fft)) > > pil_array = scipy.misc.pilutil.toimage(fft_2d) > > > > scipy.misc.pilutil.imsave('fft.tif', pil_array) > > > > If you could provide the input image that would be helpful for people > trying to determine what the issue is. > > As a stab in the dark -- have you checked (from python) what the > values of at the "black points" are? Are they NaNs? (Perhaps from the > log10 operation?) > > Zach > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > Hi Zachary, The image of input: http://img846.imageshack.us/i/faceg.jpg/ Image before moving on log10. http://img215.imageshack.us/i/fftthreshold2.png/ Thx, Paulo -------------- next part -------------- An HTML attachment was scrubbed... URL: From Pierre.RAYBAUT at CEA.FR Tue Apr 26 07:49:33 2011 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Tue, 26 Apr 2011 13:49:33 +0200 Subject: [SciPy-User] [ANN] guiqwt v2.1.2 Message-ID: Hi all, I am pleased to announce that `guiqwt` v2.1.2 has been released. This release mainly fixes a critical bug when running the GUI-based test launcher (except from the test launcher itself, this bug had absolutely no impact on the library -- it was however considered critical as many regular users were not able to run the test launcher). Main changes since `guiqwt` v2.1.0: * added support for NaNs in image plot items (default behaviour: NaN pixels are transparents) * added "oblique averaged cross section" feature * bugfixes This version of `guiqwt` includes a demo software, Sift (for Signal and Image Filtering Tool), based on `guidata` and `guiqwt`: http://packages.python.org/guiqwt/sift.html Windows users may even download the portable version of Sift 0.23 to test it without having to install anything: http://code.google.com/p/guiqwt/downloads/detail?name=sift023_portable.zip The `guiqwt` documentation with examples, API reference, etc. is available here: http://packages.python.org/guiqwt/ Based on PyQwt (plotting widgets for PyQt4 graphical user interfaces) and on the scientific modules NumPy and SciPy, guiqwt is a Python library providing efficient 2D data-plotting features (curve/image visualization and related tools) for interactive computing and signal/image processing application development. When compared to the excellent module `matplotlib`, the main advantage of `guiqwt` is performance: see http://packages.python.org/guiqwt/overview.html#performances. But `guiqwt` is more than a plotting library; it also provides: * Helper functions for data processing: see the example http://packages.python.org/guiqwt/examples.html#curve-fitting * Framework for signal/image processing application development: see http://packages.python.org/guiqwt/examples.html * And many other features like making executable Windows programs easily (py2exe helpers): see http://packages.python.org/guiqwt/disthelpers.html guiqwt plotting features are the following: guiqwt.pyplot: equivalent to matplotlib's pyplot module (pylab) supported plot items: * curves, error bar curves and 1-D histograms * images (RGB images are not supported), images with non-linear x/y scales, images with specified pixel size (e.g. loaded from DICOM files), 2-D histograms, pseudo-color images (pcolor) * labels, curve plot legends * shapes: polygon, polylines, rectangle, circle, ellipse and segment * annotated shapes (shapes with labels showing position and dimensions): rectangle with center position and size, circle with center position and diameter, ellipse with center position and diameters (these items are very useful to measure things directly on displayed images) curves, images and shapes: * multiple object selection for moving objects or editing their properties through automatically generated dialog boxes (guidata) * item list panel: move objects from foreground to background, show/hide objects, remove objects, ... * customizable aspect ratio * a lot of ready-to-use tools: plot canvas export to image file, image snapshot, image rectangular filter, etc. curves: * interval selection tools with labels showing results of computing on selected area * curve fitting tool with automatic fit, manual fit with sliders, ... images: * contrast adjustment panel: select the LUT by moving a range selection object on the image levels histogram, eliminate outliers, ... * X-axis and Y-axis cross-sections: support for multiple images, average cross-section tool on a rectangular area, ... * apply any affine transform to displayed images in real-time (rotation, magnification, translation, horizontal/vertical flip, ...) application development helpers: * ready-to-use curve and image plot widgets and dialog boxes * load/save graphical objects (curves, images, shapes) * a lot of test scripts which demonstrate guiqwt features guiqwt has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guiqwt/ Documentation, screenshots: http://packages.python.org/guiqwt/ Downloads (source + Python(x,y) plugin): http://guiqwt.googlecode.com Cheers, Pierre --- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From Pierre.RAYBAUT at CEA.FR Tue Apr 26 07:50:59 2011 From: Pierre.RAYBAUT at CEA.FR (Pierre.RAYBAUT at CEA.FR) Date: Tue, 26 Apr 2011 13:50:59 +0200 Subject: [SciPy-User] [ANN] guidata v1.3.1 Message-ID: Hi all, I am pleased to announce that `guidata` v1.3.1 has been released. Note that the project has recently been moved to GoogleCode: http://guidata.googlecode.com Main changes since `guidata` v1.3.0: * gettext_helpers module was not working on Linux * bugfixes The `guidata` documentation with examples, API reference, etc. is available here: http://packages.python.org/guidata/ Based on the Qt Python binding module PyQt4, guidata is a Python library generating graphical user interfaces for easy dataset editing and display. It also provides helpers and application development tools for PyQt4. guidata also provides the following features: * guidata.qthelpers: PyQt4 helpers * guidata.disthelpers: py2exe helpers * guidata.userconfig: .ini configuration management helpers (based on Python standard module ConfigParser) * guidata.configtools: library/application data management * guidata.gettext_helpers: translation helpers (based on the GNU tool gettext) * guidata.guitest: automatic GUI-based test launcher * guidata.utils: miscelleneous utilities guidata has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guidata/ Documentation, screenshots: http://packages.python.org/guidata/ Downloads (source + Python(x,y) plugin): http://guidata.googlecode.com Cheers, Pierre --- Dr. Pierre Raybaut CEA - Commissariat ? l'Energie Atomique et aux Energies Alternatives From denis-bz-gg at t-online.de Tue Apr 26 09:12:03 2011 From: denis-bz-gg at t-online.de (denis) Date: Tue, 26 Apr 2011 06:12:03 -0700 (PDT) Subject: [SciPy-User] Moving median code In-Reply-To: <4DB593D5.4010506@cs.wisc.edu> References: <4DB593D5.4010506@cs.wisc.edu> Message-ID: Folks, Fwiw, pure python for a different algorithm is under http://stackoverflow.com/questions/1309263/rolling-median-algorithm-in-c Method: cache the median, so that wider windows are faster. The code is simple -- no heaps, no trees. See Perreault + Hebert, Median Filtering in Constant Time, 2007, http://nomis80.org/ctmf.html: nice 6-page paper and C code, mainly for 2d images cheers -- denis On Apr 25, 5:31?pm, "J. David Lee" wrote: > In working on a project recently, I wrote a moving median code that is > about 10x faster than scipy.ndimage.median_filter. It utilizes a linked > list structure to store values and track the median value. From zachary.pincus at yale.edu Tue Apr 26 10:49:59 2011 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Tue, 26 Apr 2011 10:49:59 -0400 Subject: [SciPy-User] Image with black points In-Reply-To: References: Message-ID: I don't know if it's image IO differences (I don't use PIL) or what, but the results I get look quite different than what you've shown. Did "face.jpg" get re-encoded by ImageShack (I downloaded a "faceg.jpg" -- that might have altered some frequency bands that you see in the FFT which are absent in what I calculated. Nevertheless, I see some "dark spots" too, but nothing seems wrong with them. Are you sure it's an error? Why don't you figure out the pixel coordinates of a dark spot in your output, and then look at the calculated values in the various intermediate numpy arrays at that position (don't forget to transpose the coords) to satisfy yourself that the calculation is correct? Zach On Apr 25, 2011, at 7:32 PM, Paulo Henrique Junqueira Amorim wrote: > On 25 April 2011 14:15, Zachary Pincus > wrote: > On Apr 24, 2011, at 10:22 PM, Paulo Henrique Junqueira Amorim wrote: > > > Hi, > > > > I'm computing the 1D fft from image, and saving the image of > > frequency. > > > > But the saved image appears black point's (example image), it seems > > that do not belong. > > > > Is this correct? How could I remove it? > > > > Example image - http://img818.imageshack.us/i/fftthreshold.png/ > > > > Code: > > > > import scipy > > import numpy > > > > r = scipy.misc.pilutil.imread('face.jpg', 'F') > > > > fft = numpy.fft.fftshift(numpy.fft.fft(r).real) > > fft_2d = numpy.log10(numpy.abs(fft)) > > pil_array = scipy.misc.pilutil.toimage(fft_2d) > > > > scipy.misc.pilutil.imsave('fft.tif', pil_array) > > > > If you could provide the input image that would be helpful for people > trying to determine what the issue is. > > As a stab in the dark -- have you checked (from python) what the > values of at the "black points" are? Are they NaNs? (Perhaps from the > log10 operation?) > > Zach > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > Hi Zachary, > > The image of input: http://img846.imageshack.us/i/faceg.jpg/ > > Image before moving on log10.http://img215.imageshack.us/i/fftthreshold2.png/ > > Thx, > Paulo From nwerneck at gmail.com Tue Apr 26 20:40:54 2011 From: nwerneck at gmail.com (Nicolau Werneck) Date: Tue, 26 Apr 2011 21:40:54 -0300 Subject: [SciPy-User] Image with black points In-Reply-To: References: Message-ID: My guess is any problems could be related to properly scaling the values of the images. Pay attention to when they should be between 0..1 and 0..255... ++nic On Tue, Apr 26, 2011 at 11:49 AM, Zachary Pincus wrote: > I don't know if it's image IO differences (I don't use PIL) or what, > but the results I get look quite different than what you've shown. Did > "face.jpg" get re-encoded by ImageShack (I downloaded a "faceg.jpg" -- > that might have altered some frequency bands that you see in the FFT > which are absent in what I calculated. > > Nevertheless, I see some "dark spots" too, but nothing seems wrong > with them. Are you sure it's an error? > > Why don't you figure out the pixel coordinates of a dark spot in your > output, and then look at the calculated values in the various > intermediate numpy arrays at that position (don't forget to transpose > the coords) to satisfy yourself that the calculation is correct? > > Zach > > > On Apr 25, 2011, at 7:32 PM, Paulo Henrique Junqueira Amorim wrote: > >> On 25 April 2011 14:15, Zachary Pincus >> wrote: >> On Apr 24, 2011, at 10:22 PM, Paulo Henrique Junqueira Amorim wrote: >> >> > Hi, >> > >> > I'm computing the 1D fft from image, and saving the image of >> > frequency. >> > >> > But the saved image appears black point's (example image), it seems >> > that do not belong. >> > >> > Is this correct? How could I remove it? >> > >> > Example image - http://img818.imageshack.us/i/fftthreshold.png/ >> > >> > Code: >> > >> > import scipy >> > import numpy >> > >> > r = scipy.misc.pilutil.imread('face.jpg', 'F') >> > >> > fft = numpy.fft.fftshift(numpy.fft.fft(r).real) >> > fft_2d = numpy.log10(numpy.abs(fft)) >> > pil_array = scipy.misc.pilutil.toimage(fft_2d) >> > >> > scipy.misc.pilutil.imsave('fft.tif', pil_array) >> > >> >> If you could provide the input image that would be helpful for people >> trying to determine what the issue is. >> >> As a stab in the dark -- have you checked (from python) what the >> values of at the "black points" are? Are they NaNs? (Perhaps from the >> log10 operation?) >> >> Zach >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> >> Hi Zachary, >> >> The image of input: ?http://img846.imageshack.us/i/faceg.jpg/ >> >> Image before moving on log10.http://img215.imageshack.us/i/fftthreshold2.png/ >> >> Thx, >> Paulo > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -- Nicolau Werneck ? ? ? ?? C3CF E29F 5350 5DAA 3705 http://www.lti.pcs.usp.br/~nwerneck? ? ? ? ? ? ? ? ?? 7B9E D6C4 37BB DA64 6F15 Linux user #460716 From arsbbr at gmx.net Wed Apr 27 04:50:06 2011 From: arsbbr at gmx.net (arsbbr at gmx.net) Date: Wed, 27 Apr 2011 10:50:06 +0200 Subject: [SciPy-User] Speedup with integrate.romberg() and vector arguments Message-ID: <20110427085006.325480@gmx.net> Hi, I need to integrate a function with an [1xn] array argument q. For this I used integrate.quad(), but then I have to loop through every element in q. In http://www.astrobetter.com/interpolation-and-integration-in-python/ I found a hint how to speed up the integration. Unfortunately I can't seem to get romberg() to work: q is a [1 x n] numpy array, so I went from def phi_quad(q, a, alpha, beta, r_0, r=r): # phi(q,...,r) y = zeros(r.size) for k in range(r.size): y[k] = integrate.quad(dy_dr, r[0], r[-1], # dy(r,...) args=(q[k], A, a, alpha, beta, r_0))[0] return y to def phi_romb_vec(q, a, alpha, beta, r_0, r=r): # phi(q,...,r) y = integrate.romberg(dy_dr, r[0], r[-1], # dy(r,...) args=(q, A, a, alpha, beta, r_0), vec_func=True) return y which raises an error, if q is an array. The other arguments are all floats. I attached a working example of the problem. Is there any way to speed up the integration? Best regards, Arne -- NEU: FreePhone - kostenlos mobil telefonieren und surfen! Jetzt informieren: http://www.gmx.net/de/go/freephone -------------- next part -------------- A non-text attachment was scrubbed... Name: nonlin_sans.py Type: text/x-python Size: 2247 bytes Desc: not available URL: From bgeorge98121 at gmail.com Tue Apr 26 22:10:32 2011 From: bgeorge98121 at gmail.com (Bruno George) Date: Tue, 26 Apr 2011 19:10:32 -0700 Subject: [SciPy-User] Image with black points In-Reply-To: References: Message-ID: Hi, You know the two images are mirroring (flopped) to each other? Bruno George On Mon, Apr 25, 2011 at 10:15 AM, Zachary Pincus wrote: > On Apr 24, 2011, at 10:22 PM, Paulo Henrique Junqueira Amorim wrote: > > > Hi, > > > > I'm computing the 1D fft from image, and saving the image of > > frequency. > > > > But the saved image appears black point's (example image), it seems > > that do not belong. > > > > Is this correct? How could I remove it? > > > > Example image - http://img818.imageshack.us/i/fftthreshold.png/ > > > > Code: > > > > import scipy > > import numpy > > > > r = scipy.misc.pilutil.imread('face.jpg', 'F') > > > > fft = numpy.fft.fftshift(numpy.fft.fft(r).real) > > fft_2d = numpy.log10(numpy.abs(fft)) > > pil_array = scipy.misc.pilutil.toimage(fft_2d) > > > > scipy.misc.pilutil.imsave('fft.tif', pil_array) > > > > If you could provide the input image that would be helpful for people > trying to determine what the issue is. > > As a stab in the dark -- have you checked (from python) what the > values of at the "black points" are? Are they NaNs? (Perhaps from the > log10 operation?) > > Zach > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Apr 27 09:14:52 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 27 Apr 2011 09:14:52 -0400 Subject: [SciPy-User] Speedup with integrate.romberg() and vector arguments In-Reply-To: <20110427085006.325480@gmx.net> References: <20110427085006.325480@gmx.net> Message-ID: On Wed, Apr 27, 2011 at 4:50 AM, wrote: > Hi, > > I need to integrate a function with an [1xn] array argument q. > For this I used integrate.quad(), but then I have to loop through every element in q. > > In http://www.astrobetter.com/interpolation-and-integration-in-python/ I found a hint how to speed up the integration. > > Unfortunately I can't seem to get romberg() to work: > > q is a [1 x n] numpy array, so I went from > > def phi_quad(q, a, alpha, beta, r_0, r=r): # phi(q,...,r) > ? ?y = zeros(r.size) > ? ?for k in range(r.size): > ? ? ? ?y[k] = integrate.quad(dy_dr, r[0], r[-1], # dy(r,...) > ? ? ? ? ? ?args=(q[k], A, a, alpha, beta, r_0))[0] > ? ?return y > > to > > def phi_romb_vec(q, a, alpha, beta, r_0, r=r): # phi(q,...,r) > ? ?y = integrate.romberg(dy_dr, r[0], r[-1], # dy(r,...) > ? ? ? ? args=(q, A, a, alpha, beta, r_0), vec_func=True) > ? ?return y > > which raises an error, if q is an array. The other arguments are all floats. I attached a working example of the problem. Is there any way to speed up the integration? I never tried this, but from the blog and the documentation, this is my interpretation It looks like romberg takes a vector argument but handles only a single integration problem (scalar output). in the function to be integrated, romberg uses a vector of r to speed up the integration dy_dr(r, q, A, a, alpha, beta, r_0) however, q and the other parameters need to be scalars So you still need to loop over q, doing one integration problem at a time. It might be possible to enhance romberg to proper broadcasting, but I think it would require that the number of iterations, number of subdivisions is the same for all integration problems. If you are willing to work with a fixed grid, then integrate.romb might be useful, since from the docs it takes an axis argument, so a broadcasted (r, q) grid might work. Josef > > Best regards, > Arne > -- > NEU: FreePhone - kostenlos mobil telefonieren und surfen! > Jetzt informieren: http://www.gmx.net/de/go/freephone > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From josef.pktd at gmail.com Wed Apr 27 10:18:54 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 27 Apr 2011 10:18:54 -0400 Subject: [SciPy-User] Speedup with integrate.romberg() and vector arguments In-Reply-To: References: <20110427085006.325480@gmx.net> Message-ID: On Wed, Apr 27, 2011 at 9:14 AM, wrote: > On Wed, Apr 27, 2011 at 4:50 AM, ? wrote: >> Hi, >> >> I need to integrate a function with an [1xn] array argument q. >> For this I used integrate.quad(), but then I have to loop through every element in q. >> >> In http://www.astrobetter.com/interpolation-and-integration-in-python/ I found a hint how to speed up the integration. >> >> Unfortunately I can't seem to get romberg() to work: >> >> q is a [1 x n] numpy array, so I went from >> >> def phi_quad(q, a, alpha, beta, r_0, r=r): # phi(q,...,r) >> ? ?y = zeros(r.size) >> ? ?for k in range(r.size): >> ? ? ? ?y[k] = integrate.quad(dy_dr, r[0], r[-1], # dy(r,...) >> ? ? ? ? ? ?args=(q[k], A, a, alpha, beta, r_0))[0] >> ? ?return y >> >> to >> >> def phi_romb_vec(q, a, alpha, beta, r_0, r=r): # phi(q,...,r) >> ? ?y = integrate.romberg(dy_dr, r[0], r[-1], # dy(r,...) >> ? ? ? ? args=(q, A, a, alpha, beta, r_0), vec_func=True) >> ? ?return y >> >> which raises an error, if q is an array. The other arguments are all floats. I attached a working example of the problem. Is there any way to speed up the integration? > > I never tried this, but from the blog and the documentation, this is > my interpretation > > It looks like romberg takes a vector argument but handles only a > single integration problem (scalar output). > > in the function to be integrated, romberg uses a vector of r to speed > up the integration > dy_dr(r, q, A, a, alpha, beta, r_0) > however, q and the other parameters need to be scalars > > So you still need to loop over q, doing one integration problem at a time. > > It might be possible to enhance romberg to proper broadcasting, but I > think it would require that the number of iterations, number of > subdivisions is the same for all integration problems. this actually seems to work with minimal changes if np.all(err < tol) or np.all(err < rtol*abs(result)): but needs some broadcasting checks. >>> np.concatenate([phi_romb_vec(qi, a, alpha, beta, r_0, r=r) for qi in q]) array([ 0.00065848, 0.00127215, 0.00216926, 0.00339121, 0.00497182, 0.00693667, 0.0093027 , 0.01207799, 0.01526184, 0.01884502, 0.02281024, 0.02713285, 0.03178164, 0.03671976, 0.04190578, 0.0472947 , 0.05283912, 0.05849023, 0.06419887, 0.06991646, 0.07559587, 0.08119216, 0.08666319, 0.09197017, 0.09707798, 0.1019555 , 0.1065757 , 0.11091574, 0.11495692, 0.11868458, 0.12208791, 0.12515975, 0.12789636, 0.13029711, 0.13236424, 0.13410255, 0.1355191 , 0.13662298, 0.137425 , 0.13793743, 0.13817382, 0.13814868, 0.13787734, 0.13737571, 0.13666008, 0.13574698, 0.13465298, 0.13339455, 0.13198792, 0.13044894, 0.128793 , 0.12703493, 0.12518888, 0.12326832, 0.12128595, 0.11925369, 0.11718264]) >>> phi_romb_vec(q[None,:], a, alpha, beta, r_0, r=r) array([[ 0.00065848, 0.00127215, 0.00216926, 0.00339121, 0.00497182, 0.00693667, 0.0093027 , 0.01207799, 0.01526184, 0.01884502, 0.02281024, 0.02713285, 0.03178164, 0.03671976, 0.04190578, 0.0472947 , 0.05283912, 0.05849023, 0.06419887, 0.06991646, 0.07559587, 0.08119216, 0.08666319, 0.09197017, 0.09707798, 0.1019555 , 0.1065757 , 0.11091574, 0.11495692, 0.11868458, 0.1220879 , 0.12515975, 0.12789636, 0.13029711, 0.13236424, 0.13410255, 0.1355191 , 0.13662298, 0.137425 , 0.13793743, 0.13817382, 0.13814868, 0.13787734, 0.13737571, 0.13666008, 0.13574698, 0.13465298, 0.13339455, 0.13198792, 0.13044894, 0.128793 , 0.12703493, 0.12518888, 0.12326832, 0.12128595, 0.11925369, 0.11718264]]) I haven't managed the call to curve_fit with the vectorized romberg Josef > > If you are willing to work with a fixed grid, then integrate.romb > might be useful, since from the docs it takes an axis argument, so a > broadcasted (r, q) grid might work. > > Josef > > >> >> Best regards, >> Arne >> -- >> NEU: FreePhone - kostenlos mobil telefonieren und surfen! >> Jetzt informieren: http://www.gmx.net/de/go/freephone >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > From denis-bz-gg at t-online.de Wed Apr 27 11:32:41 2011 From: denis-bz-gg at t-online.de (denis) Date: Wed, 27 Apr 2011 08:32:41 -0700 (PDT) Subject: [SciPy-User] Is anyone using the Lq aka fractional metric ? Message-ID: Is anyone using the Lq aka fractional or near-Hamming metric sum |a_j - b_j|^q 0 <= q < 1 ? This up-weights close matches in a few features, which make sense for feature matching in high dimensions; otherwise a sum of even 10 terms is normally distributed with no contrast at all, distance whiteout. (Yes near-Hamming is not a norm but it is a metric, satisfies the triangle inequality -- note there's no outer ^ 1/q ). (A WIBNI, wouldn't it be nice if, scipy *uniformly* used this wherever Lp, p >= 1, can be used: scipy.spatial.distance, scipy.spatial.cKDTree ... p == 0: Hamming distance, sum( aj != bj ) 0 < p < 1: Lq, near-Hamming p >= 1: L1 Manhattan, L2 Euclidean, Lmax, Lp Minkowski. I've modified ckdtree.pyx for this, trivial, but is there any demand / any experience ?) cheers -- denis From Nikolaus at rath.org Wed Apr 27 17:31:29 2011 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 27 Apr 2011 17:31:29 -0400 Subject: [SciPy-User] SciPy 0.9 install: _fitpack.so: undefined symbol: jwe_ilst Message-ID: <87hb9jz8i6.fsf@inspiron.ap.columbia.edu> Hello, I'm trying to build Scipy 0.9 from source. The installation with "setup.py install" seemed to work just fine, but when I try to.. In [1]: import scipy.interpolate ..then I get... ImportError: /home/byrne/.local/lib/python2.6/site-packages/scipy/interpolate/_fitpack.so: undefined symbol: jwe_ilst Any suggestions what may have gone wrong and how I could fix it? Thanks, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From paulojamorim at gmail.com Wed Apr 27 20:04:53 2011 From: paulojamorim at gmail.com (Paulo Henrique Junqueira Amorim) Date: Wed, 27 Apr 2011 21:04:53 -0300 Subject: [SciPy-User] Image with black points In-Reply-To: References: Message-ID: Hi Bruno, Yeah Bruno, how could only one of them? Thx, Paulo On 26 April 2011 23:10, Bruno George wrote: > Hi, > You know the two images are mirroring (flopped) to each other? > > Bruno George > > > On Mon, Apr 25, 2011 at 10:15 AM, Zachary Pincus wrote: > >> On Apr 24, 2011, at 10:22 PM, Paulo Henrique Junqueira Amorim wrote: >> >> > Hi, >> > >> > I'm computing the 1D fft from image, and saving the image of >> > frequency. >> > >> > But the saved image appears black point's (example image), it seems >> > that do not belong. >> > >> > Is this correct? How could I remove it? >> > >> > Example image - http://img818.imageshack.us/i/fftthreshold.png/ >> > >> > Code: >> > >> > import scipy >> > import numpy >> > >> > r = scipy.misc.pilutil.imread('face.jpg', 'F') >> > >> > fft = numpy.fft.fftshift(numpy.fft.fft(r).real) >> > fft_2d = numpy.log10(numpy.abs(fft)) >> > pil_array = scipy.misc.pilutil.toimage(fft_2d) >> > >> > scipy.misc.pilutil.imsave('fft.tif', pil_array) >> > >> >> If you could provide the input image that would be helpful for people >> trying to determine what the issue is. >> >> As a stab in the dark -- have you checked (from python) what the >> values of at the "black points" are? Are they NaNs? (Perhaps from the >> log10 operation?) >> >> Zach >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwerneck at gmail.com Wed Apr 27 22:28:55 2011 From: nwerneck at gmail.com (Nicolau Werneck) Date: Wed, 27 Apr 2011 23:28:55 -0300 Subject: [SciPy-User] Image with black points In-Reply-To: References: Message-ID: What result are you expecting anyway? This all looks very natural to me. The FFT is always symmetric, and your black points are the locations where the curves cross the zero from negative to positive values. They are also happening a lot more in the noisy, high-frequency regions, that probably don't matter much to your application. (Face image analysis?) ++nic On Wed, Apr 27, 2011 at 9:04 PM, Paulo Henrique Junqueira Amorim wrote: > Hi Bruno, > > Yeah Bruno, how could only one of them? > > Thx, > Paulo > > > On 26 April 2011 23:10, Bruno George wrote: >> >> Hi, >> You know the two images are mirroring (flopped) to each other? >> >> Bruno George >> >> On Mon, Apr 25, 2011 at 10:15 AM, Zachary Pincus >> wrote: >>> >>> On Apr 24, 2011, at 10:22 PM, Paulo Henrique Junqueira Amorim wrote: >>> >>> > Hi, >>> > >>> > I'm computing the 1D fft from image, and saving the image of >>> > frequency. >>> > >>> > But the saved image appears black point's (example image), it seems >>> > that do not belong. >>> > >>> > Is this correct? How could I remove it? >>> > >>> > Example image - http://img818.imageshack.us/i/fftthreshold.png/ >>> > >>> > Code: >>> > >>> > import scipy >>> > import numpy >>> > >>> > r = scipy.misc.pilutil.imread('face.jpg', 'F') >>> > >>> > fft = numpy.fft.fftshift(numpy.fft.fft(r).real) >>> > fft_2d = numpy.log10(numpy.abs(fft)) >>> > pil_array = scipy.misc.pilutil.toimage(fft_2d) >>> > >>> > scipy.misc.pilutil.imsave('fft.tif', pil_array) >>> > >>> >>> If you could provide the input image that would be helpful for people >>> trying to determine what the issue is. >>> >>> As a stab in the dark -- have you checked (from python) what the >>> values of at the "black points" are? Are they NaNs? (Perhaps from the >>> log10 operation?) >>> >>> Zach >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- Nicolau Werneck ? ? ? ?? C3CF E29F 5350 5DAA 3705 http://www.lti.pcs.usp.br/~nwerneck? ? ? ? ? ? ? ? ?? 7B9E D6C4 37BB DA64 6F15 Linux user #460716 From dima.osin at gmail.com Wed Apr 27 23:08:54 2011 From: dima.osin at gmail.com (dima osin) Date: Wed, 27 Apr 2011 23:08:54 -0400 Subject: [SciPy-User] confidence interval for leastsq fit Message-ID: How to calculate confidence interval for scipy.optimize.leastsq fit using the Student's t distribution and NOT the bootstrapping method? -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Thu Apr 28 06:00:11 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 28 Apr 2011 12:00:11 +0200 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: Message-ID: <20110428100011.GD17045@phare.normalesup.org> On Wed, Apr 27, 2011 at 11:08:54PM -0400, dima osin wrote: > How to calculate confidence interval for scipy.optimize.leastsq fit using > the Student's t distribution and NOT the bootstrapping method? I suspect that you are the same person that I just replied to on stack overflow. I'll copy my answer here: """ I am not sure what you mean by confidence interval. In general, leastsq doesn't know much about the function that you are trying to minimize, so it can't really give a confidence interval. However, it does return an estimate of the Hessian, in other word the generalization of 2nd derivatives to multidimensional problems. As hinted in the docstring of the function, you could use that information along with the residuals (the difference between your fitted solution and the actual data) to computed the covariance of parameter estimates, which is a local guess of the confidence interval. Note that it is only a local information, and I suspect that you can strictly speaking come to a conclusion only if your objective function is strictly convex. I don't have any proofs or references on that statement :). """ That said, you give precisions above "using the Student's t distribution and NOT the bootstrapping method". I am a bit worried that you employ the concept of Student's t distribution. Strictly speaking, a Student's t is applicable only under Gaussian hypothesis, in a linear regression setting. In other words, it comes with assumptions that could very well be violated by your cost function and your data. If your cost function is well behaved (i.e. the optimizer always finds the same mimimum, and it's Hessian is well conditionned near this minimum) and if you have a data-fitting problem, I suspect that assymptotic normality will apply, and you will be able to use Gaussian test statistics. These are big ifs. HTH, Gael From josef.pktd at gmail.com Thu Apr 28 06:09:14 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 06:09:14 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: Message-ID: On Wed, Apr 27, 2011 at 11:08 PM, dima osin wrote: > How to calculate confidence interval for scipy.optimize.leastsq fit using > the Student's t distribution and NOT the bootstrapping method? > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > if you mean the confidence interval for the parameter estimates formula and calulation for the covariance matrix of the parameter estimate is in optimize curvefit bse = np.sqrt(np.diag(cov) confint = params + np.array([-1, +1]) * stats.t.ppf(alpha/ 2) * bse[:,None] (from scikits.statsmodels LikelihoodModel confint: ) lower = self.params[cols] - dist.ppf(1-\ alpha/2,self.model.df_resid) * bse[cols] upper = self.params[cols] + dist.ppf(1-\ alpha/2,self.model.df_resid) * bse[cols] there is also a scikits.statsmodels.miscmodels.NonlinearLS which is similar to scipy.optimize curvefit but a class with all regression results using the gradient information. Caution: doesn't have complete test suite yet, only partially tested on some examples. If you mean confidence interval for prediction, then I'm not completely sure the analogy to the linear models works unchanged. Josef From josef.pktd at gmail.com Thu Apr 28 06:18:45 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 06:18:45 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: <20110428100011.GD17045@phare.normalesup.org> References: <20110428100011.GD17045@phare.normalesup.org> Message-ID: On Thu, Apr 28, 2011 at 6:00 AM, Gael Varoquaux wrote: > On Wed, Apr 27, 2011 at 11:08:54PM -0400, dima osin wrote: >> ? ?How to calculate confidence interval for scipy.optimize.leastsq fit using >> ? ?the Student's t distribution and NOT the bootstrapping method? > > I suspect that you are the same person that I just replied to on stack > overflow. I'll copy my answer here: > > """ > I am not sure what you mean by confidence interval. > > In general, leastsq doesn't know much about the function that you are > trying to minimize, so it can't really give a confidence interval. > However, it does return an estimate of the Hessian, in other word the > generalization of 2nd derivatives to multidimensional problems. > > As hinted in the docstring of the function, you could use that > information along with the residuals (the difference between your fitted > solution and the actual data) to computed the covariance of parameter > estimates, which is a local guess of the confidence interval. > > Note that it is only a local information, and I suspect that you can > strictly speaking come to a conclusion only if your objective function is > strictly convex. I don't have any proofs or references on that statement > :). > """ > > That said, you give precisions above "using the Student's t distribution > and NOT the bootstrapping method". I am a bit worried that you employ the > concept of Student's t distribution. Strictly speaking, a Student's t is > applicable only under Gaussian hypothesis, in a linear regression > setting. In other words, it comes with assumptions that could very well > be violated by your cost function and your data. If your cost function is > well behaved (i.e. the optimizer always finds the same mimimum, and it's > Hessian is well conditionned near this minimum) and if you have a > data-fitting problem, I suspect that assymptotic normality will apply, > and you will be able to use Gaussian test statistics. These are big ifs. given the question is about optimize.leastsq, I assume this applies : http://en.wikipedia.org/wiki/Non-linear_least_squares#Parameter_errors.2C_confidence_limits.2C_residuals_etc. non-linear model with additive error y = f(x, params) + err function f is differentiable. minimize quadratic loss parameter is in the interior of the parameter space if there are bounds all regression results of the linear model apply based on local (derivatives), asymptotic (normality) distribution using the Jacobian instead of the design matrix. Josef > > HTH, > > Gael > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Thu Apr 28 06:28:12 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 06:28:12 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: <20110428100011.GD17045@phare.normalesup.org> Message-ID: On Thu, Apr 28, 2011 at 6:18 AM, wrote: > On Thu, Apr 28, 2011 at 6:00 AM, Gael Varoquaux > wrote: >> On Wed, Apr 27, 2011 at 11:08:54PM -0400, dima osin wrote: >>> ? ?How to calculate confidence interval for scipy.optimize.leastsq fit using >>> ? ?the Student's t distribution and NOT the bootstrapping method? >> >> I suspect that you are the same person that I just replied to on stack >> overflow. I'll copy my answer here: >> >> """ >> I am not sure what you mean by confidence interval. >> >> In general, leastsq doesn't know much about the function that you are >> trying to minimize, so it can't really give a confidence interval. >> However, it does return an estimate of the Hessian, in other word the >> generalization of 2nd derivatives to multidimensional problems. >> >> As hinted in the docstring of the function, you could use that >> information along with the residuals (the difference between your fitted >> solution and the actual data) to computed the covariance of parameter >> estimates, which is a local guess of the confidence interval. >> >> Note that it is only a local information, and I suspect that you can >> strictly speaking come to a conclusion only if your objective function is >> strictly convex. I don't have any proofs or references on that statement >> :). >> """ >> >> That said, you give precisions above "using the Student's t distribution >> and NOT the bootstrapping method". I am a bit worried that you employ the >> concept of Student's t distribution. Strictly speaking, a Student's t is >> applicable only under Gaussian hypothesis, in a linear regression >> setting. In other words, it comes with assumptions that could very well >> be violated by your cost function and your data. If your cost function is >> well behaved (i.e. the optimizer always finds the same mimimum, and it's >> Hessian is well conditionned near this minimum) and if you have a >> data-fitting problem, I suspect that assymptotic normality will apply, >> and you will be able to use Gaussian test statistics. These are big ifs. > > > given the question is about optimize.leastsq, I assume this applies : > > http://en.wikipedia.org/wiki/Non-linear_least_squares#Parameter_errors.2C_confidence_limits.2C_residuals_etc. > > non-linear model with additive error ?y = f(x, params) + err > function f is differentiable. > minimize quadratic loss > parameter is in the interior of the parameter space if there are bounds > > all regression results of the linear model apply based on local > (derivatives), asymptotic (normality) distribution using the Jacobian > instead of the design matrix. small correction: if the errors are distributed as normal, then it's only a local argument, based on delta method http://en.wikipedia.org/wiki/Delta_method#Motivation_of_multivariate_delta_method and we use the t distribution for the confidence interval. asymptotic results would be required if distribution of error is not assumed to be normal, then the confidence interval would be based on the normal distribution, similar to the hessian based calculations for a generic maximum likelihood calculation (I think). Josef > > Josef > >> >> HTH, >> >> Gael >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > From e.antero.tammi at gmail.com Thu Apr 28 08:01:48 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 28 Apr 2011 15:01:48 +0300 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: Message-ID: Hi On Thu, Apr 28, 2011 at 1:09 PM, wrote: > On Wed, Apr 27, 2011 at 11:08 PM, dima osin wrote: > > How to calculate confidence interval for scipy.optimize.leastsq fit using > > the Student's t distribution and NOT the bootstrapping method? > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > > if you mean the confidence interval for the parameter estimates > > formula and calulation for the covariance matrix of the parameter > estimate is in optimize curvefit > bse = np.sqrt(np.diag(cov) > confint = params + np.array([-1, +1]) * stats.t.ppf(alpha/ 2) * bse[:,None] > According to leastsq documentation (v0.10.dev) " cov_x : ndarray ... This matrix must be multiplied by the residual standard deviation to get the covariance of the parameter estimates ? see curve_fit." However curve_fit doc states "pcov : 2d array The estimated covariance of popt. The diagonals provide the variance of the parameter estimate." Shouldn't bse (based on leastsq) be claculated like: bse= sqrt(sqrt(sum(residual** 2)/ (M- N))* dig(cov)) # assumed residual zero mean normally distributed Regards, eat > > > (from scikits.statsmodels LikelihoodModel confint: ) > lower = self.params[cols] - dist.ppf(1-\ > alpha/2,self.model.df_resid) * bse[cols] > upper = self.params[cols] + dist.ppf(1-\ > alpha/2,self.model.df_resid) * bse[cols] > > there is also a scikits.statsmodels.miscmodels.NonlinearLS which is > similar to scipy.optimize curvefit but a class with all regression > results using the gradient information. Caution: doesn't have complete > test suite yet, only partially tested on some examples. > > If you mean confidence interval for prediction, then I'm not > completely sure the analogy to the linear models works unchanged. > > Josef > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail.till at gmx.de Thu Apr 28 08:11:35 2011 From: mail.till at gmx.de (Till Stensitzki) Date: Thu, 28 Apr 2011 12:11:35 +0000 (UTC) Subject: [SciPy-User] confidence interval for leastsq fit References: Message-ID: In ultrafast physics, we often use, that the ration of the reduced chi^2 follows a F-Distribution. Most of the time, we are a fitting a sum-of- exponentials model to our data, in which the covariance between the parameters is quite hight. Using the cov-diagonal to calculate the confidence interval underestimates the size and form of the interval, so we are using a exhaustive search method. See http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1260379/?page=4 for details. From e.antero.tammi at gmail.com Thu Apr 28 08:12:27 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 28 Apr 2011 15:12:27 +0300 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: Message-ID: On Thu, Apr 28, 2011 at 3:01 PM, eat wrote: > Hi > > On Thu, Apr 28, 2011 at 1:09 PM, wrote: > >> On Wed, Apr 27, 2011 at 11:08 PM, dima osin wrote: >> > How to calculate confidence interval for scipy.optimize.leastsq fit >> using >> > the Student's t distribution and NOT the bootstrapping method? >> > >> > _______________________________________________ >> > SciPy-User mailing list >> > SciPy-User at scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-user >> > >> > >> >> if you mean the confidence interval for the parameter estimates >> >> formula and calulation for the covariance matrix of the parameter >> estimate is in optimize curvefit >> bse = np.sqrt(np.diag(cov) >> confint = params + np.array([-1, +1]) * stats.t.ppf(alpha/ 2) * >> bse[:,None] >> > According to leastsq documentation (v0.10.dev) " cov_x : ndarray ... This > matrix must be multiplied by the residual standard deviation to get the > covariance of the parameter estimates ? see curve_fit." However curve_fit > doc states "pcov : 2d array The estimated covariance of popt. The diagonals > provide the variance of the parameter estimate." > Shouldn't bse (based on leastsq) be claculated like: > bse= sqrt(sqrt(sum(residual** 2)/ (M- N))* dig(cov)) # assumed residual > zero mean normally distributed > Oops meant to be like: bse= sqrt(sum(residual** 2)/ (M- N))* dig(cov) # assumed residual zero mean normally distributed > > Regards, > eat > >> >> (from scikits.statsmodels LikelihoodModel confint: ) >> lower = self.params[cols] - dist.ppf(1-\ >> alpha/2,self.model.df_resid) * bse[cols] >> upper = self.params[cols] + dist.ppf(1-\ >> alpha/2,self.model.df_resid) * bse[cols] >> >> there is also a scikits.statsmodels.miscmodels.NonlinearLS which is >> similar to scipy.optimize curvefit but a class with all regression >> results using the gradient information. Caution: doesn't have complete >> test suite yet, only partially tested on some examples. >> >> If you mean confidence interval for prediction, then I'm not >> completely sure the analogy to the linear models works unchanged. >> >> Josef >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 28 08:20:27 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 08:20:27 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: Message-ID: On Thu, Apr 28, 2011 at 8:12 AM, eat wrote: > > > On Thu, Apr 28, 2011 at 3:01 PM, eat wrote: >> >> Hi >> >> On Thu, Apr 28, 2011 at 1:09 PM, wrote: >>> >>> On Wed, Apr 27, 2011 at 11:08 PM, dima osin wrote: >>> > How to calculate confidence interval for scipy.optimize.leastsq fit >>> > using >>> > the Student's t distribution and NOT the bootstrapping method? >>> > >>> > _______________________________________________ >>> > SciPy-User mailing list >>> > SciPy-User at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/scipy-user >>> > >>> > >>> >>> if you mean the confidence interval for the parameter estimates >>> >>> formula and calulation for the covariance matrix of the parameter >>> estimate is in optimize curvefit >>> bse = np.sqrt(np.diag(cov) >>> confint = params + np.array([-1, +1]) * stats.t.ppf(alpha/ 2) * >>> bse[:,None] >> >> According to leastsq documentation (v0.10.dev)?"?cov_x?: ndarray?... This >> matrix must be multiplied by the residual standard deviation to get the >> covariance of the parameter estimates ? see curve_fit." However curve_fit >> doc states "pcov?: 2d array?The estimated covariance of popt. The diagonals >> provide the variance of the parameter estimate." >> Shouldn't bse (based on leastsq) be claculated like: >> bse= sqrt(sqrt(sum(residual** 2)/ (M- N))* dig(cov)) # assumed residual >> zero mean normally distributed > > Oops > meant to be like: > bse= sqrt(sum(residual** 2)/ (M- N))* dig(cov) # assumed residual zero mean > normally distributed optimize.leastsq returns cov_x which would be the cov in this calculation and needs the multiplication. optimize.curve_fit already does the multiplication with the sum squared residual, so the cov that curve_fit returns is the actual parameter covariance, which is the one I meant originally cf_params, cf_pcov = optimize.curve_fit(func0, x, y) cf_bse = np.sqrt(np.diag(cf_pcov)) Josef >> >> Regards, >> eat >>> >>> (from scikits.statsmodels LikelihoodModel confint: ) >>> ? ? ? ? ? ?lower = self.params[cols] - dist.ppf(1-\ >>> ? ? ? ? ? ? ? ? ? ? ? ?alpha/2,self.model.df_resid) * bse[cols] >>> ? ? ? ? ? ?upper = self.params[cols] + dist.ppf(1-\ >>> ? ? ? ? ? ? ? ? ? ? ? ?alpha/2,self.model.df_resid) * bse[cols] >>> >>> there is also a scikits.statsmodels.miscmodels.NonlinearLS which ?is >>> similar to scipy.optimize curvefit but a class with all regression >>> results using the gradient information. Caution: doesn't have complete >>> test suite yet, only partially tested on some examples. >>> >>> If you mean confidence interval for prediction, then I'm not >>> completely sure the analogy to the linear models works unchanged. >>> >>> Josef >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >> > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From e.antero.tammi at gmail.com Thu Apr 28 08:29:20 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 28 Apr 2011 15:29:20 +0300 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: Message-ID: On Thu, Apr 28, 2011 at 3:12 PM, eat wrote: > > > On Thu, Apr 28, 2011 at 3:01 PM, eat wrote: > >> Hi >> >> On Thu, Apr 28, 2011 at 1:09 PM, wrote: >> >>> On Wed, Apr 27, 2011 at 11:08 PM, dima osin wrote: >>> > How to calculate confidence interval for scipy.optimize.leastsq fit >>> using >>> > the Student's t distribution and NOT the bootstrapping method? >>> > >>> > _______________________________________________ >>> > SciPy-User mailing list >>> > SciPy-User at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/scipy-user >>> > >>> > >>> >>> if you mean the confidence interval for the parameter estimates >>> >>> formula and calulation for the covariance matrix of the parameter >>> estimate is in optimize curvefit >>> bse = np.sqrt(np.diag(cov) >>> confint = params + np.array([-1, +1]) * stats.t.ppf(alpha/ 2) * >>> bse[:,None] >>> >> According to leastsq documentation (v0.10.dev) " cov_x : ndarray ... This >> matrix must be multiplied by the residual standard deviation to get the >> covariance of the parameter estimates ? see curve_fit." However curve_fit >> doc states "pcov : 2d array The estimated covariance of popt. The diagonals >> provide the variance of the parameter estimate." >> Shouldn't bse (based on leastsq) be claculated like: >> bse= sqrt(sqrt(sum(residual** 2)/ (M- N))* dig(cov)) # assumed residual >> zero mean normally distributed >> > Oops > meant to be like: > bse= sqrt(sum(residual** 2)/ (M- N))* dig(cov) # assumed residual zero mean > normally distributed > Sorry for the noise, please just ignore my last post. > >> Regards, >> eat >> >>> >>> (from scikits.statsmodels LikelihoodModel confint: ) >>> lower = self.params[cols] - dist.ppf(1-\ >>> alpha/2,self.model.df_resid) * bse[cols] >>> upper = self.params[cols] + dist.ppf(1-\ >>> alpha/2,self.model.df_resid) * bse[cols] >>> >>> there is also a scikits.statsmodels.miscmodels.NonlinearLS which is >>> similar to scipy.optimize curvefit but a class with all regression >>> results using the gradient information. Caution: doesn't have complete >>> test suite yet, only partially tested on some examples. >>> >>> If you mean confidence interval for prediction, then I'm not >>> completely sure the analogy to the linear models works unchanged. >>> >>> Josef >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Thu Apr 28 08:31:51 2011 From: e.antero.tammi at gmail.com (eat) Date: Thu, 28 Apr 2011 15:31:51 +0300 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: Message-ID: On Thu, Apr 28, 2011 at 3:20 PM, wrote: > On Thu, Apr 28, 2011 at 8:12 AM, eat wrote: > > > > > > On Thu, Apr 28, 2011 at 3:01 PM, eat wrote: > >> > >> Hi > >> > >> On Thu, Apr 28, 2011 at 1:09 PM, wrote: > >>> > >>> On Wed, Apr 27, 2011 at 11:08 PM, dima osin > wrote: > >>> > How to calculate confidence interval for scipy.optimize.leastsq fit > >>> > using > >>> > the Student's t distribution and NOT the bootstrapping method? > >>> > > >>> > _______________________________________________ > >>> > SciPy-User mailing list > >>> > SciPy-User at scipy.org > >>> > http://mail.scipy.org/mailman/listinfo/scipy-user > >>> > > >>> > > >>> > >>> if you mean the confidence interval for the parameter estimates > >>> > >>> formula and calulation for the covariance matrix of the parameter > >>> estimate is in optimize curvefit > >>> bse = np.sqrt(np.diag(cov) > >>> confint = params + np.array([-1, +1]) * stats.t.ppf(alpha/ 2) * > >>> bse[:,None] > >> > >> According to leastsq documentation (v0.10.dev) " cov_x : ndarray ... > This > >> matrix must be multiplied by the residual standard deviation to get the > >> covariance of the parameter estimates ? see curve_fit." However > curve_fit > >> doc states "pcov : 2d array The estimated covariance of popt. The > diagonals > >> provide the variance of the parameter estimate." > >> Shouldn't bse (based on leastsq) be claculated like: > >> bse= sqrt(sqrt(sum(residual** 2)/ (M- N))* dig(cov)) # assumed residual > >> zero mean normally distributed > > > > Oops > > meant to be like: > > bse= sqrt(sum(residual** 2)/ (M- N))* dig(cov) # assumed residual zero > mean > > normally distributed > > optimize.leastsq returns cov_x which would be the cov in this > calculation and needs the multiplication. > > optimize.curve_fit already does the multiplication with the sum > squared residual, so the cov that curve_fit returns is the actual > parameter covariance, which is the one I meant originally > > cf_params, cf_pcov = optimize.curve_fit(func0, x, y) > cf_bse = np.sqrt(np.diag(cf_pcov)) > Yes, just my poor attempt to make OP aware of this difference between leastsq and curve_fit. Thanks, eat > > Josef > > > >> > >> Regards, > >> eat > >>> > >>> (from scikits.statsmodels LikelihoodModel confint: ) > >>> lower = self.params[cols] - dist.ppf(1-\ > >>> alpha/2,self.model.df_resid) * bse[cols] > >>> upper = self.params[cols] + dist.ppf(1-\ > >>> alpha/2,self.model.df_resid) * bse[cols] > >>> > >>> there is also a scikits.statsmodels.miscmodels.NonlinearLS which is > >>> similar to scipy.optimize curvefit but a class with all regression > >>> results using the gradient information. Caution: doesn't have complete > >>> test suite yet, only partially tested on some examples. > >>> > >>> If you mean confidence interval for prediction, then I'm not > >>> completely sure the analogy to the linear models works unchanged. > >>> > >>> Josef > >>> _______________________________________________ > >>> SciPy-User mailing list > >>> SciPy-User at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/scipy-user > >> > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 28 08:42:24 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 08:42:24 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: Message-ID: On Thu, Apr 28, 2011 at 8:11 AM, Till Stensitzki wrote: > In ultrafast physics, we often use, that the ration of the reduced chi^2 > follows a ?F-Distribution. Most of the time, we are a fitting a sum-of- > exponentials model to our data, in which the covariance between the > parameters is quite hight. Using the cov-diagonal to calculate the > confidence interval underestimates the size and form of the interval, so we are > using a exhaustive search method. > > See > > http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1260379/?page=4 interesting, it sounds similar to profile log-likelihood, which I haven't quite figured out yet how to program efficiently, but I have never seen it in the least-squares context. In the profile loglikelihood discussion the emphasis is on the non-linearity and not so much on multi-collinearity. If there is strong correlation between parameter estimates then it should also blow up the covariance matrix (similar to a near singular Hessian) and the standard errors, I think. But I guess multi-collinearity can have an even more drastic effect in non-linear models than in the linear model. Josef > > for details. > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From bsouthey at gmail.com Thu Apr 28 09:46:46 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 28 Apr 2011 08:46:46 -0500 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: Message-ID: <4DB96FC6.4070605@gmail.com> On 04/28/2011 07:11 AM, Till Stensitzki wrote: > In ultrafast physics, we often use, that the ration of the reduced chi^2 > follows a F-Distribution. Most of the time, we are a fitting a sum-of- > exponentials model to our data, in which the covariance between the > parameters is quite hight. Using the cov-diagonal to calculate the > confidence interval underestimates the size and form of the interval, so we are > using a exhaustive search method. > > See > > http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1260379/?page=4 > > for details. > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user That is just because the confidence intervals require an assumption of linearized version of the nonlinear model when computing confidence intervals. So it will be wrong if when the model is non-linear - but gets better as this linearized form approximates the true model and correct if the model is linear. By exhaustive search I presume you mean boostrapping which is going to better provided that there are enough points, samples, method used and how appropriate the model actually is. Bruce From dima.osin at gmail.com Thu Apr 28 10:26:39 2011 From: dima.osin at gmail.com (dima osin) Date: Thu, 28 Apr 2011 10:26:39 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: <4DB96FC6.4070605@gmail.com> References: <4DB96FC6.4070605@gmail.com> Message-ID: Hello, everybody Thank you all for your replies and such a lively discussion. I did not expect so many people will respond. Special thanks to Gael, he gave me direct instructions with the bootstrapping method on stack overflow site, and also participated here. Thanks a lot. I have seen the bootstrapping method in action before: http://www.variousconsequences.com/2010/02/visualizing-confidence-intervals.html but I do not exactly understand it, and, hence, cannot use it. I trust though MathWorks, OriginLab and QtiPlot, they all use the same method to calculate the confidence interval. Here is a citation from the Mathworks help pages: ""The confidence bounds for fitted coefficients are given by C = b +/- t*sqrt(S), where b are the coefficients produced by the fit, t depends on the confidence level, and is computed using the inverse of Student's t cumulative distribution function, and S is a vector of the diagonal elements from the estimated covariance matrix of the coefficient estimates, (X^TX)^-1*s^2. In a linear fit, X is the design matrix, while for a nonlinear fit X is the Jacobian of the fitted values with respect to the coefficients. X^T is the transpose of X, and s^2 is the mean squared error."" So, to narrow down the question, I know how to calculate the S matrix, but I do not know how to calculate t. If you know how to do it, please, advice. If the answer was given already here in the mail-list, please, point me to there. Thanks a lot. Dima -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 28 10:30:37 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 10:30:37 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: <4DB96FC6.4070605@gmail.com> Message-ID: On Thu, Apr 28, 2011 at 10:26 AM, dima osin wrote: > Hello, everybody > > Thank you all for your replies and such a lively discussion. I did not > expect so many people will respond. > Special thanks to Gael, he gave me direct instructions with the > bootstrapping method on stack > overflow site, and also participated here. Thanks a lot. > I have seen the bootstrapping method in action before: > http://www.variousconsequences.com/2010/02/visualizing-confidence-intervals.html > but I do not exactly understand it, and, hence, cannot use it. > I trust though MathWorks, OriginLab and QtiPlot, they all use the same > method to calculate the confidence interval. > Here is a citation from the Mathworks help pages: > ""The confidence bounds for fitted coefficients are given by > C = b +/- t*sqrt(S), > > where b are the coefficients produced by the fit, t depends on the > confidence level, and is computed using the inverse of Student's t > cumulative distribution function, and S is a vector of the diagonal elements > from the estimated covariance matrix of the coefficient estimates, > (X^TX)^-1*s^2. In a linear fit, X is the design matrix, while for a > nonlinear fit X is the Jacobian of the fitted values with respect to the > coefficients. X^T is the transpose of X, and s^2 is the mean squared > error."" > > So, to narrow down the question, I know how to calculate the S matrix, but I > do not know how to calculate t. > If you know how to do it, please, advice. If the answer was given already > here in the mail-list, please, point me to there. my first reply, scipy.stats.t.ppf Josef > Thanks a lot. > Dima > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From mail.till at gmx.de Thu Apr 28 11:31:01 2011 From: mail.till at gmx.de (Till Stensitzki) Date: Thu, 28 Apr 2011 15:31:01 +0000 (UTC) Subject: [SciPy-User] confidence interval for leastsq fit References: <4DB96FC6.4070605@gmail.com> Message-ID: > gets better as this linearized form approximates the true model and > correct if the model is linear. True, but in some cases the approximation is quite bad, e.g. the mentioned sum of exponentials. > By exhaustive search I presume you mean > boostrapping which is going to better provided that there are enough > points, samples, method used and how appropriate the model actually is. Bootstrapping is to take random samples and compare the results of fitting the samples? Nope, by exhaustive search i am the procedure described by the linked paper. You change the optimal parameter by a small margin, repeat the fit with the one parameter hold and than compare the chi^2 of this fit with the chi^2 of the optimal fit. Till From bsouthey at gmail.com Thu Apr 28 12:01:52 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 28 Apr 2011 11:01:52 -0500 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: <4DB96FC6.4070605@gmail.com> Message-ID: <4DB98F70.9060000@gmail.com> On 04/28/2011 10:31 AM, Till Stensitzki wrote: >> gets better as this linearized form approximates the true model and >> correct if the model is linear. > True, but in some cases the approximation is quite bad, > e.g. the mentioned sum of exponentials. > >> By exhaustive search I presume you mean >> boostrapping which is going to better provided that there are enough >> points, samples, method used and how appropriate the model actually is. > Bootstrapping is to take random samples and compare the results of > fitting the samples? Nope, by exhaustive search i am the procedure > described by the linked paper. You change the optimal parameter > by a small margin, repeat the fit with the one parameter hold and > than compare the chi^2 of this fit with the chi^2 of the optimal fit. > > Till > If you can estimate this fixed parameter, then you really need to include in your model. That type of 'messing about' does not provide the correct values so of course your confidence intervals will be 'wrong'. Wrong because the variances and covariances are *conditional* on the fixed value and do not account for the uncertainty in estimating the 'fixed' parameter. If you can not estimate that parameter, then 'whatever' you is rather meaningless because there are an infinite number of solutions. This is not a problem if your desired function is estimable (as in analysis variance with dummy variables) where you estimate the differences not actual values). Otherwise your answer will depend on the constraint imposed so if the constraint is wrong so will be the answer. Bruce From josef.pktd at gmail.com Thu Apr 28 12:16:52 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 12:16:52 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: <4DB98F70.9060000@gmail.com> References: <4DB96FC6.4070605@gmail.com> <4DB98F70.9060000@gmail.com> Message-ID: On Thu, Apr 28, 2011 at 12:01 PM, Bruce Southey wrote: > On 04/28/2011 10:31 AM, Till Stensitzki wrote: >>> gets better as this linearized form approximates the true model and >>> correct if the model is linear. >> True, but in some cases the approximation is quite bad, >> e.g. the mentioned sum of exponentials. >> >>> By exhaustive search I presume you mean >>> boostrapping which is going to better provided that there are enough >>> points, samples, method used and how appropriate the model actually is. >> Bootstrapping is to take random samples and compare the results of >> fitting the samples? Nope, by exhaustive search i am the procedure >> described by the linked paper. You change the optimal parameter >> by a small margin, repeat the fit with the one parameter hold and >> than compare the chi^2 of this fit with the chi^2 of the optimal fit. >> >> Till >> > If you can estimate this fixed parameter, then you really need to > include in your model. That type of 'messing about' does not provide the > correct values so of course your confidence intervals will be 'wrong'. > Wrong because the variances and covariances are *conditional* on the > fixed value and do not account for the uncertainty in estimating the > 'fixed' parameter. > > If you can not estimate that parameter, then 'whatever' you is rather > meaningless because there are an infinite number of solutions. This is > not a problem if your desired function is estimable (as in analysis > variance with dummy variables) where you estimate the differences not > actual values). Otherwise your answer will depend on the constraint > imposed so if the constraint is wrong so will be the answer. The approach sounds to me like the same principle as behind profile likelihood, which is just a way to trace the likelihood contour. e.g. http://support.sas.com/documentation/cdl/en/imlug/59656/HTML/default/nonlinearoptexpls_sect19.htm# My guess is under the assumption of a normal additive error, using the F-statistic or the loglikelihood level or ratio might be pretty much equivalent. Josef > > Bruce > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From mail.till at gmx.de Thu Apr 28 12:23:41 2011 From: mail.till at gmx.de (Till Stensitzki) Date: Thu, 28 Apr 2011 16:23:41 +0000 (UTC) Subject: [SciPy-User] confidence interval for leastsq fit References: <4DB96FC6.4070605@gmail.com> <4DB98F70.9060000@gmail.com> Message-ID: > If you can estimate this fixed parameter, then you really need to > include in your model. That type of 'messing about' does not provide the > correct values so of course your confidence intervals will be 'wrong'. > Wrong because the variances and covariances are *conditional* on the > fixed value and do not account for the uncertainty in estimating the > 'fixed' parameter. Please read the reference, its only half a page long, maybe my explanation weren't clear enough. The principal of method is, that as long the chi^2 values of a parameter set are not significantly different from the chi^2 value of the optimal set (in a statistical sense, can be estimated by using the fisher-distribution.) the parameters are "valid". From bsouthey at gmail.com Thu Apr 28 12:42:13 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 28 Apr 2011 11:42:13 -0500 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: <4DB96FC6.4070605@gmail.com> <4DB98F70.9060000@gmail.com> Message-ID: <4DB998E5.9010505@gmail.com> On 04/28/2011 11:16 AM, josef.pktd at gmail.com wrote: > On Thu, Apr 28, 2011 at 12:01 PM, Bruce Southey wrote: >> On 04/28/2011 10:31 AM, Till Stensitzki wrote: >>>> gets better as this linearized form approximates the true model and >>>> correct if the model is linear. >>> True, but in some cases the approximation is quite bad, >>> e.g. the mentioned sum of exponentials. >>> >>>> By exhaustive search I presume you mean >>>> boostrapping which is going to better provided that there are enough >>>> points, samples, method used and how appropriate the model actually is. >>> Bootstrapping is to take random samples and compare the results of >>> fitting the samples? Nope, by exhaustive search i am the procedure >>> described by the linked paper. You change the optimal parameter >>> by a small margin, repeat the fit with the one parameter hold and >>> than compare the chi^2 of this fit with the chi^2 of the optimal fit. >>> >>> Till >>> >> If you can estimate this fixed parameter, then you really need to >> include in your model. That type of 'messing about' does not provide the >> correct values so of course your confidence intervals will be 'wrong'. >> Wrong because the variances and covariances are *conditional* on the >> fixed value and do not account for the uncertainty in estimating the >> 'fixed' parameter. >> >> If you can not estimate that parameter, then 'whatever' you is rather >> meaningless because there are an infinite number of solutions. This is >> not a problem if your desired function is estimable (as in analysis >> variance with dummy variables) where you estimate the differences not >> actual values). Otherwise your answer will depend on the constraint >> imposed so if the constraint is wrong so will be the answer. > The approach sounds to me like the same principle as behind profile > likelihood, which is just a way to trace the likelihood contour. > e.g. > http://support.sas.com/documentation/cdl/en/imlug/59656/HTML/default/nonlinearoptexpls_sect19.htm# > > My guess is under the assumption of a normal additive error, using the > F-statistic or the loglikelihood level or ratio might be pretty much > equivalent. > > Josef > > Perhaps the 'decidedly inferior' marginal profile confidence intervals is closer as you fix all but one parameter and compare that to the full likelihood. http://www.unc.edu/courses/2010fall/ecol/563/001/docs/lectures/lecture11.htm#marginal It also relies rather heavily on the assumption that the likelihood ratio is chi-squared. Bruce From josef.pktd at gmail.com Thu Apr 28 12:56:13 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 12:56:13 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: <4DB998E5.9010505@gmail.com> References: <4DB96FC6.4070605@gmail.com> <4DB98F70.9060000@gmail.com> <4DB998E5.9010505@gmail.com> Message-ID: On Thu, Apr 28, 2011 at 12:42 PM, Bruce Southey wrote: > On 04/28/2011 11:16 AM, josef.pktd at gmail.com wrote: >> On Thu, Apr 28, 2011 at 12:01 PM, Bruce Southey ?wrote: >>> On 04/28/2011 10:31 AM, Till Stensitzki wrote: >>>>> gets better as this linearized form approximates the true model and >>>>> correct if the model is linear. >>>> True, but in some cases the approximation is quite bad, >>>> e.g. the mentioned sum of exponentials. >>>> >>>>> By exhaustive search I presume you mean >>>>> boostrapping which is going to better provided that there are enough >>>>> points, samples, method used and how appropriate the model actually is. >>>> Bootstrapping is to take random samples and compare the results of >>>> fitting the samples? Nope, by exhaustive search i am the procedure >>>> described by the linked paper. You change the optimal parameter >>>> by a small margin, repeat the fit with the one parameter hold and >>>> than compare the chi^2 of this fit with the chi^2 of the optimal fit. >>>> >>>> Till >>>> >>> If you can estimate this fixed parameter, then you really need to >>> include in your model. That type of 'messing about' does not provide the >>> correct values so of course your confidence intervals will be 'wrong'. >>> Wrong because the variances and covariances are *conditional* on the >>> fixed value and do not account for the uncertainty in estimating the >>> 'fixed' parameter. >>> >>> If you can not estimate that parameter, then 'whatever' you is rather >>> meaningless because there are an infinite number of solutions. This is >>> not a problem if your desired function is estimable (as in analysis >>> variance with dummy variables) where you estimate the differences not >>> actual values). Otherwise your answer will depend on the constraint >>> imposed so if the constraint is wrong so will be the answer. >> The approach sounds to me like the same principle as behind profile >> likelihood, which is just a way to trace the likelihood contour. >> e.g. >> http://support.sas.com/documentation/cdl/en/imlug/59656/HTML/default/nonlinearoptexpls_sect19.htm# >> >> My guess is under the assumption of a normal additive error, using the >> F-statistic or the loglikelihood level or ratio might be pretty much >> equivalent. >> >> Josef >> >> > Perhaps the 'decidedly inferior' marginal profile confidence intervals > is closer as you fix all but one parameter and compare that to the full > likelihood. > http://www.unc.edu/courses/2010fall/ecol/563/001/docs/lectures/lecture11.htm#marginal > > It also relies rather heavily on the assumption that the likelihood > ratio is chi-squared. from Tills reference """ In the exhaustive search procedure the rate constant ki was changed (increased or decreased) from its optimal value by a certain fraction. Subsequently, a new minimization of x2 was performed in which ki is fixed, whereas all other fit parameters were allowed to relax, in order to find a new minimum on the reduced x2-surface. The rate constant ki was thus increased/decreased stepwise until the new minimal value for the reduced x2 was significantly worse than X2 . This procedure mapped the complete x2-surface around Xmin It provided error ranges for each rate constant, at a given statistical accuracy. """ hold one parameter fixed, maximize chi2, or loglikelihood with respect to all other parameters (same idea as concentrated likelihood) Josef > > Bruce > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From bsouthey at gmail.com Thu Apr 28 12:55:04 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 28 Apr 2011 11:55:04 -0500 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: <4DB96FC6.4070605@gmail.com> <4DB98F70.9060000@gmail.com> <4DB998E5.9010505@gmail.com> Message-ID: <4DB99BE8.8060203@gmail.com> On 04/28/2011 11:56 AM, josef.pktd at gmail.com wrote: > On Thu, Apr 28, 2011 at 12:42 PM, Bruce Southey wrote: >> On 04/28/2011 11:16 AM, josef.pktd at gmail.com wrote: >>> On Thu, Apr 28, 2011 at 12:01 PM, Bruce Southey wrote: >>>> On 04/28/2011 10:31 AM, Till Stensitzki wrote: >>>>>> gets better as this linearized form approximates the true model and >>>>>> correct if the model is linear. >>>>> True, but in some cases the approximation is quite bad, >>>>> e.g. the mentioned sum of exponentials. >>>>> >>>>>> By exhaustive search I presume you mean >>>>>> boostrapping which is going to better provided that there are enough >>>>>> points, samples, method used and how appropriate the model actually is. >>>>> Bootstrapping is to take random samples and compare the results of >>>>> fitting the samples? Nope, by exhaustive search i am the procedure >>>>> described by the linked paper. You change the optimal parameter >>>>> by a small margin, repeat the fit with the one parameter hold and >>>>> than compare the chi^2 of this fit with the chi^2 of the optimal fit. >>>>> >>>>> Till >>>>> >>>> If you can estimate this fixed parameter, then you really need to >>>> include in your model. That type of 'messing about' does not provide the >>>> correct values so of course your confidence intervals will be 'wrong'. >>>> Wrong because the variances and covariances are *conditional* on the >>>> fixed value and do not account for the uncertainty in estimating the >>>> 'fixed' parameter. >>>> >>>> If you can not estimate that parameter, then 'whatever' you is rather >>>> meaningless because there are an infinite number of solutions. This is >>>> not a problem if your desired function is estimable (as in analysis >>>> variance with dummy variables) where you estimate the differences not >>>> actual values). Otherwise your answer will depend on the constraint >>>> imposed so if the constraint is wrong so will be the answer. >>> The approach sounds to me like the same principle as behind profile >>> likelihood, which is just a way to trace the likelihood contour. >>> e.g. >>> http://support.sas.com/documentation/cdl/en/imlug/59656/HTML/default/nonlinearoptexpls_sect19.htm# >>> >>> My guess is under the assumption of a normal additive error, using the >>> F-statistic or the loglikelihood level or ratio might be pretty much >>> equivalent. >>> >>> Josef >>> >>> >> Perhaps the 'decidedly inferior' marginal profile confidence intervals >> is closer as you fix all but one parameter and compare that to the full >> likelihood. >> http://www.unc.edu/courses/2010fall/ecol/563/001/docs/lectures/lecture11.htm#marginal >> >> It also relies rather heavily on the assumption that the likelihood >> ratio is chi-squared. > from Tills reference > """ > In the exhaustive search procedure the rate constant ki was changed > (increased or decreased) from its optimal value by a certain fraction. > Subsequently, a new minimization of x2 was performed in which ki is > fixed, whereas all other fit parameters were allowed to relax, in order > to find a new minimum on the reduced x2-surface. The rate constant ki > was thus increased/decreased stepwise until the new minimal value for > the reduced x2 was significantly worse than X2 . This procedure > mapped the complete x2-surface around Xmin It provided error ranges > for each rate constant, at a given statistical accuracy. > """ > > hold one parameter fixed, maximize chi2, or loglikelihood with respect > to all other parameters > (same idea as concentrated likelihood) > > Josef > > >> Bruce >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user Are we talking confidence intervals (OP) or parameter estimation (essentially a grid search)? (I have nothing more to say on the latter as it still holds.) Bruce From josef.pktd at gmail.com Thu Apr 28 13:14:41 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 13:14:41 -0400 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: <4DB99BE8.8060203@gmail.com> References: <4DB96FC6.4070605@gmail.com> <4DB98F70.9060000@gmail.com> <4DB998E5.9010505@gmail.com> <4DB99BE8.8060203@gmail.com> Message-ID: On Thu, Apr 28, 2011 at 12:55 PM, Bruce Southey wrote: > On 04/28/2011 11:56 AM, josef.pktd at gmail.com wrote: >> On Thu, Apr 28, 2011 at 12:42 PM, Bruce Southey ?wrote: >>> On 04/28/2011 11:16 AM, josef.pktd at gmail.com wrote: >>>> On Thu, Apr 28, 2011 at 12:01 PM, Bruce Southey ? ?wrote: >>>>> On 04/28/2011 10:31 AM, Till Stensitzki wrote: >>>>>>> gets better as this linearized form approximates the true model and >>>>>>> correct if the model is linear. >>>>>> True, but in some cases the approximation is quite bad, >>>>>> e.g. the mentioned sum of exponentials. >>>>>> >>>>>>> By exhaustive search I presume you mean >>>>>>> boostrapping which is going to better provided that there are enough >>>>>>> points, samples, method used and how appropriate the model actually is. >>>>>> Bootstrapping is to take random samples and compare the results of >>>>>> fitting the samples? Nope, by exhaustive search i am the procedure >>>>>> described by the linked paper. You change the optimal parameter >>>>>> by a small margin, repeat the fit with the one parameter hold and >>>>>> than compare the chi^2 of this fit with the chi^2 of the optimal fit. >>>>>> >>>>>> Till >>>>>> >>>>> If you can estimate this fixed parameter, then you really need to >>>>> include in your model. That type of 'messing about' does not provide the >>>>> correct values so of course your confidence intervals will be 'wrong'. >>>>> Wrong because the variances and covariances are *conditional* on the >>>>> fixed value and do not account for the uncertainty in estimating the >>>>> 'fixed' parameter. >>>>> >>>>> If you can not estimate that parameter, then 'whatever' you is rather >>>>> meaningless because there are an infinite number of solutions. This is >>>>> not a problem if your desired function is estimable (as in analysis >>>>> variance with dummy variables) where you estimate the differences not >>>>> actual values). Otherwise your answer will depend on the constraint >>>>> imposed so if the constraint is wrong so will be the answer. >>>> The approach sounds to me like the same principle as behind profile >>>> likelihood, which is just a way to trace the likelihood contour. >>>> e.g. >>>> http://support.sas.com/documentation/cdl/en/imlug/59656/HTML/default/nonlinearoptexpls_sect19.htm# >>>> >>>> My guess is under the assumption of a normal additive error, using the >>>> F-statistic or the loglikelihood level or ratio might be pretty much >>>> equivalent. >>>> >>>> Josef >>>> >>>> >>> Perhaps the 'decidedly inferior' marginal profile confidence intervals >>> is closer as you fix all but one parameter and compare that to the full >>> likelihood. >>> http://www.unc.edu/courses/2010fall/ecol/563/001/docs/lectures/lecture11.htm#marginal >>> >>> It also relies rather heavily on the assumption that the likelihood >>> ratio is chi-squared. >> from Tills reference >> """ >> In the exhaustive search procedure the rate constant ki was changed >> (increased or decreased) from its optimal value by a certain fraction. >> Subsequently, a new minimization of x2 was performed in which ki is >> fixed, whereas all other fit parameters were allowed to relax, in order >> to find a new minimum on the reduced x2-surface. The rate constant ki >> was thus increased/decreased stepwise until the new minimal value for >> the reduced x2 was significantly worse than X2 . This procedure >> mapped the complete x2-surface around Xmin It provided error ranges >> for each rate constant, at a given statistical accuracy. >> """ >> >> hold one parameter fixed, maximize chi2, or loglikelihood with respect >> to all other parameters >> (same idea as concentrated likelihood) >> >> Josef >> >> >>> Bruce >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > Are we talking confidence intervals (OP) or parameter estimation > (essentially a grid search)? > (I have nothing more to say on the latter as it still holds.) the constraint parameter estimation is just a tool to trace the log-likelihood, the purpose is to construct a confidence interval that takes the non-linearity or non-normality (for example in generalized linear model) into account. Per Brodtkorb has it implemented in his version of stats.distributions (which got me started to look into this). Since the local linear approximation in the standard parameter variance version might not be very good for larger non-linear changes, tracing the loglikelihood should give better confidence intervals. >From the description of one implementation in R, they use a grid to calculate the loglikelihood at different points (holding one parameter constant), approximate it with a polynomial or a spline to get an estimate of the (profile) loglikelihood, which is then inverted for given Likelihood-Ratio or F-test critical value. This way they get the extreme points of the parameter estimates for which the LR-test is not rejected. >From what I have seen, the theory is pretty clear if there is only 1 parameter of interest and the others are considered nuisance parameters. But the explanations usually contain "the profile likelihood is not a true likelihood" function, and I'm not quite sure what the interpretation should be if there are many parameters of interest. Josef > > Bruce > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From gael.varoquaux at normalesup.org Thu Apr 28 13:23:53 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 28 Apr 2011 19:23:53 +0200 Subject: [SciPy-User] confidence interval for leastsq fit In-Reply-To: References: <20110428100011.GD17045@phare.normalesup.org> Message-ID: <20110428172353.GC32604@phare.normalesup.org> On Thu, Apr 28, 2011 at 06:18:45AM -0400, josef.pktd at gmail.com wrote: > given the question is about optimize.leastsq, I assume this applies : > http://en.wikipedia.org/wiki/Non-linear_least_squares#Parameter_errors.2C_confidence_limits.2C_residuals_etc. > non-linear model with additive error y = f(x, params) + err > function f is differentiable. > minimize quadratic loss > parameter is in the interior of the parameter space if there are bounds > all regression results of the linear model apply based on local > (derivatives), asymptotic (normality) distribution using the Jacobian > instead of the design matrix. Correct but a few caveats: - These are asymptotic results: you need a lot of data - The Jacobian should be non singular - One should be sure that that there is only one minimum, or that the optimizer always falls in the same minimum G From eg at fft.be Thu Apr 28 15:33:27 2011 From: eg at fft.be (Eloi Gaudry) Date: Thu, 28 Apr 2011 21:33:27 +0200 Subject: [SciPy-User] [msvc] scipy/numpy build with visual studio and intel fortran compiler Message-ID: <4DB9C107.1090709@fft.be> hi, i'd like to discuss these two patches with people that needs to build current/recent versions of numpy and scipy using Visual Studio (200x) and the Intel Visual Fortran compiler. so far, i've been able to build numpy and i've facing one remaining issue with the scipy/special/cephes module which seems to occur because i'm using shared and static libraries. C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:c:\ fft\bin\python-2.6.6\libs /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LIBPATH:build\temp .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib mach.lib /EXPORT:init_quadpack build \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj /OUT:build\lib.win32-2.6\scipy\integrate \_quadpack.pyd /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib /MANIFESTFILE:buil d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:c:\ fft\bin\python-2.6.6\libs /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LIBPATH:build\temp .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib mach.lib /EXPORT:init_quadpack build \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj /OUT:build\lib.win32-2.6\scipy\integrate \_quadpack.pyd /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib /MANIFESTFILE:buil d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest boom ! 1120 LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already defined in MSVCRT.lib(MSVCR80.dll ) LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already defined in MSVCRT.lib(MSVCR80.dll ) LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in MSVCRT.lib(cinitexe.obj) LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in MSVCRT.lib(cinitexe.obj) LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in MSVCRT.lib(cinitexe.obj) LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in MSVCRT.lib(cinitexe.obj) LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already defined in MSVCRT.lib(MSVCR80.dll ) LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in MSVCRT.lib(MSVCR80.dll) Creating library build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object build\temp .win32-2.6\Release\scipy\integrate\_quadpack.exp LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:libr ary LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:libr ary libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external symbol _MAIN__ referenced in func tion _main build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error LNK1120: 1 unresolved externals LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already defined in MSVCRT.lib(MSVCR80.dll ) LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already defined in MSVCRT.lib(MSVCR80.dll ) LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in MSVCRT.lib(cinitexe.obj) LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in MSVCRT.lib(cinitexe.obj) LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in MSVCRT.lib(cinitexe.obj) LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in MSVCRT.lib(cinitexe.obj) LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already defined in MSVCRT.lib(MSVCR80.dll ) LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in MSVCRT.lib(MSVCR80.dll) LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in MSVCRT.lib(MSVCR80.dll) Creating library build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object build\temp .win32-2.6\Release\scipy\integrate\_quadpack.exp LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:libr ary LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:libr ary libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external symbol _MAIN__ referenced in func tion _main build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error LNK1120: 1 unresolved externals error: Command "C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL /nologo /INCREMENTAL :NO /LIBPATH:c:\fft\bin\python-2.6.6\libs /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LI BPATH:build\temp.win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs /LIBPATH:c:\fft\bin\python-2.6.6\PC \VS8.0\win32release /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib mach.lib /EXPORT:ini t_quadpack build\temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj /OUT:build\lib.win32-2.6 \scipy\integrate\_quadpack.pyd /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib /M ANIFESTFILE:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest" failed with exit st atus 1120 thanks, ?loi -- Eloi Gaudry Senior Product Development Engineer Free Field Technologies Company Website: http://www.fft.be -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.numpy-1.5.0.msvc.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.scipy-0.9.0.msvc.diff URL: From ralf.gommers at googlemail.com Thu Apr 28 16:22:49 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 28 Apr 2011 22:22:49 +0200 Subject: [SciPy-User] SciPy 0.9 install: _fitpack.so: undefined symbol: jwe_ilst In-Reply-To: <87hb9jz8i6.fsf@inspiron.ap.columbia.edu> References: <87hb9jz8i6.fsf@inspiron.ap.columbia.edu> Message-ID: On Wed, Apr 27, 2011 at 11:31 PM, Nikolaus Rath wrote: > Hello, > > I'm trying to build Scipy 0.9 from source. The installation with > "setup.py install" seemed to work just fine, but when I try to.. > > In [1]: import scipy.interpolate > > ..then I get... > > ImportError: /home/byrne/.local/lib/python2.6/site-packages/scipy/interpolate/_fitpack.so: undefined symbol: jwe_ilst > > Any suggestions what may have gone wrong and how I could fix it? Very little turns up when googling for "jwe_ilst", the only perhaps related useful info is http://www.laheyforum.com/showthread.php?t=10777. Are you also using the Lahey/Fujitsu Fortran compiler, or perhaps a LAPACK built with it? Complete info on your OS and compilers would be helpful. Ralf From ralf.gommers at googlemail.com Thu Apr 28 16:32:33 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 28 Apr 2011 22:32:33 +0200 Subject: [SciPy-User] Unable to build scipy with python3.2 In-Reply-To: References: Message-ID: 2011/4/24 ?ngel Vel?squez : > Hi, > > Let me make a short introduction, my name is Angel Vel?squez from > Argentina, I am the maintainer of scipy on Arch Linux distribution. Hi Angel. That doesn't look familiar, can you send us your site.cfg and the paths to your BLAS/LAPACK? > I am having this issue [1] > > I've tried several options (creating a site.cfg with [blas]) export > the BLAS variable, etc, but I unfortunately can make it. > > I'm following these steps: > > $ python setup.py config_fc --fcompiler=gnu95 build Do you get the same result when replacing build by install here? Cheers, Ralf > > The breakage comes in the first step, so I cannot do the following step: > > # python setup.py config_fc --fcompiler=gnu95 install --prefix=/usr --optimize=1 > > I was able to package scipy with python 2.7 in the same enviroment but > not with python 3.2 > > Any help is appreciated, thanks in advance > > [1] http://dpaste.com/535077/ > > -- > Angel Vel?squez > angvp @ irc.freenode.net > Arch Linux Developer / Trusted User > Linux Counter: #359909 > http://www.angvp.com > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From kwgoodman at gmail.com Thu Apr 28 16:52:39 2011 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 28 Apr 2011 13:52:39 -0700 Subject: [SciPy-User] [ANN] la 0.5.0, labeled array Message-ID: The fifth release of la is faster and adds fast, moving window methods to larry. The cost of the speed and new functionality is that la now requires the Bottleneck package (http://pypi.python.org/pypi/Bottleneck). The main class of the la package is a labeled array, larry. A larry consists of data and labels. The data is stored as a NumPy array and the labels as a list of lists (one list per dimension). Faster - sum, mean, std, var, min, max, median, ranking - correlation, group_median, demean, demedian, zscore Moving window - fast (Bottleneck): move_sum, move_mean, move_std, move_min, move_max - slow (Python): move_ranking, move_median, move_func New methods and functions - la.data.yahoo.quotes(): Given a ticker sequence, return historical Yahoo! quotes (open, close, high, low, volume) as a 3d larry - Numpy array functions that ignore NaNs: demean, demedian, and zscore - la.unique(): Find the unique elements of a larry - larry.tofile(): Save 1d or 2d larry to text file Enhancements - Add optional `ddof` input parameter to larry.std() and larry.var() - Cut import time by 15% Breakage from la 0.4 - Bottleneck is now a dependency of la - ranking() and group_rank() no longer take a `ties` input parameter - movingsum no longer treats Inf and -Inf as missing values - movingsum and movingrank have been deprecated and will be removed in la 0.6; use move_sum and move_ranking instead Bugs fixes - Please report bugs at https://github.com/kwgoodman/la/issues - #1 Due to a typo, la.info() crashed if h5py could not be imported - #2 larry.sortaxis(None) chopped off singleton dimensions - #5 la.farray.lastrank() choked on empty array input - #7 larry.quantile() choked on axis=None - #8 demean, demedian, zscore choked on 1d input when axis=-1 - #9 cross_validation docstring refers to old name of function (cv) - #10 Unit tests: "Warning: invalid value encountered in..." URLs download http://pypi.python.org/pypi/la docs http://berkeleyanalytics.com/la code https://github.com/kwgoodman/la mailing list http://groups.google.com/group/labeled-array issue tracker https://github.com/kwgoodman/la/issues From eg at fft.be Thu Apr 28 17:26:18 2011 From: eg at fft.be (Eloi Gaudry) Date: Thu, 28 Apr 2011 23:26:18 +0200 Subject: [SciPy-User] [msvc] scipy/numpy build with visual studio and intel fortran compiler In-Reply-To: <4DB9C107.1090709@fft.be> References: <4DB9C107.1090709@fft.be> Message-ID: <4DB9DB7A.4080907@fft.be> well, i somehow messed up with my setup. using the enclosed patches, i can build numpy and scipy using msvc8/9 and intel visual fortran compiler 11.1 (you need to apply the patches and use the matching build_*msvc.x*.bat file). 'hope this would help people that need to stick to visual studio compilers... ?loi On 28/04/2011 21:33, Eloi Gaudry wrote: > hi, > > i'd like to discuss these two patches with people that needs to build > current/recent versions of numpy and scipy using Visual Studio (200x) > and the Intel Visual Fortran compiler. > > so far, i've been able to build numpy and i've facing one remaining > issue with the scipy/special/cephes module which seems to occur > because i'm using shared and static libraries. > > C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL > /nologo /INCREMENTAL:NO /LIBPATH:c:\ > fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release > /LIBPATH:build\temp > .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele > ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib > mach.lib /EXPORT:init_quadpack build > \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj > /OUT:build\lib.win32-2.6\scipy\integrate > \_quadpack.pyd > /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib > /MANIFESTFILE:buil > d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest > C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL > /nologo /INCREMENTAL:NO /LIBPATH:c:\ > fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release > /LIBPATH:build\temp > .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele > ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib > mach.lib /EXPORT:init_quadpack build > \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj > /OUT:build\lib.win32-2.6\scipy\integrate > \_quadpack.pyd > /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib > /MANIFESTFILE:buil > d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest > > boom ! > 1120 > LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined > in MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined > in MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already > defined in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already > defined in MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already > defined in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already > defined in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined > in MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in > MSVCRT.lib(MSVCR80.dll) > Creating library > build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object > build\temp > .win32-2.6\Release\scipy\integrate\_quadpack.exp > LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of > other libs; use /NODEFAULTLIB:libr > ary > LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of > other libs; use /NODEFAULTLIB:libr > ary > libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external > symbol _MAIN__ referenced in func > tion _main > build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error > LNK1120: 1 unresolved externals > > LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined > in MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined > in MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already > defined in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already > defined in MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already > defined in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already > defined in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined > in MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in > MSVCRT.lib(MSVCR80.dll) > Creating library > build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object > build\temp > .win32-2.6\Release\scipy\integrate\_quadpack.exp > LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of > other libs; use /NODEFAULTLIB:libr > ary > LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of > other libs; use /NODEFAULTLIB:libr > ary > libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external > symbol _MAIN__ referenced in func > tion _main > build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error > LNK1120: 1 unresolved externals > error: Command "C:\Program Files\Microsoft Visual Studio > 8\VC\BIN\link.exe /DLL /nologo /INCREMENTAL > :NO /LIBPATH:c:\fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LI > BPATH:build\temp.win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC > \VS8.0\win32release /LIBPATH:build\temp.win32-2.6 quadpack.lib > linpack_lite.lib mach.lib /EXPORT:ini > t_quadpack > build\temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj > /OUT:build\lib.win32-2.6 > \scipy\integrate\_quadpack.pyd > /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib /M > ANIFESTFILE:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest" > failed with exit st > atus 1120 > > thanks, > ?loi > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Eloi Gaudry Senior Product Development Engineer Free Field Technologies Company Website: http://www.fft.be Direct Phone Number: +32 10 495 147 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.numpy-1.5.0.msvc.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.scipy-0.9.0.msvc.diff URL: From ralf.gommers at googlemail.com Thu Apr 28 17:38:04 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 28 Apr 2011 23:38:04 +0200 Subject: [SciPy-User] [msvc] scipy/numpy build with visual studio and intel fortran compiler In-Reply-To: <4DB9DB7A.4080907@fft.be> References: <4DB9C107.1090709@fft.be> <4DB9DB7A.4080907@fft.be> Message-ID: On Thu, Apr 28, 2011 at 11:26 PM, Eloi Gaudry wrote: > well, i somehow messed up with my setup. using the enclosed patches, i can > build numpy and scipy using msvc8/9 and intel visual fortran compiler 11.1 > (you need to apply the patches and use the matching build_*msvc.x*.bat > file). > 'hope this would help people that need to stick to visual studio > compilers... For your numpy patch: - The version string detection is already fixed in master. - The CPU detection you changed confuses me a bit. It looks like you are running 32-bit Windows on PPC64? - No idea why those f2py changes would be necessary either. For your scipy patch: - optimization level in distutils -O1 is already fixed in master - why are all those defines in __fitpack.h, __quadpack.h and __minpack.h needed? Ralf > On 28/04/2011 21:33, Eloi Gaudry wrote: > > hi, > > i'd like to discuss these two patches with people that needs to build > current/recent versions of numpy and scipy using Visual Studio (200x) and > the Intel Visual Fortran compiler. > > so far, i've been able to build numpy and i've facing one remaining issue > with the scipy/special/cephes module which seems to occur because i'm using > shared and static libraries. > > C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL /nologo > /INCREMENTAL:NO /LIBPATH:c:\ > fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LIBPATH:build\temp > .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele > ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib mach.lib > /EXPORT:init_quadpack build > \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj > /OUT:build\lib.win32-2.6\scipy\integrate > \_quadpack.pyd > /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib > /MANIFESTFILE:buil > d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest > C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL /nologo > /INCREMENTAL:NO /LIBPATH:c:\ > fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LIBPATH:build\temp > .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele > ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib mach.lib > /EXPORT:init_quadpack build > \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj > /OUT:build\lib.win32-2.6\scipy\integrate > \_quadpack.pyd > /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib > /MANIFESTFILE:buil > d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest > > boom ! > 1120 > LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already defined > in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already defined > in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already defined > in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in > MSVCRT.lib(MSVCR80.dll) > ?? Creating library > build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object > build\temp > .win32-2.6\Release\scipy\integrate\_quadpack.exp > LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other > libs; use /NODEFAULTLIB:libr > ary > LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other > libs; use /NODEFAULTLIB:libr > ary > libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external symbol > _MAIN__ referenced in func > tion _main > build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error LNK1120: 1 > unresolved externals > > LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already defined > in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already defined > in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in > MSVCRT.lib(cinitexe.obj) > LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already defined > in MSVCRT.lib(MSVCR80.dll > ) > LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in > MSVCRT.lib(MSVCR80.dll) > LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in > MSVCRT.lib(MSVCR80.dll) > ?? Creating library > build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object > build\temp > .win32-2.6\Release\scipy\integrate\_quadpack.exp > LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other > libs; use /NODEFAULTLIB:libr > ary > LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other > libs; use /NODEFAULTLIB:libr > ary > libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external symbol > _MAIN__ referenced in func > tion _main > build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error LNK1120: 1 > unresolved externals > error: Command "C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe > /DLL /nologo /INCREMENTAL > :NO /LIBPATH:c:\fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LI > BPATH:build\temp.win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs > /LIBPATH:c:\fft\bin\python-2.6.6\PC > \VS8.0\win32release /LIBPATH:build\temp.win32-2.6 quadpack.lib > linpack_lite.lib mach.lib /EXPORT:ini > t_quadpack build\temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj > /OUT:build\lib.win32-2.6 > \scipy\integrate\_quadpack.pyd > /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib /M > ANIFESTFILE:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest" > failed with exit st > atus 1120 > > thanks, > ?loi > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > -- > Eloi Gaudry > Senior Product Development Engineer > > Free Field Technologies > Company Website: http://www.fft.be > Direct Phone Number: +32 10 495 147 > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From cool-rr at cool-rr.com Thu Apr 28 17:46:41 2011 From: cool-rr at cool-rr.com (cool-RR) Date: Thu, 28 Apr 2011 17:46:41 -0400 Subject: [SciPy-User] Do people here still use Python 2.5? Message-ID: Hey everyone, I'm considering dropping Python 2.5 support in GarlicSim. This would solve a compatibility bug I have now with zipimports, and generally make me a happier person. Do you think that people have generally moved off Python 2.5 already? Python 2.6 was released two and a half years ago, but I'm not sure whether 2.5 is safe to ignore already. I understand that GAE still runs on 2.5, which is a huge bummer... Though 2.7 support seems to be in the "Features on Deck" category in their roadmap, so I hope they do it soon so 2.5 would die already. What do you think? Are there any people here who are forced to work with Python 2.5 for some reason? Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eg at fft.be Thu Apr 28 17:49:06 2011 From: eg at fft.be (Eloi Gaudry) Date: Thu, 28 Apr 2011 23:49:06 +0200 Subject: [SciPy-User] [msvc] scipy/numpy build with visual studio and intel fortran compiler In-Reply-To: References: <4DB9C107.1090709@fft.be> <4DB9DB7A.4080907@fft.be> Message-ID: <4DB9E0D2.2000501@fft.be> On 28/04/2011 23:38, Ralf Gommers wrote: > On Thu, Apr 28, 2011 at 11:26 PM, Eloi Gaudry wrote: >> well, i somehow messed up with my setup. using the enclosed patches, i can >> build numpy and scipy using msvc8/9 and intel visual fortran compiler 11.1 >> (you need to apply the patches and use the matching build_*msvc.x*.bat >> file). >> 'hope this would help people that need to stick to visual studio >> compilers... > For your numpy patch: > - The version string detection is already fixed in master. ok > - The CPU detection you changed confuses me a bit. It looks like you > are running 32-bit Windows on PPC64? sorry, that just another patch for aix using ppc64 processors (64-bits kernel). without, gcc will define NPY_CPU_PPC instead of NPY_CPU_PPC64 on this host : there is a token missing i guess. > - No idea why those f2py changes would be necessary either. without those changes, the F_FUNC_US and F_WRAPPEDFUNC* are not defined properly when using Intel Visual Fortran: as a result of the PREPEND_FORTRAN, NO_APPEND_FORTRAN, UPPERCASE_FORTRAN macros being not properly set. > For your scipy patch: > - optimization level in distutils -O1 is already fixed in master ok (i'd better have a look at the svn next time !) thanks. > - why are all those defines in __fitpack.h, __quadpack.h and __minpack.h needed? for the same reason as for the f2py changes. as for the longjmp, i should double-check that. thanks for your feedback ralf, i appreciate. > Ralf > > >> On 28/04/2011 21:33, Eloi Gaudry wrote: >> >> hi, >> >> i'd like to discuss these two patches with people that needs to build >> current/recent versions of numpy and scipy using Visual Studio (200x) and >> the Intel Visual Fortran compiler. >> >> so far, i've been able to build numpy and i've facing one remaining issue >> with the scipy/special/cephes module which seems to occur because i'm using >> shared and static libraries. >> >> C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL /nologo >> /INCREMENTAL:NO /LIBPATH:c:\ >> fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LIBPATH:build\temp >> .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele >> ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib mach.lib >> /EXPORT:init_quadpack build >> \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj >> /OUT:build\lib.win32-2.6\scipy\integrate >> \_quadpack.pyd >> /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib >> /MANIFESTFILE:buil >> d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest >> C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL /nologo >> /INCREMENTAL:NO /LIBPATH:c:\ >> fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LIBPATH:build\temp >> .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele >> ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib mach.lib >> /EXPORT:init_quadpack build >> \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj >> /OUT:build\lib.win32-2.6\scipy\integrate >> \_quadpack.pyd >> /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib >> /MANIFESTFILE:buil >> d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest >> >> boom ! >> 1120 >> LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already defined >> in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already defined >> in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already defined >> in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in >> MSVCRT.lib(MSVCR80.dll) >> Creating library >> build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object >> build\temp >> .win32-2.6\Release\scipy\integrate\_quadpack.exp >> LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other >> libs; use /NODEFAULTLIB:libr >> ary >> LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other >> libs; use /NODEFAULTLIB:libr >> ary >> libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external symbol >> _MAIN__ referenced in func >> tion _main >> build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error LNK1120: 1 >> unresolved externals >> >> LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already defined >> in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already defined >> in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already defined >> in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in >> MSVCRT.lib(MSVCR80.dll) >> Creating library >> build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object >> build\temp >> .win32-2.6\Release\scipy\integrate\_quadpack.exp >> LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other >> libs; use /NODEFAULTLIB:libr >> ary >> LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other >> libs; use /NODEFAULTLIB:libr >> ary >> libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external symbol >> _MAIN__ referenced in func >> tion _main >> build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error LNK1120: 1 >> unresolved externals >> error: Command "C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe >> /DLL /nologo /INCREMENTAL >> :NO /LIBPATH:c:\fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LI >> BPATH:build\temp.win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC >> \VS8.0\win32release /LIBPATH:build\temp.win32-2.6 quadpack.lib >> linpack_lite.lib mach.lib /EXPORT:ini >> t_quadpack build\temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj >> /OUT:build\lib.win32-2.6 >> \scipy\integrate\_quadpack.pyd >> /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib /M >> ANIFESTFILE:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest" >> failed with exit st >> atus 1120 >> >> thanks, >> ?loi >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> >> -- >> Eloi Gaudry >> Senior Product Development Engineer >> >> Free Field Technologies >> Company Website: http://www.fft.be >> Direct Phone Number: +32 10 495 147 >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user >> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user -- Eloi Gaudry Senior Product Development Engineer Free Field Technologies Company Website: http://www.fft.be Direct Phone Number: +32 10 495 147 From cgohlke at uci.edu Thu Apr 28 19:37:37 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Thu, 28 Apr 2011 16:37:37 -0700 Subject: [SciPy-User] [msvc] scipy/numpy build with visual studio and intel fortran compiler In-Reply-To: <4DB9DB7A.4080907@fft.be> References: <4DB9C107.1090709@fft.be> <4DB9DB7A.4080907@fft.be> Message-ID: <4DB9FA41.3060204@uci.edu> On 4/28/2011 2:26 PM, Eloi Gaudry wrote: > well, i somehow messed up with my setup. using the enclosed patches, i > can build numpy and scipy using msvc8/9 and intel visual fortran > compiler 11.1 (you need to apply the patches and use the matching > build_*msvc.x*.bat file). > 'hope this would help people that need to stick to visual studio > compilers... I have no such problems building numpy 1.5.1 and scipy 0.9 from the official source releases using Visual Studio 2008 (msvc9) and Intel ifort/MKL 11.1. No need to patch sources. Using Visual Studio 2005 (msvc8) to build Python extensions is not a good idea unless you target a custom CPython built with that compiler. Christoph > > ?loi > > On 28/04/2011 21:33, Eloi Gaudry wrote: >> hi, >> >> i'd like to discuss these two patches with people that needs to build >> current/recent versions of numpy and scipy using Visual Studio (200x) >> and the Intel Visual Fortran compiler. >> >> so far, i've been able to build numpy and i've facing one remaining >> issue with the scipy/special/cephes module which seems to occur >> because i'm using shared and static libraries. >> >> C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL >> /nologo /INCREMENTAL:NO /LIBPATH:c:\ >> fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release >> /LIBPATH:build\temp >> .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele >> ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib >> mach.lib /EXPORT:init_quadpack build >> \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj >> /OUT:build\lib.win32-2.6\scipy\integrate >> \_quadpack.pyd >> /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib >> /MANIFESTFILE:buil >> d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest >> C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe /DLL >> /nologo /INCREMENTAL:NO /LIBPATH:c:\ >> fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release >> /LIBPATH:build\temp >> .win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32rele >> ase /LIBPATH:build\temp.win32-2.6 quadpack.lib linpack_lite.lib >> mach.lib /EXPORT:init_quadpack build >> \temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj >> /OUT:build\lib.win32-2.6\scipy\integrate >> \_quadpack.pyd >> /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib >> /MANIFESTFILE:buil >> d\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest >> >> boom ! >> 1120 >> LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined >> in MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined >> in MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already >> defined in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already >> defined in MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already >> defined in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already >> defined in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined >> in MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in >> MSVCRT.lib(MSVCR80.dll) >> Creating library >> build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object >> build\temp >> .win32-2.6\Release\scipy\integrate\_quadpack.exp >> LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of >> other libs; use /NODEFAULTLIB:libr >> ary >> LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of >> other libs; use /NODEFAULTLIB:libr >> ary >> libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external >> symbol _MAIN__ referenced in func >> tion _main >> build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error >> LNK1120: 1 unresolved externals >> >> LIBCMT.lib(crt0dat.obj) : error LNK2005: __amsg_exit already defined >> in MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(crt0dat.obj) : error LNK2005: __initterm_e already defined >> in MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __encode_pointer already >> defined in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __encoded_null already >> defined in MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(tidtable.obj) : error LNK2005: __decode_pointer already >> defined in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_a already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xi_z already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_a already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(crt0init.obj) : error LNK2005: ___xc_z already defined in >> MSVCRT.lib(cinitexe.obj) >> LIBCMT.lib(winxfltr.obj) : error LNK2005: ___CppXcptFilter already >> defined in MSVCRT.lib(MSVCR80.dll >> ) >> LIBCMT.lib(crtheap.obj) : error LNK2005: __malloc_crt already defined >> in MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(mlock.obj) : error LNK2005: __unlock already defined in >> MSVCRT.lib(MSVCR80.dll) >> LIBCMT.lib(mlock.obj) : error LNK2005: __lock already defined in >> MSVCRT.lib(MSVCR80.dll) >> Creating library >> build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib and object >> build\temp >> .win32-2.6\Release\scipy\integrate\_quadpack.exp >> LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of >> other libs; use /NODEFAULTLIB:libr >> ary >> LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of >> other libs; use /NODEFAULTLIB:libr >> ary >> libifcoremt.lib(for_main.obj) : error LNK2019: unresolved external >> symbol _MAIN__ referenced in func >> tion _main >> build\lib.win32-2.6\scipy\integrate\_quadpack.pyd : fatal error >> LNK1120: 1 unresolved externals >> error: Command "C:\Program Files\Microsoft Visual Studio >> 8\VC\BIN\link.exe /DLL /nologo /INCREMENTAL >> :NO /LIBPATH:c:\fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC\VS8.0\win32release /LI >> BPATH:build\temp.win32-2.6 /LIBPATH:c:\fft\bin\python-2.6.6\libs >> /LIBPATH:c:\fft\bin\python-2.6.6\PC >> \VS8.0\win32release /LIBPATH:build\temp.win32-2.6 quadpack.lib >> linpack_lite.lib mach.lib /EXPORT:ini >> t_quadpack >> build\temp.win32-2.6\Release\scipy\integrate\_quadpackmodule.obj >> /OUT:build\lib.win32-2.6 >> \scipy\integrate\_quadpack.pyd >> /IMPLIB:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.lib /M >> ANIFESTFILE:build\temp.win32-2.6\Release\scipy\integrate\_quadpack.pyd.manifest" >> failed with exit st >> atus 1120 >> >> thanks, >> ?loi >> >> >> _______________________________________________ >> SciPy-User mailing list >> SciPy-User at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-user > > > -- > Eloi Gaudry > Senior Product Development Engineer > > Free Field Technologies > Company Website:http://www.fft.be > Direct Phone Number: +32 10 495 147 > > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user From josef.pktd at gmail.com Thu Apr 28 20:27:22 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 20:27:22 -0400 Subject: [SciPy-User] all unique permutations Message-ID: is there a copyright on basic algorithms ? from http://www.cut-the-knot.org/do_you_know/AllPerm.shtml I translated 2.Lexicographic order and finding the next permutation into python (because I couldn't get the matlab translation to work) looks useful for some permutation test some examples >>> pprint(list(permit([1,1,1,1,0]))) [[0, 1, 1, 1, 1], [1, 0, 1, 1, 1], [1, 1, 0, 1, 1], [1, 1, 1, 0, 1], [1, 1, 1, 1, 0]] >>> for i in permit([1,0,1,0]): print i ... [0, 0, 1, 1] [0, 1, 0, 1] [0, 1, 1, 0] [1, 0, 0, 1] [1, 0, 1, 0] [1, 1, 0, 0] >>> for i in permit([1,0,2,2]): print i ... [0, 1, 2, 2] [0, 2, 1, 2] [0, 2, 2, 1] [1, 0, 2, 2] [1, 2, 0, 2] [1, 2, 2, 0] [2, 0, 1, 2] [2, 0, 2, 1] [2, 1, 0, 2] [2, 1, 2, 0] [2, 2, 0, 1] [2, 2, 1, 0] >>> for i in permit([1,3]): print i ... [1, 3] [3, 1] >>> for i in permit([1,2.3]): print i ... [1, 2.2999999999999998] [2.2999999999999998, 1] >>> for i in permit([1,2.5,3.5]): print i ... [1, 2.5, 3.5] [1, 3.5, 2.5] [2.5, 1, 3.5] [2.5, 3.5, 1] [3.5, 1, 2.5] [3.5, 2.5, 1] Josef From josef.pktd at gmail.com Thu Apr 28 20:40:27 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 20:40:27 -0400 Subject: [SciPy-User] all unique permutations In-Reply-To: References: Message-ID: On Thu, Apr 28, 2011 at 8:27 PM, wrote: > is there a copyright on basic algorithms ? > > from > http://www.cut-the-knot.org/do_you_know/AllPerm.shtml > I translated > 2.Lexicographic order and finding the next permutation > > into python (because I couldn't get the matlab translation to work) > > looks useful for some permutation test > > some examples > >>>> pprint(list(permit([1,1,1,1,0]))) > [[0, 1, 1, 1, 1], > ?[1, 0, 1, 1, 1], > ?[1, 1, 0, 1, 1], > ?[1, 1, 1, 0, 1], > ?[1, 1, 1, 1, 0]] > >>>> for i in permit([1,0,1,0]): print i > ... > [0, 0, 1, 1] > [0, 1, 0, 1] > [0, 1, 1, 0] > [1, 0, 0, 1] > [1, 0, 1, 0] > [1, 1, 0, 0] > >>>> for i in permit([1,0,2,2]): print i > ... > [0, 1, 2, 2] > [0, 2, 1, 2] > [0, 2, 2, 1] > [1, 0, 2, 2] > [1, 2, 0, 2] > [1, 2, 2, 0] > [2, 0, 1, 2] > [2, 0, 2, 1] > [2, 1, 0, 2] > [2, 1, 2, 0] > [2, 2, 0, 1] > [2, 2, 1, 0] > >>>> for i in permit([1,3]): print i > ... > [1, 3] > [3, 1] > >>>> for i in permit([1,2.3]): print i > ... > [1, 2.2999999999999998] > [2.2999999999999998, 1] > >>>> for i in permit([1,2.5,3.5]): print i > ... > [1, 2.5, 3.5] > [1, 3.5, 2.5] > [2.5, 1, 3.5] > [2.5, 3.5, 1] > [3.5, 1, 2.5] > [3.5, 2.5, 1] and >>> for i in permit(['aa','bbb','c','aa']): print i ... ['aa', 'aa', 'bbb', 'c'] ['aa', 'aa', 'c', 'bbb'] ['aa', 'bbb', 'aa', 'c'] ['aa', 'bbb', 'c', 'aa'] ['aa', 'c', 'aa', 'bbb'] ['aa', 'c', 'bbb', 'aa'] ['bbb', 'aa', 'aa', 'c'] ['bbb', 'aa', 'c', 'aa'] ['bbb', 'c', 'aa', 'aa'] ['c', 'aa', 'aa', 'bbb'] ['c', 'aa', 'bbb', 'aa'] ['c', 'bbb', 'aa', 'aa'] > > Josef > From odysseus9672 at gmail.com Thu Apr 28 20:47:50 2011 From: odysseus9672 at gmail.com (Sean Lake) Date: Thu, 28 Apr 2011 17:47:50 -0700 Subject: [SciPy-User] Request for Interface Modification Message-ID: Hello all, Would it be possible to make a slight modification to numpy/scipy? Specifically, the list class in python has member functions insert, append and extend (probably others, but these are the ones I use the most). I see that numpy has equivalent functions insert, append, and concatenate, but they're not usable as member functions that modify the array they're being called on. I guess I'm basically asking if it would be trivial to enable the usage of more pythonesque syntax seen here http://docs.python.org/tutorial/datastructures.html , with arrays. It would seem that since many of these functions already exist it would be a simple matter of adding some statements to the class definitions would take care of this. Statements like: def function(self, other args): self = function( self, otherargs) Thanks, Sean Lake From ptittmann at gmail.com Thu Apr 28 20:59:10 2011 From: ptittmann at gmail.com (Peter Tittmann) Date: Thu, 28 Apr 2011 17:59:10 -0700 Subject: [SciPy-User] all unique permutations In-Reply-To: References: Message-ID: Hi Josef, Not sure if this is relevant but i've used itertools.permutations (http://docs.python.org/library/itertools.html#itertools.permutations) Best, Peter -- Peter Tittmann c 707 849 4135 On Thursday, April 28, 2011 at 5:40 PM, josef.pktd at gmail.com wrote: > On Thu, Apr 28, 2011 at 8:27 PM, wrote: > > is there a copyright on basic algorithms ? > > > > from > > http://www.cut-the-knot.org/do_you_know/AllPerm.shtml > > I translated > > 2.Lexicographic order and finding the next permutation > > > > into python (because I couldn't get the matlab translation to work) > > > > looks useful for some permutation test > > > > some examples > > > > > > > pprint(list(permit([1,1,1,1,0]))) > > [[0, 1, 1, 1, 1], > > [1, 0, 1, 1, 1], > > [1, 1, 0, 1, 1], > > [1, 1, 1, 0, 1], > > [1, 1, 1, 1, 0]] > > > > > > > for i in permit([1,0,1,0]): print i > > ... > > [0, 0, 1, 1] > > [0, 1, 0, 1] > > [0, 1, 1, 0] > > [1, 0, 0, 1] > > [1, 0, 1, 0] > > [1, 1, 0, 0] > > > > > > > for i in permit([1,0,2,2]): print i > > ... > > [0, 1, 2, 2] > > [0, 2, 1, 2] > > [0, 2, 2, 1] > > [1, 0, 2, 2] > > [1, 2, 0, 2] > > [1, 2, 2, 0] > > [2, 0, 1, 2] > > [2, 0, 2, 1] > > [2, 1, 0, 2] > > [2, 1, 2, 0] > > [2, 2, 0, 1] > > [2, 2, 1, 0] > > > > > > > for i in permit([1,3]): print i > > ... > > [1, 3] > > [3, 1] > > > > > > > for i in permit([1,2.3]): print i > > ... > > [1, 2.2999999999999998] > > [2.2999999999999998, 1] > > > > > > > for i in permit([1,2.5,3.5]): print i > > ... > > [1, 2.5, 3.5] > > [1, 3.5, 2.5] > > [2.5, 1, 3.5] > > [2.5, 3.5, 1] > > [3.5, 1, 2.5] > > [3.5, 2.5, 1] > > and > > > > > for i in permit(['aa','bbb','c','aa']): print i > ... > ['aa', 'aa', 'bbb', 'c'] > ['aa', 'aa', 'c', 'bbb'] > ['aa', 'bbb', 'aa', 'c'] > ['aa', 'bbb', 'c', 'aa'] > ['aa', 'c', 'aa', 'bbb'] > ['aa', 'c', 'bbb', 'aa'] > ['bbb', 'aa', 'aa', 'c'] > ['bbb', 'aa', 'c', 'aa'] > ['bbb', 'c', 'aa', 'aa'] > ['c', 'aa', 'aa', 'bbb'] > ['c', 'aa', 'bbb', 'aa'] > ['c', 'bbb', 'aa', 'aa'] > > > > > Josef > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 28 21:12:42 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 21:12:42 -0400 Subject: [SciPy-User] all unique permutations In-Reply-To: References: Message-ID: On Thu, Apr 28, 2011 at 8:59 PM, Peter Tittmann wrote: > Hi Josef, > Not sure if this is relevant but i've used itertools.permutations > (http://docs.python.org/library/itertools.html#itertools.permutations) I didn't find any iterator that would give unique values >>> for i in itertools.permutations([0,0,1]): print i ... (0, 0, 1) (0, 1, 0) (0, 0, 1) (0, 1, 0) (1, 0, 0) (1, 0, 0) >>> for i in permit([0,0,1]): print i ... [0, 0, 1] [0, 1, 0] [1, 0, 0] Thanks for the pointer Josef > Best, > Peter > -- > Peter Tittmann > c 707 849 4135 > > On Thursday, April 28, 2011 at 5:40 PM, josef.pktd at gmail.com wrote: > > On Thu, Apr 28, 2011 at 8:27 PM, wrote: > > is there a copyright on basic algorithms ? > > from > http://www.cut-the-knot.org/do_you_know/AllPerm.shtml > I translated > 2.Lexicographic order and finding the next permutation > > into python (because I couldn't get the matlab translation to work) > > looks useful for some permutation test > > some examples > > pprint(list(permit([1,1,1,1,0]))) > > [[0, 1, 1, 1, 1], > ?[1, 0, 1, 1, 1], > ?[1, 1, 0, 1, 1], > ?[1, 1, 1, 0, 1], > ?[1, 1, 1, 1, 0]] > > for i in permit([1,0,1,0]): print i > > ... > [0, 0, 1, 1] > [0, 1, 0, 1] > [0, 1, 1, 0] > [1, 0, 0, 1] > [1, 0, 1, 0] > [1, 1, 0, 0] > > for i in permit([1,0,2,2]): print i > > ... > [0, 1, 2, 2] > [0, 2, 1, 2] > [0, 2, 2, 1] > [1, 0, 2, 2] > [1, 2, 0, 2] > [1, 2, 2, 0] > [2, 0, 1, 2] > [2, 0, 2, 1] > [2, 1, 0, 2] > [2, 1, 2, 0] > [2, 2, 0, 1] > [2, 2, 1, 0] > > for i in permit([1,3]): print i > > ... > [1, 3] > [3, 1] > > for i in permit([1,2.3]): print i > > ... > [1, 2.2999999999999998] > [2.2999999999999998, 1] > > for i in permit([1,2.5,3.5]): print i > > ... > [1, 2.5, 3.5] > [1, 3.5, 2.5] > [2.5, 1, 3.5] > [2.5, 3.5, 1] > [3.5, 1, 2.5] > [3.5, 2.5, 1] > > and > > for i in permit(['aa','bbb','c','aa']): print i > > ... > ['aa', 'aa', 'bbb', 'c'] > ['aa', 'aa', 'c', 'bbb'] > ['aa', 'bbb', 'aa', 'c'] > ['aa', 'bbb', 'c', 'aa'] > ['aa', 'c', 'aa', 'bbb'] > ['aa', 'c', 'bbb', 'aa'] > ['bbb', 'aa', 'aa', 'c'] > ['bbb', 'aa', 'c', 'aa'] > ['bbb', 'c', 'aa', 'aa'] > ['c', 'aa', 'aa', 'bbb'] > ['c', 'aa', 'bbb', 'aa'] > ['c', 'bbb', 'aa', 'aa'] > > > Josef > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From alan.isaac at gmail.com Thu Apr 28 21:25:03 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 28 Apr 2011 21:25:03 -0400 Subject: [SciPy-User] all unique permutations In-Reply-To: References: Message-ID: <4DBA136F.40102@gmail.com> On 4/28/2011 9:12 PM, josef.pktd at gmail.com wrote: > I didn't find any iterator that would give unique values It's true that itertools.permutations is based on index permutations, but when the number of permuations is not huge, it is pretty cheap to form a set or a Counter. fwiw, Alan From alan.isaac at gmail.com Thu Apr 28 21:27:42 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Thu, 28 Apr 2011 21:27:42 -0400 Subject: [SciPy-User] Request for Interface Modification In-Reply-To: References: Message-ID: <4DBA140E.7020602@gmail.com> On 4/28/2011 8:47 PM, Sean Lake wrote: > numpy has equivalent functions insert, append, and > concatenate, but they're not usable as member functions > that modify the array they're being called on. NumPy arrays have fixed (total) length. Cheers, Alan Isaac From gruben at bigpond.net.au Thu Apr 28 21:45:22 2011 From: gruben at bigpond.net.au (gary ruben) Date: Fri, 29 Apr 2011 11:45:22 +1000 Subject: [SciPy-User] all unique permutations In-Reply-To: References: Message-ID: How about this? In [18]: from itertools import permutations In [19]: np.asarray(list(set(permutations([1,1,1,1,0])))) Out[19]: array([[0, 1, 1, 1, 1], [1, 1, 1, 0, 1], [1, 0, 1, 1, 1], [1, 1, 1, 1, 0], [1, 1, 0, 1, 1]]) It generates a different permutation ordering because sets are inherently unordered. Also, it's not a generator so your solution should be better for iterating over very large lists of items. Gary R. On Fri, Apr 29, 2011 at 10:40 AM, wrote: > On Thu, Apr 28, 2011 at 8:27 PM, ? wrote: >> is there a copyright on basic algorithms ? >> >> from >> http://www.cut-the-knot.org/do_you_know/AllPerm.shtml >> I translated >> 2.Lexicographic order and finding the next permutation >> >> into python (because I couldn't get the matlab translation to work) >> >> looks useful for some permutation test >> >> some examples >> >>>>> pprint(list(permit([1,1,1,1,0]))) >> [[0, 1, 1, 1, 1], >> ?[1, 0, 1, 1, 1], >> ?[1, 1, 0, 1, 1], >> ?[1, 1, 1, 0, 1], >> ?[1, 1, 1, 1, 0]] >> >>>>> for i in permit([1,0,1,0]): print i >> ... >> [0, 0, 1, 1] >> [0, 1, 0, 1] >> [0, 1, 1, 0] >> [1, 0, 0, 1] >> [1, 0, 1, 0] >> [1, 1, 0, 0] >> >>>>> for i in permit([1,0,2,2]): print i >> ... >> [0, 1, 2, 2] >> [0, 2, 1, 2] >> [0, 2, 2, 1] >> [1, 0, 2, 2] >> [1, 2, 0, 2] >> [1, 2, 2, 0] >> [2, 0, 1, 2] >> [2, 0, 2, 1] >> [2, 1, 0, 2] >> [2, 1, 2, 0] >> [2, 2, 0, 1] >> [2, 2, 1, 0] >> >>>>> for i in permit([1,3]): print i >> ... >> [1, 3] >> [3, 1] >> >>>>> for i in permit([1,2.3]): print i >> ... >> [1, 2.2999999999999998] >> [2.2999999999999998, 1] >> >>>>> for i in permit([1,2.5,3.5]): print i >> ... >> [1, 2.5, 3.5] >> [1, 3.5, 2.5] >> [2.5, 1, 3.5] >> [2.5, 3.5, 1] >> [3.5, 1, 2.5] >> [3.5, 2.5, 1] > > and > >>>> for i in permit(['aa','bbb','c','aa']): print i > ... > ['aa', 'aa', 'bbb', 'c'] > ['aa', 'aa', 'c', 'bbb'] > ['aa', 'bbb', 'aa', 'c'] > ['aa', 'bbb', 'c', 'aa'] > ['aa', 'c', 'aa', 'bbb'] > ['aa', 'c', 'bbb', 'aa'] > ['bbb', 'aa', 'aa', 'c'] > ['bbb', 'aa', 'c', 'aa'] > ['bbb', 'c', 'aa', 'aa'] > ['c', 'aa', 'aa', 'bbb'] > ['c', 'aa', 'bbb', 'aa'] > ['c', 'bbb', 'aa', 'aa'] > >> >> Josef >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Thu Apr 28 21:49:10 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 21:49:10 -0400 Subject: [SciPy-User] all unique permutations In-Reply-To: <4DBA136F.40102@gmail.com> References: <4DBA136F.40102@gmail.com> Message-ID: On Thu, Apr 28, 2011 at 9:25 PM, Alan G Isaac wrote: > On 4/28/2011 9:12 PM, josef.pktd at gmail.com wrote: >> I didn't find any iterator that would give unique values > > It's true that itertools.permutations is ?based on index > permutations, but when the number of permuations is not > huge, it is pretty cheap to form a set or a Counter. But it won't be an iterator anymore, and we need the entire set at once. I would like to go for big, maybe not huge, (and I'm still missing several other pieces and I still don't have an efficient code.) The target are exact permutation tests where the number of permutations grows very fast, (some papers by Mielke that I don't find right now.) >>> len(list(itertools.permutations([0,0,0,0,0,1,1,1,1]))) 362880 >>> len(list(permit([0,0,0,0,0,1,1,1,1]))) 126 >>> >>> len(list(permit([0,0,0,0,0,1,1,1,1,1,1]))) 462 >>> len(list(itertools.permutations([0,0,0,0,0,1,1,1,1,1,1]))) Traceback (most recent call last): File "", line 1, in MemoryError >>> it stopped at around 1.5 gigabyte according to taskmanager Josef > > fwiw, > Alan > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From bsouthey at gmail.com Thu Apr 28 22:18:05 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 28 Apr 2011 21:18:05 -0500 Subject: [SciPy-User] all unique permutations In-Reply-To: References: <4DBA136F.40102@gmail.com> Message-ID: On Thu, Apr 28, 2011 at 8:49 PM, wrote: > On Thu, Apr 28, 2011 at 9:25 PM, Alan G Isaac wrote: >> On 4/28/2011 9:12 PM, josef.pktd at gmail.com wrote: >>> I didn't find any iterator that would give unique values >> >> It's true that itertools.permutations is ?based on index >> permutations, but when the number of permuations is not >> huge, it is pretty cheap to form a set or a Counter. > > But it won't be an iterator anymore, and we need the entire set at once. > > I would like to go for big, maybe not huge, (and I'm still missing > several other pieces and I still don't have an efficient code.) ?The > target are exact permutation tests where the number of permutations > grows very fast, (some papers by Mielke that I don't find right now.) > >>>> len(list(itertools.permutations([0,0,0,0,0,1,1,1,1]))) > 362880 >>>> len(list(permit([0,0,0,0,0,1,1,1,1]))) > 126 >>>> >>>> len(list(permit([0,0,0,0,0,1,1,1,1,1,1]))) > 462 >>>> len(list(itertools.permutations([0,0,0,0,0,1,1,1,1,1,1]))) > Traceback (most recent call last): > ?File "", line 1, in > MemoryError >>>> > > it stopped at around 1.5 gigabyte according to taskmanager > > Josef > > >> >> fwiw, >> Alan >> >> _______________________________________________ > is there a copyright on basic algorithms ? IANAL but yes because translation from one [computer] language to another is still a derivative work. Rather the real question should be if there is in fact a valid copyright in the first place. In these cases probably not because many of these are probably already in public domain. I have not looked for ages, but I did find some recipes at activestate. A simple search gives many recipes such as: http://code.activestate.com/recipes/66463-permutation/ Actually the comments are often more helpful... Bruce From robert.kern at gmail.com Thu Apr 28 22:51:44 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 28 Apr 2011 21:51:44 -0500 Subject: [SciPy-User] all unique permutations In-Reply-To: References: <4DBA136F.40102@gmail.com> Message-ID: On Thu, Apr 28, 2011 at 21:18, Bruce Southey wrote: > On Thu, Apr 28, 2011 at 8:49 PM, ? wrote: >> is there a copyright on basic algorithms ? > IANAL but yes because translation from one [computer] language to > another is still a derivative work. Well, no, not in the US or any other jurisdiction that I am aware of. At least in the US, copyright protects the expression of ideas, not the ideas themselves. The actual algorithm is an idea. Code implementing the algorithm is the expression of that idea. Redistribution of a mechanical translation (and I mean the term loosely, not literally a translation done by a machine) of one form expression to another would be a violation of copyright. That said, for many such algorithms, there is really only one reasonable way to code it in a particular language, so there is no creative content in the actual expression of the idea, and so implementing it in your choice of language using another public implementation as a guide is usually fine. Copying comments, for example, is not fine. In this case, I think it's probably going to be fine. IANAL. TINLA. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Thu Apr 28 23:27:49 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 23:27:49 -0400 Subject: [SciPy-User] all unique permutations In-Reply-To: References: <4DBA136F.40102@gmail.com> Message-ID: On Thu, Apr 28, 2011 at 10:18 PM, Bruce Southey wrote: > On Thu, Apr 28, 2011 at 8:49 PM, ? wrote: >> On Thu, Apr 28, 2011 at 9:25 PM, Alan G Isaac wrote: >>> On 4/28/2011 9:12 PM, josef.pktd at gmail.com wrote: >>>> I didn't find any iterator that would give unique values >>> >>> It's true that itertools.permutations is ?based on index >>> permutations, but when the number of permuations is not >>> huge, it is pretty cheap to form a set or a Counter. >> >> But it won't be an iterator anymore, and we need the entire set at once. >> >> I would like to go for big, maybe not huge, (and I'm still missing >> several other pieces and I still don't have an efficient code.) ?The >> target are exact permutation tests where the number of permutations >> grows very fast, (some papers by Mielke that I don't find right now.) >> >>>>> len(list(itertools.permutations([0,0,0,0,0,1,1,1,1]))) >> 362880 >>>>> len(list(permit([0,0,0,0,0,1,1,1,1]))) >> 126 >>>>> >>>>> len(list(permit([0,0,0,0,0,1,1,1,1,1,1]))) >> 462 >>>>> len(list(itertools.permutations([0,0,0,0,0,1,1,1,1,1,1]))) >> Traceback (most recent call last): >> ?File "", line 1, in >> MemoryError >>>>> >> >> it stopped at around 1.5 gigabyte according to taskmanager >> >> Josef >> >> >>> >>> fwiw, >>> Alan >>> >>> _______________________________________________ > >> is there a copyright on basic algorithms ? > IANAL but yes because translation from one [computer] language to > another is still a derivative work. Rather the real question should be > if there is in fact a valid copyright in the first place. In these > cases probably not because many of these are probably already in > public domain. > > > I have not looked for ages, but I did find some recipes at > activestate. A simple search gives many recipes such as: > http://code.activestate.com/recipes/66463-permutation/ > > Actually the comments are often more helpful... I forgot to look there most algorithms go by positions but I found two that drop duplicates http://code.activestate.com/recipes/496819/ http://code.activestate.com/recipes/496869-computing-permutations-with-duplicates/ They are cleaner than mine, and I didn't know or remember that recursive function can "yield" The stop criterion in the unittests in the second recipe is even worse than mine, but should work well if I figure out how to calculate the number of unique permutations. Thanks Josef > > Bruce > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From josef.pktd at gmail.com Thu Apr 28 23:31:26 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 28 Apr 2011 23:31:26 -0400 Subject: [SciPy-User] all unique permutations In-Reply-To: References: <4DBA136F.40102@gmail.com> Message-ID: On Thu, Apr 28, 2011 at 10:51 PM, Robert Kern wrote: > On Thu, Apr 28, 2011 at 21:18, Bruce Southey wrote: >> On Thu, Apr 28, 2011 at 8:49 PM, ? wrote: > >>> is there a copyright on basic algorithms ? >> IANAL but yes because translation from one [computer] language to >> another is still a derivative work. > > Well, no, not in the US or any other jurisdiction that I am aware of. > At least in the US, copyright protects the expression of ideas, not > the ideas themselves. The actual algorithm is an idea. Code > implementing the algorithm is the expression of that idea. > Redistribution of a mechanical translation (and I mean the term > loosely, not literally a translation done by a machine) of one form > expression to another would be a violation of copyright. > > That said, for many such algorithms, there is really only one > reasonable way to code it in a particular language, so there is no > creative content in the actual expression of the idea, and so > implementing it in your choice of language using another public > implementation as a guide is usually fine. Copying comments, for > example, is not fine. In this case, I think it's probably going to be > fine. > > IANAL. TINLA. Thanks for answering our copyright questions, even IANAL and TINLA. Josef http://www.ianyl.net/TINLA.html > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From david at silveregg.co.jp Fri Apr 29 01:29:51 2011 From: david at silveregg.co.jp (David) Date: Fri, 29 Apr 2011 14:29:51 +0900 Subject: [SciPy-User] Do people here still use Python 2.5? In-Reply-To: References: Message-ID: <4DBA4CCF.30606@silveregg.co.jp> On 04/29/2011 06:46 AM, cool-RR wrote: > Hey everyone, > > I'm considering dropping Python 2.5 support in GarlicSim. This would > solve a compatibility bug I have now with zipimports, and generally make > me a happier person. > > Do you think that people have generally moved off Python 2.5 already? Depends on your audience. People who have to use "enterprise" distros like RHEL < 6 still depend on python 2.4. OTOH, updating to a new python is not too hard on windows and mac os x (or more exactly not harder than it already is :) ). > What do you think? Are there any people here who are forced to work with > Python 2.5 for some reason? There are people who still have to use python 1.x, so the question is really one of tradeoffs. Python 2.5 does not have huge advantages over 2.4 unless you use __future__ features (context managers, etc...). cheers, David From arsbbr at gmx.net Fri Apr 29 02:44:38 2011 From: arsbbr at gmx.net (arsbbr) Date: Thu, 28 Apr 2011 23:44:38 -0700 (PDT) Subject: [SciPy-User] [SciPy-user] Speedup with integrate.romberg() and vector arguments In-Reply-To: References: <20110427085006.325480@gmx.net> Message-ID: <31502972.post@talk.nabble.com> Thank you for your tips. A fixed grid wouldn't be a real option. But by writing the romberg() loop your way, the time for one curve_fit() call just takes 3.8 s instead of 15.7 s. def phi_romb_vec(q, a, alpha, beta, r_0, r=r): # phi(q,...,r) y = array([integrate.romberg(dy_dr, r[0], r[-1], # dy(r,...) args=(qi, A, a, alpha, beta, r_0), vec_func=True) for qi in q]) return y Interestingly, quad() "vectorized" doesn't change a thing: 9.24 s --> 9.22 s. I am new to this and still have to figure out what you do with np.concatenate(). It didn't seem to work that way. Regards, Arne josef.pktd wrote: > > On Wed, Apr 27, 2011 at 9:14 AM, wrote: >> On Wed, Apr 27, 2011 at 4:50 AM, ? wrote: >>> Hi, >>> >>> I need to integrate a function with an [1xn] array argument q. >>> For this I used integrate.quad(), but then I have to loop through every >>> element in q. >>> >>> In http://www.astrobetter.com/interpolation-and-integration-in-python/ I >>> found a hint how to speed up the integration. >>> >>> Unfortunately I can't seem to get romberg() to work: >>> >>> q is a [1 x n] numpy array, so I went from >>> >>> def phi_quad(q, a, alpha, beta, r_0, r=r): # phi(q,...,r) >>> ? ?y = zeros(r.size) >>> ? ?for k in range(r.size): >>> ? ? ? ?y[k] = integrate.quad(dy_dr, r[0], r[-1], # dy(r,...) >>> ? ? ? ? ? ?args=(q[k], A, a, alpha, beta, r_0))[0] >>> ? ?return y >>> >>> to >>> >>> def phi_romb_vec(q, a, alpha, beta, r_0, r=r): # phi(q,...,r) >>> ? ?y = integrate.romberg(dy_dr, r[0], r[-1], # dy(r,...) >>> ? ? ? ? args=(q, A, a, alpha, beta, r_0), vec_func=True) >>> ? ?return y >>> >>> which raises an error, if q is an array. The other arguments are all >>> floats. I attached a working example of the problem. Is there any way to >>> speed up the integration? >> >> I never tried this, but from the blog and the documentation, this is >> my interpretation >> >> It looks like romberg takes a vector argument but handles only a >> single integration problem (scalar output). >> >> in the function to be integrated, romberg uses a vector of r to speed >> up the integration >> dy_dr(r, q, A, a, alpha, beta, r_0) >> however, q and the other parameters need to be scalars >> >> So you still need to loop over q, doing one integration problem at a >> time. >> >> It might be possible to enhance romberg to proper broadcasting, but I >> think it would require that the number of iterations, number of >> subdivisions is the same for all integration problems. > > this actually seems to work with minimal changes > if np.all(err < tol) or np.all(err < rtol*abs(result)): > > but needs some broadcasting checks. > >>>> np.concatenate([phi_romb_vec(qi, a, alpha, beta, r_0, r=r) for qi in >>>> q]) > array([ 0.00065848, 0.00127215, 0.00216926, 0.00339121, 0.00497182, > 0.00693667, 0.0093027 , 0.01207799, 0.01526184, 0.01884502, > 0.02281024, 0.02713285, 0.03178164, 0.03671976, 0.04190578, > 0.0472947 , 0.05283912, 0.05849023, 0.06419887, 0.06991646, > 0.07559587, 0.08119216, 0.08666319, 0.09197017, 0.09707798, > 0.1019555 , 0.1065757 , 0.11091574, 0.11495692, 0.11868458, > 0.12208791, 0.12515975, 0.12789636, 0.13029711, 0.13236424, > 0.13410255, 0.1355191 , 0.13662298, 0.137425 , 0.13793743, > 0.13817382, 0.13814868, 0.13787734, 0.13737571, 0.13666008, > 0.13574698, 0.13465298, 0.13339455, 0.13198792, 0.13044894, > 0.128793 , 0.12703493, 0.12518888, 0.12326832, 0.12128595, > 0.11925369, 0.11718264]) >>>> phi_romb_vec(q[None,:], a, alpha, beta, r_0, r=r) > array([[ 0.00065848, 0.00127215, 0.00216926, 0.00339121, 0.00497182, > 0.00693667, 0.0093027 , 0.01207799, 0.01526184, 0.01884502, > 0.02281024, 0.02713285, 0.03178164, 0.03671976, 0.04190578, > 0.0472947 , 0.05283912, 0.05849023, 0.06419887, 0.06991646, > 0.07559587, 0.08119216, 0.08666319, 0.09197017, 0.09707798, > 0.1019555 , 0.1065757 , 0.11091574, 0.11495692, 0.11868458, > 0.1220879 , 0.12515975, 0.12789636, 0.13029711, 0.13236424, > 0.13410255, 0.1355191 , 0.13662298, 0.137425 , 0.13793743, > 0.13817382, 0.13814868, 0.13787734, 0.13737571, 0.13666008, > 0.13574698, 0.13465298, 0.13339455, 0.13198792, 0.13044894, > 0.128793 , 0.12703493, 0.12518888, 0.12326832, 0.12128595, > 0.11925369, 0.11718264]]) > > I haven't managed the call to curve_fit with the vectorized romberg > > Josef > >> >> If you are willing to work with a fixed grid, then integrate.romb >> might be useful, since from the docs it takes an axis argument, so a >> broadcasted (r, q) grid might work. >> >> Josef >> >> >>> >>> Best regards, >>> Arne >>> -- >>> NEU: FreePhone - kostenlos mobil telefonieren und surfen! >>> Jetzt informieren: http://www.gmx.net/de/go/freephone >>> >>> _______________________________________________ >>> SciPy-User mailing list >>> SciPy-User at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-user >>> >>> >> > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > -- View this message in context: http://old.nabble.com/Speedup-with-integrate.romberg%28%29-and-vector-arguments-tp31485594p31502972.html Sent from the Scipy-User mailing list archive at Nabble.com. From eg at fft.be Fri Apr 29 02:49:01 2011 From: eg at fft.be (Eloi Gaudry) Date: Fri, 29 Apr 2011 08:49:01 +0200 Subject: [SciPy-User] [msvc] scipy/numpy build with visual studio and intel fortran compiler In-Reply-To: <4DB9FA41.3060204@uci.edu> References: <4DB9C107.1090709@fft.be> <4DB9DB7A.4080907@fft.be> <4DB9FA41.3060204@uci.edu> Message-ID: <4DBA5F5D.4030704@fft.be> On 04/29/2011 01:37 AM, Christoph Gohlke wrote: > > On 4/28/2011 2:26 PM, Eloi Gaudry wrote: >> well, i somehow messed up with my setup. using the enclosed patches, i >> can build numpy and scipy using msvc8/9 and intel visual fortran >> compiler 11.1 (you need to apply the patches and use the matching >> build_*msvc.x*.bat file). >> 'hope this would help people that need to stick to visual studio >> compilers... > I have no such problems building numpy 1.5.1 and scipy 0.9 from the > official source releases using Visual Studio 2008 (msvc9) and Intel > ifort/MKL 11.1. No need to patch sources. christoph, you're right, those patches are useless (except the one targeting at scipy/sparse/linalg/dsolve/SuperLU/SRC/lsame.c) : those are obsolete modifications used for building scipy with a bad fcompiler setup, i shouldn't have sent those. > Using Visual Studio 2005 (msvc8) to build Python extensions is not a > good idea unless you target a custom CPython built with that compiler. i build python and python extensions using msvc8 for various reasons. what make you think this is a bad idea ? thanks > Christoph -- Eloi Gaudry Senior Product Development Engineer Free Field Technologies Company Website: http://www.fft.be From david at silveregg.co.jp Fri Apr 29 03:22:09 2011 From: david at silveregg.co.jp (David) Date: Fri, 29 Apr 2011 16:22:09 +0900 Subject: [SciPy-User] [msvc] scipy/numpy build with visual studio and intel fortran compiler In-Reply-To: <4DBA5F5D.4030704@fft.be> References: <4DB9C107.1090709@fft.be> <4DB9DB7A.4080907@fft.be> <4DB9FA41.3060204@uci.edu> <4DBA5F5D.4030704@fft.be> Message-ID: <4DBA6721.8010009@silveregg.co.jp> On 04/29/2011 03:49 PM, Eloi Gaudry wrote: > On 04/29/2011 01:37 AM, Christoph Gohlke wrote: >> >> On 4/28/2011 2:26 PM, Eloi Gaudry wrote: >>> well, i somehow messed up with my setup. using the enclosed patches, i >>> can build numpy and scipy using msvc8/9 and intel visual fortran >>> compiler 11.1 (you need to apply the patches and use the matching >>> build_*msvc.x*.bat file). >>> 'hope this would help people that need to stick to visual studio >>> compilers... >> I have no such problems building numpy 1.5.1 and scipy 0.9 from the >> official source releases using Visual Studio 2008 (msvc9) and Intel >> ifort/MKL 11.1. No need to patch sources. > christoph, > you're right, those patches are useless (except the one targeting at > scipy/sparse/linalg/dsolve/SuperLU/SRC/lsame.c) : those are obsolete > modifications used for building scipy with a bad fcompiler setup, i > shouldn't have sent those. >> Using Visual Studio 2005 (msvc8) to build Python extensions is not a >> good idea unless you target a custom CPython built with that compiler. > i build python and python extensions using msvc8 for various reasons. > what make you think this is a bad idea ? You need to use the exact same C library for both python interpreters and any extension used afterwards. The only way to do that is to use the exact same compiler to build python and all the extensions you want to use. Python 2.6 and above official releases are built with MSVC 2008. You are almost guaranteed to have weird crashes if you don't take that precaution while building numpy, cheers, David From cgohlke at uci.edu Fri Apr 29 03:35:40 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Fri, 29 Apr 2011 00:35:40 -0700 Subject: [SciPy-User] [msvc] scipy/numpy build with visual studio and intel fortran compiler In-Reply-To: <4DBA5F5D.4030704@fft.be> References: <4DB9C107.1090709@fft.be> <4DB9DB7A.4080907@fft.be> <4DB9FA41.3060204@uci.edu> <4DBA5F5D.4030704@fft.be> Message-ID: <4DBA6A4C.6070800@uci.edu> On 4/28/2011 11:49 PM, Eloi Gaudry wrote: > On 04/29/2011 01:37 AM, Christoph Gohlke wrote: >> >> On 4/28/2011 2:26 PM, Eloi Gaudry wrote: >>> well, i somehow messed up with my setup. using the enclosed patches, i >>> can build numpy and scipy using msvc8/9 and intel visual fortran >>> compiler 11.1 (you need to apply the patches and use the matching >>> build_*msvc.x*.bat file). >>> 'hope this would help people that need to stick to visual studio >>> compilers... >> I have no such problems building numpy 1.5.1 and scipy 0.9 from the >> official source releases using Visual Studio 2008 (msvc9) and Intel >> ifort/MKL 11.1. No need to patch sources. > christoph, > you're right, those patches are useless (except the one targeting at > scipy/sparse/linalg/dsolve/SuperLU/SRC/lsame.c) : those are obsolete > modifications used for building scipy with a bad fcompiler setup, i > shouldn't have sent those. lsame.c should not used on Windows when building with ifort/MKL: Probably the FPATH environment variable is not set in your case. Instead of manually setting the Intel compiler variables, better call the provided batch files, e.g.: call "C:\Program Files (x86)\Intel\Compiler\11.1\070\bin\iclvars.bat" ia32 vs2008 call "C:\Program Files (x86)\Intel\Compiler\11.1\070\bin\ifortvars.bat" ia32 vs2008 Christoph >> Using Visual Studio 2005 (msvc8) to build Python extensions is not a >> good idea unless you target a custom CPython built with that compiler. > i build python and python extensions using msvc8 for various reasons. > what make you think this is a bad idea ? > thanks >> Christoph > From alan.isaac at gmail.com Fri Apr 29 08:11:09 2011 From: alan.isaac at gmail.com (Alan G Isaac) Date: Fri, 29 Apr 2011 08:11:09 -0400 Subject: [SciPy-User] all unique permutations In-Reply-To: References: <4DBA136F.40102@gmail.com> Message-ID: <4DBAAADD.1030706@gmail.com> On 4/28/2011 9:49 PM, josef.pktd at gmail.com wrote: > But it won't be an iterator anymore, and we need the entire set at once. I found this ^ statement confusing: I think you are saying that you want an iterator, *not* the whole set at once. (?) Looking at your example, you seem to be working a different problem than I thought: it looks like your list are always zeros and ones? If so, you are just looking for k-combinations of positions. Take for example your list [0,0,0,0,0,1,1,1,1]. This places k=4 ones in any of n=9 positions. So we get the indexes you want as idx = itertools.combinations(range(9), 4) So you can have the iterator you want (?) as def idxvec(n, k): for idx in itertools.combinations(range(n), k): z = numpy.zeros(n) z[list(idx)] = 1 yield z If this does what you want, you can add a few efficiencies. (E.g., most importantly, test whether k>n/2, but also e.g., don't keep recreating the range). fwiw, Alan From josef.pktd at gmail.com Fri Apr 29 08:48:48 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 29 Apr 2011 08:48:48 -0400 Subject: [SciPy-User] all unique permutations In-Reply-To: <4DBAAADD.1030706@gmail.com> References: <4DBA136F.40102@gmail.com> <4DBAAADD.1030706@gmail.com> Message-ID: On Fri, Apr 29, 2011 at 8:11 AM, Alan G Isaac wrote: > On 4/28/2011 9:49 PM, josef.pktd at gmail.com wrote: >> But it won't be an iterator anymore, and we need the entire set at once. > > I found this ^ statement confusing: > I think you are saying that you want an iterator, > *not* the whole set at once. (?) Yes I meant, we need the entire set at once to construct the set. > > Looking at your example, you seem to be working > a different problem than I thought: it looks > like your list are always zeros and ones? ?If so, > you are just looking for k-combinations of positions. > Take for example your list [0,0,0,0,0,1,1,1,1]. > This places k=4 ones in any of n=9 positions. > So we get the indexes you want as > idx = itertools.combinations(range(9), 4) > > So you can have the iterator you want (?) as > > def idxvec(n, k): > ? for idx in itertools.combinations(range(n), k): > ? ? z = numpy.zeros(n) > ? ? z[list(idx)] = 1 > ? ? yield z 0s and 1s will be the most common case, but I might use the k-sample iterator. So, this is nice, similar to what I do for random permutations of 0s and 1s. Since I just recently switched to python 2.6, and statsmodels is on python 2.5, I haven't looked at the itertools enough yet. My combinatorics is rusty, I still have to figure out how many permutations there are in the different versions. Thanks, Josef > > If this does what you want, you can add a few efficiencies. > (E.g., most importantly, test whether k>n/2, but also e.g., > don't keep recreating the range). > > fwiw, > Alan > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > From Nikolaus at rath.org Fri Apr 29 09:16:52 2011 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 29 Apr 2011 09:16:52 -0400 Subject: [SciPy-User] SciPy 0.9 install: _fitpack.so: undefined symbol: jwe_ilst In-Reply-To: (Ralf Gommers's message of "Thu, 28 Apr 2011 22:22:49 +0200") References: <87hb9jz8i6.fsf@inspiron.ap.columbia.edu> Message-ID: <87iptxmc3f.fsf@inspiron.ap.columbia.edu> Ralf Gommers writes: > On Wed, Apr 27, 2011 at 11:31 PM, Nikolaus Rath wrote: >> Hello, >> >> I'm trying to build Scipy 0.9 from source. The installation with >> "setup.py install" seemed to work just fine, but when I try to.. >> >> In [1]: import scipy.interpolate >> >> ..then I get... >> >> ImportError: /home/byrne/.local/lib/python2.6/site-packages/scipy/interpolate/_fitpack.so: undefined symbol: jwe_ilst >> >> Any suggestions what may have gone wrong and how I could fix it? > > Very little turns up when googling for "jwe_ilst", the only perhaps > related useful info is > http://www.laheyforum.com/showthread.php?t=10777. Are you also using > the Lahey/Fujitsu Fortran compiler, or perhaps a LAPACK built with it? > Complete info on your OS and compilers would be helpful. Good intuition you have there. I was using GCC, but setup.py picked up the Lahey LAPACK libraries nevertheless. After I installed the atlas package, setup.py seems to use these instead and everything works fine. Thanks! -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From jturner at gemini.edu Fri Apr 29 11:50:52 2011 From: jturner at gemini.edu (James Turner) Date: Fri, 29 Apr 2011 12:50:52 -0300 Subject: [SciPy-User] Seg. fault from scipy.interpolate or numpy In-Reply-To: <4D9DE75F.9030203@gmail.com> References: <4D8CF861.9030302@gemini.edu> <4D9DDFA3.8070400@gemini.edu> <4D9DE75F.9030203@gmail.com> Message-ID: <4DBADE5C.80006@gemini.edu> >>> So can you determine exactly which line it crashes on (especially the >>> value yc) and the shapes of the arrays? >> >> Yes, it crashes on the last line, beginning "noise=", not in the loop. >> >> Good question about yc; it's value is 127, which is correct, as the >> test array has a shape of (128, 2048). >> >> I just tried splitting up the last line into separate operations and >> the one that crashes is objfit.clip(min=0.0). If I replace the last >> line with just that call, I still get the crash. >> >> I thought I'd try printing objfit.min() and max() just for fun and >> those work without crashing, returning -185.079 and 711.543. >> >> Thanks, >> >> James. > Good! > I think that you should be using 'a_min' not 'min' an the argument to an ndarray: > clip(...) > a.clip(a_min, a_max, out=None) Sorry for the slow reply again. I'm running NumPy 1.5.1 and when I use a_min=0., I get an error that it's an invalid keyword argument, but min=0 seems to work. That's odd! Anyway, I can give the minimum as a positional argument and it makes no difference to the crash. > Also you could try using np.clip(objfit, 0.0) as it should be the same > as objfit.clip(0.0). That doesn't work as it says 3 arguments are required -- but that made me think to try adding a maximum to the clip method as well as a minimum and voila, it doesn't crash! I don't want a maximum though :-(. Does this make any sense to you? Maybe there is something wrong with the way the function is wrapped (by the look if it) as a method, but I can't see how it can be using an uninitialized maximum value, as if my data were getting clipped to some random maximum it should be obvious. > But I am still curious is why it is crashing on your system. The ticket says > Python 2.5 on a 64-bit Linux but your backtrace includes 'i686' which suggests > that it is 32-bit. Can you be more specific of the versions of the software > regarding 32 vs 64 bit? I am running 32-bit binaries on a 64-bit machine (our server installation supports a variety of machines). Thanks, James. From ldavid at fas.harvard.edu Fri Apr 29 12:31:52 2011 From: ldavid at fas.harvard.edu (Lawrence David) Date: Fri, 29 Apr 2011 12:31:52 -0400 Subject: [SciPy-User] Doubled distances on UPGMA trees? Message-ID: <0DB23CB7-C4BC-45CC-AC44-4DFBBFF3F38A@fas.harvard.edu> Hi everyone, I'm trying to build UPGMA trees using scipy.cluster, but I'm running into a problem where my branch lengths appear to be doubled in length. As a test case, I'd like to build the tree from the following tutorial: http://www.nmsr.org/upgma.htm. >>> import scipy.cluster.hierarchy as sch >>> import scipy.spatial.distance as ssd >>> d_M = array([[0,19,27,8,33,18,13],[19,0,31,18,36,1,13],[27,31,0,26,41,32,29],[8,18,26,0,31,17,14],[33,36,41,31,0,35,28],[18,1,32,17,35,0,12],[13,13,29,14,28,12,0]]) >>> Y = ssd.squareform(d_M) >>> Z = sch.linkage(Y,method='average') >>> sch.dendrogram(Z) The outputted tree has branch lengths that are roughly double the ones expected according to the UPGMA guide. Can anyone please explain to me how I might want to go about recovering the desired tree? Is dendrogram the wrong plotting function to use? Many thanks! - Lawrence From pierre.raybaut at gmail.com Fri Apr 29 13:07:23 2011 From: pierre.raybaut at gmail.com (Pierre Raybaut) Date: Fri, 29 Apr 2011 19:07:23 +0200 Subject: [SciPy-User] ANN: Spyder v2.0.11 Message-ID: <09317DC5AFAD46FE8AA719F390F4C918@asus> Hi all, I am pleased to announced that Spyder v2.0.11 has just been released. As this is mostly a maintenance release, a lot of bugs were fixed and some minor features were added (see below). Embedding an interactive Python console into your GUI-based application (Qt): This version includes an example of application using the Spyder's internal shell as a debugging console which also demonstrates the py2exe deployment procedure. Spyder (previously known as Pydee) is a free open-source Python development environment providing MATLAB-like features in a simple and light-weighted software, available for Windows XP/Vista/7, GNU/Linux and MacOS X: http://spyderlib.googlecode.com/ Spyder is part of spyderlib, a Python module based on PyQt4, pyflakes and rope (QScintilla's dependency has been removed in version 2.0 and rope features have been integrated since this version as well). Some of Spyder basic features: * Python, C/C++, Fortran source editor with class/function browser, code completion and calltips * consoles: o open as many Python interpreters, IPython consoles or command windows as you need o code completion and calltips o variable explorer with GUI-based editors for a lot of data types (numbers, strings, lists, arrays, dictionaries, ...) * object inspector: provide documentation or source code on any Python object (class, function, module, ...) * online documentation: automatically generated html documentation on installed Python modules * find in files * file explorer * project manager * MATLAB-like PYTHONPATH management dialog box (works with all consoles) * Windows only: current user environment variables editor * direct links to documentation (Python, Qt, Matplotlib, NumPy, Scipy, etc.) * direct link to Python(x,y) launcher * direct links to QtDesigner, QtLinguist and QtAssistant (Qt documentation) Bug fixes (since v2.0.9): (Fixes Issue 616) Pylint plugin: tree widget header text was not updated when analyizing a new script (the last analyzed script name was still shown) Editor/completion widget/bugfix: pressing shift was hiding the completion combo box (Fixes Issue 630) Added missing default settings for "Spyder light" (only necessary when installing from scratch and without any remaining .spyder.ini file) Editor/Console-bugfix: info tooltips (calltips) were hidden right after being shown (i.e. when typing any character after the left parenthesis) (Fixes Issue 631) Drag and drop of files into editor on Linux was pasting path instead of opening the file (Fixes Issue 640) Editor: block comment was not working correctly at end of file Code completion widget (Editor/Console) - bugfix: parenting to the ancestor widget was necessary on Linux (Fixes Issue 546) [Contributor: Alex Fargus] C/Cpp syntax highlighting bugfix (Fixes Issue 646) IPython integration: fixed pyreadline monkey-patch for pyreadline v1.7 Enhancements (since v2.0.9): File explorer widget/plugin: improved performances (widget is now populated in a separate thread) Spyder crash dialog: warning the user about the '--reset' option (this will remove all configuration files) Cheers, Pierre From bsouthey at gmail.com Fri Apr 29 14:20:55 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 29 Apr 2011 13:20:55 -0500 Subject: [SciPy-User] Seg. fault from scipy.interpolate or numpy In-Reply-To: <4DBADE5C.80006@gemini.edu> References: <4D8CF861.9030302@gemini.edu> <4D9DDFA3.8070400@gemini.edu> <4D9DE75F.9030203@gmail.com> <4DBADE5C.80006@gemini.edu> Message-ID: <4DBB0187.3080600@gmail.com> On 04/29/2011 10:50 AM, James Turner wrote: >>>> So can you determine exactly which line it crashes on (especially the >>>> value yc) and the shapes of the arrays? >>> >>> Yes, it crashes on the last line, beginning "noise=", not in the loop. >>> >>> Good question about yc; it's value is 127, which is correct, as the >>> test array has a shape of (128, 2048). >>> >>> I just tried splitting up the last line into separate operations and >>> the one that crashes is objfit.clip(min=0.0). If I replace the last >>> line with just that call, I still get the crash. >>> >>> I thought I'd try printing objfit.min() and max() just for fun and >>> those work without crashing, returning -185.079 and 711.543. >>> >>> Thanks, >>> >>> James. >> Good! >> I think that you should be using 'a_min' not 'min' an the argument to >> an ndarray: >> clip(...) >> a.clip(a_min, a_max, out=None) > > Sorry for the slow reply again. I'm running NumPy 1.5.1 and when I > use a_min=0., I get an error that it's an invalid keyword argument, > but min=0 seems to work. That's odd! Anyway, I can give the minimum > as a positional argument and it makes no difference to the crash. Okay that is a bug in 1.6 as well! >>> a.clip(a_min=0, a_max=numpy.inf) Traceback (most recent call last): File "", line 1, in TypeError: 'a_max' is an invalid keyword argument for this function >>> help(a.clip) help on built-in function clip: clip(...) a.clip(a_min, a_max, out=None) Return an array whose values are limited to ``[a_min, a_max]``. > >> Also you could try using np.clip(objfit, 0.0) as it should be the same >> as objfit.clip(0.0). > > That doesn't work as it says 3 arguments are required -- but that > made me think to try adding a maximum to the clip method as well as > a minimum and voila, it doesn't crash! I don't want a maximum > though :-(. Does this make any sense to you? Maybe there is something > wrong with the way the function is wrapped (by the look if it) as a > method, but I can't see how it can be using an uninitialized maximum > value, as if my data were getting clipped to some random maximum it > should be obvious. Yes because it does not know the dtype in advance. But surely you can use positive and negative infinity: Python 2.5.5 (r255:77872, Nov 8 2010, 09:10:42) [GCC 4.5.1 20100924 (Red Hat 4.5.1-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '1.5.1' >>> help(np.clip) >>> a=np.arange(-10,10, dtype=float) >>> help(a.clip) >>> a.clip(a_min=0) Traceback (most recent call last): File "", line 1, in TypeError: 'a_min' is an invalid keyword argument for this function >>> a.clip(min=0) array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) >>> np.clip(a,a_min=0, a_max=np.inf) array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) >>> >>> a=np.arange(-10,10, dtype=float) >>> a array([-10., -9., -8., -7., -6., -5., -4., -3., -2., -1., 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) >>> a.clip(0) array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) >>> np.clip(a,0, np.PINF) array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) > >> But I am still curious is why it is crashing on your system. The >> ticket says >> Python 2.5 on a 64-bit Linux but your backtrace includes 'i686' which >> suggests >> that it is 32-bit. Can you be more specific of the versions of the >> software >> regarding 32 vs 64 bit? > > I am running 32-bit binaries on a 64-bit machine (our server > installation supports a variety of machines). > > Thanks, > > James. Bruce From bsouthey at gmail.com Fri Apr 29 14:36:25 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 29 Apr 2011 13:36:25 -0500 Subject: [SciPy-User] Seg. fault from scipy.interpolate or numpy In-Reply-To: <4DBB0187.3080600@gmail.com> References: <4D8CF861.9030302@gemini.edu> <4D9DDFA3.8070400@gemini.edu> <4D9DE75F.9030203@gmail.com> <4DBADE5C.80006@gemini.edu> <4DBB0187.3080600@gmail.com> Message-ID: On Fri, Apr 29, 2011 at 1:20 PM, Bruce Southey wrote: > On 04/29/2011 10:50 AM, James Turner wrote: >>>>> >>>>> So can you determine exactly which line it crashes on (especially the >>>>> value yc) and the shapes of the arrays? >>>> >>>> Yes, it crashes on the last line, beginning "noise=", not in the loop. >>>> >>>> Good question about yc; it's value is 127, which is correct, as the >>>> test array has a shape of (128, 2048). >>>> >>>> I just tried splitting up the last line into separate operations and >>>> the one that crashes is objfit.clip(min=0.0). If I replace the last >>>> line with just that call, I still get the crash. >>>> >>>> I thought I'd try printing objfit.min() and max() just for fun and >>>> those work without crashing, returning -185.079 and 711.543. >>>> >>>> Thanks, >>>> >>>> James. >>> >>> Good! >>> I think that you should be using 'a_min' not 'min' an the argument to an >>> ndarray: >>> clip(...) >>> a.clip(a_min, a_max, out=None) >> >> Sorry for the slow reply again. I'm running NumPy 1.5.1 and when I >> use a_min=0., I get an error that it's an invalid keyword argument, >> but min=0 seems to work. That's odd! Anyway, I can give the minimum >> as a positional argument and it makes no difference to the crash. > > Okay that is a bug in 1.6 as well! > >>>> a.clip(a_min=0, a_max=numpy.inf) > Traceback (most recent call last): > ?File "", line 1, in > TypeError: 'a_max' is an invalid keyword argument for this function >>>> help(a.clip) > help on built-in function clip: > > clip(...) > ? ?a.clip(a_min, a_max, out=None) > > ? ?Return an array whose values are limited to ``[a_min, a_max]``. > > >> >>> Also you could try using np.clip(objfit, 0.0) as it should be the same >>> as objfit.clip(0.0). >> >> That doesn't work as it says 3 arguments are required -- but that >> made me think to try adding a maximum to the clip method as well as >> a minimum and voila, it doesn't crash! I don't want a maximum >> though :-(. Does this make any sense to you? Maybe there is something >> wrong with the way the function is wrapped (by the look if it) as a >> method, but I can't see how it can be using an uninitialized maximum >> value, as if my data were getting clipped to some random maximum it >> should be obvious. > > Yes because it does not know the dtype in advance. > But surely you can use positive and negative infinity: > > Python 2.5.5 (r255:77872, Nov ?8 2010, 09:10:42) > [GCC 4.5.1 20100924 (Red Hat 4.5.1-4)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy as np >>>> np.__version__ > '1.5.1' >>>> help(np.clip) > >>>> a=np.arange(-10,10, dtype=float) >>>> help(a.clip) > >>>> a.clip(a_min=0) > Traceback (most recent call last): > ?File "", line 1, in > TypeError: 'a_min' is an invalid keyword argument for this function >>>> a.clip(min=0) > array([ 0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?1., ?2., > ? ? ? ?3., ?4., ?5., ?6., ?7., ?8., ?9.]) >>>> np.clip(a,a_min=0, a_max=np.inf) > array([ 0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?1., ?2., > ? ? ? ?3., ?4., ?5., ?6., ?7., ?8., ?9.]) >>>> > > >>>> a=np.arange(-10,10, dtype=float) >>>> a > array([-10., ?-9., ?-8., ?-7., ?-6., ?-5., ?-4., ?-3., ?-2., ?-1., ? 0., > ? ? ? ? 1., ? 2., ? 3., ? 4., ? 5., ? 6., ? 7., ? 8., ? 9.]) >>>> a.clip(0) > array([ 0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?1., ?2., > ? ? ? ?3., ?4., ?5., ?6., ?7., ?8., ?9.]) >>>> np.clip(a,0, np.PINF) > array([ 0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?0., ?1., ?2., > ? ? ? ?3., ?4., ?5., ?6., ?7., ?8., ?9.]) > >> >>> But I am still curious is why it is crashing on your system. The ticket >>> says >>> Python 2.5 on a 64-bit Linux but your backtrace includes 'i686' which >>> suggests >>> that it is 32-bit. Can you be more specific of the versions of the >>> software >>> regarding 32 vs 64 bit? >> >> I am running 32-bit binaries on a 64-bit machine (our server >> installation supports a variety of machines). >> >> Thanks, >> >> James. > > Bruce > Anyhow the clip bug was previously reported as ticket 1745: http://projects.scipy.org/numpy/ticket/1745 So I will add this example to that ticket. Bruce From hg+scipy at schaathun.net Fri Apr 29 15:01:35 2011 From: hg+scipy at schaathun.net (Hans Georg Schaathun) Date: Fri, 29 Apr 2011 21:01:35 +0200 Subject: [SciPy-User] Probability Density Estimation In-Reply-To: References: <20110406112947.GA1994@macwilliams.local> Message-ID: <20110429190135.GA9143@macwilliams.local> Dear all, this is a bit overdue; thanks a lot to everyone who helped me with my KDE trouble three weeks ago. On Wed, Apr 06, 2011 at 11:47:26AM -0400, josef.pktd at gmail.com wrote: > If you have or find some results and are willing to share, then I will > be very interested. The solution to my problem was to make sure that I use the same bandwidth for all the KDE estimations relating to the same variable, i.e. for P(X), P(X|Y=1) and P(X|Y=0). Now the results seem plausible, even though I did not get the results I was hoping for :-) I attach a monkey-patched and annotated version of gaussian_kde making it easier to choose the bandwidth. The main reason I monkey-patched rather than inherit is that I needed to annotate it to understand what I was doing. The comments are intended for pylit and sphinx, but do not work as well as I would have hoped. The second attachment is a derived class supporting entropy estimation. I have not done any really serious testing, and may have made mistakes. Any feedback would be welcome. -- :-- Hans Georg -------------- next part -------------- ## $Id: kde.py 2476 2011-04-29 18:51:57Z georg $ ## -*- coding: utf-8 -*- """ Define classes for (uni/multi)-variate kernel density estimation. Currently, only Gaussian kernels are implemented. :Module: itml.kde :Author: Robert Kern :Date: $Date: 2011-04-29 20:51:57 +0200 (Fri, 29 Apr 2011) $ :Revision: $Revision: 2476 $ :Copyright: 2004-2005 by Enthought, Inc. ? 2011: Hans Georg Schaathun :Original Date: 2004-08-09 :Modified: 2005-02-10 by Robert Kern. Contributed to Scipy 2005-10-07 by Robert Kern. Some fixes to match the new scipy_core 2011-04-13 by Hans Georg Schaathun. """ print "[itml.kde] $Id: kde.py 2476 2011-04-29 18:51:57Z georg $" # ######################### # Kernel density estimation # ######################### # # .. automodule:: itml.kde # # Standard library imports:: import warnings # Scipy imports:: from scipy import linalg, special from numpy import atleast_2d, reshape, zeros, newaxis, dot, exp, pi, sqrt, \ ravel, power, atleast_1d, squeeze, sum, transpose import numpy as np from numpy.random import randint, multivariate_normal # Local imports:: #import stats from scipy.stats import mvn from . import * __all__ = ['gaussian_kde', ] # Auxiliary functions # =================== # # .. autofunction:: scotts_factor # # .. autofunction:: silverman_factor # # Both functions calculate a bandwidth scaling factor based on the sample # size n and dimensionality d. # # :: def scotts_factor(n,d): return power(n, -1./(d+4)) def silverman_factor(n,d): return power(n*(d+2.0)/4.0, -1./(d+4)) # The class # ========= # # .. autoclass:: gaussian_kde class gaussian_kde(object): """ Representation of a kernel-density estimate using Gaussian kernels. Attributes ---------- d : int number of dimensions n : int number of datapoints Methods ------- kde.evaluate(points) : array evaluate the estimated pdf on a provided set of points kde(points) : array same as kde.evaluate(points) kde.setBandwidth(bandwidth) : array sets the bandwidth matrix, or if H is not specified, recalculates a bandwidth matrix from the covariance kde.setBandwidthFactor(factor) : array sets the scaling factor and recalculates the bandwidth matrix from the covariance. kde.integrate_gaussian(mean, cov) : float multiply pdf with a specified Gaussian and integrate over the whole domain kde.integrate_box_1d(low, high) : float integrate pdf (1D only) between two bounds kde.integrate_box(low_bounds, high_bounds) : float integrate pdf over a rectangular space between low_bounds and high_bounds kde.integrate_kde(other_kde) : float integrate two kernel density estimates multiplied together kde.resample(size=None) : array randomly sample a dataset from the estimated pdf. Internal Methods ---------------- kde.covariance_factor() : float computes the coefficient that multiplies the data covariance matrix to obtain the kernel covariance matrix. Parameters ---------- dataset : (# of dims, # of data)-array datapoints to estimate from factor : (optional) function, scalar, or string optionally specify a scaling factor to use to calculate the bandwidth matrix. It may be a function taking sample size and dimension as inputs, a scalar, or a string "scotts" or "silverman" for the standard scaling factors. If a bandwidth matrix (below) is specified, then factor is ignored. bandwidth : optional, (# dims, # dims)-array optionally specify the bandwidth matrix to use with the kernel """ # The following dictionary is used to interpret bandwidth factors specified # as prescriptive strings, and look up corresponding functions. # :: bwfactor = { "scott" : scotts_factor, "scotts" : scotts_factor, "silverman" : silverman_factor, "sc" : scotts_factor, "si" : silverman_factor, } # The constructor will only store the data set and compute sample size, # dimension, and covariance matrix. All the important calculations are # made later, upon evaluation or integration. # # :: def __init__(self, dataset, factor="scotts", bandwidth=None ): self.dataset = atleast_2d(dataset) self.d, self.n = self.dataset.shape if bandwidth != None: self.setBandwidth( bandwidth ) else: self.setBandwidthFactor( factor ) # This is the most important method, evaluating the estimated function # at given points. # # .. automethod:: gaussian_kde.evaluate def evaluate(self, points): """ Evaluate the estimated pdf on a set of points. Parameters ---------- points : (# of dimensions, # of points)-array Alternatively, a (# of dimensions,) vector can be passed in and treated as a single point. Returns ------- values : (# of points,)-array The values at each point. Raises ------ ValueError if the dimensionality of the input points is different than the dimensionality of the KDE. """ # The given evaluation point(s) are converted to a 2-D array of the # same type as the original sample points. # # :: points = atleast_2d(points).astype(self.dataset.dtype) # We check that the dimensions match. If a single point was passed # as a row vector, we can simply transpose it. Any other mismatch # leads to an exception. # # :: d, m = points.shape if d != self.d: if d == 1 and m == self.d: # points was passed in as a row vector points = reshape(points, (self.d, 1)) m = 1 else: msg = "points have dimension %s, dataset has dimension %s" % (d, self.d) raise ValueError(msg) # We initialise the return array with all zeros. # # :: result = zeros((m,), points.dtype) # For the sake of computational efficiency, we distinguish # the cases where the sample is smaller or larger than the # set of evaluation points. # If there are more points than data, then we loop over data, # otherwise we will loop over the points. # The algorithms for the two cases are equivalent, # but the second case is easier to explain so we leave this # one uncommented. # # :: if m >= self.n: for i in range(self.n): diff = self.dataset[:,i,newaxis] - points tdiff = dot(self.inv_bandwidth, diff) energy = sum(diff*tdiff,axis=0)/2.0 result += exp(-energy) else: for i in range(m): # Now, we loop over points. # Each iteration considers a single evaluation point :math:`x_i`, # and we calculate the following formula which is the definition # of the Gaussian KDE. # # .. math:: # # \hat f(x_i) = \frac1{n}\sum_{j=1}^n s_H(X_j-x_i), # # where :math:`s_H` is the scaled Gaussian kernel defined, # for a given bandwidth matrix :math:`H` as # # .. math:: # # s_H(\mathbf{x}) = \frac1{\sqrt{|2\pi H^2|}} # \exp\big( \frac12 \mathbf{x}^T\cdot (H^2)^{-1}\cdot\mathbf{x} \big). # # The bandwidth :math:`H` is calculated by the :meth:`setBandwidth` # method, and is usually given as :math:`H^2 = h^2\cdot\Sigma^2` where # :math:`\Sigma^2` is the covariance matrix and :math:`h` is a scalar # scaling factor. # # First we calculate the differences :math:`X_j-x_i` for # each sample point :math:`X_j` and the evaluation point :math:`x_i`. # Each column of diff is a difference :math:`X_j - x_i`. # # :: diff = self.dataset - points[:,i,newaxis] # Now we want to calculate # :math:`(X_j-x_i)^T\dot\Sigma^{-1}\dot (X_j-x_i)/2` for each :math:`j` # where :math:`\Sigma` is the covariance matrix. # # :: tdiff = dot(self.inv_bandwidth, diff) energy = sum(diff*tdiff,axis=0)/2.0 # Each entry in energy corresponds to one sample point :math:`X_j`. # We apply the exponential function and sum over :math:`j` obtaining the # result :math:`\hat f(x_i)` as result[i]. # # :: result[i] = sum(exp(-energy),axis=0) # Finally, we divide by the normalisation factor # :math:`n\sqrt{|2\pi\Sigma|}`. # # :: result /= self._norm_factor return result # Just for syntactic sugar, we make the object callable. # Calling the object is equivalent to calling :meth:`evaluate`. # # :: __call__ = evaluate # Covariance and Bandwidth # ======================== # # :: def covariance_factor(self): """ Calculate the scaling factor for the bandwidth matrix. This is intended for internal use only. """ if callable(self._bwfactor): return self._bwfactor(self.n,self.d) else: return self._bwfactor def setBandwidthFactor(self,factor="scotts",recalculate=True): if self.bwfactor.has_key(factor): self._bwfactor = self.bwfactor[factor] elif callable(factor): self._bwfactor = factor else: self._bwfactor = float(factor) if recalculate: self.setBandwidth() def getBandwidth(self): return self.bandwidth def setBandwidth(self,bandwidth=None,covariance=None): """Computes the covariance matrix for each Gaussian kernel using covariance_factor """ # If the bandwidth is given, we use it. Otherwise we calculate # a recommended bandwidth from the covariance matrix. # # :: if bandwidth != None: self.covariance = None self.bandwidth = bandwidth else: factor = self.covariance_factor() # The covariance attribute is the covariance matrix :math:`\Sigma^2`, # and the bandwidth attribute is the squared bandwidth matrix # :math:`\vec{H} = (h\Sigma)^2`. # # :: if covariance == None: self.covariance = atleast_2d( np.cov(self.dataset, rowvar=1, bias=False) ) else: self.covariance = covariance self.bandwidth = self.covariance * factor * factor # We invert the squared bandwidth matrix here, to make sure it is done # only once. # # :: try: self.inv_bandwidth = linalg.inv(self.bandwidth) except Exception as e: print_exc(e) print "Bandwidth:", self.bandwidth print "Covariance:", self.covariance print "Factor:", factor print "Dataset:", self.dataset raise e # The normalisation factor consists of three elements. # Firstly, there is the factor :math:`\sqrt{2\pi}^d` which # is found in the :math:`d`-variate Gaussian kernel. # Secondly, there is the square root of the determinant of the bandwidth # matrix :math:`\sqrt{|h^2\Sigma^2|}`. And finally we have the sample # size :math:`n` from the KDE itself. Thus, the _norm_factor below # is equal to # # .. math:: # # 2\pi^{d/2}\sqrt{|h^2\Sigma^2|} \cdot n = # \sqrt{|2\pi h^2\Sigma^2|} \cdot n. # # :: self._norm_factor = sqrt(linalg.det(2*pi*self.bandwidth)) * self.n # Other methods # ============= # # No changes have been made to the following methods compared to # :class:`scipy.stats.guassian_kde`. # # :: def resample(self, size=None): """Randomly sample a dataset from the estimated pdf. Parameters ---------- size : int, optional The number of samples to draw. If not provided, then the size is the same as the underlying dataset. Returns ------- dataset : (self.d, size)-array sampled dataset """ if size is None: size = self.n norm = transpose(multivariate_normal(zeros((self.d,), float), self.bandwidth, size=size)) indices = randint(0, self.n, size=size) means = self.dataset[:,indices] return means + norm # Integration methods # =================== # # No changes have been made to the integration methods compared to # :class:`scipy.stats.guassian_kde`. # # :: def integrate_gaussian(self, mean, cov): """Multiply estimated density by a multivariate Gaussian and integrate over the wholespace. Parameters ---------- mean : vector the mean of the Gaussian cov : matrix the covariance matrix of the Gaussian Returns ------- result : scalar the value of the integral Raises ------ ValueError if the mean or covariance of the input Gaussian differs from the KDE's dimensionality. """ mean = atleast_1d(squeeze(mean)) cov = atleast_2d(cov) if mean.shape != (self.d,): raise ValueError("mean does not have dimension %s" % self.d) if cov.shape != (self.d, self.d): raise ValueError("covariance does not have dimension %s" % self.d) # make mean a column vector mean = mean[:,newaxis] sum_cov = self.bandwidth + cov diff = self.dataset - mean tdiff = dot(linalg.inv(sum_cov), diff) energies = sum(diff*tdiff,axis=0)/2.0 result = sum(exp(-energies),axis=0)/sqrt(linalg.det(2*pi*sum_cov))/self.n return result def integrate_box_1d(self, low, high): """Computes the integral of a 1D pdf between two bounds. Parameters ---------- low : scalar lower bound of integration high : scalar upper bound of integration Returns ------- value : scalar the result of the integral Raises ------ ValueError if the KDE is over more than one dimension. """ if self.d != 1: raise ValueError("integrate_box_1d() only handles 1D pdfs") stdev = ravel(sqrt(self.bandwidth))[0] normalized_low = ravel((low - self.dataset)/stdev) normalized_high = ravel((high - self.dataset)/stdev) value = np.mean(special.ndtr(normalized_high) - special.ndtr(normalized_low)) return value def integrate_box(self, low_bounds, high_bounds, maxpts=None): """Computes the integral of a pdf over a rectangular interval. Parameters ---------- low_bounds : vector lower bounds of integration high_bounds : vector upper bounds of integration maxpts=None : int maximum number of points to use for integration Returns ------- value : scalar the result of the integral """ if maxpts is not None: extra_kwds = {'maxpts': maxpts} else: extra_kwds = {} value, inform = mvn.mvnun(low_bounds, high_bounds, self.dataset, self.bandwidth, **extra_kwds) if inform: msg = ('an integral in mvn.mvnun requires more points than %s' % (self.d*1000)) warnings.warn(msg) return value def integrate_kde(self, other): """Computes the integral of the product of this kernel density estimate with another. Parameters ---------- other : gaussian_kde instance the other kde Returns ------- value : scalar the result of the integral Raises ------ ValueError if the KDEs have different dimensionality. """ if other.d != self.d: raise ValueError("KDEs are not the same dimensionality") # we want to iterate over the smallest number of points if other.n < self.n: small = other large = self else: small = self large = other sum_cov = small.covariance + large.covariance result = 0.0 for i in range(small.n): mean = small.dataset[:,i,newaxis] diff = large.dataset - mean tdiff = dot(linalg.inv(sum_cov), diff) energies = sum(diff*tdiff,axis=0)/2.0 result += sum(exp(-energies),axis=0) result /= sqrt(linalg.det(2*pi*sum_cov))*large.n*small.n return result -------------- next part -------------- ## $Id: kdetools.py 2290 2011-04-13 19:46:36Z georg $ ## -*- coding: utf-8 -*- """ Define additional tools for kernel density estimation, including methods to estimate the entropy. :Module: itml.kdetools :Author: Hans Georg Schaathun :Date: $Date: 2011-04-13 21:46:36 +0200 (Wed, 13 Apr 2011) $ :Revision: $Revision: 2290 $ :Copyright: ? 2011: Hans Georg Schaathun """ print "[itml.kdetools] $Id: kdetools.py 2290 2011-04-13 19:46:36Z georg $" # ####################### # KDE Additions and Tools # ####################### # # .. automodule:: itml.kdetools # # :: import numpy as np from .kde import gaussian_kde as gkde # The main class # ============== # # :: class xkde(gkde): """ Representation of a kernel-density estimate using Gaussian kernels, including methods to estimate entropy. """ # The constructor is completely overridden because other methods have # changed their signatures. # # :: def __init__(self,dataset,exclude=None,factor="scotts",bandwidth=None, verbosity=2): self._verbosity = verbosity # Firstly, store the dataset and its shape as in the :class:`gkde` # constructor. We use self.nprime to count the number of points # used in the bandwidth calculation. # # :: self.dataset = np.atleast_2d(dataset) self.d, self.n = self.dataset.shape self.nprime = self.n # Then we set the bandwidth using the relevant input arguments. # # :: self.setBandwidthFactor( factor, recalculate=False ) if bandwidth != None: self.setBandwidth( bandwidth=bandwidth ) else: self.setBandwidth( exclude ) # We override :meth:`covariance_factor` because we do not use the # full sample in the bandwidth calculation. Therefore we use self.nprime # in lieu of self.n. # # :: def covariance_factor(self): """ Calculate the scaling factor for the bandwidth matrix. This is intended for internal use only. """ if callable(self._bwfactor): return self._bwfactor(self.nprime,self.d) else: return self._bwfactor # Overriding the :meth:`setBandwidth` method is essential. The new # bandwidth calculation is the main feature of the derived class. # # :: def setBandwidth(self,exclude=None,*a,**kw): """Computes the covariance matrix for each Gaussian kernel using covariance_factor """ # If exclude is not given, we revert to the method of the parent class. # We reset self.nprime just in case we are recalculating the bandwidth # after having used a different bandwidth. # # :: if exclude == None: self.nprime = self.n return gkde.setBandwidth(self,*a,**kw) # Calculate the variance for each feature, and build the boolean # matrix B indicating entries which at most equal to exclude times # the standard deviation. Squaring the data is faster, and thus # preferred over calculating the standard deviation. # # :: sigma2 = np.var( self.dataset, axis=1 ) mu = np.mean( self.dataset, axis=1 ) D = (self.dataset - mu) ** 2 S = np.repeat(sigma2[:,np.newaxis],self.n,axis=1) B = ( D <= exclude * S ) # Now, we get the indices of all the data points with no feature # exceeding exclude times the standard deviation, and include # these points only when we calculate the covariance. # # :: idx = [ i for i in xrange(self.n) if B[:,i].all() ] if self._verbosity > 0: print "[setBandwidth] Excluding:", self.n - len(idx), \ (np.min(self.dataset,axis=1), np.max(self.dataset,axis=1) ), self.covariance_factor() self.nprime = len(idx) cov = np.atleast_2d( np.cov(self.dataset[:,idx], rowvar=1, bias=False) ) # Check for NaN entries. This should not happen, but in case it does, # we provide verbose debug information. # # :: if np.isnan(cov).any(): print "[setBandwidth] Covariance matrix contains NaN entries." print "Original Covariance Matrix:", sigma2 print "Dataset:", self.nprime print "Covariance Matrix with Exclusions:", cov raise Exception, "Covariance matrix contains NaN entries." # Now, we can use the parent class, feeding it the covariance matrix # we have calculated. # # :: gkde.setBandwidth(self,covariance=cov) # Entropy Estimation # ------------------ # # There are several different formul? for estimating entropy. # Each of them is based on having an estimate of the PDF first, # for instance in the form of a kernel density estimate. # # An intuitive estimate of the entropy 2 is to substitute the estimated density # for the theoretical density in the formula for entropy. However, this # is not quite the favoured method in the literature. # # A very simple estimate to calculate is that of Ahmad and Lin, # which we implement below. # Contrary to other methods, it requires no integration, and thus # it avoids the challenge of finding the optimal way of numerical # integration for the given dimensionality and sample size. # We implement it as follows. # # :: def ahmadlin(self,verbosity=0): """ Estimate the entropy using the formula of Ahmad and Lin. """ # First we evaluate the estimated PDF at all the sample points. # # :: y = self.evaluate(np.atleast_2d(self.dataset)) if verbosity > 2: print "[ahmadlin]", len(y), np.var(self.dataset), np.max(y), np.min(y) # Then we take the logarithm of all the evaluations and add up to # get the estimated entropy r. # # :: return - np.sum(np.log(y)) / self.n contEntropy = ahmadlin # Functions # ========= # # :: def contEntropy(X,*a,**kw): """ Estimate the entropy of a probability distribution based on the given sample X. """ return xkde(X,*a,**kw).contEntropy() From josef.pktd at gmail.com Fri Apr 29 16:06:46 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 29 Apr 2011 16:06:46 -0400 Subject: [SciPy-User] Probability Density Estimation In-Reply-To: <20110429190135.GA9143@macwilliams.local> References: <20110406112947.GA1994@macwilliams.local> <20110429190135.GA9143@macwilliams.local> Message-ID: On Fri, Apr 29, 2011 at 3:01 PM, Hans Georg Schaathun wrote: > Dear all, > > this is a bit overdue; thanks a lot to everyone who helped me > with my KDE trouble three weeks ago. > > On Wed, Apr 06, 2011 at 11:47:26AM -0400, josef.pktd at gmail.com wrote: >> If you have or find some results and are willing to share, then I will >> be very interested. > > The solution to my problem was to make sure that I use the same > bandwidth for all the KDE estimations relating to the same > variable, i.e. for P(X), P(X|Y=1) and P(X|Y=0). ?Now the results > seem plausible, even though I did not get the results I was hoping > for :-) > > I attach a monkey-patched and annotated version of gaussian_kde making > it easier to choose the bandwidth. ?The main reason I monkey-patched > rather than inherit is that I needed to annotate it to understand what > I was doing. ?The comments are intended for pylit and sphinx, but do > not work as well as I would have hoped. > > The second attachment is a derived class supporting entropy estimation. > > I have not done any really serious testing, and may have made mistakes. > > Any feedback would be welcome. Thanks, I need to play with it to see where it differs from the scipy or my older version (except for the exclude trimming). I didn't see yet how you set the bandwidth from 2 samples to use for each individual, but it sounds like an interesting idea that I need to check for my case in mutual information. You didn't specify a license, which would be good so that readers know what they are allowed to do with the code. And as an observation: I find the literate programming comments pretty distracting (at least reading it in the browser without a highlighting editor, wrong intend for my taste) and would prefer numpy style docstrings, e.g. I didn't find a description of exclude. some details I'm not sure about when reading it, for example idx = [ i for i in xrange(self.n) if B[:,i].all() ] isn't this the same as idx = np.nonzero(B.all(0)) ahmadlin looks like it would be a nice extension to scipy.stats also as a standalone function (like your contEntropy), if the automatic bandwidth choice can be made robust enough. Was the outlier trimming enough to solve your problem with estimating a kernel density, or did you try also try other kernels? Josef > -- > :-- Hans Georg > > _______________________________________________ > SciPy-User mailing list > SciPy-User at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user > > From hg+scipy at schaathun.net Fri Apr 29 17:46:25 2011 From: hg+scipy at schaathun.net (Hans Georg Schaathun) Date: Fri, 29 Apr 2011 23:46:25 +0200 Subject: [SciPy-User] Probability Density Estimation In-Reply-To: References: <20110406112947.GA1994@macwilliams.local> <20110429190135.GA9143@macwilliams.local> Message-ID: <20110429214625.GA9631@macwilliams.local> On Fri, Apr 29, 2011 at 04:06:46PM -0400, josef.pktd at gmail.com wrote: > I need to play with it to see where it differs from the scipy or my > older version (except for the exclude trimming). > I didn't see yet how you set the bandwidth from 2 samples to use for > each individual, but it sounds like an interesting idea that I need to > check for my case in mutual information. I do this: fC = xkde(C) H = fC.getBandwidth() fA = xkde(A,bandwidth=H) fB = xkde(B,bandwidth=H) where A and B are the samples of X where Y=0 and Y=1 respectively, and C is the complete sample of X (i.e. the union of A and B). This calculates the three KDE-s with the same bandwidth. I experimented a bit using the exclude argument without finding any evidence that it is useful. > You didn't specify a license, which would be good so that readers know > what they are allowed to do with the code. I didn't. And I will look into this before I publish any larger quantities of codes. I simply have not got around to it. I see no distinction between source code and other text; in other words, I expect proper citations/acknowledgement/etc... > And as an observation: I find the literate programming comments pretty > distracting (at least reading it in the browser without a highlighting > editor, wrong intend for my taste) and would prefer numpy style > docstrings, e.g. I didn't find a description of exclude. Agreed. It is really written to be compiled and read from PDF. I am not satisfied with the solution, but I really need this feature of literate programming. The docstrings can certainly be improved. Do you have a good pointer on good practice? You talk about numpy-style, should I look in the numpy doc's? > some details I'm not sure about when reading it, for example > > idx = [ i for i in xrange(self.n) if B[:,i].all() ] > isn't this the same as > idx = np.nonzero(B.all(0)) Probably. I am not sure how numpy.nonzero deals with Boolean values. > ahmadlin looks like it would be a nice extension to scipy.stats also > as a standalone function (like your contEntropy), if the automatic > bandwidth choice can be made robust enough. Sure. I am really not sure how to go about testing the robustness. Are you doing research in a similar area? Information Theory? There is not an awful lot in the literature about multivariate continuous entropy, so there might be room for publishing some theory here (or even experimental evaluations of the little theory there is), if we can develop it. The reason for the two identifiers: ahmadlin and contEntropy was that I was considering using other estimates, other than that of Ahmad and Lin. Thus contEntropy may change drastically while ahmadlin should not. I am not sure how useful it would be to have more than one estimate in a published version. But, well, if some scipy developers want to push some of it into scipy, I won't object as long as I am consulted. I am not, however, going to find the right channel to make the proposal myself in the foreseeable future. > Was the outlier trimming enough to solve your problem with estimating > a kernel density, or did you try also try other kernels? Actually, the effect of the outliers was extreme values for the bandwidth. Thus, using consistent bandwidth in the three calculations was sufficient to give a plausible result. In the end, I kept the outliers. As I mentioned, the purpose was feature selection for machine learning. The features with the outliers used to show up with implausibly high heuristics. With the new approach to bandwidth, they show up as worthless, which they can be confirmed to be. Thus the problem is gone, and I need to find more trouble before I know anything more about outliers. -- :-- Hans Georg From k-assem84 at hotmail.com Fri Apr 29 04:37:45 2011 From: k-assem84 at hotmail.com (suzana8447) Date: Fri, 29 Apr 2011 01:37:45 -0700 (PDT) Subject: [SciPy-User] [SciPy-user] fit program. Message-ID: <31503548.post@talk.nabble.com> Hello, I am using right now a program called least square root fit (python language) and I am fitting a function to some data. But I urgently need to fit also the first and second derivatives of the same function at the same time to some data. However I do not know how to deal with. Can you help me. Thanks in advance. -- View this message in context: http://old.nabble.com/fit-program.-tp31503548p31503548.html Sent from the Scipy-User mailing list archive at Nabble.com. From k-assem84 at hotmail.com Fri Apr 29 10:02:40 2011 From: k-assem84 at hotmail.com (suzana8447) Date: Fri, 29 Apr 2011 07:02:40 -0700 (PDT) Subject: [SciPy-User] [SciPy-user] fit program. Message-ID: <31503548.post@talk.nabble.com> Hello, I am using a program called least square root fit (python language) and I am fitting a function to some given data. But I urgently need to fit also the first and second derivatives of the same function (I have these expressions analytically) at the same time to some data. However I do not know how to deal with this. Can one help me. Thanks in advance. -- View this message in context: http://old.nabble.com/fit-program.-tp31503548p31503548.html Sent from the Scipy-User mailing list archive at Nabble.com. From charlesr.harris at gmail.com Sat Apr 30 15:59:09 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 30 Apr 2011 13:59:09 -0600 Subject: [SciPy-User] [SciPy-user] fit program. In-Reply-To: <31503548.post@talk.nabble.com> References: <31503548.post@talk.nabble.com> Message-ID: On Fri, Apr 29, 2011 at 2:37 AM, suzana8447 wrote: > > Hello, > > I am using right now a program called least square root fit (python > language) and I am fitting a function to some data. > But I urgently need to fit also the first and second derivatives of the > same > function at the same time to some data. However I do not know how to deal > with. Can you help me. > Thanks in advance. > > Probably, but you need to be a bit more precise about what you are trying to do. Also, can you supply a link to the program you are using? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sat Apr 30 16:19:50 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 30 Apr 2011 22:19:50 +0200 Subject: [SciPy-User] ANN: Numpy 1.6.0 release candidate 1 Message-ID: Hi, I am pleased to announce the availability of the first release candidate of NumPy 1.6.0. If no new problems are reported, the final release will be in one week. Sources and binaries can be found at http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc1/ For (preliminary) release notes see below. Enjoy, Ralf ========================= NumPy 1.6.0 Release Notes ========================= This release includes several new features as well as numerous bug fixes and improved documentation. It is backward compatible with the 1.5.0 release, and supports Python 2.4 - 2.7 and 3.1 - 3.2. Highlights ========== * Re-introduction of datetime dtype support to deal with dates in arrays. * A new 16-bit floating point type. * A new iterator, which improves performance of many functions. New features ============ New 16-bit floating point type ------------------------------ This release adds support for the IEEE 754-2008 binary16 format, available as the data type ``numpy.half``. Within Python, the type behaves similarly to `float` or `double`, and C extensions can add support for it with the exposed half-float API. New iterator ------------ A new iterator has been added, replacing the functionality of the existing iterator and multi-iterator with a single object and API. This iterator works well with general memory layouts different from C or Fortran contiguous, and handles both standard NumPy and customized broadcasting. The buffering, automatic data type conversion, and optional output parameters, offered by ufuncs but difficult to replicate elsewhere, are now exposed by this iterator. Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` ------------------------------------------------------------------------- Extend the number of polynomials available in the polynomial package. In addition, a new ``window`` attribute has been added to the classes in order to specify the range the ``domain`` maps to. This is mostly useful for the Laguerre, Hermite, and HermiteE polynomials whose natural domains are infinite and provides a more intuitive way to get the correct mapping of values without playing unnatural tricks with the domain. Fortran assumed shape array and size function support in ``numpy.f2py`` ----------------------------------------------------------------------- F2py now supports wrapping Fortran 90 routines that use assumed shape arrays. Before such routines could be called from Python but the corresponding Fortran routines received assumed shape arrays as zero length arrays which caused unpredicted results. Thanks to Lorenz H?depohl for pointing out the correct way to interface routines with assumed shape arrays. In addition, f2py interprets Fortran expression ``size(array, dim)`` as ``shape(array, dim-1)`` which makes it possible to automatically wrap Fortran routines that use two argument ``size`` function in dimension specifications. Before users were forced to apply this mapping manually. Other new functions ------------------- ``numpy.ravel_multi_index`` : Converts a multi-index tuple into an array of flat indices, applying boundary modes to the indices. ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the Einstein summation convention, many common multi-dimensional array operations can be represented in a simple fashion. This function provides a way compute such summations. ``numpy.count_nonzero`` : Counts the number of non-zero elements in an array. ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions expose the underlying type promotion used by the ufuncs and other operations to determine the types of outputs. These improve upon the ``numpy.common_type`` and ``numpy.mintypecode`` which provide similar functionality but do not match the ufunc implementation. Changes ======= Changes and improvements in the numpy core ------------------------------------------ ``default error handling`` -------------------------- The default error handling has been change from ``print`` to ``warn`` for all except for ``underflow``, which remains as ``ignore``. ``numpy.distutils`` ------------------- Several new compilers are supported for building Numpy: the Portland Group Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel C compiler on Linux. ``numpy.testing`` ----------------- The testing framework gained ``numpy.testing.assert_allclose``, which provides a more convenient way to compare floating point arrays than `assert_almost_equal`, `assert_approx_equal` and `assert_array_almost_equal`. ``C API`` --------- In addition to the APIs for the new iterator and half data type, a number of other additions have been made to the C API. The type promotion mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration ``NPY_CASTING`` has been added which controls what types of casts are permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` and ``PyArray_CanCastTypeTo``. A more flexible way to handle conversion of arbitrary python objects into arrays is exposed by ``PyArray_GetArrayParamsFromObject``. Deprecated features =================== The "normed" keyword in ``numpy.histogram`` is deprecated. Its functionality will be replaced by the new "density" keyword. Removed features ================ ``numpy.fft`` ------------- The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, which were aliases for the same functions without the 'e' in the name, were removed. ``numpy.memmap`` ---------------- The `sync()` and `close()` methods of memmap were removed. Use `flush()` and "del memmap" instead. ``numpy.lib`` ------------- The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. ``numpy.ma`` ------------ Several deprecated items were removed from the ``numpy.ma`` module:: * ``numpy.ma.MaskedArray`` "raw_data" method * ``numpy.ma.MaskedArray`` constructor "flag" keyword * ``numpy.ma.make_mask`` "flag" keyword * ``numpy.ma.allclose`` "fill_value" keyword ``numpy.distutils`` ------------------- The ``numpy.get_numpy_include`` function was removed, use ``numpy.get_include`` instead. From charlesr.harris at gmail.com Sat Apr 30 18:54:06 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 30 Apr 2011 16:54:06 -0600 Subject: [SciPy-User] ANN: Numpy 1.6.0 release candidate 1 In-Reply-To: References: Message-ID: On Sat, Apr 30, 2011 at 2:19 PM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the availability of the first release > candidate of NumPy 1.6.0. If no new problems are reported, the final > release will be in one week. > > Sources and binaries can be found at > http://sourceforge.net/projects/numpy/files/NumPy/1.6.0rc1/ > For (preliminary) release notes see below. > > Enjoy, > Ralf > > > ========================= > NumPy 1.6.0 Release Notes > ========================= > > This release includes several new features as well as numerous bug fixes > and > improved documentation. It is backward compatible with the 1.5.0 release, > and > supports Python 2.4 - 2.7 and 3.1 - 3.2. > > > Highlights > ========== > > * Re-introduction of datetime dtype support to deal with dates in arrays. > > * A new 16-bit floating point type. > > * A new iterator, which improves performance of many functions. > > > New features > ============ > > New 16-bit floating point type > ------------------------------ > > This release adds support for the IEEE 754-2008 binary16 format, available > as > the data type ``numpy.half``. Within Python, the type behaves similarly to > `float` or `double`, and C extensions can add support for it with the > exposed > half-float API. > > > New iterator > ------------ > > A new iterator has been added, replacing the functionality of the > existing iterator and multi-iterator with a single object and API. > This iterator works well with general memory layouts different from > C or Fortran contiguous, and handles both standard NumPy and > customized broadcasting. The buffering, automatic data type > conversion, and optional output parameters, offered by > ufuncs but difficult to replicate elsewhere, are now exposed by this > iterator. > > > Legendre, Laguerre, Hermite, HermiteE polynomials in ``numpy.polynomial`` > ------------------------------------------------------------------------- > > Extend the number of polynomials available in the polynomial package. In > addition, a new ``window`` attribute has been added to the classes in > order to specify the range the ``domain`` maps to. This is mostly useful > for the Laguerre, Hermite, and HermiteE polynomials whose natural domains > are infinite and provides a more intuitive way to get the correct mapping > of values without playing unnatural tricks with the domain. > > > Fortran assumed shape array and size function support in ``numpy.f2py`` > ----------------------------------------------------------------------- > > F2py now supports wrapping Fortran 90 routines that use assumed shape > arrays. Before such routines could be called from Python but the > corresponding Fortran routines received assumed shape arrays as zero > length arrays which caused unpredicted results. Thanks to Lorenz > H?depohl for pointing out the correct way to interface routines with > assumed shape arrays. > > In addition, f2py interprets Fortran expression ``size(array, dim)`` > as ``shape(array, dim-1)`` which makes it possible to automatically > wrap Fortran routines that use two argument ``size`` function in > dimension specifications. Before users were forced to apply this > mapping manually. > > > Other new functions > ------------------- > > ``numpy.ravel_multi_index`` : Converts a multi-index tuple into > an array of flat indices, applying boundary modes to the indices. > > ``numpy.einsum`` : Evaluate the Einstein summation convention. Using the > Einstein summation convention, many common multi-dimensional array > operations > can be represented in a simple fashion. This function provides a way > compute > such summations. > > ``numpy.count_nonzero`` : Counts the number of non-zero elements in an > array. > > ``numpy.result_type`` and ``numpy.min_scalar_type`` : These functions > expose > the underlying type promotion used by the ufuncs and other operations to > determine the types of outputs. These improve upon the > ``numpy.common_type`` > and ``numpy.mintypecode`` which provide similar functionality but do > not match the ufunc implementation. > > > Changes > ======= > > Changes and improvements in the numpy core > ------------------------------------------ > > ``default error handling`` > -------------------------- > > The default error handling has been change from ``print`` to ``warn`` for > all except for ``underflow``, which remains as ``ignore``. > > > ``numpy.distutils`` > ------------------- > > Several new compilers are supported for building Numpy: the Portland Group > Fortran compiler on OS X, the PathScale compiler suite and the 64-bit Intel > C > compiler on Linux. > > > ``numpy.testing`` > ----------------- > > The testing framework gained ``numpy.testing.assert_allclose``, which > provides > a more convenient way to compare floating point arrays than > `assert_almost_equal`, `assert_approx_equal` and > `assert_array_almost_equal`. > > > ``C API`` > --------- > > In addition to the APIs for the new iterator and half data type, a number > of other additions have been made to the C API. The type promotion > mechanism used by ufuncs is exposed via ``PyArray_PromoteTypes``, > ``PyArray_ResultType``, and ``PyArray_MinScalarType``. A new enumeration > ``NPY_CASTING`` has been added which controls what types of casts are > permitted. This is used by the new functions ``PyArray_CanCastArrayTo`` > and ``PyArray_CanCastTypeTo``. A more flexible way to handle > conversion of arbitrary python objects into arrays is exposed by > ``PyArray_GetArrayParamsFromObject``. > > > Deprecated features > =================== > > The "normed" keyword in ``numpy.histogram`` is deprecated. Its > functionality > will be replaced by the new "density" keyword. > > > Removed features > ================ > > ``numpy.fft`` > ------------- > > The functions `refft`, `refft2`, `refftn`, `irefft`, `irefft2`, `irefftn`, > which were aliases for the same functions without the 'e' in the name, were > removed. > > > ``numpy.memmap`` > ---------------- > > The `sync()` and `close()` methods of memmap were removed. Use `flush()` > and > "del memmap" instead. > > > ``numpy.lib`` > ------------- > > The deprecated functions ``numpy.unique1d``, ``numpy.setmember1d``, > ``numpy.intersect1d_nu`` and ``numpy.lib.ufunclike.log2`` were removed. > > > ``numpy.ma`` > ------------ > > Several deprecated items were removed from the ``numpy.ma`` module:: > > * ``numpy.ma.MaskedArray`` "raw_data" method > * ``numpy.ma.MaskedArray`` constructor "flag" keyword > * ``numpy.ma.make_mask`` "flag" keyword > * ``numpy.ma.allclose`` "fill_value" keyword > > > ``numpy.distutils`` > ------------------- > > The ``numpy.get_numpy_include`` function was removed, use > ``numpy.get_include`` > instead. > ________ Small nit, the file names have c1 instead of rc1. I don't think it matters enough to change at the point. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: