From kwgoodman at gmail.com  Mon Feb  1 00:07:36 2010
From: kwgoodman at gmail.com (Keith Goodman)
Date: Sun, 31 Jan 2010 21:07:36 -0800
Subject: [SciPy-User] all() and any() with array-like input
In-Reply-To: <c048da1c1001312039l75db94f0h31db62867def9355@mail.gmail.com>
References: <c048da1c1001312039l75db94f0h31db62867def9355@mail.gmail.com>
Message-ID: <f4f93d421001312107p24ef9f51i9aaeddf99cdf7ba3@mail.gmail.com>

On Sun, Jan 31, 2010 at 8:39 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> I'm sure this has come up before, but it's difficult to search of any
> and all... ?This is tripping me up, but maybe there is just something
> I am missing.
>
> In [1]: import numpy as np
>
> In [2]: X = [.2,.2,.2,.2,.2]
>
> In [3]: np.all(X <= 1)
> Out[3]: False
>
> In [4]: np.all(X >= 0)
> Out[4]: True
>
> In [5]: np.all(np.asarray(X) <= 1)
> Out[5]: True
>
> In [6]: np.any(X>1)
> Out[6]: True
>
> I guess it's simple enough to use asarray, but I was just curious what
> drives this behavior since the docs indicate that it should work with
> array-like structures.

"X <= 1", where X is a list, is not array-like, it is a bool:

>>> X <= 1
False

So I guess it is a python thing. These are weird:

>>> [] <= 1
False
>>> {} < []
True
>>> {} > []
False


From pnorthug at gmail.com  Mon Feb  1 01:30:24 2010
From: pnorthug at gmail.com (Paul)
Date: Sun, 31 Jan 2010 22:30:24 -0800 (PST)
Subject: [SciPy-User] circulant matrices
Message-ID: <886b6e61-562c-4ce7-a8b2-a6ad9c29b4eb@q2g2000pre.googlegroups.com>

I would like to find,

argmin_c   norm( x - dot(phi, c), 2)

where x, c are vectors and phi is a circulant matrix. Is there a way
to do this with a built-in numpy or scipy function? LAPACK doesn't
seem to have a circulant matrix type. There is a Fortran library
NAPACK on netlib.org that has a circulant matrix type though I don't
know anything about it. I guess I could try to add these functions
with f2py.

I tried to solve this problem using a naive deconvolution approach
with fft's but it's not robust. I guess I have to do some more
research. I can't really afford to treat phi as anything other than
circulant (like a general banded or dense matrix) as it has to be
fast.

Any pointers would be appreciated.

Thanks,
P?l


From warren.weckesser at enthought.com  Mon Feb  1 02:53:58 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Mon, 01 Feb 2010 01:53:58 -0600
Subject: [SciPy-User] circulant matrices
In-Reply-To: <886b6e61-562c-4ce7-a8b2-a6ad9c29b4eb@q2g2000pre.googlegroups.com>
References: <886b6e61-562c-4ce7-a8b2-a6ad9c29b4eb@q2g2000pre.googlegroups.com>
Message-ID: <4B668896.6000101@enthought.com>

Paul wrote:
> I would like to find,
>
> argmin_c   norm( x - dot(phi, c), 2)
>
> where x, c are vectors and phi is a circulant matrix. Is there a way
> to do this with a built-in numpy or scipy function? LAPACK doesn't
> seem to have a circulant matrix type. There is a Fortran library
> NAPACK on netlib.org that has a circulant matrix type though I don't
> know anything about it. I guess I could try to add these functions
> with f2py.
>
> I tried to solve this problem using a naive deconvolution approach
> with fft's but it's not robust. I guess I have to do some more
> research. I can't really afford to treat phi as anything other than
> circulant (like a general banded or dense matrix) as it has to be
> fast.
>
>   


Is phi singular?  If not, then wouldn't you just solve dot(phi,c) = x for c?

Based on the wikipedia article 
http://en.wikipedia.org/wiki/Circulant_matrix,
I implemented a quick test of the FFT method--see the attached script.  
(Sorry,
my notation is the usual Ax=b instead of phi*c=x.)  It seems to work, and it
is much faster than linalg.solve(), but the script is the full extent of my
experience with circulant matrices (i.e. probably even more naive than
what you have already tried), so there are probably details that I am 
missing.

Warren


> Any pointers would be appreciated.
>
> Thanks,
> P?l
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: circtest.py
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100201/abf4faed/attachment.ksh>

From pnorthug at gmail.com  Mon Feb  1 17:48:58 2010
From: pnorthug at gmail.com (Paul)
Date: Mon, 1 Feb 2010 14:48:58 -0800 (PST)
Subject: [SciPy-User] circulant matrices
In-Reply-To: <4B668896.6000101@enthought.com>
References: <886b6e61-562c-4ce7-a8b2-a6ad9c29b4eb@q2g2000pre.googlegroups.com>
	<4B668896.6000101@enthought.com>
Message-ID: <92215163-83bf-4ecd-bd10-361ff4efda91@b9g2000pri.googlegroups.com>

On Jan 31, 11:53?pm, Warren Weckesser <warren.weckes... at enthought.com>
wrote:
> Paul wrote:
> > I would like to find,
>
> > argmin_c ? norm( x - dot(phi, c), 2)
>
> > where x, c are vectors and phi is a circulant matrix. Is there a way
> > to do this with a built-in numpy or scipy function? LAPACK doesn't
> > seem to have a circulant matrix type. There is a Fortran library
> > NAPACK on netlib.org that has a circulant matrix type though I don't
> > know anything about it. I guess I could try to add these functions
> > with f2py.
>
> > I tried to solve this problem using a naive deconvolution approach
> > with fft's but it's not robust. I guess I have to do some more
> > research. I can't really afford to treat phi as anything other than
> > circulant (like a general banded or dense matrix) as it has to be
> > fast.
>
> Is phi singular? ?If not, then wouldn't you just solve dot(phi,c) = x for c?

Thank you for your example code! It helped me realize that I have been
doing something wrong.

phi is not singular and you are right that you can solve dot(phi,c) =
x directly, with the fast implementation that you provided. I am
guessing phi not being singular also means fc = fft(c) has no zeros.

My mistake is that my phi is not exactly circulant, that is, it has
several additional rows. That is why I thought my problem is
overdetermined and required an optimization. I am representing a
correlation of a vector phi with c (where len(phi) << len(c) and I
need to correct my padding so that I can solve it in the way you
suggested. In the full problem, the correlation is actually 2-d,
complicating things a little further.

>
> Based on the wikipedia articlehttp://en.wikipedia.org/wiki/Circulant_matrix,
> I implemented a quick test of the FFT method--see the attached script. ?
> (Sorry,
> my notation is the usual Ax=b instead of phi*c=x.) ?It seems to work, and it
> is much faster than linalg.solve(), but the script is the full extent of my
> experience with circulant matrices (i.e. probably even more naive than
> what you have already tried), so there are probably details that I am
> missing.

Thanks for the reference and the code again.

>
> Warren
>
> > Any pointers would be appreciated.
>
> > Thanks,
> > P?l
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-U... at scipy.org
> >http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
> [circtest.py< 1K ]import time
> import numpy as np
> from scipy import fft, ifft
>
> # Solve A*x = b when A is circulant.
>
> # Create a random circulant matrix.
> n = 512
> c = 2*np.random.random(n) - 1.0
> A = np.empty((n,n), dtype=float)
> for k in range(n):
> ? ? A[:,k] = np.roll(c,k)
>
> # Create the b vector.
> b = np.arange(n)
>
> # First use linalg.solve() for comparison.
> t = time.time()
> x1 = np.linalg.solve(A,b)
> dt1 = time.time() - t
> print "Using solve: %.3e seconds" % dt1
>
> # Next use the FFT.
> # Note that we only need c here; A was created to be used in solve().
> t = time.time()
> fb = fft(b)
> fc = fft(c)
> x2 = ifft(fb / fc)
> dt2 = time.time() - t
> print "Using FFT: ? %.3e seconds" % dt2
>
> print "Speedup: %.3g" % (dt1/dt2)
>
> # Verify that the solutions are the same.
> assert np.allclose(x1, x2.real)
>

P?l.


From josephsmidt at gmail.com  Mon Feb  1 21:19:31 2010
From: josephsmidt at gmail.com (josephsmidt at gmail.com)
Date: Tue, 02 Feb 2010 02:19:31 +0000
Subject: [SciPy-User] Theological Implications Of Mainstream Cosmological
	Ideas.
Message-ID: <0016e64b0bce6ad6c9047e94bb4f@google.com>

Third post.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100202/ebbf5dd8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Theological Implications Of Mainstream Cosmologica.doc
Type: application/msword
Size: 16384 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100202/ebbf5dd8/attachment.doc>

From josephsmidt at gmail.com  Mon Feb  1 21:43:08 2010
From: josephsmidt at gmail.com (Joseph Smidt)
Date: Mon, 1 Feb 2010 18:43:08 -0800
Subject: [SciPy-User] Theological Implications Of Mainstream
	Cosmological Ideas.
In-Reply-To: <0016e64b0bce6ad6c9047e94bb4f@google.com>
References: <0016e64b0bce6ad6c9047e94bb4f@google.com>
Message-ID: <142682e11002011843r4c0ef660wb8ba4b4b3e56dd42@mail.gmail.com>

Opps, wrong people.

On Mon, Feb 1, 2010 at 6:19 PM, josephsmidt at gmail.com
<josephsmidt at gmail.com> wrote:
>
> Third post.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From aisaac at american.edu  Tue Feb  2 10:23:10 2010
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 02 Feb 2010 10:23:10 -0500
Subject: [SciPy-User] sample from kernel density estimate
Message-ID: <4B68435E.9020105@american.edu>

I have a kernel density estimate from scipy.stats.gaussian_kde.
What's the best way to sample from it?  (Not from the underlying data.)

Thanks,
Alan Isaac


From josef.pktd at gmail.com  Tue Feb  2 10:31:11 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 2 Feb 2010 10:31:11 -0500
Subject: [SciPy-User] sample from kernel density estimate
In-Reply-To: <4B68435E.9020105@american.edu>
References: <4B68435E.9020105@american.edu>
Message-ID: <1cd32cbb1002020731k3561e714rfabce1383951a9f6@mail.gmail.com>

On Tue, Feb 2, 2010 at 10:23 AM, Alan G Isaac <aisaac at american.edu> wrote:
> I have a kernel density estimate from scipy.stats.gaussian_kde.
> What's the best way to sample from it? ?(Not from the underlying data.)

I think stats.kde.gaussian_kde.resample  does it.
If I remember correctly, it samples from the underlying data and adds
a normal noise. I think I read somewhere that that is equivalent to
sampling from the kernel density directly.

Josef


>
> Thanks,
> Alan Isaac
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Tue Feb  2 10:52:07 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 2 Feb 2010 10:52:07 -0500
Subject: [SciPy-User] sample from kernel density estimate
In-Reply-To: <1cd32cbb1002020731k3561e714rfabce1383951a9f6@mail.gmail.com>
References: <4B68435E.9020105@american.edu>
	<1cd32cbb1002020731k3561e714rfabce1383951a9f6@mail.gmail.com>
Message-ID: <1cd32cbb1002020752u17a06dbex9ec44e92e045541@mail.gmail.com>

On Tue, Feb 2, 2010 at 10:31 AM,  <josef.pktd at gmail.com> wrote:
> On Tue, Feb 2, 2010 at 10:23 AM, Alan G Isaac <aisaac at american.edu> wrote:
>> I have a kernel density estimate from scipy.stats.gaussian_kde.
>> What's the best way to sample from it? ?(Not from the underlying data.)
>
> I think stats.kde.gaussian_kde.resample ?does it.
> If I remember correctly, it samples from the underlying data and adds
> a normal noise. I think I read somewhere that that is equivalent to
> sampling from the kernel density directly.

drawing a sample and running kstest will make a nice test case, it's
still missing in the test suite.
I will put it on my todo list.

Josef


>
> Josef
>
>
>>
>> Thanks,
>> Alan Isaac
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From aisaac at american.edu  Tue Feb  2 10:57:17 2010
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 02 Feb 2010 10:57:17 -0500
Subject: [SciPy-User] sample from kernel density estimate
In-Reply-To: <1cd32cbb1002020731k3561e714rfabce1383951a9f6@mail.gmail.com>
References: <4B68435E.9020105@american.edu>
	<1cd32cbb1002020731k3561e714rfabce1383951a9f6@mail.gmail.com>
Message-ID: <4B684B5D.7040507@american.edu>

> On Tue, Feb 2, 2010 at 10:23 AM, Alan G Isaac<aisaac at american.edu>  wrote:
>> I have a kernel density estimate from scipy.stats.gaussian_kde.
>> What's the best way to sample from it?  (Not from the underlying data.)


On 2/2/2010 10:31 AM, josef.pktd at gmail.com wrote:
> I think stats.kde.gaussian_kde.resample  does it.
> If I remember correctly, it samples from the underlying data and adds
> a normal noise. I think I read somewhere that that is equivalent to
> sampling from the kernel density directly.


That appears to be correct.
I was using the online docs,
which did not show this method.
However, `help` does show it::

         |  resample(self, size=None)
         |      Randomly sample a dataset from the estimated pdf.
         |
         |      Parameters
         |      ----------
         |      size : int, optional
         |          The number of samples to draw.
         |          If not provided, then the size is the same as the underlying
         |          dataset.
         |
         |      Returns
         |      -------
         |      dataset : (self.d, size)-array
         |          sampled dataset

Thanks,
Alan



From josef.pktd at gmail.com  Tue Feb  2 11:08:03 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 2 Feb 2010 11:08:03 -0500
Subject: [SciPy-User] sample from kernel density estimate
In-Reply-To: <4B684B5D.7040507@american.edu>
References: <4B68435E.9020105@american.edu>
	<1cd32cbb1002020731k3561e714rfabce1383951a9f6@mail.gmail.com>
	<4B684B5D.7040507@american.edu>
Message-ID: <1cd32cbb1002020808k8de6b47v4fb98998581e40f0@mail.gmail.com>

On Tue, Feb 2, 2010 at 10:57 AM, Alan G Isaac <aisaac at american.edu> wrote:
>> On Tue, Feb 2, 2010 at 10:23 AM, Alan G Isaac<aisaac at american.edu> ?wrote:
>>> I have a kernel density estimate from scipy.stats.gaussian_kde.
>>> What's the best way to sample from it? ?(Not from the underlying data.)
>
>
> On 2/2/2010 10:31 AM, josef.pktd at gmail.com wrote:
>> I think stats.kde.gaussian_kde.resample ?does it.
>> If I remember correctly, it samples from the underlying data and adds
>> a normal noise. I think I read somewhere that that is equivalent to
>> sampling from the kernel density directly.
>
>
> That appears to be correct.
> I was using the online docs,
> which did not show this method.

I added it to the list of methods in the docs.

Josef

> However, `help` does show it::
>
> ? ? ? ? | ?resample(self, size=None)
> ? ? ? ? | ? ? ?Randomly sample a dataset from the estimated pdf.
> ? ? ? ? |
> ? ? ? ? | ? ? ?Parameters
> ? ? ? ? | ? ? ?----------
> ? ? ? ? | ? ? ?size : int, optional
> ? ? ? ? | ? ? ? ? ?The number of samples to draw.
> ? ? ? ? | ? ? ? ? ?If not provided, then the size is the same as the underlying
> ? ? ? ? | ? ? ? ? ?dataset.
> ? ? ? ? |
> ? ? ? ? | ? ? ?Returns
> ? ? ? ? | ? ? ?-------
> ? ? ? ? | ? ? ?dataset : (self.d, size)-array
> ? ? ? ? | ? ? ? ? ?sampled dataset
>
> Thanks,
> Alan
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From nolambar at gmail.com  Tue Feb  2 12:10:51 2010
From: nolambar at gmail.com (Ignacio Vergara)
Date: Tue, 2 Feb 2010 14:10:51 -0300
Subject: [SciPy-User] Weave and OOP
Message-ID: <10f88f6d1002020910m1253705dn6473bd38b1b3d7ca@mail.gmail.com>

Hi

I'm trying to use weave inside a class, but it throws me a compilation
error.

I would like to know if there is any kind of workaround or if it's a
limitation regarding weave

Tanks in advance

Ignacio Vergara
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100202/005e88d8/attachment.html>

From david_baddeley at yahoo.com.au  Tue Feb  2 21:32:42 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Tue, 2 Feb 2010 18:32:42 -0800 (PST)
Subject: [SciPy-User] Float32 FFT
Message-ID: <615785.75542.qm@web33004.mail.mud.yahoo.com>

Does anyone know if there is a way of doing single precision ffts (more explicitly n-dimensional ffts using fftn) in scipy/numpy? The standard functions all seem to force a conversion to double precision and with large (~1000x1000x50) images the memory starts to add up. 

I'm running numpy 1.2.1, scipy 0.7.0, python 2.6 on linux x64.

Thanks,
David


      


From david at silveregg.co.jp  Tue Feb  2 21:42:07 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Wed, 03 Feb 2010 11:42:07 +0900
Subject: [SciPy-User] Float32 FFT
In-Reply-To: <615785.75542.qm@web33004.mail.mud.yahoo.com>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>
Message-ID: <4B68E27F.6000300@silveregg.co.jp>

David Baddeley wrote:
> Does anyone know if there is a way of doing single precision ffts (more explicitly n-dimensional ffts using fftn) in scipy/numpy? The standard functions all seem to force a conversion to double precision and with large (~1000x1000x50) images the memory starts to add up. 

It is supported in scipy trunk (upcoming 0.8.0). It is transparent, that 
is if your input is single precision, the output is as well.

cheers,

David


From seb.haase at gmail.com  Wed Feb  3 03:42:54 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Wed, 3 Feb 2010 09:42:54 +0100
Subject: [SciPy-User] Float32 FFT
In-Reply-To: <615785.75542.qm@web33004.mail.mud.yahoo.com>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>
Message-ID: <bc657ead1002030042u23e05d5vc63bc72030f2318c@mail.gmail.com>

On Wed, Feb 3, 2010 at 3:32 AM, David Baddeley
<david_baddeley at yahoo.com.au> wrote:
> Does anyone know if there is a way of doing single precision ffts (more explicitly n-dimensional ffts using fftn) in scipy/numpy? The standard functions all seem to force a conversion to double precision and with large (~1000x1000x50) images the memory starts to add up.
>
> I'm running numpy 1.2.1, scipy 0.7.0, python 2.6 on linux x64.
>
> Thanks,
> David
>
I have an fftw2 wrapper (SWIGged) in Priithon (see googlecode).
It is based on Python 2.5.  A kind of "super package" binary version
(also for 64 bit Linux) is downloadable.
(FFTW is GPL). (The wrapper could be BSD).

Regards,
Sebastian Haase


From david_baddeley at yahoo.com.au  Wed Feb  3 05:43:40 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Wed, 3 Feb 2010 02:43:40 -0800 (PST)
Subject: [SciPy-User] Float32 FFT
In-Reply-To: <bc657ead1002030042u23e05d5vc63bc72030f2318c@mail.gmail.com>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>
	<bc657ead1002030042u23e05d5vc63bc72030f2318c@mail.gmail.com>
Message-ID: <668054.42551.qm@web33008.mail.mud.yahoo.com>

Thanks David & Sebastian,

will look into the scipy trunk & priithon options. Alternatively might find the motivation to dust off some old fftw3 based code I've got floating round which should in theory be a little faster (I'm doing multiple FFTs of the same size, with the potential to do the memory allocation and plan generation just once at the start).

cheers,
David


----- Original Message ----
From: Sebastian Haase <seb.haase at gmail.com>
To: David Baddeley <david_baddeley at yahoo.com.au>; SciPy Users List <scipy-user at scipy.org>
Sent: Wed, 3 February, 2010 9:42:54 PM
Subject: Re: [SciPy-User] Float32 FFT

On Wed, Feb 3, 2010 at 3:32 AM, David Baddeley
<david_baddeley at yahoo.com.au> wrote:
> Does anyone know if there is a way of doing single precision ffts (more explicitly n-dimensional ffts using fftn) in scipy/numpy? The standard functions all seem to force a conversion to double precision and with large (~1000x1000x50) images the memory starts to add up.
>
> I'm running numpy 1.2.1, scipy 0.7.0, python 2.6 on linux x64.
>
> Thanks,
> David
>
I have an fftw2 wrapper (SWIGged) in Priithon (see googlecode).
It is based on Python 2.5.  A kind of "super package" binary version
(also for 64 bit Linux) is downloadable.
(FFTW is GPL). (The wrapper could be BSD).

Regards,
Sebastian Haase



      


From ryanlists at gmail.com  Wed Feb  3 09:00:36 2010
From: ryanlists at gmail.com (Ryan Krauss)
Date: Wed, 3 Feb 2010 08:00:36 -0600
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>
	<c5b438121001281239y6b31691dq878e5ef340df96fd@mail.gmail.com>
	<4B623ECD.4040403@enthought.com>
	<1cd32cbb1001281757x14b7b581xba02f4ae21d3acfb@mail.gmail.com>
	<4B6256F5.3050802@enthought.com>
	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>
	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>
Message-ID: <c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>

FYI, I am quite happy with passing in an hmax value.  I basically
copied and pasted lsim2 from signal.ltisys and adapted it just a
little to make it a method of my derived class.  Then I added the hmas
kwarg that gets passed to odeint.

Is there any reason not to allow the user to pass in a kwargs to lsim2
that gets passed to odeint?

On Fri, Jan 29, 2010 at 6:44 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
> Thanks to Warren and Josef for their time and thoughts. ?I feel like I
> now understand the underlying problem and have some good options to
> solve my short term issues (I assigned the project last night and they
> need to be able to start working on it immediately). ?I actually use a
> TransferFunction class that derives from ltisys. ?I could override its
> lsim2 method to try out some of these solutions quickly and fairly
> easily.
>
> Ryan
>
> On Thu, Jan 28, 2010 at 10:00 PM, ?<josef.pktd at gmail.com> wrote:
>> On Thu, Jan 28, 2010 at 10:33 PM, Warren Weckesser
>> <warren.weckesser at enthought.com> wrote:
>>> josef.pktd at gmail.com wrote:
>>>> On Thu, Jan 28, 2010 at 8:50 PM, Warren Weckesser
>>>> <warren.weckesser at enthought.com> wrote:
>>>>
>>>>> Ryan,
>>>>>
>>>>> The problem is that the ODE solver used by lsim2 is too good. :)
>>>>>
>>>>> It uses scipy.integrate.odeint, which in turn uses the Fortran library
>>>>> LSODA. ?Like any good solver, LSODA is an adaptive solver--it adjusts its
>>>>> step size to be as large as possible while keeping estimates of the error
>>>>> bounded. ?For the problem you are solving, with initial condition 0, the
>>>>> exact solution is initially exactly 0. ?This is such a nice smooth solution
>>>>> that the solver's step size quickly grows--so big, in fact, that it skips
>>>>> right over your pulse and never sees it.
>>>>>
>>>>> So how does it create all those intermediate points at the requested time
>>>>> values? ?It uses interpolation between the steps that it computed to create
>>>>> the solution values at the times that you requested. ?So using a finer grid
>>>>> of time values won't help. ?(If lsim2 gave you a hook into the parameters
>>>>> passed to odeint, you could set odeint's 'hmax' to a value smaller than your
>>>>> pulse width, which would force the solver to see the pulse. ?But there is no
>>>>> way to set that parameter from lsim2.)
>>>>>
>>>>
>>>> It's something what I suspected. I don't know much about odeint, but
>>>> do you think it would be useful to let lsim2 pass through some
>>>> parameters to odeint?
>>>>
>>>>
>>>
>>> Sounds useful to me. ?A simple implementation is an optional keyword
>>> argument that is a dict of odeint arguments. ? But this would almost
>>> certainly break if lsim2 were ever reimplemented with a different
>>> solver. ?So perhaps it should allow a common set of ODE solver
>>> parameters (e.g. absolute and relative error tolerances, max and min
>>> step sizes, others?).
>>>
>>> Perhaps this should wait until after the ODE solver redesign that is
>>> occasionally discussed:
>>> ? ?http://projects.scipy.org/scipy/wiki/OdeintRedesign
>>> Then the solver itself could be an optional argument to lsim2.
>>
>> I was just thinking of adding to the argument list a **kwds argument
>> that is directly passed on to whatever ODE solver is used. This should
>> be pretty flexible for any changes and be backwards compatible.
>>
>> I've seen and used it in a similar way for calls to optimization
>> routines, e.g. also optimize.curve_fit, does it. What are actually
>> valid keywords would depend on which function is called.
>>
>> (But I'm not a user of lsim, I'm just stealing some ideas from lti and
>> friends for time series analysis.)
>>
>> Josef
>>
>>
>>
>>
>>>
>>> Warren
>>>
>>>> Josef
>>>>
>>>>
>>>>
>>>>> The basic problem is you are passing in a discontinuous function to a solver
>>>>> that expects a smooth function. ?A better way to solve this problem is to
>>>>> explicitly account for the discontinuity. One possibility is the attached
>>>>> script.
>>>>>
>>>>> This is an excellent "learning opportunity" for your students on the hazards
>>>>> of numerical computing!
>>>>>
>>>>> Warren
>>>>>
>>>>>
>>>>> Ryan Krauss wrote:
>>>>>
>>>>>> I believe I have discovered a bug in signal.lsim2. ?I believe the
>>>>>> short attached script illustrates the problem. ?I was trying to
>>>>>> predict the response of a transfer function with a pure integrator:
>>>>>>
>>>>>> ? ? ? ? ? ? g
>>>>>> G = -------------
>>>>>> ? ? ? ? s(s+p)
>>>>>>
>>>>>> to a finite width pulse. ?lsim2 seems to handle the step response just
>>>>>> fine, but says that the pulse response is exactly 0.0 for the entire
>>>>>> time of the simulation. ?Obviously, this isn't the right answer.
>>>>>>
>>>>>> I am running scipy 0.7.0 and numpy 1.2.1 on Ubuntu 9.04, but I also
>>>>>> have the same problem on Windows running 0.7.1 and 1.4.0.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Ryan
>>>>>> ?------------------------------------------------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>> from pylab import *
>>>>> from scipy import signal
>>>>>
>>>>>
>>>>> g = 100.0
>>>>> p = 15.0
>>>>> G = signal.ltisys.lti(g, [1,p,0])
>>>>>
>>>>> t = arange(0, 1.0, 0.002)
>>>>> N = len(t)
>>>>>
>>>>> # u for the whole interval (not used in lsim2, only for plotting later).
>>>>> amp = 50.0
>>>>> u = zeros(N)
>>>>> k1 = 50
>>>>> k2 = 100
>>>>> u[k1:k2] = amp
>>>>>
>>>>> # Create input functions for each smooth interval. (This could be simpler,
>>>>> since u
>>>>> # is constant on each interval.)
>>>>> a = float(k1)/N
>>>>> b = float(k2)/N
>>>>> T1 = linspace(0, a, 201)
>>>>> u1 = zeros_like(T1)
>>>>> T2 = linspace(a, b, 201)
>>>>> u2 = amp*ones_like(T2)
>>>>> T3 = linspace(b, 1.0, 201)
>>>>> u3 = zeros_like(T3)
>>>>>
>>>>> # Solve on each interval; use the final value of one solution as the
>>>>> starting
>>>>> # point of the next solution.
>>>>> # (We could skip the first calculation, since we know the solution will be
>>>>> 0.)
>>>>> (t1, y1, x1) = signal.lsim2(G,u1,T1)
>>>>> (t2, y2, x2) = signal.lsim2(G, u2, T2, X0=x1[-1])
>>>>> (t3, y3, x3) = signal.lsim2(G, u3, T3, X0=x2[-1])
>>>>>
>>>>> figure(1)
>>>>> clf()
>>>>> plot(t, u, 'k', linewidth=3)
>>>>> plot(t1, y1, 'y', linewidth=3)
>>>>> plot(t2, y2, 'b', linewidth=3)
>>>>> plot(t3, y3, 'g', linewidth=3)
>>>>>
>>>>> show()
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From josef.pktd at gmail.com  Wed Feb  3 09:16:35 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 3 Feb 2010 09:16:35 -0500
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>
	<c5b438121001281239y6b31691dq878e5ef340df96fd@mail.gmail.com>
	<4B623ECD.4040403@enthought.com>
	<1cd32cbb1001281757x14b7b581xba02f4ae21d3acfb@mail.gmail.com>
	<4B6256F5.3050802@enthought.com>
	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>
	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>
	<c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>
Message-ID: <1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>

On Wed, Feb 3, 2010 at 9:00 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
> FYI, I am quite happy with passing in an hmax value. ?I basically
> copied and pasted lsim2 from signal.ltisys and adapted it just a
> little to make it a method of my derived class. ?Then I added the hmas
> kwarg that gets passed to odeint.
>
> Is there any reason not to allow the user to pass in a kwargs to lsim2
> that gets passed to odeint?

I don't see a reason why we cannot add a **kwargs, it should be
completely backwards compatible.
Can you file a ticket and add your adjusted version or a patch? And
even better, add your original example as a test case?

Josef


>
> On Fri, Jan 29, 2010 at 6:44 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>> Thanks to Warren and Josef for their time and thoughts. ?I feel like I
>> now understand the underlying problem and have some good options to
>> solve my short term issues (I assigned the project last night and they
>> need to be able to start working on it immediately). ?I actually use a
>> TransferFunction class that derives from ltisys. ?I could override its
>> lsim2 method to try out some of these solutions quickly and fairly
>> easily.
>>
>> Ryan
>>
>> On Thu, Jan 28, 2010 at 10:00 PM, ?<josef.pktd at gmail.com> wrote:
>>> On Thu, Jan 28, 2010 at 10:33 PM, Warren Weckesser
>>> <warren.weckesser at enthought.com> wrote:
>>>> josef.pktd at gmail.com wrote:
>>>>> On Thu, Jan 28, 2010 at 8:50 PM, Warren Weckesser
>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>
>>>>>> Ryan,
>>>>>>
>>>>>> The problem is that the ODE solver used by lsim2 is too good. :)
>>>>>>
>>>>>> It uses scipy.integrate.odeint, which in turn uses the Fortran library
>>>>>> LSODA. ?Like any good solver, LSODA is an adaptive solver--it adjusts its
>>>>>> step size to be as large as possible while keeping estimates of the error
>>>>>> bounded. ?For the problem you are solving, with initial condition 0, the
>>>>>> exact solution is initially exactly 0. ?This is such a nice smooth solution
>>>>>> that the solver's step size quickly grows--so big, in fact, that it skips
>>>>>> right over your pulse and never sees it.
>>>>>>
>>>>>> So how does it create all those intermediate points at the requested time
>>>>>> values? ?It uses interpolation between the steps that it computed to create
>>>>>> the solution values at the times that you requested. ?So using a finer grid
>>>>>> of time values won't help. ?(If lsim2 gave you a hook into the parameters
>>>>>> passed to odeint, you could set odeint's 'hmax' to a value smaller than your
>>>>>> pulse width, which would force the solver to see the pulse. ?But there is no
>>>>>> way to set that parameter from lsim2.)
>>>>>>
>>>>>
>>>>> It's something what I suspected. I don't know much about odeint, but
>>>>> do you think it would be useful to let lsim2 pass through some
>>>>> parameters to odeint?
>>>>>
>>>>>
>>>>
>>>> Sounds useful to me. ?A simple implementation is an optional keyword
>>>> argument that is a dict of odeint arguments. ? But this would almost
>>>> certainly break if lsim2 were ever reimplemented with a different
>>>> solver. ?So perhaps it should allow a common set of ODE solver
>>>> parameters (e.g. absolute and relative error tolerances, max and min
>>>> step sizes, others?).
>>>>
>>>> Perhaps this should wait until after the ODE solver redesign that is
>>>> occasionally discussed:
>>>> ? ?http://projects.scipy.org/scipy/wiki/OdeintRedesign
>>>> Then the solver itself could be an optional argument to lsim2.
>>>
>>> I was just thinking of adding to the argument list a **kwds argument
>>> that is directly passed on to whatever ODE solver is used. This should
>>> be pretty flexible for any changes and be backwards compatible.
>>>
>>> I've seen and used it in a similar way for calls to optimization
>>> routines, e.g. also optimize.curve_fit, does it. What are actually
>>> valid keywords would depend on which function is called.
>>>
>>> (But I'm not a user of lsim, I'm just stealing some ideas from lti and
>>> friends for time series analysis.)
>>>
>>> Josef
>>>
>>>
>>>
>>>
>>>>
>>>> Warren
>>>>
>>>>> Josef
>>>>>
>>>>>
>>>>>
>>>>>> The basic problem is you are passing in a discontinuous function to a solver
>>>>>> that expects a smooth function. ?A better way to solve this problem is to
>>>>>> explicitly account for the discontinuity. One possibility is the attached
>>>>>> script.
>>>>>>
>>>>>> This is an excellent "learning opportunity" for your students on the hazards
>>>>>> of numerical computing!
>>>>>>
>>>>>> Warren
>>>>>>
>>>>>>
>>>>>> Ryan Krauss wrote:
>>>>>>
>>>>>>> I believe I have discovered a bug in signal.lsim2. ?I believe the
>>>>>>> short attached script illustrates the problem. ?I was trying to
>>>>>>> predict the response of a transfer function with a pure integrator:
>>>>>>>
>>>>>>> ? ? ? ? ? ? g
>>>>>>> G = -------------
>>>>>>> ? ? ? ? s(s+p)
>>>>>>>
>>>>>>> to a finite width pulse. ?lsim2 seems to handle the step response just
>>>>>>> fine, but says that the pulse response is exactly 0.0 for the entire
>>>>>>> time of the simulation. ?Obviously, this isn't the right answer.
>>>>>>>
>>>>>>> I am running scipy 0.7.0 and numpy 1.2.1 on Ubuntu 9.04, but I also
>>>>>>> have the same problem on Windows running 0.7.1 and 1.4.0.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Ryan
>>>>>>> ?------------------------------------------------------------------------
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SciPy-User mailing list
>>>>>>> SciPy-User at scipy.org
>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>
>>>>>> from pylab import *
>>>>>> from scipy import signal
>>>>>>
>>>>>>
>>>>>> g = 100.0
>>>>>> p = 15.0
>>>>>> G = signal.ltisys.lti(g, [1,p,0])
>>>>>>
>>>>>> t = arange(0, 1.0, 0.002)
>>>>>> N = len(t)
>>>>>>
>>>>>> # u for the whole interval (not used in lsim2, only for plotting later).
>>>>>> amp = 50.0
>>>>>> u = zeros(N)
>>>>>> k1 = 50
>>>>>> k2 = 100
>>>>>> u[k1:k2] = amp
>>>>>>
>>>>>> # Create input functions for each smooth interval. (This could be simpler,
>>>>>> since u
>>>>>> # is constant on each interval.)
>>>>>> a = float(k1)/N
>>>>>> b = float(k2)/N
>>>>>> T1 = linspace(0, a, 201)
>>>>>> u1 = zeros_like(T1)
>>>>>> T2 = linspace(a, b, 201)
>>>>>> u2 = amp*ones_like(T2)
>>>>>> T3 = linspace(b, 1.0, 201)
>>>>>> u3 = zeros_like(T3)
>>>>>>
>>>>>> # Solve on each interval; use the final value of one solution as the
>>>>>> starting
>>>>>> # point of the next solution.
>>>>>> # (We could skip the first calculation, since we know the solution will be
>>>>>> 0.)
>>>>>> (t1, y1, x1) = signal.lsim2(G,u1,T1)
>>>>>> (t2, y2, x2) = signal.lsim2(G, u2, T2, X0=x1[-1])
>>>>>> (t3, y3, x3) = signal.lsim2(G, u3, T3, X0=x2[-1])
>>>>>>
>>>>>> figure(1)
>>>>>> clf()
>>>>>> plot(t, u, 'k', linewidth=3)
>>>>>> plot(t1, y1, 'y', linewidth=3)
>>>>>> plot(t2, y2, 'b', linewidth=3)
>>>>>> plot(t3, y3, 'g', linewidth=3)
>>>>>>
>>>>>> show()
>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From ryanlists at gmail.com  Wed Feb  3 11:28:49 2010
From: ryanlists at gmail.com (Ryan Krauss)
Date: Wed, 3 Feb 2010 10:28:49 -0600
Subject: [SciPy-User] lsoda vs. Coulomb friction
Message-ID: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>

I am trying to use odeint (i.e. lsoda) on a mechanical system that
involves a mass and friction modeled as a viscous term plus Coulomb
friction.  The discontinuity near 0 velocity makes lsoda mad.  It
complains that it has to do excess work (lazy algorithm :).  In spite
of its complaining, the result leads to fairly good agreement between
model and experiment.  But is there a better way to handle this?

This is the func I am passing to odeint (see attached example script
for the details):

def ydot_ol(x, t, u, C):
    m = C[0]
    b1 = C[1]
    b2 = C[2]
    y1 = x[0]
    y2 = x[1]
    ydot = zeros(2,)
    ydot[0] = y2
    ydot[1] = (u - b1*y2 - b2*sign(y2))/m
    return ydot

u is an external input.

lsoda doesn't complain until the system is essentially stopping.

Thanks,

Ryan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lsoda_vs_coulomb_friction.py
Type: text/x-python
Size: 854 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100203/fe4123b0/attachment.py>

From charlesr.harris at gmail.com  Wed Feb  3 15:16:38 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 3 Feb 2010 13:16:38 -0700
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
Message-ID: <e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>

On Wed, Feb 3, 2010 at 9:28 AM, Ryan Krauss <ryanlists at gmail.com> wrote:

> I am trying to use odeint (i.e. lsoda) on a mechanical system that
> involves a mass and friction modeled as a viscous term plus Coulomb
> friction.  The discontinuity near 0 velocity makes lsoda mad.  It
> complains that it has to do excess work (lazy algorithm :).  In spite
> of its complaining, the result leads to fairly good agreement between
> model and experiment.  But is there a better way to handle this?
>
>
You might try some of the other stiff solvers. The algorithms for stiff
equations tend to be implicit, i.e., they have to solve non-linear equations
and are inherently slower for that reason. Other than that I haven't any
suggestions.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100203/aa47a9f6/attachment.html>

From d.l.goldsmith at gmail.com  Wed Feb  3 16:00:19 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Wed, 3 Feb 2010 13:00:19 -0800
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
	<e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>
Message-ID: <45d1ab481002031300x78b5eb03hc74e7bf7cb2c8aae@mail.gmail.com>

On Wed, Feb 3, 2010 at 12:16 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Wed, Feb 3, 2010 at 9:28 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>
>> I am trying to use odeint (i.e. lsoda) on a mechanical system that
>> involves a mass and friction modeled as a viscous term plus Coulomb
>> friction.  The discontinuity near 0 velocity makes lsoda mad.  It
>> complains that it has to do excess work (lazy algorithm :).  In spite
>> of its complaining, the result leads to fairly good agreement between
>> model and experiment.  But is there a better way to handle this?
>>
>>
> You might try some of the other stiff solvers. The algorithms for stiff
> equations tend to be implicit, i.e., they have to solve non-linear equations
> and are inherently slower for that reason. Other than that I haven't any
> suggestions.
>
>
Hi, Ryan.  To elaborate on Chuck's reply: first, are you using
scipy.integrate.ode.ode or scipy.integrate.odepack.odeint?  If the former,
use vode and use its set_integrator() method with keyword argument 'method'
set equal to 'bdf'.  See, e.g.,
http://docs.scipy.org/scipy/docs/scipy.integrate.ode.ode/ for more
information.  If you're using the latter, you may want to switch to the
former, as odeint's docstring has this to say about stiff systems:

"   :
    :
Returns
 -------
    :
    :
 infodict : dict, only returned if full_output == True
     Dictionary containing additional output information
        :
        :
     'mused'  a vector of method indicators for each successful time step:
              1: adams (nonstiff), 2: bdf (stiff)"

Note that it's a _return_ value: apparently with odeint, you can't specify
which method to use, you can only "hear" which method the algorithm decided
to use for you.

HTH,

DG

Chuck
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100203/a59b54a2/attachment.html>

From kwgoodman at gmail.com  Wed Feb  3 16:24:29 2010
From: kwgoodman at gmail.com (Keith Goodman)
Date: Wed, 3 Feb 2010 13:24:29 -0800
Subject: [SciPy-User] [ANN] New package for manipulating labeled arrays
Message-ID: <f4f93d421002031324r1df44831ofa27375fb9ee8b2e@mail.gmail.com>

I am pleased to announce the first release of the la package, version 0.1.

The main class of the la package is a labeled array, larry. A larry consists
of a data array and a label list. The data array is stored as a NumPy array
and the label list as a list of lists.

larry has built-in methods such as movingsum, ranking, merge, shuffle,
zscore, demean, lag as well as typical Numpy methods like sum, max, std,
sign, clip. NaNs are treated as missing data.

Alignment by label is automatic when you add (or subtract, multiply, divide)
two larrys.

larry adds the convenience of labels, provides many built-in methods, and
let's you use your existing array functions.

Download: https://launchpad.net/larry/+download
docs  http://larry.sourceforge.net
code  https://launchpad.net/larry
list  http://groups.google.ca/group/pystatsmodels


From peridot.faceted at gmail.com  Wed Feb  3 20:05:51 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 3 Feb 2010 20:05:51 -0500
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <45d1ab481002031300x78b5eb03hc74e7bf7cb2c8aae@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
	<e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>
	<45d1ab481002031300x78b5eb03hc74e7bf7cb2c8aae@mail.gmail.com>
Message-ID: <ce557a361002031705o6259762doe57a4326631709c9@mail.gmail.com>

2010/2/3 David Goldsmith <d.l.goldsmith at gmail.com>:

> "?? :
> ??? :
> Returns
> ?-------
> ??? :
> ??? :
> ?infodict : dict, only returned if full_output == True
> ???? Dictionary containing additional output information
> ??????? :
> ??????? :
> ? ?? 'mused'? a vector of method indicators for each successful time step:
> ????????????? 1: adams (nonstiff), 2: bdf (stiff)"
>
> Note that it's a _return_ value: apparently with odeint, you can't specify
> which method to use, you can only "hear" which method the algorithm decided
> to use for you.

In particular, odeint always (?) starts with a non-stiff method and
switches to a stiff method if the problem appears to warrant it. It
does not seem that odeint ever switches back to a non-stiff method
even if the problem enters a non-stiff region.

In any case, I'm not sure there's any reason to think that a stiff
solver will help with the OP's problem, which I think is coming from
the discontinuity. I suspect what's happening is that initially the
losses are dominated by the viscosity term, so that the discontinuity
doesn't bother the solver much. But when the velocities drop and the
discontinuous term begins to dominate, the solver tries to match it
better and better. Since the solver tries to approximate the solution
with a 4th or 5th order polynomial but the actual solution is probably
only differentiable once (or not at all), the solver goes crazy trying
to fit a kink with a curve, and you get zillions of basically useless
steps.

I can see three ways to solve the problem:

* Declare that when the discontinuity becomes important, the Coulomb
friction model becomes too crude an approximation, and stop.
(Obviously if the results agree with experiment this is unnecessary.)

* Come up with a model that describes in more detail what happens at
that transition; perhaps a smooth roll-off of forces from positive to
negative, perhaps an additional "stuck area" variable that increases
near zero velocity and that provides resistance to movement, or
perhaps some more sophisticated friction model from the literature.
(Unfortunately the tribology prof I used to work with went back to
Germany, or I'd ask if he had some suggestions.)

* Model it piecewise: run the solver with a RHS that assumes the
object is moving to the right, and stop when the velocity hits zero.
Then figure out whether and which way the object starts moving again,
and start up a new solver. This requires you to have a solver that can
stop when a condition is reached, which is a pain with the scipy
solvers. (I think that odeint can be used in a single-step mode that
allows you to "back up" using interpolation within the last step, but
really the right solution is to use a wrapper for lsodar, which
supports stopping conditions directly.)

Anne


From warren.weckesser at enthought.com  Wed Feb  3 20:17:52 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Wed, 03 Feb 2010 19:17:52 -0600
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
Message-ID: <4B6A2040.6030700@enthought.com>

Hi Ryan,

I have two suggestions, but the second option requires using a 
capability that the scipy ODE solvers currently don't provide.

Since the underlying problem is that you are using discontinous 
equations with a solver that expects smooth equations, you can replace 
your discontinuous function with a smooth approximation.  The attached 
script, coulomb_approx.py, includes an approximation for your friction 
terms.  It generates the attached PNG file.

Also attached is lsoda_vs_coulomb_friction2.py, which is a modified 
version of your script.  It computes the solution twice, once as in the 
original, and once with the approximation.  With the smooth 
approximation, lsoda does not complain, and--at least at the resolution 
of the pylab plot--you can not distinguish the two solutions.   Whether 
an approximation like this is good enough depends on the ultimate goals 
of your analysis.  There maybe behavior very close to v=0 that you want 
to observe but that are not demonstrated with the smooth approximation.

The second option is to use an ODE solver with "root finding" (aka 
"event detection").  You would set the solver to stop when it attempted 
to cross v=0.  Depending on the direction of the crossing, it would 
change a parameter appropriately and then resume from the point where it 
stopped.  Unfortunately, the current ODE solvers in scipy do not include 
this capability (and "rolling your own" can be hard to get right).

Warren


Ryan Krauss wrote:
> I am trying to use odeint (i.e. lsoda) on a mechanical system that
> involves a mass and friction modeled as a viscous term plus Coulomb
> friction.  The discontinuity near 0 velocity makes lsoda mad.  It
> complains that it has to do excess work (lazy algorithm :).  In spite
> of its complaining, the result leads to fairly good agreement between
> model and experiment.  But is there a better way to handle this?
>
> This is the func I am passing to odeint (see attached example script
> for the details):
>
> def ydot_ol(x, t, u, C):
>     m = C[0]
>     b1 = C[1]
>     b2 = C[2]
>     y1 = x[0]
>     y2 = x[1]
>     ydot = zeros(2,)
>     ydot[0] = y2
>     ydot[1] = (u - b1*y2 - b2*sign(y2))/m
>     return ydot
>
> u is an external input.
>
> lsoda doesn't complain until the system is essentially stopping.
>
> Thanks,
>
> Ryan
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: coulomb_approx.py
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100203/71418db3/attachment.ksh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: coulomb_approx.png
Type: image/png
Size: 20128 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100203/71418db3/attachment.png>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: lsoda_vs_coulomb_friction2.py
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100203/71418db3/attachment-0001.ksh>

From warren.weckesser at enthought.com  Wed Feb  3 20:21:59 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Wed, 03 Feb 2010 19:21:59 -0600
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <ce557a361002031705o6259762doe57a4326631709c9@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>	<e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>	<45d1ab481002031300x78b5eb03hc74e7bf7cb2c8aae@mail.gmail.com>
	<ce557a361002031705o6259762doe57a4326631709c9@mail.gmail.com>
Message-ID: <4B6A2137.1040001@enthought.com>

Looks like Anne's email arrived while I was finishing mine.  I could 
have save myself some time by simply saying "Yeah, what Anne said."  :)

Warren


Anne Archibald wrote:
> 2010/2/3 David Goldsmith <d.l.goldsmith at gmail.com>:
>
>   
>> "   :
>>     :
>> Returns
>>  -------
>>     :
>>     :
>>  infodict : dict, only returned if full_output == True
>>      Dictionary containing additional output information
>>         :
>>         :
>>      'mused'  a vector of method indicators for each successful time step:
>>               1: adams (nonstiff), 2: bdf (stiff)"
>>
>> Note that it's a _return_ value: apparently with odeint, you can't specify
>> which method to use, you can only "hear" which method the algorithm decided
>> to use for you.
>>     
>
> In particular, odeint always (?) starts with a non-stiff method and
> switches to a stiff method if the problem appears to warrant it. It
> does not seem that odeint ever switches back to a non-stiff method
> even if the problem enters a non-stiff region.
>
> In any case, I'm not sure there's any reason to think that a stiff
> solver will help with the OP's problem, which I think is coming from
> the discontinuity. I suspect what's happening is that initially the
> losses are dominated by the viscosity term, so that the discontinuity
> doesn't bother the solver much. But when the velocities drop and the
> discontinuous term begins to dominate, the solver tries to match it
> better and better. Since the solver tries to approximate the solution
> with a 4th or 5th order polynomial but the actual solution is probably
> only differentiable once (or not at all), the solver goes crazy trying
> to fit a kink with a curve, and you get zillions of basically useless
> steps.
>
> I can see three ways to solve the problem:
>
> * Declare that when the discontinuity becomes important, the Coulomb
> friction model becomes too crude an approximation, and stop.
> (Obviously if the results agree with experiment this is unnecessary.)
>
> * Come up with a model that describes in more detail what happens at
> that transition; perhaps a smooth roll-off of forces from positive to
> negative, perhaps an additional "stuck area" variable that increases
> near zero velocity and that provides resistance to movement, or
> perhaps some more sophisticated friction model from the literature.
> (Unfortunately the tribology prof I used to work with went back to
> Germany, or I'd ask if he had some suggestions.)
>
> * Model it piecewise: run the solver with a RHS that assumes the
> object is moving to the right, and stop when the velocity hits zero.
> Then figure out whether and which way the object starts moving again,
> and start up a new solver. This requires you to have a solver that can
> stop when a condition is reached, which is a pain with the scipy
> solvers. (I think that odeint can be used in a single-step mode that
> allows you to "back up" using interpolation within the last step, but
> really the right solution is to use a wrapper for lsodar, which
> supports stopping conditions directly.)
>
> Anne
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From charlesr.harris at gmail.com  Wed Feb  3 20:29:34 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 3 Feb 2010 18:29:34 -0700
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <ce557a361002031705o6259762doe57a4326631709c9@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
	<e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>
	<45d1ab481002031300x78b5eb03hc74e7bf7cb2c8aae@mail.gmail.com>
	<ce557a361002031705o6259762doe57a4326631709c9@mail.gmail.com>
Message-ID: <e06186141002031729i2d0c4018j36b65bd6d384fe6a@mail.gmail.com>

On Wed, Feb 3, 2010 at 6:05 PM, Anne Archibald <peridot.faceted at gmail.com>wrote:

> 2010/2/3 David Goldsmith <d.l.goldsmith at gmail.com>:
>
> > "   :
> >     :
> > Returns
> >  -------
> >     :
> >     :
> >  infodict : dict, only returned if full_output == True
> >      Dictionary containing additional output information
> >         :
> >         :
> >      'mused'  a vector of method indicators for each successful time
> step:
> >               1: adams (nonstiff), 2: bdf (stiff)"
> >
> > Note that it's a _return_ value: apparently with odeint, you can't
> specify
> > which method to use, you can only "hear" which method the algorithm
> decided
> > to use for you.
>
> In particular, odeint always (?) starts with a non-stiff method and
> switches to a stiff method if the problem appears to warrant it. It
> does not seem that odeint ever switches back to a non-stiff method
> even if the problem enters a non-stiff region.
>
> In any case, I'm not sure there's any reason to think that a stiff
> solver will help with the OP's problem,


Me neither, what motivated that suggestion is that if the object "sticks",
then the ODE becomes an algebraic equation. Another WAG would be to try a
low order integrator. I'm rather curious to see what works best here.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100203/e4dd14cf/attachment.html>

From charlesr.harris at gmail.com  Wed Feb  3 20:44:46 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 3 Feb 2010 18:44:46 -0700
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <e06186141002031729i2d0c4018j36b65bd6d384fe6a@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
	<e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>
	<45d1ab481002031300x78b5eb03hc74e7bf7cb2c8aae@mail.gmail.com>
	<ce557a361002031705o6259762doe57a4326631709c9@mail.gmail.com>
	<e06186141002031729i2d0c4018j36b65bd6d384fe6a@mail.gmail.com>
Message-ID: <e06186141002031744vdc8e3d7saf306a68de31aeeb@mail.gmail.com>

On Wed, Feb 3, 2010 at 6:29 PM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Wed, Feb 3, 2010 at 6:05 PM, Anne Archibald <peridot.faceted at gmail.com>wrote:
>
>> 2010/2/3 David Goldsmith <d.l.goldsmith at gmail.com>:
>>
>> > "   :
>> >     :
>> > Returns
>> >  -------
>> >     :
>> >     :
>> >  infodict : dict, only returned if full_output == True
>> >      Dictionary containing additional output information
>> >         :
>> >         :
>> >      'mused'  a vector of method indicators for each successful time
>> step:
>> >               1: adams (nonstiff), 2: bdf (stiff)"
>> >
>> > Note that it's a _return_ value: apparently with odeint, you can't
>> specify
>> > which method to use, you can only "hear" which method the algorithm
>> decided
>> > to use for you.
>>
>> In particular, odeint always (?) starts with a non-stiff method and
>> switches to a stiff method if the problem appears to warrant it. It
>> does not seem that odeint ever switches back to a non-stiff method
>> even if the problem enters a non-stiff region.
>>
>> In any case, I'm not sure there's any reason to think that a stiff
>> solver will help with the OP's problem,
>
>
> Me neither, what motivated that suggestion is that if the object "sticks",
> then the ODE becomes an algebraic equation. Another WAG would be to try a
> low order integrator. I'm rather curious to see what works best here.
>
>
Here is a bit of discussion <http://tinyurl.com/y8kxxyb>. The author says
that smoothing methods lead to stiff equations and notes that these are time
consuming.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100203/cd6c40b5/attachment.html>

From Burly.Cumberland at coherent.com  Thu Feb  4 09:14:11 2010
From: Burly.Cumberland at coherent.com (Cumberland, Burly)
Date: Thu, 4 Feb 2010 14:14:11 -0000
Subject: [SciPy-User] Matching date lists
Message-ID: <CFD8D5FB73B68A498FEF88DD16F52C8B013AEA24@glaexc02.eu.coherentinc.com>

Hi,

 

I have several datasets which are linked to date/timestamps. I import
these in and convert all the dates to python datetime objects. So for
instance I might have something like

 

Array of datetime objects with an array of data values associated with
it, say A.

Another array of datetime objects with several arrays of data values
associated, say B, C and D.

 

The time stream is not continuous, i.e. there may be 5-6 days data then
nothing for a day then 10 days data. While some of the arrays are at
regular sample intervals (frequency) at least one of them has data
generated at a higher frequency but with no fixed period. Ideally I
would like to associate the closest measurement from this list with the
datetime stamp from the first list. Thus if I have

 

array([2009-12-23 13:57:16, 2009-12-23 13:58:15, 2009-12-23 13:59:14,

      2009-12-23 14:00:14, 2009-12-23 14:01:13, 2009-12-23 14:02:13,

      2009-12-23 14:03:13, 2009-12-23 14:04:12, 2009-12-23 14:05:12,

      2009-12-23 14:06:12], dtype=object)

 

and

 

array([2009-12-23 13:57:21, 2009-12-23 13:57:28, 2009-12-23 13:57:37,

      2009-12-23 13:57:44, 2009-12-23 13:57:53, 2009-12-23 13:58:02,

      2009-12-23 13:58:09, 2009-12-23 13:58:17, 2009-12-23 13:58:25,

      2009-12-23 13:58:33], dtype=object)

 

I'd like to tie my values for the 1st and 8th values from the second
array to the first two values from the first array (assuming I'm happy
that the data is taken within 5 seconds of each other). Thus I'd mask or
disregard all the data in the second array set (B, C and D) that isn't
measured within a reasonable time period of the first.

 

Clearly I could write a loop and do comparisons and then copy the data
or pop it. The problem is there's a quarter of a million items in the
data set at the minute and it continues to grow. So I was wondering if
anyone can recommend a method or module, I'd a brief look at timeseries
and pandas but neither appears to have a tool which resolves this for
me.

 

Any suggestions welcome.

 

Regards,

Burly.

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100204/b8b82b7e/attachment.html>

From ryanlists at gmail.com  Thu Feb  4 09:33:26 2010
From: ryanlists at gmail.com (Ryan Krauss)
Date: Thu, 4 Feb 2010 08:33:26 -0600
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <e06186141002031744vdc8e3d7saf306a68de31aeeb@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
	<e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>
	<45d1ab481002031300x78b5eb03hc74e7bf7cb2c8aae@mail.gmail.com>
	<ce557a361002031705o6259762doe57a4326631709c9@mail.gmail.com>
	<e06186141002031729i2d0c4018j36b65bd6d384fe6a@mail.gmail.com>
	<e06186141002031744vdc8e3d7saf306a68de31aeeb@mail.gmail.com>
Message-ID: <c5b438121002040633l45eb814v811328000596b3d9@mail.gmail.com>

Thanks for all the excellent and thoughtful responses.  I kind of
expected Warren to yell at me to stop using smooth solvers on
discontinuous systems and leave it at that.  Your responses not only
give me somethings to try, but make me feel like my question really
was a good one.

For now, I am basically following Anne's first suggestion:

* Declare that when the discontinuity becomes important, the Coulomb
friction model becomes too crude an approximation, and stop.
(Obviously if the results agree with experiment this is unnecessary.)

But I can only get away with this because the input stops or goes very
near zero at the end of the test where the velocity approaches zero.
I have rationalized that because there is no spring force in my model,
when the external input force is zero and the velocity gets small
enough, the mass basically skids to a stop because of friction and
there is nothing to make it start moving the other direction.

I will run some more experiments (probably early next week) with near
zero velocity and see if that helps me refine the model.  If nothing
else, it should give me enough data to compare some of your
suggestions.

Thanks again.  This is very helpful.

Ryan

On Wed, Feb 3, 2010 at 7:44 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Wed, Feb 3, 2010 at 6:29 PM, Charles R Harris <charlesr.harris at gmail.com>
> wrote:
>>
>>
>> On Wed, Feb 3, 2010 at 6:05 PM, Anne Archibald <peridot.faceted at gmail.com>
>> wrote:
>>>
>>> 2010/2/3 David Goldsmith <d.l.goldsmith at gmail.com>:
>>>
>>> > "?? :
>>> > ??? :
>>> > Returns
>>> > ?-------
>>> > ??? :
>>> > ??? :
>>> > ?infodict : dict, only returned if full_output == True
>>> > ???? Dictionary containing additional output information
>>> > ??????? :
>>> > ??????? :
>>> > ? ?? 'mused'? a vector of method indicators for each successful time
>>> > step:
>>> > ????????????? 1: adams (nonstiff), 2: bdf (stiff)"
>>> >
>>> > Note that it's a _return_ value: apparently with odeint, you can't
>>> > specify
>>> > which method to use, you can only "hear" which method the algorithm
>>> > decided
>>> > to use for you.
>>>
>>> In particular, odeint always (?) starts with a non-stiff method and
>>> switches to a stiff method if the problem appears to warrant it. It
>>> does not seem that odeint ever switches back to a non-stiff method
>>> even if the problem enters a non-stiff region.
>>>
>>> In any case, I'm not sure there's any reason to think that a stiff
>>> solver will help with the OP's problem,
>>
>> Me neither, what motivated that suggestion is that if the object "sticks",
>> then the ODE becomes an algebraic equation. Another WAG would be to try a
>> low order integrator. I'm rather curious to see what works best here.
>>
>
> Here is a bit of discussion. The author says that smoothing methods lead to
> stiff equations and notes that these are time consuming.
>
> Chuck
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From wesmckinn at gmail.com  Thu Feb  4 09:58:00 2010
From: wesmckinn at gmail.com (Wes McKinney)
Date: Thu, 4 Feb 2010 09:58:00 -0500
Subject: [SciPy-User] Matching date lists
In-Reply-To: <CFD8D5FB73B68A498FEF88DD16F52C8B013AEA24@glaexc02.eu.coherentinc.com>
References: <AcqlpFL9DIgbHI7bTNarm9EJlE3poQ==>
	<CFD8D5FB73B68A498FEF88DD16F52C8B013AEA24@glaexc02.eu.coherentinc.com>
Message-ID: <6c476c8a1002040658t716085c9v6e0470beb16c42a9@mail.gmail.com>

On Thu, Feb 4, 2010 at 9:14 AM, Cumberland, Burly
<Burly.Cumberland at coherent.com> wrote:
> Hi,
>
>
>
> I have several datasets which are linked to date/timestamps. I import these
> in and convert all the dates to python datetime objects. So for instance I
> might have something like
>
>
>
> Array of datetime objects with an array of data values associated with it,
> say A.
>
> Another array of datetime objects with several arrays of data values
> associated, say B, C and D.
>
>
>
> The time stream is not continuous, i.e. there may be 5-6 days data then
> nothing for a day then 10 days data. While some of the arrays are at regular
> sample intervals (frequency) at least one of them has data generated at a
> higher frequency but with no fixed period. Ideally I would like to associate
> the closest measurement from this list with the datetime stamp from the
> first list. Thus if I have
>
>
>
> array([2009-12-23 13:57:16, 2009-12-23 13:58:15, 2009-12-23 13:59:14,
>
> ????? 2009-12-23 14:00:14, 2009-12-23 14:01:13, 2009-12-23 14:02:13,
>
> ????? 2009-12-23 14:03:13, 2009-12-23 14:04:12, 2009-12-23 14:05:12,
>
> ????? 2009-12-23 14:06:12], dtype=object)
>
>
>
> and
>
>
>
> array([2009-12-23 13:57:21, 2009-12-23 13:57:28, 2009-12-23 13:57:37,
>
> ????? 2009-12-23 13:57:44, 2009-12-23 13:57:53, 2009-12-23 13:58:02,
>
> ????? 2009-12-23 13:58:09, 2009-12-23 13:58:17, 2009-12-23 13:58:25,
>
> ????? 2009-12-23 13:58:33], dtype=object)
>
>
>
> I'd like to tie my values for the 1st and 8th values from the second array
> to the first two values from the first array (assuming I'm happy that the
> data is taken within 5 seconds of each other). Thus I'd mask or disregard
> all the data in the second array set (B, C and D) that isn't measured within
> a reasonable time period of the first.
>
>
>
> Clearly I could write a loop and do comparisons and then copy the data or
> pop it. The problem is there's a quarter of a million items in the data set
> at the minute and it continues to grow. So I was wondering if anyone can
> recommend a method or module, I'd a brief look at timeseries and pandas but
> neither appears to have a tool which resolves this for me.
>
>
>
> Any suggestions welcome.
>
>
>
> Regards,
>
> Burly.
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>

Assuming that measurements in the second array have to happen after
the timestamps in the first array, I would suggest a searchsorted
approach:

indexer = first_list.searchsorted(second_list)

closest_dates = first_list.take(indexer - 1) # need date prior-- does
not deal with date equality though

deltas = (second_list - closest_dates)

mask = deltas < timedelta(seconds=5)

# now do as you wish with the mask and indexer

I think you just need to check that edge cases (beginning and end of
the arrays) are being handled correctly. If you don't care whether the
dates in the second list come before or after the ones in the first
list, you can do a couple searchsorteds and a few more closeness
comparisons. I would be curious if this works and is sufficiently
performant.

- Wes


From wesmckinn at gmail.com  Thu Feb  4 10:01:35 2010
From: wesmckinn at gmail.com (Wes McKinney)
Date: Thu, 4 Feb 2010 10:01:35 -0500
Subject: [SciPy-User] Matching date lists
In-Reply-To: <6c476c8a1002040658t716085c9v6e0470beb16c42a9@mail.gmail.com>
References: <CFD8D5FB73B68A498FEF88DD16F52C8B013AEA24@glaexc02.eu.coherentinc.com>
	<6c476c8a1002040658t716085c9v6e0470beb16c42a9@mail.gmail.com>
Message-ID: <6c476c8a1002040701g7138c125sdfa9d0fc2b9c7459@mail.gmail.com>

On Thu, Feb 4, 2010 at 9:58 AM, Wes McKinney <wesmckinn at gmail.com> wrote:
> On Thu, Feb 4, 2010 at 9:14 AM, Cumberland, Burly
> <Burly.Cumberland at coherent.com> wrote:
>> Hi,
>>
>>
>>
>> I have several datasets which are linked to date/timestamps. I import these
>> in and convert all the dates to python datetime objects. So for instance I
>> might have something like
>>
>>
>>
>> Array of datetime objects with an array of data values associated with it,
>> say A.
>>
>> Another array of datetime objects with several arrays of data values
>> associated, say B, C and D.
>>
>>
>>
>> The time stream is not continuous, i.e. there may be 5-6 days data then
>> nothing for a day then 10 days data. While some of the arrays are at regular
>> sample intervals (frequency) at least one of them has data generated at a
>> higher frequency but with no fixed period. Ideally I would like to associate
>> the closest measurement from this list with the datetime stamp from the
>> first list. Thus if I have
>>
>>
>>
>> array([2009-12-23 13:57:16, 2009-12-23 13:58:15, 2009-12-23 13:59:14,
>>
>> ????? 2009-12-23 14:00:14, 2009-12-23 14:01:13, 2009-12-23 14:02:13,
>>
>> ????? 2009-12-23 14:03:13, 2009-12-23 14:04:12, 2009-12-23 14:05:12,
>>
>> ????? 2009-12-23 14:06:12], dtype=object)
>>
>>
>>
>> and
>>
>>
>>
>> array([2009-12-23 13:57:21, 2009-12-23 13:57:28, 2009-12-23 13:57:37,
>>
>> ????? 2009-12-23 13:57:44, 2009-12-23 13:57:53, 2009-12-23 13:58:02,
>>
>> ????? 2009-12-23 13:58:09, 2009-12-23 13:58:17, 2009-12-23 13:58:25,
>>
>> ????? 2009-12-23 13:58:33], dtype=object)
>>
>>
>>
>> I'd like to tie my values for the 1st and 8th values from the second array
>> to the first two values from the first array (assuming I'm happy that the
>> data is taken within 5 seconds of each other). Thus I'd mask or disregard
>> all the data in the second array set (B, C and D) that isn't measured within
>> a reasonable time period of the first.
>>
>>
>>
>> Clearly I could write a loop and do comparisons and then copy the data or
>> pop it. The problem is there's a quarter of a million items in the data set
>> at the minute and it continues to grow. So I was wondering if anyone can
>> recommend a method or module, I'd a brief look at timeseries and pandas but
>> neither appears to have a tool which resolves this for me.
>>
>>
>>
>> Any suggestions welcome.
>>
>>
>>
>> Regards,
>>
>> Burly.
>>
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> Assuming that measurements in the second array have to happen after
> the timestamps in the first array, I would suggest a searchsorted
> approach:
>
> indexer = first_list.searchsorted(second_list)
>
> closest_dates = first_list.take(indexer - 1) # need date prior-- does
> not deal with date equality though
>
> deltas = (second_list - closest_dates)
>
> mask = deltas < timedelta(seconds=5)
>
> # now do as you wish with the mask and indexer
>
> I think you just need to check that edge cases (beginning and end of
> the arrays) are being handled correctly. If you don't care whether the
> dates in the second list come before or after the ones in the first
> list, you can do a couple searchsorteds and a few more closeness
> comparisons. I would be curious if this works and is sufficiently
> performant.
>
> - Wes
>

I should add that this seems like a case where the numpy datetime
dtype would make a huge performance difference.


From charlesr.harris at gmail.com  Thu Feb  4 11:37:07 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 4 Feb 2010 09:37:07 -0700
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <c5b438121002040633l45eb814v811328000596b3d9@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
	<e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>
	<45d1ab481002031300x78b5eb03hc74e7bf7cb2c8aae@mail.gmail.com>
	<ce557a361002031705o6259762doe57a4326631709c9@mail.gmail.com>
	<e06186141002031729i2d0c4018j36b65bd6d384fe6a@mail.gmail.com>
	<e06186141002031744vdc8e3d7saf306a68de31aeeb@mail.gmail.com>
	<c5b438121002040633l45eb814v811328000596b3d9@mail.gmail.com>
Message-ID: <e06186141002040837l453477f4ga6fd236aad6647f3@mail.gmail.com>

On Thu, Feb 4, 2010 at 7:33 AM, Ryan Krauss <ryanlists at gmail.com> wrote:

> Thanks for all the excellent and thoughtful responses.  I kind of
> expected Warren to yell at me to stop using smooth solvers on
> discontinuous systems and leave it at that.  Your responses not only
> give me somethings to try, but make me feel like my question really
> was a good one.
>
> For now, I am basically following Anne's first suggestion:
>
> * Declare that when the discontinuity becomes important, the Coulomb
> friction model becomes too crude an approximation, and stop.
> (Obviously if the results agree with experiment this is unnecessary.)
>
>
That is sort of like the "switch" method in the reference I linked. That
looks like the easiest way to go for simple systems.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100204/c7208ea1/attachment.html>

From lroubeyrie at limair.asso.fr  Thu Feb  4 11:31:22 2010
From: lroubeyrie at limair.asso.fr (Lionel Roubeyrie)
Date: Thu, 04 Feb 2010 17:31:22 +0100
Subject: [SciPy-User] Matching date lists
In-Reply-To: <6c476c8a1002040658t716085c9v6e0470beb16c42a9@mail.gmail.com>
References: <AcqlpFL9DIgbHI7bTNarm9EJlE3poQ==>	<CFD8D5FB73B68A498FEF88DD16F52C8B013AEA24@glaexc02.eu.coherentinc.com>
	<6c476c8a1002040658t716085c9v6e0470beb16c42a9@mail.gmail.com>
Message-ID: <4B6AF65A.9030103@limair.asso.fr>

Hi,
a possible way is to use KDTree by converting datetimes to seconds from the epoch :
######################################
import datetime
import time
from scipy.spatial import KDTree

[~]|87> a
    <87>
[datetime.datetime(2009, 12, 23, 13, 57, 16),
  datetime.datetime(2009, 12, 23, 13, 58, 15),
  datetime.datetime(2009, 12, 23, 13, 59, 14),
  datetime.datetime(2009, 12, 23, 14, 0, 14),
  datetime.datetime(2009, 12, 23, 14, 1, 13),
  datetime.datetime(2009, 12, 23, 14, 2, 13),
  datetime.datetime(2009, 12, 23, 14, 3, 13),
  datetime.datetime(2009, 12, 23, 14, 4, 12),
  datetime.datetime(2009, 12, 23, 14, 5, 12),
  datetime.datetime(2009, 12, 23, 14, 6, 12)]
[~]|88> b
    <88>
[datetime.datetime(2009, 12, 23, 13, 57, 21),
  datetime.datetime(2009, 12, 23, 13, 57, 28),
  datetime.datetime(2009, 12, 23, 13, 57, 37),
  datetime.datetime(2009, 12, 23, 13, 57, 44),
  datetime.datetime(2009, 12, 23, 13, 57, 53),
  datetime.datetime(2009, 12, 23, 13, 58, 2),
  datetime.datetime(2009, 12, 23, 13, 58, 9),
  datetime.datetime(2009, 12, 23, 13, 58, 17),
  datetime.datetime(2009, 12, 23, 13, 58, 25),
  datetime.datetime(2009, 12, 23, 13, 58, 33)]
[~]|89> asec=[time.mktime(i.timetuple()) for i in a]
[~]|90> bsec=[time.mktime(i.timetuple()) for i in b]
[~]|91> atree=KDTree(zip(asec, zeros(len(asec))))
[~]|92> btree=KDTree(zip(bsec, zeros(len(bsec))))
[~]|93> atree.query(btree.data[0] )[1]
    <93> 0
[~]|94> atree.query(btree.data[7] )[1]
    <94> 1
######################################

Yes, you can join space and time \o/






From kwgoodman at gmail.com  Thu Feb  4 11:47:04 2010
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 4 Feb 2010 08:47:04 -0800
Subject: [SciPy-User] Matching date lists
In-Reply-To: <4B6AF65A.9030103@limair.asso.fr>
References: <CFD8D5FB73B68A498FEF88DD16F52C8B013AEA24@glaexc02.eu.coherentinc.com>
	<6c476c8a1002040658t716085c9v6e0470beb16c42a9@mail.gmail.com>
	<4B6AF65A.9030103@limair.asso.fr>
Message-ID: <f4f93d421002040847s76684e9fl8a6d6df0ef6751ac@mail.gmail.com>

On Thu, Feb 4, 2010 at 8:31 AM, Lionel Roubeyrie
<lroubeyrie at limair.asso.fr> wrote:
> Hi,
> a possible way is to use KDTree by converting datetimes to seconds from the epoch :
> ######################################
> import datetime
> import time
> from scipy.spatial import KDTree
>
> [~]|87> a
> ? ?<87>
> [datetime.datetime(2009, 12, 23, 13, 57, 16),
> ?datetime.datetime(2009, 12, 23, 13, 58, 15),
> ?datetime.datetime(2009, 12, 23, 13, 59, 14),
> ?datetime.datetime(2009, 12, 23, 14, 0, 14),
> ?datetime.datetime(2009, 12, 23, 14, 1, 13),
> ?datetime.datetime(2009, 12, 23, 14, 2, 13),
> ?datetime.datetime(2009, 12, 23, 14, 3, 13),
> ?datetime.datetime(2009, 12, 23, 14, 4, 12),
> ?datetime.datetime(2009, 12, 23, 14, 5, 12),
> ?datetime.datetime(2009, 12, 23, 14, 6, 12)]
> [~]|88> b
> ? ?<88>
> [datetime.datetime(2009, 12, 23, 13, 57, 21),
> ?datetime.datetime(2009, 12, 23, 13, 57, 28),
> ?datetime.datetime(2009, 12, 23, 13, 57, 37),
> ?datetime.datetime(2009, 12, 23, 13, 57, 44),
> ?datetime.datetime(2009, 12, 23, 13, 57, 53),
> ?datetime.datetime(2009, 12, 23, 13, 58, 2),
> ?datetime.datetime(2009, 12, 23, 13, 58, 9),
> ?datetime.datetime(2009, 12, 23, 13, 58, 17),
> ?datetime.datetime(2009, 12, 23, 13, 58, 25),
> ?datetime.datetime(2009, 12, 23, 13, 58, 33)]
> [~]|89> asec=[time.mktime(i.timetuple()) for i in a]
> [~]|90> bsec=[time.mktime(i.timetuple()) for i in b]
> [~]|91> atree=KDTree(zip(asec, zeros(len(asec))))
> [~]|92> btree=KDTree(zip(bsec, zeros(len(bsec))))
> [~]|93> atree.query(btree.data[0] )[1]
> ? ?<93> 0
> [~]|94> atree.query(btree.data[7] )[1]
> ? ?<94> 1
> ######################################
>
> Yes, you can join space and time \o/

I thought of that too. But I noticed that KDTree doesn't like 1d
arrays. It crashes here:

--> 127         self.n, self.m = np.shape(self.data)

Would an np.atleast_2d() fix that?


From darryl.wallace at prosensus.ca  Thu Feb  4 15:25:51 2010
From: darryl.wallace at prosensus.ca (Darryl Wallace)
Date: Thu, 4 Feb 2010 15:25:51 -0500
Subject: [SciPy-User] [build] visual studio 2008 and intel fortran 11.1
Message-ID: <7953e1d26b4dbb9af90d0aa784efe9e7@mail.gmail.com>

Hello Eloi,



> I just finished building scipy on both our 32-bit & 64-bit windows

> platforms, using disutils (patched python and numpy modules). I'll

> (hopefully) switch to scons for building the next official releases.



I just saw your note here on how you built scipy on both 32-bit and 64-bit
windows with Visual Studio 2008 and intel fortran 11.1.  I?m interested in
doing this as well but have encountered similar problems.



You mentioned using distutils (patched python and numpy modules).  Would you
care to share what steps you took to patch those modules?  I?m trying to
build these myself and have spent a number of hours fiddling with the
distutils modules.



Any help would be much appreciated.



Regards,

Darryl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100204/946d5c04/attachment.html>

From gemma.atkinson at gmail.com  Fri Feb  5 07:21:31 2010
From: gemma.atkinson at gmail.com (Gemma Atkinson)
Date: Fri, 5 Feb 2010 13:21:31 +0100
Subject: [SciPy-User] installing SciPy on Mac OS 10.6
Message-ID: <A24DA88A-C59E-4E4D-97F9-B37273782510@gmail.com>

Hi,

I'm trying to install Scipy with little success. I already had numpy installed, but to make sure I was using the most current version, I installed it again. I got both Numy and Scipy from SVN. These are the commands I used:

svn co http://svn.scipy.org/svn/numpy/trunk numpy
svn co http://svn.scipy.org/svn/scipy/trunk scipy

Then for each of these:

python setup.py build
sudo python setup.py install

This seemed to work fine for numpy, but for Scipy I got this error with python setup.py build:

from numpy.distutils.misc_util import get_numpy_include_dirs, get_info
ImportError: cannot import name get_info

(Full output pasted at end of message).

I also tried installing Numpy and Scipy from the dmgs, but both failed saying that I need Python 2.6 (I have 2.6.1).

My OS version is 10.6.2
gcc --version = i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5646) (dot 1)
gfortran --version = GNU Fortran (GCC) 4.2.3 


Many thanks for your help,

Gemma


syst13:scipy Gem$ python setup.py build
Warning: No configuration returned, assuming unavailable.
blas_opt_info:
  FOUND:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
    extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers']

lapack_opt_info:
  FOUND:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
    extra_compile_args = ['-faltivec']

umfpack_info:
  libraries umfpack not found in /System/Library/Frameworks/Python.framework/Versions/2.6/lib
  libraries umfpack not found in /usr/local/lib
  libraries umfpack not found in /usr/lib
  libraries umfpack not found in /sw/lib
/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/distutils/system_info.py:414: UserWarning: 
    UMFPACK sparse solver (http://www.cise.ufl.edu/research/sparse/umfpack/)
    not found. Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [umfpack]) or by setting
    the UMFPACK environment variable.
  warnings.warn(self.notfounderror.__doc__)
  NOT AVAILABLE

Traceback (most recent call last):
  File "setup.py", line 160, in <module>
    setup_package()
  File "setup.py", line 152, in setup_package
    configuration=configuration )
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/distutils/core.py", line 150, in setup
    config = configuration()
  File "setup.py", line 118, in configuration
    config.add_subpackage('scipy')
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/distutils/misc_util.py", line 851, in add_subpackage
    caller_level = 2)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/distutils/misc_util.py", line 834, in get_subpackage
    caller_level = caller_level + 1)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/distutils/misc_util.py", line 781, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "scipy/setup.py", line 20, in configuration
    config.add_subpackage('special')
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/distutils/misc_util.py", line 851, in add_subpackage
    caller_level = 2)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/distutils/misc_util.py", line 834, in get_subpackage
    caller_level = caller_level + 1)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/distutils/misc_util.py", line 766, in _get_configuration_from_setup_py
    ('.py', 'U', 1))
  File "scipy/special/setup.py", line 7, in <module>
    from numpy.distutils.misc_util import get_numpy_include_dirs, get_info
ImportError: cannot import name get_info





From eiffleduarte at gmail.com  Fri Feb  5 07:34:18 2010
From: eiffleduarte at gmail.com (Marcus Vinicius Eiffle Duarte)
Date: Fri, 5 Feb 2010 10:34:18 -0200
Subject: [SciPy-User] installing SciPy on Mac OS 10.6
In-Reply-To: <A24DA88A-C59E-4E4D-97F9-B37273782510@gmail.com>
References: <A24DA88A-C59E-4E4D-97F9-B37273782510@gmail.com>
Message-ID: <280253811002050434ka24d28em9b57c59c805843f0@mail.gmail.com>

Gemma,

have you tried the SciPy Superpack from http://macinscience.org/?page_id=6?
It's the easiest way to install numpy+scipy on OSX. If you don't need the
latest SVN code, it is worth a try.

Marcus Vinicius Eiffle Duarte
eiffleduarte at gmail.com
Niter?i, RJ, Brasil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100205/460b8e45/attachment.html>

From cournape at gmail.com  Fri Feb  5 08:00:55 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 5 Feb 2010 22:00:55 +0900
Subject: [SciPy-User] installing SciPy on Mac OS 10.6
In-Reply-To: <A24DA88A-C59E-4E4D-97F9-B37273782510@gmail.com>
References: <A24DA88A-C59E-4E4D-97F9-B37273782510@gmail.com>
Message-ID: <5b8d13221002050500s2956e15aw80c09bad3e365f2b@mail.gmail.com>

On Fri, Feb 5, 2010 at 9:21 PM, Gemma Atkinson <gemma.atkinson at gmail.com> wrote:
> Hi,
>
> I'm trying to install Scipy with little success. I already had numpy installed, but to make sure I was using the most current version, I installed it again. I got both Numy and Scipy from SVN. These are the commands I used:
>
> svn co http://svn.scipy.org/svn/numpy/trunk numpy
> svn co http://svn.scipy.org/svn/scipy/trunk scipy
>
> Then for each of these:
>
> python setup.py build
> sudo python setup.py install
>
> This seemed to work fine for numpy, but for Scipy I got this error with python setup.py build:
>
> from numpy.distutils.misc_util import get_numpy_include_dirs, get_info
> ImportError: cannot import name get_info

You are not using the numpy you just installed, but the one given with Mac OS X.

>
> (Full output pasted at end of message).
>
> I also tried installing Numpy and Scipy from the dmgs, but both failed saying that I need Python 2.6 (I have 2.6.1).

You should install the python from python.org. The dmgs do not work
with Apple python.

cheers,

David


From gemma.atkinson at gmail.com  Fri Feb  5 08:02:37 2010
From: gemma.atkinson at gmail.com (Gemma Atkinson)
Date: Fri, 5 Feb 2010 14:02:37 +0100
Subject: [SciPy-User] installing SciPy on Mac OS 10.6
In-Reply-To: <280253811002050434ka24d28em9b57c59c805843f0@mail.gmail.com>
References: <A24DA88A-C59E-4E4D-97F9-B37273782510@gmail.com>
	<280253811002050434ka24d28em9b57c59c805843f0@mail.gmail.com>
Message-ID: <AC960057-B559-4C3A-9D6D-F6C33C8E8350@gmail.com>

Thank you for your help. The SciPy superpack worked perfectly.

What a quick and easy solution for something I've been struggling with for ages!

Thanks again,

Gemma

On 5 Feb 2010, at 13:34, Marcus Vinicius Eiffle Duarte wrote:

> Gemma,
> 
> have you tried the SciPy Superpack from http://macinscience.org/?page_id=6?
> It's the easiest way to install numpy+scipy on OSX. If you don't need the latest SVN code, it is worth a try.
> 
> Marcus Vinicius Eiffle Duarte
> eiffleduarte at gmail.com
> Niter?i, RJ, Brasil
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100205/b832c8bd/attachment.html>

From tsyu80 at gmail.com  Fri Feb  5 17:31:30 2010
From: tsyu80 at gmail.com (Tony S Yu)
Date: Fri, 5 Feb 2010 17:31:30 -0500
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu>
	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com>
	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
Message-ID: <76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com>


On Jan 28, 2010, at 5:14 PM, Tony S Yu wrote:

> 
> On Jan 28, 2010, at 9:20 AM, denis wrote:
> 
>> 
>> 
>> On Jan 20, 11:56 pm, Tony S Yu <ton... at MIT.EDU> wrote:
>>> I'm having trouble making splines from scipy.signal work with those in scipy.interpolation.
>>> 
>>> Both packages have functions for creating (`signal.cspline1d`/`interpolate.splrep`) and evaluating (`signal.cspline1d_eval`/`interpolate.splev`) splines. There are, of course, huge differences between these functions, which is why I'm trying to get them to talk to each other.
>>> 
>>> In particular, I'd like to create a smoothing spline using `cspline1d` (which allows easier smoothing) and evaluate using `splev` (which allows me to get derivatives of the spline).
>> 
>> Tony,
>> bouncing between two murky packages doesn't sound as though it'll
>> converge ...
> 
> Agreed. This was more of a naive attempt to try and get the results that I wanted.
> 
>> interpolate though has both smoothing and derivs --
>>       interpolator = interpolate.UnivariateSpline( x, y, k=3, s=s )
>> # s=0 interpolates
>>       yy = interpolator( xx )
>>       y1 = interpolator( xx, 1 )  # deriv
> 
> You're right. When I originally read the docs for splrep, I had it in my head that the splines in scipy.interpolation didn't provide the "right" type of smoothing (don't ask me what "right" means---I have no idea). After taking some time to understand the interpolation module, I realize it does what I want. Thanks, Denis!

I think I spoke too soon. I believe what I need are called a smoothing splines, which is what scipy.signal provides; on the other I hand, scipy.interpolate creates what I think are called regression splines. The difference being that regression splines smooth the data using fewer knots than the number of observed data, while smoothing splines have knots which match the (abscissa) location of the original data. (I don't use splines often, so please correct me if I'm misinterpreting what I've read about the topic.)

It's possible to specify knots or smoothing with interpolate.splrep, but when specifying both, the smoothing parameter is ignored and the knot placement (plus the error metric) determines the smoothness of the spline. 

I'm trying to re-implement some matlab code (which uses matlab's spaps function) for spline optimization, and the algorithm requires smoothing splines (as opposed to regression splines). Any suggestions?

Thanks
-Tony





From dagss at student.matnat.uio.no  Fri Feb  5 18:00:03 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 06 Feb 2010 00:00:03 +0100
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu>	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com>	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
	<76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com>
Message-ID: <4B6CA2F3.10008@student.matnat.uio.no>

Tony S Yu wrote:
> On Jan 28, 2010, at 5:14 PM, Tony S Yu wrote:
>
>   
>> On Jan 28, 2010, at 9:20 AM, denis wrote:
>>
>>     
>>> On Jan 20, 11:56 pm, Tony S Yu <ton... at MIT.EDU> wrote:
>>>       
>>>> I'm having trouble making splines from scipy.signal work with those in scipy.interpolation.
>>>>
>>>> Both packages have functions for creating (`signal.cspline1d`/`interpolate.splrep`) and evaluating (`signal.cspline1d_eval`/`interpolate.splev`) splines. There are, of course, huge differences between these functions, which is why I'm trying to get them to talk to each other.
>>>>
>>>> In particular, I'd like to create a smoothing spline using `cspline1d` (which allows easier smoothing) and evaluate using `splev` (which allows me to get derivatives of the spline).
>>>>         
>>> Tony,
>>> bouncing between two murky packages doesn't sound as though it'll
>>> converge ...
>>>       
>> Agreed. This was more of a naive attempt to try and get the results that I wanted.
>>
>>     
>>> interpolate though has both smoothing and derivs --
>>>       interpolator = interpolate.UnivariateSpline( x, y, k=3, s=s )
>>> # s=0 interpolates
>>>       yy = interpolator( xx )
>>>       y1 = interpolator( xx, 1 )  # deriv
>>>       
>> You're right. When I originally read the docs for splrep, I had it in my head that the splines in scipy.interpolation didn't provide the "right" type of smoothing (don't ask me what "right" means---I have no idea). After taking some time to understand the interpolation module, I realize it does what I want. Thanks, Denis!
>>     
>
> I think I spoke too soon. I believe what I need are called a smoothing splines, which is what scipy.signal provides; on the other I hand, scipy.interpolate creates what I think are called regression splines. The difference being that regression splines smooth the data using fewer knots than the number of observed data, while smoothing splines have knots which match the (abscissa) location of the original data. (I don't use splines often, so please correct me if I'm misinterpreting what I've read about the topic.)
>
> It's possible to specify knots or smoothing with interpolate.splrep, but when specifying both, the smoothing parameter is ignored and the knot placement (plus the error metric) determines the smoothness of the spline. 
>
> I'm trying to re-implement some matlab code (which uses matlab's spaps function) for spline optimization, and the algorithm requires smoothing splines (as opposed to regression splines). Any suggestions?
>   
I literally *just now* (the last hour) wrapped a small subset of the GSL 
splines in Cython. If you're interested tell me and I'll make available 
what I've got. Note that GSL is GPL.

http://www.gnu.org/software/gsl/manual/html_node/Interpolation.html

Dag Sverre


From cycomanic at gmail.com  Sat Feb  6 00:11:31 2010
From: cycomanic at gmail.com (Jochen Schroeder)
Date: Sat, 6 Feb 2010 16:11:31 +1100
Subject: [SciPy-User] Float32 FFT
In-Reply-To: <668054.42551.qm@web33008.mail.mud.yahoo.com>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>
	<bc657ead1002030042u23e05d5vc63bc72030f2318c@mail.gmail.com>
	<668054.42551.qm@web33008.mail.mud.yahoo.com>
Message-ID: <20100206051129.GA3756@jschrod-laptop>

On 02/03/10 02:43, David Baddeley wrote:
> Thanks David & Sebastian,
> 
> will look into the scipy trunk & priithon options. Alternatively might find the motivation to dust off some old fftw3 based code I've got floating round which should in theory be a little faster (I'm doing multiple FFTs of the same size, with the potential to do the memory allocation and plan generation just once at the start).
> 
> cheers,
> David
> 

Hi David,

I've written a python wrapper around fftw3 in ctypes. It has quite a
pythonish interface around the way of doing things in fftw3 (using plans).
I'm planning to rewrite the detection of the c-library, before I do a release
but you can find the code at http://launchpad.net/pyfftw

Cheers
Jochen


> 
> ----- Original Message ----
> From: Sebastian Haase <seb.haase at gmail.com>
> To: David Baddeley <david_baddeley at yahoo.com.au>; SciPy Users List <scipy-user at scipy.org>
> Sent: Wed, 3 February, 2010 9:42:54 PM
> Subject: Re: [SciPy-User] Float32 FFT
> 
> On Wed, Feb 3, 2010 at 3:32 AM, David Baddeley
> <david_baddeley at yahoo.com.au> wrote:
> > Does anyone know if there is a way of doing single precision ffts (more explicitly n-dimensional ffts using fftn) in scipy/numpy? The standard functions all seem to force a conversion to double precision and with large (~1000x1000x50) images the memory starts to add up.
> >
> > I'm running numpy 1.2.1, scipy 0.7.0, python 2.6 on linux x64.
> >
> > Thanks,
> > David
> >
> I have an fftw2 wrapper (SWIGged) in Priithon (see googlecode).
> It is based on Python 2.5.  A kind of "super package" binary version
> (also for 64 bit Linux) is downloadable.
> (FFTW is GPL). (The wrapper could be BSD).
> 
> Regards,
> Sebastian Haase
> 
> 
> 
>       
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From oliphant at enthought.com  Sat Feb  6 02:34:33 2010
From: oliphant at enthought.com (Travis Oliphant)
Date: Sat, 6 Feb 2010 01:34:33 -0600
Subject: [SciPy-User] Weave and OOP
In-Reply-To: <10f88f6d1002020910m1253705dn6473bd38b1b3d7ca@mail.gmail.com>
References: <10f88f6d1002020910m1253705dn6473bd38b1b3d7ca@mail.gmail.com>
Message-ID: <D3DC0AE3-C15B-4A3F-89C2-1EF9A5D39C3D@enthought.com>


On Feb 2, 2010, at 11:10 AM, Ignacio Vergara wrote:

> Hi
>
> I'm trying to use weave inside a class, but it throws me a  
> compilation error.
>
> I would like to know if there is any kind of workaround or if it's a  
> limitation regarding weave

Please post the code and error.   There should not be a problem as far  
as I know.

Thanks,

-Travis



From nahumoz at gmail.com  Sat Feb  6 12:32:16 2010
From: nahumoz at gmail.com (Oz Nahum)
Date: Sat, 6 Feb 2010 18:32:16 +0100
Subject: [SciPy-User] weired results with ode solver
Message-ID: <6ec71d091002060932v17c05fa2r5f1c2b810aa30036@mail.gmail.com>

Hi Everyone,

I'm trying to convert the following matlab code into python, the code
simulates reduction of organic matter by bacteria in a batch system.
It has to files in the matlab version, and 1 in the python version.
I'm not sure I did the conversion correctly, but I'm sure the result
I'm getting from python is wrong - probably because I did not really
understand how to use the solver. Any insights would be appreciated.

Here are the codes:
% This file is a matlab script to simulate a batch problem as described

% in problem 1a.

% This file calls batch1_rate.m


%clear all;

clc;


%Define variables

Coi=3;              % initial concentration of oxigen [mg/l]

Csi=10;             % initial concentration of substrat [mg/l]

Cbi=0.2;            % initial concentration of biomass [mg/l]


%define vector of concentrations

c1=[Coi;Csi;Cbi];


ko=0.1;             % Monod coefficients Oxigen [mg/l]

ks=0.1;             % Monod coefficients Substract [mg/l]


umax=0.125/86400;   % Maximun growth rate [1/d]


Yo=0.125;           % Oxigen Yield [mg X / mg O]

Ys=0.25;            % Substract Yield [mg X / mg S]


Kdec=0.01/86400;    % Decay rate

kb=1;              % Biomass Inhibition constant [mg/L]


%Solve ODE

timerange=[0 43*86400];

[t,c]=ode15s(@batch1_rate,timerange,c1,[],ko,ks,Ys,Yo,Kdec,umax,kb);


%Plot

hf=figure(1);

clf;

%plot(,c(:,1:3),'linewidth',2);

days=t/86400;

plot(days,c(:,1),'b-',days,c(:,2),'k--','linewidth',2);

hold on

plot(days,c(:,3),'-','Color',[0 0.498 0],'linewidth',1);


xlabel('T [days]','fontsize',14);

ylabel('C [mg/L]','fontsize',14);

ylim([0 10]);

xlim([0 22])


#end of 1st matlab file.
% This file is an aid function to the script batch1.m which describes

% a batch system relations of substrate, oxygen and one microbial specie

% over time.

% Oz Nahum, Anibal Perez, Georgia Kastanioti, Januar 2010.





function f=batch1_rate(t,c1,ko,ks,Ys,Yo,Kdec,umax,kb)

co=c1(1); %concentration of Oxygen

cs=c1(2); %concentration of Substrate

X=c1(3); %concentration of bacteria



kgr=umax*(co/(ko+co))*(cs/(ks+cs)); % specific uptake rate [mol/m3/s]

Ib=1+X/kb;  				 		% inhibiton caused by biomass



foxi=((-kgr/Yo/Ib)*X);  %dc/dt of Oxygen

fsub=((-kgr/Ys/Ib)*X);  %dc/dt of Substrate

fbio=((kgr/Ib-Kdec)*X); %inhibition for growth only



f=[foxi;fsub;fbio];

end

#end of second matlab file

#python script:
from scipy.integrate import odeint
from numpy import *
#%clear all;
#clc;

#%Define variables
Coi=3              # initial concentration of oxigen [mg/l]
Csi=10             # initial concentration of substrat [mg/l]
Cbi=0.2            # initial concentration of biomass [mg/l]

#define vector of concentrations
#c1 = array((Coi,Csi,Cbi))
c1=(Coi,Csi,Cbi)
y0=c1
ko=0.1             #% Monod coefficients Oxigen [mg/l]
ks=0.1             #% Monod coefficients Substract [mg/l]

umax=0.125/86400 #  % Maximun growth rate [1/d]

Yo=0.125#;           % Oxigen Yield [mg X / mg O]
Ys=0.25#;            % Substract Yield [mg X / mg S]

Kdec=0.01/86400#;    % Decay rate
kb=1#;              % Biomass Inhibition constant [mg/L]


def rate_func(c,time):
	ko=0.1             #% Monod coefficients Oxigen [mg/l]
	ks=0.1             #% Monod coefficients Substract [mg/l]
	umax=0.125/86400 #  % Maximun growth rate [1/d]
	Yo=0.125#;           % Oxigen Yield [mg X / mg O]
	Ys=0.25#;            % Substract Yield [mg X / mg S]
	Kdec=0.01/86400#;    % Decay rate
	kb=1#;              % Biomass Inhibition constant [mg/L]
	co=c1[0]#; %concentration of Oxygen
	cs=c1[1]#; %concentration of Substrate
	X=c1[2]#; %concentration of bacteria
	kgr=umax*(co/(ko+co))*(cs/(ks+cs))#; % specific uptake rate [mol/m3/s]
	Ib=1+X/kb#;  				 		% inhibiton caused by biomass
	foxi=((-kgr/Yo/Ib)*X)#;  %dc/dt of Oxygen
	fsub=((-kgr/Ys/Ib)*X)#;  %dc/dt of Substrate
	fbio=((kgr/Ib-Kdec)*X)#; %inhibition for growth only
	f=array((foxi,fsub,fbio))
	return  f

#Solve ODE
timerange=arange(1,43*86400,60)
parms=(c1,ko,ks,Ys,Yo,Kdec,umax,kb)
#[t,c]=ode15s(@batch1_rate,timerange,c1,[],ko,ks,Ys,Yo,Kdec,umax,kb);
#c=odeint(rate_func,parms,timerange)
c=odeint(rate_func,c1,timerange)
#sprint c

from pylab import *
plot(timerange, c[:,0], 'b-')
plot(timerange, c[:,1], 'g-')
plot(timerange, c[:,2], 'k-')
show()



Many thanks in advance.


Oz Nahum
Graduate Student
Zentrum f?r Angewandte Geologie
Universit?t T?bingen

---

Imagine there's no countries
it isn't hard to do
Nothing to kill or die for
And no religion too
Imagine all the people
Living life in peace


From warren.weckesser at enthought.com  Sat Feb  6 12:45:26 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sat, 06 Feb 2010 11:45:26 -0600
Subject: [SciPy-User] weired results with ode solver
In-Reply-To: <6ec71d091002060932v17c05fa2r5f1c2b810aa30036@mail.gmail.com>
References: <6ec71d091002060932v17c05fa2r5f1c2b810aa30036@mail.gmail.com>
Message-ID: <4B6DAAB6.1010508@enthought.com>

Oz,

In rate_func(), try changing this:

	co=c1[0]#; %concentration of Oxygen
	cs=c1[1]#; %concentration of Substrate
	X=c1[2]#; %concentration of bacteria


to this

	co = c[0] #; %concentration of Oxygen
	cs = c[1] #; %concentration of Substrate
	X  = c[2] #; %concentration of bacteria


Also, you have all your parameters hard-coded in rate_func().   That's 
fine if you want to work that way, but you could also add additional 
arguments to rate_func(), and use the 'args' keyword argument of odeint 
to tell the solver what the additional arguments are.  There is an 
example in the SciPy cookbook:

    http://www.scipy.org/Cookbook/CoupledSpringMassSystem


Regards,

Warren


Oz Nahum wrote:
> Hi Everyone,
>
> I'm trying to convert the following matlab code into python, the code
> simulates reduction of organic matter by bacteria in a batch system.
> It has to files in the matlab version, and 1 in the python version.
> I'm not sure I did the conversion correctly, but I'm sure the result
> I'm getting from python is wrong - probably because I did not really
> understand how to use the solver. Any insights would be appreciated.
>
> Here are the codes:
> % This file is a matlab script to simulate a batch problem as described
>
> % in problem 1a.
>
> % This file calls batch1_rate.m
>
>
> %clear all;
>
> clc;
>
>
> %Define variables
>
> Coi=3;              % initial concentration of oxigen [mg/l]
>
> Csi=10;             % initial concentration of substrat [mg/l]
>
> Cbi=0.2;            % initial concentration of biomass [mg/l]
>
>
> %define vector of concentrations
>
> c1=[Coi;Csi;Cbi];
>
>
> ko=0.1;             % Monod coefficients Oxigen [mg/l]
>
> ks=0.1;             % Monod coefficients Substract [mg/l]
>
>
> umax=0.125/86400;   % Maximun growth rate [1/d]
>
>
> Yo=0.125;           % Oxigen Yield [mg X / mg O]
>
> Ys=0.25;            % Substract Yield [mg X / mg S]
>
>
> Kdec=0.01/86400;    % Decay rate
>
> kb=1;              % Biomass Inhibition constant [mg/L]
>
>
> %Solve ODE
>
> timerange=[0 43*86400];
>
> [t,c]=ode15s(@batch1_rate,timerange,c1,[],ko,ks,Ys,Yo,Kdec,umax,kb);
>
>
> %Plot
>
> hf=figure(1);
>
> clf;
>
> %plot(,c(:,1:3),'linewidth',2);
>
> days=t/86400;
>
> plot(days,c(:,1),'b-',days,c(:,2),'k--','linewidth',2);
>
> hold on
>
> plot(days,c(:,3),'-','Color',[0 0.498 0],'linewidth',1);
>
>
> xlabel('T [days]','fontsize',14);
>
> ylabel('C [mg/L]','fontsize',14);
>
> ylim([0 10]);
>
> xlim([0 22])
>
>
> #end of 1st matlab file.
> % This file is an aid function to the script batch1.m which describes
>
> % a batch system relations of substrate, oxygen and one microbial specie
>
> % over time.
>
> % Oz Nahum, Anibal Perez, Georgia Kastanioti, Januar 2010.
>
>
>
>
>
> function f=batch1_rate(t,c1,ko,ks,Ys,Yo,Kdec,umax,kb)
>
> co=c1(1); %concentration of Oxygen
>
> cs=c1(2); %concentration of Substrate
>
> X=c1(3); %concentration of bacteria
>
>
>
> kgr=umax*(co/(ko+co))*(cs/(ks+cs)); % specific uptake rate [mol/m3/s]
>
> Ib=1+X/kb;  				 		% inhibiton caused by biomass
>
>
>
> foxi=((-kgr/Yo/Ib)*X);  %dc/dt of Oxygen
>
> fsub=((-kgr/Ys/Ib)*X);  %dc/dt of Substrate
>
> fbio=((kgr/Ib-Kdec)*X); %inhibition for growth only
>
>
>
> f=[foxi;fsub;fbio];
>
> end
>
> #end of second matlab file
>
> #python script:
> from scipy.integrate import odeint
> from numpy import *
> #%clear all;
> #clc;
>
> #%Define variables
> Coi=3              # initial concentration of oxigen [mg/l]
> Csi=10             # initial concentration of substrat [mg/l]
> Cbi=0.2            # initial concentration of biomass [mg/l]
>
> #define vector of concentrations
> #c1 = array((Coi,Csi,Cbi))
> c1=(Coi,Csi,Cbi)
> y0=c1
> ko=0.1             #% Monod coefficients Oxigen [mg/l]
> ks=0.1             #% Monod coefficients Substract [mg/l]
>
> umax=0.125/86400 #  % Maximun growth rate [1/d]
>
> Yo=0.125#;           % Oxigen Yield [mg X / mg O]
> Ys=0.25#;            % Substract Yield [mg X / mg S]
>
> Kdec=0.01/86400#;    % Decay rate
> kb=1#;              % Biomass Inhibition constant [mg/L]
>
>
> def rate_func(c,time):
> 	ko=0.1             #% Monod coefficients Oxigen [mg/l]
> 	ks=0.1             #% Monod coefficients Substract [mg/l]
> 	umax=0.125/86400 #  % Maximun growth rate [1/d]
> 	Yo=0.125#;           % Oxigen Yield [mg X / mg O]
> 	Ys=0.25#;            % Substract Yield [mg X / mg S]
> 	Kdec=0.01/86400#;    % Decay rate
> 	kb=1#;              % Biomass Inhibition constant [mg/L]
> 	co=c1[0]#; %concentration of Oxygen
> 	cs=c1[1]#; %concentration of Substrate
> 	X=c1[2]#; %concentration of bacteria
> 	kgr=umax*(co/(ko+co))*(cs/(ks+cs))#; % specific uptake rate [mol/m3/s]
> 	Ib=1+X/kb#;  				 		% inhibiton caused by biomass
> 	foxi=((-kgr/Yo/Ib)*X)#;  %dc/dt of Oxygen
> 	fsub=((-kgr/Ys/Ib)*X)#;  %dc/dt of Substrate
> 	fbio=((kgr/Ib-Kdec)*X)#; %inhibition for growth only
> 	f=array((foxi,fsub,fbio))
> 	return  f
>
> #Solve ODE
> timerange=arange(1,43*86400,60)
> parms=(c1,ko,ks,Ys,Yo,Kdec,umax,kb)
> #[t,c]=ode15s(@batch1_rate,timerange,c1,[],ko,ks,Ys,Yo,Kdec,umax,kb);
> #c=odeint(rate_func,parms,timerange)
> c=odeint(rate_func,c1,timerange)
> #sprint c
>
> from pylab import *
> plot(timerange, c[:,0], 'b-')
> plot(timerange, c[:,1], 'g-')
> plot(timerange, c[:,2], 'k-')
> show()
>
>
>
> Many thanks in advance.
>
>
> Oz Nahum
> Graduate Student
> Zentrum f?r Angewandte Geologie
> Universit?t T?bingen
>
> ---
>
> Imagine there's no countries
> it isn't hard to do
> Nothing to kill or die for
> And no religion too
> Imagine all the people
> Living life in peace
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From nahumoz at gmail.com  Sat Feb  6 13:33:33 2010
From: nahumoz at gmail.com (Oz Nahum)
Date: Sat, 6 Feb 2010 19:33:33 +0100
Subject: [SciPy-User] weired results with ode solver
Message-ID: <6ec71d091002061033r7f9a15d4iade3404429a1a545@mail.gmail.com>

Hi Warren,
Thanks for your answer,

here is my updated code:
#% This file is a matlab script to simulate a batch problem as described
#% in problem 1a.

from scipy.integrate import odeint
from numpy import *
#%clear all;
#clc;

#%Define variables
Coi=3              # initial concentration of oxigen [mg/l]
Csi=10             # initial concentration of substrat [mg/l]
Cbi=0.2            # initial concentration of biomass [mg/l]

#define vector of concentrations
#c1 = array((Coi,Csi,Cbi))
c1=(Coi,Csi,Cbi)
y0=c1
ko=0.1             #% Monod coefficients Oxigen [mg/l]
ks=0.1             #% Monod coefficients Substract [mg/l]

umax=0.125/86400 #  % Maximun growth rate [1/d]

Yo=0.125#;           % Oxigen Yield [mg X / mg O]
Ys=0.25#;            % Substract Yield [mg X / mg S]

Kdec=0.01/86400#;    % Decay rate
kb=1#;              % Biomass Inhibition constant [mg/L]

def rate_func(c,time,ko,ks,umax,Kdec,Yo,Ys,kb):
	co=c[0]#; %concentration of Oxygen
	cs=c[1]#; %concentration of Substrate
	X=c[2]#; %concentration of bacteria
	kgr=umax*(co/(ko+co))*(cs/(ks+cs))#; % specific uptake rate [mol/m3/s]
	Ib=1+X/kb#;  				 		% inhibiton caused by biomass
	foxi=((-kgr/Yo/Ib)*X)#;  %dc/dt of Oxygen
	fsub=((-kgr/Ys/Ib)*X)#;  %dc/dt of Substrate
	fbio=((kgr/Ib-Kdec)*X)#; %inhibition for growth only
	f=array((foxi,fsub,fbio))
	return  f

#Solve ODE
timerange=arange(1,43*86400,3600)
c=odeint(rate_func,c1,timerange,args=(ko,ks,umax,Kdec,Yo,Ys,kb))

from pylab import *
plot(timerange/86400, c[:,0], 'bo')
plot(timerange/86400, c[:,1], 'go')
plot(timerange/86400, c[:,2], 'ko')
show()


I seems that it runs much faster than matlab (though I have not really
checked). It's syntacticly clearer to read. Much more fun.
There is one issue that bothers me still. I don't want to show the
concentrations every second. So I divide by 86400 sec perday. However,
this results in smeared concentrations, where every day has  a few of
them. Why is that ? how can I remove it ?
A line plot would be nice.

Thanks again,

-- 
Oz Nahum
Graduate Student
Zentrum f?r Angewandte Geologie
Universit?t T?bingen

---

Imagine there's no countries
it isn't hard to do
Nothing to kill or die for
And no religion too
Imagine all the people
Living life in peace


From warren.weckesser at enthought.com  Sat Feb  6 13:55:31 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sat, 06 Feb 2010 12:55:31 -0600
Subject: [SciPy-User] [Fwd: Re: weired results with ode solver]
Message-ID: <4B6DBB23.2010705@enthought.com>

I replied to Oz, but didn't "Reply to all", so for completeness, I'm 
forwarding this to the list.

-------------- next part --------------
An embedded message was scrubbed...
From: Warren Weckesser <warren.weckesser at enthought.com>
Subject: Re: weired results with ode solver
Date: Sat, 06 Feb 2010 12:51:38 -0600
Size: 1550
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100206/7ee7a445/attachment.eml>

From warren.weckesser at enthought.com  Sat Feb  6 13:58:27 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sat, 06 Feb 2010 12:58:27 -0600
Subject: [SciPy-User] weired results with ode solver
In-Reply-To: <6ec71d091002061033r7f9a15d4iade3404429a1a545@mail.gmail.com>
References: <6ec71d091002061033r7f9a15d4iade3404429a1a545@mail.gmail.com>
Message-ID: <4B6DBBD3.1080705@enthought.com>

Oz Nahum wrote:

<snip>
> from pylab import *
> plot(timerange/86400, c[:,0], 'bo')
> plot(timerange/86400, c[:,1], 'go')
> plot(timerange/86400, c[:,2], 'ko')
> show()
>
>
>   
<snip>
> A line plot would be nice.
>
>   

P.S.

To get a line plot, just change the style arguments of the plot 
functions.  E.g.:

plot(timerange/86400.0, c[:,0], 'b')
plot(timerange/86400.0, c[:,1], 'g')
plot(timerange/86400.0, c[:,2], 'k')




From nahumoz at gmail.com  Sat Feb  6 14:12:00 2010
From: nahumoz at gmail.com (Oz Nahum)
Date: Sat, 6 Feb 2010 20:12:00 +0100
Subject: [SciPy-User] weired results with ode solver
In-Reply-To: <4B6DBBD3.1080705@enthought.com>
References: <6ec71d091002061033r7f9a15d4iade3404429a1a545@mail.gmail.com> 
	<4B6DBBD3.1080705@enthought.com>
Message-ID: <6ec71d091002061112t19d1c6f3m733fb8419aa430f9@mail.gmail.com>

AWESOME,
That is wonderful, i got this matlab conversion done in less then 1 hour.

This is absolutely great.

The behavior of array int devision is a bit funny, but after you get
used to it, it's fine.

>>> timerange=arange(1,12)
>>> timerange/2
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5]) #this is why i got multiple values
>>> timerange/2.0
array([ 0.5,  1. ,  1.5,  2. ,  2.5,  3. ,  3.5,  4. ,  4.5,  5. ,  5.5])

 Thank you very much. I'm trying to push forward the use of python in
our department, and this for sure will help.

Oz.

On Sat, Feb 6, 2010 at 7:58 PM, Warren Weckesser
<warren.weckesser at enthought.com> wrote:
> Oz Nahum wrote:
>
> <snip>
>>
>> from pylab import *
>> plot(timerange/86400, c[:,0], 'bo')
>> plot(timerange/86400, c[:,1], 'go')
>> plot(timerange/86400, c[:,2], 'ko')
>> show()
>>
>>
>>
>
> <snip>
>>
>> A line plot would be nice.
>>
>>
>
> P.S.
>
> To get a line plot, just change the style arguments of the plot functions.
> ?E.g.:
>
> plot(timerange/86400.0, c[:,0], 'b')
> plot(timerange/86400.0, c[:,1], 'g')
> plot(timerange/86400.0, c[:,2], 'k')
>
>
>


From tsyu80 at gmail.com  Sat Feb  6 14:25:48 2010
From: tsyu80 at gmail.com (Tony S Yu)
Date: Sat, 6 Feb 2010 14:25:48 -0500
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <4B6CA2F3.10008@student.matnat.uio.no>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu>	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com>	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
	<76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com>
	<4B6CA2F3.10008@student.matnat.uio.no>
Message-ID: <E6517CF3-C57C-436D-BAF4-E5C9D0BF61AF@gmail.com>


On Feb 5, 2010, at 6:00 PM, Dag Sverre Seljebotn wrote:

> Tony S Yu wrote:
>> 
>> 
>> I think I spoke too soon. I believe what I need are called a smoothing splines, which is what scipy.signal provides; on the other I hand, scipy.interpolate creates what I think are called regression splines. The difference being that regression splines smooth the data using fewer knots than the number of observed data, while smoothing splines have knots which match the (abscissa) location of the original data. (I don't use splines often, so please correct me if I'm misinterpreting what I've read about the topic.)
>> 
>> It's possible to specify knots or smoothing with interpolate.splrep, but when specifying both, the smoothing parameter is ignored and the knot placement (plus the error metric) determines the smoothness of the spline. 
>> 
>> I'm trying to re-implement some matlab code (which uses matlab's spaps function) for spline optimization, and the algorithm requires smoothing splines (as opposed to regression splines). Any suggestions?
>> 
> I literally *just now* (the last hour) wrapped a small subset of the GSL 
> splines in Cython. If you're interested tell me and I'll make available 
> what I've got. Note that GSL is GPL.
> 
> http://www.gnu.org/software/gsl/manual/html_node/Interpolation.html
> 
> Dag Sverre


Thanks for your response.

I'm definitely interested to see what you have, but to be honest, the link you sent just made me more confused (as far as terminology goes). It sounds like the interpolating splines in GSL serve the same function as splines in scipy.interpolate without the ability to smooth the data; the spline passes through the data points and the spline is useful for taking derivatives, or as the name suggests, interpolating. 

On the other hand, it sounds like GSL's basis splines act like splines in scipy.interpolate with smoothing turned on: smoothing is achieved using a reduced number of knots and a least squares fit.

What I think I need (and supposedly matlab's spaps provides) is a spline with knots at the data's x-values, plus smoothing---which means the spline doesn't necessarily go through the y-values of the data. (I wish I had a better understanding of all this so I could use the correct terminology.) Does GSL provide this sort of spline?

Thanks,
-Tony



From rob.clewley at gmail.com  Sat Feb  6 14:30:49 2010
From: rob.clewley at gmail.com (Rob Clewley)
Date: Sat, 6 Feb 2010 14:30:49 -0500
Subject: [SciPy-User] weired results with ode solver
In-Reply-To: <6ec71d091002061112t19d1c6f3m733fb8419aa430f9@mail.gmail.com>
References: <6ec71d091002061033r7f9a15d4iade3404429a1a545@mail.gmail.com>
	<4B6DBBD3.1080705@enthought.com>
	<6ec71d091002061112t19d1c6f3m733fb8419aa430f9@mail.gmail.com>
Message-ID: <a749952d1002061130s647e45e0j1b6e74aa8fbf9395@mail.gmail.com>

Hi,

> The behavior of array int devision is a bit funny, but after you get
> used to it, it's fine.

You can always use

from __future__ import division

at the top of all your files and you'll be safe to do this division
(at an almost negligible performance hit). The mathematically
intuitive version of division is standard in Python 3000.

>>>> timerange=arange(1,12)
>>>> timerange/2
> array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5]) #this is why i got multiple values
>>>> timerange/2.0
> array([ 0.5, ?1. , ?1.5, ?2. , ?2.5, ?3. , ?3.5, ?4. , ?4.5, ?5. , ?5.5])

You may wish to check out numpy's linspace command for generating
float arrays with arbitrary step sizes in a numerically safe manner
(and with endpoints included by default).

-Rob


From jsseabold at gmail.com  Sat Feb  6 15:10:26 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sat, 6 Feb 2010 15:10:26 -0500
Subject: [SciPy-User] Wrapping a lapack function (attempt results seg fault)
Message-ID: <c048da1c1002061210g4e2fde84j486c3fb9ebd90b45@mail.gmail.com>

All,

I'm trying to wrap the lapack function dtgsen for the ordered generalized
Schur decomposition given real valued arrays <
http://linux.die.net/man/l/dtgsen>.  However, I'm getting a segmentation
fault, and I have no idea why.  I was hoping someone might look at my work
and see where I've gone wrong, as this is my first attempt at trying to do
something like this.  One specific thing I wasn't sure about is when the
lapack function calls for a LOGICAL type can this just be

from ctypes import c_bool
logical_true = c_bool(True)

?

I've attached my functions and arrays that make a self-contained example.
 Just put the two files in a directory and run.  When I do a gdb backtrace I
receive the following output.

(gdb) run ordqztest.py

Starting program: /usr/bin/python ordqztest.py

[Thread debugging using libthread_db enabled]


Program received signal SIGSEGV, Segmentation fault.
0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
(gdb) backtrace
#0  0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
#1  0x00007ffff6da8bec in ffi_call_unix64 ()
   at
/build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/unix64.S:75
#2  0x00007ffff6da82d3 in ffi_call (cif=0x7fffffffd810, fn=<value optimized
out>,
   rvalue=<value optimized out>, avalue=<value optimized out>)

   at
/build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/ffi64.c:430
#3  0x00007ffff6da2d64 in _call_function_pointer (pProc=<value optimized
out>,
   argtuple=<value optimized out>, flags=<value optimized out>,
argtypes=<value optimized out>,
   restype=<value optimized out>, checker=<value optimized out>)

   at /build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:816

#4  _CallProc (pProc=<value optimized out>, argtuple=<value optimized out>,

   flags=<value optimized out>, argtypes=<value optimized out>,
restype=<value optimized out>,
   checker=<value optimized out>) at
/build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:1163
#5  0x00007ffff6d9abf7 in CFuncPtr_call (self=0xc2c2c0, inargs=0x18,
kwds=<value optimized out>)
   at /build/buildd/python2.6-2.6.4/Modules/_ctypes/_ctypes.c:3860

#6  0x000000000041d6e7 in PyObject_Call (func=0xc2c2c0, arg=0xcb8360,
kw=0xc63520)
   at ../Objects/abstract.c:2492

#7  0x00000000004a199c in do_call (f=0xcb98c0, throwflag=<value optimized
out>)
   at ../Python/ceval.c:3924

#8  call_function (f=0xcb98c0, throwflag=<value optimized out>) at
../Python/ceval.c:3729
#9  PyEval_EvalFrameEx (f=0xcb98c0, throwflag=<value optimized out>) at
../Python/ceval.c:2389
#10 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f01f30,
globals=<value optimized out>,
   locals=<value optimized out>, args=0x9, argcount=<value optimized out>,
kws=<value optimized out>,
   kwcount=2, defs=0x7ffff7f0a7f8, defcount=4, closure=0x0) at
../Python/ceval.c:2968
#11 0x00000000004a245f in fast_function (f=0xcb9660, throwflag=<value
optimized out>)
   at ../Python/ceval.c:3802

#12 call_function (f=0xcb9660, throwflag=<value optimized out>) at
../Python/ceval.c:3727
---Type <return> to continue, or q <return> to quit---

#13 PyEval_EvalFrameEx (f=0xcb9660, throwflag=<value optimized out>) at
../Python/ceval.c:2389
#14 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f026c0,
globals=<value optimized out>,
   locals=<value optimized out>, args=0x8, argcount=<value optimized out>,
kws=<value optimized out>,
   kwcount=2, defs=0x7ffff3ea9e78, defcount=3, closure=0x0) at
../Python/ceval.c:2968
#15 0x00000000004a245f in fast_function (f=0x928e00, throwflag=<value
optimized out>)
   at ../Python/ceval.c:3802
#16 call_function (f=0x928e00, throwflag=<value optimized out>) at
../Python/ceval.c:3727
#17 PyEval_EvalFrameEx (f=0x928e00, throwflag=<value optimized out>) at
../Python/ceval.c:2389
#18 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f02378,
globals=<value optimized out>,
   locals=<value optimized out>, args=0x0, argcount=<value optimized out>,
kws=<value optimized out>,
   kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2968
#19 0x00000000004a41b2 in PyEval_EvalCode (co=0x0, globals=0xcb8360,
locals=0xc63520)
   at ../Python/ceval.c:522
#20 0x00000000004c33a0 in run_mod (fp=0x919900, filename=<value optimized
out>,
   start=<value optimized out>, globals=<value optimized out>,
locals=0x8b9270, closeit=1,
   flags=0x7fffffffe100) at ../Python/pythonrun.c:1335
#21 PyRun_FileExFlags (fp=0x919900, filename=<value optimized out>,
start=<value optimized out>,
   globals=<value optimized out>, locals=0x8b9270, closeit=1,
flags=0x7fffffffe100)
   at ../Python/pythonrun.c:1321
#22 0x00000000004c3564 in PyRun_SimpleFileExFlags (fp=<value optimized out>,
   filename=0x7fffffffe518 "ordqztest.py", closeit=1, flags=0x7fffffffe100)
   at ../Python/pythonrun.c:931
#23 0x0000000000418ab7 in Py_Main (argc=-135384960, argv=<value optimized
out>) at ../Modules/main.c:599
#24 0x00007ffff6fd0abd in __libc_start_main (main=<value optimized out>,
argc=<value optimized out>,
   ubp_av=<value optimized out>, init=<value optimized out>, fini=<value
optimized out>,
   rtld_fini=<value optimized out>, stack_end=0x7fffffffe218) at
libc-start.c:220
#25 0x0000000000417ca9 in _start () at ../sysdeps/x86_64/elf/start.S:113

Any help or pointers would be appreciated, as I am a bit out of my depth.

-Skipper
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100206/ed14f35f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ordqztest.py
Type: text/x-python
Size: 4544 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100206/ed14f35f/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testvals.npz
Type: application/octet-stream
Size: 6942 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100206/ed14f35f/attachment.obj>

From josef.pktd at gmail.com  Sat Feb  6 17:45:26 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 6 Feb 2010 17:45:26 -0500
Subject: [SciPy-User] Wrapping a lapack function (attempt results seg
	fault)
In-Reply-To: <c048da1c1002061210g4e2fde84j486c3fb9ebd90b45@mail.gmail.com>
References: <c048da1c1002061210g4e2fde84j486c3fb9ebd90b45@mail.gmail.com>
Message-ID: <1cd32cbb1002061445t55e1973bl302ca28cb4240638@mail.gmail.com>

On Sat, Feb 6, 2010 at 3:10 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> All,
>
> I'm trying to wrap the lapack function dtgsen for the ordered generalized
> Schur decomposition given real valued arrays
> <http://linux.die.net/man/l/dtgsen>. ?However, I'm getting a segmentation
> fault, and I have no idea why. ?I was hoping someone might look at my work
> and see where I've gone wrong, as this is my first attempt at trying to do
> something like this.? One specific thing I wasn't sure about is when the
> lapack function calls for a LOGICAL type can this just be
>
> from ctypes import c_bool
> logical_true = c_bool(True)
>
> ?
>
> I've attached my functions and arrays that make a self-contained example.
> ?Just put the two files in a directory and run. ?When I do a gdb backtrace I
> receive the following output.
>
> (gdb) run ordqztest.py
>
> Starting program: /usr/bin/python ordqztest.py
>
> [Thread debugging using libthread_db enabled]
>
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
> (gdb) backtrace
> #0 ?0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
> #1 ?0x00007ffff6da8bec in ffi_call_unix64 ()
> ? ?at
> /build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/unix64.S:75
> #2 ?0x00007ffff6da82d3 in ffi_call (cif=0x7fffffffd810, fn=<value optimized
> out>,
> ? ?rvalue=<value optimized out>, avalue=<value optimized out>)
>
> ? ?at
> /build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/ffi64.c:430
> #3 ?0x00007ffff6da2d64 in _call_function_pointer (pProc=<value optimized
> out>,
> ? ?argtuple=<value optimized out>, flags=<value optimized out>,
> argtypes=<value optimized out>,
> ? ?restype=<value optimized out>, checker=<value optimized out>)
>
> ? ?at /build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:816
>
> #4 ?_CallProc (pProc=<value optimized out>, argtuple=<value optimized out>,
>
> ? ?flags=<value optimized out>, argtypes=<value optimized out>,
> restype=<value optimized out>,
> ? ?checker=<value optimized out>) at
> /build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:1163
> #5 ?0x00007ffff6d9abf7 in CFuncPtr_call (self=0xc2c2c0, inargs=0x18,
> kwds=<value optimized out>)
> ? ?at /build/buildd/python2.6-2.6.4/Modules/_ctypes/_ctypes.c:3860
>
> #6 ?0x000000000041d6e7 in PyObject_Call (func=0xc2c2c0, arg=0xcb8360,
> kw=0xc63520)
> ? ?at ../Objects/abstract.c:2492
>
> #7 ?0x00000000004a199c in do_call (f=0xcb98c0, throwflag=<value optimized
> out>)
> ? ?at ../Python/ceval.c:3924
>
> #8 ?call_function (f=0xcb98c0, throwflag=<value optimized out>) at
> ../Python/ceval.c:3729
> #9 ?PyEval_EvalFrameEx (f=0xcb98c0, throwflag=<value optimized out>) at
> ../Python/ceval.c:2389
> #10 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f01f30,
> globals=<value optimized out>,
> ? ?locals=<value optimized out>, args=0x9, argcount=<value optimized out>,
> kws=<value optimized out>,
> ? ?kwcount=2, defs=0x7ffff7f0a7f8, defcount=4, closure=0x0) at
> ../Python/ceval.c:2968
> #11 0x00000000004a245f in fast_function (f=0xcb9660, throwflag=<value
> optimized out>)
> ? ?at ../Python/ceval.c:3802
>
> #12 call_function (f=0xcb9660, throwflag=<value optimized out>) at
> ../Python/ceval.c:3727
> ---Type <return> to continue, or q <return> to quit---
>
> #13 PyEval_EvalFrameEx (f=0xcb9660, throwflag=<value optimized out>) at
> ../Python/ceval.c:2389
> #14 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f026c0,
> globals=<value optimized out>,
> ? ?locals=<value optimized out>, args=0x8, argcount=<value optimized out>,
> kws=<value optimized out>,
> ? ?kwcount=2, defs=0x7ffff3ea9e78, defcount=3, closure=0x0) at
> ../Python/ceval.c:2968
> #15 0x00000000004a245f in fast_function (f=0x928e00, throwflag=<value
> optimized out>)
> ? ?at ../Python/ceval.c:3802
> #16 call_function (f=0x928e00, throwflag=<value optimized out>) at
> ../Python/ceval.c:3727
> #17 PyEval_EvalFrameEx (f=0x928e00, throwflag=<value optimized out>) at
> ../Python/ceval.c:2389
> #18 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f02378,
> globals=<value optimized out>,
> ? ?locals=<value optimized out>, args=0x0, argcount=<value optimized out>,
> kws=<value optimized out>,
> ? ?kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2968
> #19 0x00000000004a41b2 in PyEval_EvalCode (co=0x0, globals=0xcb8360,
> locals=0xc63520)
> ? ?at ../Python/ceval.c:522
> #20 0x00000000004c33a0 in run_mod (fp=0x919900, filename=<value optimized
> out>,
> ? ?start=<value optimized out>, globals=<value optimized out>,
> locals=0x8b9270, closeit=1,
> ? ?flags=0x7fffffffe100) at ../Python/pythonrun.c:1335
> #21 PyRun_FileExFlags (fp=0x919900, filename=<value optimized out>,
> start=<value optimized out>,
> ? ?globals=<value optimized out>, locals=0x8b9270, closeit=1,
> flags=0x7fffffffe100)
> ? ?at ../Python/pythonrun.c:1321
> #22 0x00000000004c3564 in PyRun_SimpleFileExFlags (fp=<value optimized out>,
> ? ?filename=0x7fffffffe518 "ordqztest.py", closeit=1, flags=0x7fffffffe100)
> ? ?at ../Python/pythonrun.c:931
> #23 0x0000000000418ab7 in Py_Main (argc=-135384960, argv=<value optimized
> out>) at ../Modules/main.c:599
> #24 0x00007ffff6fd0abd in __libc_start_main (main=<value optimized out>,
> argc=<value optimized out>,
> ? ?ubp_av=<value optimized out>, init=<value optimized out>, fini=<value
> optimized out>,
> ? ?rtld_fini=<value optimized out>, stack_end=0x7fffffffe218) at
> libc-start.c:220
> #25 0x0000000000417ca9 in _start () at ../sysdeps/x86_64/elf/start.S:113
>
> Any help or pointers would be appreciated, as I am a bit out of my depth.
>
> -Skipper
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>

according to http://www.netlib.org/lapack/double/dtgsen.f  you are
missing alphai  in the argument list.

I don't have (or find) c_bool in python 2.5, so my tries are also unsuccessful.

Maybe wrapping with f2py like the routines in scipy.linalg might be a
better choice than ctypes? However, I haven't seen a recipe how to do
it.

Josef


From jsseabold at gmail.com  Sat Feb  6 18:02:14 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sat, 6 Feb 2010 18:02:14 -0500
Subject: [SciPy-User] Wrapping a lapack function (attempt results seg
	fault)
In-Reply-To: <1cd32cbb1002061445t55e1973bl302ca28cb4240638@mail.gmail.com>
References: <c048da1c1002061210g4e2fde84j486c3fb9ebd90b45@mail.gmail.com> 
	<1cd32cbb1002061445t55e1973bl302ca28cb4240638@mail.gmail.com>
Message-ID: <c048da1c1002061502j4978cfe8u96abedbc7313771d@mail.gmail.com>

On Sat, Feb 6, 2010 at 5:45 PM,  <josef.pktd at gmail.com> wrote:
> On Sat, Feb 6, 2010 at 3:10 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> All,
>>
>> I'm trying to wrap the lapack function dtgsen for the ordered generalized
>> Schur decomposition given real valued arrays
>> <http://linux.die.net/man/l/dtgsen>. ?However, I'm getting a segmentation
>> fault, and I have no idea why. ?I was hoping someone might look at my work
>> and see where I've gone wrong, as this is my first attempt at trying to do
>> something like this.? One specific thing I wasn't sure about is when the
>> lapack function calls for a LOGICAL type can this just be
>>
>> from ctypes import c_bool
>> logical_true = c_bool(True)
>>
>> ?
>>
>> I've attached my functions and arrays that make a self-contained example.
>> ?Just put the two files in a directory and run. ?When I do a gdb backtrace I
>> receive the following output.
>>
>> (gdb) run ordqztest.py
>>
>> Starting program: /usr/bin/python ordqztest.py
>>
>> [Thread debugging using libthread_db enabled]
>>
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
>> (gdb) backtrace
>> #0 ?0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
>> #1 ?0x00007ffff6da8bec in ffi_call_unix64 ()
>> ? ?at
>> /build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/unix64.S:75
>> #2 ?0x00007ffff6da82d3 in ffi_call (cif=0x7fffffffd810, fn=<value optimized
>> out>,
>> ? ?rvalue=<value optimized out>, avalue=<value optimized out>)
>>
>> ? ?at
>> /build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/ffi64.c:430
>> #3 ?0x00007ffff6da2d64 in _call_function_pointer (pProc=<value optimized
>> out>,
>> ? ?argtuple=<value optimized out>, flags=<value optimized out>,
>> argtypes=<value optimized out>,
>> ? ?restype=<value optimized out>, checker=<value optimized out>)
>>
>> ? ?at /build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:816
>>
>> #4 ?_CallProc (pProc=<value optimized out>, argtuple=<value optimized out>,
>>
>> ? ?flags=<value optimized out>, argtypes=<value optimized out>,
>> restype=<value optimized out>,
>> ? ?checker=<value optimized out>) at
>> /build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:1163
>> #5 ?0x00007ffff6d9abf7 in CFuncPtr_call (self=0xc2c2c0, inargs=0x18,
>> kwds=<value optimized out>)
>> ? ?at /build/buildd/python2.6-2.6.4/Modules/_ctypes/_ctypes.c:3860
>>
>> #6 ?0x000000000041d6e7 in PyObject_Call (func=0xc2c2c0, arg=0xcb8360,
>> kw=0xc63520)
>> ? ?at ../Objects/abstract.c:2492
>>
>> #7 ?0x00000000004a199c in do_call (f=0xcb98c0, throwflag=<value optimized
>> out>)
>> ? ?at ../Python/ceval.c:3924
>>
>> #8 ?call_function (f=0xcb98c0, throwflag=<value optimized out>) at
>> ../Python/ceval.c:3729
>> #9 ?PyEval_EvalFrameEx (f=0xcb98c0, throwflag=<value optimized out>) at
>> ../Python/ceval.c:2389
>> #10 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f01f30,
>> globals=<value optimized out>,
>> ? ?locals=<value optimized out>, args=0x9, argcount=<value optimized out>,
>> kws=<value optimized out>,
>> ? ?kwcount=2, defs=0x7ffff7f0a7f8, defcount=4, closure=0x0) at
>> ../Python/ceval.c:2968
>> #11 0x00000000004a245f in fast_function (f=0xcb9660, throwflag=<value
>> optimized out>)
>> ? ?at ../Python/ceval.c:3802
>>
>> #12 call_function (f=0xcb9660, throwflag=<value optimized out>) at
>> ../Python/ceval.c:3727
>> ---Type <return> to continue, or q <return> to quit---
>>
>> #13 PyEval_EvalFrameEx (f=0xcb9660, throwflag=<value optimized out>) at
>> ../Python/ceval.c:2389
>> #14 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f026c0,
>> globals=<value optimized out>,
>> ? ?locals=<value optimized out>, args=0x8, argcount=<value optimized out>,
>> kws=<value optimized out>,
>> ? ?kwcount=2, defs=0x7ffff3ea9e78, defcount=3, closure=0x0) at
>> ../Python/ceval.c:2968
>> #15 0x00000000004a245f in fast_function (f=0x928e00, throwflag=<value
>> optimized out>)
>> ? ?at ../Python/ceval.c:3802
>> #16 call_function (f=0x928e00, throwflag=<value optimized out>) at
>> ../Python/ceval.c:3727
>> #17 PyEval_EvalFrameEx (f=0x928e00, throwflag=<value optimized out>) at
>> ../Python/ceval.c:2389
>> #18 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f02378,
>> globals=<value optimized out>,
>> ? ?locals=<value optimized out>, args=0x0, argcount=<value optimized out>,
>> kws=<value optimized out>,
>> ? ?kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2968
>> #19 0x00000000004a41b2 in PyEval_EvalCode (co=0x0, globals=0xcb8360,
>> locals=0xc63520)
>> ? ?at ../Python/ceval.c:522
>> #20 0x00000000004c33a0 in run_mod (fp=0x919900, filename=<value optimized
>> out>,
>> ? ?start=<value optimized out>, globals=<value optimized out>,
>> locals=0x8b9270, closeit=1,
>> ? ?flags=0x7fffffffe100) at ../Python/pythonrun.c:1335
>> #21 PyRun_FileExFlags (fp=0x919900, filename=<value optimized out>,
>> start=<value optimized out>,
>> ? ?globals=<value optimized out>, locals=0x8b9270, closeit=1,
>> flags=0x7fffffffe100)
>> ? ?at ../Python/pythonrun.c:1321
>> #22 0x00000000004c3564 in PyRun_SimpleFileExFlags (fp=<value optimized out>,
>> ? ?filename=0x7fffffffe518 "ordqztest.py", closeit=1, flags=0x7fffffffe100)
>> ? ?at ../Python/pythonrun.c:931
>> #23 0x0000000000418ab7 in Py_Main (argc=-135384960, argv=<value optimized
>> out>) at ../Modules/main.c:599
>> #24 0x00007ffff6fd0abd in __libc_start_main (main=<value optimized out>,
>> argc=<value optimized out>,
>> ? ?ubp_av=<value optimized out>, init=<value optimized out>, fini=<value
>> optimized out>,
>> ? ?rtld_fini=<value optimized out>, stack_end=0x7fffffffe218) at
>> libc-start.c:220
>> #25 0x0000000000417ca9 in _start () at ../sysdeps/x86_64/elf/start.S:113
>>
>> Any help or pointers would be appreciated, as I am a bit out of my depth.
>>
>> -Skipper
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> according to http://www.netlib.org/lapack/double/dtgsen.f ?you are
> missing alphai ?in the argument list.
>

Ah, that is true.  I wonder why it didn't complain about my arguments
list.  I originally tried to do the complex version, but then realized
I didn't need it as of yet.  Still seg faults though.

> I don't have (or find) c_bool in python 2.5, so my tries are also unsuccessful.
>

This is where I think it might be tripping up, but I don't know if
LOGICAL type can just be a c_int.

> Maybe wrapping with f2py like the routines in scipy.linalg might be a
> better choice than ctypes? However, I haven't seen a recipe how to do
> it.
>

I guess I do have the fortran source file.  I will try this.  I think
I should just have to add the f2py directives for the inputs/outputs
and (fingers-crossed) it will work.

Skipper


From josef.pktd at gmail.com  Sat Feb  6 18:07:57 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 6 Feb 2010 18:07:57 -0500
Subject: [SciPy-User] Wrapping a lapack function (attempt results seg
	fault)
In-Reply-To: <c048da1c1002061502j4978cfe8u96abedbc7313771d@mail.gmail.com>
References: <c048da1c1002061210g4e2fde84j486c3fb9ebd90b45@mail.gmail.com>
	<1cd32cbb1002061445t55e1973bl302ca28cb4240638@mail.gmail.com>
	<c048da1c1002061502j4978cfe8u96abedbc7313771d@mail.gmail.com>
Message-ID: <1cd32cbb1002061507mbe83bcbn116988584c7e9594@mail.gmail.com>

On Sat, Feb 6, 2010 at 6:02 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Sat, Feb 6, 2010 at 5:45 PM, ?<josef.pktd at gmail.com> wrote:
>> On Sat, Feb 6, 2010 at 3:10 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>>> All,
>>>
>>> I'm trying to wrap the lapack function dtgsen for the ordered generalized
>>> Schur decomposition given real valued arrays
>>> <http://linux.die.net/man/l/dtgsen>. ?However, I'm getting a segmentation
>>> fault, and I have no idea why. ?I was hoping someone might look at my work
>>> and see where I've gone wrong, as this is my first attempt at trying to do
>>> something like this.? One specific thing I wasn't sure about is when the
>>> lapack function calls for a LOGICAL type can this just be
>>>
>>> from ctypes import c_bool
>>> logical_true = c_bool(True)
>>>
>>> ?
>>>
>>> I've attached my functions and arrays that make a self-contained example.
>>> ?Just put the two files in a directory and run. ?When I do a gdb backtrace I
>>> receive the following output.
>>>
>>> (gdb) run ordqztest.py
>>>
>>> Starting program: /usr/bin/python ordqztest.py
>>>
>>> [Thread debugging using libthread_db enabled]
>>>
>>>
>>> Program received signal SIGSEGV, Segmentation fault.
>>> 0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
>>> (gdb) backtrace
>>> #0 ?0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
>>> #1 ?0x00007ffff6da8bec in ffi_call_unix64 ()
>>> ? ?at
>>> /build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/unix64.S:75
>>> #2 ?0x00007ffff6da82d3 in ffi_call (cif=0x7fffffffd810, fn=<value optimized
>>> out>,
>>> ? ?rvalue=<value optimized out>, avalue=<value optimized out>)
>>>
>>> ? ?at
>>> /build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/ffi64.c:430
>>> #3 ?0x00007ffff6da2d64 in _call_function_pointer (pProc=<value optimized
>>> out>,
>>> ? ?argtuple=<value optimized out>, flags=<value optimized out>,
>>> argtypes=<value optimized out>,
>>> ? ?restype=<value optimized out>, checker=<value optimized out>)
>>>
>>> ? ?at /build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:816
>>>
>>> #4 ?_CallProc (pProc=<value optimized out>, argtuple=<value optimized out>,
>>>
>>> ? ?flags=<value optimized out>, argtypes=<value optimized out>,
>>> restype=<value optimized out>,
>>> ? ?checker=<value optimized out>) at
>>> /build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:1163
>>> #5 ?0x00007ffff6d9abf7 in CFuncPtr_call (self=0xc2c2c0, inargs=0x18,
>>> kwds=<value optimized out>)
>>> ? ?at /build/buildd/python2.6-2.6.4/Modules/_ctypes/_ctypes.c:3860
>>>
>>> #6 ?0x000000000041d6e7 in PyObject_Call (func=0xc2c2c0, arg=0xcb8360,
>>> kw=0xc63520)
>>> ? ?at ../Objects/abstract.c:2492
>>>
>>> #7 ?0x00000000004a199c in do_call (f=0xcb98c0, throwflag=<value optimized
>>> out>)
>>> ? ?at ../Python/ceval.c:3924
>>>
>>> #8 ?call_function (f=0xcb98c0, throwflag=<value optimized out>) at
>>> ../Python/ceval.c:3729
>>> #9 ?PyEval_EvalFrameEx (f=0xcb98c0, throwflag=<value optimized out>) at
>>> ../Python/ceval.c:2389
>>> #10 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f01f30,
>>> globals=<value optimized out>,
>>> ? ?locals=<value optimized out>, args=0x9, argcount=<value optimized out>,
>>> kws=<value optimized out>,
>>> ? ?kwcount=2, defs=0x7ffff7f0a7f8, defcount=4, closure=0x0) at
>>> ../Python/ceval.c:2968
>>> #11 0x00000000004a245f in fast_function (f=0xcb9660, throwflag=<value
>>> optimized out>)
>>> ? ?at ../Python/ceval.c:3802
>>>
>>> #12 call_function (f=0xcb9660, throwflag=<value optimized out>) at
>>> ../Python/ceval.c:3727
>>> ---Type <return> to continue, or q <return> to quit---
>>>
>>> #13 PyEval_EvalFrameEx (f=0xcb9660, throwflag=<value optimized out>) at
>>> ../Python/ceval.c:2389
>>> #14 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f026c0,
>>> globals=<value optimized out>,
>>> ? ?locals=<value optimized out>, args=0x8, argcount=<value optimized out>,
>>> kws=<value optimized out>,
>>> ? ?kwcount=2, defs=0x7ffff3ea9e78, defcount=3, closure=0x0) at
>>> ../Python/ceval.c:2968
>>> #15 0x00000000004a245f in fast_function (f=0x928e00, throwflag=<value
>>> optimized out>)
>>> ? ?at ../Python/ceval.c:3802
>>> #16 call_function (f=0x928e00, throwflag=<value optimized out>) at
>>> ../Python/ceval.c:3727
>>> #17 PyEval_EvalFrameEx (f=0x928e00, throwflag=<value optimized out>) at
>>> ../Python/ceval.c:2389
>>> #18 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f02378,
>>> globals=<value optimized out>,
>>> ? ?locals=<value optimized out>, args=0x0, argcount=<value optimized out>,
>>> kws=<value optimized out>,
>>> ? ?kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2968
>>> #19 0x00000000004a41b2 in PyEval_EvalCode (co=0x0, globals=0xcb8360,
>>> locals=0xc63520)
>>> ? ?at ../Python/ceval.c:522
>>> #20 0x00000000004c33a0 in run_mod (fp=0x919900, filename=<value optimized
>>> out>,
>>> ? ?start=<value optimized out>, globals=<value optimized out>,
>>> locals=0x8b9270, closeit=1,
>>> ? ?flags=0x7fffffffe100) at ../Python/pythonrun.c:1335
>>> #21 PyRun_FileExFlags (fp=0x919900, filename=<value optimized out>,
>>> start=<value optimized out>,
>>> ? ?globals=<value optimized out>, locals=0x8b9270, closeit=1,
>>> flags=0x7fffffffe100)
>>> ? ?at ../Python/pythonrun.c:1321
>>> #22 0x00000000004c3564 in PyRun_SimpleFileExFlags (fp=<value optimized out>,
>>> ? ?filename=0x7fffffffe518 "ordqztest.py", closeit=1, flags=0x7fffffffe100)
>>> ? ?at ../Python/pythonrun.c:931
>>> #23 0x0000000000418ab7 in Py_Main (argc=-135384960, argv=<value optimized
>>> out>) at ../Modules/main.c:599
>>> #24 0x00007ffff6fd0abd in __libc_start_main (main=<value optimized out>,
>>> argc=<value optimized out>,
>>> ? ?ubp_av=<value optimized out>, init=<value optimized out>, fini=<value
>>> optimized out>,
>>> ? ?rtld_fini=<value optimized out>, stack_end=0x7fffffffe218) at
>>> libc-start.c:220
>>> #25 0x0000000000417ca9 in _start () at ../sysdeps/x86_64/elf/start.S:113
>>>
>>> Any help or pointers would be appreciated, as I am a bit out of my depth.
>>>
>>> -Skipper
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> according to http://www.netlib.org/lapack/double/dtgsen.f ?you are
>> missing alphai ?in the argument list.
>>
>
> Ah, that is true. ?I wonder why it didn't complain about my arguments
> list. ?I originally tried to do the complex version, but then realized
> I didn't need it as of yet. ?Still seg faults though.
>
>> I don't have (or find) c_bool in python 2.5, so my tries are also unsuccessful.
>>
>
> This is where I think it might be tripping up, but I don't know if
> LOGICAL type can just be a c_int.

I tried this, but it wasn't successful, and I have no idea about these types

>
>> Maybe wrapping with f2py like the routines in scipy.linalg might be a
>> better choice than ctypes? However, I haven't seen a recipe how to do
>> it.
>>
>
> I guess I do have the fortran source file. ?I will try this. ?I think
> I should just have to add the f2py directives for the inputs/outputs
> and (fingers-crossed) it will work.

I didn't look carefully, but the netlib with dependencies has a lot of
files, so it might not be so easy to do it standalone.

Here is a open ticket that wraps a new function

http://projects.scipy.org/scipy/ticket/964

maybe good for inspiration

Josef

>
> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From jsseabold at gmail.com  Sat Feb  6 18:12:53 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sat, 6 Feb 2010 18:12:53 -0500
Subject: [SciPy-User] Wrapping a lapack function (attempt results seg
	fault)
In-Reply-To: <1cd32cbb1002061507mbe83bcbn116988584c7e9594@mail.gmail.com>
References: <c048da1c1002061210g4e2fde84j486c3fb9ebd90b45@mail.gmail.com> 
	<1cd32cbb1002061445t55e1973bl302ca28cb4240638@mail.gmail.com> 
	<c048da1c1002061502j4978cfe8u96abedbc7313771d@mail.gmail.com> 
	<1cd32cbb1002061507mbe83bcbn116988584c7e9594@mail.gmail.com>
Message-ID: <c048da1c1002061512q63120672ia6c22b6f693b2e71@mail.gmail.com>

On Sat, Feb 6, 2010 at 6:07 PM,  <josef.pktd at gmail.com> wrote:
> On Sat, Feb 6, 2010 at 6:02 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> On Sat, Feb 6, 2010 at 5:45 PM, ?<josef.pktd at gmail.com> wrote:
>>> On Sat, Feb 6, 2010 at 3:10 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>>>> All,
>>>>
>>>> I'm trying to wrap the lapack function dtgsen for the ordered generalized
>>>> Schur decomposition given real valued arrays
>>>> <http://linux.die.net/man/l/dtgsen>. ?However, I'm getting a segmentation
>>>> fault, and I have no idea why. ?I was hoping someone might look at my work
>>>> and see where I've gone wrong, as this is my first attempt at trying to do
>>>> something like this.? One specific thing I wasn't sure about is when the
>>>> lapack function calls for a LOGICAL type can this just be
>>>>
>>>> from ctypes import c_bool
>>>> logical_true = c_bool(True)
>>>>
>>>> ?
>>>>
>>>> I've attached my functions and arrays that make a self-contained example.
>>>> ?Just put the two files in a directory and run. ?When I do a gdb backtrace I
>>>> receive the following output.
>>>>
>>>> (gdb) run ordqztest.py
>>>>
>>>> Starting program: /usr/bin/python ordqztest.py
>>>>
>>>> [Thread debugging using libthread_db enabled]
>>>>
>>>>
>>>> Program received signal SIGSEGV, Segmentation fault.
>>>> 0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
>>>> (gdb) backtrace
>>>> #0 ?0x00007ffff5015d7d in dtgsen_ () from /usr/lib/liblapack.so
>>>> #1 ?0x00007ffff6da8bec in ffi_call_unix64 ()
>>>> ? ?at
>>>> /build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/unix64.S:75
>>>> #2 ?0x00007ffff6da82d3 in ffi_call (cif=0x7fffffffd810, fn=<value optimized
>>>> out>,
>>>> ? ?rvalue=<value optimized out>, avalue=<value optimized out>)
>>>>
>>>> ? ?at
>>>> /build/buildd/python2.6-2.6.4/Modules/_ctypes/libffi/src/x86/ffi64.c:430
>>>> #3 ?0x00007ffff6da2d64 in _call_function_pointer (pProc=<value optimized
>>>> out>,
>>>> ? ?argtuple=<value optimized out>, flags=<value optimized out>,
>>>> argtypes=<value optimized out>,
>>>> ? ?restype=<value optimized out>, checker=<value optimized out>)
>>>>
>>>> ? ?at /build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:816
>>>>
>>>> #4 ?_CallProc (pProc=<value optimized out>, argtuple=<value optimized out>,
>>>>
>>>> ? ?flags=<value optimized out>, argtypes=<value optimized out>,
>>>> restype=<value optimized out>,
>>>> ? ?checker=<value optimized out>) at
>>>> /build/buildd/python2.6-2.6.4/Modules/_ctypes/callproc.c:1163
>>>> #5 ?0x00007ffff6d9abf7 in CFuncPtr_call (self=0xc2c2c0, inargs=0x18,
>>>> kwds=<value optimized out>)
>>>> ? ?at /build/buildd/python2.6-2.6.4/Modules/_ctypes/_ctypes.c:3860
>>>>
>>>> #6 ?0x000000000041d6e7 in PyObject_Call (func=0xc2c2c0, arg=0xcb8360,
>>>> kw=0xc63520)
>>>> ? ?at ../Objects/abstract.c:2492
>>>>
>>>> #7 ?0x00000000004a199c in do_call (f=0xcb98c0, throwflag=<value optimized
>>>> out>)
>>>> ? ?at ../Python/ceval.c:3924
>>>>
>>>> #8 ?call_function (f=0xcb98c0, throwflag=<value optimized out>) at
>>>> ../Python/ceval.c:3729
>>>> #9 ?PyEval_EvalFrameEx (f=0xcb98c0, throwflag=<value optimized out>) at
>>>> ../Python/ceval.c:2389
>>>> #10 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f01f30,
>>>> globals=<value optimized out>,
>>>> ? ?locals=<value optimized out>, args=0x9, argcount=<value optimized out>,
>>>> kws=<value optimized out>,
>>>> ? ?kwcount=2, defs=0x7ffff7f0a7f8, defcount=4, closure=0x0) at
>>>> ../Python/ceval.c:2968
>>>> #11 0x00000000004a245f in fast_function (f=0xcb9660, throwflag=<value
>>>> optimized out>)
>>>> ? ?at ../Python/ceval.c:3802
>>>>
>>>> #12 call_function (f=0xcb9660, throwflag=<value optimized out>) at
>>>> ../Python/ceval.c:3727
>>>> ---Type <return> to continue, or q <return> to quit---
>>>>
>>>> #13 PyEval_EvalFrameEx (f=0xcb9660, throwflag=<value optimized out>) at
>>>> ../Python/ceval.c:2389
>>>> #14 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f026c0,
>>>> globals=<value optimized out>,
>>>> ? ?locals=<value optimized out>, args=0x8, argcount=<value optimized out>,
>>>> kws=<value optimized out>,
>>>> ? ?kwcount=2, defs=0x7ffff3ea9e78, defcount=3, closure=0x0) at
>>>> ../Python/ceval.c:2968
>>>> #15 0x00000000004a245f in fast_function (f=0x928e00, throwflag=<value
>>>> optimized out>)
>>>> ? ?at ../Python/ceval.c:3802
>>>> #16 call_function (f=0x928e00, throwflag=<value optimized out>) at
>>>> ../Python/ceval.c:3727
>>>> #17 PyEval_EvalFrameEx (f=0x928e00, throwflag=<value optimized out>) at
>>>> ../Python/ceval.c:2389
>>>> #18 0x00000000004a40e0 in PyEval_EvalCodeEx (co=0x7ffff7f02378,
>>>> globals=<value optimized out>,
>>>> ? ?locals=<value optimized out>, args=0x0, argcount=<value optimized out>,
>>>> kws=<value optimized out>,
>>>> ? ?kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:2968
>>>> #19 0x00000000004a41b2 in PyEval_EvalCode (co=0x0, globals=0xcb8360,
>>>> locals=0xc63520)
>>>> ? ?at ../Python/ceval.c:522
>>>> #20 0x00000000004c33a0 in run_mod (fp=0x919900, filename=<value optimized
>>>> out>,
>>>> ? ?start=<value optimized out>, globals=<value optimized out>,
>>>> locals=0x8b9270, closeit=1,
>>>> ? ?flags=0x7fffffffe100) at ../Python/pythonrun.c:1335
>>>> #21 PyRun_FileExFlags (fp=0x919900, filename=<value optimized out>,
>>>> start=<value optimized out>,
>>>> ? ?globals=<value optimized out>, locals=0x8b9270, closeit=1,
>>>> flags=0x7fffffffe100)
>>>> ? ?at ../Python/pythonrun.c:1321
>>>> #22 0x00000000004c3564 in PyRun_SimpleFileExFlags (fp=<value optimized out>,
>>>> ? ?filename=0x7fffffffe518 "ordqztest.py", closeit=1, flags=0x7fffffffe100)
>>>> ? ?at ../Python/pythonrun.c:931
>>>> #23 0x0000000000418ab7 in Py_Main (argc=-135384960, argv=<value optimized
>>>> out>) at ../Modules/main.c:599
>>>> #24 0x00007ffff6fd0abd in __libc_start_main (main=<value optimized out>,
>>>> argc=<value optimized out>,
>>>> ? ?ubp_av=<value optimized out>, init=<value optimized out>, fini=<value
>>>> optimized out>,
>>>> ? ?rtld_fini=<value optimized out>, stack_end=0x7fffffffe218) at
>>>> libc-start.c:220
>>>> #25 0x0000000000417ca9 in _start () at ../sysdeps/x86_64/elf/start.S:113
>>>>
>>>> Any help or pointers would be appreciated, as I am a bit out of my depth.
>>>>
>>>> -Skipper
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>
>>> according to http://www.netlib.org/lapack/double/dtgsen.f ?you are
>>> missing alphai ?in the argument list.
>>>
>>
>> Ah, that is true. ?I wonder why it didn't complain about my arguments
>> list. ?I originally tried to do the complex version, but then realized
>> I didn't need it as of yet. ?Still seg faults though.
>>
>>> I don't have (or find) c_bool in python 2.5, so my tries are also unsuccessful.
>>>
>>
>> This is where I think it might be tripping up, but I don't know if
>> LOGICAL type can just be a c_int.
>
> I tried this, but it wasn't successful, and I have no idea about these types
>
>>
>>> Maybe wrapping with f2py like the routines in scipy.linalg might be a
>>> better choice than ctypes? However, I haven't seen a recipe how to do
>>> it.
>>>
>>
>> I guess I do have the fortran source file. ?I will try this. ?I think
>> I should just have to add the f2py directives for the inputs/outputs
>> and (fingers-crossed) it will work.
>
> I didn't look carefully, but the netlib with dependencies has a lot of
> files, so it might not be so easy to do it standalone.
>

As I just discovered...

> Here is a open ticket that wraps a new function
>
> http://projects.scipy.org/scipy/ticket/964
>
> maybe good for inspiration
>
> Josef
>
>>
>> Skipper
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From nancy.leelee at yahoo.com  Sat Feb  6 21:23:10 2010
From: nancy.leelee at yahoo.com (Nancy Lee)
Date: Sat, 6 Feb 2010 18:23:10 -0800 (PST)
Subject: [SciPy-User] Install scikits.timeseries error: Unable to find
	vcvarsall.bat'
Message-ID: <611757.25677.qm@web111913.mail.gq1.yahoo.com>



Hi,
I am very new with Python and sikits.timeseries package. I tired to install scikits.timeseries-0.91.3.tar.gz, I unzip it and 
use 'python setup.py install ' command. However, it shows error msg as

'No module named msvccompiler in numpy.distutils; trying from distutils
customize MSVCCompiler
customize MSVCCompiler using build_ext
building 'scikits.timeseries.cseries' extension
compiling C sources
error: Unable to find vcvarsall.bat'

My OS is win 32 and the?scikits.timeseries-0.91.3.win32-py2.6.exe file works fine for me , but for some specifit reason, I?need to use the setup.py to install the package.
Could anyone pls help me? Thanks!

Nancy


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100206/81a49829/attachment.html>

From mattknox.ca at gmail.com  Sun Feb  7 11:33:48 2010
From: mattknox.ca at gmail.com (Matt Knox)
Date: Sun, 7 Feb 2010 16:33:48 +0000 (UTC)
Subject: [SciPy-User]
	=?utf-8?q?Install_scikits=2Etimeseries_error=3A_Unab?=
	=?utf-8?q?le_to_find=09vcvarsall=2Ebat=27?=
References: <611757.25677.qm@web111913.mail.gq1.yahoo.com>
Message-ID: <loom.20100207T171934-153@post.gmane.org>

Nancy Lee <nancy.leelee <at> yahoo.com> writes:

> Hi,
> I am very new with Python and sikits.timeseries package. I tired to install
> scikits.timeseries-0.91.3.tar.gz, I unzip it and use
> 'python setup.py install ' command. However, it shows error msg as
> ?
> 'No module named msvccompiler in numpy.distutils; trying from
> distutilscustomize MSVCCompilercustomize MSVCCompiler using build_extbuilding
> 'scikits.timeseries.cseries' extensioncompiling C sourceserror: Unable to
> find vcvarsall.bat'
> ?
> My OS is win 32 and the?scikits.timeseries-0.91.3.win32-py2.6.exe file works
> fine for me , but for some specifit reason, I?need to use the setup.py to
> install the package. Could anyone pls help me? Thanks!
> ?
> Nancy

The time series module includes extensions written in C which require that
you have a compiler installed to build them. If you are using the official
python 2.6 release from python.org, and the official pre-compiled numpy
releases from sourceforge, then it *should* work with either visual studio
2008 (including the express edition), or mingw. Personally, I have been using
TDM's mingw releases from here: http://www.tdragon.net/recentgcc/ and it works
fine for me. I believe if you are using the activestate python distribution, it
won't work out of the box with mingw, you'd have to use Visual studio for that.

Note that by default, setup.py will try to compile with a microsoft compiler
on windows, you need to configure python to use mingw instead if you want to do
that. I think there is a command line argument you can pass to setup.py to make
it use mingw, but I just create a "distutils.cfg" file. Just create a text file
with the following two lines in it:

[build]
compiler = mingw32


Then save it as "C:\Python26\Lib\distutils\distutils.cfg".

- Matt




From jsseabold at gmail.com  Sun Feb  7 16:17:42 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sun, 7 Feb 2010 16:17:42 -0500
Subject: [SciPy-User] help with f2py directives for lapack function wrapper
Message-ID: <c048da1c1002071317o139f5f07q2868c1d13b042eb4@mail.gmail.com>

I abandoned the ctypes route to exposing dtgsen and am trying the f2py
route.  I'm hoping someone might be able to answer my questions.  The f2py
documentation is pretty good, and I am beginning to feel like I have some
idea of what I am doing, but I am running into problems with a few f2py
directives.  For the record, I am only interested in the function for the
case when IJOB=0.

I have produced a file ordqz.so that I can import and look at the docstring,
but when I try to use the function I get

ValueError: failed to create intent(cache|hide)|optional array-- must have
defined dimensions but got (-1,)

Which I assume comes from the LWORK argument as stated in the docs.

I have defined M, PL, PR, WORK, LWORK, IWORK, and LIWORK as follows in
ordqz.pyf based on the docs <http://linux.die.net/man/l/dtgsen> (in-line
below).  These should be defined based on the value of IJOB.  Is there some
other way to go about this, or can someone just point me to a relevant
function that's already in the numpy/scipy source as a reference for a
similar situation?

!xxx - made up dimensions for pl and pr, and dependency of m
           integer depend(pl),intent(hide):: m = shape(pl,0)
           double precision intent(out),dimension(n):: pl
           double precision intent(out),dimension(n):: pr
           double precision dimension(2),intent(out):: dif
           double precision
intent(hide,cache),dimension(lwork),depend(lwork):: work
!Made up a value for lwork (its lower bound) 4*n+16
           integer intent(hide),depend(n):: lwork=4*n+16
           integer intent(hide,cache),dimension(liwork),depend(liwork)::
iwork
!Made up a value for liwork (its lower bound) n+6
           integer intent(hide):: liwork=n+6


Relevant part of docs
-------
M (output) INTEGER
   The dimension of the specified pair of left and right eigen- spaces
(deflating subspaces). 0 <= M <= N.

   PL, PR (output) DOUBLE PRECISION If IJOB = 1, 4 or 5, PL, PR are lower
bounds on the reciprocal of the norm of "projections" onto left and right
eigenspaces with respect to the selected cluster. 0 < PL, PR <= 1. If M = 0
or M = N, PL = PR = 1. If IJOB = 0, 2 or 3, PL and PR are not referenced.
DIF (output) DOUBLE PRECISION array, dimension (2).
   If IJOB >= 2, DIF(1:2) store the estimates of Difu and Difl.
   If IJOB = 2 or 4, DIF(1:2) are F-norm-based upper bounds on
   Difu and Difl. If IJOB = 3 or 5, DIF(1:2) are 1-norm-based estimates of
Difu and Difl. If M = 0 or N, DIF(1:2) = F-norm([A, B]). If IJOB = 0 or 1,
DIF is not referenced.
WORK (workspace/output) DOUBLE PRECISION array, dimension (LWORK)
   IF IJOB = 0, WORK is not referenced. Otherwise, on exit, if INFO = 0,
WORK(1) returns the optimal LWORK.
LWORK (input) INTEGER
   The dimension of the array WORK. LWORK >= 4*N+16. If IJOB = 1, 2 or 4,
LWORK >= MAX(4*N+16, 2*M*(N-M)). If IJOB = 3 or 5, LWORK >= MAX(4*N+16,
4*M*(N-M)).

   If LWORK = -1, then a workspace query is assumed; the routine only
calculates the optimal size of the WORK array, returns this value as the
first entry of the WORK array, and no error message related to LWORK is
issued by XERBLA.
IWORK (workspace/output) INTEGER array, dimension (LIWORK)
   IF IJOB = 0, IWORK is not referenced. Otherwise, on exit, if INFO = 0,
IWORK(1) returns the optimal LIWORK.
LIWORK (input) INTEGER
   The dimension of the array IWORK. LIWORK >= 1. If IJOB = 1, 2 or 4,
LIWORK >= N+6. If IJOB = 3 or 5, LIWORK >= MAX(2*M*(N-M), N+6).

   If LIWORK = -1, then a workspace query is assumed; the routine only
calculates the optimal size of the IWORK array, returns this value as the
first entry of the IWORK array, and no error message related to LIWORK is
issued by XERBLA.

Thanks for any help,

Skipper
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100207/ee605c1d/attachment.html>

From jsseabold at gmail.com  Sun Feb  7 21:06:46 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sun, 7 Feb 2010 21:06:46 -0500
Subject: [SciPy-User] help with f2py directives for lapack function
	wrapper
In-Reply-To: <c048da1c1002071317o139f5f07q2868c1d13b042eb4@mail.gmail.com>
References: <c048da1c1002071317o139f5f07q2868c1d13b042eb4@mail.gmail.com>
Message-ID: <c048da1c1002071806r23ae6f4fna72dd71a22f127fd@mail.gmail.com>

On Sun, Feb 7, 2010 at 4:17 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
<snip>
> !xxx - made up dimensions for pl and pr, and dependency of m
> ? ? ? ? ? ?integer depend(pl),intent(hide):: m = shape(pl,0)
> ? ? ? ? ? ?double precision intent(out),dimension(n):: pl
> ? ? ? ? ? ?double precision intent(out),dimension(n):: pr
> ? ? ? ? ? ?double precision dimension(2),intent(out):: dif
> ? ? ? ? ? ?double precision

D'oh, took a break and came back and realized that pl and pr are not
arrays.  Cutting out the dimension argument and everything works as
expected.  Now I just have to clean up the call sig, etc..  Sorry for
the noise (again).

Skipper


From klaus.kopec at tuebingen.mpg.de  Mon Feb  8 07:40:14 2010
From: klaus.kopec at tuebingen.mpg.de (Klaus Kopec)
Date: Mon, 8 Feb 2010 04:40:14 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Float32 FFT
In-Reply-To: <4B68E27F.6000300@silveregg.co.jp>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>
	<4B68E27F.6000300@silveregg.co.jp>
Message-ID: <27499232.post@talk.nabble.com>



David Cournapeau-3 wrote:
> 
> It is supported in scipy trunk (upcoming 0.8.0). It is transparent, that 
> is if your input is single precision, the output is as well.
> 

Maybe I am mistaken, but I think that only fft, but not fftn, is capable of
fixed-point computations. I checked out the current SVN of scipy a couple of
minutes ago, and a simple test with a float32 array as input returned a
complex64 array for fft and a complex128 array for fftn. 

I am not sure, whether this is a bug, wrong use by me, or a not yet
implemented feature, as the scipy changelog under
http://projects.scipy.org/scipy/milestone/0.8.0 is not very specific: "fft
functions can now handle single precision inputs". 'fft' could refer to ALL
fftpack-functions (=> fftn not doing it would be a bug), or it could really
just mean the 1D ones (=> fixed point would be a missing feature for
N-dimensional FFTs). 

Please help me understand this, as I would like to use fftn with fixed
precision as well.

Best regards, 
Klaus
-- 
View this message in context: http://old.nabble.com/Float32-FFT-tp27430622p27499232.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From cournape at gmail.com  Mon Feb  8 10:19:58 2010
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 9 Feb 2010 00:19:58 +0900
Subject: [SciPy-User] [SciPy-user] Float32 FFT
In-Reply-To: <27499232.post@talk.nabble.com>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>
	<4B68E27F.6000300@silveregg.co.jp> <27499232.post@talk.nabble.com>
Message-ID: <5b8d13221002080719m39f4f8dcq4d442f285e4af823@mail.gmail.com>

On Mon, Feb 8, 2010 at 9:40 PM, Klaus Kopec
<klaus.kopec at tuebingen.mpg.de> wrote:
>
>
> David Cournapeau-3 wrote:
>>
>> It is supported in scipy trunk (upcoming 0.8.0). It is transparent, that
>> is if your input is single precision, the output is as well.
>>
>
> Maybe I am mistaken, but I think that only fft, but not fftn, is capable of
> fixed-point computations. I checked out the current SVN of scipy a couple of
> minutes ago, and a simple test with a float32 array as input returned a
> complex64 array for fft and a complex128 array for fftn.

Note that neither float32 nor float64 are fixed point types - they are
float point. Scipy does not have any fft support for fixed point.

>
> I am not sure, whether this is a bug, wrong use by me, or a not yet
> implemented feature, as the scipy changelog under
> http://projects.scipy.org/scipy/milestone/0.8.0 is not very specific: "fft
> functions can now handle single precision inputs". 'fft' could refer to ALL
> fftpack-functions (=> fftn not doing it would be a bug), or it could really
> just mean the 1D ones (=> fixed point would be a missing feature for
> N-dimensional FFTs).

The double precision support for fftn was a bug, this is now fixed.

David


From klaus.kopec at tuebingen.mpg.de  Mon Feb  8 10:49:11 2010
From: klaus.kopec at tuebingen.mpg.de (Klaus Kopec)
Date: Mon, 08 Feb 2010 16:49:11 +0100
Subject: [SciPy-User] [SciPy-user] Float32 FFT
In-Reply-To: <5b8d13221002080719m39f4f8dcq4d442f285e4af823@mail.gmail.com>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>	<4B68E27F.6000300@silveregg.co.jp>
	<27499232.post@talk.nabble.com>
	<5b8d13221002080719m39f4f8dcq4d442f285e4af823@mail.gmail.com>
Message-ID: <4B703277.50305@tuebingen.mpg.de>

On 02/08/2010 04:19 PM, David Cournapeau wrote:
> The double precision support for fftn was a bug, this is now fixed.
Thank you for fixing the bug so quickly.

The only remaining problem with fftn now is ifftn, which is apparently 
not yet implemented for complex64 input (according to the 
NotYetImplemented error and the fftpack.basic source code for ifftn).

Do you (or anybody else reading this thread) know, whether this will be 
implemented soon?


From klaus.kopec at tuebingen.mpg.de  Mon Feb  8 11:01:30 2010
From: klaus.kopec at tuebingen.mpg.de (Klaus Kopec)
Date: Mon, 08 Feb 2010 17:01:30 +0100
Subject: [SciPy-User] [SciPy-user] Float32 FFT
In-Reply-To: <4B703277.50305@tuebingen.mpg.de>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>	<4B68E27F.6000300@silveregg.co.jp>	<27499232.post@talk.nabble.com>	<5b8d13221002080719m39f4f8dcq4d442f285e4af823@mail.gmail.com>
	<4B703277.50305@tuebingen.mpg.de>
Message-ID: <4B70355A.1060203@tuebingen.mpg.de>

On 02/08/2010 04:49 PM, Klaus Kopec wrote:
> On 02/08/2010 04:19 PM, David Cournapeau wrote:
>    
>> The double precision support for fftn was a bug, this is now fixed.
>>      
> The only remaining problem with fftn now is ifftn, which is apparently
> not yet implemented for complex64 input (according to the
> NotYetImplemented error and the fftpack.basic source code for ifftn).
Sorry for posting twice, but if I am not mistaken (even though I have no 
clue what I am doing but just copy/pasted from normal ifft), ifftn for 
complex64 seems to work after replacing the code throwing the 
NotImplementedError with

overwrite_x = overwrite_x or (tmp is not x and not \
                                                   hasattr(x,'__array__'))
work_function = fftpack.cfftnd

Is this the correct way of doing it?

Bye,
Klaus


From nancy.leelee at yahoo.com  Mon Feb  8 14:54:10 2010
From: nancy.leelee at yahoo.com (Nancy Lee)
Date: Mon, 8 Feb 2010 11:54:10 -0800 (PST)
Subject: [SciPy-User] Install scikits.timeseries error: Unable to
	find	vcvarsall.bat'
In-Reply-To: <loom.20100207T171934-153@post.gmane.org>
References: <611757.25677.qm@web111913.mail.gq1.yahoo.com>
	<loom.20100207T171934-153@post.gmane.org>
Message-ID: <910258.38473.qm@web111903.mail.gq1.yahoo.com>

Matt, Thanks a lot. 
It works!



________________________________
From: Matt Knox <mattknox.ca at gmail.com>
To: scipy-user at scipy.org
Sent: Sun, February 7, 2010 10:33:48 AM
Subject: Re: [SciPy-User] Install scikits.timeseries error: Unable to find vcvarsall.bat'

Nancy Lee <nancy.leelee <at> yahoo.com> writes:

> Hi,
> I am very new with Python and sikits.timeseries package. I tired to install
> scikits.timeseries-0.91.3.tar.gz, I unzip it and use
> 'python setup.py install ' command. However, it shows error msg as
> ?
> 'No module named msvccompiler in numpy.distutils; trying from
> distutilscustomize MSVCCompilercustomize MSVCCompiler using build_extbuilding
> 'scikits.timeseries.cseries' extensioncompiling C sourceserror: Unable to
> find vcvarsall.bat'
> ?
> My OS is win 32 and the?scikits.timeseries-0.91.3.win32-py2.6.exe file works
> fine for me , but for some specifit reason, I?need to use the setup.py to
> install the package. Could anyone pls help me? Thanks!
> ?
> Nancy

The time series module includes extensions written in C which require that
you have a compiler installed to build them. If you are using the official
python 2.6 release from python.org, and the official pre-compiled numpy
releases from sourceforge, then it *should* work with either visual studio
2008 (including the express edition), or mingw. Personally, I have been using
TDM's mingw releases from here: http://www.tdragon.net/recentgcc/ and it works
fine for me. I believe if you are using the activestate python distribution, it
won't work out of the box with mingw, you'd have to use Visual studio for that.

Note that by default, setup.py will try to compile with a microsoft compiler
on windows, you need to configure python to use mingw instead if you want to do
that. I think there is a command line argument you can pass to setup.py to make
it use mingw, but I just create a "distutils.cfg" file. Just create a text file
with the following two lines in it:

[build]
compiler = mingw32


Then save it as "C:\Python26\Lib\distutils\distutils.cfg".

- Matt


_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100208/097a71a0/attachment.html>

From nancy.leelee at yahoo.com  Mon Feb  8 15:09:28 2010
From: nancy.leelee at yahoo.com (Nancy Lee)
Date: Mon, 8 Feb 2010 12:09:28 -0800 (PST)
Subject: [SciPy-User] scikits.timeseries 'Combine TimeSeries objects'
Message-ID: <180695.59792.qm@web111905.mail.gq1.yahoo.com>



Hi everyone,

I have one question about the scikits.timeseries TimeSeries objects.
Is there any function that can combine multiple timeseries objects into one in the correct chronological order?
For example,? I have A timeseries object with '2009-02-03' data, and B timeseries object with?'2009-02-04' data, and C with '2009-02-05' data, how can I get one timeseries from 2009-02-03 to 2009-02-05 ?
Any suggestion? thanks a lot!

Nancy


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100208/11418839/attachment.html>

From nwagner at iam.uni-stuttgart.de  Mon Feb  8 15:14:50 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 08 Feb 2010 21:14:50 +0100
Subject: [SciPy-User] Roots  of a signal
Message-ID: <web-130081776@uni-stuttgart.de>

Hi all,

How can I compute the roots of a signal ?

The following didn't work due to a memory error.

from scipy.interpolate import splrep, splev, sproot
from numpy import exp, sin, linspace, pi
from pylab import plot, show, legend
t = linspace(0,4,20)
y = exp(-0.1*t)*sin(2*pi*t)
plot(t,y,label='Signal')
tck = splrep(t,y)
t_new = linspace(0,10,40)
y2 = splev(t_new,tck)
plot(t_new,y2,label='Spline')
roots = sproot(tck,10)
legend(loc=0)
show()


Any pointer would be appreciated.

Thanks in advance

                                  Nils


From pgmdevlist at gmail.com  Mon Feb  8 15:15:50 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 8 Feb 2010 15:15:50 -0500
Subject: [SciPy-User] scikits.timeseries 'Combine TimeSeries objects'
In-Reply-To: <180695.59792.qm@web111905.mail.gq1.yahoo.com>
References: <180695.59792.qm@web111905.mail.gq1.yahoo.com>
Message-ID: <309FC0FB-1CF2-4D41-B6B6-4D7190FE787B@gmail.com>

On Feb 8, 2010, at 3:09 PM, Nancy Lee wrote:
> 
>  
> Hi everyone,
>  
> I have one question about the scikits.timeseries TimeSeries objects.
> Is there any function that can combine multiple timeseries objects into one in the correctchronological order?
> For example,  I have A timeseries object with '2009-02-03' data, and B timeseries object with '2009-02-04' data, and C with '2009-02-05' data, how can I get one timeseries from 2009-02-03 to 2009-02-05 ?
> Any suggestion? thanks a lot!

Only one date per TimeSeries ? Mmh, that's not really how it's supposed to work, but hey...
Try ts.concatenate
>>>  A=ts.time_series([1],start_date="2010-01",freq="M")
>>> B=ts.time_series([2],start_date="2009-12",freq="M")
>>> C=ts.time_series([3],start_date="2010-02",freq="M")
>>> ts.concatenate((A,B,C))
timeseries([2 1 3],
   dates = [Dec-2009 ... Feb-2010],
   freq  = M)

From josef.pktd at gmail.com  Mon Feb  8 16:10:48 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 8 Feb 2010 16:10:48 -0500
Subject: [SciPy-User] Roots of a signal
In-Reply-To: <web-130081776@uni-stuttgart.de>
References: <web-130081776@uni-stuttgart.de>
Message-ID: <1cd32cbb1002081310y24675fc0i7848a634abd64da7@mail.gmail.com>

On Mon, Feb 8, 2010 at 3:14 PM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> Hi all,
>
> How can I compute the roots of a signal ?
>
> The following didn't work due to a memory error.
>
> from scipy.interpolate import splrep, splev, sproot
> from numpy import exp, sin, linspace, pi
> from pylab import plot, show, legend
> t = linspace(0,4,20)
> y = exp(-0.1*t)*sin(2*pi*t)
> plot(t,y,label='Signal')
> tck = splrep(t,y)
> t_new = linspace(0,10,40)
> y2 = splev(t_new,tck)
> plot(t_new,y2,label='Spline')
> roots = sproot(tck,10)
> legend(loc=0)
> show()
>
>
> Any pointer would be appreciated.

your script finishes without problem for me

>>> roots
array([  1.32135351e-18,   5.01642799e-01,   9.99383967e-01,
         1.50051340e+00,   2.00014353e+00,   2.49953905e+00,
         3.00076843e+00,   3.49870336e+00])

from the plot (of only the first half) and

>>> np.diff(y2>0).sum()
9

it looks like there is one root missing

Josef


>
> Thanks in advance
>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Nils
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From nancy.leelee at yahoo.com  Mon Feb  8 16:12:08 2010
From: nancy.leelee at yahoo.com (Nancy Lee)
Date: Mon, 8 Feb 2010 13:12:08 -0800 (PST)
Subject: [SciPy-User] scikits.timeseries 'Combine TimeSeries objects'
In-Reply-To: <309FC0FB-1CF2-4D41-B6B6-4D7190FE787B@gmail.com>
References: <180695.59792.qm@web111905.mail.gq1.yahoo.com>
	<309FC0FB-1CF2-4D41-B6B6-4D7190FE787B@gmail.com>
Message-ID: <417999.95665.qm@web111911.mail.gq1.yahoo.com>

Thanks, Pierre, it helps a lot.
Actually, I am working on second or micro-second data, so timeseries for one date might contain big data-set.
I noticed that the minimum frequency for scikites.timeseries package?is per second. Is it possible to?convert it to per micro second??Is there any modified package?available for?the micro seconds timeseries?
Thank you :)

Nancy
?



________________________________
From: Pierre GM <pgmdevlist at gmail.com>
To: SciPy Users List <scipy-user at scipy.org>
Sent: Mon, February 8, 2010 2:15:50 PM
Subject: Re: [SciPy-User] scikits.timeseries 'Combine TimeSeries objects'

On Feb 8, 2010, at 3:09 PM, Nancy Lee wrote:
> 
>? 
> Hi everyone,
>? 
> I have one question about the scikits.timeseries TimeSeries objects.
> Is there any function that can combine multiple timeseries objects into one in the correctchronological order?
> For example,? I have A timeseries object with '2009-02-03' data, and B timeseries object with '2009-02-04' data, and C with '2009-02-05' data, how can I get one timeseries from 2009-02-03 to 2009-02-05 ?
> Any suggestion? thanks a lot!

Only one date per TimeSeries ? Mmh, that's not really how it's supposed to work, but hey...
Try ts.concatenate
>>>? A=ts.time_series([1],start_date="2010-01",freq="M")
>>> B=ts.time_series([2],start_date="2009-12",freq="M")
>>> C=ts.time_series([3],start_date="2010-02",freq="M")
>>> ts.concatenate((A,B,C))
timeseries([2 1 3],
? dates = [Dec-2009 ... Feb-2010],
? freq? = M)
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100208/40b20bfd/attachment.html>

From nwagner at iam.uni-stuttgart.de  Mon Feb  8 16:17:22 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 08 Feb 2010 22:17:22 +0100
Subject: [SciPy-User] Roots of a signal
In-Reply-To: <1cd32cbb1002081310y24675fc0i7848a634abd64da7@mail.gmail.com>
References: <web-130081776@uni-stuttgart.de>
	<1cd32cbb1002081310y24675fc0i7848a634abd64da7@mail.gmail.com>
Message-ID: <web-130082991@uni-stuttgart.de>

Strange.

python -i test_spline.py
/home/nwagner/local/lib64/python2.6/site-packages/scikits/__init__.py:1: 
UserWarning: Module pkg_resources was already imported 
from 
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/pkg_resources.py, 
but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
   __import__('pkg_resources').declare_namespace(__name__)
/home/nwagner/local/lib64/python2.6/site-packages/scikits/__init__.py:1: 
UserWarning: Module site was already imported from 
/usr/lib64/python2.6/site.pyc, but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
   __import__('pkg_resources').declare_namespace(__name__)
/home/nwagner/local/lib64/python2.6/site-packages/matplotlib/backends/backend_gtk.py:621: 
DeprecationWarning: Use the new widget gtk.Tooltip
   self.tooltips = gtk.Tooltips()
Traceback (most recent call last):
   File "test_spline.py", line 11, in <module>
     roots = sproot(tck,10)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/interpolate/fitpack.py", 
line 576, in sproot
     z,ier=_fitpack._sproot(t,c,k,mest)
MemoryError

>>> scipy.__version__
'0.8.0.dev6212'
>>> numpy.__version__
'1.5.0.dev8087'
>>> show_config()                                                                                                  
atlas_threads_info:
     libraries = ['lapack', 'ptf77blas', 'ptcblas', 
'atlas']
     library_dirs = 
['/home/nwagner/src/ATLAS3.8.2/mybuild/lib']
     language = f77
     include_dirs = 
['/home/nwagner/src/ATLAS3.8.2/include']
blas_opt_info:
     libraries = ['ptf77blas', 'ptcblas', 'atlas']
     library_dirs = 
['/home/nwagner/src/ATLAS3.8.2/mybuild/lib']
     define_macros = [('ATLAS_INFO', '"\\"3.8.2\\""')]
     language = c
     include_dirs = 
['/home/nwagner/src/ATLAS3.8.2/include']
atlas_blas_threads_info:
     libraries = ['ptf77blas', 'ptcblas', 'atlas']
     library_dirs = 
['/home/nwagner/src/ATLAS3.8.2/mybuild/lib']
     language = c
     include_dirs = 
['/home/nwagner/src/ATLAS3.8.2/include']
lapack_opt_info:
     libraries = ['lapack', 'ptf77blas', 'ptcblas', 
'atlas']
     library_dirs = 
['/home/nwagner/src/ATLAS3.8.2/mybuild/lib']
     define_macros = [('ATLAS_INFO', '"\\"3.8.2\\""')]
     language = f77
     include_dirs = 
['/home/nwagner/src/ATLAS3.8.2/include']
lapack_mkl_info:
   NOT AVAILABLE
blas_mkl_info:
   NOT AVAILABLE
mkl_info:
   NOT AVAILABLE


cat /proc/meminfo
MemTotal:        1793140 kB

> 
> your script finishes without problem for me
> 
>>>> roots
> array([  1.32135351e-18,   5.01642799e-01, 
>  9.99383967e-01,
>         1.50051340e+00,   2.00014353e+00, 
>  2.49953905e+00,
>         3.00076843e+00,   3.49870336e+00])
> 
> from the plot (of only the first half) and
> 
>>>> np.diff(y2>0).sum()
> 9
> 
> it looks like there is one root missing
> 
> Josef
> 


From pgmdevlist at gmail.com  Mon Feb  8 16:27:39 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 8 Feb 2010 16:27:39 -0500
Subject: [SciPy-User] scikits.timeseries 'Combine TimeSeries objects'
In-Reply-To: <417999.95665.qm@web111911.mail.gq1.yahoo.com>
References: <180695.59792.qm@web111905.mail.gq1.yahoo.com>
	<309FC0FB-1CF2-4D41-B6B6-4D7190FE787B@gmail.com>
	<417999.95665.qm@web111911.mail.gq1.yahoo.com>
Message-ID: <2CF09586-CA69-4093-8429-598B4947FA68@gmail.com>

On Feb 8, 2010, at 4:12 PM, Nancy Lee wrote:
> 
> Thanks, Pierre, it helps a lot.
> Actually, I am working on second or micro-second data, so timeseries for one date might contain big data-set.

I figured that would be the case ;)

> I noticed that the minimum frequency for scikites.timeseries package is per second. Is it possible to convert it to per micro second? Is there any modified package available for the micro seconds timeseries?

No, not yet. It's on the to-do list, but we haven't come to it yet. It's not especially complicated (we'd just have to write some conversion functions, mostly), just time consuming. 
An alternative would be to use the upcoming "datetime74" dtype in numpy, that will allow far more flexibility in terms of frequency definition. Unfortunately, it's still very experimental, and I haven't started working on mergng it w/ our DateArray class.
So, sorry but you're unfortunately out of luck for the moment.
In any case, let us know how it goes.



From josef.pktd at gmail.com  Mon Feb  8 16:32:26 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 8 Feb 2010 16:32:26 -0500
Subject: [SciPy-User] scikits.timeseries 'Combine TimeSeries objects'
In-Reply-To: <2CF09586-CA69-4093-8429-598B4947FA68@gmail.com>
References: <180695.59792.qm@web111905.mail.gq1.yahoo.com>
	<309FC0FB-1CF2-4D41-B6B6-4D7190FE787B@gmail.com>
	<417999.95665.qm@web111911.mail.gq1.yahoo.com>
	<2CF09586-CA69-4093-8429-598B4947FA68@gmail.com>
Message-ID: <1cd32cbb1002081332p8f1796bt929a4d632954d792@mail.gmail.com>

On Mon, Feb 8, 2010 at 4:27 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> On Feb 8, 2010, at 4:12 PM, Nancy Lee wrote:
>>
>> Thanks, Pierre, it helps a lot.
>> Actually, I am working on second or micro-second data, so timeseries for one date might contain big data-set.
>
> I figured that would be the case ;)
>
>> I noticed that the minimum frequency for scikites.timeseries package is per second. Is it possible to convert it to per micro second? Is there any modified package available for the micro seconds timeseries?
>
> No, not yet. It's on the to-do list, but we haven't come to it yet. It's not especially complicated (we'd just have to write some conversion functions, mostly), just time consuming.
> An alternative would be to use the upcoming "datetime74" dtype in numpy, that will allow far more flexibility in terms of frequency definition. Unfortunately, it's still very experimental, and I haven't started working on mergng it w/ our DateArray class.
> So, sorry but you're unfortunately out of luck for the moment.
> In any case, let us know how it goes.


http://nipy.sourceforge.net/nitime/

is also working on a time series object that is designed for
experiment time (with very high frequency) instead of calender time
and might help in this case.

Josef
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From jsalvati at u.washington.edu  Tue Feb  9 00:12:42 2010
From: jsalvati at u.washington.edu (John Salvatier)
Date: Mon, 8 Feb 2010 21:12:42 -0800
Subject: [SciPy-User] What optimization function should I use?
Message-ID: <113e17f21002082112u7893cd40k6781117473c62932@mail.gmail.com>

Hello,

I am writing an adaptive MCMC sampler for PyMC and I have found that
providing a good starting point improves its performance dramatically, so I
have used scipy.optimize.fmin_cfg to find a mode of the posterior
distribution to use as the starting point. However, I don't know much about
different optimization routines, so I don't know if this is a good good
choice or not.

Does anyone have advice on which of the optimization routines I should use?
I don't care too much about performance because I only have to run the
optimization routine a few times at the start of sampling. I should also
mention that I have access to the analytical gradient. My algorithm is
intended to be for general use, so I can't say too much about the functions
on which it will be used, other than they will likely be approximately
normal (so the function will be approximately multi-quadradic) near the
mode. I don't care if it does not work well for multiple optima.

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100208/0cc2728d/attachment.html>

From cournape at gmail.com  Tue Feb  9 04:24:02 2010
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 9 Feb 2010 18:24:02 +0900
Subject: [SciPy-User] [SciPy-user] Float32 FFT
In-Reply-To: <4B703277.50305@tuebingen.mpg.de>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>
	<4B68E27F.6000300@silveregg.co.jp> <27499232.post@talk.nabble.com>
	<5b8d13221002080719m39f4f8dcq4d442f285e4af823@mail.gmail.com>
	<4B703277.50305@tuebingen.mpg.de>
Message-ID: <5b8d13221002090124o71005736q6f18ea5fb9e6b45e@mail.gmail.com>

On Tue, Feb 9, 2010 at 12:49 AM, Klaus Kopec
<klaus.kopec at tuebingen.mpg.de> wrote:
> On 02/08/2010 04:19 PM, David Cournapeau wrote:
>> The double precision support for fftn was a bug, this is now fixed.
> Thank you for fixing the bug so quickly.
>
> The only remaining problem with fftn now is ifftn, which is apparently
> not yet implemented for complex64 input (according to the
> NotYetImplemented error and the fftpack.basic source code for ifftn).

Yep, it is a bug. Should be fixed in the trunk. It is not super
tested, though, and may be slightly inefficient (I have been lazy in
the reuse of input buffers),

cheers,

David


From klaus.kopec at tuebingen.mpg.de  Tue Feb  9 09:43:25 2010
From: klaus.kopec at tuebingen.mpg.de (Klaus Kopec)
Date: Tue, 09 Feb 2010 15:43:25 +0100
Subject: [SciPy-User] [SciPy-user] Float32 FFT
In-Reply-To: <5b8d13221002090124o71005736q6f18ea5fb9e6b45e@mail.gmail.com>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>	<4B68E27F.6000300@silveregg.co.jp>
	<27499232.post@talk.nabble.com>	<5b8d13221002080719m39f4f8dcq4d442f285e4af823@mail.gmail.com>	<4B703277.50305@tuebingen.mpg.de>
	<5b8d13221002090124o71005736q6f18ea5fb9e6b45e@mail.gmail.com>
Message-ID: <4B71748D.1020903@tuebingen.mpg.de>


> Yep, it is a bug. Should be fixed in the trunk.
Sorry for bothering you again, but I think something with your checkin 
might have gone wrong.
I checked out the trunk a couple of minutes ago and 
scipy.fftpack.basic.ifftn still has the old code:

elif istype(tmp, numpy.complex64):
     raise NotImplementedError

Thanks for solving all my problems with this module so quickly!
Klaus


From jgomezdans at gmail.com  Tue Feb  9 11:12:55 2010
From: jgomezdans at gmail.com (Jose Gomez-Dans)
Date: Tue, 9 Feb 2010 16:12:55 +0000
Subject: [SciPy-User] Efficiently applying a function over a large array
Message-ID: <91d218431002090812p574fbb6chf52fa3cc2a8c56c9@mail.gmail.com>

Hi!
I have a function that does some fairly involved linear algebra using some
vectors that are defined over 2D arrays. You can think of it as the 3rd
dimension of a 3D array. For each "pixel" in the 2D array, I want to appy my
function to [i,j,:]. Now, that's all very easy to do in theory using a loop,
eg
for i in ny:
  for j in nx:
     Out[i,j] = MyFunc ( arr1[i,j,:], arr2[i,j,:], arr3[i,j,:] )

but the array size is quite large (>1000x1000 elements), so I would like to
know what the most efficient way of doing this would be. I came accross
numpy.apply_along_axis, but also about the fact that is probably quite slow
for these large arrays?

Thanks!
Jose
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100209/ce9399b5/attachment.html>

From dwf at cs.toronto.edu  Tue Feb  9 11:42:17 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 9 Feb 2010 11:42:17 -0500
Subject: [SciPy-User] Efficiently applying a function over a large array
In-Reply-To: <91d218431002090812p574fbb6chf52fa3cc2a8c56c9@mail.gmail.com>
References: <91d218431002090812p574fbb6chf52fa3cc2a8c56c9@mail.gmail.com>
Message-ID: <0288FFAD-D1BF-4FD7-BF84-0F0831D24E3F@cs.toronto.edu>


On 9-Feb-10, at 11:12 AM, Jose Gomez-Dans wrote:

> for i in ny:
>   for j in nx:
>      Out[i,j] = MyFunc ( arr1[i,j,:], arr2[i,j,:], arr3[i,j,:] )
>
> but the array size is quite large (>1000x1000 elements), so I would  
> like to know what the most efficient way of doing this would be.

You'll have to provide more details about what MyFunc is doing, as how  
efficient you can make it will depend critically on this.

David


From jgomezdans at gmail.com  Tue Feb  9 11:54:40 2010
From: jgomezdans at gmail.com (Jose Gomez-Dans)
Date: Tue, 9 Feb 2010 16:54:40 +0000
Subject: [SciPy-User] Efficiently applying a function over a large array
In-Reply-To: <0288FFAD-D1BF-4FD7-BF84-0F0831D24E3F@cs.toronto.edu>
References: <91d218431002090812p574fbb6chf52fa3cc2a8c56c9@mail.gmail.com>
	<0288FFAD-D1BF-4FD7-BF84-0F0831D24E3F@cs.toronto.edu>
Message-ID: <91d218431002090854l6c94b7a4l4fc42844fe35b325@mail.gmail.com>

David, thanks for your reply

On 9 February 2010 16:42, David Warde-Farley <dwf at cs.toronto.edu> wrote:

>
> On 9-Feb-10, at 11:12 AM, Jose Gomez-Dans wrote:
>
> > for i in ny:
> >   for j in nx:
> >      Out[i,j] = MyFunc ( arr1[i,j,:], arr2[i,j,:], arr3[i,j,:] )
> >
> > but the array size is quite large (>1000x1000 elements), so I would
> > like to know what the most efficient way of doing this would be.
>
> You'll have to provide more details about what MyFunc is doing, as how
> efficient you can make it will depend critically on this.
>

It is doing a fair bit of work, but in essence, from the three vectors it
gets, it generates a set of matrices, and performs a number of matrix
operations (sums and products), and a matrix inversion too. What you get out
of the function is a scalar.

Thinking about it, I thought about list comprehensions:

arr1=numpy.ones(25).reshape(5,5)
arr2=numpy.arange(25).reshape(5,5)
x=[[ func(arr1[i0,i1], arr2[i0,i1]) for i1 in xrange(5)] for i0 in
xrange(5)]

This works, but I don't know how well it's going to apply to my case. I
understand that my function might take some time to evaluate, and that's
fine, but I was worried of the looping taking most of the time.

Thanks!
J
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100209/ee440064/attachment.html>

From denis-bz-gg at t-online.de  Tue Feb  9 11:46:48 2010
From: denis-bz-gg at t-online.de (denis)
Date: Tue, 9 Feb 2010 08:46:48 -0800 (PST)
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <E6517CF3-C57C-436D-BAF4-E5C9D0BF61AF@gmail.com>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu> 
	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com> 
	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
	<76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com> 
	<4B6CA2F3.10008@student.matnat.uio.no>
	<E6517CF3-C57C-436D-BAF4-E5C9D0BF61AF@gmail.com>
Message-ID: <2ca69d04-0b1f-4f54-87a6-eb953bc2b57c@b7g2000yqd.googlegroups.com>

Tony, Dag Sverre,

  let me try to give a bit of notation and background
(as far as I know -- experts please correct me):

    - interpolating spline: goes through the data points
    - smoothing spline: may not.

Say we have points or knots p0 p1 p2 ... labelled by integers,
and want piecewise cubic p(t) for real t.
NB the points can be in 1d 2d kd, and can be spaced any old way --
they can even overlap, p4 == p5.  Common are (j, yj) and (xj, yj).

Local splines are those for which p(t), 0 <= t <= 1,
depends only on the 4 nearest points p_1 p0 p1 p2.
If p(0) == p0 and p(1) == p1 we're interpolating, if not then not.
Local splines are very simple, and the only kind I use.
Global / least-squares-weighted splines
which depend or more points are common;
they're smoother (C2) but can surprise you  (see extrapolation
plot ...)
scipy.interpolate wraps fitpack which does least-squares weighting,
don't know how.

Of local splines, two kinds are common:
    - interpolating: Catmull-Rom
    - smoothing: B-spline.
These are defined by 4x4 "blending matrices", see the link below.
(By the way one can mix: (C-R matrix + Bspline matrix) / 2
gives curves between the two.)
For derivatives, any p(t) is cubic on intervals j <= t <= j+1
so one can just take dp(t)/dt (p_1 p0 p1 p2).

Given the many different kinds of splines,
the only way to find out what a given package does
(if its doc isn't clear)
is to plot its impulse response to ... 0 0 1 0 0 ...

Numpy code for local C-R and B-splines can be found at
http://advice.mechanicalkern.com/question/22/basic-spline-interpolation-in-a-few-lines-of-numpy
C-R interpolates, Bspline smooths; no derivatives.
Comments would be welcome.
After setup, the p(t) are fast _dotblas.dot s, the higher "times" the
faster.


So after all that, a menu for splines:
    - interpolating / smoothing
    - local / various global
    - work on Nxk reals / work on anything numpy
    - 1d as above, 2d image processing
    - derivatives
    - Bezier curves with control points

And we have rather sad choices on doc.

Before jumping into "speed up ..." I'd like to hear from a few more
numpy/scipy users --
what do users want, do any want doc-first ?

Re timing, has anyone timed np.dot (_dotblas) vs cython vs straight C
say for n in (3, 10, 100, 1000) ?

cheers
  -- denis


"Whenever something is a matter of taste, discussions can drag on
forever."


From zachary.pincus at yale.edu  Tue Feb  9 12:05:12 2010
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Tue, 9 Feb 2010 12:05:12 -0500
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <2ca69d04-0b1f-4f54-87a6-eb953bc2b57c@b7g2000yqd.googlegroups.com>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu>
	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com>
	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
	<76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com>
	<4B6CA2F3.10008@student.matnat.uio.no>
	<E6517CF3-C57C-436D-BAF4-E5C9D0BF61AF@gmail.com>
	<2ca69d04-0b1f-4f54-87a6-eb953bc2b57c@b7g2000yqd.googlegroups.com>
Message-ID: <D977291C-4891-44BA-BC68-9743A9D98309@yale.edu>

> Local splines are very simple, and the only kind I use.
> Global / least-squares-weighted splines
> which depend or more points are common;
> they're smoother (C2) but can surprise you  (see extrapolation
> plot ...)
> scipy.interpolate wraps fitpack which does least-squares weighting,
> don't know how.
>
> Of local splines, two kinds are common:
>    - interpolating: Catmull-Rom
>    - smoothing: B-spline.


I had always thought that the splines produced by fitpack were plain  
(and local) non-uniform B-splines (as opposed to the uniform kind,  
which Denis's really helpful demo code uses). However, the way that  
those B-splines are fit to the given data by fitpack is definitely  
global and can lead to odd artifacts if you're incautious.

Is this correct?

Then is reason that the B-splines from scipy.interpolate.fitpack and  
from scipy.signal are a bit different is that the latter are also  
strictly uniform? (E.g. evenly-spaced knot vector.) Probably this is  
easy to test...

Zach


From dwf at cs.toronto.edu  Tue Feb  9 12:42:36 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 9 Feb 2010 12:42:36 -0500
Subject: [SciPy-User] Efficiently applying a function over a large array
In-Reply-To: <91d218431002090854l6c94b7a4l4fc42844fe35b325@mail.gmail.com>
References: <91d218431002090812p574fbb6chf52fa3cc2a8c56c9@mail.gmail.com>
	<0288FFAD-D1BF-4FD7-BF84-0F0831D24E3F@cs.toronto.edu>
	<91d218431002090854l6c94b7a4l4fc42844fe35b325@mail.gmail.com>
Message-ID: <B8990487-B440-4965-A472-7C9644ED00F3@cs.toronto.edu>

On 9-Feb-10, at 11:54 AM, Jose Gomez-Dans wrote:

> David, thanks for your reply
>
> On 9 February 2010 16:42, David Warde-Farley <dwf at cs.toronto.edu>  
> wrote:
>
> On 9-Feb-10, at 11:12 AM, Jose Gomez-Dans wrote:
>
> > for i in ny:
> >   for j in nx:
> >      Out[i,j] = MyFunc ( arr1[i,j,:], arr2[i,j,:], arr3[i,j,:] )
> >
> > but the array size is quite large (>1000x1000 elements), so I would
> > like to know what the most efficient way of doing this would be.
>
> You'll have to provide more details about what MyFunc is doing, as how
> efficient you can make it will depend critically on this.
>
> It is doing a fair bit of work, but in essence, from the three  
> vectors it gets, it generates a set of matrices, and performs a  
> number of matrix operations (sums and products), and a matrix  
> inversion too. What you get out of the function is a scalar.

The reason I ask is that depending on the specific operations it may  
be possible to write them as array-level operations on the entirety of  
arr1, arr2, arr3 without need for a loop/list comprehension/etc.

If you can't, you're stuck with a loop, but Cython would be an option  
for speeding it up if it turns out to be a bottleneck.

David


From tsyu80 at gmail.com  Tue Feb  9 12:47:18 2010
From: tsyu80 at gmail.com (Tony S Yu)
Date: Tue, 9 Feb 2010 12:47:18 -0500
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <2ca69d04-0b1f-4f54-87a6-eb953bc2b57c@b7g2000yqd.googlegroups.com>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu>
	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com>
	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
	<76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com>
	<4B6CA2F3.10008@student.matnat.uio.no>
	<E6517CF3-C57C-436D-BAF4-E5C9D0BF61AF@gmail.com>
	<2ca69d04-0b1f-4f54-87a6-eb953bc2b57c@b7g2000yqd.googlegroups.com>
Message-ID: <F1099DFF-8A2C-4B7B-991A-448E6940A79E@gmail.com>


On Feb 9, 2010, at 11:46 AM, denis wrote:

> Tony, Dag Sverre,
> 
>  let me try to give a bit of notation and background
> (as far as I know -- experts please correct me):
> 
>    - interpolating spline: goes through the data points
>    - smoothing spline: may not.

<snip>

> So after all that, a menu for splines:
>    - interpolating / smoothing
>    - local / various global
>    - work on Nxk reals / work on anything numpy
>    - 1d as above, 2d image processing
>    - derivatives
>    - Bezier curves with control points

Thanks Denis! Reading through this explanation has helped a lot (although I'll need to spend more time understanding everything in it). 

I'm still a bit confused on what seems to be subclasses of smoothing splines: 1) those with points or knots at the x-values (but not necessarily the y-values) of original data points and 2) those with fewer points than the original data (which is what scipy.interpolate provides). Is there a term for these subclasses? I'm having a difficult time distinguishing the two in the literature.

Thanks,
-Tony

From mdekauwe at gmail.com  Tue Feb  9 13:06:32 2010
From: mdekauwe at gmail.com (mdekauwe)
Date: Tue, 9 Feb 2010 10:06:32 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Compiling scipy on Solaris
Message-ID: <27519446.post@talk.nabble.com>


Hi,

I am trying to install the latest scipy libraries and whilst everything (i
think) builds without issues I have encountered some relocation issues
finding libraries.

A number of the basic functionalities work find e.g.

import scipy

a = scipy.arange(10)

However 

if I tried to import something from the optimise library, for example 

from scipy.optimize import leastsq

ImportError: ld.so.1: python: fatal: relocation error: file
/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so: symbol etime_:
referenced symbol not found

Now I am not sure to be honest what I am not doing. If it helps...
ldd -d /users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so
       
        libfsu.so.1 =>   /nerc/packages/studio/12.0/SUNWspro/lib/libfsu.so.1
        libsunmath.so.1 =>      
/nerc/packages/studio/12.0/SUNWspro/lib/libsunmath.so.1
        libf77compat.so.1 =>    
/nerc/packages/studio/12.0/SUNWspro/lib/libf77compat.so.1
        libmtsk.so.1 =>  /usr/lib/libmtsk.so.1
        libc.so.1 =>     /usr/lib/libc.so.1
        libm.so.1 =>     /usr/lib/libm.so.1
        libdl.so.1 =>    /usr/lib/libdl.so.1
        libpthread.so.1 =>       /usr/lib/libpthread.so.1
        /usr/platform/SUNW,A70/lib/libc_psr.so.1
        symbol not found: PyCObject_Type               
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyExc_ImportError            
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyExc_RuntimeError           
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyString_Type        
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: _Py_NoneStruct               
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyInt_Type           
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyType_Type          
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyComplex_Type               
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyFloat_Type         
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyExc_MemoryError            
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyExc_ValueError             
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyExc_AttributeError         
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
        symbol not found: PyExc_TypeError              
(/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)


any suggestions would be most appreciated. Many thanks.

Martin


-- 
View this message in context: http://old.nabble.com/Compiling-scipy-on-Solaris-tp27519446p27519446.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From warren.weckesser at enthought.com  Tue Feb  9 13:17:33 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Tue, 09 Feb 2010 12:17:33 -0600
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>	<c5b438121001281239y6b31691dq878e5ef340df96fd@mail.gmail.com>	<4B623ECD.4040403@enthought.com>	<1cd32cbb1001281757x14b7b581xba02f4ae21d3acfb@mail.gmail.com>	<4B6256F5.3050802@enthought.com>	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>	<c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>
	<1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>
Message-ID: <4B71A6BD.9090002@enthought.com>

josef.pktd at gmail.com wrote:
> On Wed, Feb 3, 2010 at 9:00 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>   
>> FYI, I am quite happy with passing in an hmax value.  I basically
>> copied and pasted lsim2 from signal.ltisys and adapted it just a
>> little to make it a method of my derived class.  Then I added the hmas
>> kwarg that gets passed to odeint.
>>
>> Is there any reason not to allow the user to pass in a kwargs to lsim2
>> that gets passed to odeint?
>>     
>
> I don't see a reason why we cannot add a **kwargs, it should be
> completely backwards compatible.
> Can you file a ticket and add your adjusted version or a patch? And
> even better, add your original example as a test case?
>
>   

Josef,

I just created ticket #1112 for this.  Unless Ryan wants to adapt his 
change to lsim2, I can make a patch this week to implement the enhancement.

Warren


> Josef
>
>
>   
>> On Fri, Jan 29, 2010 at 6:44 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>     
>>> Thanks to Warren and Josef for their time and thoughts.  I feel like I
>>> now understand the underlying problem and have some good options to
>>> solve my short term issues (I assigned the project last night and they
>>> need to be able to start working on it immediately).  I actually use a
>>> TransferFunction class that derives from ltisys.  I could override its
>>> lsim2 method to try out some of these solutions quickly and fairly
>>> easily.
>>>
>>> Ryan
>>>
>>> On Thu, Jan 28, 2010 at 10:00 PM,  <josef.pktd at gmail.com> wrote:
>>>       
>>>> On Thu, Jan 28, 2010 at 10:33 PM, Warren Weckesser
>>>> <warren.weckesser at enthought.com> wrote:
>>>>         
>>>>> josef.pktd at gmail.com wrote:
>>>>>           
>>>>>> On Thu, Jan 28, 2010 at 8:50 PM, Warren Weckesser
>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>
>>>>>>             
>>>>>>> Ryan,
>>>>>>>
>>>>>>> The problem is that the ODE solver used by lsim2 is too good. :)
>>>>>>>
>>>>>>> It uses scipy.integrate.odeint, which in turn uses the Fortran library
>>>>>>> LSODA.  Like any good solver, LSODA is an adaptive solver--it adjusts its
>>>>>>> step size to be as large as possible while keeping estimates of the error
>>>>>>> bounded.  For the problem you are solving, with initial condition 0, the
>>>>>>> exact solution is initially exactly 0.  This is such a nice smooth solution
>>>>>>> that the solver's step size quickly grows--so big, in fact, that it skips
>>>>>>> right over your pulse and never sees it.
>>>>>>>
>>>>>>> So how does it create all those intermediate points at the requested time
>>>>>>> values?  It uses interpolation between the steps that it computed to create
>>>>>>> the solution values at the times that you requested.  So using a finer grid
>>>>>>> of time values won't help.  (If lsim2 gave you a hook into the parameters
>>>>>>> passed to odeint, you could set odeint's 'hmax' to a value smaller than your
>>>>>>> pulse width, which would force the solver to see the pulse.  But there is no
>>>>>>> way to set that parameter from lsim2.)
>>>>>>>
>>>>>>>               
>>>>>> It's something what I suspected. I don't know much about odeint, but
>>>>>> do you think it would be useful to let lsim2 pass through some
>>>>>> parameters to odeint?
>>>>>>
>>>>>>
>>>>>>             
>>>>> Sounds useful to me.  A simple implementation is an optional keyword
>>>>> argument that is a dict of odeint arguments.   But this would almost
>>>>> certainly break if lsim2 were ever reimplemented with a different
>>>>> solver.  So perhaps it should allow a common set of ODE solver
>>>>> parameters (e.g. absolute and relative error tolerances, max and min
>>>>> step sizes, others?).
>>>>>
>>>>> Perhaps this should wait until after the ODE solver redesign that is
>>>>> occasionally discussed:
>>>>>    http://projects.scipy.org/scipy/wiki/OdeintRedesign
>>>>> Then the solver itself could be an optional argument to lsim2.
>>>>>           
>>>> I was just thinking of adding to the argument list a **kwds argument
>>>> that is directly passed on to whatever ODE solver is used. This should
>>>> be pretty flexible for any changes and be backwards compatible.
>>>>
>>>> I've seen and used it in a similar way for calls to optimization
>>>> routines, e.g. also optimize.curve_fit, does it. What are actually
>>>> valid keywords would depend on which function is called.
>>>>
>>>> (But I'm not a user of lsim, I'm just stealing some ideas from lti and
>>>> friends for time series analysis.)
>>>>
>>>> Josef
>>>>
>>>>
>>>>
>>>>
>>>>         
>>>>> Warren
>>>>>
>>>>>           
>>>>>> Josef
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> The basic problem is you are passing in a discontinuous function to a solver
>>>>>>> that expects a smooth function.  A better way to solve this problem is to
>>>>>>> explicitly account for the discontinuity. One possibility is the attached
>>>>>>> script.
>>>>>>>
>>>>>>> This is an excellent "learning opportunity" for your students on the hazards
>>>>>>> of numerical computing!
>>>>>>>
>>>>>>> Warren
>>>>>>>
>>>>>>>
>>>>>>> Ryan Krauss wrote:
>>>>>>>
>>>>>>>               
>>>>>>>> I believe I have discovered a bug in signal.lsim2.  I believe the
>>>>>>>> short attached script illustrates the problem.  I was trying to
>>>>>>>> predict the response of a transfer function with a pure integrator:
>>>>>>>>
>>>>>>>>             g
>>>>>>>> G = -------------
>>>>>>>>         s(s+p)
>>>>>>>>
>>>>>>>> to a finite width pulse.  lsim2 seems to handle the step response just
>>>>>>>> fine, but says that the pulse response is exactly 0.0 for the entire
>>>>>>>> time of the simulation.  Obviously, this isn't the right answer.
>>>>>>>>
>>>>>>>> I am running scipy 0.7.0 and numpy 1.2.1 on Ubuntu 9.04, but I also
>>>>>>>> have the same problem on Windows running 0.7.1 and 1.4.0.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Ryan
>>>>>>>>  ------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SciPy-User mailing list
>>>>>>>> SciPy-User at scipy.org
>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>
>>>>>>>>                 
>>>>>>> from pylab import *
>>>>>>> from scipy import signal
>>>>>>>
>>>>>>>
>>>>>>> g = 100.0
>>>>>>> p = 15.0
>>>>>>> G = signal.ltisys.lti(g, [1,p,0])
>>>>>>>
>>>>>>> t = arange(0, 1.0, 0.002)
>>>>>>> N = len(t)
>>>>>>>
>>>>>>> # u for the whole interval (not used in lsim2, only for plotting later).
>>>>>>> amp = 50.0
>>>>>>> u = zeros(N)
>>>>>>> k1 = 50
>>>>>>> k2 = 100
>>>>>>> u[k1:k2] = amp
>>>>>>>
>>>>>>> # Create input functions for each smooth interval. (This could be simpler,
>>>>>>> since u
>>>>>>> # is constant on each interval.)
>>>>>>> a = float(k1)/N
>>>>>>> b = float(k2)/N
>>>>>>> T1 = linspace(0, a, 201)
>>>>>>> u1 = zeros_like(T1)
>>>>>>> T2 = linspace(a, b, 201)
>>>>>>> u2 = amp*ones_like(T2)
>>>>>>> T3 = linspace(b, 1.0, 201)
>>>>>>> u3 = zeros_like(T3)
>>>>>>>
>>>>>>> # Solve on each interval; use the final value of one solution as the
>>>>>>> starting
>>>>>>> # point of the next solution.
>>>>>>> # (We could skip the first calculation, since we know the solution will be
>>>>>>> 0.)
>>>>>>> (t1, y1, x1) = signal.lsim2(G,u1,T1)
>>>>>>> (t2, y2, x2) = signal.lsim2(G, u2, T2, X0=x1[-1])
>>>>>>> (t3, y3, x3) = signal.lsim2(G, u3, T3, X0=x2[-1])
>>>>>>>
>>>>>>> figure(1)
>>>>>>> clf()
>>>>>>> plot(t, u, 'k', linewidth=3)
>>>>>>> plot(t1, y1, 'y', linewidth=3)
>>>>>>> plot(t2, y2, 'b', linewidth=3)
>>>>>>> plot(t3, y3, 'g', linewidth=3)
>>>>>>>
>>>>>>> show()
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SciPy-User mailing list
>>>>>>> SciPy-User at scipy.org
>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>>             
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>           
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>         
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>     
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From josef.pktd at gmail.com  Tue Feb  9 15:22:14 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 9 Feb 2010 15:22:14 -0500
Subject: [SciPy-User] miso_filter
Message-ID: <1cd32cbb1002091222p4cae1ca9j2316696a63e2f852@mail.gmail.com>

In case anyone is interested: how to do a multi-input single-output
linear filter in 2 lines of code and 50 lines of tests

    inp = signal.correlate(x, ma[::-1,:])[:, (x.shape[1]+1)//2]
    signal.lfilter([1], ar, inp), inp

(I needed the tests to clear up a misunderstanding between me and convolve.)

details at
http://bazaar.launchpad.net/~josef-pktd/statsmodels/statsmodels-josef-experimental/annotate/head%3A/scikits/statsmodels/sandbox/regression/mle.py#L363

with application to simulating GARCH models (those are not verified yet)

Josef


From robert.pickel at gmail.com  Tue Feb  9 16:25:23 2010
From: robert.pickel at gmail.com (Robert Pickel)
Date: Tue, 9 Feb 2010 16:25:23 -0500
Subject: [SciPy-User] [SciPy-user] Compiling scipy on Solaris
In-Reply-To: <27519446.post@talk.nabble.com>
References: <27519446.post@talk.nabble.com>
Message-ID: <e11a47651002091325h31f34657p7648692408d8db0d@mail.gmail.com>

I'm not too sure on this, but hope it helps....

Looking at the "symbol not found" stuff looks like the python library hasn't
been linked properly.

You may want to make sure that libpython is in the path for your linker, it
might be someplace else on your system.







On Tue, Feb 9, 2010 at 1:06 PM, mdekauwe <mdekauwe at gmail.com> wrote:

>
> Hi,
>
> I am trying to install the latest scipy libraries and whilst everything (i
> think) builds without issues I have encountered some relocation issues
> finding libraries.
>
> A number of the basic functionalities work find e.g.
>
> import scipy
>
> a = scipy.arange(10)
>
> However
>
> if I tried to import something from the optimise library, for example
>
> from scipy.optimize import leastsq
>
> ImportError: ld.so.1: python: fatal: relocation error: file
> /users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so: symbol etime_:
> referenced symbol not found
>
> Now I am not sure to be honest what I am not doing. If it helps...
> ldd -d /users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so
>
>        libfsu.so.1 =>   /nerc/packages/studio/12.0/SUNWspro/lib/libfsu.so.1
>        libsunmath.so.1 =>
> /nerc/packages/studio/12.0/SUNWspro/lib/libsunmath.so.1
>        libf77compat.so.1 =>
> /nerc/packages/studio/12.0/SUNWspro/lib/libf77compat.so.1
>        libmtsk.so.1 =>  /usr/lib/libmtsk.so.1
>        libc.so.1 =>     /usr/lib/libc.so.1
>        libm.so.1 =>     /usr/lib/libm.so.1
>        libdl.so.1 =>    /usr/lib/libdl.so.1
>        libpthread.so.1 =>       /usr/lib/libpthread.so.1
>        /usr/platform/SUNW,A70/lib/libc_psr.so.1
>        symbol not found: PyCObject_Type
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyExc_ImportError
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyExc_RuntimeError
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyString_Type
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: _Py_NoneStruct
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyInt_Type
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyType_Type
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyComplex_Type
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyFloat_Type
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyExc_MemoryError
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyExc_ValueError
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyExc_AttributeError
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>        symbol not found: PyExc_TypeError
> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>
>
> any suggestions would be most appreciated. Many thanks.
>
> Martin
>
>
> --
> View this message in context:
> http://old.nabble.com/Compiling-scipy-on-Solaris-tp27519446p27519446.html
> Sent from the Scipy-User mailing list archive at Nabble.com.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100209/17f7f7bb/attachment.html>

From mdekauwe at gmail.com  Tue Feb  9 16:47:48 2010
From: mdekauwe at gmail.com (mdekauwe)
Date: Tue, 9 Feb 2010 13:47:48 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Compiling scipy on Solaris
In-Reply-To: <e11a47651002091325h31f34657p7648692408d8db0d@mail.gmail.com>
References: <27519446.post@talk.nabble.com>
	<e11a47651002091325h31f34657p7648692408d8db0d@mail.gmail.com>
Message-ID: <27522587.post@talk.nabble.com>


Thanks but libpython is definatley in the path, numpy works fine and as I
said it seems all the scipy functions that don't require an extension, I.e
scipy.optimize stuff it struggles to find for example



Robert Pickel wrote:
> 
> I'm not too sure on this, but hope it helps....
> 
> Looking at the "symbol not found" stuff looks like the python library
> hasn't
> been linked properly.
> 
> You may want to make sure that libpython is in the path for your linker,
> it
> might be someplace else on your system.
> 
> 
> 
> 
> 
> 
> 
> On Tue, Feb 9, 2010 at 1:06 PM, mdekauwe <mdekauwe at gmail.com> wrote:
> 
>>
>> Hi,
>>
>> I am trying to install the latest scipy libraries and whilst everything
>> (i
>> think) builds without issues I have encountered some relocation issues
>> finding libraries.
>>
>> A number of the basic functionalities work find e.g.
>>
>> import scipy
>>
>> a = scipy.arange(10)
>>
>> However
>>
>> if I tried to import something from the optimise library, for example
>>
>> from scipy.optimize import leastsq
>>
>> ImportError: ld.so.1: python: fatal: relocation error: file
>> /users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so: symbol
>> etime_:
>> referenced symbol not found
>>
>> Now I am not sure to be honest what I am not doing. If it helps...
>> ldd -d /users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so
>>
>>        libfsu.so.1 =>  
>> /nerc/packages/studio/12.0/SUNWspro/lib/libfsu.so.1
>>        libsunmath.so.1 =>
>> /nerc/packages/studio/12.0/SUNWspro/lib/libsunmath.so.1
>>        libf77compat.so.1 =>
>> /nerc/packages/studio/12.0/SUNWspro/lib/libf77compat.so.1
>>        libmtsk.so.1 =>  /usr/lib/libmtsk.so.1
>>        libc.so.1 =>     /usr/lib/libc.so.1
>>        libm.so.1 =>     /usr/lib/libm.so.1
>>        libdl.so.1 =>    /usr/lib/libdl.so.1
>>        libpthread.so.1 =>       /usr/lib/libpthread.so.1
>>        /usr/platform/SUNW,A70/lib/libc_psr.so.1
>>        symbol not found: PyCObject_Type
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyExc_ImportError
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyExc_RuntimeError
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyString_Type
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: _Py_NoneStruct
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyInt_Type
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyType_Type
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyComplex_Type
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyFloat_Type
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyExc_MemoryError
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyExc_ValueError
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyExc_AttributeError
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>        symbol not found: PyExc_TypeError
>> (/users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so)
>>
>>
>> any suggestions would be most appreciated. Many thanks.
>>
>> Martin
>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Compiling-scipy-on-Solaris-tp27519446p27519446.html
>> Sent from the Scipy-User mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 

-- 
View this message in context: http://old.nabble.com/Compiling-scipy-on-Solaris-tp27519446p27522587.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From david at silveregg.co.jp  Tue Feb  9 20:22:36 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Wed, 10 Feb 2010 10:22:36 +0900
Subject: [SciPy-User] [SciPy-user] Compiling scipy on Solaris
In-Reply-To: <27519446.post@talk.nabble.com>
References: <27519446.post@talk.nabble.com>
Message-ID: <4B720A5C.8090807@silveregg.co.jp>

mdekauwe wrote:
> Hi,
> 
> I am trying to install the latest scipy libraries and whilst everything (i
> think) builds without issues I have encountered some relocation issues
> finding libraries.
> 
> A number of the basic functionalities work find e.g.
> 
> import scipy
> 
> a = scipy.arange(10)
> 
> However 
> 
> if I tried to import something from the optimise library, for example 
> 
> from scipy.optimize import leastsq
> 
> ImportError: ld.so.1: python: fatal: relocation error: file
> /users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so: symbol etime_:
> referenced symbol not found

Most likely, that's the problem. Etime is a function which depending on 
your toolchain, is an intrinsic or not.

Could you provide us a build log ? Which fortran compiler and which 
LAPACK are you using (it does not look like you are using the sun 
performance library) ?

cheers,

David


From cournape at gmail.com  Tue Feb  9 20:58:00 2010
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 10 Feb 2010 10:58:00 +0900
Subject: [SciPy-User] [SciPy-user] Float32 FFT
In-Reply-To: <4B71748D.1020903@tuebingen.mpg.de>
References: <615785.75542.qm@web33004.mail.mud.yahoo.com>
	<4B68E27F.6000300@silveregg.co.jp> <27499232.post@talk.nabble.com>
	<5b8d13221002080719m39f4f8dcq4d442f285e4af823@mail.gmail.com>
	<4B703277.50305@tuebingen.mpg.de>
	<5b8d13221002090124o71005736q6f18ea5fb9e6b45e@mail.gmail.com>
	<4B71748D.1020903@tuebingen.mpg.de>
Message-ID: <5b8d13221002091758y3b1aed75gaae5caeb03294fc1@mail.gmail.com>

On Tue, Feb 9, 2010 at 11:43 PM, Klaus Kopec
<klaus.kopec at tuebingen.mpg.de> wrote:
>
>> Yep, it is a bug. Should be fixed in the trunk.
> Sorry for bothering you again, but I think something with your checkin
> might have gone wrong.

It is just that I forgot to push the changes from my git repo to the
svn repo. All should be fine now.

There seems to be a serious memory issue for > 2d arrays,  though,
making fftn almost unusable for large arrays. Maybe this falls into
the "A cache with a bad policy is another name for a memory leak"
motto,

cheers,

David


From mdekauwe at gmail.com  Wed Feb 10 05:05:01 2010
From: mdekauwe at gmail.com (mdekauwe)
Date: Wed, 10 Feb 2010 02:05:01 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Compiling scipy on Solaris
In-Reply-To: <4B720A5C.8090807@silveregg.co.jp>
References: <27519446.post@talk.nabble.com> <4B720A5C.8090807@silveregg.co.jp>
Message-ID: <27527116.post@talk.nabble.com>


Below is the output of the build (I have removed some of it as it wouldn't
let me post the message), I hope that was what you meant by log, if it isn't
and there was a specific command then please let me know. Thanks.

I should probably add I have managed to get scipy 0.6.1 to build and run
without issues.

blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /users/eow/mgdk/sun4u/lib
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/users/eow/mgdk/sun4u/lib']
    language = c

customize SunFCompiler
Could not locate executable f90
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize Gnu95FCompiler
Found executable /nerc/packages/gcc/4.2.3/5.9/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
gcc -pthread _configtest.o -L/users/eow/mgdk/sun4u/lib -llapack -lf77blas
-lcblas -latlas -o _configtest
ATLAS version 3.8.3 built by mgdk on Fri Jul 17 19:10:13 BST 2009:
   UNAME    : SunOS wlsn098 5.9 Generic_122300-14 sun4u sparc SUNW,A70
   INSTFLG  : -1 0 -a 1
   ARCHDEFS : -DATL_OS_SunOS -DATL_ARCH_USIII -DATL_CPUMHZ=1600 -DSUN_HR
-DATL_GAS_SPARC
   F2CDEFS  : -DAdd_ -DF77_INTEGER=int -DStringSunStyle
   CACHEEDGE: 786432
   F77      : gfortran, version GNU Fortran (GCC) 4.2.3
   F77FLAGS : -O -mcpu=ultrasparc3 -mtune=ultrasparc3 -fPIC -m32
   SMC      : gcc, version gcc (GCC) 4.2.3
   SMCFLAGS : -mcpu=ultrasparc3 -mtune=ultrasparc3 -O3 -funroll-all-loops
-fPIC -m32
   SKC      : gcc, version gcc (GCC) 4.2.3
   SKCFLAGS : -mcpu=ultrasparc3 -mtune=ultrasparc3 -O3 -funroll-all-loops
-fPIC -m32
success!
removing: _configtest.c _configtest.o _configtest
  FOUND:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/users/eow/mgdk/sun4u/lib']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]

ATLAS version 3.8.3
lapack_opt_info:
lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in /users/eow/mgdk/sun4u/lib
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /users/eow/mgdk/sun4u/lib
numpy.distutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/users/eow/mgdk/sun4u/lib']
    language = f77

customize SunFCompiler
customize GnuFCompiler
customize Gnu95FCompiler
customize Gnu95FCompiler
customize Gnu95FCompiler using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
gcc -pthread _configtest.o -L/users/eow/mgdk/sun4u/lib -llapack -llapack
-lf77blas -lcblas -latlas -o _configtest
ATLAS version 3.8.3 built by mgdk on Fri Jul 17 19:10:13 BST 2009:
   UNAME    : SunOS wlsn098 5.9 Generic_122300-14 sun4u sparc SUNW,A70
   INSTFLG  : -1 0 -a 1
   ARCHDEFS : -DATL_OS_SunOS -DATL_ARCH_USIII -DATL_CPUMHZ=1600 -DSUN_HR
-DATL_GAS_SPARC
   F2CDEFS  : -DAdd_ -DF77_INTEGER=int -DStringSunStyle
   CACHEEDGE: 786432
   F77      : gfortran, version GNU Fortran (GCC) 4.2.3
   F77FLAGS : -O -mcpu=ultrasparc3 -mtune=ultrasparc3 -fPIC -m32
   SMC      : gcc, version gcc (GCC) 4.2.3
   SMCFLAGS : -mcpu=ultrasparc3 -mtune=ultrasparc3 -O3 -funroll-all-loops
-fPIC -m32
   SKC      : gcc, version gcc (GCC) 4.2.3
   SKCFLAGS : -mcpu=ultrasparc3 -mtune=ultrasparc3 -O3 -funroll-all-loops
-fPIC -m32
success!
removing: _configtest.c _configtest.o _configtest
  FOUND:
    libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/users/eow/mgdk/sun4u/lib']
    language = f77
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]

ATLAS version 3.8.3
ATLAS version 3.8.3
umfpack_info:
  libraries umfpack not found in /users/eow/mgdk/sun4u/lib
  NOT AVAILABLE

running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler
options
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler
options
running build_src
building py_modules sources
building library "dfftpack" sources
building library "linpack_lite" sources
building library "mach" sources
building library "quadpack" sources
building library "odepack" sources
building library "fitpack" sources
building library "odrpack" sources
building library "minpack" sources
building library "rootfind" sources
building library "superlu_src" sources
building library "arpack" sources
building library "sc_c_misc" sources
building library "sc_cephes" sources
building library "sc_mach" sources
building library "sc_toms" sources
building library "sc_amos" sources
building library "sc_cdf" sources
building library "sc_specfun" sources
building library "statlib" sources
building extension "scipy.cluster._vq" sources
building extension "scipy.cluster._hierarchy_wrap" sources
building extension "scipy.fftpack._fftpack" sources
f2py options: []
  adding 'build/src.solaris-2.9-sun4u-2.5/fortranobject.c' to sources.
  adding 'build/src.solaris-2.9-sun4u-2.5' to include_dirs.
building extension "scipy.fftpack.convolve" sources
f2py options: []
  adding 'build/src.solaris-2.9-sun4u-2.5/fortranobject.c' to sources.
  adding 'build/src.solaris-2.9-sun4u-2.5' to include_dirs.
building extension "scipy.integrate._quadpack" sources
building extension "scipy.integrate._odepack" sources
building extension "scipy.integrate.vode" sources
f2py options: []
  adding 'build/src.solaris-2.9-sun4u-2.5/fortranobject.c' to sources.
  adding 'build/src.solaris-2.9-sun4u-2.5' to include_dirs.
building extension "scipy.interpolate._fitpack" sources
building extension "scipy.interpolate.dfitpack" sources
f2py options: []
  adding 'build/src.solaris-2.9-sun4u-2.5/fortranobject.c' to sources.
  adding 'build/src.solaris-2.9-sun4u-2.5' to include_dirs.
  adding
'build/src.solaris-2.9-sun4u-2.5/scipy/interpolate/src/dfitpack-f2pywrappers.f'
to sources.
building extension "scipy.interpolate._interpolate" sources
building extension "scipy.io.numpyio" sources
building extension "scipy.lib.blas.fblas" sources
f2py options: ['skip:', ':']
  adding 'build/src.solaris-2.9-sun4u-2.5/fortranobject.c' to sources.
  adding 'build/src.solaris-2.9-sun4u-2.5' to include_dirs.

etc etc


David Cournapeau-3 wrote:
> 
> mdekauwe wrote:
>> Hi,
>> 
>> I am trying to install the latest scipy libraries and whilst everything
>> (i
>> think) builds without issues I have encountered some relocation issues
>> finding libraries.
>> 
>> A number of the basic functionalities work find e.g.
>> 
>> import scipy
>> 
>> a = scipy.arange(10)
>> 
>> However 
>> 
>> if I tried to import something from the optimise library, for example 
>> 
>> from scipy.optimize import leastsq
>> 
>> ImportError: ld.so.1: python: fatal: relocation error: file
>> /users/eow/mgdk/sun4u/lib/python/scipy/optimize/_lbfgsb.so: symbol
>> etime_:
>> referenced symbol not found
> 
> Most likely, that's the problem. Etime is a function which depending on 
> your toolchain, is an intrinsic or not.
> 
> Could you provide us a build log ? Which fortran compiler and which 
> LAPACK are you using (it does not look like you are using the sun 
> performance library) ?
> 
> cheers,
> 
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 

-- 
View this message in context: http://old.nabble.com/Compiling-scipy-on-Solaris-tp27519446p27527116.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From almar.klein at gmail.com  Wed Feb 10 08:29:42 2010
From: almar.klein at gmail.com (Almar Klein)
Date: Wed, 10 Feb 2010 14:29:42 +0100
Subject: [SciPy-User] ANN: visvis v1.1
Message-ID: <cc38d75f1002100529i28f7318an6d614641df01fc5b@mail.gmail.com>

Hi all,

I am pleased to announce version 1.1 of visvis, a Python
visualization library for of 1D to 4D data.

Website: http://code.google.com/p/visvis/
Discussion group: http://groups.google.com/group/visvis/
Documentation: http://code.google.com/p/visvis/wiki/Visvis_basics

=== Description ===

Visvis is a pure Python visualization library that uses OpenGl to
display 1D to 4D data; it can be used from simple plotting tasks to
rendering 3D volumetric data that moves in time.

Visvis can be used in Python scripts, interactive Python sessions (as
with IPython) and can be embedded in applications.

Visvis employs an object oriented structure; each object being
visualized (e.g. a line or a texture) has various properties that can
be modified to change its behavior or appearance. A Matlab-like
interface in the form of a set of functions allows easy creation of
these objects (e.g. plot(), imshow(), volshow()).

=== Changes ===
A complete list can be found at
http://code.google.com/p/visvis/wiki/releaseNotes.

- Colormaps are now supported for texture objects. A colorbar and
colormapEditor wibject are also available.
- The performance of rendering 3D volumes has significantly improved. A new
shader (raycast) is also available).
- Fonts are rendered more smoothly and faster.
- Data for textures is automatically padded to a factor two if necessary,
and down-sampled if too large to fit in OpenGl memory.
- Visvis now works for any OpenGl version from OpenGl 1.1 (some features
might not be supported for lower versions though).

Regards,
  Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/ea943456/attachment.html>

From m.perrin at me.com  Wed Feb 10 10:08:29 2010
From: m.perrin at me.com (Mark P)
Date: Wed, 10 Feb 2010 10:08:29 -0500
Subject: [SciPy-User] odeint (python vs mathematica)
Message-ID: <18B95518-6CA6-4C5F-BB57-3BBECC863C0B@me.com>

Hello all,

(my first post to the list)

I have been performing modelling of ionic currents in Mathematica for the past three years as a student. I have recently passed into the hallowed halls of post-grad, and am keen to be done with Mathematica and use a free and open solution for my projects, hence Python and numpy/scipy. 

I've made some progress with the transition; the cookbooks are especially helpful. There is one in particular I have been looking at - the rabbit/fox model ( Lotka-Volterra Tutorial), that is helping me to understand odeint.

=====
from numpy import *
import pylab as p
# Definition of parameters
a = 1.
b = 0.1
c = 1.5
d = 0.75
def dX_dt(X, t=0):
    """ Return the growth rate of fox and rabbit populations. """
    return array([ a*X[0] -   b*X[0]*X[1] ,
                  -c*X[1] + d*b*X[0]*X[1] ]) 

from scipy import integrate

t = linspace(0, 10000,  10000)              # time
X0 = array([10, 5])                     		# initials conditions: 10 rabbits and 5 foxes
X = integrate.odeint(dX_dt, X0, t)
===

A simple form of this code in Mathematica is:

===
a = 1
b = 0.1
c = 1.5
d = 0.75

eqn1 := r'[t] == a * r[t] - b * r[t] * f[t]
eqn2 := f'[t] == -c * f[t] + d * b * r[t] * f[t]

solution = NDSolve[{eqn1, eqn2, r[0] == 10, f[0] == 5}, {r[t], f[t]}, {t, 0, 10000}, MaxSteps -> Infinity]
===

My question is about speed. When computed in Mathematica 7 the code is roughly about 10x faster than odeint (I've increased t from 15 in the cookbook to 10000 to allow the measurement of speed to be made). But I know from the web that numpy/scipy can be of an equivalent or faster speed. I was wondering whether the solution can be sped up; and would that require the use of C or Fortran.

I see the inherent advantage of using Python with numpy/scipy and I do not expect the speeds of a compiled language but I was hoping for equivalent speeds to other interpreted solutions.

Thank you,

Mark Perrin
Electrophysiology Research Fellow
Ottawa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/0e2124a7/attachment.html>

From schut at sarvision.nl  Wed Feb 10 10:45:41 2010
From: schut at sarvision.nl (Vincent Schut)
Date: Wed, 10 Feb 2010 16:45:41 +0100
Subject: [SciPy-User] moving window (2D) correlation coefficient
Message-ID: <hkukb3$k0h$1@ger.gmane.org>

Hi,

I need to calculate the correlation coefficient of a (simultaneously) 
moving window over 2 images, such that the resulting pixel x,y (center 
of the window) is the corrcoef((a 5x5 window around x,y for image A), (a 
5x5 window around x,y for image B)).
Currently, I just loop over y, x, and then call corrcoef for each x,y 
window. Would there be a better way, other than converting the loop to 
cython?


For clarity (or not), the relevant part from my code:


for y in range(d500shape[2]):
     for x in range(d500shape[3]):
         if valid500[d,y,x]:
             window = spectral500Bordered[d,:,y:y+5, x:x+5].reshape(7, -1)
             for b in range(5):
                 nonzeroMask = (window[0] > 0)
                 b01corr[0,b,y,x] = 
numpy.corrcoef(window[0][nonzeroMask], window[b+2][nonzeroMask])[0,1]
                 b01corr[1,b,y,x] = 
numpy.corrcoef(window[1][nonzeroMask], window[b+2][nonzeroMask])[0,1]


forget the 'if valid500' and 'nonzeroMask', those are to prevent 
calculating pixels that don't need to be calculated, and to feed only 
valid pixels from the window into corrcoef
spectral500Bordered is essentially a [d dates, 7 images, ysize, xsize] 
array. I work per date (d), then calculate the corrcoef for images[0] 
versus images[2:], and for images[1] versus images[2:]

Thanks,
Vincent.



From ryanlists at gmail.com  Wed Feb 10 10:57:08 2010
From: ryanlists at gmail.com (Ryan Krauss)
Date: Wed, 10 Feb 2010 09:57:08 -0600
Subject: [SciPy-User] lsoda vs. Coulomb friction
In-Reply-To: <e06186141002040837l453477f4ga6fd236aad6647f3@mail.gmail.com>
References: <c5b438121002030828q708089e5k43d85bc8b36432e3@mail.gmail.com>
	<e06186141002031216scfaa9d1mb558a18839f1835@mail.gmail.com>
	<45d1ab481002031300x78b5eb03hc74e7bf7cb2c8aae@mail.gmail.com>
	<ce557a361002031705o6259762doe57a4326631709c9@mail.gmail.com>
	<e06186141002031729i2d0c4018j36b65bd6d384fe6a@mail.gmail.com>
	<e06186141002031744vdc8e3d7saf306a68de31aeeb@mail.gmail.com>
	<c5b438121002040633l45eb814v811328000596b3d9@mail.gmail.com>
	<e06186141002040837l453477f4ga6fd236aad6647f3@mail.gmail.com>
Message-ID: <c5b438121002100757x25fe579doc4c104e1775ea422@mail.gmail.com>

FYI, I am moving to a slightly more sophisticated approach, similar to
Anne's third recommendation or the top of page 99 in the book Chuck
referenced.  The system is a DC motor with internal friction.
Originally, I was doing open-loop testing with a pulse input.  Now I
am putting the system under proportional control and it needs to be
possible to change directions.

So, I am going to write a case that tests for the possibility of
either sticking or changing directions.  Since the experimental
closed-loop system calculates an input that remains constant for each
fixed-width time step, I think I can handle this cleanly in a for
loop, looping over the time vector.

On Thu, Feb 4, 2010 at 10:37 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Thu, Feb 4, 2010 at 7:33 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>
>> Thanks for all the excellent and thoughtful responses. ?I kind of
>> expected Warren to yell at me to stop using smooth solvers on
>> discontinuous systems and leave it at that. ?Your responses not only
>> give me somethings to try, but make me feel like my question really
>> was a good one.
>>
>> For now, I am basically following Anne's first suggestion:
>>
>> * Declare that when the discontinuity becomes important, the Coulomb
>> friction model becomes too crude an approximation, and stop.
>> (Obviously if the results agree with experiment this is unnecessary.)
>>
>
> That is sort of like the "switch" method in the reference I linked. That
> looks like the easiest way to go for simple systems.
>
> <snip>
>
> Chuck
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From fredmfp at gmail.com  Wed Feb 10 10:58:01 2010
From: fredmfp at gmail.com (fred)
Date: Wed, 10 Feb 2010 16:58:01 +0100
Subject: [SciPy-User] interpolation grid/scatter
Message-ID: <4B72D789.8090209@gmail.com>

Hi all,

Say I have data grid like this:

xmin, xmax, dx = 0, 5, 1
ymin, ymax, dy = 0, 4, 1

data.shape : (6,5)

I would like to get scalar value at points that are _not_ on the grid
nodes, say at x=1.213, y=3.24.

How could I do that?

Is there a scipy function to do what I want?

PS : I have about 100 points on which I want the scalar value

PS2 : my array can be 3D array.

TIA.

Cheers,

-- 
Fred



From jkington at wisc.edu  Wed Feb 10 11:03:50 2010
From: jkington at wisc.edu (Joe Kington)
Date: Wed, 10 Feb 2010 10:03:50 -0600
Subject: [SciPy-User] interpolation grid/scatter
In-Reply-To: <4B72D789.8090209@gmail.com>
References: <4B72D789.8090209@gmail.com>
Message-ID: <f6951adf1002100803w25d210d9rc5650c215e7c2926@mail.gmail.com>

If I'm understanding you correctly,
scipy.ndimage.interpolation.map_coordinates does exactly what you want.

 It takes a list of arbirtary (i.e. not integer) indexes and interpolates a
grid at those coordinates.

Hope that helps,
-Joe

On Wed, Feb 10, 2010 at 9:58 AM, fred <fredmfp at gmail.com> wrote:

> Hi all,
>
> Say I have data grid like this:
>
> xmin, xmax, dx = 0, 5, 1
> ymin, ymax, dy = 0, 4, 1
>
> data.shape : (6,5)
>
> I would like to get scalar value at points that are _not_ on the grid
> nodes, say at x=1.213, y=3.24.
>
> How could I do that?
>
> Is there a scipy function to do what I want?
>
> PS : I have about 100 points on which I want the scalar value
>
> PS2 : my array can be 3D array.
>
> TIA.
>
> Cheers,
>
> --
> Fred
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/0d5a0f5c/attachment.html>

From nasser at udel.edu  Wed Feb 10 10:58:02 2010
From: nasser at udel.edu (Nasser Mohieddin Abukhdeir)
Date: Wed, 10 Feb 2010 10:58:02 -0500
Subject: [SciPy-User] SciPy-User Digest, Vol 78, Issue 19
In-Reply-To: <mailman.4907.1265814526.2454.scipy-user@scipy.org>
References: <mailman.4907.1265814526.2454.scipy-user@scipy.org>
Message-ID: <1265817482.2185.32.camel@nasser-desktop>

Hello Mark:
    I am quite new to Python and Scipy, but I have some experience with
SciPy's ODE functionality and it has been comparable to compiled code,
depending on my implementation of the integrand. There are a few
important parameters for the integrator that you should take into
account in order to assure a fair comparison. Based upon odeint's
default values, you should confirm that the relative and absolute error
tolerances are the same for both computations. Something else that is
important is the integration scheme, it looks like both methods use
LSODA, so it should be an even comparison. Can you rerun both
simulations and check the number of time steps and function evaluations
from each implementation?


Nasser Mohieddin Abukhdeir
Postdoctoral Fellow (Vlachos Research Group)
University of Delaware - Department of Chemical Engineering
(Personal Web Page) http://udel.edu/~nasser/
(Group Web Page)    http://www.dion.che.udel.edu/

On Wed, 2010-02-10 at 09:08 -0600, scipy-user-request at scipy.org wrote:


> Message: 5
> Date: Wed, 10 Feb 2010 10:08:29 -0500
> From: Mark P <m.perrin at me.com>
> Subject: [SciPy-User] odeint (python vs mathematica)
> To: Scipy <scipy-user at scipy.org>
> Message-ID: <18B95518-6CA6-4C5F-BB57-3BBECC863C0B at me.com>
> Content-Type: text/plain; charset="us-ascii"
> 
> Hello all,
> 
> (my first post to the list)
> 
> I have been performing modelling of ionic currents in Mathematica for the past three years as a student. I have recently passed into the hallowed halls of post-grad, and am keen to be done with Mathematica and use a free and open solution for my projects, hence Python and numpy/scipy. 
> 
> I've made some progress with the transition; the cookbooks are especially helpful. There is one in particular I have been looking at - the rabbit/fox model ( Lotka-Volterra Tutorial), that is helping me to understand odeint.
> 
> =====
> from numpy import *
> import pylab as p
> # Definition of parameters
> a = 1.
> b = 0.1
> c = 1.5
> d = 0.75
> def dX_dt(X, t=0):
>     """ Return the growth rate of fox and rabbit populations. """
>     return array([ a*X[0] -   b*X[0]*X[1] ,
>                   -c*X[1] + d*b*X[0]*X[1] ]) 
> 
> from scipy import integrate
> 
> t = linspace(0, 10000,  10000)              # time
> X0 = array([10, 5])                     		# initials conditions: 10 rabbits and 5 foxes
> X = integrate.odeint(dX_dt, X0, t)
> ===
> 
> A simple form of this code in Mathematica is:
> 
> ===
> a = 1
> b = 0.1
> c = 1.5
> d = 0.75
> 
> eqn1 := r'[t] == a * r[t] - b * r[t] * f[t]
> eqn2 := f'[t] == -c * f[t] + d * b * r[t] * f[t]
> 
> solution = NDSolve[{eqn1, eqn2, r[0] == 10, f[0] == 5}, {r[t], f[t]}, {t, 0, 10000}, MaxSteps -> Infinity]
> ===
> 
> My question is about speed. When computed in Mathematica 7 the code is roughly about 10x faster than odeint (I've increased t from 15 in the cookbook to 10000 to allow the measurement of speed to be made). But I know from the web that numpy/scipy can be of an equivalent or faster speed. I was wondering whether the solution can be sped up; and would that require the use of C or Fortran.
> 
> I see the inherent advantage of using Python with numpy/scipy and I do not expect the speeds of a compiled language but I was hoping for equivalent speeds to other interpreted solutions.
> 
> Thank you,
> 
> Mark Perrin
> Electrophysiology Research Fellow
> Ottawa
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/0e2124a7/attachment.html 
> 
> ------------------------------
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 
> End of SciPy-User Digest, Vol 78, Issue 19
> ******************************************





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/90429071/attachment.html>

From schut at sarvision.nl  Wed Feb 10 11:06:15 2010
From: schut at sarvision.nl (Vincent Schut)
Date: Wed, 10 Feb 2010 17:06:15 +0100
Subject: [SciPy-User] interpolation grid/scatter
In-Reply-To: <4B72D789.8090209@gmail.com>
References: <4B72D789.8090209@gmail.com>
Message-ID: <hkulhm$p0j$1@ger.gmane.org>

On 02/10/2010 04:58 PM, fred wrote:
> Hi all,
>
> Say I have data grid like this:
>
> xmin, xmax, dx = 0, 5, 1
> ymin, ymax, dy = 0, 4, 1
>
> data.shape : (6,5)
>
> I would like to get scalar value at points that are _not_ on the grid
> nodes, say at x=1.213, y=3.24.
>
> How could I do that?
>
> Is there a scipy function to do what I want?
>
> PS : I have about 100 points on which I want the scalar value
>
> PS2 : my array can be 3D array.
>
> TIA.
>
> Cheers,
>

Fred,

As long as your points are all within the data grid boundaries (so: no 
extrapolation) and your data grid is regular and without voids, 
scipy.ndimage.map_coordinates should do what you want.
Be careful for spline interpolation artifacts though. Under/overshoots 
might result when using order=2 or highter. Order=1 will interpolate 
linearly, order=0 does nearest neighbour mapping.

Vincent.



From denis-bz-gg at t-online.de  Wed Feb 10 11:27:43 2010
From: denis-bz-gg at t-online.de (denis)
Date: Wed, 10 Feb 2010 08:27:43 -0800 (PST)
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <F1099DFF-8A2C-4B7B-991A-448E6940A79E@gmail.com>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu> 
	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com> 
	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
	<76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com> 
	<4B6CA2F3.10008@student.matnat.uio.no>
	<E6517CF3-C57C-436D-BAF4-E5C9D0BF61AF@gmail.com> 
	<2ca69d04-0b1f-4f54-87a6-eb953bc2b57c@b7g2000yqd.googlegroups.com> 
	<F1099DFF-8A2C-4B7B-991A-448E6940A79E@gmail.com>
Message-ID: <f5a4114d-5a4e-4868-9b2e-16af31dd8343@g28g2000yqh.googlegroups.com>

Hi Tony,
  yet more notation:

you start with "data points", and add some "control points" or "knots"
or "handles"
to bend or shape the curve or surface towards the knots.
Visually, you add and move "control points" with a mouse to shape the
curve.
(The starting curve may go through the data points, or may not,
e.g. Catmull-Rom / Bspline.)OB

Here's a confusion, experts correct me: "knots" can mean
1) "control points" added as above,
2) or t values where cubic pieces join, initial (for me 0 1 2) or
added.

For any cloud of data points, you can generate any number of spline
pieces,
with various smoothing criteria.
That is, Ndata and Nsplinepiece can be related any which way;
the simplest case, in dotspline, has e.g. Ndata=11, Npiece=10,
but you can have Ndata > Nsplinepiece if Ndata is big
and Ndata < Nsplinepiece too.

There are zillions of papers on this, each with its own thicket of
notation.
Scipy.interpolate is based on fitpack by Prof. Dierckx, who has a
book;
books.google.com "spline" in title => 302 books since 1990 ?!
Sederberg, http://tom.cs.byu.edu/~557/text/cagd.pdf is 250 pages, v
readable.

An interactive applet combining C-R, Bspline, and knots would be nice:
anyone ?
For fun, look at the applets in http://en.wikipedia.org/wiki/Bezier_curve
also http://en.wikipedia.org/wiki/Bezier_spline for control points.


I'll add derivatives to the code on http://advice.mechanicalkern.com
(if I can edit it, getting login - claimid - timeout.)
Example:
    deriv = dotspline1( x, nsample, spline="Catmull-Rom-derivative" )

cheers
  -- denis


From fredmfp at gmail.com  Wed Feb 10 11:32:47 2010
From: fredmfp at gmail.com (fred)
Date: Wed, 10 Feb 2010 17:32:47 +0100
Subject: [SciPy-User] interpolation grid/scatter
In-Reply-To: <hkulhm$p0j$1@ger.gmane.org>
References: <4B72D789.8090209@gmail.com> <hkulhm$p0j$1@ger.gmane.org>
Message-ID: <4B72DFAF.20608@gmail.com>

Joe & Vincent,

I already use map_coordinates() for others purposes, and I did not
understand I can and how I can use it for this purpose here.

Thanks to have pointed it.


Cheers,

-- 
Fred


From zachary.pincus at yale.edu  Wed Feb 10 11:42:19 2010
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 10 Feb 2010 11:42:19 -0500
Subject: [SciPy-User] moving window (2D) correlation coefficient
In-Reply-To: <hkukb3$k0h$1@ger.gmane.org>
References: <hkukb3$k0h$1@ger.gmane.org>
Message-ID: <0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>

I bet that you could construct an array with shape (x,y,5,5), where  
array[i,j,:,:] would give the 5x5 window around (i,j), as some sort of  
mind-bending view on an array of shape (x,y), using a positive offset  
and some dimensions having negative strides. Then you could compute  
the correlation coefficient between the two arrays directly. Maybe?

Probably cython would be more maintainable...

Zach


On Feb 10, 2010, at 10:45 AM, Vincent Schut wrote:

> Hi,
>
> I need to calculate the correlation coefficient of a (simultaneously)
> moving window over 2 images, such that the resulting pixel x,y (center
> of the window) is the corrcoef((a 5x5 window around x,y for image  
> A), (a
> 5x5 window around x,y for image B)).
> Currently, I just loop over y, x, and then call corrcoef for each x,y
> window. Would there be a better way, other than converting the loop to
> cython?
>
>
> For clarity (or not), the relevant part from my code:
>
>
> for y in range(d500shape[2]):
>     for x in range(d500shape[3]):
>         if valid500[d,y,x]:
>             window = spectral500Bordered[d,:,y:y+5, x:x 
> +5].reshape(7, -1)
>             for b in range(5):
>                 nonzeroMask = (window[0] > 0)
>                 b01corr[0,b,y,x] =
> numpy.corrcoef(window[0][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>                 b01corr[1,b,y,x] =
> numpy.corrcoef(window[1][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>
>
> forget the 'if valid500' and 'nonzeroMask', those are to prevent
> calculating pixels that don't need to be calculated, and to feed only
> valid pixels from the window into corrcoef
> spectral500Bordered is essentially a [d dates, 7 images, ysize, xsize]
> array. I work per date (d), then calculate the corrcoef for images[0]
> versus images[2:], and for images[1] versus images[2:]
>
> Thanks,
> Vincent.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From josef.pktd at gmail.com  Wed Feb 10 11:53:28 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 10 Feb 2010 11:53:28 -0500
Subject: [SciPy-User] moving window (2D) correlation coefficient
In-Reply-To: <0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>
References: <hkukb3$k0h$1@ger.gmane.org>
	<0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>
Message-ID: <1cd32cbb1002100853t43bca77cxb93ca1242451204b@mail.gmail.com>

On Wed, Feb 10, 2010 at 11:42 AM, Zachary Pincus
<zachary.pincus at yale.edu> wrote:
> I bet that you could construct an array with shape (x,y,5,5), where
> array[i,j,:,:] would give the 5x5 window around (i,j), as some sort of
> mind-bending view on an array of shape (x,y), using a positive offset
> and some dimensions having negative strides. Then you could compute
> the correlation coefficient between the two arrays directly. Maybe?
>
> Probably cython would be more maintainable...
>
> Zach
>
>
> On Feb 10, 2010, at 10:45 AM, Vincent Schut wrote:
>
>> Hi,
>>
>> I need to calculate the correlation coefficient of a (simultaneously)
>> moving window over 2 images, such that the resulting pixel x,y (center
>> of the window) is the corrcoef((a 5x5 window around x,y for image
>> A), (a
>> 5x5 window around x,y for image B)).
>> Currently, I just loop over y, x, and then call corrcoef for each x,y
>> window. Would there be a better way, other than converting the loop to
>> cython?
>>
>>
>> For clarity (or not), the relevant part from my code:
>>
>>
>> for y in range(d500shape[2]):
>> ? ? for x in range(d500shape[3]):
>> ? ? ? ? if valid500[d,y,x]:
>> ? ? ? ? ? ? window = spectral500Bordered[d,:,y:y+5, x:x
>> +5].reshape(7, -1)
>> ? ? ? ? ? ? for b in range(5):
>> ? ? ? ? ? ? ? ? nonzeroMask = (window[0] > 0)
>> ? ? ? ? ? ? ? ? b01corr[0,b,y,x] =
>> numpy.corrcoef(window[0][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>> ? ? ? ? ? ? ? ? b01corr[1,b,y,x] =
>> numpy.corrcoef(window[1][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>
>>
>> forget the 'if valid500' and 'nonzeroMask', those are to prevent
>> calculating pixels that don't need to be calculated, and to feed only
>> valid pixels from the window into corrcoef
>> spectral500Bordered is essentially a [d dates, 7 images, ysize, xsize]
>> array. I work per date (d), then calculate the corrcoef for images[0]
>> versus images[2:], and for images[1] versus images[2:]

I wrote a moving correlation for time series last november (scipy user
and preceding discussion on numpy mailing list)
I don't work with pictures, so I don't know if this can be extended to
your case. Since signal.correlate or convolve work in all directions
it might be possible

   def yxcov(self):
       xys = signal.correlate(self.x*self.y, self.kern,
mode='same')[self.sslice]
       return xys/self.n - self.ymean*self.xmean

Josef

>>
>> Thanks,
>> Vincent.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From denis-bz-gg at t-online.de  Wed Feb 10 12:01:03 2010
From: denis-bz-gg at t-online.de (denis)
Date: Wed, 10 Feb 2010 09:01:03 -0800 (PST)
Subject: [SciPy-User] Efficiently applying a function over a large array
In-Reply-To: <91d218431002090812p574fbb6chf52fa3cc2a8c56c9@mail.gmail.com>
References: <91d218431002090812p574fbb6chf52fa3cc2a8c56c9@mail.gmail.com>
Message-ID: <d371c99d-6d91-444e-add1-443bb60b8fb2@l19g2000yqb.googlegroups.com>

Jose,
  it sounds as though MyFunction is doing quite a lot, say 100 times
more flops
than the outer loop ? If so, speeding up the loop would gain < 1 %.
You could timeit for i: for j: dummyfunc()
but in general describe, code, test, *then* look at the whole
picture / speedup.
Bytheway arr1[i,j,:] is fast too, no copy.

cheers
  -- denis


From m.perrin at me.com  Wed Feb 10 12:17:47 2010
From: m.perrin at me.com (Mark P)
Date: Wed, 10 Feb 2010 12:17:47 -0500
Subject: [SciPy-User] odeint python and mathematica
In-Reply-To: <1265817482.2185.32.camel@nasser-desktop>
References: <mailman.4907.1265814526.2454.scipy-user@scipy.org>
	<1265817482.2185.32.camel@nasser-desktop>
Message-ID: <66939327-ACDC-4120-B978-94028F1A7655@me.com>

In the Python routine:
	- number of time step evaluations 	50,000
	- number of function evaluations 	120,000

In the Mathematica routine:
	- time steps 					231,770
	- function evaluations 			500,000

I'm not sure how to work out the relative and absolute error tolerances, but I did set the 'atol' in Python to a number just before the answer became inaccurate and it did speed it up 2.5x but still 4x slower than Mathematica.

Thank you,

Mark


On 2010-02-10, at 10:58 AM, Nasser Mohieddin Abukhdeir wrote:

> Hello Mark:
>     I am quite new to Python and Scipy, but I have some experience with SciPy's ODE functionality and it has been comparable to compiled code, depending on my implementation of the integrand. There are a few important parameters for the integrator that you should take into account in order to assure a fair comparison. Based upon odeint's default values, you should confirm that the relative and absolute error tolerances are the same for both computations. Something else that is important is the integration scheme, it looks like both methods use LSODA, so it should be an even comparison. Can you rerun both simulations and check the number of time steps and function evaluations from each implementation?
> 
> 
> Nasser Mohieddin Abukhdeir
> Postdoctoral Fellow (Vlachos Research Group)
> University of Delaware - Department of Chemical Engineering
> (Personal Web Page) http://udel.edu/~nasser/
> (Group Web Page)    http://www.dion.che.udel.edu/
> 
> On Wed, 2010-02-10 at 09:08 -0600, scipy-user-request at scipy.org wrote:
> 
>> Message: 5
>> Date: Wed, 10 Feb 2010 10:08:29 -0500
>> From: Mark P <m.perrin at me.com>
>> Subject: [SciPy-User] odeint (python vs mathematica)
>> To: Scipy <scipy-user at scipy.org>
>> Message-ID: <18B95518-6CA6-4C5F-BB57-3BBECC863C0B at me.com>
>> Content-Type: text/plain; charset="us-ascii"
>> 
>> Hello all,
>> 
>> (my first post to the list)
>> 
>> I have been performing modelling of ionic currents in Mathematica for the past three years as a student. I have recently passed into the hallowed halls of post-grad, and am keen to be done with Mathematica and use a free and open solution for my projects, hence Python and numpy/scipy. 
>> 
>> I've made some progress with the transition; the cookbooks are especially helpful. There is one in particular I have been looking at - the rabbit/fox model ( Lotka-Volterra Tutorial), that is helping me to understand odeint.
>> 
>> =====
>> from numpy import *
>> import pylab as p
>> # Definition of parameters
>> a = 1.
>> b = 0.1
>> c = 1.5
>> d = 0.75
>> def dX_dt(X, t=0):
>>     """ Return the growth rate of fox and rabbit populations. """
>>     return array([ a*X[0] -   b*X[0]*X[1] ,
>>                   -c*X[1] + d*b*X[0]*X[1] ]) 
>> 
>> from scipy import integrate
>> 
>> t = linspace(0, 10000,  10000)              # time
>> X0 = array([10, 5])                     		# initials conditions: 10 rabbits and 5 foxes
>> X = integrate.odeint(dX_dt, X0, t)
>> ===
>> 
>> A simple form of this code in Mathematica is:
>> 
>> ===
>> a = 1
>> b = 0.1
>> c = 1.5
>> d = 0.75
>> 
>> eqn1 := r'[t] == a * r[t] - b * r[t] * f[t]
>> eqn2 := f'[t] == -c * f[t] + d * b * r[t] * f[t]
>> 
>> solution = NDSolve[{eqn1, eqn2, r[0] == 10, f[0] == 5}, {r[t], f[t]}, {t, 0, 10000}, MaxSteps -> Infinity]
>> ===
>> 
>> My question is about speed. When computed in Mathematica 7 the code is roughly about 10x faster than odeint (I've increased t from 15 in the cookbook to 10000 to allow the measurement of speed to be made). But I know from the web that numpy/scipy can be of an equivalent or faster speed. I was wondering whether the solution can be sped up; and would that require the use of C or Fortran.
>> 
>> I see the inherent advantage of using Python with numpy/scipy and I do not expect the speeds of a compiled language but I was hoping for equivalent speeds to other interpreted solutions.
>> 
>> Thank you,
>> 
>> Mark Perrin
>> Electrophysiology Research Fellow
>> Ottawa
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/0e2124a7/attachment.html 
>> 
>> ------------------------------
>> 
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> 
>> 
>> End of SciPy-User Digest, Vol 78, Issue 19
>> ******************************************
> 
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/db339972/attachment.html>

From denis-bz-gg at t-online.de  Wed Feb 10 12:38:14 2010
From: denis-bz-gg at t-online.de (denis)
Date: Wed, 10 Feb 2010 09:38:14 -0800 (PST)
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <D977291C-4891-44BA-BC68-9743A9D98309@yale.edu>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu> 
	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com> 
	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
	<76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com> 
	<4B6CA2F3.10008@student.matnat.uio.no>
	<E6517CF3-C57C-436D-BAF4-E5C9D0BF61AF@gmail.com> 
	<2ca69d04-0b1f-4f54-87a6-eb953bc2b57c@b7g2000yqd.googlegroups.com> 
	<D977291C-4891-44BA-BC68-9743A9D98309@yale.edu>
Message-ID: <23239434-1a26-4df5-9dae-013dd548a0f9@o28g2000yqh.googlegroups.com>


On Feb 9, 6:05?pm, Zachary Pincus <zachary.pin... at yale.edu> wrote:

>
> I had always thought that the splines produced by fitpack were plain ?
> (and local) non-uniform B-splines (as opposed to the uniform kind, ?
> which Denis's really helpful demo code uses). However, the way that ?
> those B-splines are fit to the given data by fitpack is definitely ?
> global and can lead to odd artifacts if you're incautious.
>
> Is this correct?
>
> Then is reason that the B-splines from scipy.interpolate.fitpack and ?
> from scipy.signal are a bit different is that the latter are also ?
> strictly uniform? (E.g. evenly-spaced knot vector.) Probably this is ?
> easy to test...

Zach,
  if UnivariateSpline were local, its response to 0 0 1 0 0 1000
should be 0 0 1 0 ...
i.e. shouldn't see the 1000, right ?  Doesn't look so --

""" is scipy UnivariateSpline local ?  1 0 0 1000 """
from __future__ import division
import numpy as np
from scipy.interpolate import UnivariateSpline  # fitpack
import pylab as pl

N = 10
H = 10
NH = N * H
N2 = N//2
x = np.arange(N+1)
y = np.zeros(N+1);  y[N2] = 1;  y[N2+3] = 1000
xup = np.arange( 0, N, 1/H )  # N * H
np.set_printoptions( 1, threshold=100, edgeitems=3*H, suppress=True )
# .1f

#..............................................................................
title = __doc__
interpolator = UnivariateSpline( x, y, k=3, s=0 )  # s=0 interpolates
yup = interpolator( xup )
print "yup:", yup[N2*H - 1 : (N2+3)*H]

pl.title( title )
pl.plot( xup, yup )
ax = pl.gca()
ax.set_xlim( N2 - 1, N2 + 5 )
ax.set_ylim( -10, 10 )
pl.show()

The real problem imho is that the doc doesn't say
"we have more software, with less doc, than at any time in history"

cheers
  -- denis


From zachary.pincus at yale.edu  Wed Feb 10 12:56:04 2010
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 10 Feb 2010 12:56:04 -0500
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <23239434-1a26-4df5-9dae-013dd548a0f9@o28g2000yqh.googlegroups.com>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu>
	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com>
	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
	<76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com>
	<4B6CA2F3.10008@student.matnat.uio.no>
	<E6517CF3-C57C-436D-BAF4-E5C9D0BF61AF@gmail.com>
	<2ca69d04-0b1f-4f54-87a6-eb953bc2b57c@b7g2000yqd.googlegroups.com>
	<D977291C-4891-44BA-BC68-9743A9D98309@yale.edu>
	<23239434-1a26-4df5-9dae-013dd548a0f9@o28g2000yqh.googlegroups.com>
Message-ID: <5578B53D-6403-42F7-836A-2948BB2674F8@yale.edu>

>> I had always thought that the splines produced by fitpack were plain
>> (and local) non-uniform B-splines (as opposed to the uniform kind,
>> which Denis's really helpful demo code uses). However, the way that
>> those B-splines are fit to the given data by fitpack is definitely
>> global and can lead to odd artifacts if you're incautious.
>>
>> Is this correct?
>>
>> Then is reason that the B-splines from scipy.interpolate.fitpack and
>> from scipy.signal are a bit different is that the latter are also
>> strictly uniform? (E.g. evenly-spaced knot vector.) Probably this is
>> easy to test...
>
> Zach,
>  if UnivariateSpline were local, its response to 0 0 1 0 0 1000
> should be 0 0 1 0 ...
> i.e. shouldn't see the 1000, right ?  Doesn't look so --

Yeah -- the *fitting* procedure (fitpack.splrep / fitpack.splprep) is  
global, as above. (Note that if you give s=0, this forces the spline  
to interpolate the data, and you get as many knots as data points.)  
However, I'm pretty sure that the splines that are then produced are  
just regular, local non-uniform b-splines, in that altering the  
control points can't alter the curve positions on knot spans that  
aren't active.

As for your original question, I would figure that fitpack should be  
able to evaluate the bsplines produced by scipy.signal properly,  
assuming you are able to give it the right knot vector. But not vice- 
versa, since fitpack produces non-uniform knot vectors in general, and  
signal is just for uniform ones. Right?

Zach


From nasser at udel.edu  Wed Feb 10 13:10:53 2010
From: nasser at udel.edu (Nasser Mohieddin Abukhdeir)
Date: Wed, 10 Feb 2010 13:10:53 -0500
Subject: [SciPy-User] odeint python and mathematica
In-Reply-To: <66939327-ACDC-4120-B978-94028F1A7655@me.com>
References: <mailman.4907.1265814526.2454.scipy-user@scipy.org>
	<1265817482.2185.32.camel@nasser-desktop>
	<66939327-ACDC-4120-B978-94028F1A7655@me.com>
Message-ID: <1265825453.2185.44.camel@nasser-desktop>

It seems as though there are two different integration methods being
used, so you really need to get to the bottom of that. There is one
other important difference between SciPy and Mathematica in this case,
Mathematica has just-in-time compilation functionality:

http://www.wolfram.com/technology/guide/TransparentAutoCompilation/

while SciPy does not, although it is using compiled FORTRAN modules.
Your case is also an extreme one, a system of two ODEs, where I
typically deal with systems of 10^3+. The Python overhead is much less
of a problem for me since all of the matrix operations are done in
C-compiled code. Look into SciPy's Weave functionality, I have not had
much success with it and am working on using Cython to speed-up my code
and so I can easily interface with external C-code. It might seem like
alot of work now, but switching from Matlab to Python (and not C) has
benefited me greatly.  I think it is well worth the effort, access to
VODE through scipy.integrate.ode, excellent 3D plotting using Mayavi,
and wrappers exist for the SUNDIALS ODE package if you ever need a
matrix-free implicit ODE solver (see PySundials).

Nasser

On Wed, 2010-02-10 at 12:17 -0500, Mark P wrote:
> In the Python routine:
> - number of time step evaluations 50,000
> - number of function evaluations 120,000
> 
> 
> In the Mathematica routine:
> - time steps  231,770
> - function evaluations 500,000
> 
> 
> I'm not sure how to work out the relative and absolute error
> tolerances, but I did set the 'atol' in Python to a number just before
> the answer became inaccurate and it did speed it up 2.5x but still 4x
> slower than Mathematica.
> 
> 
> Thank you,
> 
> 
> Mark
> 
> 
> 
> 
> 
> On 2010-02-10, at 10:58 AM, Nasser Mohieddin Abukhdeir wrote:
> 
> 
> 
> > Hello Mark:
> >     I am quite new to Python and Scipy, but I have some experience
> > with SciPy's ODE functionality and it has been comparable to
> > compiled code, depending on my implementation of the integrand.
> > There are a few important parameters for the integrator that you
> > should take into account in order to assure a fair comparison. Based
> > upon odeint's default values, you should confirm that the relative
> > and absolute error tolerances are the same for both computations.
> > Something else that is important is the integration scheme, it looks
> > like both methods use LSODA, so it should be an even comparison. Can
> > you rerun both simulations and check the number of time steps and
> > function evaluations from each implementation?
> > 
> > 
> > Nasser Mohieddin Abukhdeir
> > Postdoctoral Fellow (Vlachos Research Group)
> > University of Delaware - Department of Chemical Engineering
> > (Personal Web Page) http://udel.edu/~nasser/
> > (Group Web Page)    http://www.dion.che.udel.edu/
> > 
> > On Wed, 2010-02-10 at 09:08 -0600, scipy-user-request at scipy.org
> > wrote:
> > 
> > 
> > > Message: 5
> > > Date: Wed, 10 Feb 2010 10:08:29 -0500
> > > From: Mark P <m.perrin at me.com>
> > > Subject: [SciPy-User] odeint (python vs mathematica)
> > > To: Scipy <scipy-user at scipy.org>
> > > Message-ID: <18B95518-6CA6-4C5F-BB57-3BBECC863C0B at me.com>
> > > Content-Type: text/plain; charset="us-ascii"
> > > 
> > > Hello all,
> > > 
> > > (my first post to the list)
> > > 
> > > I have been performing modelling of ionic currents in Mathematica for the past three years as a student. I have recently passed into the hallowed halls of post-grad, and am keen to be done with Mathematica and use a free and open solution for my projects, hence Python and numpy/scipy. 
> > > 
> > > I've made some progress with the transition; the cookbooks are especially helpful. There is one in particular I have been looking at - the rabbit/fox model ( Lotka-Volterra Tutorial), that is helping me to understand odeint.
> > > 
> > > =====
> > > from numpy import *
> > > import pylab as p
> > > # Definition of parameters
> > > a = 1.
> > > b = 0.1
> > > c = 1.5
> > > d = 0.75
> > > def dX_dt(X, t=0):
> > >     """ Return the growth rate of fox and rabbit populations. """
> > >     return array([ a*X[0] -   b*X[0]*X[1] ,
> > >                   -c*X[1] + d*b*X[0]*X[1] ]) 
> > > 
> > > from scipy import integrate
> > > 
> > > t = linspace(0, 10000,  10000)              # time
> > > X0 = array([10, 5])                     		# initials conditions: 10 rabbits and 5 foxes
> > > X = integrate.odeint(dX_dt, X0, t)
> > > ===
> > > 
> > > A simple form of this code in Mathematica is:
> > > 
> > > ===
> > > a = 1
> > > b = 0.1
> > > c = 1.5
> > > d = 0.75
> > > 
> > > eqn1 := r'[t] == a * r[t] - b * r[t] * f[t]
> > > eqn2 := f'[t] == -c * f[t] + d * b * r[t] * f[t]
> > > 
> > > solution = NDSolve[{eqn1, eqn2, r[0] == 10, f[0] == 5}, {r[t], f[t]}, {t, 0, 10000}, MaxSteps -> Infinity]
> > > ===
> > > 
> > > My question is about speed. When computed in Mathematica 7 the code is roughly about 10x faster than odeint (I've increased t from 15 in the cookbook to 10000 to allow the measurement of speed to be made). But I know from the web that numpy/scipy can be of an equivalent or faster speed. I was wondering whether the solution can be sped up; and would that require the use of C or Fortran.
> > > 
> > > I see the inherent advantage of using Python with numpy/scipy and I do not expect the speeds of a compiled language but I was hoping for equivalent speeds to other interpreted solutions.
> > > 
> > > Thank you,
> > > 
> > > Mark Perrin
> > > Electrophysiology Research Fellow
> > > Ottawa
> > > -------------- next part --------------
> > > An HTML attachment was scrubbed...
> > > URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/0e2124a7/attachment.html 
> > > 
> > > ------------------------------
> > > 
> > > _______________________________________________
> > > SciPy-User mailing list
> > > SciPy-User at scipy.org
> > > http://mail.scipy.org/mailman/listinfo/scipy-user
> > > 
> > > 
> > > End of SciPy-User Digest, Vol 78, Issue 19
> > > ******************************************
> > 
> > 
> > 
> > 
> > 
> > 
> 
> 



Nasser Mohieddin Abukhdeir
Postdoctoral Fellow (Vlachos Research Group)
University of Delaware - Department of Chemical Engineering
(Personal Web Page) http://udel.edu/~nasser/
(Group Web Page)    http://www.dion.che.udel.edu/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/b9f429d1/attachment.html>

From silva at lma.cnrs-mrs.fr  Wed Feb 10 13:24:52 2010
From: silva at lma.cnrs-mrs.fr (Fabrice Silva)
Date: Wed, 10 Feb 2010 19:24:52 +0100
Subject: [SciPy-User] Splines in scipy.signal vs scipy.interpolation
In-Reply-To: <f5a4114d-5a4e-4868-9b2e-16af31dd8343@g28g2000yqh.googlegroups.com>
References: <9AF13441-AFE5-4568-9438-4E98D6E99EDF@mit.edu>
	<8b9578e2-7308-4c8b-95cf-57bb49572029@v25g2000yqk.googlegroups.com>
	<733616BB-EC5C-43E5-B8F9-FA26EB538B9E@gmail.com>
	<76005EB5-58B7-4CE3-8244-99B44DDE2CF7@gmail.com>
	<4B6CA2F3.10008@student.matnat.uio.no>
	<E6517CF3-C57C-436D-BAF4-E5C9D0BF61AF@gmail.com>
	<2ca69d04-0b1f-4f54-87a6-eb953bc2b57c@b7g2000yqd.googlegroups.com>
	<F1099DFF-8A2C-4B7B-991A-448E6940A79E@gmail.com>
	<f5a4114d-5a4e-4868-9b2e-16af31dd8343@g28g2000yqh.googlegroups.com>
Message-ID: <1265826292.1776.9.camel@Portable-s2m.cnrs-mrs.fr>

Le mercredi 10 f?vrier 2010 ? 08:27 -0800, denis a ?crit :
> An interactive applet combining C-R, Bspline, and knots would be nice:
> anyone ?
> For fun, look at the applets in http://en.wikipedia.org/wiki/Bezier_curve
> also http://en.wikipedia.org/wiki/Bezier_spline for control points.

For my own needs, I just ended a (smoothing) parametric Bspline class
that, through a matplotlib gui, let you :
- fitting a Bspline with the standard value of s=N-2sqrt(N) where N is
the number of sample of the data (measured in my case)
- change the value of the fitting parameter s
- modify manually the position of the knots and control points.

I'll post it in the next few days, after cleaning it.


Imho (and mainly in what I have read from Dierckx), the knots are the
values of the 1-D parameter and the control points may be N-D (so your
second option)





From schut at sarvision.nl  Wed Feb 10 14:31:16 2010
From: schut at sarvision.nl (Vincent Schut)
Date: Wed, 10 Feb 2010 20:31:16 +0100
Subject: [SciPy-User] moving window (2D) correlation coefficient
In-Reply-To: <0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>
References: <hkukb3$k0h$1@ger.gmane.org>
	<0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>
Message-ID: <hkv1i3$7a3$1@ger.gmane.org>

On 02/10/2010 05:42 PM, Zachary Pincus wrote:
> I bet that you could construct an array with shape (x,y,5,5), where
> array[i,j,:,:] would give the 5x5 window around (i,j), as some sort of
> mind-bending view on an array of shape (x,y), using a positive offset
> and some dimensions having negative strides.

I tried that, but then less clever (just reshaping and transposing); it 
made a copy, and I ended up with an array that took 5*5 times as much 
space as my initial array... And it took a lot of time.
But I don't see how this reshape would gain me much speed?

  Then you could compute
> the correlation coefficient between the two arrays directly. Maybe?

Ah, like that. I thought numpy.corrcoef(x,y) only worked on flat x and 
flat y. There is no axis keyword... Is there another function that would 
work on nd arrays?

>
> Probably cython would be more maintainable...

Well, speed is more important than readability this time :-) It's 
terabytes of data I'll need to push through this function...
What I understood till now (just starting to look at cython) is that in 
this kind of thing, the repeated array sub-indexing (and its bounds 
checking etc.) is the main bottleneck. I presume the corrcoef function 
is pretty much optimized? So I thought if I could do the loop in cython 
and implement some clever code for the window indexing (only replace the 
new elements), I'd relatively simple get the major speed gain. Does that 
sound correct?
Only still have to find out now how to call the corrcoef function from 
cython...

Thanks for your thoughts on this,
Vincent.
>
> Zach
>
>
> On Feb 10, 2010, at 10:45 AM, Vincent Schut wrote:
>
>> Hi,
>>
>> I need to calculate the correlation coefficient of a (simultaneously)
>> moving window over 2 images, such that the resulting pixel x,y (center
>> of the window) is the corrcoef((a 5x5 window around x,y for image
>> A), (a
>> 5x5 window around x,y for image B)).
>> Currently, I just loop over y, x, and then call corrcoef for each x,y
>> window. Would there be a better way, other than converting the loop to
>> cython?
>>
>>
>> For clarity (or not), the relevant part from my code:
>>
>>
>> for y in range(d500shape[2]):
>>      for x in range(d500shape[3]):
>>          if valid500[d,y,x]:
>>              window = spectral500Bordered[d,:,y:y+5, x:x
>> +5].reshape(7, -1)
>>              for b in range(5):
>>                  nonzeroMask = (window[0]>  0)
>>                  b01corr[0,b,y,x] =
>> numpy.corrcoef(window[0][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>                  b01corr[1,b,y,x] =
>> numpy.corrcoef(window[1][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>
>>
>> forget the 'if valid500' and 'nonzeroMask', those are to prevent
>> calculating pixels that don't need to be calculated, and to feed only
>> valid pixels from the window into corrcoef
>> spectral500Bordered is essentially a [d dates, 7 images, ysize, xsize]
>> array. I work per date (d), then calculate the corrcoef for images[0]
>> versus images[2:], and for images[1] versus images[2:]
>>
>> Thanks,
>> Vincent.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user




From schut at sarvision.nl  Wed Feb 10 14:36:44 2010
From: schut at sarvision.nl (Vincent Schut)
Date: Wed, 10 Feb 2010 20:36:44 +0100
Subject: [SciPy-User] moving window (2D) correlation coefficient
In-Reply-To: <1cd32cbb1002100853t43bca77cxb93ca1242451204b@mail.gmail.com>
References: <hkukb3$k0h$1@ger.gmane.org>	<0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>
	<1cd32cbb1002100853t43bca77cxb93ca1242451204b@mail.gmail.com>
Message-ID: <hkv1sb$8ut$1@ger.gmane.org>

On 02/10/2010 05:53 PM, josef.pktd at gmail.com wrote:
> On Wed, Feb 10, 2010 at 11:42 AM, Zachary Pincus
> <zachary.pincus at yale.edu>  wrote:
>> I bet that you could construct an array with shape (x,y,5,5), where
>> array[i,j,:,:] would give the 5x5 window around (i,j), as some sort of
>> mind-bending view on an array of shape (x,y), using a positive offset
>> and some dimensions having negative strides. Then you could compute
>> the correlation coefficient between the two arrays directly. Maybe?
>>
>> Probably cython would be more maintainable...
>>
>> Zach
>>
>>
>> On Feb 10, 2010, at 10:45 AM, Vincent Schut wrote:
>>
>>> Hi,
>>>
>>> I need to calculate the correlation coefficient of a (simultaneously)
>>> moving window over 2 images, such that the resulting pixel x,y (center
>>> of the window) is the corrcoef((a 5x5 window around x,y for image
>>> A), (a
>>> 5x5 window around x,y for image B)).
>>> Currently, I just loop over y, x, and then call corrcoef for each x,y
>>> window. Would there be a better way, other than converting the loop to
>>> cython?
>>>
>>>
>>> For clarity (or not), the relevant part from my code:
>>>
>>>
>>> for y in range(d500shape[2]):
>>>      for x in range(d500shape[3]):
>>>          if valid500[d,y,x]:
>>>              window = spectral500Bordered[d,:,y:y+5, x:x
>>> +5].reshape(7, -1)
>>>              for b in range(5):
>>>                  nonzeroMask = (window[0]>  0)
>>>                  b01corr[0,b,y,x] =
>>> numpy.corrcoef(window[0][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>>                  b01corr[1,b,y,x] =
>>> numpy.corrcoef(window[1][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>>
>>>
>>> forget the 'if valid500' and 'nonzeroMask', those are to prevent
>>> calculating pixels that don't need to be calculated, and to feed only
>>> valid pixels from the window into corrcoef
>>> spectral500Bordered is essentially a [d dates, 7 images, ysize, xsize]
>>> array. I work per date (d), then calculate the corrcoef for images[0]
>>> versus images[2:], and for images[1] versus images[2:]
>
> I wrote a moving correlation for time series last november (scipy user
> and preceding discussion on numpy mailing list)
> I don't work with pictures, so I don't know if this can be extended to
> your case. Since signal.correlate or convolve work in all directions
> it might be possible
>
>     def yxcov(self):
>         xys = signal.correlate(self.x*self.y, self.kern,
> mode='same')[self.sslice]
>         return xys/self.n - self.ymean*self.xmean
>
> Josef

I saw that when searching on this topic, but didn't think it would work 
for me as I supposed it was purely 1-dimensional, and I thought that in 
your implementation, though the window moves, the kernel is the same all 
the time? I'm no signal processing pro (alas) so please correct me if 
I'm wrong. I'll try to find the discussion you mentioned tomorrow. Damn 
time zones... ;-)
>
>>>
>>> Thanks,
>>> Vincent.
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>




From josef.pktd at gmail.com  Wed Feb 10 14:48:23 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 10 Feb 2010 14:48:23 -0500
Subject: [SciPy-User] moving window (2D) correlation coefficient
In-Reply-To: <hkv1sb$8ut$1@ger.gmane.org>
References: <hkukb3$k0h$1@ger.gmane.org>
	<0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>
	<1cd32cbb1002100853t43bca77cxb93ca1242451204b@mail.gmail.com>
	<hkv1sb$8ut$1@ger.gmane.org>
Message-ID: <1cd32cbb1002101148x594cfa6t5ded16b26948eb7c@mail.gmail.com>

On Wed, Feb 10, 2010 at 2:36 PM, Vincent Schut <schut at sarvision.nl> wrote:
> On 02/10/2010 05:53 PM, josef.pktd at gmail.com wrote:
>> On Wed, Feb 10, 2010 at 11:42 AM, Zachary Pincus
>> <zachary.pincus at yale.edu> ?wrote:
>>> I bet that you could construct an array with shape (x,y,5,5), where
>>> array[i,j,:,:] would give the 5x5 window around (i,j), as some sort of
>>> mind-bending view on an array of shape (x,y), using a positive offset
>>> and some dimensions having negative strides. Then you could compute
>>> the correlation coefficient between the two arrays directly. Maybe?
>>>
>>> Probably cython would be more maintainable...
>>>
>>> Zach
>>>
>>>
>>> On Feb 10, 2010, at 10:45 AM, Vincent Schut wrote:
>>>
>>>> Hi,
>>>>
>>>> I need to calculate the correlation coefficient of a (simultaneously)
>>>> moving window over 2 images, such that the resulting pixel x,y (center
>>>> of the window) is the corrcoef((a 5x5 window around x,y for image
>>>> A), (a
>>>> 5x5 window around x,y for image B)).
>>>> Currently, I just loop over y, x, and then call corrcoef for each x,y
>>>> window. Would there be a better way, other than converting the loop to
>>>> cython?
>>>>
>>>>
>>>> For clarity (or not), the relevant part from my code:
>>>>
>>>>
>>>> for y in range(d500shape[2]):
>>>> ? ? ?for x in range(d500shape[3]):
>>>> ? ? ? ? ?if valid500[d,y,x]:
>>>> ? ? ? ? ? ? ?window = spectral500Bordered[d,:,y:y+5, x:x
>>>> +5].reshape(7, -1)
>>>> ? ? ? ? ? ? ?for b in range(5):
>>>> ? ? ? ? ? ? ? ? ?nonzeroMask = (window[0]> ?0)
>>>> ? ? ? ? ? ? ? ? ?b01corr[0,b,y,x] =
>>>> numpy.corrcoef(window[0][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>>> ? ? ? ? ? ? ? ? ?b01corr[1,b,y,x] =
>>>> numpy.corrcoef(window[1][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>>>
>>>>
>>>> forget the 'if valid500' and 'nonzeroMask', those are to prevent
>>>> calculating pixels that don't need to be calculated, and to feed only
>>>> valid pixels from the window into corrcoef
>>>> spectral500Bordered is essentially a [d dates, 7 images, ysize, xsize]
>>>> array. I work per date (d), then calculate the corrcoef for images[0]
>>>> versus images[2:], and for images[1] versus images[2:]
>>
>> I wrote a moving correlation for time series last november (scipy user
>> and preceding discussion on numpy mailing list)
>> I don't work with pictures, so I don't know if this can be extended to
>> your case. Since signal.correlate or convolve work in all directions
>> it might be possible
>>
>> ? ? def yxcov(self):
>> ? ? ? ? xys = signal.correlate(self.x*self.y, self.kern,
>> mode='same')[self.sslice]
>> ? ? ? ? return xys/self.n - self.ymean*self.xmean
>>
>> Josef
>
> I saw that when searching on this topic, but didn't think it would work
> for me as I supposed it was purely 1-dimensional, and I thought that in
> your implementation, though the window moves, the kernel is the same all
> the time? I'm no signal processing pro (alas) so please correct me if
> I'm wrong. I'll try to find the discussion you mentioned tomorrow. Damn
> time zones... ;-)

correlation is just  sum(x*y)/sqrt(sum(x*x))/sqrt(sum(y*y))   but
subtracting moving mean which makes it slightly tricky.

the moving sum can be done with any nd convolve or correlate (ndimage
or signal) which goes in all directions

the window is just np.ones((windowsize,windowsize)) for symmetric
picture, or if nan handling is not necessary than the averaging window
can be used directly.

I'm pretty sure now it works, with 5 or so intermediate arrays in the
same size as the original array

Josef



>>
>>>>
>>>> Thanks,
>>>> Vincent.
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From rob.clewley at gmail.com  Wed Feb 10 14:50:35 2010
From: rob.clewley at gmail.com (Rob Clewley)
Date: Wed, 10 Feb 2010 14:50:35 -0500
Subject: [SciPy-User] odeint python and mathematica
In-Reply-To: <1265825453.2185.44.camel@nasser-desktop>
References: <mailman.4907.1265814526.2454.scipy-user@scipy.org>
	<1265817482.2185.32.camel@nasser-desktop>
	<66939327-ACDC-4120-B978-94028F1A7655@me.com>
	<1265825453.2185.44.camel@nasser-desktop>
Message-ID: <a749952d1002101150q40a546b9v7665041102a8c8d2@mail.gmail.com>

Hi,

On Wed, Feb 10, 2010 at 1:10 PM, Nasser Mohieddin Abukhdeir
<nasser at udel.edu> wrote:
>
> while SciPy does not, although it is using compiled FORTRAN modules. Your case is also an extreme one, a system of two ODEs, where I typically deal with systems of 10^3+. The Python overhead is much less of a problem for me since all of the matrix operations are done in C-compiled code. Look into SciPy's Weave functionality, I have not had much success with it and am working on using Cython to speed-up my code and so I can easily interface with external C-code. It might seem like alot of work now, but switching from Matlab to Python (and not C) has benefited me greatly.? I think it is well worth the effort, access to VODE through scipy.integrate.ode, excellent 3D plotting using Mayavi, and wrappers exist for the SUNDIALS ODE package if you ever need a matrix-free implicit ODE solver (see PySundials).
>
> Nasser

Just FYI, if speed is really of the essence, but you still want the
ease of setting up and/or changing the equations easily or adding
fancy functionality (e.g. accurate event detection) without hard
coding stuff yourself in C, my PyDSTool software automatically writes
out C code from your textual specifications and links it to
well-established ODE integrators (Dopri and Radau) on any platform.
This is a very fast way to solve regular, stiff, and implicit ODEs.
You even have the option to delay the final compilation and linking
and make your own bespoke changes to the auto-produced C code so that
you can add special features or tweak the code however you like. So
you get *all* of the speedup from using C code with none of the
hassle, IMHO :)  Also, I concur that Mayavi is certainly looking great
as a 3D plotting system.


You'll find more information at pydstool.sourceforge.net. Of course,
the whole environment is built over native python classes and numpy.

Best,
Rob


From zachary.pincus at yale.edu  Wed Feb 10 15:02:22 2010
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 10 Feb 2010 15:02:22 -0500
Subject: [SciPy-User] moving window (2D) correlation coefficient
In-Reply-To: <hkv1i3$7a3$1@ger.gmane.org>
References: <hkukb3$k0h$1@ger.gmane.org>
	<0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>
	<hkv1i3$7a3$1@ger.gmane.org>
Message-ID: <B7886502-AFF9-4DD0-AE76-059CC62C5B31@yale.edu>

> I tried that, but then less clever (just reshaping and transposing);  
> it
> made a copy, and I ended up with an array that took 5*5 times as much
> space as my initial array... And it took a lot of time.
> But I don't see how this reshape would gain me much speed?

Well, the feindishly-clever view (which might not even be possible, I  
haven't thought through it super-clearly) would at least use no more  
memory than the original array.

>  Then you could compute
>> the correlation coefficient between the two arrays directly. Maybe?
>
> Ah, like that. I thought numpy.corrcoef(x,y) only worked on flat x and
> flat y. There is no axis keyword... Is there another function that  
> would
> work on nd arrays?

I'd follow Joseph's lead here and implement your own corrcoef that (in  
the above case) works on (x,y,n,n)-shape arrays, or, in the cython  
case, is just a cdef function that computes it directly.

> Well, speed is more important than readability this time :-) It's
> terabytes of data I'll need to push through this function...

OK, then cython is likely preferable here. Just make sure to properly  
declare the array and the index variables, e.g.:
cdef np.ndarray[np.float32_t, ndim=2, negative_indices=False,  
mode='c'] a = np.asarray(a_in, dtype=np.float32, order='C')
(or could use fortran-order if that's what the arrays are natively)
and make sure after debugging to @cython.boundscheck(False) the  
function.

With these steps, your indexing will be maximally fast.

> What I understood till now (just starting to look at cython) is that  
> in
> this kind of thing, the repeated array sub-indexing (and its bounds
> checking etc.) is the main bottleneck. I presume the corrcoef function
> is pretty much optimized? So I thought if I could do the loop in  
> cython
> and implement some clever code for the window indexing (only replace  
> the
> new elements), I'd relatively simple get the major speed gain. Does  
> that
> sound correct?
> Only still have to find out now how to call the corrcoef function from
> cython...

Definitely implement your own corrcoef in cython in this case. Or just  
inline it in the inner loop of the function you're writing. The killer  
overhead will be in calling a python function on every pixel and  
converting the arguments and return value otherwise.

Zach





From david_baddeley at yahoo.com.au  Wed Feb 10 16:24:53 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Wed, 10 Feb 2010 13:24:53 -0800 (PST)
Subject: [SciPy-User] moving window (2D) correlation coefficient
In-Reply-To: <1cd32cbb1002101148x594cfa6t5ded16b26948eb7c@mail.gmail.com>
References: <hkukb3$k0h$1@ger.gmane.org>
	<0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>
	<1cd32cbb1002100853t43bca77cxb93ca1242451204b@mail.gmail.com>
	<hkv1sb$8ut$1@ger.gmane.org>
	<1cd32cbb1002101148x594cfa6t5ded16b26948eb7c@mail.gmail.com>
Message-ID: <734153.26153.qm@web33002.mail.mud.yahoo.com>

I think Josef's nailed it - if you can live with subtracting a moving mean from each pixel [calculated using a correlation/convolution with (1.0/(windowsize*windowsize))*ones((windowsize, windowsize))] rather than taking the mean across the actual window used for the correlation, you'll be home free. Not only do you save yourself the loops, but you're also going to be doing significantly less multiplications than if you calculated the correlation separately for each window. 

I suspect that this approach would even be considerably faster than coding your loops and the windowed correlation up in cython.

cheers,
David


----- Original Message ----
From: "josef.pktd at gmail.com" <josef.pktd at gmail.com>
To: SciPy Users List <scipy-user at scipy.org>
Sent: Thu, 11 February, 2010 8:48:23 AM
Subject: Re: [SciPy-User] moving window (2D) correlation coefficient

On Wed, Feb 10, 2010 at 2:36 PM, Vincent Schut <schut at sarvision.nl> wrote:
> On 02/10/2010 05:53 PM, josef.pktd at gmail.com wrote:
>> On Wed, Feb 10, 2010 at 11:42 AM, Zachary Pincus
>> <zachary.pincus at yale.edu>  wrote:
>>> I bet that you could construct an array with shape (x,y,5,5), where
>>> array[i,j,:,:] would give the 5x5 window around (i,j), as some sort of
>>> mind-bending view on an array of shape (x,y), using a positive offset
>>> and some dimensions having negative strides. Then you could compute
>>> the correlation coefficient between the two arrays directly. Maybe?
>>>
>>> Probably cython would be more maintainable...
>>>
>>> Zach
>>>
>>>
>>> On Feb 10, 2010, at 10:45 AM, Vincent Schut wrote:
>>>
>>>> Hi,
>>>>
>>>> I need to calculate the correlation coefficient of a (simultaneously)
>>>> moving window over 2 images, such that the resulting pixel x,y (center
>>>> of the window) is the corrcoef((a 5x5 window around x,y for image
>>>> A), (a
>>>> 5x5 window around x,y for image B)).
>>>> Currently, I just loop over y, x, and then call corrcoef for each x,y
>>>> window. Would there be a better way, other than converting the loop to
>>>> cython?
>>>>
>>>>
>>>> For clarity (or not), the relevant part from my code:
>>>>
>>>>
>>>> for y in range(d500shape[2]):
>>>>      for x in range(d500shape[3]):
>>>>          if valid500[d,y,x]:
>>>>              window = spectral500Bordered[d,:,y:y+5, x:x
>>>> +5].reshape(7, -1)
>>>>              for b in range(5):
>>>>                  nonzeroMask = (window[0]>  0)
>>>>                  b01corr[0,b,y,x] =
>>>> numpy.corrcoef(window[0][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>>>                  b01corr[1,b,y,x] =
>>>> numpy.corrcoef(window[1][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>>>
>>>>
>>>> forget the 'if valid500' and 'nonzeroMask', those are to prevent
>>>> calculating pixels that don't need to be calculated, and to feed only
>>>> valid pixels from the window into corrcoef
>>>> spectral500Bordered is essentially a [d dates, 7 images, ysize, xsize]
>>>> array. I work per date (d), then calculate the corrcoef for images[0]
>>>> versus images[2:], and for images[1] versus images[2:]
>>
>> I wrote a moving correlation for time series last november (scipy user
>> and preceding discussion on numpy mailing list)
>> I don't work with pictures, so I don't know if this can be extended to
>> your case. Since signal.correlate or convolve work in all directions
>> it might be possible
>>
>>     def yxcov(self):
>>         xys = signal.correlate(self.x*self.y, self.kern,
>> mode='same')[self.sslice]
>>         return xys/self.n - self.ymean*self.xmean
>>
>> Josef
>
> I saw that when searching on this topic, but didn't think it would work
> for me as I supposed it was purely 1-dimensional, and I thought that in
> your implementation, though the window moves, the kernel is the same all
> the time? I'm no signal processing pro (alas) so please correct me if
> I'm wrong. I'll try to find the discussion you mentioned tomorrow. Damn
> time zones... ;-)

correlation is just  sum(x*y)/sqrt(sum(x*x))/sqrt(sum(y*y))   but
subtracting moving mean which makes it slightly tricky.

the moving sum can be done with any nd convolve or correlate (ndimage
or signal) which goes in all directions

the window is just np.ones((windowsize,windowsize)) for symmetric
picture, or if nan handling is not necessary than the averaging window
can be used directly.

I'm pretty sure now it works, with 5 or so intermediate arrays in the
same size as the original array

Josef



>>
>>>>
>>>> Thanks,
>>>> Vincent.
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From mcmcclur at unca.edu  Wed Feb 10 16:52:04 2010
From: mcmcclur at unca.edu (Mark McClure)
Date: Wed, 10 Feb 2010 16:52:04 -0500
Subject: [SciPy-User] odeint (python vs mathematica)
In-Reply-To: <18B95518-6CA6-4C5F-BB57-3BBECC863C0B@me.com>
References: <18B95518-6CA6-4C5F-BB57-3BBECC863C0B@me.com>
Message-ID: <7414ba0d1002101352l2027ae0ve00504dd572d14f@mail.gmail.com>

On Wed, Feb 10, 2010 at 10:08 AM, Mark P <m.perrin at me.com> wrote:
> I have been performing modelling of ionic currents in Mathematica for the
> past three years as a student. ... I have been looking at - the rabbit/fox
> model (?Lotka-Volterra Tutorial), that is helping me to understand odeint.
>
> ... SciPy and Mathematica ODE code snipped ...
>
> My question is about speed. When computed in Mathematica 7 the code is
> roughly about 10x faster than odeint ... But I know
> from the web that numpy/scipy can be of an equivalent or faster speed. I was
> wondering whether the solution can be sped up; and would that require the
> use of C or Fortran.

As I understand it, claims (typically correct) that Python is several
times faster than Mathematica or Matlab, this is usually in reference
to procedural code.  My experience indeed confirms that complicated,
nested loops in Python usually run much faster than equivalent
Mathematica or Matlab code.  The code you present, however, simply
calls a library function, quite possibly the same library in both
cases.  The speed difference is probably caused by default parameters
(that you could likely fiddle with) and/or the quality of the
optimization of the library at compile time (that you probably can't
fiddle with).

Mark McClure


From nolambar at gmail.com  Wed Feb 10 18:44:23 2010
From: nolambar at gmail.com (Ignacio Vergara)
Date: Wed, 10 Feb 2010 20:44:23 -0300
Subject: [SciPy-User] SWIG + numpy and data types
Message-ID: <10f88f6d1002101544x18c5b97xf37a1e359ea19612@mail.gmail.com>

Hi

I hope this is the correct mailing list for this.

I'm using SWIG to create a module for Python+numpy for a legacy and highly
optimized C code.

The function takes 2 array arguments (different sizes) and returns a third
array. But I've been unable to define the interface file.

The function definition is

void cLandscape(int N, int* interactions, float* contributions, float*
landscape);

where interactions is a N*(N+1) array, contributions is another array of
length 2^(K+1) [0 <= K <= N-1] and landscape is the returned array of length
2^N. So N is the only relevant length.

The function loops over 2^N to fill the landscape array.

The interface typemap line looks like this

%apply (int DIM1, int* IN_ARRAY1, float* IN_ARRAY1, float* ARGOUT_ARRAY1)
{(int N, int* interactions, float* contributions, float* landscape)}

Thanks in advance for any tip and help ;)

Ignacio Vergara Kausel
Bachelor in Science, Physics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100210/e0cf7a5f/attachment.html>

From rpg.314 at gmail.com  Wed Feb 10 23:20:24 2010
From: rpg.314 at gmail.com (Rohit Garg)
Date: Thu, 11 Feb 2010 09:50:24 +0530
Subject: [SciPy-User] SWIG + numpy and data types
In-Reply-To: <10f88f6d1002101544x18c5b97xf37a1e359ea19612@mail.gmail.com>
References: <10f88f6d1002101544x18c5b97xf37a1e359ea19612@mail.gmail.com>
Message-ID: <4d5dd8c21002102020x2e47d8ccs24de9de200d65e8@mail.gmail.com>

AFAIK, this will not be possible without some serious gymnastics. The
typemaps available in SWIG's numpy.i allow for both input and output
arrays to be the same size only.

One easy way (a hack, TBH)  to achieve this to make a struct with int
N, int* interactions, float* contributions, float* landscape as
members and define functions on that struct which pass in numpy arrays
corresponding to  interactions and contributions arrays and which just
copy this data into the struct.

Then you can define another function over this struct in C which is a
proxy to call your highly optimized function.

One last function could be one which just passes in the float
*landscape and copies the data back into numpy array.

I admit I have not figured out how to use the SWIG's support for
buffer protocol yet. It is supposed to be very clean and meant for
tasks like this only. If someone could point out a simple example, I'd
be very interested in having a look.

On Thu, Feb 11, 2010 at 5:14 AM, Ignacio Vergara <nolambar at gmail.com> wrote:
>
> Hi
> I hope this is the correct mailing list for this.
> I'm using SWIG to create a module for Python+numpy for a legacy and highly optimized C code.
> The function takes 2 array arguments (different sizes) and returns a third array. But I've been unable to define the interface file.
> The function definition is
> void cLandscape(int N, int* interactions, float* contributions, float* landscape);
> where interactions is a N*(N+1) array, contributions is another array of length 2^(K+1) [0 <= K <= N-1] and landscape is the returned array of length 2^N. So N is the only relevant length.
> The function loops over 2^N to fill the landscape array.
> The interface typemap line looks like this
> %apply (int DIM1, int* IN_ARRAY1, float* IN_ARRAY1, float* ARGOUT_ARRAY1) {(int N, int* interactions, float* contributions, float* landscape)}
> Thanks in advance for any tip and help ;)
> Ignacio Vergara Kausel
> Bachelor in Science, Physics
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



--
Rohit Garg

http://rpg-314.blogspot.com/

Senior Undergraduate
Department of Physics
Indian Institute of Technology
Bombay


From stefan at sun.ac.za  Thu Feb 11 01:35:55 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 11 Feb 2010 08:35:55 +0200
Subject: [SciPy-User] moving window (2D) correlation coefficient
In-Reply-To: <B7886502-AFF9-4DD0-AE76-059CC62C5B31@yale.edu>
References: <hkukb3$k0h$1@ger.gmane.org>
	<0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu> 
	<hkv1i3$7a3$1@ger.gmane.org>
	<B7886502-AFF9-4DD0-AE76-059CC62C5B31@yale.edu>
Message-ID: <9457e7c81002102235wbbb29a0u3d6023a410486fbd@mail.gmail.com>

On 10 February 2010 22:02, Zachary Pincus <zachary.pincus at yale.edu> wrote:
>> Well, speed is more important than readability this time :-) It's
>> terabytes of data I'll need to push through this function...
>
> OK, then cython is likely preferable here.

I'd say the Cython version is bound to be more readable than any
vectorised form one could contrive.  As an example of how to calculate
correlation coefficients in Cython using Summed Area Tables, have a
look at

http://dip.sun.ac.za/~stefan/code/supreme.git/?p=stefan/supreme.git;a=blob;f=supreme/register/ncorr.pyx;hb=HEAD#l21

(sorry for the horrible URL)

Further down, the function "ncc" declares normalised cross-correlation.

Regards
St?fan


From matthieu.brucher at gmail.com  Thu Feb 11 03:21:29 2010
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 11 Feb 2010 09:21:29 +0100
Subject: [SciPy-User] SWIG + numpy and data types
In-Reply-To: <10f88f6d1002101544x18c5b97xf37a1e359ea19612@mail.gmail.com>
References: <10f88f6d1002101544x18c5b97xf37a1e359ea19612@mail.gmail.com>
Message-ID: <e76aa17f1002110021l1374c791jc82254979cb1ddbf@mail.gmail.com>

Hi,

I think your best bet is to add an additional C layer that will expose
the dimension of each array :

void cLandscape_wrap(int* interactions, int size_interaction, float*
contributions, int size_contributions, float* landscape, int
size_landscape);

and then apply the SWIG %apply on each pair. This is the easiest way
IMO to achieve your goal.

Matthieu

2010/2/11 Ignacio Vergara <nolambar at gmail.com>:
> Hi
> I hope this is the correct mailing list for this.
> I'm using SWIG to create a module for Python+numpy for a legacy and highly
> optimized C code.
> The function takes 2 array arguments (different sizes) and returns a third
> array. But I've been unable to define the interface file.
> The function definition is
> void cLandscape(int N, int* interactions, float* contributions, float*
> landscape);
> where interactions is a N*(N+1) array, contributions is another array of
> length 2^(K+1) [0 <= K <= N-1] and landscape is the returned array of length
> 2^N. So N is the only relevant length.
> The function loops over 2^N to fill the landscape array.
> The interface typemap line looks like this
> %apply (int DIM1, int* IN_ARRAY1, float* IN_ARRAY1, float* ARGOUT_ARRAY1)
> {(int N, int* interactions, float* contributions, float* landscape)}
> Thanks in advance for any tip and help ;)
> Ignacio Vergara Kausel
> Bachelor in Science, Physics
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From schut at sarvision.nl  Thu Feb 11 04:40:41 2010
From: schut at sarvision.nl (Vincent Schut)
Date: Thu, 11 Feb 2010 10:40:41 +0100
Subject: [SciPy-User] moving window (2D) correlation coefficient
In-Reply-To: <734153.26153.qm@web33002.mail.mud.yahoo.com>
References: <hkukb3$k0h$1@ger.gmane.org>	<0C0B0CFF-75DA-4A7B-870B-6D253EBD9A97@yale.edu>	<1cd32cbb1002100853t43bca77cxb93ca1242451204b@mail.gmail.com>	<hkv1sb$8ut$1@ger.gmane.org>	<1cd32cbb1002101148x594cfa6t5ded16b26948eb7c@mail.gmail.com>
	<734153.26153.qm@web33002.mail.mud.yahoo.com>
Message-ID: <hl0jao$3e9$1@ger.gmane.org>

On 02/10/2010 10:24 PM, David Baddeley wrote:
> I think Josef's nailed it - if you can live with subtracting a moving mean from each pixel [calculated using a correlation/convolution with (1.0/(windowsize*windowsize))*ones((windowsize, windowsize))] rather than taking the mean across the actual window used for the correlation, you'll be home free. Not only do you save yourself the loops, but you're also going to be doing significantly less multiplications than if you calculated the correlation separately for each window.
>
> I suspect that this approach would even be considerably faster than coding your loops and the windowed correlation up in cython.
>
> cheers,
> David
>

Guys, thanks a lot! It works well, and executing times have decreased 
with approx a factor 6. Pity I still don't get to learn cython, but I 
gained some insight. Be asured I'd buy you a beer if we lived a bit 
closer :-)

thanks again,
Vincent.

>
> ----- Original Message ----
> From: "josef.pktd at gmail.com"<josef.pktd at gmail.com>
> To: SciPy Users List<scipy-user at scipy.org>
> Sent: Thu, 11 February, 2010 8:48:23 AM
> Subject: Re: [SciPy-User] moving window (2D) correlation coefficient
>
> On Wed, Feb 10, 2010 at 2:36 PM, Vincent Schut<schut at sarvision.nl>  wrote:
>> On 02/10/2010 05:53 PM, josef.pktd at gmail.com wrote:
>>> On Wed, Feb 10, 2010 at 11:42 AM, Zachary Pincus
>>> <zachary.pincus at yale.edu>   wrote:
>>>> I bet that you could construct an array with shape (x,y,5,5), where
>>>> array[i,j,:,:] would give the 5x5 window around (i,j), as some sort of
>>>> mind-bending view on an array of shape (x,y), using a positive offset
>>>> and some dimensions having negative strides. Then you could compute
>>>> the correlation coefficient between the two arrays directly. Maybe?
>>>>
>>>> Probably cython would be more maintainable...
>>>>
>>>> Zach
>>>>
>>>>
>>>> On Feb 10, 2010, at 10:45 AM, Vincent Schut wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I need to calculate the correlation coefficient of a (simultaneously)
>>>>> moving window over 2 images, such that the resulting pixel x,y (center
>>>>> of the window) is the corrcoef((a 5x5 window around x,y for image
>>>>> A), (a
>>>>> 5x5 window around x,y for image B)).
>>>>> Currently, I just loop over y, x, and then call corrcoef for each x,y
>>>>> window. Would there be a better way, other than converting the loop to
>>>>> cython?
>>>>>
>>>>>
>>>>> For clarity (or not), the relevant part from my code:
>>>>>
>>>>>
>>>>> for y in range(d500shape[2]):
>>>>>       for x in range(d500shape[3]):
>>>>>           if valid500[d,y,x]:
>>>>>               window = spectral500Bordered[d,:,y:y+5, x:x
>>>>> +5].reshape(7, -1)
>>>>>               for b in range(5):
>>>>>                   nonzeroMask = (window[0]>   0)
>>>>>                   b01corr[0,b,y,x] =
>>>>> numpy.corrcoef(window[0][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>>>>                   b01corr[1,b,y,x] =
>>>>> numpy.corrcoef(window[1][nonzeroMask], window[b+2][nonzeroMask])[0,1]
>>>>>
>>>>>
>>>>> forget the 'if valid500' and 'nonzeroMask', those are to prevent
>>>>> calculating pixels that don't need to be calculated, and to feed only
>>>>> valid pixels from the window into corrcoef
>>>>> spectral500Bordered is essentially a [d dates, 7 images, ysize, xsize]
>>>>> array. I work per date (d), then calculate the corrcoef for images[0]
>>>>> versus images[2:], and for images[1] versus images[2:]
>>>
>>> I wrote a moving correlation for time series last november (scipy user
>>> and preceding discussion on numpy mailing list)
>>> I don't work with pictures, so I don't know if this can be extended to
>>> your case. Since signal.correlate or convolve work in all directions
>>> it might be possible
>>>
>>>      def yxcov(self):
>>>          xys = signal.correlate(self.x*self.y, self.kern,
>>> mode='same')[self.sslice]
>>>          return xys/self.n - self.ymean*self.xmean
>>>
>>> Josef
>>
>> I saw that when searching on this topic, but didn't think it would work
>> for me as I supposed it was purely 1-dimensional, and I thought that in
>> your implementation, though the window moves, the kernel is the same all
>> the time? I'm no signal processing pro (alas) so please correct me if
>> I'm wrong. I'll try to find the discussion you mentioned tomorrow. Damn
>> time zones... ;-)
>
> correlation is just  sum(x*y)/sqrt(sum(x*x))/sqrt(sum(y*y))   but
> subtracting moving mean which makes it slightly tricky.
>
> the moving sum can be done with any nd convolve or correlate (ndimage
> or signal) which goes in all directions
>
> the window is just np.ones((windowsize,windowsize)) for symmetric
> picture, or if nan handling is not necessary than the averaging window
> can be used directly.
>
> I'm pretty sure now it works, with 5 or so intermediate arrays in the
> same size as the original array
>
> Josef
>
>
>
>>>
>>>>>
>>>>> Thanks,
>>>>> Vincent.
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user




From josef.pktd at gmail.com  Thu Feb 11 09:38:46 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 11 Feb 2010 09:38:46 -0500
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <4B71A6BD.9090002@enthought.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>
	<c5b438121001281239y6b31691dq878e5ef340df96fd@mail.gmail.com>
	<4B623ECD.4040403@enthought.com>
	<1cd32cbb1001281757x14b7b581xba02f4ae21d3acfb@mail.gmail.com>
	<4B6256F5.3050802@enthought.com>
	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>
	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>
	<c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>
	<1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>
	<4B71A6BD.9090002@enthought.com>
Message-ID: <1cd32cbb1002110638kcdaae10kc34c5a451e38badb@mail.gmail.com>

On Tue, Feb 9, 2010 at 1:17 PM, Warren Weckesser
<warren.weckesser at enthought.com> wrote:
> josef.pktd at gmail.com wrote:
>> On Wed, Feb 3, 2010 at 9:00 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>
>>> FYI, I am quite happy with passing in an hmax value. ?I basically
>>> copied and pasted lsim2 from signal.ltisys and adapted it just a
>>> little to make it a method of my derived class. ?Then I added the hmas
>>> kwarg that gets passed to odeint.
>>>
>>> Is there any reason not to allow the user to pass in a kwargs to lsim2
>>> that gets passed to odeint?
>>>
>>
>> I don't see a reason why we cannot add a **kwargs, it should be
>> completely backwards compatible.
>> Can you file a ticket and add your adjusted version or a patch? And
>> even better, add your original example as a test case?
>>
>>
>
> Josef,
>
> I just created ticket #1112 for this. ?Unless Ryan wants to adapt his
> change to lsim2, I can make a patch this week to implement the enhancement.

Thanks, Stefan (I think) or I will look at it.

For statsmodels (maximum likelihood) I was looking at a second pattern
instead of **kw that keeps the options for the optimizer, or ode in
this case, separate.

It's just a taste difference. **kw is the more usual pattern in scipy.
Using a separate keyword for the options instead, keeps the signature
a bit cleaner (and I have been reading more gauss and matlab lately),
but it requires slightly more typing in the call.

Josef

def fun1a(a, opt1=3, **kw):
    #options to second function as keywords
    print '\nfun1a a',a
    print 'fun1a opt1', opt1
    print 'fun1a kw', kw
    fun2(a, **kw)
    return 'end fun1a'

def fun1b(a, opt1=3, optfun={}):
    #options to second function as separate dictionary
    print '\nfun1b a',a
    print 'fun1 opt1', opt1
    print 'fun1b optfun', optfun
    fun2(a, **optfun)
    return 'end fun1b'


def fun2(a, tol=0, maxiter=1):
    print 'fun2 a',a
    print 'fun2 kw: tol, maxiter', tol, maxiter
    return 'end fun2'

print fun1a('1a')
print fun1a('1a', tol=5, maxiter=10)
print fun1a('1a', opt1=4, tol=5, maxiter=10)
my_options = dict(tol=5, maxiter=10)
print fun1a('1a', opt1=4, **my_options)

print fun1a('1b')
print fun1b('1b', optfun=dict(tol=5, maxiter=10))
print fun1b('1b', opt1=4, optfun={ 'tol':5, 'maxiter':10})
my_options = dict(tol=5, maxiter=10)
print fun1b('1b', opt1=4, optfun=my_options)



>
> Warren
>
>
>> Josef
>>
>>
>>
>>> On Fri, Jan 29, 2010 at 6:44 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>
>>>> Thanks to Warren and Josef for their time and thoughts. ?I feel like I
>>>> now understand the underlying problem and have some good options to
>>>> solve my short term issues (I assigned the project last night and they
>>>> need to be able to start working on it immediately). ?I actually use a
>>>> TransferFunction class that derives from ltisys. ?I could override its
>>>> lsim2 method to try out some of these solutions quickly and fairly
>>>> easily.
>>>>
>>>> Ryan
>>>>
>>>> On Thu, Jan 28, 2010 at 10:00 PM, ?<josef.pktd at gmail.com> wrote:
>>>>
>>>>> On Thu, Jan 28, 2010 at 10:33 PM, Warren Weckesser
>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>
>>>>>> josef.pktd at gmail.com wrote:
>>>>>>
>>>>>>> On Thu, Jan 28, 2010 at 8:50 PM, Warren Weckesser
>>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Ryan,
>>>>>>>>
>>>>>>>> The problem is that the ODE solver used by lsim2 is too good. :)
>>>>>>>>
>>>>>>>> It uses scipy.integrate.odeint, which in turn uses the Fortran library
>>>>>>>> LSODA. ?Like any good solver, LSODA is an adaptive solver--it adjusts its
>>>>>>>> step size to be as large as possible while keeping estimates of the error
>>>>>>>> bounded. ?For the problem you are solving, with initial condition 0, the
>>>>>>>> exact solution is initially exactly 0. ?This is such a nice smooth solution
>>>>>>>> that the solver's step size quickly grows--so big, in fact, that it skips
>>>>>>>> right over your pulse and never sees it.
>>>>>>>>
>>>>>>>> So how does it create all those intermediate points at the requested time
>>>>>>>> values? ?It uses interpolation between the steps that it computed to create
>>>>>>>> the solution values at the times that you requested. ?So using a finer grid
>>>>>>>> of time values won't help. ?(If lsim2 gave you a hook into the parameters
>>>>>>>> passed to odeint, you could set odeint's 'hmax' to a value smaller than your
>>>>>>>> pulse width, which would force the solver to see the pulse. ?But there is no
>>>>>>>> way to set that parameter from lsim2.)
>>>>>>>>
>>>>>>>>
>>>>>>> It's something what I suspected. I don't know much about odeint, but
>>>>>>> do you think it would be useful to let lsim2 pass through some
>>>>>>> parameters to odeint?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Sounds useful to me. ?A simple implementation is an optional keyword
>>>>>> argument that is a dict of odeint arguments. ? But this would almost
>>>>>> certainly break if lsim2 were ever reimplemented with a different
>>>>>> solver. ?So perhaps it should allow a common set of ODE solver
>>>>>> parameters (e.g. absolute and relative error tolerances, max and min
>>>>>> step sizes, others?).
>>>>>>
>>>>>> Perhaps this should wait until after the ODE solver redesign that is
>>>>>> occasionally discussed:
>>>>>> ? ?http://projects.scipy.org/scipy/wiki/OdeintRedesign
>>>>>> Then the solver itself could be an optional argument to lsim2.
>>>>>>
>>>>> I was just thinking of adding to the argument list a **kwds argument
>>>>> that is directly passed on to whatever ODE solver is used. This should
>>>>> be pretty flexible for any changes and be backwards compatible.
>>>>>
>>>>> I've seen and used it in a similar way for calls to optimization
>>>>> routines, e.g. also optimize.curve_fit, does it. What are actually
>>>>> valid keywords would depend on which function is called.
>>>>>
>>>>> (But I'm not a user of lsim, I'm just stealing some ideas from lti and
>>>>> friends for time series analysis.)
>>>>>
>>>>> Josef
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Warren
>>>>>>
>>>>>>
>>>>>>> Josef
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> The basic problem is you are passing in a discontinuous function to a solver
>>>>>>>> that expects a smooth function. ?A better way to solve this problem is to
>>>>>>>> explicitly account for the discontinuity. One possibility is the attached
>>>>>>>> script.
>>>>>>>>
>>>>>>>> This is an excellent "learning opportunity" for your students on the hazards
>>>>>>>> of numerical computing!
>>>>>>>>
>>>>>>>> Warren
>>>>>>>>
>>>>>>>>
>>>>>>>> Ryan Krauss wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> I believe I have discovered a bug in signal.lsim2. ?I believe the
>>>>>>>>> short attached script illustrates the problem. ?I was trying to
>>>>>>>>> predict the response of a transfer function with a pure integrator:
>>>>>>>>>
>>>>>>>>> ? ? ? ? ? ? g
>>>>>>>>> G = -------------
>>>>>>>>> ? ? ? ? s(s+p)
>>>>>>>>>
>>>>>>>>> to a finite width pulse. ?lsim2 seems to handle the step response just
>>>>>>>>> fine, but says that the pulse response is exactly 0.0 for the entire
>>>>>>>>> time of the simulation. ?Obviously, this isn't the right answer.
>>>>>>>>>
>>>>>>>>> I am running scipy 0.7.0 and numpy 1.2.1 on Ubuntu 9.04, but I also
>>>>>>>>> have the same problem on Windows running 0.7.1 and 1.4.0.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Ryan
>>>>>>>>> ?------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SciPy-User mailing list
>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>> from pylab import *
>>>>>>>> from scipy import signal
>>>>>>>>
>>>>>>>>
>>>>>>>> g = 100.0
>>>>>>>> p = 15.0
>>>>>>>> G = signal.ltisys.lti(g, [1,p,0])
>>>>>>>>
>>>>>>>> t = arange(0, 1.0, 0.002)
>>>>>>>> N = len(t)
>>>>>>>>
>>>>>>>> # u for the whole interval (not used in lsim2, only for plotting later).
>>>>>>>> amp = 50.0
>>>>>>>> u = zeros(N)
>>>>>>>> k1 = 50
>>>>>>>> k2 = 100
>>>>>>>> u[k1:k2] = amp
>>>>>>>>
>>>>>>>> # Create input functions for each smooth interval. (This could be simpler,
>>>>>>>> since u
>>>>>>>> # is constant on each interval.)
>>>>>>>> a = float(k1)/N
>>>>>>>> b = float(k2)/N
>>>>>>>> T1 = linspace(0, a, 201)
>>>>>>>> u1 = zeros_like(T1)
>>>>>>>> T2 = linspace(a, b, 201)
>>>>>>>> u2 = amp*ones_like(T2)
>>>>>>>> T3 = linspace(b, 1.0, 201)
>>>>>>>> u3 = zeros_like(T3)
>>>>>>>>
>>>>>>>> # Solve on each interval; use the final value of one solution as the
>>>>>>>> starting
>>>>>>>> # point of the next solution.
>>>>>>>> # (We could skip the first calculation, since we know the solution will be
>>>>>>>> 0.)
>>>>>>>> (t1, y1, x1) = signal.lsim2(G,u1,T1)
>>>>>>>> (t2, y2, x2) = signal.lsim2(G, u2, T2, X0=x1[-1])
>>>>>>>> (t3, y3, x3) = signal.lsim2(G, u3, T3, X0=x2[-1])
>>>>>>>>
>>>>>>>> figure(1)
>>>>>>>> clf()
>>>>>>>> plot(t, u, 'k', linewidth=3)
>>>>>>>> plot(t1, y1, 'y', linewidth=3)
>>>>>>>> plot(t2, y2, 'b', linewidth=3)
>>>>>>>> plot(t3, y3, 'g', linewidth=3)
>>>>>>>>
>>>>>>>> show()
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SciPy-User mailing list
>>>>>>>> SciPy-User at scipy.org
>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SciPy-User mailing list
>>>>>>> SciPy-User at scipy.org
>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From ryanlists at gmail.com  Thu Feb 11 10:21:01 2010
From: ryanlists at gmail.com (Ryan Krauss)
Date: Thu, 11 Feb 2010 09:21:01 -0600
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <4B71A6BD.9090002@enthought.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>
	<c5b438121001281239y6b31691dq878e5ef340df96fd@mail.gmail.com>
	<4B623ECD.4040403@enthought.com>
	<1cd32cbb1001281757x14b7b581xba02f4ae21d3acfb@mail.gmail.com>
	<4B6256F5.3050802@enthought.com>
	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>
	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>
	<c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>
	<1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>
	<4B71A6BD.9090002@enthought.com>
Message-ID: <c5b438121002110721g66eda059ve86d199f7beed0fd@mail.gmail.com>

Yeah, **kwds is a little harder to understand and slightly less user
friendly, but it is easier to maintain and less work for the patch
writer.  And it is more future proof if the integrator ever changes.

On Tue, Feb 9, 2010 at 12:17 PM, Warren Weckesser
<warren.weckesser at enthought.com> wrote:
> josef.pktd at gmail.com wrote:
>> On Wed, Feb 3, 2010 at 9:00 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>
>>> FYI, I am quite happy with passing in an hmax value. ?I basically
>>> copied and pasted lsim2 from signal.ltisys and adapted it just a
>>> little to make it a method of my derived class. ?Then I added the hmas
>>> kwarg that gets passed to odeint.
>>>
>>> Is there any reason not to allow the user to pass in a kwargs to lsim2
>>> that gets passed to odeint?
>>>
>>
>> I don't see a reason why we cannot add a **kwargs, it should be
>> completely backwards compatible.
>> Can you file a ticket and add your adjusted version or a patch? And
>> even better, add your original example as a test case?
>>
>>
>
> Josef,
>
> I just created ticket #1112 for this. ?Unless Ryan wants to adapt his
> change to lsim2, I can make a patch this week to implement the enhancement.
>
> Warren
>
>
>> Josef
>>
>>
>>
>>> On Fri, Jan 29, 2010 at 6:44 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>
>>>> Thanks to Warren and Josef for their time and thoughts. ?I feel like I
>>>> now understand the underlying problem and have some good options to
>>>> solve my short term issues (I assigned the project last night and they
>>>> need to be able to start working on it immediately). ?I actually use a
>>>> TransferFunction class that derives from ltisys. ?I could override its
>>>> lsim2 method to try out some of these solutions quickly and fairly
>>>> easily.
>>>>
>>>> Ryan
>>>>
>>>> On Thu, Jan 28, 2010 at 10:00 PM, ?<josef.pktd at gmail.com> wrote:
>>>>
>>>>> On Thu, Jan 28, 2010 at 10:33 PM, Warren Weckesser
>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>
>>>>>> josef.pktd at gmail.com wrote:
>>>>>>
>>>>>>> On Thu, Jan 28, 2010 at 8:50 PM, Warren Weckesser
>>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Ryan,
>>>>>>>>
>>>>>>>> The problem is that the ODE solver used by lsim2 is too good. :)
>>>>>>>>
>>>>>>>> It uses scipy.integrate.odeint, which in turn uses the Fortran library
>>>>>>>> LSODA. ?Like any good solver, LSODA is an adaptive solver--it adjusts its
>>>>>>>> step size to be as large as possible while keeping estimates of the error
>>>>>>>> bounded. ?For the problem you are solving, with initial condition 0, the
>>>>>>>> exact solution is initially exactly 0. ?This is such a nice smooth solution
>>>>>>>> that the solver's step size quickly grows--so big, in fact, that it skips
>>>>>>>> right over your pulse and never sees it.
>>>>>>>>
>>>>>>>> So how does it create all those intermediate points at the requested time
>>>>>>>> values? ?It uses interpolation between the steps that it computed to create
>>>>>>>> the solution values at the times that you requested. ?So using a finer grid
>>>>>>>> of time values won't help. ?(If lsim2 gave you a hook into the parameters
>>>>>>>> passed to odeint, you could set odeint's 'hmax' to a value smaller than your
>>>>>>>> pulse width, which would force the solver to see the pulse. ?But there is no
>>>>>>>> way to set that parameter from lsim2.)
>>>>>>>>
>>>>>>>>
>>>>>>> It's something what I suspected. I don't know much about odeint, but
>>>>>>> do you think it would be useful to let lsim2 pass through some
>>>>>>> parameters to odeint?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Sounds useful to me. ?A simple implementation is an optional keyword
>>>>>> argument that is a dict of odeint arguments. ? But this would almost
>>>>>> certainly break if lsim2 were ever reimplemented with a different
>>>>>> solver. ?So perhaps it should allow a common set of ODE solver
>>>>>> parameters (e.g. absolute and relative error tolerances, max and min
>>>>>> step sizes, others?).
>>>>>>
>>>>>> Perhaps this should wait until after the ODE solver redesign that is
>>>>>> occasionally discussed:
>>>>>> ? ?http://projects.scipy.org/scipy/wiki/OdeintRedesign
>>>>>> Then the solver itself could be an optional argument to lsim2.
>>>>>>
>>>>> I was just thinking of adding to the argument list a **kwds argument
>>>>> that is directly passed on to whatever ODE solver is used. This should
>>>>> be pretty flexible for any changes and be backwards compatible.
>>>>>
>>>>> I've seen and used it in a similar way for calls to optimization
>>>>> routines, e.g. also optimize.curve_fit, does it. What are actually
>>>>> valid keywords would depend on which function is called.
>>>>>
>>>>> (But I'm not a user of lsim, I'm just stealing some ideas from lti and
>>>>> friends for time series analysis.)
>>>>>
>>>>> Josef
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Warren
>>>>>>
>>>>>>
>>>>>>> Josef
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> The basic problem is you are passing in a discontinuous function to a solver
>>>>>>>> that expects a smooth function. ?A better way to solve this problem is to
>>>>>>>> explicitly account for the discontinuity. One possibility is the attached
>>>>>>>> script.
>>>>>>>>
>>>>>>>> This is an excellent "learning opportunity" for your students on the hazards
>>>>>>>> of numerical computing!
>>>>>>>>
>>>>>>>> Warren
>>>>>>>>
>>>>>>>>
>>>>>>>> Ryan Krauss wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> I believe I have discovered a bug in signal.lsim2. ?I believe the
>>>>>>>>> short attached script illustrates the problem. ?I was trying to
>>>>>>>>> predict the response of a transfer function with a pure integrator:
>>>>>>>>>
>>>>>>>>> ? ? ? ? ? ? g
>>>>>>>>> G = -------------
>>>>>>>>> ? ? ? ? s(s+p)
>>>>>>>>>
>>>>>>>>> to a finite width pulse. ?lsim2 seems to handle the step response just
>>>>>>>>> fine, but says that the pulse response is exactly 0.0 for the entire
>>>>>>>>> time of the simulation. ?Obviously, this isn't the right answer.
>>>>>>>>>
>>>>>>>>> I am running scipy 0.7.0 and numpy 1.2.1 on Ubuntu 9.04, but I also
>>>>>>>>> have the same problem on Windows running 0.7.1 and 1.4.0.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Ryan
>>>>>>>>> ?------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SciPy-User mailing list
>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>> from pylab import *
>>>>>>>> from scipy import signal
>>>>>>>>
>>>>>>>>
>>>>>>>> g = 100.0
>>>>>>>> p = 15.0
>>>>>>>> G = signal.ltisys.lti(g, [1,p,0])
>>>>>>>>
>>>>>>>> t = arange(0, 1.0, 0.002)
>>>>>>>> N = len(t)
>>>>>>>>
>>>>>>>> # u for the whole interval (not used in lsim2, only for plotting later).
>>>>>>>> amp = 50.0
>>>>>>>> u = zeros(N)
>>>>>>>> k1 = 50
>>>>>>>> k2 = 100
>>>>>>>> u[k1:k2] = amp
>>>>>>>>
>>>>>>>> # Create input functions for each smooth interval. (This could be simpler,
>>>>>>>> since u
>>>>>>>> # is constant on each interval.)
>>>>>>>> a = float(k1)/N
>>>>>>>> b = float(k2)/N
>>>>>>>> T1 = linspace(0, a, 201)
>>>>>>>> u1 = zeros_like(T1)
>>>>>>>> T2 = linspace(a, b, 201)
>>>>>>>> u2 = amp*ones_like(T2)
>>>>>>>> T3 = linspace(b, 1.0, 201)
>>>>>>>> u3 = zeros_like(T3)
>>>>>>>>
>>>>>>>> # Solve on each interval; use the final value of one solution as the
>>>>>>>> starting
>>>>>>>> # point of the next solution.
>>>>>>>> # (We could skip the first calculation, since we know the solution will be
>>>>>>>> 0.)
>>>>>>>> (t1, y1, x1) = signal.lsim2(G,u1,T1)
>>>>>>>> (t2, y2, x2) = signal.lsim2(G, u2, T2, X0=x1[-1])
>>>>>>>> (t3, y3, x3) = signal.lsim2(G, u3, T3, X0=x2[-1])
>>>>>>>>
>>>>>>>> figure(1)
>>>>>>>> clf()
>>>>>>>> plot(t, u, 'k', linewidth=3)
>>>>>>>> plot(t1, y1, 'y', linewidth=3)
>>>>>>>> plot(t2, y2, 'b', linewidth=3)
>>>>>>>> plot(t3, y3, 'g', linewidth=3)
>>>>>>>>
>>>>>>>> show()
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SciPy-User mailing list
>>>>>>>> SciPy-User at scipy.org
>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SciPy-User mailing list
>>>>>>> SciPy-User at scipy.org
>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From amenity at enthought.com  Thu Feb 11 17:35:31 2010
From: amenity at enthought.com (Amenity Applewhite)
Date: Thu, 11 Feb 2010 16:35:31 -0600
Subject: [SciPy-User] Save the date: SciPy 2010 June 28 - July 3
Message-ID: <E0BE6677-8B2A-4B6A-B0AF-DD67F1FB123C@enthought.com>

The annual US Scientific Computing with Python Conference, SciPy, has  
been held at Caltech since it began in 2001. While we always love an  
excuse to go to California, it?s also important to make sure that we  
allow everyone an opportunity to attend the conference. So, as Jarrod  
Millman announced last fall, we?ll begin rotating the conference  
location and hold the 2010 conference in Austin, Texas.

As you may know, Enthought is headquartered in Austin. So in addition  
to our standard SciPy sponsorship, this year we?ll also be undertaking  
a great deal of the planning and organization.

To begin with, we?re thrilled to announce that we?ve secured several  
corporate sponsorships that will allow us to host the conference at  
the brand new AT&T Executive Education and Conference Center on campus  
at the University of Texas. It?s a wonderful facility in Central  
Austin and provides easy access to an array of great restaurants,  
parks, and music venues.

We will also be able to provide stipends for our Tutorial presenters  
for the first time. These community members provide us with an  
invaluable learning experience every year, and we?re very excited to  
be able to compensate them for their efforts. And of course, we?ll  
also continue to offer student sponsorships to support active academic  
contributors who want to attend the conference.

So mark your calendars, folks! June 28 ? July 3. Early registration  
open now.

Thanks,
The Enthought Team
--
Amenity Applewhite
Enthought, Inc.
Scientific Computing Solutions
www.enthought.com









-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100211/65479331/attachment.html>

From meesters at gmx.de  Fri Feb 12 04:41:35 2010
From: meesters at gmx.de (Christian Meesters)
Date: Fri, 12 Feb 2010 10:41:35 +0100
Subject: [SciPy-User] swig and OpenMP
Message-ID: <1265967695.2649.3.camel@meesters>

Hi,

Sorry for being a bit off-topic, but I thought somebody here might have
the relevant info for me.

A while ago I already asked about integrating OpenMP-code in C/C++-code
to be wrapped for python on the swig mailing list. Unfortunately nobody
there knew an answer to my problem.

Please find a minimal example in this pastebin:
pi_test.h : http://paste.pocoo.org/show/174174/ 
pi_test.c: http://paste.pocoo.org/show/174175/ 
pi_test.i: http://paste.pocoo.org/show/174177/ 
and my setup.py: http://paste.pocoo.org/show/175890/

When I do
$ python setup.py build

I get no compiler warnings or error - except for the unused variables.

But when importing the module I get this as a Traceback:

ImportError: ./pi_test.so: undefined symbol: GOMP_parallel_end

So, apparently OpenMP did not make it correctly into the wrapped code.

Any pointers for me? What's going wrong here?
Does anyone know a working example for OpenMP-code with SWIG (or
Cython)?

For the record: I'm using python 2.6.4, swig 1.3.36, and gcc 4.4.1.

TIA
Christian




From tatiana.alchueyr at gmail.com  Fri Feb 12 05:17:42 2010
From: tatiana.alchueyr at gmail.com (Tatiana Al-Chueyr)
Date: Fri, 12 Feb 2010 08:17:42 -0200
Subject: [SciPy-User] Open source 3D medical imaging software
Message-ID: <e4b8e4a61002120217v5c3a1c2nb97639e3d86555dc@mail.gmail.com>

Dear All,

I'd like to announce the release of InVesalius 3.0 Beta 1, an open
source 3D medical imaging reconstruction software - available under
GNU GPL 2.

We've used  PIL, VTK, wxPython, among other technologies. Screenshots are
available in the Trac [1].

InVesalius is currently available for Windows and GNU Linux, both 32
bits and 64 bits (soon for MacOS X). Also, it has already been
translated to English, Spanish, Portuguese and French.

Read all about the release [2] and download [3] it.

Any feedbak is welcome!

Best regards,

Tatiana

[1] http://svn.softwarepublico.gov.br/trac/invesalius
[2] http://svn.softwarepublico.gov.br/trac/invesalius/wiki/releases/3.0-beta-1
[3] http://svn.softwarepublico.gov.br/trac/invesalius/wiki/downloads/3.0-beta-1


From cimrman3 at ntc.zcu.cz  Fri Feb 12 05:30:00 2010
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Fri, 12 Feb 2010 11:30:00 +0100
Subject: [SciPy-User] Open source 3D medical imaging software
In-Reply-To: <e4b8e4a61002120217v5c3a1c2nb97639e3d86555dc@mail.gmail.com>
References: <e4b8e4a61002120217v5c3a1c2nb97639e3d86555dc@mail.gmail.com>
Message-ID: <4B752DA8.7050904@ntc.zcu.cz>

Hi Tatiana!

Tatiana Al-Chueyr wrote:
> Dear All,
> 
> I'd like to announce the release of InVesalius 3.0 Beta 1, an open
> source 3D medical imaging reconstruction software - available under
> GNU GPL 2.
> 
> We've used  PIL, VTK, wxPython, among other technologies. Screenshots are
> available in the Trac [1].
> 
> InVesalius is currently available for Windows and GNU Linux, both 32
> bits and 64 bits (soon for MacOS X). Also, it has already been
> translated to English, Spanish, Portuguese and French.
> 
> Read all about the release [2] and download [3] it.
> 
> Any feedbak is welcome!
> 
> Best regards,
> 
> Tatiana
> 
> [1] http://svn.softwarepublico.gov.br/trac/invesalius
> [2] http://svn.softwarepublico.gov.br/trac/invesalius/wiki/releases/3.0-beta-1
> [3] http://svn.softwarepublico.gov.br/trac/invesalius/wiki/downloads/3.0-beta-1

This is really cool news! I am sure some of my colleagues working with such 
images will be eager to use your software.

cheers & thanks for the release!
r.


From tatiana.alchueyr at gmail.com  Fri Feb 12 06:28:52 2010
From: tatiana.alchueyr at gmail.com (Tatiana Al-Chueyr)
Date: Fri, 12 Feb 2010 09:28:52 -0200
Subject: [SciPy-User] Open source 3D medical imaging software
In-Reply-To: <4B752DA8.7050904@ntc.zcu.cz>
References: <e4b8e4a61002120217v5c3a1c2nb97639e3d86555dc@mail.gmail.com> 
	<4B752DA8.7050904@ntc.zcu.cz>
Message-ID: <e4b8e4a61002120328x3e7c47bbg9f82730adc698d73@mail.gmail.com>

Hi Robert!

On 12 February 2010 08:30, Robert Cimrman <cimrman3 at ntc.zcu.cz> wrote:
> Hi Tatiana!
>
> Tatiana Al-Chueyr wrote:
>> Dear All,
>>
>> I'd like to announce the release of InVesalius 3.0 Beta 1, an open
>> source 3D medical imaging reconstruction software - available under
>> GNU GPL 2.
>>
>> We've used ?PIL, VTK, wxPython, among other technologies. Screenshots are
>> available in the Trac [1].
>>
>> InVesalius is currently available for Windows and GNU Linux, both 32
>> bits and 64 bits (soon for MacOS X). Also, it has already been
>> translated to English, Spanish, Portuguese and French.
>>
>> Read all about the release [2] and download [3] it.
>>
>> Any feedbak is welcome!
>>
>> Best regards,
>>
>> Tatiana
>>
>> [1] http://svn.softwarepublico.gov.br/trac/invesalius
>> [2] http://svn.softwarepublico.gov.br/trac/invesalius/wiki/releases/3.0-beta-1
>> [3] http://svn.softwarepublico.gov.br/trac/invesalius/wiki/downloads/3.0-beta-1
>
> This is really cool news! I am sure some of my colleagues working with such
> images will be eager to use your software.
>
> cheers & thanks for the release!

Thanks!
Just a detail... We used NumPy on InVesalius development... ;)

-- 
Tatiana Al-Chueyr
Sent from Campinas, SP, Brazil


From wizzard028wise at gmail.com  Fri Feb 12 08:58:48 2010
From: wizzard028wise at gmail.com (Dorian)
Date: Fri, 12 Feb 2010 14:58:48 +0100
Subject: [SciPy-User] Open source 3D medical imaging software
In-Reply-To: <e4b8e4a61002120328x3e7c47bbg9f82730adc698d73@mail.gmail.com>
References: <e4b8e4a61002120217v5c3a1c2nb97639e3d86555dc@mail.gmail.com>
	<4B752DA8.7050904@ntc.zcu.cz>
	<e4b8e4a61002120328x3e7c47bbg9f82730adc698d73@mail.gmail.com>
Message-ID: <674a602a1002120558u4816dfb1p41b3c47840af94de@mail.gmail.com>

Hi  Tatiana,
Nice and good job from the screen-shot.
I was excited to go on the website you have indicated to download
"InVesalius".
After struggling with the language on this website ( Spanish or Portuguese )
, now I'm registered
for the forum. But until now I can't find  documentation or tutorial in
English
"How to use InVesalius"  step by step .

Could you point me one ?

Thanks in advance

Dorian  Ng.






Is there any documentation or tutorial in English to

On Fri, Feb 12, 2010 at 12:28 PM, Tatiana Al-Chueyr <
tatiana.alchueyr at gmail.com> wrote:

> Hi Robert!
>
> On 12 February 2010 08:30, Robert Cimrman <cimrman3 at ntc.zcu.cz> wrote:
> > Hi Tatiana!
> >
> > Tatiana Al-Chueyr wrote:
> >> Dear All,
> >>
> >> I'd like to announce the release of InVesalius 3.0 Beta 1, an open
> >> source 3D medical imaging reconstruction software - available under
> >> GNU GPL 2.
> >>
> >> We've used  PIL, VTK, wxPython, among other technologies. Screenshots
> are
> >> available in the Trac [1].
> >>
> >> InVesalius is currently available for Windows and GNU Linux, both 32
> >> bits and 64 bits (soon for MacOS X). Also, it has already been
> >> translated to English, Spanish, Portuguese and French.
> >>
> >> Read all about the release [2] and download [3] it.
> >>
> >> Any feedbak is welcome!
> >>
> >> Best regards,
> >>
> >> Tatiana
> >>
> >> [1] http://svn.softwarepublico.gov.br/trac/invesalius
> >> [2]
> http://svn.softwarepublico.gov.br/trac/invesalius/wiki/releases/3.0-beta-1
> >> [3]
> http://svn.softwarepublico.gov.br/trac/invesalius/wiki/downloads/3.0-beta-1
> >
> > This is really cool news! I am sure some of my colleagues working with
> such
> > images will be eager to use your software.
> >
> > cheers & thanks for the release!
>
> Thanks!
> Just a detail... We used NumPy on InVesalius development... ;)
>
> --
> Tatiana Al-Chueyr
> Sent from Campinas, SP, Brazil
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100212/b33f0c62/attachment.html>

From rpg.314 at gmail.com  Fri Feb 12 09:04:00 2010
From: rpg.314 at gmail.com (Rohit Garg)
Date: Fri, 12 Feb 2010 19:34:00 +0530
Subject: [SciPy-User] swig and OpenMP
In-Reply-To: <1265967695.2649.3.camel@meesters>
References: <1265967695.2649.3.camel@meesters>
Message-ID: <4d5dd8c21002120604w762d5905l2b642b1415f7abba@mail.gmail.com>

It's another symptom of a much bigger disease.

Whenever you ask your compiler (atleast GCC) to instrument your code,
certain symbols are added. These symbols are
undocumented/subject-to-change.

Whenever you try to import such a module into python, the dynamic
library loader will complain about these symbols.

For instance, if you try to  compile your extension module with
Profile guided optimization, you'll land into the same problems.

The only reasonable fix I see for these kinds of problems is to embed
Python instead of extending it. Unfortunately, it also seems to be a
lot less popular/easy/common.

On Fri, Feb 12, 2010 at 3:11 PM, Christian Meesters <meesters at gmx.de> wrote:
> Hi,
>
> Sorry for being a bit off-topic, but I thought somebody here might have
> the relevant info for me.
>
> A while ago I already asked about integrating OpenMP-code in C/C++-code
> to be wrapped for python on the swig mailing list. Unfortunately nobody
> there knew an answer to my problem.
>
> Please find a minimal example in this pastebin:
> pi_test.h : http://paste.pocoo.org/show/174174/
> pi_test.c: http://paste.pocoo.org/show/174175/
> pi_test.i: http://paste.pocoo.org/show/174177/
> and my setup.py: http://paste.pocoo.org/show/175890/
>
> When I do
> $ python setup.py build
>
> I get no compiler warnings or error - except for the unused variables.
>
> But when importing the module I get this as a Traceback:
>
> ImportError: ./pi_test.so: undefined symbol: GOMP_parallel_end
>
> So, apparently OpenMP did not make it correctly into the wrapped code.
>
> Any pointers for me? What's going wrong here?
> Does anyone know a working example for OpenMP-code with SWIG (or
> Cython)?
>
> For the record: I'm using python 2.6.4, swig 1.3.36, and gcc 4.4.1.
>
> TIA
> Christian
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Rohit Garg

http://rpg-314.blogspot.com/

Senior Undergraduate
Department of Physics
Indian Institute of Technology
Bombay


From dagss at student.matnat.uio.no  Fri Feb 12 10:15:30 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Feb 2010 16:15:30 +0100
Subject: [SciPy-User] swig and OpenMP
In-Reply-To: <4d5dd8c21002120604w762d5905l2b642b1415f7abba@mail.gmail.com>
References: <1265967695.2649.3.camel@meesters>
	<4d5dd8c21002120604w762d5905l2b642b1415f7abba@mail.gmail.com>
Message-ID: <4B757092.1050709@student.matnat.uio.no>

Rohit Garg wrote:
> It's another symptom of a much bigger disease.
>
> Whenever you ask your compiler (atleast GCC) to instrument your code,
> certain symbols are added. These symbols are
> undocumented/subject-to-change.
>
> Whenever you try to import such a module into python, the dynamic
> library loader will complain about these symbols.
>
> For instance, if you try to  compile your extension module with
> Profile guided optimization, you'll land into the same problems.
>
> The only reasonable fix I see for these kinds of problems is to embed
> Python instead of extending it. Unfortunately, it also seems to be a
> lot less popular/easy/common.
>   
In some situations, it might help to simply call sys.setdlopenflags though:

import sys
import ctypes

try:
    _old_rtld = sys.getdlopenflags()
    sys.setdlopenflags(_old_rtld|ctypes.RTLD_GLOBAL)
    import yourmod
finally:
    sys.setdlopenflags(_old_rtld)



Dag Sverre


From dagss at student.matnat.uio.no  Fri Feb 12 10:18:55 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Feb 2010 16:18:55 +0100
Subject: [SciPy-User] swig and OpenMP
In-Reply-To: <1265967695.2649.3.camel@meesters>
References: <1265967695.2649.3.camel@meesters>
Message-ID: <4B75715F.8060904@student.matnat.uio.no>

Christian Meesters wrote:
> Hi,
>
> Sorry for being a bit off-topic, but I thought somebody here might have
> the relevant info for me.
>
> A while ago I already asked about integrating OpenMP-code in C/C++-code
> to be wrapped for python on the swig mailing list. Unfortunately nobody
> there knew an answer to my problem.
>
> Please find a minimal example in this pastebin:
> pi_test.h : http://paste.pocoo.org/show/174174/ 
> pi_test.c: http://paste.pocoo.org/show/174175/ 
> pi_test.i: http://paste.pocoo.org/show/174177/ 
> and my setup.py: http://paste.pocoo.org/show/175890/
>
> When I do
> $ python setup.py build
>
> I get no compiler warnings or error - except for the unused variables.
>
> But when importing the module I get this as a Traceback:
>
> ImportError: ./pi_test.so: undefined symbol: GOMP_parallel_end
>
> So, apparently OpenMP did not make it correctly into the wrapped code.
>
> Any pointers for me? What's going wrong here?
> Does anyone know a working example for OpenMP-code with SWIG (or
> Cython)?
>   

Hmm...you have extra_compile_args=-fopenmp, but not extra_link_args. 
Perhaps the module is only compiled, but not linked, with -fopenmp? I'd 
try to add extra_link_args as well. Inspect the build messages and make 
sure that the OpenMP libraries get linked in.

Dag Sverre


From y.copin at ipnl.in2p3.fr  Fri Feb 12 09:59:34 2010
From: y.copin at ipnl.in2p3.fr (Yannick Copin)
Date: Fri, 12 Feb 2010 15:59:34 +0100
Subject: [SciPy-User] Convert a time-frequency array to sound file
Message-ID: <4B756CD6.3000407@ipnl.in2p3.fr>

Hi,

for some illustration (!) purposes, I would like to convert a time-frequency 
array (originating from a time series of spectra f(wavelegnth)) into an 
audible sound clip, e.g. a WAV file. I quickly looked at what's available 
under numpy/python (http://wiki.python.org/moin/Audio/, or Audiolab), but I 
couldn't find a single procedure to do this conversion.

Disclaimer: I don't know anything about sound processing, and this is just for 
fun, similar to what has been done there: 
http://newscenter.lbl.gov/feature-stories/2010/01/25/music-of-the-universe/

Any hints or links on how to do that in a simple way?

Cheers,
-- 
    .~.   Yannick COPIN  (o:>*  Doctus cum libro
    /V\   Institut de physique nucl?aire de Lyon
   // \\  Universit? de Lyon - CNRS-IN2P3
  /(   )\ AIM: YnCopin ICQ: 236931013
   ^`~'^  http://snovae.in2p3.fr/ycopin/




From josef.pktd at gmail.com  Fri Feb 12 11:44:04 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 12 Feb 2010 11:44:04 -0500
Subject: [SciPy-User] rpy/rpy2 for tseries and fGarch
Message-ID: <1cd32cbb1002120844o795547c0rfe6368671a8e9088@mail.gmail.com>

Does anyone have some examples how to use arma and garch from R
tseries package and/or garchFit and garchSim from the fGarch package
through either rpy or rpy2? I'm still working mostly with rpy when I
manage, because I find rpy2 even more confusing.

I'm trying to write some validation scripts for my first versions of
arma and garch, but I'm not able to feed any model or order options
with rpy without getting an RPy_RException. Using only the default
settings, which work for fGarch, is boring.

Thanks,

Josef


From david_baddeley at yahoo.com.au  Fri Feb 12 15:48:18 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Fri, 12 Feb 2010 12:48:18 -0800 (PST)
Subject: [SciPy-User] Convert a time-frequency array to sound file
In-Reply-To: <4B756CD6.3000407@ipnl.in2p3.fr>
References: <4B756CD6.3000407@ipnl.in2p3.fr>
Message-ID: <603193.62738.qm@web33004.mail.mud.yahoo.com>

You could just take the inverse fourier transform of each spectra and then just patch them end to end, but I suspect this will end up sounding pretty awful as you'd get lots of phase discontinuities at the end of each segment. A better strategy might be to generate a continuous sin wave for each frequency (with a good number of samples per segment), multiply each of these sin waves by the corresponding, interpolated, spectral amplitude, and then sum over the different frequencies.

This should be easy enough to do with the standard numpy functions.

David

 
 

----- Original Message ----
From: Yannick Copin <y.copin at ipnl.in2p3.fr>
To: scipy-user at scipy.org
Sent: Sat, 13 February, 2010 3:59:34 AM
Subject: [SciPy-User] Convert a time-frequency array to sound file

Hi,

for some illustration (!) purposes, I would like to convert a time-frequency 
array (originating from a time series of spectra f(wavelegnth)) into an 
audible sound clip, e.g. a WAV file. I quickly looked at what's available 
under numpy/python (http://wiki.python.org/moin/Audio/, or Audiolab), but I 
couldn't find a single procedure to do this conversion.

Disclaimer: I don't know anything about sound processing, and this is just for 
fun, similar to what has been done there: 
http://newscenter.lbl.gov/feature-stories/2010/01/25/music-of-the-universe/

Any hints or links on how to do that in a simple way?

Cheers,
-- 
    .~.   Yannick COPIN  (o:>*  Doctus cum libro
    /V\   Institut de physique nucl?aire de Lyon
   // \\  Universit? de Lyon - CNRS-IN2P3
  /(   )\ AIM: YnCopin ICQ: 236931013
   ^`~'^  http://snovae.in2p3.fr/ycopin/


_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From david_baddeley at yahoo.com.au  Fri Feb 12 16:09:17 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Fri, 12 Feb 2010 13:09:17 -0800 (PST)
Subject: [SciPy-User] Convert a time-frequency array to sound file
In-Reply-To: <603193.62738.qm@web33004.mail.mud.yahoo.com>
References: <4B756CD6.3000407@ipnl.in2p3.fr>
	<603193.62738.qm@web33004.mail.mud.yahoo.com>
Message-ID: <706673.87560.qm@web33005.mail.mud.yahoo.com>

On second thoughts, phase shouldn't be a problem with FT method, although amplitude discontinuities might be. If you go with the frequency synthesis method you may want to consider randomising the initial phases of each oscillator so that you don't get a big bang/crash at the beginning when everything is in phase.


----- Original Message ----
From: David Baddeley <david_baddeley at yahoo.com.au>
To: SciPy Users List <scipy-user at scipy.org>
Sent: Sat, 13 February, 2010 9:48:18 AM
Subject: Re: [SciPy-User] Convert a time-frequency array to sound file

You could just take the inverse fourier transform of each spectra and then just patch them end to end, but I suspect this will end up sounding pretty awful as you'd get lots of phase discontinuities at the end of each segment. A better strategy might be to generate a continuous sin wave for each frequency (with a good number of samples per segment), multiply each of these sin waves by the corresponding, interpolated, spectral amplitude, and then sum over the different frequencies.

This should be easy enough to do with the standard numpy functions.

David




----- Original Message ----
From: Yannick Copin <y.copin at ipnl.in2p3.fr>
To: scipy-user at scipy.org
Sent: Sat, 13 February, 2010 3:59:34 AM
Subject: [SciPy-User] Convert a time-frequency array to sound file

Hi,

for some illustration (!) purposes, I would like to convert a time-frequency 
array (originating from a time series of spectra f(wavelegnth)) into an 
audible sound clip, e.g. a WAV file. I quickly looked at what's available 
under numpy/python (http://wiki.python.org/moin/Audio/, or Audiolab), but I 
couldn't find a single procedure to do this conversion.

Disclaimer: I don't know anything about sound processing, and this is just for 
fun, similar to what has been done there: 
http://newscenter.lbl.gov/feature-stories/2010/01/25/music-of-the-universe/

Any hints or links on how to do that in a simple way?

Cheers,
-- 
    .~.   Yannick COPIN  (o:>*  Doctus cum libro
    /V\   Institut de physique nucl?aire de Lyon
   // \\  Universit? de Lyon - CNRS-IN2P3
  /(   )\ AIM: YnCopin ICQ: 236931013
   ^`~'^  http://snovae.in2p3.fr/ycopin/


_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From mdekauwe at gmail.com  Fri Feb 12 16:14:43 2010
From: mdekauwe at gmail.com (mdekauwe)
Date: Fri, 12 Feb 2010 13:14:43 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Compiling scipy on Solaris
Message-ID: <27527114.post@talk.nabble.com>


Below is the output of the build, I hope that was what you meant by log, if
it isn't and there was a specific command then please let me know. Thanks

I should probably add I have managed to get scipy 0.6.1 to build and run
without issues.


  FOUND:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/users/eow/mgdk/sun4u/lib']
    language = c

customize SunFCompiler
Could not locate executable f90
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize Gnu95FCompiler
Found executable /nerc/packages/gcc/4.2.3/5.9/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using config
compiling '_configtest.c':


compile options: '-c'
gcc: _configtest.c
gcc -pthread _configtest.o -L/users/eow/mgdk/sun4u/lib -llapack -lf77blas
-lcblas -latlas -o _configtest
ATLAS version 3.8.3 built by mgdk on Fri Jul 17 19:10:13 BST 2009:
   UNAME    : SunOS wlsn098 5.9 Generic_122300-14 sun4u sparc SUNW,A70
   INSTFLG  : -1 0 -a 1
   ARCHDEFS : -DATL_OS_SunOS -DATL_ARCH_USIII -DATL_CPUMHZ=1600 -DSUN_HR
-DATL_GAS_SPARC
   F2CDEFS  : -DAdd_ -DF77_INTEGER=int -DStringSunStyle
   CACHEEDGE: 786432
   F77      : gfortran, version GNU Fortran (GCC) 4.2.3
   F77FLAGS : -O -mcpu=ultrasparc3 -mtune=ultrasparc3 -fPIC -m32
   SMC      : gcc, version gcc (GCC) 4.2.3
   SMCFLAGS : -mcpu=ultrasparc3 -mtune=ultrasparc3 -O3 -funroll-all-loops
-fPIC -m32
   SKC      : gcc, version gcc (GCC) 4.2.3
   SKCFLAGS : -mcpu=ultrasparc3 -mtune=ultrasparc3 -O3 -funroll-all-loops
-fPIC -m32
success!
removing: _configtest.c _configtest.o _configtest
  FOUND:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/users/eow/mgdk/sun4u/lib']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]

ATLAS version 3.8.3
lapack_opt_info:
lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in /users/eow/mgdk/sun4u/lib
  NOT AVAILABLE

  NOT AVAILABLE

NOTE I snipped the full build as the mailing list wouldn't let me paste it
all, so I picked the top bit which I thought might help the most. If there
is something specific I can post that.

Thanks

-- 
View this message in context: http://old.nabble.com/Compiling-scipy-on-Solaris-tp27519446p27527114.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From jsseabold at gmail.com  Fri Feb 12 19:37:54 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Fri, 12 Feb 2010 19:37:54 -0500
Subject: [SciPy-User] asanyarray behavior with sparse matrices?
Message-ID: <c048da1c1002121637s3bfceed1r17ba0be423731e59@mail.gmail.com>

Bug or wishful thinking on my part?

from scipy import sparse
import numpy as np

X = np.random.rand(50,3)
spX = sparse.csr_matrix(X) # any sparse matrix really
newX = np.asanyarray(spX)

spX is newX
spX.ndim == newX.ndim

Cheers,

Skipper


From robert.kern at gmail.com  Fri Feb 12 19:44:04 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 12 Feb 2010 18:44:04 -0600
Subject: [SciPy-User] asanyarray behavior with sparse matrices?
In-Reply-To: <c048da1c1002121637s3bfceed1r17ba0be423731e59@mail.gmail.com>
References: <c048da1c1002121637s3bfceed1r17ba0be423731e59@mail.gmail.com>
Message-ID: <3d375d731002121644x766ca51paa5c5ccb0cd4295f@mail.gmail.com>

On Fri, Feb 12, 2010 at 18:37, Skipper Seabold <jsseabold at gmail.com> wrote:
> Bug or wishful thinking on my part?

Wishful thinking. asanyarray() leaves subclasses of ndarray alone, but
sparse matrices are not subclasses of ndarray. Not being something
that asanyarray() can recognize as a sequence (len(spX) raises a
TypeError), it interprets it as a scalar object.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From agile.aspect at gmail.com  Fri Feb 12 21:21:50 2010
From: agile.aspect at gmail.com (Agile Aspect)
Date: Fri, 12 Feb 2010 18:21:50 -0800
Subject: [SciPy-User] Open source 3D medical imaging software
In-Reply-To: <674a602a1002120558u4816dfb1p41b3c47840af94de@mail.gmail.com>
References: <e4b8e4a61002120217v5c3a1c2nb97639e3d86555dc@mail.gmail.com>
	<4B752DA8.7050904@ntc.zcu.cz>
	<e4b8e4a61002120328x3e7c47bbg9f82730adc698d73@mail.gmail.com>
	<674a602a1002120558u4816dfb1p41b3c47840af94de@mail.gmail.com>
Message-ID: <c6a6f63b1002121821w50053db0p3098e91889771e4f@mail.gmail.com>

Where's the source?

Or did you mean there are Windows and Debian binaries but no Open
Source source code?

-- 
      Enjoy global warming while it lasts.


From tatiana.alchueyr at gmail.com  Fri Feb 12 21:25:46 2010
From: tatiana.alchueyr at gmail.com (Tatiana Al-Chueyr)
Date: Sat, 13 Feb 2010 00:25:46 -0200
Subject: [SciPy-User] Open source 3D medical imaging software
In-Reply-To: <674a602a1002120558u4816dfb1p41b3c47840af94de@mail.gmail.com>
References: <e4b8e4a61002120217v5c3a1c2nb97639e3d86555dc@mail.gmail.com> 
	<4B752DA8.7050904@ntc.zcu.cz>
	<e4b8e4a61002120328x3e7c47bbg9f82730adc698d73@mail.gmail.com> 
	<674a602a1002120558u4816dfb1p41b3c47840af94de@mail.gmail.com>
Message-ID: <e4b8e4a61002121825m18b48f34k90f695a664342dfa@mail.gmail.com>

Hi Dorian,

Aswers inlined..!

On 12 February 2010 11:58, Dorian <wizzard028wise at gmail.com> wrote:
> Hi? Tatiana,
> Nice and good job from the screen-shot.
> I was excited to go on the website you have indicated to download
> "InVesalius".

:)

> After struggling with the language on this website ( Spanish or Portuguese )

Portuguese

> , now I'm registered
> for the forum. But until now I can't find? documentation or tutorial in
> English
> "How to use InVesalius"? step by step .
>

Well, up to now, we haven't written an English version yet. We have
intentions of making some videos, step by step. Would this help you
initially?

> Could you point me one ?
>

Otherwise, we could talk on Skype or some similar tool, and I could
teach you, while we don't have other materials.

> Thanks in advance
>
> Dorian? Ng.
>
>
>
>
>
>
> Is there any documentation or tutorial in English to
>
> On Fri, Feb 12, 2010 at 12:28 PM, Tatiana Al-Chueyr
> <tatiana.alchueyr at gmail.com> wrote:
>>
>> Hi Robert!
>>
>> On 12 February 2010 08:30, Robert Cimrman <cimrman3 at ntc.zcu.cz> wrote:
>> > Hi Tatiana!
>> >
>> > Tatiana Al-Chueyr wrote:
>> >> Dear All,
>> >>
>> >> I'd like to announce the release of InVesalius 3.0 Beta 1, an open
>> >> source 3D medical imaging reconstruction software - available under
>> >> GNU GPL 2.
>> >>
>> >> We've used ?PIL, VTK, wxPython, among other technologies. Screenshots
>> >> are
>> >> available in the Trac [1].
>> >>
>> >> InVesalius is currently available for Windows and GNU Linux, both 32
>> >> bits and 64 bits (soon for MacOS X). Also, it has already been
>> >> translated to English, Spanish, Portuguese and French.
>> >>
>> >> Read all about the release [2] and download [3] it.
>> >>
>> >> Any feedbak is welcome!
>> >>
>> >> Best regards,
>> >>
>> >> Tatiana
>> >>
>> >> [1] http://svn.softwarepublico.gov.br/trac/invesalius
>> >> [2]
>> >> http://svn.softwarepublico.gov.br/trac/invesalius/wiki/releases/3.0-beta-1
>> >> [3]
>> >> http://svn.softwarepublico.gov.br/trac/invesalius/wiki/downloads/3.0-beta-1
>> >
>> > This is really cool news! I am sure some of my colleagues working with
>> > such
>> > images will be eager to use your software.
>> >
>> > cheers & thanks for the release!
>>
>> Thanks!
>> Just a detail... We used NumPy on InVesalius development... ;)
>>
>> --
>> Tatiana Al-Chueyr
>> Sent from Campinas, SP, Brazil
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>



-- 
Tatiana Al-Chueyr
Sent from Campinas, SP, Brazil


From tatiana.alchueyr at gmail.com  Fri Feb 12 21:32:44 2010
From: tatiana.alchueyr at gmail.com (Tatiana Al-Chueyr)
Date: Sat, 13 Feb 2010 00:32:44 -0200
Subject: [SciPy-User] Open source 3D medical imaging software
In-Reply-To: <c6a6f63b1002121821w50053db0p3098e91889771e4f@mail.gmail.com>
References: <e4b8e4a61002120217v5c3a1c2nb97639e3d86555dc@mail.gmail.com> 
	<4B752DA8.7050904@ntc.zcu.cz>
	<e4b8e4a61002120328x3e7c47bbg9f82730adc698d73@mail.gmail.com> 
	<674a602a1002120558u4816dfb1p41b3c47840af94de@mail.gmail.com> 
	<c6a6f63b1002121821w50053db0p3098e91889771e4f@mail.gmail.com>
Message-ID: <e4b8e4a61002121832pef0364bg20bcbbbb6bad019e@mail.gmail.com>

Answers are bellow

On 13 February 2010 00:21, Agile Aspect <agile.aspect at gmail.com> wrote:
> Where's the source?
>
It is in our Trac:
http://svn.softwarepublico.gov.br/trac/invesalius/wiki

To be able to checkout and browse the source code, you only need to
register. Please, follow this instructions:
http://svn.softwarepublico.gov.br/trac/invesalius/wiki/InVesalius/Register

Let me know if you have problems.

> Or did you mean there are Windows and Debian binaries but no Open
> Source source code?

I really meant open source / free software as in freedom... Even Ohloh
follows our commits:
https://www.ohloh.net/p/invesalius
;)

> --
> ? ? ?Enjoy global warming while it lasts.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Tatiana Al-Chueyr
Sent from Campinas, SP, Brazil


From warren.weckesser at enthought.com  Sun Feb 14 20:02:53 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sun, 14 Feb 2010 19:02:53 -0600
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <c5b438121002110721g66eda059ve86d199f7beed0fd@mail.gmail.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>	<c5b438121001281239y6b31691dq878e5ef340df96fd@mail.gmail.com>	<4B623ECD.4040403@enthought.com>	<1cd32cbb1001281757x14b7b581xba02f4ae21d3acfb@mail.gmail.com>	<4B6256F5.3050802@enthought.com>	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>	<c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>	<1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>	<4B71A6BD.9090002@enthought.com>
	<c5b438121002110721g66eda059ve86d199f7beed0fd@mail.gmail.com>
Message-ID: <4B789D3D.1060506@enthought.com>

I added a patch to ticket #1112.  In addition to adding the suggested 
change to pass keyword args to odeint, I also added a new funciton, 
impulse2(), that computes the impulse response by using odeint.  See 
this thread for details:

    http://mail.scipy.org/pipermail/scipy-user/2009-November/023416.html


Warren


Ryan Krauss wrote:
> Yeah, **kwds is a little harder to understand and slightly less user
> friendly, but it is easier to maintain and less work for the patch
> writer.  And it is more future proof if the integrator ever changes.
>
> On Tue, Feb 9, 2010 at 12:17 PM, Warren Weckesser
> <warren.weckesser at enthought.com> wrote:
>   
>> josef.pktd at gmail.com wrote:
>>     
>>> On Wed, Feb 3, 2010 at 9:00 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>
>>>       
>>>> FYI, I am quite happy with passing in an hmax value.  I basically
>>>> copied and pasted lsim2 from signal.ltisys and adapted it just a
>>>> little to make it a method of my derived class.  Then I added the hmas
>>>> kwarg that gets passed to odeint.
>>>>
>>>> Is there any reason not to allow the user to pass in a kwargs to lsim2
>>>> that gets passed to odeint?
>>>>
>>>>         
>>> I don't see a reason why we cannot add a **kwargs, it should be
>>> completely backwards compatible.
>>> Can you file a ticket and add your adjusted version or a patch? And
>>> even better, add your original example as a test case?
>>>
>>>
>>>       
>> Josef,
>>
>> I just created ticket #1112 for this.  Unless Ryan wants to adapt his
>> change to lsim2, I can make a patch this week to implement the enhancement.
>>
>> Warren
>>
>>
>>     
>>> Josef
>>>
>>>
>>>
>>>       
>>>> On Fri, Jan 29, 2010 at 6:44 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>>
>>>>         
>>>>> Thanks to Warren and Josef for their time and thoughts.  I feel like I
>>>>> now understand the underlying problem and have some good options to
>>>>> solve my short term issues (I assigned the project last night and they
>>>>> need to be able to start working on it immediately).  I actually use a
>>>>> TransferFunction class that derives from ltisys.  I could override its
>>>>> lsim2 method to try out some of these solutions quickly and fairly
>>>>> easily.
>>>>>
>>>>> Ryan
>>>>>
>>>>> On Thu, Jan 28, 2010 at 10:00 PM,  <josef.pktd at gmail.com> wrote:
>>>>>
>>>>>           
>>>>>> On Thu, Jan 28, 2010 at 10:33 PM, Warren Weckesser
>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>
>>>>>>             
>>>>>>> josef.pktd at gmail.com wrote:
>>>>>>>
>>>>>>>               
>>>>>>>> On Thu, Jan 28, 2010 at 8:50 PM, Warren Weckesser
>>>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> Ryan,
>>>>>>>>>
>>>>>>>>> The problem is that the ODE solver used by lsim2 is too good. :)
>>>>>>>>>
>>>>>>>>> It uses scipy.integrate.odeint, which in turn uses the Fortran library
>>>>>>>>> LSODA.  Like any good solver, LSODA is an adaptive solver--it adjusts its
>>>>>>>>> step size to be as large as possible while keeping estimates of the error
>>>>>>>>> bounded.  For the problem you are solving, with initial condition 0, the
>>>>>>>>> exact solution is initially exactly 0.  This is such a nice smooth solution
>>>>>>>>> that the solver's step size quickly grows--so big, in fact, that it skips
>>>>>>>>> right over your pulse and never sees it.
>>>>>>>>>
>>>>>>>>> So how does it create all those intermediate points at the requested time
>>>>>>>>> values?  It uses interpolation between the steps that it computed to create
>>>>>>>>> the solution values at the times that you requested.  So using a finer grid
>>>>>>>>> of time values won't help.  (If lsim2 gave you a hook into the parameters
>>>>>>>>> passed to odeint, you could set odeint's 'hmax' to a value smaller than your
>>>>>>>>> pulse width, which would force the solver to see the pulse.  But there is no
>>>>>>>>> way to set that parameter from lsim2.)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> It's something what I suspected. I don't know much about odeint, but
>>>>>>>> do you think it would be useful to let lsim2 pass through some
>>>>>>>> parameters to odeint?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> Sounds useful to me.  A simple implementation is an optional keyword
>>>>>>> argument that is a dict of odeint arguments.   But this would almost
>>>>>>> certainly break if lsim2 were ever reimplemented with a different
>>>>>>> solver.  So perhaps it should allow a common set of ODE solver
>>>>>>> parameters (e.g. absolute and relative error tolerances, max and min
>>>>>>> step sizes, others?).
>>>>>>>
>>>>>>> Perhaps this should wait until after the ODE solver redesign that is
>>>>>>> occasionally discussed:
>>>>>>>    http://projects.scipy.org/scipy/wiki/OdeintRedesign
>>>>>>> Then the solver itself could be an optional argument to lsim2.
>>>>>>>
>>>>>>>               
>>>>>> I was just thinking of adding to the argument list a **kwds argument
>>>>>> that is directly passed on to whatever ODE solver is used. This should
>>>>>> be pretty flexible for any changes and be backwards compatible.
>>>>>>
>>>>>> I've seen and used it in a similar way for calls to optimization
>>>>>> routines, e.g. also optimize.curve_fit, does it. What are actually
>>>>>> valid keywords would depend on which function is called.
>>>>>>
>>>>>> (But I'm not a user of lsim, I'm just stealing some ideas from lti and
>>>>>> friends for time series analysis.)
>>>>>>
>>>>>> Josef
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> Warren
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Josef
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> The basic problem is you are passing in a discontinuous function to a solver
>>>>>>>>> that expects a smooth function.  A better way to solve this problem is to
>>>>>>>>> explicitly account for the discontinuity. One possibility is the attached
>>>>>>>>> script.
>>>>>>>>>
>>>>>>>>> This is an excellent "learning opportunity" for your students on the hazards
>>>>>>>>> of numerical computing!
>>>>>>>>>
>>>>>>>>> Warren
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ryan Krauss wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> I believe I have discovered a bug in signal.lsim2.  I believe the
>>>>>>>>>> short attached script illustrates the problem.  I was trying to
>>>>>>>>>> predict the response of a transfer function with a pure integrator:
>>>>>>>>>>
>>>>>>>>>>             g
>>>>>>>>>> G = -------------
>>>>>>>>>>         s(s+p)
>>>>>>>>>>
>>>>>>>>>> to a finite width pulse.  lsim2 seems to handle the step response just
>>>>>>>>>> fine, but says that the pulse response is exactly 0.0 for the entire
>>>>>>>>>> time of the simulation.  Obviously, this isn't the right answer.
>>>>>>>>>>
>>>>>>>>>> I am running scipy 0.7.0 and numpy 1.2.1 on Ubuntu 9.04, but I also
>>>>>>>>>> have the same problem on Windows running 0.7.1 and 1.4.0.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Ryan
>>>>>>>>>>  ------------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> SciPy-User mailing list
>>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>> from pylab import *
>>>>>>>>> from scipy import signal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> g = 100.0
>>>>>>>>> p = 15.0
>>>>>>>>> G = signal.ltisys.lti(g, [1,p,0])
>>>>>>>>>
>>>>>>>>> t = arange(0, 1.0, 0.002)
>>>>>>>>> N = len(t)
>>>>>>>>>
>>>>>>>>> # u for the whole interval (not used in lsim2, only for plotting later).
>>>>>>>>> amp = 50.0
>>>>>>>>> u = zeros(N)
>>>>>>>>> k1 = 50
>>>>>>>>> k2 = 100
>>>>>>>>> u[k1:k2] = amp
>>>>>>>>>
>>>>>>>>> # Create input functions for each smooth interval. (This could be simpler,
>>>>>>>>> since u
>>>>>>>>> # is constant on each interval.)
>>>>>>>>> a = float(k1)/N
>>>>>>>>> b = float(k2)/N
>>>>>>>>> T1 = linspace(0, a, 201)
>>>>>>>>> u1 = zeros_like(T1)
>>>>>>>>> T2 = linspace(a, b, 201)
>>>>>>>>> u2 = amp*ones_like(T2)
>>>>>>>>> T3 = linspace(b, 1.0, 201)
>>>>>>>>> u3 = zeros_like(T3)
>>>>>>>>>
>>>>>>>>> # Solve on each interval; use the final value of one solution as the
>>>>>>>>> starting
>>>>>>>>> # point of the next solution.
>>>>>>>>> # (We could skip the first calculation, since we know the solution will be
>>>>>>>>> 0.)
>>>>>>>>> (t1, y1, x1) = signal.lsim2(G,u1,T1)
>>>>>>>>> (t2, y2, x2) = signal.lsim2(G, u2, T2, X0=x1[-1])
>>>>>>>>> (t3, y3, x3) = signal.lsim2(G, u3, T3, X0=x2[-1])
>>>>>>>>>
>>>>>>>>> figure(1)
>>>>>>>>> clf()
>>>>>>>>> plot(t, u, 'k', linewidth=3)
>>>>>>>>> plot(t1, y1, 'y', linewidth=3)
>>>>>>>>> plot(t2, y2, 'b', linewidth=3)
>>>>>>>>> plot(t3, y3, 'g', linewidth=3)
>>>>>>>>>
>>>>>>>>> show()
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SciPy-User mailing list
>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> _______________________________________________
>>>>>>>> SciPy-User mailing list
>>>>>>>> SciPy-User at scipy.org
>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> _______________________________________________
>>>>>>> SciPy-User mailing list
>>>>>>> SciPy-User at scipy.org
>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>>
>>>>>>             
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>>         
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>       
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>     
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From sccolbert at gmail.com  Sun Feb 14 21:08:35 2010
From: sccolbert at gmail.com (Chris Colbert)
Date: Sun, 14 Feb 2010 21:08:35 -0500
Subject: [SciPy-User] Save the date: SciPy 2010 June 28 - July 3
In-Reply-To: <E0BE6677-8B2A-4B6A-B0AF-DD67F1FB123C@enthought.com>
References: <E0BE6677-8B2A-4B6A-B0AF-DD67F1FB123C@enthought.com>
Message-ID: <7f014ea61002141808j4041628fk70a640310636cbf2@mail.gmail.com>

Do you know when the official call for papers will come out?

On Thu, Feb 11, 2010 at 5:35 PM, Amenity Applewhite
<amenity at enthought.com>wrote:

> The annual US Scientific Computing with Python Conference, SciPy, has been
> held at Caltech <http://www.caltech.edu/> since it began in 2001. While we
> always love an excuse to go to California, it?s also important to make sure
> that we allow everyone an opportunity to attend the conference. So, as
> Jarrod Millman announced<http://blog.jarrodmillman.com/2009/11/scipy-2010-coming-to-austin-tx-628-74.html> last
> fall, we?ll begin rotating the conference location and hold the 2010
> conference <http://conference.scipy.org/scipy2010/> in Austin, Texas.
>
> As you may know, Enthought is headquartered in Austin. So in addition to
> our standard SciPy sponsorship, this year we?ll also be undertaking a great
> deal of the planning and organization.
>
> To begin with, we?re thrilled to announce that we?ve secured several
> corporate sponsorships that will allow us to host the conference at the
> brand new AT&T Executive Education and Conference Center<http://conference.scipy.org/scipy2010/location.html> on
> campus at the University of Texas <http://www.utexas.edu/>. It?s a
> wonderful facility in Central Austin and provides easy access to an array of
> great restaurants, parks, and music venues<http://conference.scipy.org/scipy2010/location.html>
> .
>
> We will also be able to provide stipends for our Tutorial presenters<http://conference.scipy.org/scipy2010/tutorials.html> for
> the first time. These community members provide us with an invaluable
> learning experience every year, and we?re very excited to be able to
> compensate them for their efforts. And of course, we?ll also continue to
> offer student sponsorships<http://conference.scipy.org/scipy2010/student.html> to
> support active academic contributors who want to attend the conference.
>
> So mark your calendars, folks! June 28 ? July 3. Early registration<https://conference.scipy.org/scipy2010/registration.html> open
> now.
>
> Thanks,
> The Enthought Team
> --
> Amenity Applewhite
> Enthought, Inc.
> Scientific Computing Solutions
> www.enthought.com
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100214/45038005/attachment.html>

From millman at berkeley.edu  Sun Feb 14 21:32:11 2010
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 14 Feb 2010 20:32:11 -0600
Subject: [SciPy-User] Save the date: SciPy 2010 June 28 - July 3
In-Reply-To: <7f014ea61002141808j4041628fk70a640310636cbf2@mail.gmail.com>
References: <E0BE6677-8B2A-4B6A-B0AF-DD67F1FB123C@enthought.com>
	<7f014ea61002141808j4041628fk70a640310636cbf2@mail.gmail.com>
Message-ID: <c7009a551002141832l7291be39sf0f9c4841bfccd04@mail.gmail.com>

On Sun, Feb 14, 2010 at 8:08 PM, Chris Colbert <sccolbert at gmail.com> wrote:
> Do you know when the official call for papers will come out?

The call for papers will be out by the beginning of March.  Stefan van
der Walt is the program chair this year and his dissertation is due
this month (so he is slightly occupied at the moment ;)).  We aren't
planning any radical changes from last year, so last year's call will
give you a very good idea of what it will look like this year:
http://conference.scipy.org/call_for_papers

Thanks for your interest,

-- 
Jarrod Millman
Helen Wills Neuroscience Institute
10 Giannini Hall, UC Berkeley
http://cirl.berkeley.edu/


From josef.pktd at gmail.com  Sun Feb 14 21:43:22 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 14 Feb 2010 21:43:22 -0500
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <4B789D3D.1060506@enthought.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>
	<1cd32cbb1001281757x14b7b581xba02f4ae21d3acfb@mail.gmail.com>
	<4B6256F5.3050802@enthought.com>
	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>
	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>
	<c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>
	<1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>
	<4B71A6BD.9090002@enthought.com>
	<c5b438121002110721g66eda059ve86d199f7beed0fd@mail.gmail.com>
	<4B789D3D.1060506@enthought.com>
Message-ID: <1cd32cbb1002141843u261a7acbx5951e0374d0b5d48@mail.gmail.com>

On Sun, Feb 14, 2010 at 8:02 PM, Warren Weckesser
<warren.weckesser at enthought.com> wrote:
> I added a patch to ticket #1112. ?In addition to adding the suggested
> change to pass keyword args to odeint, I also added a new funciton,
> impulse2(), that computes the impulse response by using odeint. ?See
> this thread for details:
>
> ? ?http://mail.scipy.org/pipermail/scipy-user/2009-November/023416.html

Thanks, the test examples look nice and serve also as some
documentation for ltisys. The lsim2 test05 is interesting, I was
convinced last year (and there were some previous threads) that ltisys
cannot handle multi-input systems. I always got shape errors when I
tried and without documentation it's difficult to figure out what is
supposed to work and what not. (I will look at it more closely when I
have more time.)

I think the changes are reasonable including turning u and t into
keywords instead of required arguments. Maybe Ryan or someone who is
using this functions can comment on this.

Just one improvement to the tests, assert_almost_equal takes a decimal
argument. If you know the precision of your tests, then it would be
useful to increase it from the default decimal=7.

Josef


>
> Warren
>
>
> Ryan Krauss wrote:
>> Yeah, **kwds is a little harder to understand and slightly less user
>> friendly, but it is easier to maintain and less work for the patch
>> writer. ?And it is more future proof if the integrator ever changes.
>>
>> On Tue, Feb 9, 2010 at 12:17 PM, Warren Weckesser
>> <warren.weckesser at enthought.com> wrote:
>>
>>> josef.pktd at gmail.com wrote:
>>>
>>>> On Wed, Feb 3, 2010 at 9:00 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>>
>>>>
>>>>> FYI, I am quite happy with passing in an hmax value. ?I basically
>>>>> copied and pasted lsim2 from signal.ltisys and adapted it just a
>>>>> little to make it a method of my derived class. ?Then I added the hmas
>>>>> kwarg that gets passed to odeint.
>>>>>
>>>>> Is there any reason not to allow the user to pass in a kwargs to lsim2
>>>>> that gets passed to odeint?
>>>>>
>>>>>
>>>> I don't see a reason why we cannot add a **kwargs, it should be
>>>> completely backwards compatible.
>>>> Can you file a ticket and add your adjusted version or a patch? And
>>>> even better, add your original example as a test case?
>>>>
>>>>
>>>>
>>> Josef,
>>>
>>> I just created ticket #1112 for this. ?Unless Ryan wants to adapt his
>>> change to lsim2, I can make a patch this week to implement the enhancement.
>>>
>>> Warren
>>>
>>>
>>>
>>>> Josef
>>>>
>>>>
>>>>
>>>>
>>>>> On Fri, Jan 29, 2010 at 6:44 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>>>
>>>>>
>>>>>> Thanks to Warren and Josef for their time and thoughts. ?I feel like I
>>>>>> now understand the underlying problem and have some good options to
>>>>>> solve my short term issues (I assigned the project last night and they
>>>>>> need to be able to start working on it immediately). ?I actually use a
>>>>>> TransferFunction class that derives from ltisys. ?I could override its
>>>>>> lsim2 method to try out some of these solutions quickly and fairly
>>>>>> easily.
>>>>>>
>>>>>> Ryan
>>>>>>
>>>>>> On Thu, Jan 28, 2010 at 10:00 PM, ?<josef.pktd at gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>> On Thu, Jan 28, 2010 at 10:33 PM, Warren Weckesser
>>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> josef.pktd at gmail.com wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Thu, Jan 28, 2010 at 8:50 PM, Warren Weckesser
>>>>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Ryan,
>>>>>>>>>>
>>>>>>>>>> The problem is that the ODE solver used by lsim2 is too good. :)
>>>>>>>>>>
>>>>>>>>>> It uses scipy.integrate.odeint, which in turn uses the Fortran library
>>>>>>>>>> LSODA. ?Like any good solver, LSODA is an adaptive solver--it adjusts its
>>>>>>>>>> step size to be as large as possible while keeping estimates of the error
>>>>>>>>>> bounded. ?For the problem you are solving, with initial condition 0, the
>>>>>>>>>> exact solution is initially exactly 0. ?This is such a nice smooth solution
>>>>>>>>>> that the solver's step size quickly grows--so big, in fact, that it skips
>>>>>>>>>> right over your pulse and never sees it.
>>>>>>>>>>
>>>>>>>>>> So how does it create all those intermediate points at the requested time
>>>>>>>>>> values? ?It uses interpolation between the steps that it computed to create
>>>>>>>>>> the solution values at the times that you requested. ?So using a finer grid
>>>>>>>>>> of time values won't help. ?(If lsim2 gave you a hook into the parameters
>>>>>>>>>> passed to odeint, you could set odeint's 'hmax' to a value smaller than your
>>>>>>>>>> pulse width, which would force the solver to see the pulse. ?But there is no
>>>>>>>>>> way to set that parameter from lsim2.)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> It's something what I suspected. I don't know much about odeint, but
>>>>>>>>> do you think it would be useful to let lsim2 pass through some
>>>>>>>>> parameters to odeint?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Sounds useful to me. ?A simple implementation is an optional keyword
>>>>>>>> argument that is a dict of odeint arguments. ? But this would almost
>>>>>>>> certainly break if lsim2 were ever reimplemented with a different
>>>>>>>> solver. ?So perhaps it should allow a common set of ODE solver
>>>>>>>> parameters (e.g. absolute and relative error tolerances, max and min
>>>>>>>> step sizes, others?).
>>>>>>>>
>>>>>>>> Perhaps this should wait until after the ODE solver redesign that is
>>>>>>>> occasionally discussed:
>>>>>>>> ? ?http://projects.scipy.org/scipy/wiki/OdeintRedesign
>>>>>>>> Then the solver itself could be an optional argument to lsim2.
>>>>>>>>
>>>>>>>>
>>>>>>> I was just thinking of adding to the argument list a **kwds argument
>>>>>>> that is directly passed on to whatever ODE solver is used. This should
>>>>>>> be pretty flexible for any changes and be backwards compatible.
>>>>>>>
>>>>>>> I've seen and used it in a similar way for calls to optimization
>>>>>>> routines, e.g. also optimize.curve_fit, does it. What are actually
>>>>>>> valid keywords would depend on which function is called.
>>>>>>>
>>>>>>> (But I'm not a user of lsim, I'm just stealing some ideas from lti and
>>>>>>> friends for time series analysis.)
>>>>>>>
>>>>>>> Josef
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Warren
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Josef
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> The basic problem is you are passing in a discontinuous function to a solver
>>>>>>>>>> that expects a smooth function. ?A better way to solve this problem is to
>>>>>>>>>> explicitly account for the discontinuity. One possibility is the attached
>>>>>>>>>> script.
>>>>>>>>>>
>>>>>>>>>> This is an excellent "learning opportunity" for your students on the hazards
>>>>>>>>>> of numerical computing!
>>>>>>>>>>
>>>>>>>>>> Warren
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ryan Krauss wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I believe I have discovered a bug in signal.lsim2. ?I believe the
>>>>>>>>>>> short attached script illustrates the problem. ?I was trying to
>>>>>>>>>>> predict the response of a transfer function with a pure integrator:
>>>>>>>>>>>
>>>>>>>>>>> ? ? ? ? ? ? g
>>>>>>>>>>> G = -------------
>>>>>>>>>>> ? ? ? ? s(s+p)
>>>>>>>>>>>
>>>>>>>>>>> to a finite width pulse. ?lsim2 seems to handle the step response just
>>>>>>>>>>> fine, but says that the pulse response is exactly 0.0 for the entire
>>>>>>>>>>> time of the simulation. ?Obviously, this isn't the right answer.
>>>>>>>>>>>
>>>>>>>>>>> I am running scipy 0.7.0 and numpy 1.2.1 on Ubuntu 9.04, but I also
>>>>>>>>>>> have the same problem on Windows running 0.7.1 and 1.4.0.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Ryan
>>>>>>>>>>> ?------------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> SciPy-User mailing list
>>>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> from pylab import *
>>>>>>>>>> from scipy import signal
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> g = 100.0
>>>>>>>>>> p = 15.0
>>>>>>>>>> G = signal.ltisys.lti(g, [1,p,0])
>>>>>>>>>>
>>>>>>>>>> t = arange(0, 1.0, 0.002)
>>>>>>>>>> N = len(t)
>>>>>>>>>>
>>>>>>>>>> # u for the whole interval (not used in lsim2, only for plotting later).
>>>>>>>>>> amp = 50.0
>>>>>>>>>> u = zeros(N)
>>>>>>>>>> k1 = 50
>>>>>>>>>> k2 = 100
>>>>>>>>>> u[k1:k2] = amp
>>>>>>>>>>
>>>>>>>>>> # Create input functions for each smooth interval. (This could be simpler,
>>>>>>>>>> since u
>>>>>>>>>> # is constant on each interval.)
>>>>>>>>>> a = float(k1)/N
>>>>>>>>>> b = float(k2)/N
>>>>>>>>>> T1 = linspace(0, a, 201)
>>>>>>>>>> u1 = zeros_like(T1)
>>>>>>>>>> T2 = linspace(a, b, 201)
>>>>>>>>>> u2 = amp*ones_like(T2)
>>>>>>>>>> T3 = linspace(b, 1.0, 201)
>>>>>>>>>> u3 = zeros_like(T3)
>>>>>>>>>>
>>>>>>>>>> # Solve on each interval; use the final value of one solution as the
>>>>>>>>>> starting
>>>>>>>>>> # point of the next solution.
>>>>>>>>>> # (We could skip the first calculation, since we know the solution will be
>>>>>>>>>> 0.)
>>>>>>>>>> (t1, y1, x1) = signal.lsim2(G,u1,T1)
>>>>>>>>>> (t2, y2, x2) = signal.lsim2(G, u2, T2, X0=x1[-1])
>>>>>>>>>> (t3, y3, x3) = signal.lsim2(G, u3, T3, X0=x2[-1])
>>>>>>>>>>
>>>>>>>>>> figure(1)
>>>>>>>>>> clf()
>>>>>>>>>> plot(t, u, 'k', linewidth=3)
>>>>>>>>>> plot(t1, y1, 'y', linewidth=3)
>>>>>>>>>> plot(t2, y2, 'b', linewidth=3)
>>>>>>>>>> plot(t3, y3, 'g', linewidth=3)
>>>>>>>>>>
>>>>>>>>>> show()
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> SciPy-User mailing list
>>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SciPy-User mailing list
>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SciPy-User mailing list
>>>>>>>> SciPy-User at scipy.org
>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> SciPy-User mailing list
>>>>>>> SciPy-User at scipy.org
>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From cycomanic at gmail.com  Sun Feb 14 23:03:41 2010
From: cycomanic at gmail.com (Jochen Schroeder)
Date: Mon, 15 Feb 2010 15:03:41 +1100
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
Message-ID: <20100215040339.GA2115@cudos0803>

Hi all,

I'm pleased to announce version 0.2 of pyfftw, a python module providing access 
to the FFTW3 library. 

New features:

- pyfftw can now create advanced plans (needs testing)
- provide location of fftw libraries at runtime with an environment variable
- better detection of fftw location at install time

Note:
Pyfftw moved to launchpad, new url is http://launchpad.net/pyfftw.

Cheers
Jochen




From seb.haase at gmail.com  Mon Feb 15 04:35:40 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Mon, 15 Feb 2010 10:35:40 +0100
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <20100215040339.GA2115@cudos0803>
References: <20100215040339.GA2115@cudos0803>
Message-ID: <bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com>

On Mon, Feb 15, 2010 at 5:03 AM, Jochen Schroeder <cycomanic at gmail.com> wrote:
> Hi all,
>
> I'm pleased to announce version 0.2 of pyfftw, a python module providing access
> to the FFTW3 library.
>
> New features:
>
> - pyfftw can now create advanced plans (needs testing)
> - provide location of fftw libraries at runtime with an environment variable
> - better detection of fftw location at install time
>
> Note:
> Pyfftw moved to launchpad, new url is http://launchpad.net/pyfftw.
>
> Cheers
> Jochen
>
Hi Jochen,
this is great news and I'm eager to try it out.

I have a few questions:
1.) Do you support both single-threaded and multi-threaded ffts in one
installation, or would it work (only) with the kind of fft library
pyfftw finds at installation time ?
2.) I would like to incorporate pyfftw into my image analysis package
(Priithon). For that I provide binary packages for Linux 32bit + 64
bit + windows + Mac-OSX. The question now is, if I could share the
same pyfftw files - as generated by "python setup.py install" on one
system - between those systems ?  I could see this working because you
are using ctypes. On the other hand, the README file talks about many
things that are done at installation time.
It even says, that the FFTW_PATH environment variable is overwritten
if the situation at installation time would somehow compete with it.


Thanks for making your work available to us.
- Sebastian Haase


From cycomanic at gmail.com  Mon Feb 15 05:20:10 2010
From: cycomanic at gmail.com (Jochen Schroeder)
Date: Mon, 15 Feb 2010 21:20:10 +1100
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com>
Message-ID: <20100215102007.GA3762@jschrod-laptop>

On 02/15/10 10:35, Sebastian Haase wrote:
> On Mon, Feb 15, 2010 at 5:03 AM, Jochen Schroeder <cycomanic at gmail.com> wrote:
> > Hi all,
> >
> > I'm pleased to announce version 0.2 of pyfftw, a python module providing access
> > to the FFTW3 library.
> >
> > New features:
> >
> > - pyfftw can now create advanced plans (needs testing)
> > - provide location of fftw libraries at runtime with an environment variable
> > - better detection of fftw location at install time
> >
> > Note:
> > Pyfftw moved to launchpad, new url is http://launchpad.net/pyfftw.
> >
> > Cheers
> > Jochen
> >
> Hi Jochen,
> this is great news and I'm eager to try it out.
> 
> I have a few questions:
> 1.) Do you support both single-threaded and multi-threaded ffts in one
> installation, or would it work (only) with the kind of fft library
> pyfftw finds at installation time ?
Single and multi-threaded ffts are supported. Currently pyfftw assumes that
the single- and multithreaded libraries are in the same directory (on windows
it's actually the same file). If it can't find the threaded library at runtime
threaded mode just does not work.

> 2.) I would like to incorporate pyfftw into my image analysis package
> (Priithon). For that I provide binary packages for Linux 32bit + 64
> bit + windows + Mac-OSX. The question now is, if I could share the
> same pyfftw files - as generated by "python setup.py install" on one
> system - between those systems ?  I could see this working because you
> are using ctypes. On the other hand, the README file talks about many
> things that are done at installation time.
> It even says, that the FFTW_PATH environment variable is overwritten
> if the situation at installation time would somehow compete with it.
> 
FFTW_PATH at runtime always overrides the installtime location, maybe I need to
make this clearer in the README. However, this
probably does not really help your case. The problem is that it's not really
simple to detect the fftw library with ctypes in a cross-platform way.
Especially if fftw is not installed in a system directory (which somehow seems
to be the norm on Windows). I'm still not entirely sure that the way I'm doing
things currently is the best way, so I'm always open to suggestions. I'll
think about if there's a better way of doing this

Cheers
Jochen

> 
> Thanks for making your work available to us.
> - Sebastian Haase
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From seb.haase at gmail.com  Mon Feb 15 05:27:42 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Mon, 15 Feb 2010 11:27:42 +0100
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <20100215102007.GA3762@jschrod-laptop>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com> 
	<20100215102007.GA3762@jschrod-laptop>
Message-ID: <bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com>

On Mon, Feb 15, 2010 at 11:20 AM, Jochen Schroeder <cycomanic at gmail.com> wrote:
> On 02/15/10 10:35, Sebastian Haase wrote:
>> On Mon, Feb 15, 2010 at 5:03 AM, Jochen Schroeder <cycomanic at gmail.com> wrote:
>> > Hi all,
>> >
>> > I'm pleased to announce version 0.2 of pyfftw, a python module providing access
>> > to the FFTW3 library.
>> >
>> > New features:
>> >
>> > - pyfftw can now create advanced plans (needs testing)
>> > - provide location of fftw libraries at runtime with an environment variable
>> > - better detection of fftw location at install time
>> >
>> > Note:
>> > Pyfftw moved to launchpad, new url is http://launchpad.net/pyfftw.
>> >
>> > Cheers
>> > Jochen
>> >
>> Hi Jochen,
>> this is great news and I'm eager to try it out.
>>
>> I have a few questions:
>> 1.) Do you support both single-threaded and multi-threaded ffts in one
>> installation, or would it work (only) with the kind of fft library
>> pyfftw finds at installation time ?
> Single and multi-threaded ffts are supported. Currently pyfftw assumes that
> the single- and multithreaded libraries are in the same directory (on windows
> it's actually the same file). If it can't find the threaded library at runtime
> threaded mode just does not work.
>
>> 2.) I would like to incorporate pyfftw into my image analysis package
>> (Priithon). For that I provide binary packages for Linux 32bit + 64
>> bit + windows + Mac-OSX. The question now is, if I could share the
>> same pyfftw files - as generated by "python setup.py install" on one
>> system - between those systems ? ?I could see this working because you
>> are using ctypes. On the other hand, the README file talks about many
>> things that are done at installation time.
>> It even says, that the FFTW_PATH environment variable is overwritten
>> if the situation at installation time would somehow compete with it.
>>
> FFTW_PATH at runtime always overrides the installtime location, maybe I need to
> make this clearer in the README. However, this
> probably does not really help your case. The problem is that it's not really
> simple to detect the fftw library with ctypes in a cross-platform way.
> Especially if fftw is not installed in a system directory (which somehow seems
> to be the norm on Windows). I'm still not entirely sure that the way I'm doing
> things currently is the best way, so I'm always open to suggestions. I'll
> think about if there's a better way of doing this
>
> Cheers
> Jochen
>

Thanks for the reply,
another question: quote from the README:
"""
!IMPORTANT!
Because the plan uses pointers to the data of the arrays you cannot perform
operations on the arrays that change the data pointer.
"""
However, the fftw docs that I remember (found) says:
"Once the plan has been created, you can use it as many times as you
like for transforms on arrays of the same size."
( http://www.fftw.org/fftw2_doc/fftw_2.html )

Has this changed from FFTW2 to FFTW3 ?
It would really limit the use of plans, and make overall FFTs much
slower. In my specific case I very often have 512x512 single-precision
real arrays (images), that I would do ffts over and over again. But
the pointers would change ....

-Sebastian


From david at silveregg.co.jp  Mon Feb 15 05:46:55 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Mon, 15 Feb 2010 19:46:55 +0900
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com> 
	<20100215102007.GA3762@jschrod-laptop>
	<bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com>
Message-ID: <4B79261F.6060706@silveregg.co.jp>

Sebastian Haase wrote:

> 
> Has this changed from FFTW2 to FFTW3 ?
> It would really limit the use of plans, and make overall FFTs much
> slower. In my specific case I very often have 512x512 single-precision
> real arrays (images), that I would do ffts over and over again. But
> the pointers would change ....

You can, but you need to use the advanced plan API, or use the recently 
added new-array execute function:

http://www.fftw.org/fftw3_doc/New_002darray-Execute-Functions.html#New_002darray-Execute-Functions

cheers,

David


From seb.haase at gmail.com  Mon Feb 15 06:06:38 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Mon, 15 Feb 2010 12:06:38 +0100
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <4B79261F.6060706@silveregg.co.jp>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com> 
	<20100215102007.GA3762@jschrod-laptop>
	<bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com> 
	<4B79261F.6060706@silveregg.co.jp>
Message-ID: <bc657ead1002150306l228a4748u1f509e672ef5a57f@mail.gmail.com>

On Mon, Feb 15, 2010 at 11:46 AM, David Cournapeau
<david at silveregg.co.jp> wrote:
> Sebastian Haase wrote:
>
>>
>> Has this changed from FFTW2 to FFTW3 ?
>> It would really limit the use of plans, and make overall FFTs much
>> slower. In my specific case I very often have 512x512 single-precision
>> real arrays (images), that I would do ffts over and over again. But
>> the pointers would change ....
>
> You can, but you need to use the advanced plan API, or use the recently
> added new-array execute function:
>
> http://www.fftw.org/fftw3_doc/New_002darray-Execute-Functions.html#New_002darray-Execute-Functions
>
so it sounds like the alignment is the "killer" argument for the whole idea:
quote:
"""
In general, only arrays allocated with fftw_malloc are guaranteed to
be equally aligned (see SIMD alignment and fftw_malloc).

The alignment issue is especially critical, because if you don't use
fftw_malloc then you may have little control over the alignment of
arrays in memory. For example, neither the C++ new function nor the
Fortran allocate statement provide strong enough guarantees about data
alignment. If you don't use fftw_malloc, therefore, you probably have
to use FFTW_UNALIGNED (which disables most SIMD support). If possible,
it is probably better for you to simply create multiple plans
(creating a new plan is quick once one exists for a given size), or
better yet re-use the same array for your transforms.
"""

I guess this is really all new with version 3 of FFTW. I hope that
"reating a new plan is quick once one exists for a given size" means
"neglectable" for 512x512 arrays !?

-S.


From seb.haase at gmail.com  Mon Feb 15 06:09:32 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Mon, 15 Feb 2010 12:09:32 +0100
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <bc657ead1002150306l228a4748u1f509e672ef5a57f@mail.gmail.com>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com> 
	<20100215102007.GA3762@jschrod-laptop>
	<bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com> 
	<4B79261F.6060706@silveregg.co.jp>
	<bc657ead1002150306l228a4748u1f509e672ef5a57f@mail.gmail.com>
Message-ID: <bc657ead1002150309k2a300fden6ea72076b4161ade@mail.gmail.com>

On Mon, Feb 15, 2010 at 12:06 PM, Sebastian Haase <seb.haase at gmail.com> wrote:
> On Mon, Feb 15, 2010 at 11:46 AM, David Cournapeau
> <david at silveregg.co.jp> wrote:
>> Sebastian Haase wrote:
>>
>>>
>>> Has this changed from FFTW2 to FFTW3 ?
>>> It would really limit the use of plans, and make overall FFTs much
>>> slower. In my specific case I very often have 512x512 single-precision
>>> real arrays (images), that I would do ffts over and over again. But
>>> the pointers would change ....
>>
>> You can, but you need to use the advanced plan API, or use the recently
>> added new-array execute function:
>>
>> http://www.fftw.org/fftw3_doc/New_002darray-Execute-Functions.html#New_002darray-Execute-Functions
>>
> so it sounds like the alignment is the "killer" argument for the whole idea:
> quote:
> """
> In general, only arrays allocated with fftw_malloc are guaranteed to
> be equally aligned (see SIMD alignment and fftw_malloc).
>
> The alignment issue is especially critical, because if you don't use
> fftw_malloc then you may have little control over the alignment of
> arrays in memory. For example, neither the C++ new function nor the
> Fortran allocate statement provide strong enough guarantees about data
> alignment. If you don't use fftw_malloc, therefore, you probably have
> to use FFTW_UNALIGNED (which disables most SIMD support). If possible,
> it is probably better for you to simply create multiple plans
> (creating a new plan is quick once one exists for a given size), or
> better yet re-use the same array for your transforms.
> """
>
> I guess this is really all new with version 3 of FFTW. I hope that
> "reating a new plan is quick once one exists for a given size" means
> "neglectable" for 512x512 arrays !?
>
> -S.
>
Oh, and it also seems to imply that FFTW3 is internally doing some
sort of plan-caching ... how else could it be faster the second time
!?
David, do you have experience regarding this topic - or anyone else here ?
-S.


From david at silveregg.co.jp  Mon Feb 15 06:17:45 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Mon, 15 Feb 2010 20:17:45 +0900
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <bc657ead1002150306l228a4748u1f509e672ef5a57f@mail.gmail.com>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com> 
	<20100215102007.GA3762@jschrod-laptop>
	<bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com> 
	<4B79261F.6060706@silveregg.co.jp>
	<bc657ead1002150306l228a4748u1f509e672ef5a57f@mail.gmail.com>
Message-ID: <4B792D59.2040608@silveregg.co.jp>

Sebastian Haase wrote:
> On Mon, Feb 15, 2010 at 11:46 AM, David Cournapeau
> <david at silveregg.co.jp> wrote:
>> Sebastian Haase wrote:
>>
>>> Has this changed from FFTW2 to FFTW3 ?
>>> It would really limit the use of plans, and make overall FFTs much
>>> slower. In my specific case I very often have 512x512 single-precision
>>> real arrays (images), that I would do ffts over and over again. But
>>> the pointers would change ....
>> You can, but you need to use the advanced plan API, or use the recently
>> added new-array execute function:
>>
>> http://www.fftw.org/fftw3_doc/New_002darray-Execute-Functions.html#New_002darray-Execute-Functions
>>
> so it sounds like the alignment is the "killer" argument for the whole idea:
> quote

Well, yes, you need aligned pointers, there is no way around it if you 
want to (significantly) benefit from SSE - that's why I proposed some 
time ago now an aligned allocator to be used inside NumPy, so that many 
numpy arrays would be aligned by default.

Note that you can align them by yourself if you want to (there are 
several recipes on how to do that, one from Travis on Enthought blog 
IIRC, and one from Anne in the NumPy ML). Or explicitly create plans for 
unaligned arrays (this is significantly slower, though, but should be at 
least as fast as fftw2).

Also, most arrays allocated by malloc are *not* 16 bytes aligned on 
Linux, because for allocated areas above a certain size, the glibc 
malloc use mmap, and always "disalign" the allocated buffer. The 
threshold is easily reached when working with big data.

> I guess this is really all new with version 3 of FFTW. I hope that
> "reating a new plan is quick once one exists for a given size" means
> "neglectable" for 512x512 arrays !?

You would have to test, but IIRC, the cost is not negligeable. Creating 
an API around those plans should not be very difficult - at worse, you 
can take a look at how scipy used to do it when scipy was supporting 
FFTW backend. The problem is designing a fast API - especially for small 
size arrays (~ 2**10), fft is so fast that you cannot afford a lot while 
looking for cached plans :)

cheers,

David


From cclarke at chrisdev.com  Mon Feb 15 07:57:06 2010
From: cclarke at chrisdev.com (Chris Clarke)
Date: Mon, 15 Feb 2010 08:57:06 -0400
Subject: [SciPy-User] Problems viewing http://pytseries.sourceforge.net/
	with Safari on OSX
Message-ID: <70B21120-1839-4D3F-8F9E-36E8E3D224B6@chrisdev.com>

Hi
Sent someone a link about scikits.timeseries and they said the pages  
were "blank".  Apparently there is an issue on OS.X w.r.t this and  
other linked pages.   I think it is also the same with the other  
scikits pages.  I've been able  reproduce with OSX Leopard and Safari  
4.0.4.
Of course everything works with Firefox.  I have not tried with Chrome
Regards
Chris


From y.copin at ipnl.in2p3.fr  Mon Feb 15 08:35:37 2010
From: y.copin at ipnl.in2p3.fr (Yannick Copin)
Date: Mon, 15 Feb 2010 14:35:37 +0100
Subject: [SciPy-User] Convert a time-frequency array to sound file
In-Reply-To: <mailman.11.1266084004.16738.scipy-user@scipy.org>
References: <mailman.11.1266084004.16738.scipy-user@scipy.org>
Message-ID: <4B794DA9.4040105@ipnl.in2p3.fr>

Hi,

> You could just take the inverse fourier transform of each spectra and then
> just patch them end to end, but I suspect this will end up sounding pretty
> awful as you'd get lots of phase discontinuities at the end of each
> segment. A better strategy might be to generate a continuous sin wave for
> each frequency (with a good number of samples per segment), multiply each
> of these sin waves by the corresponding, interpolated, spectral amplitude,
> and then sum over the different frequencies.
> 
> This should be easy enough to do with the standard numpy functions.

Thanks for the tips. Alas, I guess I have to conclude that such a conversion 
procedure does not already exist...

Cheers,
-- 
    .~.   Yannick COPIN  (o:>*  Doctus cum libro
    /V\   Institut de physique nucl?aire de Lyon
   // \\  Universit? de Lyon - CNRS-IN2P3
  /(   )\ AIM: YnCopin ICQ: 236931013
   ^`~'^  http://snovae.in2p3.fr/ycopin/




From josef.pktd at gmail.com  Mon Feb 15 08:52:31 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 15 Feb 2010 08:52:31 -0500
Subject: [SciPy-User] Convert a time-frequency array to sound file
In-Reply-To: <4B794DA9.4040105@ipnl.in2p3.fr>
References: <mailman.11.1266084004.16738.scipy-user@scipy.org>
	<4B794DA9.4040105@ipnl.in2p3.fr>
Message-ID: <1cd32cbb1002150552h5d6d9779s4d99bbf342a8eff@mail.gmail.com>

On Mon, Feb 15, 2010 at 8:35 AM, Yannick Copin <y.copin at ipnl.in2p3.fr> wrote:
> Hi,
>
>> You could just take the inverse fourier transform of each spectra and then
>> just patch them end to end, but I suspect this will end up sounding pretty
>> awful as you'd get lots of phase discontinuities at the end of each
>> segment. A better strategy might be to generate a continuous sin wave for
>> each frequency (with a good number of samples per segment), multiply each
>> of these sin waves by the corresponding, interpolated, spectral amplitude,
>> and then sum over the different frequencies.
>>
>> This should be easy enough to do with the standard numpy functions.
>
> Thanks for the tips. Alas, I guess I have to conclude that such a conversion
> procedure does not already exist...

Wav sound files (scipy.io.wavfile)
write (filename, rate, data) Write a numpy array as a WAV file

I don't know what this does, I'm just browsing the docs.
Disclaimer: I don't know anything about sound processing

Josef

>
> Cheers,
> --
> ? ?.~. ? Yannick COPIN ?(o:>* ?Doctus cum libro
> ? ?/V\ ? Institut de physique nucl?aire de Lyon
> ? // \\ ?Universit? de Lyon - CNRS-IN2P3
> ?/( ? )\ AIM: YnCopin ICQ: 236931013
> ? ^`~'^ ?http://snovae.in2p3.fr/ycopin/
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From zallen at urologicresearchinstitute.org  Mon Feb 15 09:17:07 2010
From: zallen at urologicresearchinstitute.org (URI)
Date: Mon, 15 Feb 2010 14:17:07 +0000 (UTC)
Subject: [SciPy-User] creating a 3D surface plot from collected data
Message-ID: <loom.20100215T145249-313@post.gmane.org>

I am trying to figure out how to create a 3D surface plot from some x,y,z data
that I have.  I don't have any particular preference as to how to do it, but
from poking around on the web it seems like mplot3d should be able to do it,
based on what I see in the tutorial (look at the wireframe or surface plot
sections):

http://matplotlib.sourceforge.net/mpl_toolkits/mplot3d/tutorial.html

The problem I have is that none of the example source code is based on working
with numbers from actual data.  The only thing I could find that addresses this
issue is at the very bottom of this page under the heading "Another Example":
http://www.scipy.org/Cookbook/Matplotlib/mplot3D

I've tried to duplicate this method but I get errors, and I don't really
understand it anyway.  The place I get lost is here:

Z = numpy.zeros((len(Y), len(X)), 'Float32')
for d in data:
   x, y, z = d
   ix = int(x * 10)
   iy = int(y * 10)
   Z[iy, ix] = z


I don't understand how ix and iy are supposed to give me valid indeces for the Z
array (and Python doesn't seem to understand it either, it gives me an error at
that last line).

Can someone either explain this method to me, or point me in a different
direction?



From baker.alexander at gmail.com  Mon Feb 15 11:47:18 2010
From: baker.alexander at gmail.com (alexander baker)
Date: Mon, 15 Feb 2010 16:47:18 +0000
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <loom.20100215T145249-313@post.gmane.org>
References: <loom.20100215T145249-313@post.gmane.org>
Message-ID: <270620221002150847q5f3e0cbexb517cb7a2d782606@mail.gmail.com>

Example here that might help.

Regards

Alex Baker

http://www.alexfb.com/cgi-bin/twiki/view/PtPhysics/Spherical-Polar-Coordinates-Python

Mobile: 07788 872118
Blog: www.alexfb.com

--
All science is either physics or stamp collecting.


On 15 February 2010 14:17, URI <zallen at urologicresearchinstitute.org> wrote:

> I am trying to figure out how to create a 3D surface plot from some x,y,z
> data
> that I have.  I don't have any particular preference as to how to do it,
> but
> from poking around on the web it seems like mplot3d should be able to do
> it,
> based on what I see in the tutorial (look at the wireframe or surface plot
> sections):
>
> http://matplotlib.sourceforge.net/mpl_toolkits/mplot3d/tutorial.html
>
> The problem I have is that none of the example source code is based on
> working
> with numbers from actual data.  The only thing I could find that addresses
> this
> issue is at the very bottom of this page under the heading "Another
> Example":
> http://www.scipy.org/Cookbook/Matplotlib/mplot3D
>
> I've tried to duplicate this method but I get errors, and I don't really
> understand it anyway.  The place I get lost is here:
>
> Z = numpy.zeros((len(Y), len(X)), 'Float32')
> for d in data:
>   x, y, z = d
>   ix = int(x * 10)
>   iy = int(y * 10)
>   Z[iy, ix] = z
>
>
> I don't understand how ix and iy are supposed to give me valid indeces for
> the Z
> array (and Python doesn't seem to understand it either, it gives me an
> error at
> that last line).
>
> Can someone either explain this method to me, or point me in a different
> direction?
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100215/ca47efe8/attachment.html>

From jkington at wisc.edu  Mon Feb 15 12:20:30 2010
From: jkington at wisc.edu (Joe Kington)
Date: Mon, 15 Feb 2010 11:20:30 -0600
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <loom.20100215T145249-313@post.gmane.org>
References: <loom.20100215T145249-313@post.gmane.org>
Message-ID: <f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>

Are your data already gridded? Or are they irregularly spaced?  Seeing as
how you mentioned "real" data, I'm assuming they're irregularly spaced.

The iy, ix example assumes that you have a regular grid that has been
exported as an x,y,z file instead of a grid of z coordinates.

If they're not already a regular grid, you'll need to interpolate them onto
a regular grid before using any of the 3D surface plotting routines.

(I'll skip my "interpolation is an artform" rant, here...  For a quick look
splines are fine.)

Look into using the various functions in sp.interpolate.  The example I
threw together below uses a thin-plate spline, but there are lots of other
options.

import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import scipy.interpolate
from mpl_toolkits.mplot3d import Axes3D

x = np.random.random(10)
y = np.random.random(10)
z = np.random.random(10)

spline = sp.interpolate.Rbf(x,y,z,function='thin-plate')

xi = np.linspace(x.min(), x.max(), 50)
yi = np.linspace(y.min(), y.max(), 50)
xi, yi = np.meshgrid(xi, yi)

zi = spline(xi,yi)

fig = plt.figure()
ax = Axes3D(fig)
ax.plot_surface(xi,yi,zi)
plt.show()

Hope that helps,
-Joe

On Mon, Feb 15, 2010 at 8:17 AM, URI
<zallen at urologicresearchinstitute.org>wrote:

> I am trying to figure out how to create a 3D surface plot from some x,y,z
> data
> that I have.  I don't have any particular preference as to how to do it,
> but
> from poking around on the web it seems like mplot3d should be able to do
> it,
> based on what I see in the tutorial (look at the wireframe or surface plot
> sections):
>
> http://matplotlib.sourceforge.net/mpl_toolkits/mplot3d/tutorial.html
>
> The problem I have is that none of the example source code is based on
> working
> with numbers from actual data.  The only thing I could find that addresses
> this
> issue is at the very bottom of this page under the heading "Another
> Example":
> http://www.scipy.org/Cookbook/Matplotlib/mplot3D
>
> I've tried to duplicate this method but I get errors, and I don't really
> understand it anyway.  The place I get lost is here:
>
> Z = numpy.zeros((len(Y), len(X)), 'Float32')
> for d in data:
>   x, y, z = d
>   ix = int(x * 10)
>   iy = int(y * 10)
>   Z[iy, ix] = z
>
>
> I don't understand how ix and iy are supposed to give me valid indeces for
> the Z
> array (and Python doesn't seem to understand it either, it gives me an
> error at
> that last line).
>
> Can someone either explain this method to me, or point me in a different
> direction?
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100215/3aee2783/attachment.html>

From peridot.faceted at gmail.com  Mon Feb 15 12:30:26 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 15 Feb 2010 12:30:26 -0500
Subject: [SciPy-User] Convert a time-frequency array to sound file
In-Reply-To: <4B794DA9.4040105@ipnl.in2p3.fr>
References: <mailman.11.1266084004.16738.scipy-user@scipy.org>
	<4B794DA9.4040105@ipnl.in2p3.fr>
Message-ID: <ce557a361002150930n48199225se1f3fbf1b5370dc@mail.gmail.com>

On 15 February 2010 08:35, Yannick Copin <y.copin at ipnl.in2p3.fr> wrote:
> Hi,
>
>> You could just take the inverse fourier transform of each spectra and then
>> just patch them end to end, but I suspect this will end up sounding pretty
>> awful as you'd get lots of phase discontinuities at the end of each
>> segment. A better strategy might be to generate a continuous sin wave for
>> each frequency (with a good number of samples per segment), multiply each
>> of these sin waves by the corresponding, interpolated, spectral amplitude,
>> and then sum over the different frequencies.
>>
>> This should be easy enough to do with the standard numpy functions.
>
> Thanks for the tips. Alas, I guess I have to conclude that such a conversion
> procedure does not already exist...

Without phase information, there's no well-defined way to do this.
It's easiest to think of this the other way around: suppose you start
with a sound file. To get frequency-as-a-function-of-time information,
you need to break the file up into chunks and take an FFT of each
chunk. The size of each chunk is where you switch from a time
representation to a frequency representation - that is, suppose you go
with 20 chunks a second. Then a tone higher than 20 Hz will appear in
your spectra, while a tone lower than 20 Hz will appear as variations
from one spectrum to the next. Once you've decided on the chunk size,
you probably want to avoid discontinuities at the ends of the chunks,
so you probably want the chunks to overlap. Now having done this, you
have a series of spectra, containing phase and amplitude information
for each frequency. You can reconstruct the original signal exactly
from this.

What you have amounts to only the amplitude information for each
chunk. This is not necessarily a problem, as the ear can't really hear
phases of sounds, although you have to be careful when combining
pieces so that you don't get too much constructive or destructive
interference. Choosing random phases will probably be fine. So to get
a sound file, you could inverse FFT these spectra, and then piece them
together with overlap. The result will probably sound horrible. There
are some things you can tweak:

* The temporal spacing of the chunks. Depending on what your initial
information looks like, you might want to either average it down in
the time direction, or interpolate it up, so that the sounds happen at
a reasonable pace.

* The frequency range you generate. In a raw FFT approach the lowest
frequency in your image will be the chunk rate and the highest will be
set by the sampling rate, probably inaudibly high. You may well want
to rearrange the frequency range so it's comfortable, say from
100-1000 Hz.

* The atonality of the output. Especially if you have very little
information in the frequency direction, you may want to ensure that
every frequency that's generated corresponds to a note on some musical
scale, possibly even using some instrument more elaborate than a sine
wave. For this you should look at software (music) synthesizers.

The general family of techniques you'd be looking at for this would be
"granular synthesis"; in particular, there are utilities out there to
turn images into sound files. I don't know the name of any particular
one, but some Googling ought to turn up some choices with varying
degrees of flexibility.

The more general problem, of "graphical visualization" as applied to
sound rather than images, seems to be a sadly understudied field.

Anne

P.S. Here's an example of this sort of thing, although sadly very
little documentation exists:
http://www.amazon.com/Symphonies-Planets-NASA-Voyager-Recordings/dp/B000001V2O
-A


From nahumoz at gmail.com  Mon Feb 15 13:32:22 2010
From: nahumoz at gmail.com (Oz Nahum)
Date: Mon, 15 Feb 2010 19:32:22 +0100
Subject: [SciPy-User] creating a 3D surface plot from collected data
Message-ID: <6ec71d091002151032u74f5f8beof39b2d6d1e9a5bfd@mail.gmail.com>

> The problem I have is that none of the example source code is based on working
with numbers from actual data.

This is my usual criticism to most of the scipy documentation which is
written to mathematicians not for users (check this in my blog
http://www.tabula0rasa.org/2008/07/crazy-matplotlib-tutorial/ )

In anycase -
here is any example with hydraulic head data which is easy to understand

http://www.tabula0rasa.org/2009/05/3d-plots-with-matplotlib/

let me know if the example works for you or not (I didn't check it for a while)

-- 
Oz Nahum
Graduate Student
Zentrum f?r Angewandte Geologie
Universit?t T?bingen

---

Imagine there's no countries
it isn't hard to do
Nothing to kill or die for
And no religion too
Imagine all the people
Living life in peace


From nahumoz at gmail.com  Mon Feb 15 14:10:45 2010
From: nahumoz at gmail.com (Oz Nahum)
Date: Mon, 15 Feb 2010 20:10:45 +0100
Subject: [SciPy-User] creating a 3D surface plot from collected data
Message-ID: <6ec71d091002151110v19b10679m60e5c5c2cd853915@mail.gmail.com>

BTW,
the example give by Alex, is nice, but there is a small error:
The line
import matplotlib.axes3d as p3   # 3D axes class
should be
import mpl_toolkits.mplot3d as p3

And then it plot a lovely diamond ;-)

-- 
Oz Nahum
Graduate Student
Zentrum f?r Angewandte Geologie
Universit?t T?bingen

---

Imagine there's no countries
it isn't hard to do
Nothing to kill or die for
And no religion too
Imagine all the people
Living life in peace


From R.Springuel at umit.maine.edu  Mon Feb  8 15:40:13 2010
From: R.Springuel at umit.maine.edu (R. Padraic Springuel)
Date: Mon, 08 Feb 2010 15:40:13 -0500
Subject: [SciPy-User] New installation
Message-ID: <4B7076AD.8010900@umit.maine.edu>

I just tried to upgrade one of the computers I'm responsible for from
Python 2.5 to Python 2.6 (including all the packages that are used) and
while all the installations seemed to work just fine, I get the
following error when I try to import numpy after install:

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/__init__.py", 

line 132, in <module>
     import add_newdocs
   File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/add_newdocs.py", 

line 9, in <module>
     from lib import add_newdoc
   File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/lib/__init__.py", 

line 4, in <module>
     from type_check import *
   File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/lib/type_check.py", 

line 8, in <module>
     import numpy.core.numeric as _nx
   File
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/__init__.py", 

line 5, in <module>
     import multiarray
ImportError:
dlopen(/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/multiarray.so, 

2): no suitable image found.  Did find:
	/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/multiarray.so: 

unknown required load command 0x80000022


Any idea what might be causing this error and what I can do to fix it?

Python: 2.6.4
Numpy: 1.4.0
Mac OS 10.5.8
-- 

R. Padraic Springuel
Research Assistant
Department of Physics and Astronomy
University of Maine
Bennett 309
Office Hours: Thursdays, 12-2pm



From adam.ginsburg at colorado.edu  Mon Feb 15 17:22:42 2010
From: adam.ginsburg at colorado.edu (Adam Ginsburg)
Date: Mon, 15 Feb 2010 15:22:42 -0700
Subject: [SciPy-User] Install fail on Mac 10.6.2 in fftpack._fftpack
Message-ID: <fceb3d4a1002151422m7759ec86k7d279c6cfa717003@mail.gmail.com>

Hi, I'm trying to get scipy running on my new Snow Leopard system.  I
was able to get scipy built successfully yesterday, but not so much
today.  I'd appreciate any assistance in getting scipy installed.

I built fftw with the following setup:
./configure CC="gcc -arch i386 -arch x86_64" CXX="g++ -arch i386 -arch
x86_64" CPP="gcc -E" CXXCPP="g++ -E"
make
sudo make install

I've also tried multiple variations of architection for fftw (e.g. not
using -arch i386 and passing no compiler flags). I built UMFTPACK
following the instructions on: http://blog.hyperjeff.net/?p=160

I get the following error:
building 'scipy.fftpack._fftpack' extension
compiling C sources
C compiler: gcc-4.2 -DNDEBUG -g -O3 -arch i386 -arch x86_64

creating build/temp.macosx-10.6-intel-2.6/build
creating build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6
creating build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6/scipy
creating build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6/scipy/fftpack
creating build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6/scipy/fftpack/src
compile options: '-Iscipy/fftpack/src
-Ibuild/src.macosx-10.6-intel-2.6
-I/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include
-I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
-c'
gcc-4.2: build/src.macosx-10.6-intel-2.6/scipy/fftpack/src/dct.c
gcc-4.2: build/src.macosx-10.6-intel-2.6/scipy/fftpack/_fftpackmodule.c
gcc-4.2: scipy/fftpack/src/drfft.c
gcc-4.2: scipy/fftpack/src/zfft.c
gcc-4.2: scipy/fftpack/src/zrfft.c
gcc-4.2: scipy/fftpack/src/zfftnd.c
gcc-4.2: build/src.macosx-10.6-intel-2.6/fortranobject.c
/usr/local/bin/gfortran -Wall -lgfortran -arch x86_64
build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6/scipy/fftpack/_fftpackmodule.o
build/temp.macosx-10.6-intel-2.6/scipy/fftpack/src/zfft.o
build/temp.macosx-10.6-intel-2.6/scipy/fftpack/src/drfft.o
build/temp.macosx-10.6-intel-2.6/scipy/fftpack/src/zrfft.o
build/temp.macosx-10.6-intel-2.6/scipy/fftpack/src/zfftnd.o
build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6/scipy/fftpack/src/dct.o
build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6/fortranobject.o
-L/usr/local/lib/gcc/i686-apple-darwin8/4.2.3/x86_64
-Lbuild/temp.macosx-10.6-intel-2.6 -ldfftpack -lfftpack -lgfortran -o
build/lib.macosx-10.6-intel-2.6/scipy/fftpack/_fftpack.so
Undefined symbols:
  "_Py_BuildValue", referenced from:
      _f2py_rout__fftpack_destroy_dct1_cache in _fftpackmodule.o

(...cut for brevity...)

  "_PyErr_Print", referenced from:
      _init_fftpack in _fftpackmodule.o
      _F2PyDict_SetItemString in fortranobject.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
error: Command "/usr/local/bin/gfortran -Wall -lgfortran -arch x86_64
build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6/scipy/fftpack/_fftpackmodule.o
build/temp.macosx-10.6-intel-2.6/scipy/fftpack/src/zfft.o
build/temp.macosx-10.6-intel-2.6/scipy/fftpack/src/drfft.o
build/temp.macosx-10.6-intel-2.6/scipy/fftpack/src/zrfft.o
build/temp.macosx-10.6-intel-2.6/scipy/fftpack/src/zfftnd.o
build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6/scipy/fftpack/src/dct.o
build/temp.macosx-10.6-intel-2.6/build/src.macosx-10.6-intel-2.6/fortranobject.o
-L/usr/local/lib/gcc/i686-apple-darwin8/4.2.3/x86_64
-Lbuild/temp.macosx-10.6-intel-2.6 -ldfftpack -lfftpack -lgfortran -o
build/lib.macosx-10.6-intel-2.6/scipy/fftpack/_fftpack.so" failed with
exit status 1

My numpy setup information:
$ python -c 'from numpy.f2py.diagnose import run; run()'
------
os.name='posix'
------
sys.platform='darwin'
------
sys.version:
2.6.4 (r264:75706, Feb  1 2010, 13:22:43)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)]
------
sys.prefix:
/Library/Frameworks/Python.framework/Versions/2.6
------
sys.path=':/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pip-0.6.3-py2.6.egg:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/readline-2.6.4-py2.6-macosx-10.6-intel.egg:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pytz-2010b-py2.6.egg:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/h5py-1.2.0-py2.6-macosx-10.6-intel.egg:/Users/adam/Downloads/yt-i386/src/yt-trunk-svn:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pyregion-0.1c-py2.6-macosx-10.6-intel.egg:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pyparsing-1.5.2-py2.6.egg:/Users/adam/Downloads/gildas-exe-jul09a/pc-darwin-g95/python:/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload:/Users/adam/.local/lib/python2.6/site-packages:/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages'
------
Failed to import Numeric: No module named Numeric
Failed to import numarray: No module named numarray
Found new numpy version '1.4.0rc2' in
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/__init__.pyc
Found f2py2e version '2_8112' in
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/f2py/f2py2e.pyc
Found numpy.distutils version '0.4.0' in
'/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/distutils/__init__.pyc'
------
Importing numpy.distutils.fcompiler ... ok
------
Checking availability of supported Fortran compilers:
Gnu95FCompiler instance properties:
  archiver        = ['/usr/local/bin/gfortran', '-cr']
  compile_switch  = '-c'
  compiler_f77    = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '-
                    fno-second-underscore', '-arch', 'ppc', '-arch', 'i686', '
                    -arch', 'x86_64', '-fPIC', '-O3', '-funroll-loops']
  compiler_f90    = ['/usr/local/bin/gfortran', '-Wall', '-fno-second-
                    underscore', '-arch', 'ppc', '-arch', 'i686', '-arch',
                    'x86_64', '-fPIC', '-O3', '-funroll-loops']
  compiler_fix    = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '-
                    fno-second-underscore', '-Wall', '-fno-second-underscore',
                    '-arch', 'ppc', '-arch', 'i686', '-arch', 'x86_64', '-
                    fPIC', '-O3', '-funroll-loops']
  libraries       = ['gfortran']
  library_dirs    = []
  linker_exe      = ['/usr/local/bin/gfortran', '-Wall', '-Wall']
  linker_so       = ['/usr/local/bin/gfortran', '-Wall', '-arch', 'ppc', '-
                    arch', 'i686', '-arch', 'x86_64', '-Wall', '-undefined',
                    'dynamic_lookup', '-bundle']
  object_switch   = '-o '
  ranlib          = ['/usr/local/bin/gfortran']
  version         = LooseVersion ('4.2.3')
  version_cmd     = ['/usr/local/bin/gfortran', '--version']
G95FCompiler instance properties:
  archiver        = None
  compile_switch  = '-c'
  compiler_f77    = ['/usr/local/bin/g95', '-ffixed-form', '-fno-second-
                    underscore', '-O']
  compiler_f90    = ['/usr/local/bin/g95', '-fno-second-underscore', '-O']
  compiler_fix    = ['/usr/local/bin/g95', '-ffixed-form', '-fno-second-
                    underscore', '-O']
  libraries       = []
  library_dirs    = []
  linker_exe      = None
  linker_so       = ['/usr/local/bin/g95', '-shared', '-shared']
  object_switch   = '-o '
  ranlib          = None
  version         = LooseVersion ('0.92')
  version_cmd     = ['/usr/local/bin/g95', '--version']
Fortran compilers found:
  --fcompiler=g95    G95 Fortran Compiler (0.92)
  --fcompiler=gnu95  GNU Fortran 95 compiler (4.2.3)
Compilers available for this platform, but not found:
  --fcompiler=absoft  Absoft Corp Fortran Compiler
  --fcompiler=gnu     GNU Fortran 77 compiler
  --fcompiler=ibm     IBM XL Fortran Compiler
  --fcompiler=intel   Intel Fortran Compiler for 32-bit apps
  --fcompiler=nag     NAGWare Fortran 95 Compiler
Compilers not available on this platform:
  --fcompiler=compaq   Compaq Fortran Compiler
  --fcompiler=hpux     HP Fortran 90 Compiler
  --fcompiler=intele   Intel Fortran Compiler for Itanium apps
  --fcompiler=intelem  Intel Fortran Compiler for EM64T-based apps
  --fcompiler=intelev  Intel Visual Fortran Compiler for Itanium apps
  --fcompiler=intelv   Intel Visual Fortran Compiler for 32-bit apps
  --fcompiler=lahey    Lahey/Fujitsu Fortran 95 Compiler
  --fcompiler=mips     MIPSpro Fortran Compiler
  --fcompiler=none     Fake Fortran compiler
  --fcompiler=pg       Portland Group Fortran Compiler
  --fcompiler=sun      Sun or Forte Fortran 95 Compiler
  --fcompiler=vast     Pacific-Sierra Research Fortran 90 Compiler
For compiler details, run 'config_fc --verbose' setup command.
------
Importing numpy.distutils.cpuinfo ... ok
------
CPU information: CPUInfoBase__get_nbits getNCPUs is_64bit is_i386 ------


Thanks,
-- 
Adam Ginsburg
Graduate Student
Center for Astrophysics and Space Astronomy
University of Colorado at Boulder
http://casa.colorado.edu/~ginsbura/


From cycomanic at gmail.com  Mon Feb 15 17:23:52 2010
From: cycomanic at gmail.com (Jochen Schroeder)
Date: Tue, 16 Feb 2010 09:23:52 +1100
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <4B792D59.2040608@silveregg.co.jp>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com>
	<20100215102007.GA3762@jschrod-laptop>
	<bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com>
	<4B79261F.6060706@silveregg.co.jp>
	<bc657ead1002150306l228a4748u1f509e672ef5a57f@mail.gmail.com>
	<4B792D59.2040608@silveregg.co.jp>
Message-ID: <20100215222350.GA2759@cudos0803>

On 02/15/10 20:17, David Cournapeau wrote:
> Sebastian Haase wrote:
> > On Mon, Feb 15, 2010 at 11:46 AM, David Cournapeau
> > <david at silveregg.co.jp> wrote:
> >> Sebastian Haase wrote:
> >>
> >>> Has this changed from FFTW2 to FFTW3 ?
> >>> It would really limit the use of plans, and make overall FFTs much
> >>> slower. In my specific case I very often have 512x512 single-precision
> >>> real arrays (images), that I would do ffts over and over again. But
> >>> the pointers would change ....
> >> You can, but you need to use the advanced plan API, or use the recently
> >> added new-array execute function:
> >>
> >> http://www.fftw.org/fftw3_doc/New_002darray-Execute-Functions.html#New_002darray-Execute-Functions
> >>
> > so it sounds like the alignment is the "killer" argument for the whole idea:
> > quote
> 
> Well, yes, you need aligned pointers, there is no way around it if you 
> want to (significantly) benefit from SSE - that's why I proposed some 
> time ago now an aligned allocator to be used inside NumPy, so that many 
> numpy arrays would be aligned by default.
> 
I still think this is a very good idea. What were the main objections around
this at the time?

> Note that you can align them by yourself if you want to (there are 
> several recipes on how to do that, one from Travis on Enthought blog 
> IIRC, and one from Anne in the NumPy ML). Or explicitly create plans for 
> unaligned arrays (this is significantly slower, though, but should be at 
> least as fast as fftw2).

There is a function in pyfftw to create aligned arrays, and it does cause a
significant performance benefit to use aligned arrays.

> 
> Also, most arrays allocated by malloc are *not* 16 bytes aligned on 
> Linux, because for allocated areas above a certain size, the glibc 
> malloc use mmap, and always "disalign" the allocated buffer. The 
> threshold is easily reached when working with big data.

Just to clarify this is 32bit Linux, on 64bit malloc automatically aligns to
16bytes.

> 
> > I guess this is really all new with version 3 of FFTW. I hope that
> > "reating a new plan is quick once one exists for a given size" means
> > "neglectable" for 512x512 arrays !?
> 
> You would have to test, but IIRC, the cost is not negligeable. Creating 
> an API around those plans should not be very difficult - at worse, you 
> can take a look at how scipy used to do it when scipy was supporting 
> FFTW backend. The problem is designing a fast API - especially for small 
> size arrays (~ 2**10), fft is so fast that you cannot afford a lot while 
> looking for cached plans :)

Well I looked at creating a more "traditional" API around fftw (something like
y=fft(x)) but the performance benefit for relatively small arrays (in my
experience ~2**12) was mainly eaten up by the creation of the output array.
Because most of the stuff I do uses arrays of around that size and does a lot of ffts
back and forth between two arrays (pulse propagation simulations if anyone is
interested), I went with the current approach (it's probably possible to create
some sort of memory pool to avoid the time of allocating arrays, is there
something like this in numpy already?) 

Cheers
Jochen

> 
> cheers,
> 
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From cycomanic at gmail.com  Mon Feb 15 17:27:11 2010
From: cycomanic at gmail.com (Jochen Schroeder)
Date: Tue, 16 Feb 2010 09:27:11 +1100
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <4B79261F.6060706@silveregg.co.jp>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com>
	<20100215102007.GA3762@jschrod-laptop>
	<bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com>
	<4B79261F.6060706@silveregg.co.jp>
Message-ID: <20100215222709.GB2759@cudos0803>

On 02/15/10 19:46, David Cournapeau wrote:
> Sebastian Haase wrote:
> 
> > 
> > Has this changed from FFTW2 to FFTW3 ?
> > It would really limit the use of plans, and make overall FFTs much
> > slower. In my specific case I very often have 512x512 single-precision
> > real arrays (images), that I would do ffts over and over again. But
> > the pointers would change ....
> 
> You can, but you need to use the advanced plan API, or use the recently 
> added new-array execute function:
> 
> http://www.fftw.org/fftw3_doc/New_002darray-Execute-Functions.html#New_002darray-Execute-Functions

Ah I didn't see that yet. Well there's also the guru_execute_dft function which
pyfftw supports. So if you created a plan for an in and out array, you can use
Plan.execute_guru_dft(in2, out2), provided that in2 and out2 are the same type
and size and have the same alignment as in and out.

Cheers
Jochen

> 
> cheers,
> 
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From david at silveregg.co.jp  Mon Feb 15 22:10:12 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Tue, 16 Feb 2010 12:10:12 +0900
Subject: [SciPy-User] Install fail on Mac 10.6.2 in fftpack._fftpack
In-Reply-To: <fceb3d4a1002151422m7759ec86k7d279c6cfa717003@mail.gmail.com>
References: <fceb3d4a1002151422m7759ec86k7d279c6cfa717003@mail.gmail.com>
Message-ID: <4B7A0C94.60607@silveregg.co.jp>

Adam Ginsburg wrote:
> Hi, I'm trying to get scipy running on my new Snow Leopard system.  I
> was able to get scipy built successfully yesterday, but not so much
> today.  I'd appreciate any assistance in getting scipy installed.
> 
> I built fftw with the following setup:
> ./configure CC="gcc -arch i386 -arch x86_64" CXX="g++ -arch i386 -arch
> x86_64" CPP="gcc -E" CXXCPP="g++ -E"
> make
> sudo make install
> 
> I've also tried multiple variations of architection for fftw (e.g. not
> using -arch i386 and passing no compiler flags). I built UMFTPACK
> following the instructions on: http://blog.hyperjeff.net/?p=160

Neither UMFPACK nor fftw are needed for scipy anymore, but that's not 
why your build fails.

You should unset LDFLAGS, CFLAGS, FFLAGS and CXXFLAGS when building 
scipy: setting them up does not do what you think it does,

cheers,

David


From m.abdollahi at gmail.com  Tue Feb 16 03:47:20 2010
From: m.abdollahi at gmail.com (Mohammad Abdollahi)
Date: Tue, 16 Feb 2010 09:47:20 +0100
Subject: [SciPy-User] audio recording GUI ?
Message-ID: <f72d1a261002160047u63136fccw97a7ac1a7087d4ec@mail.gmail.com>

Hi all,

I'm trying to make a user interface for audio recording in a way that i show
user a sentence and he will start recording his voice by pressing the button
Start for example and after he finished reading the sentence he presses the
button STOP. i was looking into pyaudio for the audio interface but
apparently i always have to specify a fixed duration for recording a sound
and it can not be user -dependent or in another words left-to-user. does
anybody know aNYWAY AROUND THAT ? besides do you have recommendations for
sucha GUI platform, which package ?

thanks inadvance

regards
M.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100216/baca19f5/attachment.html>

From d.l.goldsmith at gmail.com  Tue Feb 16 04:20:56 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Tue, 16 Feb 2010 01:20:56 -0800
Subject: [SciPy-User] audio recording GUI ?
In-Reply-To: <f72d1a261002160047u63136fccw97a7ac1a7087d4ec@mail.gmail.com>
References: <f72d1a261002160047u63136fccw97a7ac1a7087d4ec@mail.gmail.com>
Message-ID: <45d1ab481002160120x4432e1cgde996e0e90e16bd2@mail.gmail.com>

It's probably much more than you need, but you might take a look at
Audacity: audacity.sourceforge.net; the beta version (1.3.11) includes some
built-in scripting, and if nothing else, you could post a similar query on
their forum - I'm sure people there would have some good suggestions.

DG

On Tue, Feb 16, 2010 at 12:47 AM, Mohammad Abdollahi
<m.abdollahi at gmail.com>wrote:

> Hi all,
>
> I'm trying to make a user interface for audio recording in a way that i
> show user a sentence and he will start recording his voice by pressing the
> button Start for example and after he finished reading the sentence he
> presses the button STOP. i was looking into pyaudio for the audio interface
> but apparently i always have to specify a fixed duration for recording a
> sound and it can not be user -dependent or in another words left-to-user.
> does anybody know aNYWAY AROUND THAT ? besides do you have recommendations
> for sucha GUI platform, which package ?
>
> thanks inadvance
>
> regards
> M.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100216/f26ae392/attachment.html>

From seb.haase at gmail.com  Tue Feb 16 04:34:43 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Tue, 16 Feb 2010 10:34:43 +0100
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <20100215222350.GA2759@cudos0803>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com> 
	<20100215102007.GA3762@jschrod-laptop>
	<bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com> 
	<4B79261F.6060706@silveregg.co.jp>
	<bc657ead1002150306l228a4748u1f509e672ef5a57f@mail.gmail.com> 
	<4B792D59.2040608@silveregg.co.jp> <20100215222350.GA2759@cudos0803>
Message-ID: <bc657ead1002160134r2aba1058w3e0426d5aefa4070@mail.gmail.com>

On Mon, Feb 15, 2010 at 11:23 PM, Jochen Schroeder <cycomanic at gmail.com> wrote:
<snip>
>> Also, most arrays allocated by malloc are *not* 16 bytes aligned on
>> Linux, because for allocated areas above a certain size, the glibc
>> malloc use mmap, and always "disalign" the allocated buffer. The
>> threshold is easily reached when working with big data.
>
> Just to clarify this is 32bit Linux, on 64bit malloc automatically aligns to
> 16bytes.

thanks, for the info!

> Well I looked at creating a more "traditional" API around fftw (something like
> y=fft(x)) but the performance benefit for relatively small arrays (in my
> experience ~2**12) was mainly eaten up by the creation of the output array.
> Because most of the stuff I do uses arrays of around that size and does a lot of ffts
> back and forth between two arrays (pulse propagation simulations if anyone is
> interested), I went with the current approach (it's probably possible to create
> some sort of memory pool to avoid the time of allocating arrays, is there
> something like this in numpy already?)
>
You should add an `out=None` option to fft, so that if `out` is not
`None` it only checks the shape, dtype and contiguity requirements.
At least that's what I did for my (old, fftw2) frontend.

-Sebastian


From cournape at gmail.com  Tue Feb 16 05:37:14 2010
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 16 Feb 2010 19:37:14 +0900
Subject: [SciPy-User] [ANN] pyfftw-0.2 released
In-Reply-To: <20100215222350.GA2759@cudos0803>
References: <20100215040339.GA2115@cudos0803>
	<bc657ead1002150135v35ce872dw8c1528030271099e@mail.gmail.com>
	<20100215102007.GA3762@jschrod-laptop>
	<bc657ead1002150227o464d1311nac1ffa1984d4c4ff@mail.gmail.com>
	<4B79261F.6060706@silveregg.co.jp>
	<bc657ead1002150306l228a4748u1f509e672ef5a57f@mail.gmail.com>
	<4B792D59.2040608@silveregg.co.jp> <20100215222350.GA2759@cudos0803>
Message-ID: <5b8d13221002160237x2af2c7fficbe9e28172f0e4a7@mail.gmail.com>

On Tue, Feb 16, 2010 at 7:23 AM, Jochen Schroeder <cycomanic at gmail.com> wrote:

> I still think this is a very good idea. What were the main objections around
> this at the time?

No real objection per se, but IIRC, it was made complicated for
platforms without an aligned memory allocator. For example, for
platforms with posix_memalign, you can free the returned memory block
with the standard free. If you implement your own aligned memory
allocator (which is easy), you need to match with your own free.

The only sane way to do that in NumPy is to *always* allocate with an
aligned allocator. Doing so is not easy because of realloc (an aligned
realloc is impossible to implement without going into the libc malloc
internals). So the idea was to ban realloc inside numpy altogether
(there are only a few realloc calls in numpy), but I never took the
time to look more into it.

>
> Well I looked at creating a more "traditional" API around fftw (something like
> y=fft(x)) but the performance benefit for relatively small arrays (in my
> experience ~2**12) was mainly eaten up by the creation of the output array.

It depends on the application, of course, but in audio processing, you
generally takes M fft of M signals of size N (so you do ff(x) with x a
2d array, and the fft is done on each row or column). So this is not a
big issue (as long as you can fit the whole signal in memory, that
is).

I think providing a "high" level API ala scipy.fftpack should be
doable - actually, having a scikits which can be used as a drop-in for
scipy.fftpack but using fftw would be very useful,

cheers,

David


From zallen at urologicresearchinstitute.org  Tue Feb 16 07:00:57 2010
From: zallen at urologicresearchinstitute.org (URI)
Date: Tue, 16 Feb 2010 12:00:57 +0000 (UTC)
Subject: [SciPy-User] creating a 3D surface plot from collected data
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
Message-ID: <loom.20100216T124608-106@post.gmane.org>

>Joe Kington <jkington <at> wisc.edu> writes:
>
> 
> 
> Are your data already gridded? Or are they irregularly spaced?? Seeing as how
you mentioned "real" data, I'm assuming they're irregularly spaced.The iy, ix
example assumes that you have a regular grid that has been exported as an x,y,z
file instead of a grid of z coordinates.If they're not already a regular grid,
you'll need to interpolate them onto a regular grid before using any of the 3D
surface plotting routines.(I'll skip my "interpolation is an artform" rant,
here...? For a quick look splines are fine.)Look into using the various
functions in sp.interpolate.? The example I threw together below uses a
thin-plate spline, but there are lots of other options.import numpy as npimport
scipy as spimport matplotlib.pyplot as pltimport scipy.interpolatefrom
mpl_toolkits.mplot3d import Axes3Dx = np.random.random(10)y =
np.random.random(10)z = np.random.random(10)spline =
sp.interpolate.Rbf(x,y,z,function='thin-plate')xi = np.linspace(x.min(),
x.max(), 50)yi = np.linspace(y.min(), y.max(), 50)xi, yi = np.meshgrid(xi, yi)zi
= spline(xi,yi)
> fig = plt.figure()ax = Axes3D(fig)ax.plot_surface(xi,yi,zi)plt.show()Hope that
helps,-Joe



Joe-

This is exactly what I'm looking for, thanks.  The problem is that when I read
in my data I get these errors:

Traceback (most recent call last):
  File "C:\Python26\Lib\site-packages\xy\plotscript_griddata.py", line 33, in
<module>
    spline = sp.interpolate.Rbf(xlist,ylist,zlist,function='thin-plate')
  File "C:\Python26\lib\site-packages\scipy\interpolate\rbf.py", line 138, in
__init__
    self.nodes = linalg.solve(self.A, self.di)
  File "C:\Python26\lib\site-packages\scipy\linalg\basic.py", line 151, in solve
    raise LinAlgError, "singular matrix"
LinAlgError: singular matrix


The data I'm using is 2560 x,y,z grid points representing an anatomical contour
exported from a medical imaging program.  None of the values are zero and it's
not some crazy shape - basically it looks like a kidney, so no spike or holes or
anything.  I've found that if I chop the data down to 144 lines it will then
work (not that 144 is the exact cutoff, I just kept deleting chunks of lines
from the data file until it worked).  This tells me that I've successfully
adapted your code and that I'm reading in the data properly as numpy arrays - I
was worried that I might just have some formatting problems.

Any idea why I can't run this with my entire data set?  This is just a test run,
I will potentially need to process much larger datasets in the future.

P.S. Sorry for the bottom-post, the website won't let me submit unless I post my
message below yours.  Seems stupid to me, I think it makes a lot more sense to
see the new information at the top instead of having to scroll through a bunch
of stuff I've already read.




From zallen at urologicresearchinstitute.org  Tue Feb 16 08:10:16 2010
From: zallen at urologicresearchinstitute.org (URI)
Date: Tue, 16 Feb 2010 13:10:16 +0000 (UTC)
Subject: [SciPy-User] VTK examples
Message-ID: <loom.20100216T140738-325@post.gmane.org>

Can anyone get the example on this page to work?

http://www.scipy.org/Cookbook/Matplotlib/VTK_Integration

When I run it (in Windows 7) a window opens up but then I get a "Python has
stopped working" dialoge.

It seems to have a problem with the second to last line:
iren.Initialize()



From baker.alexander at gmail.com  Tue Feb 16 08:25:21 2010
From: baker.alexander at gmail.com (alexander baker)
Date: Tue, 16 Feb 2010 13:25:21 +0000
Subject: [SciPy-User] VTK examples
In-Reply-To: <loom.20100216T140738-325@post.gmane.org>
References: <loom.20100216T140738-325@post.gmane.org>
Message-ID: <270620221002160525t7348936s802884260dc55ebf@mail.gmail.com>

Works fine for me! (XP)

from vtk import *
   2
   3 import matplotlib
   4 matplotlib.use('Agg')
   5 from matplotlib.figure import Figure
   6 from matplotlib.backends.backend_agg import FigureCanvasAgg
   7 import pylab as p
   8
   9 # The vtkImageImporter will treat a python string as a void pointer
  10 importer = vtkImageImport()
  11 importer.SetDataScalarTypeToUnsignedChar()
  12 importer.SetNumberOfScalarComponents(4)
  13
  14 # It's upside-down when loaded, so add a flip filter
  15 imflip = vtkImageFlip()
  16 imflip.SetInput(importer.GetOutput())
  17 imflip.SetFilteredAxis(1)
  18
  19 # Map the plot as a texture on a cube
  20 cube = vtkCubeSource()
  21
  22 cubeMapper = vtkPolyDataMapper()
  23 cubeMapper.SetInput(cube.GetOutput())
  24
  25 cubeActor = vtkActor()
  26 cubeActor.SetMapper(cubeMapper)
  27
  28 # Create a texture based off of the image
  29 cubeTexture = vtkTexture()
  30 cubeTexture.InterpolateOn()
  31 cubeTexture.SetInput(imflip.GetOutput())
  32 cubeActor.SetTexture(cubeTexture)
  33
  34 ren = vtkRenderer()
  35 ren.AddActor(cubeActor)
  36
  37 renWin = vtkRenderWindow()
  38 renWin.AddRenderer(ren)
  39
  40 iren = vtkRenderWindowInteractor()
  41 iren.SetRenderWindow(renWin)
  42
  43 # Now create our plot
  44 fig = Figure()
  45 canvas = FigureCanvasAgg(fig)
  46 ax = fig.add_subplot(111)
  47 ax.grid(True)
  48 ax.set_xlabel('Hello from VTK!', size=16)
  49 ax.bar(xrange(10), p.rand(10))
  50
  51 # Powers of 2 image to be clean
  52 w,h = 1024, 1024
  53 dpi = canvas.figure.get_dpi()
  54 fig.set_figsize_inches(w / dpi, h / dpi)
  55 canvas.draw() # force a draw
  56
  57 # This is where we tell the image importer about the mpl image
  58 extent = (0, w - 1, 0, h - 1, 0, 0)
  59 importer.SetWholeExtent(extent)
  60 importer.SetDataExtent(extent)
  61 importer.SetImportVoidPointer(canvas.buffer_rgba(0,0), 1)
  62 importer.Update()
  63
  64 iren.Initialize()
  65 iren.Start()


Mobile: 07788 872118
Blog: www.alexfb.com

--
All science is either physics or stamp collecting.


On 16 February 2010 13:10, URI <zallen at urologicresearchinstitute.org> wrote:

> Can anyone get the example on this page to work?
>
> http://www.scipy.org/Cookbook/Matplotlib/VTK_Integration
>
> When I run it (in Windows 7) a window opens up but then I get a "Python has
> stopped working" dialoge.
>
> It seems to have a problem with the second to last line:
> iren.Initialize()
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100216/89a6e171/attachment.html>

From zallen at urologicresearchinstitute.org  Tue Feb 16 09:02:48 2010
From: zallen at urologicresearchinstitute.org (URI)
Date: Tue, 16 Feb 2010 14:02:48 +0000 (UTC)
Subject: [SciPy-User] VTK examples
References: <loom.20100216T140738-325@post.gmane.org>
	<270620221002160525t7348936s802884260dc55ebf@mail.gmail.com>
Message-ID: <loom.20100216T150054-887@post.gmane.org>

alexander baker <baker.alexander <at> gmail.com> writes:

> 
> 
> Works fine for me! (XP)

Alex-

What method of installation did you use?  I used Python (x,y):
http://www.pythonxy.com/

The installer included a VTK module.  I just checked the box next to it during
installation, I don't think I am supposed to do anything further to make it work.



From josef.pktd at gmail.com  Tue Feb 16 09:06:55 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 09:06:55 -0500
Subject: [SciPy-User] VTK examples
In-Reply-To: <270620221002160525t7348936s802884260dc55ebf@mail.gmail.com>
References: <loom.20100216T140738-325@post.gmane.org>
	<270620221002160525t7348936s802884260dc55ebf@mail.gmail.com>
Message-ID: <1cd32cbb1002160606n5e77507dl468fe408983908a5@mail.gmail.com>

On Tue, Feb 16, 2010 at 8:25 AM, alexander baker
<baker.alexander at gmail.com> wrote:
> Works fine for me! (XP)

What version of numpy and matplotlib are you using?

The script also crashes on my WindowsXP.
But I'm not too surprised since I have a mixture of numpy 1.3 and
numpy 1.4 compiled extensions in my site-packages.

Josef

>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From jkington at wisc.edu  Tue Feb 16 09:23:54 2010
From: jkington at wisc.edu (Joe Kington)
Date: Tue, 16 Feb 2010 08:23:54 -0600
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <loom.20100216T124608-106@post.gmane.org>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
Message-ID: <f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>

No worries about the bottom post. I actually think I'm breaking protocol by
top-posting, but I use gmail, and it's just more natural to top-post.

Anyway, I think your problem is coming from the fact that you're working
with an isosurface that fully encloses a volume. I assumed that you had
scattered data points (say, elevation measurements) that you needed to
interpolate between.

The interpolation example I posted implicitly assumes that z = f(x,y). In
other words, that there is only one z value for any given x and y.  If
you're working with a 3D kidney-shape, this is clearly not the case.

Therefore, you're getting a singluar matrix when you try to interpolate
(more than one value of z for identical rows of x and y in the matrix).

The good news is that this may make the problem much simpler. (Your data is
already in some sort of triangle-strip format if it's been exported from an
imaging program. No need to interpolate.)

What happens when you just do:

fig = plt.figure()
ax = Axes3D(fig)
ax.plot_surface(x,y,z)
plt.show()

Where x,y,z are your raw data?  There's a reasonable chance that the program
you used to export the x,y,z isosurface left the verticies in the order that
plot_surface expects them in...

On a side note, you might also look into Mayavi's mlab module for 3D
plotting<http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/mlab.html>.
I prefer it to matplotlib's native 3D plotting.
If you have it installed, you'd just do:

from enthought.mayavi import mlab
s = mlab.mesh(x, y, z)
mlab.show()

Hope that helps some,
-Joe








On Tue, Feb 16, 2010 at 6:00 AM, URI
<zallen at urologicresearchinstitute.org>wrote:

> >Joe Kington <jkington <at> wisc.edu> writes:
> >
> >
> >
> > Are your data already gridded? Or are they irregularly spaced?  Seeing as
> how
> you mentioned "real" data, I'm assuming they're irregularly spaced.The iy,
> ix
> example assumes that you have a regular grid that has been exported as an
> x,y,z
> file instead of a grid of z coordinates.If they're not already a regular
> grid,
> you'll need to interpolate them onto a regular grid before using any of the
> 3D
> surface plotting routines.(I'll skip my "interpolation is an artform" rant,
> here...  For a quick look splines are fine.)Look into using the various
> functions in sp.interpolate.  The example I threw together below uses a
> thin-plate spline, but there are lots of other options.import numpy as
> npimport
> scipy as spimport matplotlib.pyplot as pltimport scipy.interpolatefrom
> mpl_toolkits.mplot3d import Axes3Dx = np.random.random(10)y =
> np.random.random(10)z = np.random.random(10)spline =
> sp.interpolate.Rbf(x,y,z,function='thin-plate')xi = np.linspace(x.min(),
> x.max(), 50)yi = np.linspace(y.min(), y.max(), 50)xi, yi = np.meshgrid(xi,
> yi)zi
> = spline(xi,yi)
> > fig = plt.figure()ax = Axes3D(fig)ax.plot_surface(xi,yi,zi)plt.show()Hope
> that
> helps,-Joe
>
>
>
> Joe-
>
> This is exactly what I'm looking for, thanks.  The problem is that when I
> read
> in my data I get these errors:
>
> Traceback (most recent call last):
>  File "C:\Python26\Lib\site-packages\xy\plotscript_griddata.py", line 33,
> in
> <module>
>    spline = sp.interpolate.Rbf(xlist,ylist,zlist,function='thin-plate')
>  File "C:\Python26\lib\site-packages\scipy\interpolate\rbf.py", line 138,
> in
> __init__
>    self.nodes = linalg.solve(self.A, self.di)
>  File "C:\Python26\lib\site-packages\scipy\linalg\basic.py", line 151, in
> solve
>    raise LinAlgError, "singular matrix"
> LinAlgError: singular matrix
>
>
> The data I'm using is 2560 x,y,z grid points representing an anatomical
> contour
> exported from a medical imaging program.  None of the values are zero and
> it's
> not some crazy shape - basically it looks like a kidney, so no spike or
> holes or
> anything.  I've found that if I chop the data down to 144 lines it will
> then
> work (not that 144 is the exact cutoff, I just kept deleting chunks of
> lines
> from the data file until it worked).  This tells me that I've successfully
> adapted your code and that I'm reading in the data properly as numpy arrays
> - I
> was worried that I might just have some formatting problems.
>
> Any idea why I can't run this with my entire data set?  This is just a test
> run,
> I will potentially need to process much larger datasets in the future.
>
> P.S. Sorry for the bottom-post, the website won't let me submit unless I
> post my
> message below yours.  Seems stupid to me, I think it makes a lot more sense
> to
> see the new information at the top instead of having to scroll through a
> bunch
> of stuff I've already read.
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100216/714b6ffb/attachment.html>

From denis-bz-gg at t-online.de  Tue Feb 16 09:29:40 2010
From: denis-bz-gg at t-online.de (denis)
Date: Tue, 16 Feb 2010 06:29:40 -0800 (PST)
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <loom.20100216T124608-106@post.gmane.org>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com> 
	<loom.20100216T124608-106@post.gmane.org>
Message-ID: <ed479af6-20c2-4e28-9bed-de3ac9188df0@v25g2000yqk.googlegroups.com>

URI,
  can you try these --
    spline = sp.interpolate.Rbf(x,y,z,function='thin-plate',
smooth=1 )
    spline = sp.interpolate.Rbf(x,y,z,function='gaussian', smooth=1 )

pydoc scipy.interpolate.Rbf says
|  smooth : float, optional
|      Values greater than zero increase the smoothness of the
|      approximation.  0 is for interpolation (default), the function
will
|      always go through the nodal points in this case.

I have no idea if "smooth=1" is reasonable, but 0 is not a good
default -- Joe ?

cheers
  -- denis


From zallen at urologicresearchinstitute.org  Tue Feb 16 10:06:34 2010
From: zallen at urologicresearchinstitute.org (URI)
Date: Tue, 16 Feb 2010 15:06:34 +0000 (UTC)
Subject: [SciPy-User] creating a 3D surface plot from collected data
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
Message-ID: <loom.20100216T160250-221@post.gmane.org>

Joe Kington <jkington <at> wisc.edu> writes:

> 
> 
> No worries about the bottom post. I actually think I'm breaking protocol by
top-posting, but I use gmail, and it's just more natural to top-post.Anyway, I
think your problem is coming from the fact that you're working with an
isosurface that fully encloses a volume. I assumed that you had scattered data
points (say, elevation measurements) that you needed to interpolate between.?
The interpolation example I posted implicitly assumes that z = f(x,y). In other
words, that there is only one z value for any given x and y.? If you're working
with a 3D kidney-shape, this is clearly not the case.? Therefore, you're getting
a singluar matrix when you try to interpolate (more than one value of z for
identical rows of x and y in the matrix).The good news is that this may make the
problem much simpler. (Your data is already in some sort of triangle-strip
format if it's been exported from an imaging program. No need to
interpolate.)What happens when you just do:fig = plt.figure()ax =
Axes3D(fig)ax.plot_surface(x,y,z)plt.show()Where x,y,z are your raw data??
There's a reasonable chance that the program you used to export the x,y,z
isosurface left the verticies in the order that plot_surface expects them
in...On a side note, you might also look into Mayavi's mlab module for 3D
plotting.? I prefer it to matplotlib's native 3D plotting. If you have it
installed, you'd just do:from enthought.mayavi import mlabs = mlab.mesh(x, y,
z)mlab.show()Hope that helps some,
> -Joe



I get these errors:

Traceback (most recent call last):
  File "C:\Python26\Lib\site-packages\xy\plotscript_griddata.py", line 61, in
<module>
    ax.plot_surface(x,y,z)
  File "C:\Python26\lib\site-packages\mpl_toolkits\mplot3d\axes3d.py", line 573,
in plot_surface
    rows, cols = Z.shape
ValueError: need more than 1 value to unpack

Maybe Mayavi is the way to go, I did come across it while researching this
problem.  I was worried that I couldn't install it alongside my current
installation, these things always seems to conflict with each other when I try
to have multiple packages installed.

"it's just more natural to top-post" - I completely agree.  I am using the web
interface to post so the admins have more control over what I'm doing.  Maybe I
should start using an email interface.



From rob.clewley at gmail.com  Tue Feb 16 10:19:32 2010
From: rob.clewley at gmail.com (Rob Clewley)
Date: Tue, 16 Feb 2010 10:19:32 -0500
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
Message-ID: <a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>

Joe,

On Tue, Feb 16, 2010 at 9:23 AM, Joe Kington <jkington at wisc.edu> wrote:
> The interpolation example I posted implicitly assumes that z = f(x,y). In
> other words, that there is only one z value for any given x and y.  If
> you're working with a 3D kidney-shape, this is clearly not the case.
>

I too have some data that has multiple z values in the x,y  domain.
Worse, my data is not remotely arranged in a proper grid as it is
temporal trajectory data from a 3D ODE that winds itself round and
round the surface, like a very carefully wrapped mummy. I.e., it
doesn't wrap back over itself.

I'm not familiar with 3D plotting techniques and trying to set up the
plot of my surface in mayavi or matplotlib is not working along the
lines of standard cookbook examples. I cannot find appropriate
information online on how to plot the surface if I can't run the
interpolator for data that's not a function of (x,y) and my data needs
"alignment".

But maybe I'm looking in the wrong place. Could you provide any pointers please?

-Rob


From nwagner at iam.uni-stuttgart.de  Tue Feb 16 10:40:15 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 16 Feb 2010 16:40:15 +0100
Subject: [SciPy-User] Converting a tetrahedral mesh into a hexahedral mesh
Message-ID: <web-130385190@uni-stuttgart.de>

Hi all,

I am looking for a tool that converts a tetrahedral mesh 
into a hexahedral mesh.

The routine should split each tetrahedral element in a 
volume into 4 hexahedral
elements by splitting each edge and face at its midpoint, 
and then forming
connections to the center of the tet.

Any pointer would be appreciated.

  Thanks in advance

                                             Nils


From gael.varoquaux at normalesup.org  Tue Feb 16 11:07:27 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 16 Feb 2010 17:07:27 +0100
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
Message-ID: <20100216160727.GA1779@phare.normalesup.org>

On Tue, Feb 16, 2010 at 10:19:32AM -0500, Rob Clewley wrote:
> I too have some data that has multiple z values in the x,y  domain.
> Worse, my data is not remotely arranged in a proper grid as it is
> temporal trajectory data from a 3D ODE that winds itself round and
> round the surface, like a very carefully wrapped mummy. I.e., it
> doesn't wrap back over itself.

> I'm not familiar with 3D plotting techniques and trying to set up the
> plot of my surface in mayavi or matplotlib is not working along the
> lines of standard cookbook examples. I cannot find appropriate
> information online on how to plot the surface if I can't run the
> interpolator for data that's not a function of (x,y) and my data needs
> "alignment".

Would this example help:
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_surface_from_irregular_data.html

Ga?l


From dwf at cs.toronto.edu  Tue Feb 16 12:50:27 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 16 Feb 2010 12:50:27 -0500
Subject: [SciPy-User] audio recording GUI ?
In-Reply-To: <f72d1a261002160047u63136fccw97a7ac1a7087d4ec@mail.gmail.com>
References: <f72d1a261002160047u63136fccw97a7ac1a7087d4ec@mail.gmail.com>
Message-ID: <58D2B5A2-633D-4C7E-A8FA-D1BFCFA43216@cs.toronto.edu>

On 16-Feb-10, at 3:47 AM, Mohammad Abdollahi wrote:

> I'm trying to make a user interface for audio recording in a way  
> that i show user a sentence and he will start recording his voice by  
> pressing the button Start for example and after he finished reading  
> the sentence he presses the button STOP. i was looking into pyaudio  
> for the audio interface but apparently i always have to specify a  
> fixed duration for recording a sound and it can not be user - 
> dependent or in another words left-to-user. does anybody know aNYWAY  
> AROUND THAT ? besides do you have recommendations for sucha GUI  
> platform, which package ?

See this example of an audio recording GUI using Chaco:

https://svn.enthought.com/enthought/browser/Chaco/trunk/examples/advanced/spectrum.py

David


From zallen at urologicresearchinstitute.org  Tue Feb 16 12:57:50 2010
From: zallen at urologicresearchinstitute.org (URI)
Date: Tue, 16 Feb 2010 17:57:50 +0000 (UTC)
Subject: [SciPy-User] creating a 3D surface plot from collected data
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
Message-ID: <loom.20100216T185446-278@post.gmane.org>

Joe Kington <jkington <at> wisc.edu> writes:

> 
> 
> No worries about the bottom post. I actually think I'm breaking protocol by
top-posting, but I use gmail, and it's just more natural to top-post.Anyway, I
think your problem is coming from the fact that you're working with an
isosurface that fully encloses a volume. I assumed that you had scattered data
points (say, elevation measurements) that you needed to interpolate between.?
The interpolation example I posted implicitly assumes that z = f(x,y). In other
words, that there is only one z value for any given x and y.? If you're working
with a 3D kidney-shape, this is clearly not the case.? Therefore, you're getting
a singluar matrix when you try to interpolate (more than one value of z for
identical rows of x and y in the matrix).The good news is that this may make the
problem much simpler. (Your data is already in some sort of triangle-strip
format if it's been exported from an imaging program. No need to
interpolate.)What happens when you just do:fig = plt.figure()ax =
Axes3D(fig)ax.plot_surface(x,y,z)plt.show()Where x,y,z are your raw data??
There's a reasonable chance that the program you used to export the x,y,z
isosurface left the verticies in the order that plot_surface expects them
in...On a side note, you might also look into Mayavi's mlab module for 3D
plotting.? I prefer it to matplotlib's native 3D plotting. If you have it
installed, you'd just do:from enthought.mayavi import mlabs = mlab.mesh(x, y,
z)mlab.show()Hope that helps some,
> -Joe

I understand what you mean about assuming that z is a proper function of x and
y, but after looking again at the examples on the tutorial page
(http://matplotlib.sourceforge.net/mpl_toolkits/mplot3d/tutorial.html#getting-started)
I remember why I thought plot_surface was the proper function: the second
example under the plot_surface heading shows a spherical surface, so I figured
plot_surface didn't assume a 1-to-1 relationship.



From m.abdollahi at gmail.com  Tue Feb 16 12:59:43 2010
From: m.abdollahi at gmail.com (Mohammad Abdollahi)
Date: Tue, 16 Feb 2010 18:59:43 +0100
Subject: [SciPy-User] audio recording GUI ?
In-Reply-To: <58D2B5A2-633D-4C7E-A8FA-D1BFCFA43216@cs.toronto.edu>
References: <f72d1a261002160047u63136fccw97a7ac1a7087d4ec@mail.gmail.com>
	<58D2B5A2-633D-4C7E-A8FA-D1BFCFA43216@cs.toronto.edu>
Message-ID: <f72d1a261002160959j7e38c5em4196be26d6734476@mail.gmail.com>

thanks for the comment. actually i had seen that example before but i can
not run that code. I have installed chaco and all ETS packages but it always
gives me error when I try to run any of the examples including the one you
mentioned.
M.

On Tue, Feb 16, 2010 at 6:50 PM, David Warde-Farley <dwf at cs.toronto.edu>wrote:

> On 16-Feb-10, at 3:47 AM, Mohammad Abdollahi wrote:
>
> > I'm trying to make a user interface for audio recording in a way
> > that i show user a sentence and he will start recording his voice by
> > pressing the button Start for example and after he finished reading
> > the sentence he presses the button STOP. i was looking into pyaudio
> > for the audio interface but apparently i always have to specify a
> > fixed duration for recording a sound and it can not be user -
> > dependent or in another words left-to-user. does anybody know aNYWAY
> > AROUND THAT ? besides do you have recommendations for sucha GUI
> > platform, which package ?
>
> See this example of an audio recording GUI using Chaco:
>
>
> https://svn.enthought.com/enthought/browser/Chaco/trunk/examples/advanced/spectrum.py
>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100216/0661d239/attachment.html>

From jlconlin at gmail.com  Tue Feb 16 13:08:53 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Tue, 16 Feb 2010 11:08:53 -0700
Subject: [SciPy-User] How to fit data with errorbars
Message-ID: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>

I have some data with some errobars that I need to fit to a line.  Is
there anyway to use scipy.polyfit with the error associated with the
data?  If not, how can I make a fit routine work with my data?

Thanks,
Jeremy


From dwf at cs.toronto.edu  Tue Feb 16 13:36:48 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 16 Feb 2010 13:36:48 -0500
Subject: [SciPy-User] audio recording GUI ?
In-Reply-To: <f72d1a261002160959j7e38c5em4196be26d6734476@mail.gmail.com>
References: <f72d1a261002160047u63136fccw97a7ac1a7087d4ec@mail.gmail.com>
	<58D2B5A2-633D-4C7E-A8FA-D1BFCFA43216@cs.toronto.edu>
	<f72d1a261002160959j7e38c5em4196be26d6734476@mail.gmail.com>
Message-ID: <A55BE52B-DF61-410A-B5EA-86183B4E6EB2@cs.toronto.edu>


On 16-Feb-10, at 12:59 PM, Mohammad Abdollahi wrote:

> thanks for the comment. actually i had seen that example before but  
> i can
> not run that code. I have installed chaco and all ETS packages but  
> it always
> gives me error when I try to run any of the examples including the  
> one you
> mentioned.
> M.

You could try searching the archives for the enthought-dev mailing list:

https://mail.enthought.com/mailman/listinfo/enthought-dev

and if you have no luck, posting the specific error to that list.

David


From robert.kern at gmail.com  Tue Feb 16 13:39:14 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 16 Feb 2010 12:39:14 -0600
Subject: [SciPy-User] audio recording GUI ?
In-Reply-To: <A55BE52B-DF61-410A-B5EA-86183B4E6EB2@cs.toronto.edu>
References: <f72d1a261002160047u63136fccw97a7ac1a7087d4ec@mail.gmail.com> 
	<58D2B5A2-633D-4C7E-A8FA-D1BFCFA43216@cs.toronto.edu>
	<f72d1a261002160959j7e38c5em4196be26d6734476@mail.gmail.com> 
	<A55BE52B-DF61-410A-B5EA-86183B4E6EB2@cs.toronto.edu>
Message-ID: <3d375d731002161039n14cc09d1tcb7d45ef3c0bfcff@mail.gmail.com>

On Tue, Feb 16, 2010 at 12:36, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>
> On 16-Feb-10, at 12:59 PM, Mohammad Abdollahi wrote:
>
>> thanks for the comment. actually i had seen that example before but
>> i can
>> not run that code. I have installed chaco and all ETS packages but
>> it always
>> gives me error when I try to run any of the examples including the
>> one you
>> mentioned.
>> M.
>
> You could try searching the archives for the enthought-dev mailing list:
>
> https://mail.enthought.com/mailman/listinfo/enthought-dev
>
> and if you have no luck, posting the specific error to that list.

He has been.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From curiousjan at gmail.com  Tue Feb 16 14:04:33 2010
From: curiousjan at gmail.com (Jan Strube)
Date: Tue, 16 Feb 2010 19:04:33 +0000
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
Message-ID: <d7d1efcf1002161104u73360c78p9962eb9344d66519@mail.gmail.com>

Hi Jeremy,
I'm not sure about scipy, but please have a look at paida.
It's a pure python package that implements what you want.
http://paida.sourceforge.net/documentation/fitter/index.html

That's usually a decent package to have in your toolkit if you're doing data
analysis.

Hope that helps,
    Jan


On Tue, Feb 16, 2010 at 6:08 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:

> I have some data with some errobars that I need to fit to a line.  Is
> there anyway to use scipy.polyfit with the error associated with the
> data?  If not, how can I make a fit routine work with my data?
>
> Thanks,
> Jeremy
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100216/05a0b87d/attachment.html>

From jsseabold at gmail.com  Tue Feb 16 14:14:11 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 16 Feb 2010 14:14:11 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
Message-ID: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>

We are happy to announce the 0.2.0 (beta) release of scikits.statsmodels.
This is both a bug-fix and new feature release.

Download
--------------

You can easy_install (or PyPI URL:
<http://pypi.python.org/pypi/scikits.statsmodels/>)

Source downloads: <http://sourceforge.net/projects/statsmodels/>

Development branches: <http://code.launchpad.net/statsmodels>

Note that the trunk branch on launchpad is almost always stable and
has the most up to date changes since our releases are so few and far
between.

Documentation
----------------------

<http://statsmodels.sourceforge.net/>

We invite you to install, kick the tires, and make bug reports
and feature requests.

Feedback can either be on scipy-user or the mailing list at
<http://groups.google.com/group/pystatsmodels?hl=en>
Bug tracker: <https://bugs.launchpad.net/statsmodels>

Main Changes in 0.2.0
---------------------------------

* Improved documentation and expanded and more examples
* Added four discrete choice models: Poisson, Probit, Logit, and
Multinomial Logit.
* Added PyDTA. Tools for reading Stata binary datasets (*.dta) and putting
  them into numpy arrays.
* Added four new datasets for examples and tests.
* Results classes have been refactored to use lazy evaluation.
* Improved support for maximum likelihood estimation.
* bugfixes
* renames for more consistency
  RLM.fitted_values -> RLM.fittedvalues
  GLMResults.resid_dev -> GLMResults.resid_deviance

Sandbox
-------------

We are continuing to work on support for systems of equations models, panel data
models, time series analysis, and information and entropy econometrics in the
sandbox. This code is often merged into trunk as it becomes more robust.

Cheers,

Josef and Skipper


From jlconlin at gmail.com  Tue Feb 16 15:08:04 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Tue, 16 Feb 2010 13:08:04 -0700
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <d7d1efcf1002161104u73360c78p9962eb9344d66519@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
	<d7d1efcf1002161104u73360c78p9962eb9344d66519@mail.gmail.com>
Message-ID: <2588da421002161208g7e0dc3a8nd2541d28ed37d160@mail.gmail.com>

On Tue, Feb 16, 2010 at 12:04 PM, Jan Strube <curiousjan at gmail.com> wrote:
> Hi Jeremy,
> I'm not sure about scipy, but please have a look at paida.
> It's a pure python package that implements what you want.
> http://paida.sourceforge.net/documentation/fitter/index.html
> That's usually a decent package to have in your toolkit if you're doing data
> analysis.
> Hope that helps,
> ?? ?Jan
>

Thanks for the suggestion.  I was hoping not to have to download
another package.  If there is no way to do it in scipy, then maybe
I'll have to download paida.

Jeremy


From rob.clewley at gmail.com  Tue Feb 16 15:13:47 2010
From: rob.clewley at gmail.com (Rob Clewley)
Date: Tue, 16 Feb 2010 15:13:47 -0500
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <20100216160727.GA1779@phare.normalesup.org>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
	<20100216160727.GA1779@phare.normalesup.org>
Message-ID: <a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>

Gael, maybe we misunderstand each other! My data is actually very
smooth, it's just that it's not aligned on a regular mesh already.
It's ordered temporally as a single curve that is wrapping itself
around a solid object's surface.

-Rob

On Tue, Feb 16, 2010 at 11:07 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Tue, Feb 16, 2010 at 10:19:32AM -0500, Rob Clewley wrote:
>> I too have some data that has multiple z values in the x,y ?domain.
>> Worse, my data is not remotely arranged in a proper grid as it is
>> temporal trajectory data from a 3D ODE that winds itself round and
>> round the surface, like a very carefully wrapped mummy. I.e., it
>> doesn't wrap back over itself.
>
>> I'm not familiar with 3D plotting techniques and trying to set up the
>> plot of my surface in mayavi or matplotlib is not working along the
>> lines of standard cookbook examples. I cannot find appropriate
>> information online on how to plot the surface if I can't run the
>> interpolator for data that's not a function of (x,y) and my data needs
>> "alignment".
>
> Would this example help:
> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_surface_from_irregular_data.html
>
> Ga?l


From josef.pktd at gmail.com  Tue Feb 16 15:21:36 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 15:21:36 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
Message-ID: <1cd32cbb1002161221s12879438od114dc63d6c4d0f6@mail.gmail.com>

On Tue, Feb 16, 2010 at 1:08 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
> I have some data with some errobars that I need to fit to a line. ?Is
> there anyway to use scipy.polyfit with the error associated with the
> data? ?If not, how can I make a fit routine work with my data?

I'm not sure what your data and your errorbars look like but I think
scipy.optimize.curve_fit might do what you want. the sigma keyword can
be used for weighted least squares fitting. You would have to specify
your own fitting function, e.g. a polynomial on np.linspace and e.g.
np.vander(x, order)


>>> from scipy import optimize
>>> help(optimize.curve_fit)
Help on function curve_fit in module scipy.optimize.minpack:

curve_fit(f, xdata, ydata, p0=None, sigma=None, **kw)
    Use non-linear least squares to fit a function, f, to data.

    Assumes ``ydata = f(xdata, *params) + eps``

...

    sigma : None or N-length sequence
        If not None, it represents the standard-deviation of ydata.
        This vector, if given, will be used as weights in the
        least-squares problem.


Josef

>
> Thanks,
> Jeremy
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From gael.varoquaux at normalesup.org  Tue Feb 16 15:58:00 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 16 Feb 2010 21:58:00 +0100
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
	<20100216160727.GA1779@phare.normalesup.org>
	<a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>
Message-ID: <20100216205800.GB587@phare.normalesup.org>

On Tue, Feb 16, 2010 at 03:13:47PM -0500, Rob Clewley wrote:
> Gael, maybe we misunderstand each other! My data is actually very
> smooth, it's just that it's not aligned on a regular mesh already.
> It's ordered temporally as a single curve that is wrapping itself
> around a solid object's surface.

Yes, the example I point to shows how to reconstruct a surface from
unconnected point, so I would think that it applies to your problem.

HTH,

Ga?l

> -Rob

> On Tue, Feb 16, 2010 at 11:07 AM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
> > On Tue, Feb 16, 2010 at 10:19:32AM -0500, Rob Clewley wrote:
> >> I too have some data that has multiple z values in the x,y ?domain.
> >> Worse, my data is not remotely arranged in a proper grid as it is
> >> temporal trajectory data from a 3D ODE that winds itself round and
> >> round the surface, like a very carefully wrapped mummy. I.e., it
> >> doesn't wrap back over itself.

> >> I'm not familiar with 3D plotting techniques and trying to set up the
> >> plot of my surface in mayavi or matplotlib is not working along the
> >> lines of standard cookbook examples. I cannot find appropriate
> >> information online on how to plot the surface if I can't run the
> >> interpolator for data that's not a function of (x,y) and my data needs
> >> "alignment".

> > Would this example help:
> > http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_surface_from_irregular_data.html



From david_baddeley at yahoo.com.au  Tue Feb 16 16:16:10 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Tue, 16 Feb 2010 13:16:10 -0800 (PST)
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
	<20100216160727.GA1779@phare.normalesup.org>
	<a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>
Message-ID: <152614.26597.qm@web33001.mail.mud.yahoo.com>

As I see it, Gael's example doesn't do any smoothing - it just uses the triangulation to do the interpolation and get around the irregular sampling, and seems like quite a reasonable way to attack the problem. 

Correct me if I'm wrong, but as I read it you've got a number of x,y,z points which lie on the surface of a 3D volume, rather than on an essentially 2D 'rubber sheet'. In this case the 2D triangulation is not going to give you what you want, but you could apply the same principle and do a 3D triangularisation, which should give you triangles on the surface of your object. The problem is that it'll also give you triangles covering the surface of the entire convex hull and on the inside of the object, which you're somehow going to need to cull. This could be as simple as just culling everything which is larger than a certain cutoff. After this you're still left with the problem of displaying the triangles somehow (there's probably a patch function in matplotlib axes3d, and you're almost certain to be able to get triangles into mayavi.

I'm not sure if mayavi has 3D triangularisation, but there's a python wrapper around qhull which does. If you want to go this route I've got some code using qhull which does the triangularisation, large face culling, and internal face culling, as well as some very limited opengl code to draw the triangles. It's slow (practically limited to a couple of thousand points), poorly commented,  buggy, and somewhat application specific, but might be useful as a template to try and write something better. It's definitely not a ready solution, but drop me a line if you want to give it a try.

cheers,
David


----- Original Message ----
From: Rob Clewley <rob.clewley at gmail.com>
To: SciPy Users List <scipy-user at scipy.org>
Sent: Wed, 17 February, 2010 9:13:47 AM
Subject: Re: [SciPy-User] creating a 3D surface plot from collected data

Gael, maybe we misunderstand each other! My data is actually very
smooth, it's just that it's not aligned on a regular mesh already.
It's ordered temporally as a single curve that is wrapping itself
around a solid object's surface.

-Rob

On Tue, Feb 16, 2010 at 11:07 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Tue, Feb 16, 2010 at 10:19:32AM -0500, Rob Clewley wrote:
>> I too have some data that has multiple z values in the x,y  domain.
>> Worse, my data is not remotely arranged in a proper grid as it is
>> temporal trajectory data from a 3D ODE that winds itself round and
>> round the surface, like a very carefully wrapped mummy. I.e., it
>> doesn't wrap back over itself.
>
>> I'm not familiar with 3D plotting techniques and trying to set up the
>> plot of my surface in mayavi or matplotlib is not working along the
>> lines of standard cookbook examples. I cannot find appropriate
>> information online on how to plot the surface if I can't run the
>> interpolator for data that's not a function of (x,y) and my data needs
>> "alignment".
>
> Would this example help:
> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_surface_from_irregular_data.html
>
> Ga?l
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From gael.varoquaux at normalesup.org  Tue Feb 16 16:30:27 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 16 Feb 2010 22:30:27 +0100
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <152614.26597.qm@web33001.mail.mud.yahoo.com>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
	<20100216160727.GA1779@phare.normalesup.org>
	<a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>
	<152614.26597.qm@web33001.mail.mud.yahoo.com>
Message-ID: <20100216213027.GC587@phare.normalesup.org>

On Tue, Feb 16, 2010 at 01:16:10PM -0800, David Baddeley wrote:
> I'm not sure if mayavi has 3D triangularisation, 

It has, or rather VTK has, thus Mayavi has. It's just a question of
replacing the call to Delaunay2D by one to Delaunay3D. However, if a
surface is wanted, I believe that 2D triangulation is the way to go.

My 2 cents,

Ga?l


From rob.clewley at gmail.com  Tue Feb 16 18:13:23 2010
From: rob.clewley at gmail.com (Rob Clewley)
Date: Tue, 16 Feb 2010 18:13:23 -0500
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <152614.26597.qm@web33001.mail.mud.yahoo.com>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
	<20100216160727.GA1779@phare.normalesup.org>
	<a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>
	<152614.26597.qm@web33001.mail.mud.yahoo.com>
Message-ID: <a749952d1002161513k33e4d8ebm792928c36389108@mail.gmail.com>

Yes, David, definitely a 3D surface, so we'll need 3D
triangularisation. I think by culling you are referring to removing
giant triangles that would cover the opening of the "horn" shape that
I know my data makes, because those triangles would be part of the
convex hull. If so, we can certainly cutoff by size. When you say
other triangles that are "inside" do you just mean the ones on the
surface of the horn's opening that are not on the convex hull? If we
get as far as a clean triangularisation then we will plot with mayavi.

So I guess I will try using Delauney3D. Thanks for the tips. I (or
rather my student who is doing this mostly) will look into this more
closely to see if we can leverage these examples more than previous
ones. I'll be back on this thread if I get stuck, so look out :)

Thanks,
Rob

On Tue, Feb 16, 2010 at 4:16 PM, David Baddeley
<david_baddeley at yahoo.com.au> wrote:
> As I see it, Gael's example doesn't do any smoothing - it just uses the triangulation to do the interpolation and get around the irregular sampling, and seems like quite a reasonable way to attack the problem.
>
> Correct me if I'm wrong, but as I read it you've got a number of x,y,z points which lie on the surface of a 3D volume, rather than on an essentially 2D 'rubber sheet'. In this case the 2D triangulation is not going to give you what you want, but you could apply the same principle and do a 3D triangularisation, which should give you triangles on the surface of your object. The problem is that it'll also give you triangles covering the surface of the entire convex hull and on the inside of the object, which you're somehow going to need to cull. This could be as simple as just culling everything which is larger than a certain cutoff. After this you're still left with the problem of displaying the triangles somehow (there's probably a patch function in matplotlib axes3d, and you're almost certain to be able to get triangles into mayavi.
>
> I'm not sure if mayavi has 3D triangularisation, but there's a python wrapper around qhull which does. If you want to go this route I've got some code using qhull which does the triangularisation, large face culling, and internal face culling, as well as some very limited opengl code to draw the triangles. It's slow (practically limited to a couple of thousand points), poorly commented, ?buggy, and somewhat application specific, but might be useful as a template to try and write something better. It's definitely not a ready solution, but drop me a line if you want to give it a try.
>
> cheers,
> David
>
>
> ----- Original Message ----
> From: Rob Clewley <rob.clewley at gmail.com>
> To: SciPy Users List <scipy-user at scipy.org>
> Sent: Wed, 17 February, 2010 9:13:47 AM
> Subject: Re: [SciPy-User] creating a 3D surface plot from collected data
>
> Gael, maybe we misunderstand each other! My data is actually very
> smooth, it's just that it's not aligned on a regular mesh already.
> It's ordered temporally as a single curve that is wrapping itself
> around a solid object's surface.
>
> -Rob
>
> On Tue, Feb 16, 2010 at 11:07 AM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>> On Tue, Feb 16, 2010 at 10:19:32AM -0500, Rob Clewley wrote:
>>> I too have some data that has multiple z values in the x,y ?domain.
>>> Worse, my data is not remotely arranged in a proper grid as it is
>>> temporal trajectory data from a 3D ODE that winds itself round and
>>> round the surface, like a very carefully wrapped mummy. I.e., it
>>> doesn't wrap back over itself.
>>
>>> I'm not familiar with 3D plotting techniques and trying to set up the
>>> plot of my surface in mayavi or matplotlib is not working along the
>>> lines of standard cookbook examples. I cannot find appropriate
>>> information online on how to plot the surface if I can't run the
>>> interpolator for data that's not a function of (x,y) and my data needs
>>> "alignment".
>>
>> Would this example help:
>> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_surface_from_irregular_data.html
>>
>> Ga?l


From gael.varoquaux at normalesup.org  Tue Feb 16 18:20:07 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 17 Feb 2010 00:20:07 +0100
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <a749952d1002161513k33e4d8ebm792928c36389108@mail.gmail.com>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
	<20100216160727.GA1779@phare.normalesup.org>
	<a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>
	<152614.26597.qm@web33001.mail.mud.yahoo.com>
	<a749952d1002161513k33e4d8ebm792928c36389108@mail.gmail.com>
Message-ID: <20100216232007.GH587@phare.normalesup.org>

On Tue, Feb 16, 2010 at 06:13:23PM -0500, Rob Clewley wrote:
> Yes, David, definitely a 3D surface, so we'll need 3D
> triangularisation. 

I think we are having a communication problem here (but I may be wrong).
When you say a 3D surface, you mean a 2D surface embedded in a 3D space,
or a 3D volume? If it is the later, you will need Delaunay3D, as it
creates volumes. If it is the former, Delaunay2D is what you are looking
for : it creates 2D surfaces embedded in a 3D world by connecting dots
with 3D coordinnates using triangles.

Ga?l

> I think by culling you are referring to removing
> giant triangles that would cover the opening of the "horn" shape that
> I know my data makes, because those triangles would be part of the
> convex hull. If so, we can certainly cutoff by size. When you say
> other triangles that are "inside" do you just mean the ones on the
> surface of the horn's opening that are not on the convex hull? If we
> get as far as a clean triangularisation then we will plot with mayavi.

> So I guess I will try using Delauney3D. Thanks for the tips. I (or
> rather my student who is doing this mostly) will look into this more
> closely to see if we can leverage these examples more than previous
> ones. I'll be back on this thread if I get stuck, so look out :)

> Thanks,
> Rob

> On Tue, Feb 16, 2010 at 4:16 PM, David Baddeley
> <david_baddeley at yahoo.com.au> wrote:
> > As I see it, Gael's example doesn't do any smoothing - it just uses the triangulation to do the interpolation and get around the irregular sampling, and seems like quite a reasonable way to attack the problem.

> > Correct me if I'm wrong, but as I read it you've got a number of x,y,z points which lie on the surface of a 3D volume, rather than on an essentially 2D 'rubber sheet'. In this case the 2D triangulation is not going to give you what you want, but you could apply the same principle and do a 3D triangularisation, which should give you triangles on the surface of your object. The problem is that it'll also give you triangles covering the surface of the entire convex hull and on the inside of the object, which you're somehow going to need to cull. This could be as simple as just culling everything which is larger than a certain cutoff. After this you're still left with the problem of displaying the triangles somehow (there's probably a patch function in matplotlib axes3d, and you're almost certain to be able to get triangles into mayavi.

> > I'm not sure if mayavi has 3D triangularisation, but there's a python wrapper around qhull which does. If you want to go this route I've got some code using qhull which does the triangularisation, large face culling, and internal face culling, as well as some very limited opengl code to draw the triangles. It's slow (practically limited to a couple of thousand points), poorly commented, ?buggy, and somewhat application specific, but might be useful as a template to try and write something better. It's definitely not a ready solution, but drop me a line if you want to give it a try.

> > cheers,
> > David


> > ----- Original Message ----
> > From: Rob Clewley <rob.clewley at gmail.com>
> > To: SciPy Users List <scipy-user at scipy.org>
> > Sent: Wed, 17 February, 2010 9:13:47 AM
> > Subject: Re: [SciPy-User] creating a 3D surface plot from collected data

> > Gael, maybe we misunderstand each other! My data is actually very
> > smooth, it's just that it's not aligned on a regular mesh already.
> > It's ordered temporally as a single curve that is wrapping itself
> > around a solid object's surface.

> > -Rob

> > On Tue, Feb 16, 2010 at 11:07 AM, Gael Varoquaux
> > <gael.varoquaux at normalesup.org> wrote:
> >> On Tue, Feb 16, 2010 at 10:19:32AM -0500, Rob Clewley wrote:
> >>> I too have some data that has multiple z values in the x,y ?domain.
> >>> Worse, my data is not remotely arranged in a proper grid as it is
> >>> temporal trajectory data from a 3D ODE that winds itself round and
> >>> round the surface, like a very carefully wrapped mummy. I.e., it
> >>> doesn't wrap back over itself.

> >>> I'm not familiar with 3D plotting techniques and trying to set up the
> >>> plot of my surface in mayavi or matplotlib is not working along the
> >>> lines of standard cookbook examples. I cannot find appropriate
> >>> information online on how to plot the surface if I can't run the
> >>> interpolator for data that's not a function of (x,y) and my data needs
> >>> "alignment".

> >> Would this example help:
> >> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_surface_from_irregular_data.html

> >> Ga?l
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-- 
    Gael Varoquaux
    Research Fellow, INRIA
    Laboratoire de Neuro-Imagerie Assistee par Ordinateur
    NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
    ++ 33-1-69-08-78-35
    http://gael-varoquaux.info


From robert.kern at gmail.com  Tue Feb 16 18:28:25 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 16 Feb 2010 17:28:25 -0600
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <20100216232007.GH587@phare.normalesup.org>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com> 
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com> 
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com> 
	<20100216160727.GA1779@phare.normalesup.org>
	<a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com> 
	<152614.26597.qm@web33001.mail.mud.yahoo.com>
	<a749952d1002161513k33e4d8ebm792928c36389108@mail.gmail.com> 
	<20100216232007.GH587@phare.normalesup.org>
Message-ID: <3d375d731002161528ydeb0e4co689824e3f833278e@mail.gmail.com>

On Tue, Feb 16, 2010 at 17:20, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Tue, Feb 16, 2010 at 06:13:23PM -0500, Rob Clewley wrote:
>> Yes, David, definitely a 3D surface, so we'll need 3D
>> triangularisation.
>
> I think we are having a communication problem here (but I may be wrong).
> When you say a 3D surface, you mean a 2D surface embedded in a 3D space,
> or a 3D volume? If it is the later, you will need Delaunay3D, as it
> creates volumes. If it is the former, Delaunay2D is what you are looking
> for : it creates 2D surfaces embedded in a 3D world by connecting dots
> with 3D coordinnates using triangles.

No, this is not true. Delaunay2D ignores the Z coordinate and returns
the 2D Delaunay triangulation of just the X,Y coordinates. When you
have an z=f(x,y) functional relationship, this usually does what you
want. It does not solve Rob's problem or that of the OP.

  http://www.vtk.org/doc/release/5.4/html/a00404.html

Instead, they need the SurfaceReconstructionFilter:

  http://www.vtk.org/doc/release/5.4/html/a01651.html

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Tue Feb 16 18:36:24 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 17 Feb 2010 00:36:24 +0100
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <3d375d731002161528ydeb0e4co689824e3f833278e@mail.gmail.com>
References: <f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
	<20100216160727.GA1779@phare.normalesup.org>
	<a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>
	<152614.26597.qm@web33001.mail.mud.yahoo.com>
	<a749952d1002161513k33e4d8ebm792928c36389108@mail.gmail.com>
	<20100216232007.GH587@phare.normalesup.org>
	<3d375d731002161528ydeb0e4co689824e3f833278e@mail.gmail.com>
Message-ID: <20100216233624.GI587@phare.normalesup.org>

On Tue, Feb 16, 2010 at 05:28:25PM -0600, Robert Kern wrote:
> No, this is not true. Delaunay2D ignores the Z coordinate and returns
> the 2D Delaunay triangulation of just the X,Y coordinates. When you
> have an z=f(x,y) functional relationship, this usually does what you
> want. It does not solve Rob's problem or that of the OP.

>   http://www.vtk.org/doc/release/5.4/html/a00404.html

Wow, I am bluffed. I was pretty sure of the contrary and didn't check.

> Instead, they need the SurfaceReconstructionFilter:

>   http://www.vtk.org/doc/release/5.4/html/a01651.html

Rob and David, you can use it by replacing the call to
mlab.pipeline.delaunay2d by a call to:

mlab.pipeline.user_defined(obj, filter='SurfaceReconstructionFilter'),

as in the following example:
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_image_cursor_filter.html

Thanks for the details, Robert.

Ga?l


From zallen at urologicresearchinstitute.org  Tue Feb 16 18:40:47 2010
From: zallen at urologicresearchinstitute.org (URI)
Date: Tue, 16 Feb 2010 23:40:47 +0000 (UTC)
Subject: [SciPy-User] New installation
References: <4B7076AD.8010900@umit.maine.edu>
Message-ID: <loom.20100217T004028-324@post.gmane.org>

R. Padraic Springuel <R.Springuel <at> umit.maine.edu> writes:

> 
> I just tried to upgrade one of the computers I'm responsible for from
> Python 2.5 to Python 2.6 (including all the packages that are used) and
> while all the installations seemed to work just fine, I get the
> following error when I try to import numpy after install:
> 
> 
> Any idea what might be causing this error and what I can do to fix it?
> 
> Python: 2.6.4
> Numpy: 1.4.0
> Mac OS 10.5.8


R.-

Did you try to completely remove Python and then do a fresh install?  Whenever I
try to do upgrades or additions I always have problems, so I like to just wipe
everything out and install from scratch.

Another thing you can try is (if you are on Windows) the Python (x,y) installer
at http://www.pythonxy.com/download.php

I've only been using it for a couple days, but it lets you put a check mark next
to the packages you want at install time and it handles everything for you, so
you don't have to worry about missing something or making a configuration error.



From silva at lma.cnrs-mrs.fr  Tue Feb 16 18:52:56 2010
From: silva at lma.cnrs-mrs.fr (Fabrice Silva)
Date: Wed, 17 Feb 2010 00:52:56 +0100
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
Message-ID: <1266364376.2238.0.camel@Portable-s2m.cnrs-mrs.fr>

Le mardi 16 f?vrier 2010 ? 11:08 -0700, Jeremy Conlin a ?crit :
> I have some data with some errobars that I need to fit to a line.  Is
> there anyway to use scipy.polyfit with the error associated with the
> data?  If not, how can I make a fit routine work with my data?

You may have a glance at 
http://docs.scipy.org/doc/scipy/reference/odr.html




From david_baddeley at yahoo.com.au  Tue Feb 16 19:08:13 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Tue, 16 Feb 2010 16:08:13 -0800 (PST)
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <a749952d1002161513k33e4d8ebm792928c36389108@mail.gmail.com>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<a749952d1002160719l2c3fea3aj56c3a1c6aee23e2c@mail.gmail.com>
	<20100216160727.GA1779@phare.normalesup.org>
	<a749952d1002161213y6ba46dcbh6ea3206140fdc73a@mail.gmail.com>
	<152614.26597.qm@web33001.mail.mud.yahoo.com>
	<a749952d1002161513k33e4d8ebm792928c36389108@mail.gmail.com>
Message-ID: <230261.10088.qm@web33005.mail.mud.yahoo.com>

The 3D triangularisation gives you a set of tetrahedra, each of which has 4 faces, some of the faces will lie on the surface of the object, but it'll also create tetrahedra within the object, which will probably have one face on the surface & 3 faces which are hidden within the volume. These are what I was refering to as internal faces. They're more of a problem for performance than for rendering unless you want to see the inside of the object.

That said, I suspect the surface reconstruction filter might be a better way to go.

cheers,
David


----- Original Message ----
From: Rob Clewley <rob.clewley at gmail.com>
To: SciPy Users List <scipy-user at scipy.org>
Sent: Wed, 17 February, 2010 12:13:23 PM
Subject: Re: [SciPy-User] creating a 3D surface plot from collected data

Yes, David, definitely a 3D surface, so we'll need 3D
triangularisation. I think by culling you are referring to removing
giant triangles that would cover the opening of the "horn" shape that
I know my data makes, because those triangles would be part of the
convex hull. If so, we can certainly cutoff by size. When you say
other triangles that are "inside" do you just mean the ones on the
surface of the horn's opening that are not on the convex hull? If we
get as far as a clean triangularisation then we will plot with mayavi.

So I guess I will try using Delauney3D. Thanks for the tips. I (or
rather my student who is doing this mostly) will look into this more
closely to see if we can leverage these examples more than previous
ones. I'll be back on this thread if I get stuck, so look out :)

Thanks,
Rob

On Tue, Feb 16, 2010 at 4:16 PM, David Baddeley
<david_baddeley at yahoo.com.au> wrote:
> As I see it, Gael's example doesn't do any smoothing - it just uses the triangulation to do the interpolation and get around the irregular sampling, and seems like quite a reasonable way to attack the problem.
>
> Correct me if I'm wrong, but as I read it you've got a number of x,y,z points which lie on the surface of a 3D volume, rather than on an essentially 2D 'rubber sheet'. In this case the 2D triangulation is not going to give you what you want, but you could apply the same principle and do a 3D triangularisation, which should give you triangles on the surface of your object. The problem is that it'll also give you triangles covering the surface of the entire convex hull and on the inside of the object, which you're somehow going to need to cull. This could be as simple as just culling everything which is larger than a certain cutoff. After this you're still left with the problem of displaying the triangles somehow (there's probably a patch function in matplotlib axes3d, and you're almost certain to be able to get triangles into mayavi.
>
> I'm not sure if mayavi has 3D triangularisation, but there's a python wrapper around qhull which does. If you want to go this route I've got some code using qhull which does the triangularisation, large face culling, and internal face culling, as well as some very limited opengl code to draw the triangles. It's slow (practically limited to a couple of thousand points), poorly commented,  buggy, and somewhat application specific, but might be useful as a template to try and write something better. It's definitely not a ready solution, but drop me a line if you want to give it a try.
>
> cheers,
> David
>
>
> ----- Original Message ----
> From: Rob Clewley <rob.clewley at gmail.com>
> To: SciPy Users List <scipy-user at scipy.org>
> Sent: Wed, 17 February, 2010 9:13:47 AM
> Subject: Re: [SciPy-User] creating a 3D surface plot from collected data
>
> Gael, maybe we misunderstand each other! My data is actually very
> smooth, it's just that it's not aligned on a regular mesh already.
> It's ordered temporally as a single curve that is wrapping itself
> around a solid object's surface.
>
> -Rob
>
> On Tue, Feb 16, 2010 at 11:07 AM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>> On Tue, Feb 16, 2010 at 10:19:32AM -0500, Rob Clewley wrote:
>>> I too have some data that has multiple z values in the x,y  domain.
>>> Worse, my data is not remotely arranged in a proper grid as it is
>>> temporal trajectory data from a 3D ODE that winds itself round and
>>> round the surface, like a very carefully wrapped mummy. I.e., it
>>> doesn't wrap back over itself.
>>
>>> I'm not familiar with 3D plotting techniques and trying to set up the
>>> plot of my surface in mayavi or matplotlib is not working along the
>>> lines of standard cookbook examples. I cannot find appropriate
>>> information online on how to plot the surface if I can't run the
>>> interpolator for data that's not a function of (x,y) and my data needs
>>> "alignment".
>>
>> Would this example help:
>> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_surface_from_irregular_data.html
>>
>> Ga?l
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From njs at pobox.com  Tue Feb 16 22:24:59 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 16 Feb 2010 19:24:59 -0800
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
Message-ID: <961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>

On Tue, Feb 16, 2010 at 10:08 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
> I have some data with some errobars that I need to fit to a line. ?Is
> there anyway to use scipy.polyfit with the error associated with the
> data? ?If not, how can I make a fit routine work with my data?

If the error is just in the y (dependent) variable, then this may be
quite simple. If your y values:
  -- have a normally distributed error
  -- whose standard deviation varies from point to point
  -- but the standard deviations are known (up to a multiplicative constant)
then it's a classic problem -- perhaps google "heterskedasticity
correction" or "weighted least squares" or "generalized least
squares". But basically, just make a model matrix X that has one row
for each observation, and one column for each predictor. If you want
to do, say, third-order polynomial fitting and you have two vectors of
independent variables x1 and x2 and you want to include an intercept,
then it's something like (all untested):

X = np.column_stack([np.ones(len(x1)), x1, x1 ** 2, x1 ** 3, x2, x2 **
2, x2 ** 3])

Now a normal least squares fit, ignoring the errorbars, would be:

beta = np.linalg.lstsq(X, y)[0]

with predicted values np.dot(X, beta).

If you have the standard deviation for each measurement in a vector
stddevs, then the proper heteroskedasticity-corrected fit is just:

beta = np.linalg.lstsq((1 / stddevs) * X, (1 / stddevs) * y)[0]

-- Nathaniel


From josef.pktd at gmail.com  Tue Feb 16 22:33:34 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 22:33:34 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
Message-ID: <1cd32cbb1002161933n7dff2a7er957a30ea37ce8957@mail.gmail.com>

On Tue, Feb 16, 2010 at 10:24 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Feb 16, 2010 at 10:08 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>> I have some data with some errobars that I need to fit to a line. ?Is
>> there anyway to use scipy.polyfit with the error associated with the
>> data? ?If not, how can I make a fit routine work with my data?
>
> If the error is just in the y (dependent) variable, then this may be
> quite simple. If your y values:
> ?-- have a normally distributed error
> ?-- whose standard deviation varies from point to point
> ?-- but the standard deviations are known (up to a multiplicative constant)
> then it's a classic problem -- perhaps google "heterskedasticity
> correction" or "weighted least squares" or "generalized least
> squares". But basically, just make a model matrix X that has one row
> for each observation, and one column for each predictor. If you want
> to do, say, third-order polynomial fitting and you have two vectors of
> independent variables x1 and x2 and you want to include an intercept,
> then it's something like (all untested):
>
> X = np.column_stack([np.ones(len(x1)), x1, x1 ** 2, x1 ** 3, x2, x2 **
> 2, x2 ** 3])
>
> Now a normal least squares fit, ignoring the errorbars, would be:
>
> beta = np.linalg.lstsq(X, y)[0]
>
> with predicted values np.dot(X, beta).
>
> If you have the standard deviation for each measurement in a vector
> stddevs, then the proper heteroskedasticity-corrected fit is just:
>
> beta = np.linalg.lstsq((1 / stddevs) * X, (1 / stddevs) * y)[0]
>
> -- Nathaniel
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Tue Feb 16 22:48:13 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 22:48:13 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
Message-ID: <1cd32cbb1002161948o521c20bdh8f677e4423ccd100@mail.gmail.com>

On Tue, Feb 16, 2010 at 10:24 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Feb 16, 2010 at 10:08 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>> I have some data with some errobars that I need to fit to a line. ?Is
>> there anyway to use scipy.polyfit with the error associated with the
>> data? ?If not, how can I make a fit routine work with my data?
>
> If the error is just in the y (dependent) variable, then this may be
> quite simple. If your y values:
> ?-- have a normally distributed error
> ?-- whose standard deviation varies from point to point
> ?-- but the standard deviations are known (up to a multiplicative constant)
> then it's a classic problem -- perhaps google "heterskedasticity
> correction" or "weighted least squares" or "generalized least
> squares". But basically, just make a model matrix X that has one row
> for each observation, and one column for each predictor. If you want
> to do, say, third-order polynomial fitting and you have two vectors of
> independent variables x1 and x2 and you want to include an intercept,
> then it's something like (all untested):
>
> X = np.column_stack([np.ones(len(x1)), x1, x1 ** 2, x1 ** 3, x2, x2 **
> 2, x2 ** 3])
>
> Now a normal least squares fit, ignoring the errorbars, would be:
>
> beta = np.linalg.lstsq(X, y)[0]
>
> with predicted values np.dot(X, beta).
>
> If you have the standard deviation for each measurement in a vector
> stddevs, then the proper heteroskedasticity-corrected fit is just:
>
> beta = np.linalg.lstsq((1 / stddevs) * X, (1 / stddevs) * y)[0]

I didn't realize that it is a problem linear in parameters if the
objective is to fit a polynomial.

Essentially the same calculations are done in statsmodels.WLS  plus
you get additional results and test statistics.

something like

wls_results = scikits.statsmodels.WLS(Y, np.vander(X,2), weights=1/stddevs)
wls_results.params
wls_results.bse
wls_results.fittedvalues

example in statsmodels\examples\tut_ols_wls.py

Josef


> -- Nathaniel
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From njs at pobox.com  Tue Feb 16 23:18:17 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 16 Feb 2010 20:18:17 -0800
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <1cd32cbb1002161948o521c20bdh8f677e4423ccd100@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
	<1cd32cbb1002161948o521c20bdh8f677e4423ccd100@mail.gmail.com>
Message-ID: <961fa2b41002162018y39b6e7fp763c1a4d72b17c96@mail.gmail.com>

On Tue, Feb 16, 2010 at 7:48 PM,  <josef.pktd at gmail.com> wrote:
> I didn't realize that it is a problem linear in parameters if the
> objective is to fit a polynomial.

I dunno, I'm just going off a quick glance at the documentation for
"polyfit", which the OP wanted to use in the first place :-).

> Essentially the same calculations are done in statsmodels.WLS ?plus
> you get additional results and test statistics.
>
> something like
>
> wls_results = scikits.statsmodels.WLS(Y, np.vander(X,2), weights=1/stddevs)
> wls_results.params
> wls_results.bse
> wls_results.fittedvalues
>
> example in statsmodels\examples\tut_ols_wls.py

Yeah, using real statistics code is always a better idea when
available. (Actually, I would use R for this. Don't tell anyone!)

-- Nathaniel


From josef.pktd at gmail.com  Tue Feb 16 23:36:53 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 23:36:53 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <961fa2b41002162018y39b6e7fp763c1a4d72b17c96@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
	<1cd32cbb1002161948o521c20bdh8f677e4423ccd100@mail.gmail.com>
	<961fa2b41002162018y39b6e7fp763c1a4d72b17c96@mail.gmail.com>
Message-ID: <1cd32cbb1002162036n373a8757u267b4ca515ba72af@mail.gmail.com>

On Tue, Feb 16, 2010 at 11:18 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Feb 16, 2010 at 7:48 PM, ?<josef.pktd at gmail.com> wrote:
>> I didn't realize that it is a problem linear in parameters if the
>> objective is to fit a polynomial.
>
> I dunno, I'm just going off a quick glance at the documentation for
> "polyfit", which the OP wanted to use in the first place :-).
>
>> Essentially the same calculations are done in statsmodels.WLS ?plus
>> you get additional results and test statistics.
>>
>> something like
>>
>> wls_results = scikits.statsmodels.WLS(Y, np.vander(X,2), weights=1/stddevs)
>> wls_results.params
>> wls_results.bse
>> wls_results.fittedvalues
>>
>> example in statsmodels\examples\tut_ols_wls.py
>
> Yeah, using real statistics code is always a better idea when
> available. (Actually, I would use R for this. Don't tell anyone!)

But it's more fun trying to figure out how to do it in python than how
to do it in R or rpy.

(But maybe not so much if I have to figure out both for doing the
validation tests. I've seen that your incremental_ls also uses R only
for validation and not for the heavy duty stuff.)

Josef

>
> -- Nathaniel
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From jsseabold at gmail.com  Tue Feb 16 23:44:59 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 16 Feb 2010 23:44:59 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <1cd32cbb1002162036n373a8757u267b4ca515ba72af@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com> 
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com> 
	<1cd32cbb1002161948o521c20bdh8f677e4423ccd100@mail.gmail.com> 
	<961fa2b41002162018y39b6e7fp763c1a4d72b17c96@mail.gmail.com> 
	<1cd32cbb1002162036n373a8757u267b4ca515ba72af@mail.gmail.com>
Message-ID: <c048da1c1002162044r140e476dp577fafdbe8e4b254@mail.gmail.com>

On Tue, Feb 16, 2010 at 11:36 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Feb 16, 2010 at 11:18 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Tue, Feb 16, 2010 at 7:48 PM, ?<josef.pktd at gmail.com> wrote:
>>> I didn't realize that it is a problem linear in parameters if the
>>> objective is to fit a polynomial.
>>
>> I dunno, I'm just going off a quick glance at the documentation for
>> "polyfit", which the OP wanted to use in the first place :-).
>>
>>> Essentially the same calculations are done in statsmodels.WLS ?plus
>>> you get additional results and test statistics.
>>>
>>> something like
>>>
>>> wls_results = scikits.statsmodels.WLS(Y, np.vander(X,2), weights=1/stddevs)
>>> wls_results.params
>>> wls_results.bse
>>> wls_results.fittedvalues
>>>
>>> example in statsmodels\examples\tut_ols_wls.py
>>
>> Yeah, using real statistics code is always a better idea when
>> available. (Actually, I would use R for this. Don't tell anyone!)
>
> But it's more fun trying to figure out how to do it in python than how
> to do it in R or rpy.
>

Exactly.

> (But maybe not so much if I have to figure out both for doing the
> validation tests. I've seen that your incremental_ls also uses R only
> for validation and not for the heavy duty stuff.)
>

But this lets you be on both sides and try to convince people that R
isn't fun and you know by experience ;)


From njs at pobox.com  Wed Feb 17 00:22:01 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 16 Feb 2010 21:22:01 -0800
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <1cd32cbb1002162036n373a8757u267b4ca515ba72af@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
	<1cd32cbb1002161948o521c20bdh8f677e4423ccd100@mail.gmail.com>
	<961fa2b41002162018y39b6e7fp763c1a4d72b17c96@mail.gmail.com>
	<1cd32cbb1002162036n373a8757u267b4ca515ba72af@mail.gmail.com>
Message-ID: <961fa2b41002162122o1de8704dp18850e458975e684@mail.gmail.com>

On Tue, Feb 16, 2010 at 8:36 PM,  <josef.pktd at gmail.com> wrote:
> But it's more fun trying to figure out how to do it in python than how
> to do it in R or rpy.

Maybe -- I already know R, so sunk costs, etc., oh well. It's much
less pointless than learning Matlab, at least...

> (But maybe not so much if I have to figure out both for doing the
> validation tests. I've seen that your incremental_ls also uses R only
> for validation and not for the heavy duty stuff.)

Yeah, no question Python is a nicer language for greenfield projects.
But in the project I wrote incremental_ls for, the higher-level API
actually still takes R model specifications (like "y ~ x1 + log(x2) +
Subject" or whatever), shoves it through rpy (or rather, my wrapper
'rnumpy'), and then uses it to construct the actual model matrix.
There's just a huge amount of statistical wisdom and tricky code
already built in there -- it's too bad the interpreter was written by
statisticians instead of language engineers.

But this is getting pretty off-topic...

-- Nathaniel


From aisaac at american.edu  Wed Feb 17 06:40:28 2010
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 17 Feb 2010 06:40:28 -0500
Subject: [SciPy-User] scipy.stats -> deprecation warning
Message-ID: <4B7BD5AC.80502@american.edu>

SciPy 0.7.1 (i.e., latest installer):
scipy.stats.gaussian_kde raises a deprecation warning
because it calls scipy.stats.cov

Alan Isaac


From ryanlists at gmail.com  Wed Feb 17 08:39:48 2010
From: ryanlists at gmail.com (Ryan Krauss)
Date: Wed, 17 Feb 2010 07:39:48 -0600
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <1cd32cbb1002141843u261a7acbx5951e0374d0b5d48@mail.gmail.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>
	<4B6256F5.3050802@enthought.com>
	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>
	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>
	<c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>
	<1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>
	<4B71A6BD.9090002@enthought.com>
	<c5b438121002110721g66eda059ve86d199f7beed0fd@mail.gmail.com>
	<4B789D3D.1060506@enthought.com>
	<1cd32cbb1002141843u261a7acbx5951e0374d0b5d48@mail.gmail.com>
Message-ID: <c5b438121002170539n3ee8ebb7vf478046b87b68c06@mail.gmail.com>

I like it alot.  It goes way beyond solving my problem to adding new
functionality, improving the documentation, and creating some well
thought out tests.  Nice work Warren.

I didn't see how to download the original version of the patch that
created test_ltisys.py (maybe it should already exist on my computer
if I had a more recent version of scipy).  So, patch yelled at me
about not finding the file.  So, I didn't actually run Warren's tests;
I just ran a couple of simple cases I made up.  But it worked
perfectly.

Thanks again,

Ryan

On Sun, Feb 14, 2010 at 8:43 PM,  <josef.pktd at gmail.com> wrote:
> On Sun, Feb 14, 2010 at 8:02 PM, Warren Weckesser
> <warren.weckesser at enthought.com> wrote:
>> I added a patch to ticket #1112. ?In addition to adding the suggested
>> change to pass keyword args to odeint, I also added a new funciton,
>> impulse2(), that computes the impulse response by using odeint. ?See
>> this thread for details:
>>
>> ? ?http://mail.scipy.org/pipermail/scipy-user/2009-November/023416.html
>
> Thanks, the test examples look nice and serve also as some
> documentation for ltisys. The lsim2 test05 is interesting, I was
> convinced last year (and there were some previous threads) that ltisys
> cannot handle multi-input systems. I always got shape errors when I
> tried and without documentation it's difficult to figure out what is
> supposed to work and what not. (I will look at it more closely when I
> have more time.)
>
> I think the changes are reasonable including turning u and t into
> keywords instead of required arguments. Maybe Ryan or someone who is
> using this functions can comment on this.
>
> Just one improvement to the tests, assert_almost_equal takes a decimal
> argument. If you know the precision of your tests, then it would be
> useful to increase it from the default decimal=7.
>
> Josef
>
>
>>
>> Warren
>>
>>
>> Ryan Krauss wrote:
>>> Yeah, **kwds is a little harder to understand and slightly less user
>>> friendly, but it is easier to maintain and less work for the patch
>>> writer. ?And it is more future proof if the integrator ever changes.
>>>
>>> On Tue, Feb 9, 2010 at 12:17 PM, Warren Weckesser
>>> <warren.weckesser at enthought.com> wrote:
>>>
>>>> josef.pktd at gmail.com wrote:
>>>>
>>>>> On Wed, Feb 3, 2010 at 9:00 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>>>
>>>>>
>>>>>> FYI, I am quite happy with passing in an hmax value. ?I basically
>>>>>> copied and pasted lsim2 from signal.ltisys and adapted it just a
>>>>>> little to make it a method of my derived class. ?Then I added the hmas
>>>>>> kwarg that gets passed to odeint.
>>>>>>
>>>>>> Is there any reason not to allow the user to pass in a kwargs to lsim2
>>>>>> that gets passed to odeint?
>>>>>>
>>>>>>
>>>>> I don't see a reason why we cannot add a **kwargs, it should be
>>>>> completely backwards compatible.
>>>>> Can you file a ticket and add your adjusted version or a patch? And
>>>>> even better, add your original example as a test case?
>>>>>
>>>>>
>>>>>
>>>> Josef,
>>>>
>>>> I just created ticket #1112 for this. ?Unless Ryan wants to adapt his
>>>> change to lsim2, I can make a patch this week to implement the enhancement.
>>>>
>>>> Warren
>>>>
>>>>
>>>>
>>>>> Josef
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On Fri, Jan 29, 2010 at 6:44 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>> Thanks to Warren and Josef for their time and thoughts. ?I feel like I
>>>>>>> now understand the underlying problem and have some good options to
>>>>>>> solve my short term issues (I assigned the project last night and they
>>>>>>> need to be able to start working on it immediately). ?I actually use a
>>>>>>> TransferFunction class that derives from ltisys. ?I could override its
>>>>>>> lsim2 method to try out some of these solutions quickly and fairly
>>>>>>> easily.
>>>>>>>
>>>>>>> Ryan
>>>>>>>
>>>>>>> On Thu, Jan 28, 2010 at 10:00 PM, ?<josef.pktd at gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Thu, Jan 28, 2010 at 10:33 PM, Warren Weckesser
>>>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> josef.pktd at gmail.com wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Thu, Jan 28, 2010 at 8:50 PM, Warren Weckesser
>>>>>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Ryan,
>>>>>>>>>>>
>>>>>>>>>>> The problem is that the ODE solver used by lsim2 is too good. :)
>>>>>>>>>>>
>>>>>>>>>>> It uses scipy.integrate.odeint, which in turn uses the Fortran library
>>>>>>>>>>> LSODA. ?Like any good solver, LSODA is an adaptive solver--it adjusts its
>>>>>>>>>>> step size to be as large as possible while keeping estimates of the error
>>>>>>>>>>> bounded. ?For the problem you are solving, with initial condition 0, the
>>>>>>>>>>> exact solution is initially exactly 0. ?This is such a nice smooth solution
>>>>>>>>>>> that the solver's step size quickly grows--so big, in fact, that it skips
>>>>>>>>>>> right over your pulse and never sees it.
>>>>>>>>>>>
>>>>>>>>>>> So how does it create all those intermediate points at the requested time
>>>>>>>>>>> values? ?It uses interpolation between the steps that it computed to create
>>>>>>>>>>> the solution values at the times that you requested. ?So using a finer grid
>>>>>>>>>>> of time values won't help. ?(If lsim2 gave you a hook into the parameters
>>>>>>>>>>> passed to odeint, you could set odeint's 'hmax' to a value smaller than your
>>>>>>>>>>> pulse width, which would force the solver to see the pulse. ?But there is no
>>>>>>>>>>> way to set that parameter from lsim2.)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> It's something what I suspected. I don't know much about odeint, but
>>>>>>>>>> do you think it would be useful to let lsim2 pass through some
>>>>>>>>>> parameters to odeint?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Sounds useful to me. ?A simple implementation is an optional keyword
>>>>>>>>> argument that is a dict of odeint arguments. ? But this would almost
>>>>>>>>> certainly break if lsim2 were ever reimplemented with a different
>>>>>>>>> solver. ?So perhaps it should allow a common set of ODE solver
>>>>>>>>> parameters (e.g. absolute and relative error tolerances, max and min
>>>>>>>>> step sizes, others?).
>>>>>>>>>
>>>>>>>>> Perhaps this should wait until after the ODE solver redesign that is
>>>>>>>>> occasionally discussed:
>>>>>>>>> ? ?http://projects.scipy.org/scipy/wiki/OdeintRedesign
>>>>>>>>> Then the solver itself could be an optional argument to lsim2.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I was just thinking of adding to the argument list a **kwds argument
>>>>>>>> that is directly passed on to whatever ODE solver is used. This should
>>>>>>>> be pretty flexible for any changes and be backwards compatible.
>>>>>>>>
>>>>>>>> I've seen and used it in a similar way for calls to optimization
>>>>>>>> routines, e.g. also optimize.curve_fit, does it. What are actually
>>>>>>>> valid keywords would depend on which function is called.
>>>>>>>>
>>>>>>>> (But I'm not a user of lsim, I'm just stealing some ideas from lti and
>>>>>>>> friends for time series analysis.)
>>>>>>>>
>>>>>>>> Josef
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Warren
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Josef
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> The basic problem is you are passing in a discontinuous function to a solver
>>>>>>>>>>> that expects a smooth function. ?A better way to solve this problem is to
>>>>>>>>>>> explicitly account for the discontinuity. One possibility is the attached
>>>>>>>>>>> script.
>>>>>>>>>>>
>>>>>>>>>>> This is an excellent "learning opportunity" for your students on the hazards
>>>>>>>>>>> of numerical computing!
>>>>>>>>>>>
>>>>>>>>>>> Warren
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ryan Krauss wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I believe I have discovered a bug in signal.lsim2. ?I believe the
>>>>>>>>>>>> short attached script illustrates the problem. ?I was trying to
>>>>>>>>>>>> predict the response of a transfer function with a pure integrator:
>>>>>>>>>>>>
>>>>>>>>>>>> ? ? ? ? ? ? g
>>>>>>>>>>>> G = -------------
>>>>>>>>>>>> ? ? ? ? s(s+p)
>>>>>>>>>>>>
>>>>>>>>>>>> to a finite width pulse. ?lsim2 seems to handle the step response just
>>>>>>>>>>>> fine, but says that the pulse response is exactly 0.0 for the entire
>>>>>>>>>>>> time of the simulation. ?Obviously, this isn't the right answer.
>>>>>>>>>>>>
>>>>>>>>>>>> I am running scipy 0.7.0 and numpy 1.2.1 on Ubuntu 9.04, but I also
>>>>>>>>>>>> have the same problem on Windows running 0.7.1 and 1.4.0.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>> ?------------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> SciPy-User mailing list
>>>>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> from pylab import *
>>>>>>>>>>> from scipy import signal
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> g = 100.0
>>>>>>>>>>> p = 15.0
>>>>>>>>>>> G = signal.ltisys.lti(g, [1,p,0])
>>>>>>>>>>>
>>>>>>>>>>> t = arange(0, 1.0, 0.002)
>>>>>>>>>>> N = len(t)
>>>>>>>>>>>
>>>>>>>>>>> # u for the whole interval (not used in lsim2, only for plotting later).
>>>>>>>>>>> amp = 50.0
>>>>>>>>>>> u = zeros(N)
>>>>>>>>>>> k1 = 50
>>>>>>>>>>> k2 = 100
>>>>>>>>>>> u[k1:k2] = amp
>>>>>>>>>>>
>>>>>>>>>>> # Create input functions for each smooth interval. (This could be simpler,
>>>>>>>>>>> since u
>>>>>>>>>>> # is constant on each interval.)
>>>>>>>>>>> a = float(k1)/N
>>>>>>>>>>> b = float(k2)/N
>>>>>>>>>>> T1 = linspace(0, a, 201)
>>>>>>>>>>> u1 = zeros_like(T1)
>>>>>>>>>>> T2 = linspace(a, b, 201)
>>>>>>>>>>> u2 = amp*ones_like(T2)
>>>>>>>>>>> T3 = linspace(b, 1.0, 201)
>>>>>>>>>>> u3 = zeros_like(T3)
>>>>>>>>>>>
>>>>>>>>>>> # Solve on each interval; use the final value of one solution as the
>>>>>>>>>>> starting
>>>>>>>>>>> # point of the next solution.
>>>>>>>>>>> # (We could skip the first calculation, since we know the solution will be
>>>>>>>>>>> 0.)
>>>>>>>>>>> (t1, y1, x1) = signal.lsim2(G,u1,T1)
>>>>>>>>>>> (t2, y2, x2) = signal.lsim2(G, u2, T2, X0=x1[-1])
>>>>>>>>>>> (t3, y3, x3) = signal.lsim2(G, u3, T3, X0=x2[-1])
>>>>>>>>>>>
>>>>>>>>>>> figure(1)
>>>>>>>>>>> clf()
>>>>>>>>>>> plot(t, u, 'k', linewidth=3)
>>>>>>>>>>> plot(t1, y1, 'y', linewidth=3)
>>>>>>>>>>> plot(t2, y2, 'b', linewidth=3)
>>>>>>>>>>> plot(t3, y3, 'g', linewidth=3)
>>>>>>>>>>>
>>>>>>>>>>> show()
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> SciPy-User mailing list
>>>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> SciPy-User mailing list
>>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> SciPy-User mailing list
>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SciPy-User mailing list
>>>>>>>> SciPy-User at scipy.org
>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Wed Feb 17 08:45:10 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 17 Feb 2010 08:45:10 -0500
Subject: [SciPy-User] scipy.stats -> deprecation warning
In-Reply-To: <4B7BD5AC.80502@american.edu>
References: <4B7BD5AC.80502@american.edu>
Message-ID: <1cd32cbb1002170545y2f5cf734o4a5d0f7cdad3f407@mail.gmail.com>

On Wed, Feb 17, 2010 at 6:40 AM, Alan G Isaac <aisaac at american.edu> wrote:
> SciPy 0.7.1 (i.e., latest installer):
> scipy.stats.gaussian_kde raises a deprecation warning
> because it calls scipy.stats.cov
>
> Alan Isaac

>From a quick look: this has already be changed in trunk

Thanks,

Josef

> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From zallen at urologicresearchinstitute.org  Wed Feb 17 09:54:10 2010
From: zallen at urologicresearchinstitute.org (URI)
Date: Wed, 17 Feb 2010 14:54:10 +0000 (UTC)
Subject: [SciPy-User] creating a 3D surface plot from collected data
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
Message-ID: <loom.20100217T154806-544@post.gmane.org>

Joe Kington <jkington <at> wisc.edu> writes:

> 
> 
> No worries about the bottom post. I actually think I'm breaking protocol by
top-posting, but I use gmail, and it's just more natural to top-post.Anyway, I
think your problem is coming from the fact that you're working with an
isosurface that fully encloses a volume. I assumed that you had scattered data
points (say, elevation measurements) that you needed to interpolate between.?
The interpolation example I posted implicitly assumes that z = f(x,y). In other
words, that there is only one z value for any given x and y.? If you're working
with a 3D kidney-shape, this is clearly not the case.? Therefore, you're getting
a singluar matrix when you try to interpolate (more than one value of z for
identical rows of x and y in the matrix).The good news is that this may make the
problem much simpler. (Your data is already in some sort of triangle-strip
format if it's been exported from an imaging program. No need to
interpolate.)What happens when you just do:fig = plt.figure()ax =
Axes3D(fig)ax.plot_surface(x,y,z)plt.show()Where x,y,z are your raw data??
There's a reasonable chance that the program you used to export the x,y,z
isosurface left the verticies in the order that plot_surface expects them
in...On a side note, you might also look into Mayavi's mlab module for 3D
plotting.? I prefer it to matplotlib's native 3D plotting. If you have it
installed, you'd just do:from enthought.mayavi import mlabs = mlab.mesh(x, y,
z)mlab.show()Hope that helps some,
> -Joe
> On Tue, Feb 16, 2010 at 6:00 AM, URI <zallen <at>
urologicresearchinstitute.org> wrote:
> >Joe Kington <jkington <at> wisc.edu> writes:
> 
> >

I tried using mlab.mesh from mayavi but apparently the x y z components need to
be 2D arrays (see
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/mlab_helper_functions.html#enthought.mayavi.mlab.mesh).

I don't understand this, how can my list of the x-positions of the vertices be a
2D array?

I did try to use mlab.plot3d (see
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/mlab_helper_functions.html#enthought.mayavi.mlab.plot3d)
and it gave me a nice line drawing of my shape, but an actual surface would be
much nicer.



From jlconlin at gmail.com  Wed Feb 17 10:24:43 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Wed, 17 Feb 2010 08:24:43 -0700
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
Message-ID: <2588da421002170724v11be8e9bh251d52c4eeefd453@mail.gmail.com>

On Tue, Feb 16, 2010 at 8:24 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Feb 16, 2010 at 10:08 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>> I have some data with some errobars that I need to fit to a line. ?Is
>> there anyway to use scipy.polyfit with the error associated with the
>> data? ?If not, how can I make a fit routine work with my data?
>
> If the error is just in the y (dependent) variable, then this may be
> quite simple. If your y values:
> ?-- have a normally distributed error
> ?-- whose standard deviation varies from point to point
> ?-- but the standard deviations are known (up to a multiplicative constant)
> then it's a classic problem -- perhaps google "heterskedasticity
> correction" or "weighted least squares" or "generalized least
> squares". But basically, just make a model matrix X that has one row
> for each observation, and one column for each predictor. If you want
> to do, say, third-order polynomial fitting and you have two vectors of
> independent variables x1 and x2 and you want to include an intercept,
> then it's something like (all untested):
>
> X = np.column_stack([np.ones(len(x1)), x1, x1 ** 2, x1 ** 3, x2, x2 **
> 2, x2 ** 3])
>
> Now a normal least squares fit, ignoring the errorbars, would be:
>
> beta = np.linalg.lstsq(X, y)[0]
>
> with predicted values np.dot(X, beta).
>
> If you have the standard deviation for each measurement in a vector
> stddevs, then the proper heteroskedasticity-corrected fit is just:
>
> beta = np.linalg.lstsq((1 / stddevs) * X, (1 / stddevs) * y)[0]

Yes this is what I have, an array of standard deviations for each data
point.  So this might work. Since I'm doing just a linear fit, I'm
guessing I'd make my X as:

X = np.column_stack([np.ones(len(x1)), x1])

right?

After reading into curve fitting, I realize I know next to nothing
about how this is done.  Can anyone recommend a good book or some sort
of reference to describe curve fitting in general?  I need to
investigate this further.

Thanks,
Jeremy


From robert.kern at gmail.com  Wed Feb 17 10:56:40 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Feb 2010 09:56:40 -0600
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <loom.20100217T154806-544@post.gmane.org>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com> 
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com> 
	<loom.20100217T154806-544@post.gmane.org>
Message-ID: <3d375d731002170756m308a9dacgeaf78704f08816a0@mail.gmail.com>

On Wed, Feb 17, 2010 at 08:54, URI <zallen at urologicresearchinstitute.org> wrote:

> I tried using mlab.mesh from mayavi but apparently the x y z components need to
> be 2D arrays (see
> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/mlab_helper_functions.html#enthought.mayavi.mlab.mesh).
>
> I don't understand this, how can my list of the x-positions of the vertices be a
> 2D array?

It is for rectilinear grids of z=f(x,y) surfaces only.

> I did try to use mlab.plot3d (see
> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/mlab_helper_functions.html#enthought.mayavi.mlab.plot3d)
> and it gave me a nice line drawing of my shape, but an actual surface would be
> much nicer.

Please see the previous posts about using the SurfaceReconstructionFilter.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jh at physics.ucf.edu  Wed Feb 17 12:12:12 2010
From: jh at physics.ucf.edu (Joe Harrington)
Date: Wed, 17 Feb 2010 12:12:12 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <mailman.5207.1266354979.2454.scipy-user@scipy.org>
	(scipy-user-request@scipy.org)
References: <mailman.5207.1266354979.2454.scipy-user@scipy.org>
Message-ID: <wl8vddvokgj.fsf@glup.physics.ucf.edu>

> I have some data with some errobars that I need to fit to a line.

Actually, you should fit the line; please don't change the data! ;-)

The routine below does the job.  I was surprised not to find this
routine in numpy or scipy as it's very standard (I'm still convinced I
must have missed it, so somebody please embarrass me and point it
out).

The routine has a number of parameters that are standard in other
numerical packages, such as IDL, for *every* fitting routine.  We
should consider adding these parameters to all our fitting routines,
as well, since having them encourages people to fit (more) properly
(i.e., actually assess the goodness of fit, etc.).

--jh--

import numpy as np
import scipy.special as ss

def linfit(y, x=None, y_unc=None):
  '''
    Fits a line to 2D data, optionally with errors in y.

    The method is robust to roundoff error.

    Parameters
    ----------
    y: ndarray
        Ordinates, any shape.
    x: ndarray
        Abcissas, same shape.  Defaults to np.indices(y.length))
    y_unc: ndarray
        Uncertainties in y.  If scalar or 1-element array, applied
        uniformly to all y values.  [NOT IMPLEMENTED YET!]  Must be
        positive.

    Returns
    -------
    a: scalar
        0 Fitted intercept
    b: scalar
        1 Fitted slope
    a_unc: scalar
        2 Uncertainty of fitted intercept
    b_unc: scalar
        3 Uncertainty of fitted slope
    chisq: scalar
        4 Chi-squared
    prob: scalar
        5 Probability of finding worse Chi-squared for this model with
          these uncertainties.
    covar: ndarray
        6 Covariance matrix: [[a_unc**2,  covar_ab],
		              [covar_ab,  b_unc**2]]
    yfit: ndarray
        7 Model array calculated for our abcissas
    
    Notes
    -----
    
    If prob > 0.1, you can believe the fit.  If prob > 0.001 and the
    errors are not Gaussian, you could believe the fit.  Otherwise
    do not believe it.

    See Also
    --------
    Press, et al., Numerical Recipes in C, 2nd ed, section 15.2,
    or any standard data analysis text.

    Examples
    --------
    >>> import linfit

    >>> a = 1.
    >>> b = 2.
    >>> nx = 10
    >>> x = np.arange(10, dtype='float')
    >>> y = a + b * x
    >>> y_unc = numpy.ones(nx)
    >>> y[::2]  += 1
    >>> y[1::2] -= 1
    >>> a, b, sa, sb, chisq, prob, covar, yfit = linfit.linfit(y, x, y_unc)
    >>> print(a, b, sa, sb, chisq, prob, covar, yfit)
(1.272727272727272, 1.9393939393939394, 0.58775381364525869, 0.11009637651263605, 9.6969696969696937, 0.28694204178663996, array([[ 0.34545455, -0.05454545],
       [-0.05454545,  0.01212121]]), array([  1.27272727,   3.21212121,   5.15151515,   7.09090909,
         9.03030303,  10.96969697,  12.90909091,  14.84848485,
        16.78787879,  18.72727273]))

    Revisons
    --------
    2007-09-23 0.1  jh at physics.ucf.edu	Initial version
    2007-09-25 0.2  jh at physics.ucf.edu	Fixed bug reported by Kevin Stevenson.
    2008-10-09 0.3  jh at physics.ucf.edu	Fixed doc bug.
    2009-10-01 0.4  jh at physics.ucf.edu  Updated docstring, imports.
  '''
  # standardize and test inputs
  if x == None:
    x = np.indices(y.length, dtype=y.dtype)
    x.shape = y.shape

  if y_unc == None:
      y_unc = np.ones(y.shape, dtype=y.dtype)
  
  # NR Eq. 15.2.4
  ryu2  = 1. / y_unc**2
  S     = np.sum(1.    * ryu2)
  Sx    = np.sum(x     * ryu2)
  Sy    = np.sum(y     * ryu2)
  # Sxx = np.sum(x**2  * ryu2) # not used in the robust method
  # Sxy = np.sum(x * y * ryu2) # not used in the robust method

  # NR Eq. 15.2.15 - 15.2.18 (i.e., the robust method)
  t = 1. / y_unc * (x - Sx / S)
  Stt = np.sum(t**2)

  b = 1. / Stt * np.sum(t * y / y_unc)
  a = (Sy - Sx * b) / S

  covab = -Sx / (S * Stt)                  # NR Eq. 15.2.21

  sa = np.sqrt(1. / S * (1. - Sx * covab)) # NR Eq. 15.2.19
  sb = np.sqrt(1. / Stt)                   # NR Eq. 15.2.20

  rab = covab / (sa * sb)                  # NR Eq. 15.2.22
  
  covar = np.array([[sa**2, covab],
                    [covab, sb**2]])
  
  yfit = a + b * x
  chisq = np.sum( ((y - yfit) / y_unc)**2 )
  
  prob = 1. - ss.gammainc( (y.size - 2.) / 2., chisq / 2.)
  
  return a, b, sa, sb, chisq, prob, covar, yfit



From josef.pktd at gmail.com  Wed Feb 17 12:30:33 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 17 Feb 2010 12:30:33 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <wl8vddvokgj.fsf@glup.physics.ucf.edu>
References: <mailman.5207.1266354979.2454.scipy-user@scipy.org>
	<wl8vddvokgj.fsf@glup.physics.ucf.edu>
Message-ID: <1cd32cbb1002170930s723af3b1ne9d24c055df9ae06@mail.gmail.com>

On Wed, Feb 17, 2010 at 12:12 PM, Joe Harrington <jh at physics.ucf.edu> wrote:
>> I have some data with some errobars that I need to fit to a line.
>
> Actually, you should fit the line; please don't change the data! ;-)
>
> The routine below does the job. ?I was surprised not to find this
> routine in numpy or scipy as it's very standard (I'm still convinced I
> must have missed it, so somebody please embarrass me and point it
> out).

scipy.stats.linregress for linear fit in one explanatory variable, but
ols only, there are no weights.
optimize.curve_curvefit uses weights but is intended for non-linear
(in parameters), and scipy.odr is very flexible.

And of course there is scikits.statsmodels for this and much more.

Josef

> The routine has a number of parameters that are standard in other
> numerical packages, such as IDL, for *every* fitting routine. ?We
> should consider adding these parameters to all our fitting routines,
> as well, since having them encourages people to fit (more) properly
> (i.e., actually assess the goodness of fit, etc.).
>
> --jh--
>
> import numpy as np
> import scipy.special as ss
>
> def linfit(y, x=None, y_unc=None):
> ?'''
> ? ?Fits a line to 2D data, optionally with errors in y.
>
> ? ?The method is robust to roundoff error.
>
> ? ?Parameters
> ? ?----------
> ? ?y: ndarray
> ? ? ? ?Ordinates, any shape.
> ? ?x: ndarray
> ? ? ? ?Abcissas, same shape. ?Defaults to np.indices(y.length))
> ? ?y_unc: ndarray
> ? ? ? ?Uncertainties in y. ?If scalar or 1-element array, applied
> ? ? ? ?uniformly to all y values. ?[NOT IMPLEMENTED YET!] ?Must be
> ? ? ? ?positive.
>
> ? ?Returns
> ? ?-------
> ? ?a: scalar
> ? ? ? ?0 Fitted intercept
> ? ?b: scalar
> ? ? ? ?1 Fitted slope
> ? ?a_unc: scalar
> ? ? ? ?2 Uncertainty of fitted intercept
> ? ?b_unc: scalar
> ? ? ? ?3 Uncertainty of fitted slope
> ? ?chisq: scalar
> ? ? ? ?4 Chi-squared
> ? ?prob: scalar
> ? ? ? ?5 Probability of finding worse Chi-squared for this model with
> ? ? ? ? ?these uncertainties.
> ? ?covar: ndarray
> ? ? ? ?6 Covariance matrix: [[a_unc**2, ?covar_ab],
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[covar_ab, ?b_unc**2]]
> ? ?yfit: ndarray
> ? ? ? ?7 Model array calculated for our abcissas
>
> ? ?Notes
> ? ?-----
>
> ? ?If prob > 0.1, you can believe the fit. ?If prob > 0.001 and the
> ? ?errors are not Gaussian, you could believe the fit. ?Otherwise
> ? ?do not believe it.
>
> ? ?See Also
> ? ?--------
> ? ?Press, et al., Numerical Recipes in C, 2nd ed, section 15.2,
> ? ?or any standard data analysis text.
>
> ? ?Examples
> ? ?--------
> ? ?>>> import linfit
>
> ? ?>>> a = 1.
> ? ?>>> b = 2.
> ? ?>>> nx = 10
> ? ?>>> x = np.arange(10, dtype='float')
> ? ?>>> y = a + b * x
> ? ?>>> y_unc = numpy.ones(nx)
> ? ?>>> y[::2] ?+= 1
> ? ?>>> y[1::2] -= 1
> ? ?>>> a, b, sa, sb, chisq, prob, covar, yfit = linfit.linfit(y, x, y_unc)
> ? ?>>> print(a, b, sa, sb, chisq, prob, covar, yfit)
> (1.272727272727272, 1.9393939393939394, 0.58775381364525869, 0.11009637651263605, 9.6969696969696937, 0.28694204178663996, array([[ 0.34545455, -0.05454545],
> ? ? ? [-0.05454545, ?0.01212121]]), array([ ?1.27272727, ? 3.21212121, ? 5.15151515, ? 7.09090909,
> ? ? ? ? 9.03030303, ?10.96969697, ?12.90909091, ?14.84848485,
> ? ? ? ?16.78787879, ?18.72727273]))
>
> ? ?Revisons
> ? ?--------
> ? ?2007-09-23 0.1 ?jh at physics.ucf.edu ?Initial version
> ? ?2007-09-25 0.2 ?jh at physics.ucf.edu ?Fixed bug reported by Kevin Stevenson.
> ? ?2008-10-09 0.3 ?jh at physics.ucf.edu ?Fixed doc bug.
> ? ?2009-10-01 0.4 ?jh at physics.ucf.edu ?Updated docstring, imports.
> ?'''
> ?# standardize and test inputs
> ?if x == None:
> ? ?x = np.indices(y.length, dtype=y.dtype)
> ? ?x.shape = y.shape
>
> ?if y_unc == None:
> ? ? ?y_unc = np.ones(y.shape, dtype=y.dtype)
>
> ?# NR Eq. 15.2.4
> ?ryu2 ?= 1. / y_unc**2
> ?S ? ? = np.sum(1. ? ?* ryu2)
> ?Sx ? ?= np.sum(x ? ? * ryu2)
> ?Sy ? ?= np.sum(y ? ? * ryu2)
> ?# Sxx = np.sum(x**2 ?* ryu2) # not used in the robust method
> ?# Sxy = np.sum(x * y * ryu2) # not used in the robust method
>
> ?# NR Eq. 15.2.15 - 15.2.18 (i.e., the robust method)
> ?t = 1. / y_unc * (x - Sx / S)
> ?Stt = np.sum(t**2)
>
> ?b = 1. / Stt * np.sum(t * y / y_unc)
> ?a = (Sy - Sx * b) / S
>
> ?covab = -Sx / (S * Stt) ? ? ? ? ? ? ? ? ?# NR Eq. 15.2.21
>
> ?sa = np.sqrt(1. / S * (1. - Sx * covab)) # NR Eq. 15.2.19
> ?sb = np.sqrt(1. / Stt) ? ? ? ? ? ? ? ? ? # NR Eq. 15.2.20
>
> ?rab = covab / (sa * sb) ? ? ? ? ? ? ? ? ?# NR Eq. 15.2.22
>
> ?covar = np.array([[sa**2, covab],
> ? ? ? ? ? ? ? ? ? ?[covab, sb**2]])
>
> ?yfit = a + b * x
> ?chisq = np.sum( ((y - yfit) / y_unc)**2 )
>
> ?prob = 1. - ss.gammainc( (y.size - 2.) / 2., chisq / 2.)
>
> ?return a, b, sa, sb, chisq, prob, covar, yfit
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From njs at pobox.com  Wed Feb 17 13:04:11 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 17 Feb 2010 10:04:11 -0800
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <wl8vddvokgj.fsf@glup.physics.ucf.edu>
References: <mailman.5207.1266354979.2454.scipy-user@scipy.org>
	<wl8vddvokgj.fsf@glup.physics.ucf.edu>
Message-ID: <961fa2b41002171004l5452ff5i90bd1fdafeb93ef8@mail.gmail.com>

On Wed, Feb 17, 2010 at 9:12 AM, Joe Harrington <jh at physics.ucf.edu> wrote:
> The routine below does the job. ?I was surprised not to find this
> routine in numpy or scipy as it's very standard (I'm still convinced I
> must have missed it, so somebody please embarrass me and point it
> out).

Err, it may be very standard, but what is it? :-) It doesn't jump out
at me as being a least-squares fitter (or if it is then it's
implemented in a strange way), and I don't have a copy of Numerical
Recipes handy. (Nor, honestly, does NR have a great reputation when it
comes to statistical stuff, AFAICT.)

> The routine has a number of parameters that are standard in other
> numerical packages, such as IDL, for *every* fitting routine. ?We
> should consider adding these parameters to all our fitting routines,
> as well, since having them encourages people to fit (more) properly
> (i.e., actually assess the goodness of fit, etc.).

Here I certainly agree. (Cf. R again, where it is just assumed that
every fitting routine no matter how ordinary or exotic will have a
'weights' argument, diagnostic plots, etc.)

-- Nathaniel


From jsseabold at gmail.com  Wed Feb 17 13:06:24 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 17 Feb 2010 13:06:24 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <2588da421002170724v11be8e9bh251d52c4eeefd453@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com> 
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com> 
	<2588da421002170724v11be8e9bh251d52c4eeefd453@mail.gmail.com>
Message-ID: <c048da1c1002171006u3489c3afw843b1ce245383c85@mail.gmail.com>

On Wed, Feb 17, 2010 at 10:24 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
<snip>
> After reading into curve fitting, I realize I know next to nothing
> about how this is done. ?Can anyone recommend a good book or some sort
> of reference to describe curve fitting in general? ?I need to
> investigate this further.
>

What is your background?  I find examples that are in terms of things
I am intuitively familiar with to be much more helpful, though as you
learn more this becomes less important.

I found this to be a decent general intro (if you can find it):
Introduction to Linear Regression Analysis by Montgomery and Peck

Though, I haven't looked through much else along these lines.  Maybe
others have ideas.

Undergrad/ Masters level texts for econometrics are usually

Stock and Watson's Introduction to Econometrics
or
Wooldridge's Introductory Econometrics: A Modern Approach

I prefer the latter but just my opinion.

Skipper


From jlconlin at gmail.com  Wed Feb 17 13:15:16 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Wed, 17 Feb 2010 11:15:16 -0700
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <c048da1c1002171006u3489c3afw843b1ce245383c85@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
	<2588da421002170724v11be8e9bh251d52c4eeefd453@mail.gmail.com>
	<c048da1c1002171006u3489c3afw843b1ce245383c85@mail.gmail.com>
Message-ID: <2588da421002171015n68127e0bu91b8413f61c1d05b@mail.gmail.com>

On Wed, Feb 17, 2010 at 11:06 AM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Wed, Feb 17, 2010 at 10:24 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
> <snip>
>> After reading into curve fitting, I realize I know next to nothing
>> about how this is done. ?Can anyone recommend a good book or some sort
>> of reference to describe curve fitting in general? ?I need to
>> investigate this further.
>>
>
> What is your background? ?I find examples that are in terms of things
> I am intuitively familiar with to be much more helpful, though as you
> learn more this becomes less important.

I should have given my background.  I have a PhD in engineering where
my work was in computational algorithms.  However, at the time, I
didn't need to fit a curve to my data to any known models.  Now that I
have a job, this becomes necessary so I need to learn it.
>
> I found this to be a decent general intro (if you can find it):
> Introduction to Linear Regression Analysis by Montgomery and Peck

Thanks for that recommendation.  I will get this.
>
> Though, I haven't looked through much else along these lines. ?Maybe
> others have ideas.
>
> Undergrad/ Masters level texts for econometrics are usually
>
> Stock and Watson's Introduction to Econometrics
> or
> Wooldridge's Introductory Econometrics: A Modern Approach
>
> I prefer the latter but just my opinion.
>
> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From jsseabold at gmail.com  Wed Feb 17 13:17:42 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 17 Feb 2010 13:17:42 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <961fa2b41002171004l5452ff5i90bd1fdafeb93ef8@mail.gmail.com>
References: <mailman.5207.1266354979.2454.scipy-user@scipy.org> 
	<wl8vddvokgj.fsf@glup.physics.ucf.edu>
	<961fa2b41002171004l5452ff5i90bd1fdafeb93ef8@mail.gmail.com>
Message-ID: <c048da1c1002171017y134df76fi9a834d4dd99f808c@mail.gmail.com>

On Wed, Feb 17, 2010 at 1:04 PM, Nathaniel Smith <njs at pobox.com> wrote:
<snip>
>> The routine has a number of parameters that are standard in other
>> numerical packages, such as IDL, for *every* fitting routine. ?We
>> should consider adding these parameters to all our fitting routines,
>> as well, since having them encourages people to fit (more) properly
>> (i.e., actually assess the goodness of fit, etc.).
>
> Here I certainly agree. (Cf. R again, where it is just assumed that
> every fitting routine no matter how ordinary or exotic will have a
> 'weights' argument, diagnostic plots, etc.)
>

FWIW, statsmodels, where appropriate for the most part, allows
'weight' arguments.  The barrier to the plotting in scipy is the no
graphics policy.  Granted statsmodels might as well have the same
policy at this point since it is a bit lacking here, but there are
statistical measures for (some) exploratory and confirmatory analysis
(though not nearly enough compared to R) if not too much as far as
plots.  It's not a giant leap to get what you need from matplotlib,
etc.  Just not many canned routines.  The nice thing is it takes very
little time to get these implemented (once the need/motivation/time is
there...).

Skipper


From robert.kern at gmail.com  Wed Feb 17 13:23:12 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Feb 2010 12:23:12 -0600
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <2588da421002171015n68127e0bu91b8413f61c1d05b@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com> 
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com> 
	<2588da421002170724v11be8e9bh251d52c4eeefd453@mail.gmail.com> 
	<c048da1c1002171006u3489c3afw843b1ce245383c85@mail.gmail.com> 
	<2588da421002171015n68127e0bu91b8413f61c1d05b@mail.gmail.com>
Message-ID: <3d375d731002171023r57c0e717v568af56a38087709@mail.gmail.com>

On Wed, Feb 17, 2010 at 12:15, Jeremy Conlin <jlconlin at gmail.com> wrote:
> On Wed, Feb 17, 2010 at 11:06 AM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> On Wed, Feb 17, 2010 at 10:24 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>> <snip>
>>> After reading into curve fitting, I realize I know next to nothing
>>> about how this is done. ?Can anyone recommend a good book or some sort
>>> of reference to describe curve fitting in general? ?I need to
>>> investigate this further.
>>>
>>
>> What is your background? ?I find examples that are in terms of things
>> I am intuitively familiar with to be much more helpful, though as you
>> learn more this becomes less important.
>
> I should have given my background. ?I have a PhD in engineering where
> my work was in computational algorithms. ?However, at the time, I
> didn't need to fit a curve to my data to any known models. ?Now that I
> have a job, this becomes necessary so I need to learn it.
>>
>> I found this to be a decent general intro (if you can find it):
>> Introduction to Linear Regression Analysis by Montgomery and Peck
>
> Thanks for that recommendation. ?I will get this.

Honestly, I'd recommend staying away from any sources that use the
word "regression". They are perfectly fine sources in general, but
they come from a very different background and have a very different
focus from what engineers need.

Bevington and Robinson's _Data Reduction and Error Analysis for The
Physical Sciences_ is a good, though old, book that has more of a
physics and engineering focus.

http://www.amazon.com/Reduction-Error-Analysis-Physical-Sciences/dp/0079112439

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jsseabold at gmail.com  Wed Feb 17 13:28:21 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 17 Feb 2010 13:28:21 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <3d375d731002171023r57c0e717v568af56a38087709@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com> 
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com> 
	<2588da421002170724v11be8e9bh251d52c4eeefd453@mail.gmail.com> 
	<c048da1c1002171006u3489c3afw843b1ce245383c85@mail.gmail.com> 
	<2588da421002171015n68127e0bu91b8413f61c1d05b@mail.gmail.com> 
	<3d375d731002171023r57c0e717v568af56a38087709@mail.gmail.com>
Message-ID: <c048da1c1002171028k33104b76r3c24944ec3485726@mail.gmail.com>

On Wed, Feb 17, 2010 at 1:23 PM, Robert Kern <robert.kern at gmail.com> wrote:
> Honestly, I'd recommend staying away from any sources that use the
> word "regression". They are perfectly fine sources in general, but
> they come from a very different background and have a very different
> focus from what engineers need.
>

Fair enough and far afield from what I know though
<http://www.eas.asu.edu/~masmlab/montgomery/> and the book covers the
OP's original question.

> Bevington and Robinson's _Data Reduction and Error Analysis for The
> Physical Sciences_ is a good, though old, book that has more of a
> physics and engineering focus.
>
> http://www.amazon.com/Reduction-Error-Analysis-Physical-Sciences/dp/0079112439


From robert.kern at gmail.com  Wed Feb 17 13:31:47 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Feb 2010 12:31:47 -0600
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <c048da1c1002171028k33104b76r3c24944ec3485726@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com> 
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com> 
	<2588da421002170724v11be8e9bh251d52c4eeefd453@mail.gmail.com> 
	<c048da1c1002171006u3489c3afw843b1ce245383c85@mail.gmail.com> 
	<2588da421002171015n68127e0bu91b8413f61c1d05b@mail.gmail.com> 
	<3d375d731002171023r57c0e717v568af56a38087709@mail.gmail.com> 
	<c048da1c1002171028k33104b76r3c24944ec3485726@mail.gmail.com>
Message-ID: <3d375d731002171031u764ff388rf8c368c6d0bf1d42@mail.gmail.com>

On Wed, Feb 17, 2010 at 12:28, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Wed, Feb 17, 2010 at 1:23 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> Honestly, I'd recommend staying away from any sources that use the
>> word "regression". They are perfectly fine sources in general, but
>> they come from a very different background and have a very different
>> focus from what engineers need.
>
> Fair enough and far afield from what I know though
> <http://www.eas.asu.edu/~masmlab/montgomery/> and the book covers the
> OP's original question.

Ah, good. A data point against the heuristic, then. :-)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Wed Feb 17 13:33:09 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 17 Feb 2010 11:33:09 -0700
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
Message-ID: <e06186141002171033k7e2e37f6tb9f06a287306d7f@mail.gmail.com>

On Tue, Feb 16, 2010 at 11:08 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:

> I have some data with some errobars that I need to fit to a line.  Is
> there anyway to use scipy.polyfit with the error associated with the
> data?  If not, how can I make a fit routine work with my data?
>
>
Least squares is the best unbiased linear estimator when all the errors have
equal variance. If you have the Vandermonde matrix for the degree of fit you
want all you need to do is scale each row so the estimated error on the rhs
is 1. Then use least squares to solve the resulting equation Ax = y for x.
Weights should probably be added as an option in the polyfit routines. In
practice, weighting the data doesn't make much difference.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100217/7fb657e9/attachment.html>

From jh at physics.ucf.edu  Wed Feb 17 17:25:47 2010
From: jh at physics.ucf.edu (Joe Harrington)
Date: Wed, 17 Feb 2010 17:25:47 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <961fa2b41002171004l5452ff5i90bd1fdafeb93ef8@mail.gmail.com>
	(message from Nathaniel Smith on Wed, 17 Feb 2010 10:04:11 -0800)
References: <mailman.5207.1266354979.2454.scipy-user@scipy.org>
	<wl8vddvokgj.fsf@glup.physics.ucf.edu>
	<961fa2b41002171004l5452ff5i90bd1fdafeb93ef8@mail.gmail.com>
Message-ID: <wl8r5ojo5xw.fsf@glup.physics.ucf.edu>

On Wed, 17 Feb 2010 10:04:11 -0800, Nathaniel Smith wrote:

> On Wed, Feb 17, 2010 at 9:12 AM, Joe Harrington <jh at physics.ucf.edu> wrote:,
> > The routine below does the job. ?I was surprised not to find this
> > routine in numpy or scipy as it's very standard (I'm still convinced I
> > must have missed it, so somebody please embarrass me and point it
> > out).
> 
> Err, it may be very standard, but what is it? :-) It doesn't jump out
> at me as being a least-squares fitter (or if it is then it's
> implemented in a strange way), and I don't have a copy of Numerical
> Recipes handy. (Nor, honestly, does NR have a great reputation when it
> comes to statistical stuff, AFAICT.)

You can get the 2nd ed. of NR (cited in the routine) free here:

http://www.nrbook.com/a/bookcpdf.php

NR is like PERL: the wrong solution for every (well, many) job(s).  It
has a basic introduction to a large number of topics that is
accessible to a general reader.  It is highly available, and older
editions are free.  Specialists in almost every discipline complain
about it, and not without justification.  Of course, they would have
you read several semesters' worth of background material before you
even attempted your first plot in their particular areas, which
generally snuffs out all interest in the beginner.  So, read NR and
get the hang of something, play with things a bit, but then DO read
some specialized sources before you do things "for real".  If you
happen to have a good intro book in a specialized area, of course,
skip NR entirely.

Regarding the routine:

There is an analytic solution to the least-squares line fit (assuming
Gaussian errors, independent measurements, etc. etc.), which is taught
in most calculus curricula.  The routine here is a variant on that
one.

For the OP:

Some more-complicated functions also have analyic solutions, but this
is not generally true of all functions.  Function minimizers
numerically explore the chi-squared surface in the phase space of the
function parameters looking for the minimum, but it is not generally
possible to determine whether the minimum they find is the global
minimum or just a local one.  Some routines are better than others,
faster than others, more easily fooled than others, etc.

Also, determining the errors on the fitted function parameters usually
requires a Monte Carlo approach; if there is any correlation among
parameters, you cannot use the diagonal elements of the covariance
matrix that most minimizers report as errors.  Markov-chain Monte
Carlo routines are popular for making such estimates, but they can be
computationally costly, depending on the number of parameters.  There
are also some Monte-Carlo approaches to be wary of, such as bootstrap.
One of the places where NR falls short is that in the 2nd ed (1992)
they presented a very simplistic view of bootstrap that did not
discuss its limitations nor mention Markov chains or other
approaches.  They do now discuss those in the 3rd edition.

--jh--


From njs at pobox.com  Wed Feb 17 22:11:11 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 17 Feb 2010 19:11:11 -0800
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <2588da421002171015n68127e0bu91b8413f61c1d05b@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com>
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com>
	<2588da421002170724v11be8e9bh251d52c4eeefd453@mail.gmail.com>
	<c048da1c1002171006u3489c3afw843b1ce245383c85@mail.gmail.com>
	<2588da421002171015n68127e0bu91b8413f61c1d05b@mail.gmail.com>
Message-ID: <961fa2b41002171911s3dff6123idc5505db993cad19@mail.gmail.com>

On Wed, Feb 17, 2010 at 10:15 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
> I should have given my background. ?I have a PhD in engineering where
> my work was in computational algorithms. ?However, at the time, I
> didn't need to fit a curve to my data to any known models. ?Now that I
> have a job, this becomes necessary so I need to learn it.

I recently enjoyed Cosma Shalizi's lecture notes on regression, which
are available at
  http://www.stat.cmu.edu/~cshalizi/350/
(lectures 16-18)

Cheaper than a book, anyway :-)

-- Nathaniel


From jsseabold at gmail.com  Thu Feb 18 00:41:55 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 18 Feb 2010 00:41:55 -0500
Subject: [SciPy-User] How to fit data with errorbars
In-Reply-To: <961fa2b41002171911s3dff6123idc5505db993cad19@mail.gmail.com>
References: <2588da421002161008p2de9f13fgd6017a0089ace953@mail.gmail.com> 
	<961fa2b41002161924s5613e816h2dd5c1bc94cd5a0e@mail.gmail.com> 
	<2588da421002170724v11be8e9bh251d52c4eeefd453@mail.gmail.com> 
	<c048da1c1002171006u3489c3afw843b1ce245383c85@mail.gmail.com> 
	<2588da421002171015n68127e0bu91b8413f61c1d05b@mail.gmail.com> 
	<961fa2b41002171911s3dff6123idc5505db993cad19@mail.gmail.com>
Message-ID: <c048da1c1002172141h3ab3ac71n68523b48b9ef08bf@mail.gmail.com>

On Wed, Feb 17, 2010 at 10:11 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Wed, Feb 17, 2010 at 10:15 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>> I should have given my background. ?I have a PhD in engineering where
>> my work was in computational algorithms. ?However, at the time, I
>> didn't need to fit a curve to my data to any known models. ?Now that I
>> have a job, this becomes necessary so I need to learn it.
>
> I recently enjoyed Cosma Shalizi's lecture notes on regression, which
> are available at
> ?http://www.stat.cmu.edu/~cshalizi/350/
> (lectures 16-18)
>
> Cheaper than a book, anyway :-)
>

This is a nice resource and looks like a fun class.  Thanks,

Skipper


From abc.mikey at googlemail.com  Thu Feb 18 08:59:56 2010
From: abc.mikey at googlemail.com (mikey)
Date: Thu, 18 Feb 2010 13:59:56 +0000
Subject: [SciPy-User] probplot's return value r or r^2?
Message-ID: <4ebfceac1002180559k4d92a4fbk1576ca4bf80b8bc7@mail.gmail.com>

Hi there,

Hopefully you can help me with a couple of things I'm not getting
about scipy's stats package. Both of which are shown on this image:
http://img198.yfrog.com/img198/8928/normalqqplotsection3tot.png

Firstly I want to know what is returned by the r value from probplot?
The documentation says that it returns r but if you have it plot the
graph for you it plots r^2. If you compare them you find that the
returned value r and the plotted result for r^2 are the same, so which
is it?

Also I'm trying to work out what the p value from the shapiro test are
showing me. I've plotted the W and p values from the test on my
probplot for my data  and the probplot shows a good correlation
between my data and the expected distribution for normal data but p is
too low to accept the null hypothesis that it is normal. Am I
interpereting what it's showing me properly, or is it just my data?

Thanks for your help,

Mikey


From josef.pktd at gmail.com  Thu Feb 18 09:55:23 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 09:55:23 -0500
Subject: [SciPy-User] probplot's return value r or r^2?
In-Reply-To: <4ebfceac1002180559k4d92a4fbk1576ca4bf80b8bc7@mail.gmail.com>
References: <4ebfceac1002180559k4d92a4fbk1576ca4bf80b8bc7@mail.gmail.com>
Message-ID: <1cd32cbb1002180655l782f7316mc623a79002a32706@mail.gmail.com>

On Thu, Feb 18, 2010 at 8:59 AM, mikey <abc.mikey at googlemail.com> wrote:
> Hi there,
>
> Hopefully you can help me with a couple of things I'm not getting
> about scipy's stats package. Both of which are shown on this image:
> http://img198.yfrog.com/img198/8928/normalqqplotsection3tot.png
>
> Firstly I want to know what is returned by the r value from probplot?
> The documentation says that it returns r but if you have it plot the
> graph for you it plots r^2. If you compare them you find that the
> returned value r and the plotted result for r^2 are the same, so which
> is it?

general
There is a gap in the docstring for linregress, no explanation for return r
I fixed probplot so that the graphs work and the results look
"reasonable" but I never verified whether all the results are correct.
I think I verified shapiro against R and it works in Monte Carlo, but
I have to look at my notes to be sure.

to the question
probplot uses stats.linregress to estimate the line in the probplot,
and r is taken from it. r is just the correlation coefficient of x and
y. I don't remember the relationship to the regression R^2 offhand,
whether it's squared or not

>>> tmpx, tmpy = np.random.randn(2,10)
>>> tmpx
array([ 0.07649761, -3.12336718, -0.10403981, -1.26699021,  0.91165385,
       -0.78344642,  1.6671443 ,  2.15933311, -1.24495897, -2.67112134])
>>> np.corrcoef(tmpx,tmpy)
array([[ 1.        ,  0.36073832],
       [ 0.36073832,  1.        ]])
>>> stats.linregress(tmpx,tmpy)
(0.24698183423380241, 0.15082677908334857, 0.36073832198115868,
0.30580467614265466, 0.22576383851968623)


>
> Also I'm trying to work out what the p value from the shapiro test are
> showing me. I've plotted the W and p values from the test on my
> probplot for my data ?and the probplot shows a good correlation
> between my data and the expected distribution for normal data but p is
> too low to accept the null hypothesis that it is normal. Am I
> interpereting what it's showing me properly, or is it just my data?

>From the graph it looks like your data is discrete, is it? In that
case, I wouldn't be surprised if the normality hypothesis is rejected.
I'm not familiar with the details of the Shapiro-Wilk test and the
source is in Fortran, but from a brief look at the Wikipedia
description, it seems to be a quite strict test on the quantiles.

You could also try the other normality tests in scipy.stats, kstest,
normaltest, anderson, chisquare ? to see whether they also reject the
hypothesis of a normal distribution. (As an aside, Skipper started to
add qqplot and residual tests in statsmodels, but they are still
sandbox code.)

I appreciate any feedback on these function, since they are still
under-documented and I'm not sure whether all the results are correct.
And for some functions, I'm not sure whether they are actually used by
anyone. (For example, the plotting was broken in stats.probplot for
quite some time and I didn't find any comments in the mailing list
archive)

Josef

>
> Thanks for your help,
>
> Mikey
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From denis-bz-gg at t-online.de  Thu Feb 18 10:19:39 2010
From: denis-bz-gg at t-online.de (denis)
Date: Thu, 18 Feb 2010 07:19:39 -0800 (PST)
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
Message-ID: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>

A followup to the 15feb thread "creating a 3D surface plot from
collected data":
the doc for scipy.interpolate.rbf says that 3 of the 7 radial
functions
use a scale paramater `epsilon`, 4 do not.
Odd -- how can some be scale-free ?
Running rbf on 100 1d random.uniform input points
(after changing the linalg.solve in rbf.py to lstsq)
shows that all but linear and gaussian are noisy,
giving interpolants way outside the input range:

    N: 100  ngrid: 50  sigma: 0.1
    # min  max  max |delta|  Rbf
     -1.0  1.0   0.5  gaussian
     -1.0  1.2   0.3  linear
     -1.4  2.1   2.5  thin-plate
     -1.0  2.2   3.0  inverse multiquadric
     -1.0  2.4   3.2  multiquadric
     -2.1 12.8   7.4  quintic
    -10.6  7.0  11.7  cubic

Should Rbf be moved off to noisymethods/...  until experts can look it
over,
or have I done something stupid ?
(A back-of-the-envelope method for choosing epsilon aka r0 would be
nice,
and a rough noise amplification model would be nice too.)

Also, the doc should have a link to
http://www.scipy.org/Cookbook/Matplotlib/Gridding_irregularly_spaced_data
and should say that the initial rbf() is N^2.

cheers
  -- denis

<pre><code>
    # from http://scipy.org/Cookbook/RadialBasisFunctions
    import sys
    import numpy as np
    from rbf import Rbf  # lstsq
    # from scipy.interpolate import Rbf
    Rbfuncs = ( 'gaussian', 'linear', 'thin-plate', 'inverse
multiquadric',
        'multiquadric', 'quintic', 'cubic', )

    N = 100
    ngrid = 50
    sigma = .1
    exec "\n".join( sys.argv[1:] )  # N=
    np.random.seed(1)
    np.set_printoptions( 2, threshold=50, edgeitems=5,
suppress=True )  # .2f

    x = np.sort( np.random.uniform( 0, 10, N+1 ))
    y = np.sin(x) + np.random.normal( 0, sigma, x.shape )
    xi = np.linspace( 0, 10, ngrid+1 )

    print "N: %d  ngrid: %d  sigma: %.2g" % (N, ngrid, sigma)
    print "# min  max  max |delta|  Rbf"
    for func in Rbfuncs:
        rbf = Rbf( x, y, function=func )
        fi = rbf(xi)
        maxdiff = np.amax( np.abs( np.diff( fi )))
        print "%5.1f %4.1f  %4.1f  %s" % (
            fi.min(), fi.max(), maxdiff, func )
</pre></code>


From robert.kern at gmail.com  Thu Feb 18 10:48:51 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 18 Feb 2010 09:48:51 -0600
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
Message-ID: <3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com>

On Thu, Feb 18, 2010 at 09:19, denis <denis-bz-gg at t-online.de> wrote:
> A followup to the 15feb thread "creating a 3D surface plot from
> collected data":
> the doc for scipy.interpolate.rbf says that 3 of the 7 radial
> functions
> use a scale paramater `epsilon`, 4 do not.
> Odd -- how can some be scale-free ?
> Running rbf on 100 1d random.uniform input points
> (after changing the linalg.solve in rbf.py to lstsq)

Why would you do this? This does not magically turn Rbf interpolation
into a smoothing approximation. Use the "smooth" keyword to specify a
smoothing parameter.

> shows that all but linear and gaussian are noisy,
> giving interpolants way outside the input range:
>
> ? ?N: 100 ?ngrid: 50 ?sigma: 0.1
> ? ?# min ?max ?max |delta| ?Rbf
> ? ? -1.0 ?1.0 ? 0.5 ?gaussian
> ? ? -1.0 ?1.2 ? 0.3 ?linear
> ? ? -1.4 ?2.1 ? 2.5 ?thin-plate
> ? ? -1.0 ?2.2 ? 3.0 ?inverse multiquadric
> ? ? -1.0 ?2.4 ? 3.2 ?multiquadric
> ? ? -2.1 12.8 ? 7.4 ?quintic
> ? ?-10.6 ?7.0 ?11.7 ?cubic
>
> Should Rbf be moved off to noisymethods/... ?until experts can look it
> over,
> or have I done something stupid ?

I'm not sure what else you would expect from interpolating uniform
random noise. Many interpolation methods tend to give such
oscillations on such data.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jkington at wisc.edu  Thu Feb 18 12:31:49 2010
From: jkington at wisc.edu (Joe Kington)
Date: Thu, 18 Feb 2010 11:31:49 -0600
Subject: [SciPy-User] creating a 3D surface plot from collected data
In-Reply-To: <loom.20100217T154806-544@post.gmane.org>
References: <loom.20100215T145249-313@post.gmane.org>
	<f6951adf1002150920g51e3a9d0jb12661ec21e910fc@mail.gmail.com>
	<loom.20100216T124608-106@post.gmane.org>
	<f6951adf1002160623k6519ba2do2afe58362c406e65@mail.gmail.com>
	<loom.20100217T154806-544@post.gmane.org>
Message-ID: <f6951adf1002180931h3134a920t9d9dd3adab5d7613@mail.gmail.com>

Hi there,

 I don't understand this, how can my list of the x-positions of the vertices
> be a
> 2D array?
>

mlab's mesh expects 2d grids of x,y, and z values so that it can infer how
the points are connected.

Because you just have lists of x,y,z verticies, there's no pre-defined way
to connect them.  Therefore, you need some way to define how they connect
into a surface.  (The program you exported them from probably stored them as
triangle-strip mesh.  You have the verticies, but don't have the definition
of how they link into individual triangles)

The surface reconstruction filter Robert mentioned is one way (and probably
the best way) of doing this.  (Credit where it's due... I had no idea vtk
had that particular filter before Robert mentioned it!)

It's slightly confusing, as it's not quite as simple as just replacing the
delaunay2d in Gael's earlier example with a call to the
SurfaceReconstructionFilter.

The surface reconstruction filter returns a volume that you need to extract
an isosurface at a value of 0 from.  (It took me awhile to figure that
out... Guess I should have read the vtk docs a bit closer!)

If this helps, here's a very artificial example using the surface
reconstruction filter.  It will produce some artifacts, but it should (?)
work for your data.... I think...  The first half just builds a set of
points on the outside of a sphere.  (capital) X,Y,Z would be representative
of your data.

from enthought.mayavi import mlab
import numpy as np

# First off, let's make 2d grids of phi and theta
phi = np.linspace(-np.pi, np.pi, 100)
theta = np.linspace(0, np.pi*2, 100)
phi, theta = np.meshgrid(phi, theta)

# Now lets make points on the outside of a sphere and
# put things back into cartesian coordinates
r = 1000
x = r * np.sin(phi) * np.cos(theta)
y = r * np.cos(phi)
z = r * np.sin(phi) * np.sin(theta)

# Since x,y,z are all regularly spaced 2d arrays, there's
# a defined way to link up each x,y,z coordinate into a
# surface (if that makes sense).  We can plot this using mesh
mlab.figure()
mlab.mesh(x,y,z)
mlab.title('The ideal surface')

# Now, lets "lose" the connectivity information by flattening the
# arrays.  This is more like what you have... A bunch of x,y,z
# coordinates of points on a surface with no clear definition
# of how to connect them.
X, Y, Z = [item.flatten() for item in [x,y,z]]

# Now we have to reconstruct how they connect into a surface
mlab.figure()
# Make a point source from the x,y,z coordinates
pts = mlab.pipeline.scalar_scatter(X,Y,Z,Z)

# This creates a volume of the distance to the reconstructed surface
vol = mlab.pipeline.user_defined(pts, filter='SurfaceReconstructionFilter')

# We can then extract an iso_surface at 0 that should be close to the
# original surface
mlab.pipeline.iso_surface(vol, contours=[0])
mlab.title('The reconstructed surface\nNotice the artifacts')
mlab.show()

Hope that helps,
-Joe

On Wed, Feb 17, 2010 at 8:54 AM, URI
<zallen at urologicresearchinstitute.org>wrote:

> Joe Kington <jkington <at> wisc.edu> writes:
>
> >
> >
> > No worries about the bottom post. I actually think I'm breaking protocol
> by
> top-posting, but I use gmail, and it's just more natural to
> top-post.Anyway, I
> think your problem is coming from the fact that you're working with an
> isosurface that fully encloses a volume. I assumed that you had scattered
> data
> points (say, elevation measurements) that you needed to interpolate
> between.
> The interpolation example I posted implicitly assumes that z = f(x,y). In
> other
> words, that there is only one z value for any given x and y.  If you're
> working
> with a 3D kidney-shape, this is clearly not the case.  Therefore, you're
> getting
> a singluar matrix when you try to interpolate (more than one value of z for
> identical rows of x and y in the matrix).The good news is that this may
> make the
> problem much simpler. (Your data is already in some sort of triangle-strip
> format if it's been exported from an imaging program. No need to
> interpolate.)What happens when you just do:fig = plt.figure()ax =
> Axes3D(fig)ax.plot_surface(x,y,z)plt.show()Where x,y,z are your raw data?
> There's a reasonable chance that the program you used to export the x,y,z
> isosurface left the verticies in the order that plot_surface expects them
> in...On a side note, you might also look into Mayavi's mlab module for 3D
> plotting.  I prefer it to matplotlib's native 3D plotting. If you have it
> installed, you'd just do:from enthought.mayavi import mlabs = mlab.mesh(x,
> y,
> z)mlab.show()Hope that helps some,
> > -Joe
> > On Tue, Feb 16, 2010 at 6:00 AM, URI <zallen <at>
> urologicresearchinstitute.org> wrote:
> > >Joe Kington <jkington <at> wisc.edu> writes:
> >
> > >
>
> I tried using mlab.mesh from mayavi but apparently the x y z components
> need to
> be 2D arrays (see
>
> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/mlab_helper_functions.html#enthought.mayavi.mlab.mesh
> ).
>
> I don't understand this, how can my list of the x-positions of the vertices
> be a
> 2D array?
>
> I did try to use mlab.plot3d (see
>
> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/mlab_helper_functions.html#enthought.mayavi.mlab.plot3d
> )
> and it gave me a nice line drawing of my shape, but an actual surface would
> be
> much nicer.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100218/91e1ae2f/attachment.html>

From e.antero.tammi at gmail.com  Thu Feb 18 16:58:00 2010
From: e.antero.tammi at gmail.com (eat)
Date: Thu, 18 Feb 2010 21:58:00 +0000 (UTC)
Subject: [SciPy-User] convertin from octave/ matlab to scipy
Message-ID: <loom.20100218T225412-217@post.gmane.org>

Hi,

I'll just have started to study and learn the python/ scipy.
I have written some inhouse (exploratory) data analysis
octave/ matlab toolboxes/ packages that I now like to convert
to python/ scipy.

I do have googled and skimmed bunch of documents, but still
I'll like to have experts opinion how to convert this (typical)
octave/ matlab indicies idiom to scipy:


"""
a simple minded conversion of idiom 1 from octave(or matlab)
<ocode>
function t1
	m= 12; n= 123456; D= randn(m, n);
	f= @(X) sqrt(sum(X.^2)); g= @(D) zm(f, D);
	norm(f(D)- g(D))

function D= zm(f, D)
	m= mean(D, 2); D= f(D- m(:, ones(1, size(D, 2))));
</ocode>
to scipy:
"""

from scipy import *
from numpy.linalg import norm #???

def zm(f, D):
    a= vstack(range(D.shape[0]))* ([1]* D.shape[1]);
    m= mean(D, 1); return f(D- m[a])

m= 12; n= 123456; D= random.randn(m, n);
f= lambda X: sqrt(sum(X**2, 0)); g= lambda D: zm(f, D)
print norm(f(D)- g(D))


I think my main question is:
do we have in scipy some 'under the hood' functionality, which
allows us not to repmat the constant vector (m)?

(Code like above is handled quite well in matlab where
(e(matlab)/ e(octave)<~ .1), compared to (e(scipy)/ e(octave)<~ .5),
where e(x) is some measure of the execution time in x.)




From Dwf at cs.toronto.edu  Thu Feb 18 17:24:58 2010
From: Dwf at cs.toronto.edu (David Warde-Farley)
Date: Thu, 18 Feb 2010 17:24:58 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
Message-ID: <79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>


On 16-Feb-10, at 2:14 PM, Skipper Seabold wrote:

> * Added four discrete choice models: Poisson, Probit, Logit, and
> Multinomial Logit.

Awesome. I look forward to trying these out (by the way, do you  
support any regularization methods? L2/L1?)

David


From gael.varoquaux at normalesup.org  Thu Feb 18 17:30:09 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 18 Feb 2010 23:30:09 +0100
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
Message-ID: <20100218223009.GE32024@phare.normalesup.org>

On Thu, Feb 18, 2010 at 05:24:58PM -0500, David Warde-Farley wrote:

> On 16-Feb-10, at 2:14 PM, Skipper Seabold wrote:

> > * Added four discrete choice models: Poisson, Probit, Logit, and
> > Multinomial Logit.

> Awesome. I look forward to trying these out (by the way, do you  
> support any regularization methods? L2/L1?)

We'll be adding these to scikit learn pretty soon :). We might use
compiled code, for efficiency, or maybe not, we haven't decided yet.

Ga?l


From josef.pktd at gmail.com  Thu Feb 18 17:34:11 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 17:34:11 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <20100218223009.GE32024@phare.normalesup.org>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
Message-ID: <1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>

On Thu, Feb 18, 2010 at 5:30 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Thu, Feb 18, 2010 at 05:24:58PM -0500, David Warde-Farley wrote:
>
>> On 16-Feb-10, at 2:14 PM, Skipper Seabold wrote:
>
>> > * Added four discrete choice models: Poisson, Probit, Logit, and
>> > Multinomial Logit.
>
>> Awesome. I look forward to trying these out (by the way, do you
>> support any regularization methods? L2/L1?)
>
> We'll be adding these to scikit learn pretty soon :). We might use
> compiled code, for efficiency, or maybe not, we haven't decided yet.
>
> Ga?l
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Thu Feb 18 18:23:55 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 18:23:55 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
Message-ID: <1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>

hit the wrong button

On Thu, Feb 18, 2010 at 5:34 PM,  <josef.pktd at gmail.com> wrote:
> On Thu, Feb 18, 2010 at 5:30 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>> On Thu, Feb 18, 2010 at 05:24:58PM -0500, David Warde-Farley wrote:
>>
>>> On 16-Feb-10, at 2:14 PM, Skipper Seabold wrote:
>>
>>> > * Added four discrete choice models: Poisson, Probit, Logit, and
>>> > Multinomial Logit.

They are still new, so some problems might still be lurking around.

>>
>>> Awesome. I look forward to trying these out (by the way, do you
>>> support any regularization methods? L2/L1?)

L2 Tychonov style penalization is planned for specific models, eg.
generalized/Bayesian Ridge Regression, Vector Autoregressive
Regression with dummy variable priors and other shrinkage estimators
when there are too many parameters to estimate.

There are no plans for Lasso, although I would like LARS as a
complement to Principal Component Regression, but it's not high on the
priority list.

Josef

>>
>> We'll be adding these to scikit learn pretty soon :). We might use
>> compiled code, for efficiency, or maybe not, we haven't decided yet.
>>
>> Ga?l
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From robert.kern at gmail.com  Thu Feb 18 18:28:41 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 18 Feb 2010 17:28:41 -0600
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com> 
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org> 
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com> 
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
Message-ID: <3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>

On Thu, Feb 18, 2010 at 17:23,  <josef.pktd at gmail.com> wrote:
> hit the wrong button
>
> On Thu, Feb 18, 2010 at 5:34 PM, ?<josef.pktd at gmail.com> wrote:
>> On Thu, Feb 18, 2010 at 5:30 PM, Gael Varoquaux
>> <gael.varoquaux at normalesup.org> wrote:
>>> On Thu, Feb 18, 2010 at 05:24:58PM -0500, David Warde-Farley wrote:
>>>
>>>> On 16-Feb-10, at 2:14 PM, Skipper Seabold wrote:
>>>
>>>> > * Added four discrete choice models: Poisson, Probit, Logit, and
>>>> > Multinomial Logit.
>
> They are still new, so some problems might still be lurking around.
>
>>>
>>>> Awesome. I look forward to trying these out (by the way, do you
>>>> support any regularization methods? L2/L1?)
>
> L2 Tychonov style penalization is planned for specific models, eg.
> generalized/Bayesian Ridge Regression, Vector Autoregressive
> Regression with dummy variable priors and other shrinkage estimators
> when there are too many parameters to estimate.
>
> There are no plans for Lasso, although I would like LARS as a
> complement to Principal Component Regression, but it's not high on the
> priority list.

There's been a recent paper that might be of interest. The associated
code is GPLed, alas.

  http://www.jstatsoft.org/v33/i01

"""
Abstract:	
We develop fast algorithms for estimation of generalized linear models
with convex penalties. The models include linear regression, two-class
logistic regression, and multi- nomial regression problems while the
penalties include L1 (the lasso), L2 (ridge regression) and mixtures
of the two (the elastic net). The algorithms use cyclical coordinate
descent, computed along a regularization path. The methods can handle
large problems and can also deal efficiently with sparse features. In
comparative timings we find that the new algorithms are considerably
faster than competing methods.
"""

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dwf at cs.toronto.edu  Thu Feb 18 18:43:50 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Thu, 18 Feb 2010 18:43:50 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
Message-ID: <8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>

On 18-Feb-10, at 6:28 PM, Robert Kern wrote:

> There's been a recent paper that might be of interest. The associated
> code is GPLed, alas.
>
>  http://www.jstatsoft.org/v33/i01
>
> """
> Abstract:	
> We develop fast algorithms for estimation of generalized linear models
> with convex penalties. The models include linear regression, two-class
> logistic regression, and multi- nomial regression problems while the
> penalties include L1 (the lasso), L2 (ridge regression) and mixtures
> of the two (the elastic net). The algorithms use cyclical coordinate
> descent, computed along a regularization path. The methods can handle
> large problems and can also deal efficiently with sparse features. In
> comparative timings we find that the new algorithms are considerably
> faster than competing methods.
> """

Yep. There's quite a bit of code out there for doing L1 chicanery if  
you're willing to acquiesce to the GPL, also notably this code: http://www.stanford.edu/~boyd/l1_logreg/

David


From josef.pktd at gmail.com  Thu Feb 18 18:59:14 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 18:59:14 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
Message-ID: <1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>

On Thu, Feb 18, 2010 at 6:43 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> On 18-Feb-10, at 6:28 PM, Robert Kern wrote:
>
>> There's been a recent paper that might be of interest. The associated
>> code is GPLed, alas.
>>
>> ?http://www.jstatsoft.org/v33/i01
>>
>> """
>> Abstract:
>> We develop fast algorithms for estimation of generalized linear models
>> with convex penalties. The models include linear regression, two-class
>> logistic regression, and multi- nomial regression problems while the
>> penalties include L1 (the lasso), L2 (ridge regression) and mixtures
>> of the two (the elastic net). The algorithms use cyclical coordinate
>> descent, computed along a regularization path. The methods can handle
>> large problems and can also deal efficiently with sparse features. In
>> comparative timings we find that the new algorithms are considerably
>> faster than competing methods.
>> """
>
> Yep. There's quite a bit of code out there for doing L1 chicanery if
> you're willing to acquiesce to the GPL, also notably this code: http://www.stanford.edu/~boyd/l1_logreg/

but I haven't seen anything yet that would be BSD compatible.

pymvpa is wrapping elastic net and lars with rpy/rpy2 from R. If I
remember then Trevor Hastie, Rob Tibshirani write their code for R.

I think lars is not too difficult, that's one of the reason I have it
in mind, and I have seen some applications in econometrics where the
forecasting performance of lars in combination with PCA was the best
but not lars alone.

Josef




>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From gael.varoquaux at normalesup.org  Thu Feb 18 19:04:43 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 19 Feb 2010 01:04:43 +0100
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
Message-ID: <20100219000443.GI32024@phare.normalesup.org>

On Thu, Feb 18, 2010 at 06:59:14PM -0500, josef.pktd at gmail.com wrote:
> but I haven't seen anything yet that would be BSD compatible.

We actually have some code in the lab. It is based on proximal iterations,
or dual optimisation. We are going to release some of it as BSD pretty
soon. I believe we will also implement a path algorithm, but I am not
sure whether it will be a lars or the new Friedman toy.

> I think lars is not too difficult, that's one of the reason I have it
> in mind, and I have seen some applications in econometrics where the
> forecasting performance of lars in combination with PCA was the best
> but not lars alone.

I have implemented a lars (not releasing it, as it is buggy). There are a
few difficulties that are not outlined in the article :). I am willing to
work with you on the lars (mostly by dumping the code to you, with the
comments pointing to the problems). But I would advice you to wait a
month, as we have a working sessions planned exactly for pushing code
that solves these problems in scikit.learn.

Ga?l


From josef.pktd at gmail.com  Thu Feb 18 19:27:35 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 19:27:35 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <20100219000443.GI32024@phare.normalesup.org>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
Message-ID: <1cd32cbb1002181627v29922ad6sef5a9d6366b207ff@mail.gmail.com>

On Thu, Feb 18, 2010 at 7:04 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Thu, Feb 18, 2010 at 06:59:14PM -0500, josef.pktd at gmail.com wrote:
>> but I haven't seen anything yet that would be BSD compatible.
>
> We actually have some code in the lab. It is based on proximal iterations,
> or dual optimisation. We are going to release some of it as BSD pretty
> soon. I believe we will also implement a path algorithm, but I am not
> sure whether it will be a lars or the new Friedman toy.
>
>> I think lars is not too difficult, that's one of the reason I have it
>> in mind, and I have seen some applications in econometrics where the
>> forecasting performance of lars in combination with PCA was the best
>> but not lars alone.
>
> I have implemented a lars (not releasing it, as it is buggy). There are a
> few difficulties that are not outlined in the article :). I am willing to
> work with you on the lars (mostly by dumping the code to you, with the
> comments pointing to the problems). But I would advice you to wait a
> month, as we have a working sessions planned exactly for pushing code
> that solves these problems in scikit.learn.

I'm not in a hurry, I have some other pet projects (in time series
analysis) that I want to get into a usable stage first. But then I
will be glad to take you up on the offer.

I haven't looked much at the details in pymvpa, but it might be useful
to test the code.

Josef

>
> Ga?l
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From jsseabold at gmail.com  Thu Feb 18 21:28:24 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 18 Feb 2010 21:28:24 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com> 
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org> 
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com> 
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com> 
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
Message-ID: <c048da1c1002181828l30ab0996w28a8c6393e043a27@mail.gmail.com>

On Thu, Feb 18, 2010 at 6:28 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Feb 18, 2010 at 17:23, ?<josef.pktd at gmail.com> wrote:
>> hit the wrong button
>>
>> On Thu, Feb 18, 2010 at 5:34 PM, ?<josef.pktd at gmail.com> wrote:
>>> On Thu, Feb 18, 2010 at 5:30 PM, Gael Varoquaux
>>> <gael.varoquaux at normalesup.org> wrote:
>>>> On Thu, Feb 18, 2010 at 05:24:58PM -0500, David Warde-Farley wrote:
>>>>
>>>>> On 16-Feb-10, at 2:14 PM, Skipper Seabold wrote:
>>>>
>>>>> > * Added four discrete choice models: Poisson, Probit, Logit, and
>>>>> > Multinomial Logit.
>>
>> They are still new, so some problems might still be lurking around.
>>
>>>>
>>>>> Awesome. I look forward to trying these out (by the way, do you
>>>>> support any regularization methods? L2/L1?)
>>
>> L2 Tychonov style penalization is planned for specific models, eg.
>> generalized/Bayesian Ridge Regression, Vector Autoregressive
>> Regression with dummy variable priors and other shrinkage estimators
>> when there are too many parameters to estimate.
>>
>> There are no plans for Lasso, although I would like LARS as a
>> complement to Principal Component Regression, but it's not high on the
>> priority list.

Funny this should come up.  I wasn't even really aware of this
implementation- and application-wise until I saw the paper Robert
mentions below on Andy Gelman's blog and started reading up.  I was
just reading this afternoon about method of regularization as part of
the generalized entropy framework, which is somewhat supported by the
scipy.maxentropy module right now (for smoothing with a Gaussian
prior).  For my end, I will be more likely to have examples of
discrete choice models (a general class of the logit at least) with
regularization in the maxent framework before anything else given my
work this semester.

As always code donations to statsmodels are welcomed and encouraged...

Skipper

>
> There's been a recent paper that might be of interest. The associated
> code is GPLed, alas.
>
> ?http://www.jstatsoft.org/v33/i01
>
> """
> Abstract:
> We develop fast algorithms for estimation of generalized linear models
> with convex penalties. The models include linear regression, two-class
> logistic regression, and multi- nomial regression problems while the
> penalties include L1 (the lasso), L2 (ridge regression) and mixtures
> of the two (the elastic net). The algorithms use cyclical coordinate
> descent, computed along a regularization path. The methods can handle
> large problems and can also deal efficiently with sparse features. In
> comparative timings we find that the new algorithms are considerably
> faster than competing methods.
> """
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Thu Feb 18 22:56:09 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 22:56:09 -0500
Subject: [SciPy-User] convertin from octave/ matlab to scipy
In-Reply-To: <loom.20100218T225412-217@post.gmane.org>
References: <loom.20100218T225412-217@post.gmane.org>
Message-ID: <1cd32cbb1002181956n107c1f13u26755017ea98ab33@mail.gmail.com>

On Thu, Feb 18, 2010 at 4:58 PM, eat <e.antero.tammi at gmail.com> wrote:
> Hi,
>
> I'll just have started to study and learn the python/ scipy.
> I have written some inhouse (exploratory) data analysis
> octave/ matlab toolboxes/ packages that I now like to convert
> to python/ scipy.
>
> I do have googled and skimmed bunch of documents, but still
> I'll like to have experts opinion how to convert this (typical)
> octave/ matlab indicies idiom to scipy:
>
>
> """
> a simple minded conversion of idiom 1 from octave(or matlab)
> <ocode>
> function t1
> ? ? ? ?m= 12; n= 123456; D= randn(m, n);
> ? ? ? ?f= @(X) sqrt(sum(X.^2)); g= @(D) zm(f, D);
> ? ? ? ?norm(f(D)- g(D))
>
> function D= zm(f, D)
> ? ? ? ?m= mean(D, 2); D= f(D- m(:, ones(1, size(D, 2))));
> </ocode>
> to scipy:
> """
>
> from scipy import *
> from numpy.linalg import norm #???
>
> def zm(f, D):
> ? ?a= vstack(range(D.shape[0]))* ([1]* D.shape[1]);
> ? ?m= mean(D, 1); return f(D- m[a])
>
> m= 12; n= 123456; D= random.randn(m, n);
> f= lambda X: sqrt(sum(X**2, 0)); g= lambda D: zm(f, D)
> print norm(f(D)- g(D))
>
>
> I think my main question is:
> do we have in scipy some 'under the hood' functionality, which
> allows us not to repmat the constant vector (m)?

broadcasting is one of the best features of numpy compared to matlab.
Usually there are no repmats necessary, but often we have to add an
axis to match up the shape of the arrays

>>> import numpy as np
>>> a = np.arange(6).reshape(2,-1)
>>> a
array([[0, 1, 2],
       [3, 4, 5]])
>>> a.mean()
2.5
>>> a.mean(0)
array([ 1.5,  2.5,  3.5])
>>> a.mean(1)
array([ 1.,  4.])

>>> a - a.mean(0)
array([[-1.5, -1.5, -1.5],
       [ 1.5,  1.5,  1.5]])
>>> a - a.mean(1)[:,np.newaxis]     #add axis back in that we lost in reduce operation mean, equivalent with [:,None]
array([[-1.,  0.,  1.],
       [-1.,  0.,  1.]])

here are some explanations:
http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html
http://www.scipy.org/EricsBroadcastingDoc

In your code the line with a= vstack(range(D.shape[0]))* ([1]* D.shape[1]);
is not necessary, but adding the newaxis if the mean is taken with
respect to an axis > 0
This should also make it quite a bit faster.

I don't understand why you use anonymous function in matlab and lambda
in python, but maybe that's because this is out of context. It's not
easy to see what the actual flow of execution is, or maybe it's
because I'm not used to having several statements on a line separated
by semicolon.

Josef

>
> (Code like above is handled quite well in matlab where
> (e(matlab)/ e(octave)<~ .1), compared to (e(scipy)/ e(octave)<~ .5),
> where e(x) is some measure of the execution time in x.)
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Thu Feb 18 23:29:06 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 23:29:06 -0500
Subject: [SciPy-User] convertin from octave/ matlab to scipy
In-Reply-To: <1cd32cbb1002181956n107c1f13u26755017ea98ab33@mail.gmail.com>
References: <loom.20100218T225412-217@post.gmane.org>
	<1cd32cbb1002181956n107c1f13u26755017ea98ab33@mail.gmail.com>
Message-ID: <1cd32cbb1002182029m25225883s81f67cd4526ec3ed@mail.gmail.com>

On Thu, Feb 18, 2010 at 10:56 PM,  <josef.pktd at gmail.com> wrote:
> On Thu, Feb 18, 2010 at 4:58 PM, eat <e.antero.tammi at gmail.com> wrote:
>> Hi,
>>
>> I'll just have started to study and learn the python/ scipy.
>> I have written some inhouse (exploratory) data analysis
>> octave/ matlab toolboxes/ packages that I now like to convert
>> to python/ scipy.
>>
>> I do have googled and skimmed bunch of documents, but still
>> I'll like to have experts opinion how to convert this (typical)
>> octave/ matlab indicies idiom to scipy:
>>
>>
>> """
>> a simple minded conversion of idiom 1 from octave(or matlab)
>> <ocode>
>> function t1
>> ? ? ? ?m= 12; n= 123456; D= randn(m, n);
>> ? ? ? ?f= @(X) sqrt(sum(X.^2)); g= @(D) zm(f, D);
>> ? ? ? ?norm(f(D)- g(D))
>>
>> function D= zm(f, D)
>> ? ? ? ?m= mean(D, 2); D= f(D- m(:, ones(1, size(D, 2))));
>> </ocode>
>> to scipy:
>> """
>>
>> from scipy import *
>> from numpy.linalg import norm #???
>>
>> def zm(f, D):
>> ? ?a= vstack(range(D.shape[0]))* ([1]* D.shape[1]);
>> ? ?m= mean(D, 1); return f(D- m[a])
>>
>> m= 12; n= 123456; D= random.randn(m, n);
>> f= lambda X: sqrt(sum(X**2, 0)); g= lambda D: zm(f, D)
>> print norm(f(D)- g(D))
>>
>>
>> I think my main question is:
>> do we have in scipy some 'under the hood' functionality, which
>> allows us not to repmat the constant vector (m)?
>
> broadcasting is one of the best features of numpy compared to matlab.
> Usually there are no repmats necessary, but often we have to add an
> axis to match up the shape of the arrays
>
>>>> import numpy as np
>>>> a = np.arange(6).reshape(2,-1)
>>>> a
> array([[0, 1, 2],
> ? ? ? [3, 4, 5]])
>>>> a.mean()
> 2.5
>>>> a.mean(0)
> array([ 1.5, ?2.5, ?3.5])
>>>> a.mean(1)
> array([ 1., ?4.])
>
>>>> a - a.mean(0)
> array([[-1.5, -1.5, -1.5],
> ? ? ? [ 1.5, ?1.5, ?1.5]])
>>>> a - a.mean(1)[:,np.newaxis] ? ? #add axis back in that we lost in reduce operation mean, equivalent with [:,None]
> array([[-1., ?0., ?1.],
> ? ? ? [-1., ?0., ?1.]])
>
> here are some explanations:
> http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html
> http://www.scipy.org/EricsBroadcastingDoc
>
> In your code the line with a= vstack(range(D.shape[0]))* ([1]* D.shape[1]);
> is not necessary, but adding the newaxis if the mean is taken with
> respect to an axis > 0
> This should also make it quite a bit faster.
>
> I don't understand why you use anonymous function in matlab and lambda
> in python, but maybe that's because this is out of context. It's not
> easy to see what the actual flow of execution is, or maybe it's
> because I'm not used to having several statements on a line separated
> by semicolon.

I take this back, it's a bit difficult to read but I have a similar
pattern of code in matlab for nested functions.

maybe something like this (typed in editor, so it might not work)

def t1():
    m, n = 12, 123456
    D = random.randn(m, n)

    def f(X):
        return sqrt(sum(X**2, 0))

    print norm(f(D) - zm(f, D))
    #or ?
    #g = lambda D: zm(f, D)
    #print norm(f(D)- g(D))

def zm(f, D):
   m = mean(D, 1)[:, None]
   return f(D- m)



>
> Josef
>
>>
>> (Code like above is handled quite well in matlab where
>> (e(matlab)/ e(octave)<~ .1), compared to (e(scipy)/ e(octave)<~ .5),
>> where e(x) is some measure of the execution time in x.)
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From e.antero.tammi at gmail.com  Fri Feb 19 08:02:13 2010
From: e.antero.tammi at gmail.com (eat)
Date: Fri, 19 Feb 2010 13:02:13 +0000 (UTC)
Subject: [SciPy-User] convertin from octave/ matlab to scipy
References: <loom.20100218T225412-217@post.gmane.org>
	<1cd32cbb1002181956n107c1f13u26755017ea98ab33@mail.gmail.com>
	<1cd32cbb1002182029m25225883s81f67cd4526ec3ed@mail.gmail.com>
Message-ID: <loom.20100219T134742-418@post.gmane.org>

> def t1():
>     m, n = 12, 123456
>     D = random.randn(m, n)
> 
>     def f(X):
>         return sqrt(sum(X**2, 0))
> 
>     print norm(f(D) - zm(f, D))
>     #or ?
>     #g = lambda D: zm(f, D)
>     #print norm(f(D)- g(D))
> 
> def zm(f, D):
>    m = mean(D, 1)[:, None]
>    return f(D- m)
Exactly what I was looking for, just couldn't figure it out myself.

Originally I planned to make the conversion mechanically, but your code gives 
me more toughts. I'll better learn python/ scipy a bit more first inorder to 
produce more readable code.

Thanks a lot,
eat




From aisaac at american.edu  Fri Feb 19 09:34:23 2010
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 19 Feb 2010 09:34:23 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>	<20100218223009.GE32024@phare.normalesup.org>	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
Message-ID: <4B7EA16F.1080905@american.edu>

On 2/18/2010 6:43 PM, David Warde-Farley wrote:
> There's quite a bit of code out there for doing L1 chicanery if
> you're willing to acquiesce to the GPL, also notably this code:http://www.stanford.edu/~boyd/l1_logreg/

Note that the authors include contact info at the bottom
of that page, if you wish to explore whether they might
consider releasing the code under a BSD license.

Alan Isaac



From josef.pktd at gmail.com  Fri Feb 19 09:35:52 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 09:35:52 -0500
Subject: [SciPy-User] convertin from octave/ matlab to scipy
In-Reply-To: <loom.20100219T134742-418@post.gmane.org>
References: <loom.20100218T225412-217@post.gmane.org>
	<1cd32cbb1002181956n107c1f13u26755017ea98ab33@mail.gmail.com>
	<1cd32cbb1002182029m25225883s81f67cd4526ec3ed@mail.gmail.com>
	<loom.20100219T134742-418@post.gmane.org>
Message-ID: <1cd32cbb1002190635l7bf149e3xa2bbd54ea62fb100@mail.gmail.com>

On Fri, Feb 19, 2010 at 8:02 AM, eat <e.antero.tammi at gmail.com> wrote:
>> def t1():
>> ? ? m, n = 12, 123456
>> ? ? D = random.randn(m, n)
>>
>> ? ? def f(X):
>> ? ? ? ? return sqrt(sum(X**2, 0))
>>
>> ? ? print norm(f(D) - zm(f, D))
>> ? ? #or ?
>> ? ? #g = lambda D: zm(f, D)
>> ? ? #print norm(f(D)- g(D))
>>
>> def zm(f, D):
>> ? ?m = mean(D, 1)[:, None]
>> ? ?return f(D- m)
> Exactly what I was looking for, just couldn't figure it out myself.
>
> Originally I planned to make the conversion mechanically, but your code gives
> me more toughts. I'll better learn python/ scipy a bit more first inorder to
> produce more readable code.

I often translate matlab functions with a simple structure (even if
the calculation is complicated) quite literally to python/numpy/scipy.
However, for the big structure and control flow using, for example,
classes is a huge advantage compared to matlab. (Although matlab is
also improving its support of classes, which I haven't used yet.)

A good book for python (for scienc) that uses numpy is very helpful.
And I found and still find it very informative to read code written by
others.

Good luck,

Josef

>
> Thanks a lot,
> eat
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Fri Feb 19 10:11:11 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 10:11:11 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <4B7EA16F.1080905@american.edu>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<4B7EA16F.1080905@american.edu>
Message-ID: <1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>

On Fri, Feb 19, 2010 at 9:34 AM, Alan G Isaac <aisaac at american.edu> wrote:
> On 2/18/2010 6:43 PM, David Warde-Farley wrote:
>> There's quite a bit of code out there for doing L1 chicanery if
>> you're willing to acquiesce to the GPL, also notably this code:http://www.stanford.edu/~boyd/l1_logreg/
>
> Note that the authors include contact info at the bottom
> of that page, if you wish to explore whether they might
> consider releasing the code under a BSD license.

Since the reference is using the interior point algorithm: there is a
generic linear programming function with interior point algorithm
available in gauss by Thierry Roncalli licensed under MIT which might
be useful as reference implementation. He uses it for LAD estimation
and in portfolio allocation (if I remember correctly). I'm not sure
whether there is an explanation in English.

Josef

>
> Alan Isaac
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From jsseabold at gmail.com  Fri Feb 19 10:22:13 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Fri, 19 Feb 2010 10:22:13 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com> 
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org> 
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com> 
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com> 
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com> 
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<4B7EA16F.1080905@american.edu> 
	<1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>
Message-ID: <c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>

On Fri, Feb 19, 2010 at 10:11 AM,  <josef.pktd at gmail.com> wrote:
> On Fri, Feb 19, 2010 at 9:34 AM, Alan G Isaac <aisaac at american.edu> wrote:
>> On 2/18/2010 6:43 PM, David Warde-Farley wrote:
>>> There's quite a bit of code out there for doing L1 chicanery if
>>> you're willing to acquiesce to the GPL, also notably this code:http://www.stanford.edu/~boyd/l1_logreg/
>>
>> Note that the authors include contact info at the bottom
>> of that page, if you wish to explore whether they might
>> consider releasing the code under a BSD license.
>
> Since the reference is using the interior point algorithm: there is a
> generic linear programming function with interior point algorithm
> available in gauss by Thierry Roncalli licensed under MIT which might
> be useful as reference implementation. He uses it for LAD estimation
> and in portfolio allocation (if I remember correctly). I'm not sure
> whether there is an explanation in English.
>

Another nice resource: http://www.thierry-roncalli.com/

Manual is in English, though the lecture notes are in French.  Lecture
notes point to the relationship between LAD and Huber's method, which
might be nice to compare to our code for the same.

Skipper


From bsouthey at gmail.com  Fri Feb 19 10:42:14 2010
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 19 Feb 2010 09:42:14 -0600
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <20100219000443.GI32024@phare.normalesup.org>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>	<20100218223009.GE32024@phare.normalesup.org>	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
Message-ID: <4B7EB156.8010409@gmail.com>

On 02/18/2010 06:04 PM, Gael Varoquaux wrote:
> On Thu, Feb 18, 2010 at 06:59:14PM -0500, josef.pktd at gmail.com wrote:
>    
>> but I haven't seen anything yet that would be BSD compatible.
>>      
> We actually have some code in the lab. It is based on proximal iterations,
> or dual optimisation. We are going to release some of it as BSD pretty
> soon. I believe we will also implement a path algorithm, but I am not
> sure whether it will be a lars or the new Friedman toy.
>
>    
>> I think lars is not too difficult, that's one of the reason I have it
>> in mind, and I have seen some applications in econometrics where the
>> forecasting performance of lars in combination with PCA was the best
>> but not lars alone.
>>      
> I have implemented a lars (not releasing it, as it is buggy). There are a
> few difficulties that are not outlined in the article :). I am willing to
> work with you on the lars (mostly by dumping the code to you, with the
> comments pointing to the problems). But I would advice you to wait a
> month, as we have a working sessions planned exactly for pushing code
> that solves these problems in scikit.learn.
>
> Ga?l
> _______________________________________________
>    
That would be great!

I really do think that the scikits learn and statsmodels must talk 
together now that learn has had a release as well ( I don't recall 
seeing it mentioned hint hint!). What would be nice is the acceptance of 
input data types between learn and statsmodels especially for things 
like logistic regression. While I understand the need for duplicate 
functions, it may be desirable share at least code since both code bases 
are still relatively 'new'.

Bruce


From josef.pktd at gmail.com  Fri Feb 19 10:49:36 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 10:49:36 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<4B7EA16F.1080905@american.edu>
	<1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>
	<c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>
Message-ID: <1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>

On Fri, Feb 19, 2010 at 10:22 AM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Fri, Feb 19, 2010 at 10:11 AM, ?<josef.pktd at gmail.com> wrote:
>> On Fri, Feb 19, 2010 at 9:34 AM, Alan G Isaac <aisaac at american.edu> wrote:
>>> On 2/18/2010 6:43 PM, David Warde-Farley wrote:
>>>> There's quite a bit of code out there for doing L1 chicanery if
>>>> you're willing to acquiesce to the GPL, also notably this code:http://www.stanford.edu/~boyd/l1_logreg/
>>>
>>> Note that the authors include contact info at the bottom
>>> of that page, if you wish to explore whether they might
>>> consider releasing the code under a BSD license.
>>
>> Since the reference is using the interior point algorithm: there is a
>> generic linear programming function with interior point algorithm
>> available in gauss by Thierry Roncalli licensed under MIT which might
>> be useful as reference implementation. He uses it for LAD estimation
>> and in portfolio allocation (if I remember correctly). I'm not sure
>> whether there is an explanation in English.
>>
>
> Another nice resource: http://www.thierry-roncalli.com/
>
> Manual is in English, though the lecture notes are in French. ?Lecture
> notes point to the relationship between LAD and Huber's method, which
> might be nice to compare to our code for the same.

He added last year the MIT license to all his code, except
unfortunately his time series code. I will eventually target his GMM
code, but some of the Gauss support functions are not yet available in
python. From a very brief look that I had at his Huber a while ago, I
thought we might have already more in statsmodels than he does,
although maybe estimated in a different way. His packages are based on
15 years of programming, there is too much to match it anytime soon.

Josef

>
> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Fri Feb 19 11:01:58 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 11:01:58 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <4B7EB156.8010409@gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com>
Message-ID: <1cd32cbb1002190801m490b2e8ay21c1ee3d7f941dd1@mail.gmail.com>

On Fri, Feb 19, 2010 at 10:42 AM, Bruce Southey <bsouthey at gmail.com> wrote:
> On 02/18/2010 06:04 PM, Gael Varoquaux wrote:
>> On Thu, Feb 18, 2010 at 06:59:14PM -0500, josef.pktd at gmail.com wrote:
>>
>>> but I haven't seen anything yet that would be BSD compatible.
>>>
>> We actually have some code in the lab. It is based on proximal iterations,
>> or dual optimisation. We are going to release some of it as BSD pretty
>> soon. I believe we will also implement a path algorithm, but I am not
>> sure whether it will be a lars or the new Friedman toy.
>>
>>
>>> I think lars is not too difficult, that's one of the reason I have it
>>> in mind, and I have seen some applications in econometrics where the
>>> forecasting performance of lars in combination with PCA was the best
>>> but not lars alone.
>>>
>> I have implemented a lars (not releasing it, as it is buggy). There are a
>> few difficulties that are not outlined in the article :). I am willing to
>> work with you on the lars (mostly by dumping the code to you, with the
>> comments pointing to the problems). But I would advice you to wait a
>> month, as we have a working sessions planned exactly for pushing code
>> that solves these problems in scikit.learn.
>>
>> Ga?l
>> _______________________________________________
>>
> That would be great!
>
> I really do think that the scikits learn and statsmodels must talk
> together now that learn has had a release as well ( I don't recall
> seeing it mentioned hint hint!). What would be nice is the acceptance of
> input data types between learn and statsmodels especially for things
> like logistic regression. While I understand the need for duplicate
> functions, it may be desirable share at least code since both code bases
> are still relatively 'new'.

We are trying to interface or relate in some way or other to closely
related packages, http://statsmodels.sourceforge.net/related.html and
sharing code is very useful among compatible licensed packages.

However, the focus is different, and statsmodels currently targets
pure python and we don't want to depend on packages that require that
users are able to install and link to boost libraries. We are a bit
careful with dependencies and other requirements if it should remain
possible to integrate statsmodels (or parts of it) into scipy.

Josef


>
> Bruce
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From wing1127aishi at gmail.com  Fri Feb 19 11:13:49 2010
From: wing1127aishi at gmail.com (Leon Sit)
Date: Fri, 19 Feb 2010 10:13:49 -0600
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release (Skipper
	Seabold)
Message-ID: <3af1998d1002190813x656d54e8h79c7eca28b1ec9b@mail.gmail.com>

Dear all:

I am not a specialist in statistical method but I have rewritten some
code on econometric models at http://github.com/wingsit/KF. Since I
saw at somewhere that the statmodels authors are interested in
econometric models, I just want to put my code on the table and see if
it is useful to anybody.

The project name 'Kalman Filter' is a bit misleading because it
contains other regression techniques to solve the state space model.

Some summary of the library
-----------------------------------------------------------------------------------------
This project was started to test different available tools to track
mutual funds and hedge fund using Capital Asset
Pricing Model (CAPM thereafter) introduced my Sharpe and Arbitrage
Pricing Theory (APT thereafter) introduced by Ross.
The purpose of the CAPM model is to use a set of generic investable
market indices to decompose the exposure of a fund.
On the other hand, APT model is to find a set of generic indices which
are not necessarily investable, to decompose the
behaviour of an indices that we wish to analyse.

The License for this project is BSD.

Documentation:
http://packages.python.org/KF/

Dependency:

This project requires
Scipy (http://www.scipy.org/)
Numpy (http://numpy.scipy.org/)
CVXOPT (http://abel.ee.ucla.edu/cvxopt/)

Optional
Matplotlib (http://matplotlib.sourceforge.net/)

The primary data structure is DataFrame in timeSeriesFrame.py although
this is not used directly. The container for time
series data is TimeSeriesFrame which are used by Regression.
Essentially, this is just a data structure with a
scipy.matrix for tabular data and 2 lists which contains row and
column header information. Later in the future I might
change it it numpy.maskedarray to handle missing. The primary features
for TimeSeriesFrame are simple slicing and
iterators in both dimensions. This was designed in a way such that it
is easy to run operation on each time series or
each date. Later I might subclass it with
http://scikits.appspot.com/scikits but this is not in near future.

Regression is a base class for all the estimation procedures. It is
implemented as basic linear least squares
estimation. Primary sub-classes of Regression is ECRegression and
ICRegression which are base-class for equality and
inequality constrained model, respectively. Sample library usage is

>>>    stock_data = list(csv.reader(open("simulated_portfolio.csv", "rb"))) #Read data from file
>>>    stock = StylusReader(stock_data) #Put the data into TimeSeriesFrame
>>>    respond = stock[:,0] #slice the first column of TimeSeriesFrame as respond
>>>    regressors = stock[:,1:] #slice the columns after first column as regressors
>>>    t,n = regressors.size()
>>>    weight = scipy.identity(t)
>>>    intercept = True
>>>    obj = Regression(respond, regressors, intercept, weight = weight) #Initialise the regression object, construct
all the necessary matrix according to the value of intercept and settings
>>>    obj.train() #perform estimation. In this case it is simple matrix calculation and inversion.
>>>    print obj.predict()  #Return the estimate in TimeSerisFrame
>>>    print obj.error() #Return the estimation error in TimeSeriesFrame

Essentially, users only need to run the constructor and train() to get
the estimate. It was designed to be very simple
to use for end users.

Naming Convention:
regression.py imposes no state constraints on the model
ec*.py imposes Equality Constraints on the model. For example,
ecRegression.py is a regression model which must satisfy
Equality Constraints.
ic*.py imposes Inequality Constraints and Equality Constraints. For
example icRegression.py is a constrained regression
model which has both equality and inequality constraints.

Organisation: Most of the computation functions are located in libregression.py



Please feel free to drop a comment, forward this project and contribute!!

Contact: Wing H. Sit (wing1127aishi at gmail.com)
Something about me: A mathematically oriented person who wants to
learn coding, finance, and software development.
-----------------------------------------------------------------------------------------

Thanks

Leon


From aisaac at american.edu  Fri Feb 19 11:25:13 2010
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 19 Feb 2010 11:25:13 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>	<20100218223009.GE32024@phare.normalesup.org>	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>	<4B7EA16F.1080905@american.edu>	<1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>	<c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>
	<1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>
Message-ID: <4B7EBB69.8040802@american.edu>

On 2/19/2010 10:49 AM, josef.pktd at gmail.com wrote:
> He added last year the MIT license to all his code, except
> unfortunately his time series code.


In a discussion last June, I found Thierry quite open
to a discussion of licenses.  In fact, I believe this was
shortly before his switch to the MIT license.  So he may (?) be
open to releasing his time-series code under the MIT-license as well,
if asked, especially if he is given a little info about SciPy.

fwiw,
Alan



From denis-bz-gg at t-online.de  Fri Feb 19 11:26:41 2010
From: denis-bz-gg at t-online.de (denis)
Date: Fri, 19 Feb 2010 08:26:41 -0800 (PST)
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com>
Message-ID: <639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com>

On Feb 18, 4:48?pm, Robert Kern <robert.k... at gmail.com> wrote:
> On Thu, Feb 18, 2010 at 09:19, denis <denis-bz... at t-online.de> wrote:

> > Running rbf on 100 1d random.uniform input points
> > (after changing the linalg.solve in rbf.py to lstsq)
>
> Why would you do this? This does not magically turn Rbf interpolation
> into a smoothing approximation. Use the "smooth" keyword to specify a
> smoothing parameter.

Robert,
  i'm interpolating  y = np.sin(x) + np.random.normal( 0,  .1 ), not
random noise.
The idea was to look at gaussian vs thin-plate;
true, that 1d snippet doesn't say much,
but my 2d plots were so noisy that I went down to 1d.

Use "smooth" ? rbf.py just does
    self.A = self._function(r) - eye(self.N)*self.smooth
and you don't know A .

Bytheway (googling), http://www.farfieldtechnology.com/products/toolbox
...
have an O(N lnN) FastRBF TM in matlab, $

cheers
  -- denis


From gael.varoquaux at normalesup.org  Fri Feb 19 11:29:52 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 19 Feb 2010 17:29:52 +0100
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <4B7EB156.8010409@gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com>
Message-ID: <20100219162952.GB31426@phare.normalesup.org>

On Fri, Feb 19, 2010 at 09:42:14AM -0600, Bruce Southey wrote:
> I really do think that the scikits learn and statsmodels must talk 
> together now that learn has had a release as well ( I don't recall 
> seeing it mentioned hint hint!). 

That's a good point. In the long run, I think I would like statsmodels to
be a dependency of scikit learn, because I hate reimplementing stuff.

The difference that I see between scikit.learn and statsmodels is that we
have C code[*], and we will most probably end up with C++ code.

Lets say that the focus between scikit.learn and statsmodel is most
probably going to be slightly different.

> What would be nice is the acceptance of input data types between learn
> and statsmodels especially for things like logistic regression. While I
> understand the need for duplicate functions, it may be desirable share
> at least code since both code bases are still relatively 'new'.

Well, as far as I am concerned, data types are numpy arrays. I am weary
of implmenting higher level abstractions. Its more the APIs that may
different, and that we will have to keep in sync.

My 2 cents,

Ga?l

[*] For instance, we are starting to get really nice libsvm bindings.


From gael.varoquaux at normalesup.org  Fri Feb 19 11:30:33 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 19 Feb 2010 17:30:33 +0100
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <1cd32cbb1002190801m490b2e8ay21c1ee3d7f941dd1@mail.gmail.com>
References: <79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com>
	<1cd32cbb1002190801m490b2e8ay21c1ee3d7f941dd1@mail.gmail.com>
Message-ID: <20100219163033.GC31426@phare.normalesup.org>

On Fri, Feb 19, 2010 at 11:01:58AM -0500, josef.pktd at gmail.com wrote:
> However, the focus is different, and statsmodels currently targets
> pure python and we don't want to depend on packages that require that
> users are able to install and link to boost libraries. We are a bit
> careful with dependencies and other requirements if it should remain
> possible to integrate statsmodels (or parts of it) into scipy.

That's what I thought. So it does seems that we are on the same
wavelength. Cool!

Ga?l


From josef.pktd at gmail.com  Fri Feb 19 11:31:44 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 11:31:44 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <4B7EBB69.8040802@american.edu>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<4B7EA16F.1080905@american.edu>
	<1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>
	<c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>
	<1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>
	<4B7EBB69.8040802@american.edu>
Message-ID: <1cd32cbb1002190831l337dace4g1eea470b6be5ccf5@mail.gmail.com>

On Fri, Feb 19, 2010 at 11:25 AM, Alan G Isaac <aisaac at american.edu> wrote:
> On 2/19/2010 10:49 AM, josef.pktd at gmail.com wrote:
>> He added last year the MIT license to all his code, except
>> unfortunately his time series code.
>
>
> In a discussion last June, I found Thierry quite open
> to a discussion of licenses. ?In fact, I believe this was
> shortly before his switch to the MIT license. ?So he may (?) be
> open to releasing his time-series code under the MIT-license as well,
> if asked, especially if he is given a little info about SciPy.

As far as I could figure out, his time series code is still
commercially distributed, or maybe a successor of it. So, if his
original code is part of the commercial package, then he will not be
able to relicense it.

(Also, I haven't been successful in changing anyone's mind about licensing.)

Josef

>
> fwiw,
> Alan
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From robert.kern at gmail.com  Fri Feb 19 11:36:44 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 19 Feb 2010 10:36:44 -0600
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com> 
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com>
Message-ID: <3d375d731002190836v2639bffcrad8e304faa216ef5@mail.gmail.com>

On Fri, Feb 19, 2010 at 10:26, denis <denis-bz-gg at t-online.de> wrote:
> On Feb 18, 4:48?pm, Robert Kern <robert.k... at gmail.com> wrote:
>> On Thu, Feb 18, 2010 at 09:19, denis <denis-bz... at t-online.de> wrote:
>
>> > Running rbf on 100 1d random.uniform input points
>> > (after changing the linalg.solve in rbf.py to lstsq)
>>
>> Why would you do this? This does not magically turn Rbf interpolation
>> into a smoothing approximation. Use the "smooth" keyword to specify a
>> smoothing parameter.
>
> Robert,
> ?i'm interpolating ?y = np.sin(x) + np.random.normal( 0, ?.1 ), not
> random noise.

Okay. You said otherwise. It would help if you simply attached the
code that you are using.

> The idea was to look at gaussian vs thin-plate;
> true, that 1d snippet doesn't say much,
> but my 2d plots were so noisy that I went down to 1d.
>
> Use "smooth" ? rbf.py just does
> ? ?self.A = self._function(r) - eye(self.N)*self.smooth
> and you don't know A .

I have no idea what you mean by the last part of your sentence. Did
you actually try using the smooth keyword? What were your results?

> Bytheway (googling), http://www.farfieldtechnology.com/products/toolbox
> ...
> have an O(N lnN) FastRBF TM in matlab, $

Okay. And?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Fri Feb 19 11:41:06 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 11:41:06 -0500
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com>
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com>
Message-ID: <1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com>

On Fri, Feb 19, 2010 at 11:26 AM, denis <denis-bz-gg at t-online.de> wrote:
> On Feb 18, 4:48?pm, Robert Kern <robert.k... at gmail.com> wrote:
>> On Thu, Feb 18, 2010 at 09:19, denis <denis-bz... at t-online.de> wrote:
>
>> > Running rbf on 100 1d random.uniform input points
>> > (after changing the linalg.solve in rbf.py to lstsq)
>>
>> Why would you do this? This does not magically turn Rbf interpolation
>> into a smoothing approximation. Use the "smooth" keyword to specify a
>> smoothing parameter.
>
> Robert,
> ?i'm interpolating ?y = np.sin(x) + np.random.normal( 0, ?.1 ), not
> random noise.
> The idea was to look at gaussian vs thin-plate;
> true, that 1d snippet doesn't say much,
> but my 2d plots were so noisy that I went down to 1d.
>
> Use "smooth" ? rbf.py just does
> ? ?self.A = self._function(r) - eye(self.N)*self.smooth
> and you don't know A .

rbf is a global interpolator, and as Robert said for some cases you
get very unstable results.

with a large number of points self._function(r)  is not very well
behaved and you need the penalization to make it smoother. In a
variation of your example, I printed out the eigenvalues for self.A
and they don't look nice without penalization.
There are still some strange things that I don't understand with the
behavior rbf, but when I looked at it in the past, I got better
results by using only local information, i.e. fit rbf only to a
neighborhood of the points, although I think thinplate did pretty well
also with a larger number of points.

Josef

>
> Bytheway (googling), http://www.farfieldtechnology.com/products/toolbox
> ...
> have an O(N lnN) FastRBF TM in matlab, $
>
> cheers
> ?-- denis
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From bsouthey at gmail.com  Fri Feb 19 11:57:01 2010
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 19 Feb 2010 10:57:01 -0600
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <20100219162952.GB31426@phare.normalesup.org>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>	<20100218223009.GE32024@phare.normalesup.org>	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>	<20100219000443.GI32024@phare.normalesup.org>	<4B7EB156.8010409@gmail.com>
	<20100219162952.GB31426@phare.normalesup.org>
Message-ID: <4B7EC2DD.5000209@gmail.com>

On 02/19/2010 10:29 AM, Gael Varoquaux wrote:
> On Fri, Feb 19, 2010 at 09:42:14AM -0600, Bruce Southey wrote:
>    
>> I really do think that the scikits learn and statsmodels must talk
>> together now that learn has had a release as well ( I don't recall
>> seeing it mentioned hint hint!).
>>      
> That's a good point. In the long run, I think I would like statsmodels to
> be a dependency of scikit learn, because I hate reimplementing stuff.
>    
I agree with that reimplementing stuff but it is hard to a common ground.

> The difference that I see between scikit.learn and statsmodels is that we
> have C code[*], and we will most probably end up with C++ code.
>    
Will it end up as cython?
(I just used the supplied Python bindings of libsvm so this could be 
interesting.)
> Lets say that the focus between scikit.learn and statsmodel is most
> probably going to be slightly different.
>    
Having done both (with papers), I find this type of comment assuming 
because underlying both is the same concepts. What I would like to avoid 
is having different user syntax for basic models for the same model. For 
example, with logistic regression in SAS you have to be careful of which 
is the default event setting as it varies across procedures. At least 
these SAS procedures use the same unmodified dataset unlike some of the 
R packages that do lars/lasso.

>> What would be nice is the acceptance of input data types between learn
>> and statsmodels especially for things like logistic regression. While I
>> understand the need for duplicate functions, it may be desirable share
>> at least code since both code bases are still relatively 'new'.
>>      
> Well, as far as I am concerned, data types are numpy arrays. I am weary
> of implmenting higher level abstractions. Its more the APIs that may
> different, and that we will have to keep in sync.
>
> My 2 cents,
>
> Ga?l
>
> [*] For instance, we are starting to get really nice libsvm bindings.
>
>    
I do agree especially now that I have learnt the 'array' approach of 
doing things.

In some way my view of integration of things is Zelig -not that I have 
really looked at it (as it is in R) :
http://gking.harvard.edu/zelig/

The seamless ability to link packages is rather appealing and both 
scikits share at least numpy.


Bruce


From aisaac at american.edu  Fri Feb 19 12:01:10 2010
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 19 Feb 2010 12:01:10 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <1cd32cbb1002190831l337dace4g1eea470b6be5ccf5@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>	<4B7EA16F.1080905@american.edu>	<1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>	<c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>	<1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>	<4B7EBB69.8040802@american.edu>
	<1cd32cbb1002190831l337dace4g1eea470b6be5ccf5@mail.gmail.com>
Message-ID: <4B7EC3D6.2090107@american.edu>

On 2/19/2010 11:31 AM, josef.pktd at gmail.com wrote:
> As far as I could figure out, his time series code is still
> commercially distributed, or maybe a successor of it. So, if his
> original code is part of the commercial package, then he will not be
> able to relicense it.
>
> (Also, I haven't been successful in changing anyone's mind about licensing.)


I have, several times.  And if you tell me *specifically* which
parts of the code you would be ready to start translating to
SciPy, I would be happy to ask Thierry if he has any flexibility
on those pieces.

Alan



From gael.varoquaux at normalesup.org  Fri Feb 19 12:19:54 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 19 Feb 2010 18:19:54 +0100
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <4B7EC2DD.5000209@gmail.com>
References: <20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com>
	<20100219162952.GB31426@phare.normalesup.org>
	<4B7EC2DD.5000209@gmail.com>
Message-ID: <20100219171954.GE31426@phare.normalesup.org>

On Fri, Feb 19, 2010 at 10:57:01AM -0600, Bruce Southey wrote:
> Will it end up as cython?

I am trying to convince the engineer who is doing the work to go down
that way but he does like cython. I am hesitent to impose my point of
view to a highly qualified engineer, but I don't like having this
hand-written C bind, I must admit.

> (I just used the supplied Python bindings of libsvm so this could be 
> interesting.)

Well, we provide much more, like access to the weights, or vectorized
predict :).

> > Lets say that the focus between scikit.learn and statsmodel is most
> > probably going to be slightly different.

> Having done both (with papers), I find this type of comment assuming 
> because underlying both is the same concepts. What I would like to avoid 
> is having different user syntax for basic models for the same model. For 
> example, with logistic regression in SAS you have to be careful of which 
> is the default event setting as it varies across procedures. At least 
> these SAS procedures use the same unmodified dataset unlike some of the 
> R packages that do lars/lasso.

Indeed, I agree. We'll try to look very closely at statsmodel and not
differ if we can. However, (rant ahead), we hear this story everywhere we
go: match our API. So we are struggling between pymvpa, mdp and statmodel
(I am probably forgetting a few) that all differ slightly. We are willing
to adapt as long as it is not damaging for our usecases, but it would be
nice to have a common discussion.

Also, there will be differences APIs, as far as I understand the
statsmodel API. For instance, I believe that constructors of models
should work without passing it the data (the data could be optional). The
reason being that on-line estimators shouldn't be passed in
initiallisation data. As a consequence, maybe the 'fit' method should
take the data... All this is quite open to me, and I don't want to draw
any premature conclusion.

We have not done any API design so far, because we are trying to
get a feal of what the existing APIs are, and because we want to have
working code to throw usecases at it. Also, we are extremely open to
comments, just subscribe to the scikit.learn mailing list (not everybody
involved with scikit learn follows this high-traffic mailing list).

> >> What would be nice is the acceptance of input data types between learn
> >> and statsmodels especially for things like logistic regression. While I
> >> understand the need for duplicate functions, it may be desirable share
> >> at least code since both code bases are still relatively 'new'.

> > Well, as far as I am concerned, data types are numpy arrays. I am weary
> > of implmenting higher level abstractions. Its more the APIs that may
> > different, and that we will have to keep in sync.

> I do agree especially now that I have learnt the 'array' approach of 
> doing things.

> In some way my view of integration of things is Zelig -not that I have 
> really looked at it (as it is in R) :
> http://gking.harvard.edu/zelig/

Well, let us try not to have to build common API and integration a
posteriori, build right from the start. A bit of API work is well worth
the effort, I believe. And please feal free to pitch in.

> The seamless ability to link packages is rather appealing and both 
> scikits share at least numpy.

And scipy, I believe.

Cheers,

Ga?l


From matthew.brett at gmail.com  Fri Feb 19 12:34:37 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 19 Feb 2010 09:34:37 -0800
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <20100219171954.GE31426@phare.normalesup.org>
References: <20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com>
	<20100219162952.GB31426@phare.normalesup.org>
	<4B7EC2DD.5000209@gmail.com>
	<20100219171954.GE31426@phare.normalesup.org>
Message-ID: <1e2af89e1002190934o76dc44b3m5fc586803bd8c4f4@mail.gmail.com>

Hi,

On Fri, Feb 19, 2010 at 9:19 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Fri, Feb 19, 2010 at 10:57:01AM -0600, Bruce Southey wrote:
>> Will it end up as cython?
>
> I am trying to convince the engineer who is doing the work to go down
> that way but he does like cython. I am hesitent to impose my point of
> view to a highly qualified engineer, but I don't like having this
> hand-written C bind, I must admit.

Oh - dear - that's a risk isn't it?  For making the code easy to
extend and maintain?

See you,

Matthew


From gael.varoquaux at normalesup.org  Fri Feb 19 12:36:05 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 19 Feb 2010 18:36:05 +0100
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <1e2af89e1002190934o76dc44b3m5fc586803bd8c4f4@mail.gmail.com>
References: <1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com>
	<20100219162952.GB31426@phare.normalesup.org>
	<4B7EC2DD.5000209@gmail.com>
	<20100219171954.GE31426@phare.normalesup.org>
	<1e2af89e1002190934o76dc44b3m5fc586803bd8c4f4@mail.gmail.com>
Message-ID: <20100219173605.GA15417@phare.normalesup.org>

On Fri, Feb 19, 2010 at 09:34:37AM -0800, Matthew Brett wrote:
> Hi,

> On Fri, Feb 19, 2010 at 9:19 AM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
> > On Fri, Feb 19, 2010 at 10:57:01AM -0600, Bruce Southey wrote:
> >> Will it end up as cython?

> > I am trying to convince the engineer who is doing the work to go down
> > that way but he does like cython. I am hesitent to impose my point of
> > view to a highly qualified engineer, but I don't like having this
> > hand-written C bind, I must admit.

> Oh - dear - that's a risk isn't it?  For making the code easy to
> extend and maintain?

Right, I agree.

Ga?l


From jsseabold at gmail.com  Fri Feb 19 12:45:04 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Fri, 19 Feb 2010 12:45:04 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <20100219171954.GE31426@phare.normalesup.org>
References: <20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com> 
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com> 
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com> 
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com> 
	<20100219162952.GB31426@phare.normalesup.org>
	<4B7EC2DD.5000209@gmail.com> 
	<20100219171954.GE31426@phare.normalesup.org>
Message-ID: <c048da1c1002190945qfe840e4oe937bcaf8a2bb10d@mail.gmail.com>

On Fri, Feb 19, 2010 at 12:19 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Fri, Feb 19, 2010 at 10:57:01AM -0600, Bruce Southey wrote:
>> Will it end up as cython?
>
> I am trying to convince the engineer who is doing the work to go down
> that way but he does like cython. I am hesitent to impose my point of
> view to a highly qualified engineer, but I don't like having this
> hand-written C bind, I must admit.
>
>> (I just used the supplied Python bindings of libsvm so this could be
>> interesting.)
>
> Well, we provide much more, like access to the weights, or vectorized
> predict :).
>
>> > Lets say that the focus between scikit.learn and statsmodel is most
>> > probably going to be slightly different.
>
>> Having done both (with papers), I find this type of comment assuming
>> because underlying both is the same concepts. What I would like to avoid
>> is having different user syntax for basic models for the same model. For
>> example, with logistic regression in SAS you have to be careful of which
>> is the default event setting as it varies across procedures. At least
>> these SAS procedures use the same unmodified dataset unlike some of the
>> R packages that do lars/lasso.
>
> Indeed, I agree. We'll try to look very closely at statsmodel and not
> differ if we can. However, (rant ahead), we hear this story everywhere we
> go: match our API. So we are struggling between pymvpa, mdp and statmodel
> (I am probably forgetting a few) that all differ slightly. We are willing
> to adapt as long as it is not damaging for our usecases, but it would be
> nice to have a common discussion.
>
> Also, there will be differences APIs, as far as I understand the
> statsmodel API. For instance, I believe that constructors of models
> should work without passing it the data (the data could be optional). The
> reason being that on-line estimators shouldn't be passed in
> initiallisation data. As a consequence, maybe the 'fit' method should
> take the data... All this is quite open to me, and I don't want to draw
> any premature conclusion.
>

Just a quick comment (disclaimer: all my own thoughts and
misunderstandings...feel free to correct me).  Historically, the
statsmodels package accepted a design during the model instantiation
then you used your dependent variable during the fit method.  To my
mind, though this didn't seem to make much sense for how I think of a
model (probably somewhat discipline specific?).  For the estimators
that we have we are usually fitting a parametric model in order to
test a given theory about the data generating process.  The model
doesn't make much sense to me without the data (my data is not
real-time and I am not data mining).  Again though I want to make the
package as useful to others as possible (without alienating those who
think of models as I do), so of course suggestions on how to improve
the API or make it more general are more than welcome.

> We have not done any API design so far, because we are trying to
> get a feal of what the existing APIs are, and because we want to have
> working code to throw usecases at it. Also, we are extremely open to
> comments, just subscribe to the scikit.learn mailing list (not everybody
> involved with scikit learn follows this high-traffic mailing list).
>
>> >> What would be nice is the acceptance of input data types between learn
>> >> and statsmodels especially for things like logistic regression. While I
>> >> understand the need for duplicate functions, it may be desirable share
>> >> at least code since both code bases are still relatively 'new'.
>
>> > Well, as far as I am concerned, data types are numpy arrays. I am weary
>> > of implmenting higher level abstractions. Its more the APIs that may
>> > different, and that we will have to keep in sync.
>
>> I do agree especially now that I have learnt the 'array' approach of
>> doing things.
>
>> In some way my view of integration of things is Zelig -not that I have
>> really looked at it (as it is in R) :
>> http://gking.harvard.edu/zelig/
>
> Well, let us try not to have to build common API and integration a
> posteriori, build right from the start. A bit of API work is well worth
> the effort, I believe. And please feal free to pitch in.
>

To my mind, the burden is probably more on statsmodels to provide an
interface to the learn code, as we would be more likely to take
advantage of your routines.

>> The seamless ability to link packages is rather appealing and both
>> scikits share at least numpy.
>
> And scipy, I believe.
>
> Cheers,
>
> Ga?l
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From gael.varoquaux at normalesup.org  Fri Feb 19 12:55:40 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 19 Feb 2010 18:55:40 +0100
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <c048da1c1002190945qfe840e4oe937bcaf8a2bb10d@mail.gmail.com>
References: <1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com>
	<20100219162952.GB31426@phare.normalesup.org>
	<4B7EC2DD.5000209@gmail.com>
	<20100219171954.GE31426@phare.normalesup.org>
	<c048da1c1002190945qfe840e4oe937bcaf8a2bb10d@mail.gmail.com>
Message-ID: <20100219175540.GF31426@phare.normalesup.org>

On Fri, Feb 19, 2010 at 12:45:04PM -0500, Skipper Seabold wrote:
> > Also, there will be differences APIs, as far as I understand the
> > statsmodel API. For instance, I believe that constructors of models
> > should work without passing it the data (the data could be optional). The
> > reason being that on-line estimators shouldn't be passed in
> > initiallisation data. As a consequence, maybe the 'fit' method should
> > take the data... All this is quite open to me, and I don't want to draw
> > any premature conclusion.


> Just a quick comment (disclaimer: all my own thoughts and
> misunderstandings...feel free to correct me).  Historically, the
> statsmodels package accepted a design during the model instantiation
> then you used your dependent variable during the fit method.  To my
> mind, though this didn't seem to make much sense for how I think of a
> model (probably somewhat discipline specific?).  For the estimators
> that we have we are usually fitting a parametric model in order to
> test a given theory about the data generating process.  The model
> doesn't make much sense to me without the data (my data is not
> real-time and I am not data mining). 

Suppose you implement recursive estimation, say Kalman filtering? There
are usecases for that, and we want to solve them.

Sometimes your data doesn't fit in memory. If you have a forward
selection regression model on huge data, say genomics data that is never
remotely going to fit in your memory and that you are fishing out of a
database, the API is also going to break down.

Also, being able to give initial guesses to the estimator, to do
warm-restart of a convex optimisation for instance, might change
significantly the computational cost of eg a cross validation.

On the other hand, my experience is that trying to solve all the possible
usecases beforehand without working code and examples just leads to
developers staring at a white board. So I'd rather move forward, and
think about API based on examples.

I just wanted to warn that we are probably not going to follow exactly
the existing APIs, and that there were reasons for that. I am not trying
to bash existing APIs, this is a pointless activity, IMHO.

Ga?l


From jsseabold at gmail.com  Fri Feb 19 13:04:50 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Fri, 19 Feb 2010 13:04:50 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <20100219175540.GF31426@phare.normalesup.org>
References: <1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com> 
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com> 
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com> 
	<20100219162952.GB31426@phare.normalesup.org>
	<4B7EC2DD.5000209@gmail.com> 
	<20100219171954.GE31426@phare.normalesup.org>
	<c048da1c1002190945qfe840e4oe937bcaf8a2bb10d@mail.gmail.com> 
	<20100219175540.GF31426@phare.normalesup.org>
Message-ID: <c048da1c1002191004pd1df37wb4c99c228a74621e@mail.gmail.com>

On Fri, Feb 19, 2010 at 12:55 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Fri, Feb 19, 2010 at 12:45:04PM -0500, Skipper Seabold wrote:
>> > Also, there will be differences APIs, as far as I understand the
>> > statsmodel API. For instance, I believe that constructors of models
>> > should work without passing it the data (the data could be optional). The
>> > reason being that on-line estimators shouldn't be passed in
>> > initiallisation data. As a consequence, maybe the 'fit' method should
>> > take the data... All this is quite open to me, and I don't want to draw
>> > any premature conclusion.
>
>
>> Just a quick comment (disclaimer: all my own thoughts and
>> misunderstandings...feel free to correct me). ?Historically, the
>> statsmodels package accepted a design during the model instantiation
>> then you used your dependent variable during the fit method. ?To my
>> mind, though this didn't seem to make much sense for how I think of a
>> model (probably somewhat discipline specific?). ?For the estimators
>> that we have we are usually fitting a parametric model in order to
>> test a given theory about the data generating process. ?The model
>> doesn't make much sense to me without the data (my data is not
>> real-time and I am not data mining).
>
> Suppose you implement recursive estimation, say Kalman filtering? There
> are usecases for that, and we want to solve them.
>
> Sometimes your data doesn't fit in memory. If you have a forward
> selection regression model on huge data, say genomics data that is never
> remotely going to fit in your memory and that you are fishing out of a
> database, the API is also going to break down.
>
> Also, being able to give initial guesses to the estimator, to do
> warm-restart of a convex optimisation for instance, might change
> significantly the computational cost of eg a cross validation.
>
> On the other hand, my experience is that trying to solve all the possible
> usecases beforehand without working code and examples just leads to
> developers staring at a white board. So I'd rather move forward, and
> think about API based on examples.
>

All valid points, and I agree.  In the past, I've found it very hard
to code for use cases that I am not aware of ;)

> I just wanted to warn that we are probably not going to follow exactly
> the existing APIs, and that there were reasons for that. I am not trying
> to bash existing APIs, this is a pointless activity, IMHO.
>
> Ga?l
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From e.antero.tammi at gmail.com  Fri Feb 19 14:17:31 2010
From: e.antero.tammi at gmail.com (eat)
Date: Fri, 19 Feb 2010 19:17:31 +0000 (UTC)
Subject: [SciPy-User] convertin from octave/ matlab to scipy
References: <loom.20100218T225412-217@post.gmane.org>
	<1cd32cbb1002181956n107c1f13u26755017ea98ab33@mail.gmail.com>
	<1cd32cbb1002182029m25225883s81f67cd4526ec3ed@mail.gmail.com>
	<loom.20100219T134742-418@post.gmane.org>
	<1cd32cbb1002190635l7bf149e3xa2bbd54ea62fb100@mail.gmail.com>
Message-ID: <loom.20100219T194529-369@post.gmane.org>

> I often translate matlab functions with a simple structure (even if
> the calculation is complicated) quite literally to python/numpy/scipy.
> However, for the big structure and control flow using, for example,
> classes is a huge advantage compared to matlab. (Although matlab is
> also improving its support of classes, which I haven't used yet.)

I think matlab has improved a lot lately with OOP. Unfortunately octave hasn't 
been able to catch matlab in this respect. They just released 3.2.x with OOP 
support comparable with matlab some 5 years ago. (Basically you need to create 
each class in separate directory with lot of boilerplate code).

Also data visualization with octave doesn't just rock.

> 
> A good book for python (for scienc) that uses numpy is very helpful.
> And I found and still find it very informative to read code written by
> others.

I'll agree. It's just your previous posts that motivates me to browse more and 
gives me the confident that scipy/ python is worthwhile to study. It seems 
that octave/ matlab do have a more specific (or narrow) scope than scipy/ 
python. I'm allready confident that learning scipy/ python pays dividens in 
long run.

Thanks,
eat
> 
> Good luck,
> 
> Josef




From josef.pktd at gmail.com  Fri Feb 19 14:49:32 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 14:49:32 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <c048da1c1002191004pd1df37wb4c99c228a74621e@mail.gmail.com>
References: <1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<1cd32cbb1002181559q93cc2b5m86850f155cdc66df@mail.gmail.com>
	<20100219000443.GI32024@phare.normalesup.org>
	<4B7EB156.8010409@gmail.com>
	<20100219162952.GB31426@phare.normalesup.org>
	<4B7EC2DD.5000209@gmail.com>
	<20100219171954.GE31426@phare.normalesup.org>
	<c048da1c1002190945qfe840e4oe937bcaf8a2bb10d@mail.gmail.com>
	<20100219175540.GF31426@phare.normalesup.org>
	<c048da1c1002191004pd1df37wb4c99c228a74621e@mail.gmail.com>
Message-ID: <1cd32cbb1002191149v256fb1b9o49fc6505047ee2ec@mail.gmail.com>

On Fri, Feb 19, 2010 at 1:04 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Fri, Feb 19, 2010 at 12:55 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>> On Fri, Feb 19, 2010 at 12:45:04PM -0500, Skipper Seabold wrote:
>>> > Also, there will be differences APIs, as far as I understand the
>>> > statsmodel API. For instance, I believe that constructors of models
>>> > should work without passing it the data (the data could be optional). The
>>> > reason being that on-line estimators shouldn't be passed in
>>> > initiallisation data. As a consequence, maybe the 'fit' method should
>>> > take the data... All this is quite open to me, and I don't want to draw
>>> > any premature conclusion.
>>
>>
>>> Just a quick comment (disclaimer: all my own thoughts and
>>> misunderstandings...feel free to correct me). ?Historically, the
>>> statsmodels package accepted a design during the model instantiation
>>> then you used your dependent variable during the fit method. ?To my
>>> mind, though this didn't seem to make much sense for how I think of a
>>> model (probably somewhat discipline specific?). ?For the estimators
>>> that we have we are usually fitting a parametric model in order to
>>> test a given theory about the data generating process. ?The model
>>> doesn't make much sense to me without the data (my data is not
>>> real-time and I am not data mining).
>>
>> Suppose you implement recursive estimation, say Kalman filtering? There
>> are usecases for that, and we want to solve them.

Or incremental least squares as the discussion with Nathaniel shows.

>>
>> Sometimes your data doesn't fit in memory. If you have a forward
>> selection regression model on huge data, say genomics data that is never
>> remotely going to fit in your memory and that you are fishing out of a
>> database, the API is also going to break down.
>>
>> Also, being able to give initial guesses to the estimator, to do
>> warm-restart of a convex optimisation for instance, might change
>> significantly the computational cost of eg a cross validation.

For many of the linear models so far we didn't need starting values,
but for the maximum likelihood estimators, fit takes a starting_value
argument. For examples of GLS, and in general for two-step or
multi-stage methods, we can call the same estimator several times with
updated starting values, e.g. GLSAR has a fit and an iterative_fit
where iterative_fit does the updating internally.
One problem is that fit returns a Results instance, but with deferred
calculations the cost might not be high anymore.

We don't have currently any updating on data in statsmodels, but that
doesn't need to stay this way. Since we don't have any
recursive/iterative (over observations) algorithms yet, it would
currently not make any difference in the calculations. I don't have
any idea yet how or how much to do with "online" estimation and batch
processing in time series. But cross-validation is a big reason to
find a way.
Depending on the use case, it would also differ considerably which
style of implementation is the best, e.g. one call to convolve or
correlate or fft, and I have a very fast c loop over the entire time
series, but not useful if I want to do Kalman filtering.

I don't think very similar API's need to be a target either, as long
as they are easy to figure out, machine learning is "training" the
model, and we are "fitting" it to the data. Since the audience and
main usecases can be rather different, the terminology also varies.

What I find more of an issue to easy and efficient usage across
packages, is how cheap it is to get the data in and the results out.
And since the core routines just require numpy arrays, in learn and
statsmodels, this should remain easy and compatible, without having to
go through lots of layers first.


>>
>> On the other hand, my experience is that trying to solve all the possible
>> usecases beforehand without working code and examples just leads to
>> developers staring at a white board. So I'd rather move forward, and
>> think about API based on examples.
>>
>
> All valid points, and I agree. ?In the past, I've found it very hard
> to code for use cases that I am not aware of ;)

Me too. The API for the current models looks ok, but new models will
require adjustments to the generic API.

Josef

>
>> I just wanted to warn that we are probably not going to follow exactly
>> the existing APIs, and that there were reasons for that. I am not trying
>> to bash existing APIs, this is a pointless activity, IMHO.
>>
>> Ga?l
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From peter.cimermancic at gmail.com  Fri Feb 19 14:58:43 2010
From: peter.cimermancic at gmail.com (=?UTF-8?Q?Peter_Cimerman=C4=8Di=C4=8D?=)
Date: Fri, 19 Feb 2010 11:58:43 -0800
Subject: [SciPy-User] Fsolve Question
Message-ID: <18d53ca61002191158x65b2e09ft960361335b230cca@mail.gmail.com>

Hi,

I'm trying to solve a set (let's say two) of equations, where multiple
solutions are possible. Like for example in intersections of quadratic and
linear curve. How can I do that?

Thank you in advance,

Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100219/2283b24c/attachment.html>

From bjracine at glosten.com  Fri Feb 19 15:21:10 2010
From: bjracine at glosten.com (Benjamin J. Racine)
Date: Fri, 19 Feb 2010 12:21:10 -0800
Subject: [SciPy-User] Valid array assignment argument
Message-ID: <971B2F33-C499-4B20-99C2-F6416FD3F9D3@glosten.com>

Anybody know the pythonic way to get what I'm seeking with this line of invalid code...

desired_columns = np.array([np.arange(10), 13, 15])

Many thanks,
Ben R.

From josef.pktd at gmail.com  Fri Feb 19 15:25:53 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 15:25:53 -0500
Subject: [SciPy-User] Valid array assignment argument
In-Reply-To: <971B2F33-C499-4B20-99C2-F6416FD3F9D3@glosten.com>
References: <971B2F33-C499-4B20-99C2-F6416FD3F9D3@glosten.com>
Message-ID: <1cd32cbb1002191225p1529fdbfr60d4960f50ec2efe@mail.gmail.com>

On Fri, Feb 19, 2010 at 3:21 PM, Benjamin J. Racine
<bjracine at glosten.com> wrote:
> Anybody know the pythonic way to get what I'm seeking with this line of invalid code...
>
> desired_columns = np.array([np.arange(10), 13, 15])

indices need to be concatenated so they are not nested list/array

>>> np.array(range(10) + [13, 15])
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 13, 15])
>>> np.array(np.r_[np.arange(10), 13, 15])
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 13, 15])

Josef


>
> Many thanks,
> Ben R.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From dwf at cs.toronto.edu  Fri Feb 19 15:29:59 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 19 Feb 2010 15:29:59 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <4B7EA16F.1080905@american.edu>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>	<20100218223009.GE32024@phare.normalesup.org>	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<4B7EA16F.1080905@american.edu>
Message-ID: <0C8BE572-49D9-4375-A075-FCBF6F739D19@cs.toronto.edu>

On 19-Feb-10, at 9:34 AM, Alan G Isaac wrote:

> Note that the authors include contact info at the bottom
> of that page, if you wish to explore whether they might
> consider releasing the code under a BSD license.

I'm hesitant, given that the implication is "GPL-or-commercial". It  
might be different if there were one or two nontrivial functions that  
were not central to the problem that the entire package solves, but I  
expect we'd be asking them to relicense nearly the entire thing.

David


From ivo.maljevic at gmail.com  Fri Feb 19 15:31:47 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Fri, 19 Feb 2010 15:31:47 -0500
Subject: [SciPy-User] Valid array assignment argument
In-Reply-To: <1cd32cbb1002191225p1529fdbfr60d4960f50ec2efe@mail.gmail.com>
References: <971B2F33-C499-4B20-99C2-F6416FD3F9D3@glosten.com>
	<1cd32cbb1002191225p1529fdbfr60d4960f50ec2efe@mail.gmail.com>
Message-ID: <826c64da1002191231x8c6f25r5feed309be6704cf@mail.gmail.com>

It looks like the second solution can be written in a shorter form:

np.r_[np.arange(10), 13, 15]

Ivo

On 19 February 2010 15:25, <josef.pktd at gmail.com> wrote:

> On Fri, Feb 19, 2010 at 3:21 PM, Benjamin J. Racine
> <bjracine at glosten.com> wrote:
> > Anybody know the pythonic way to get what I'm seeking with this line of
> invalid code...
> >
> > desired_columns = np.array([np.arange(10), 13, 15])
>
> indices need to be concatenated so they are not nested list/array
>
> >>> np.array(range(10) + [13, 15])
> array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 13, 15])
> >>> np.array(np.r_[np.arange(10), 13, 15])
> array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 13, 15])
>
> Josef
>
>
> >
> > Many thanks,
> > Ben R.
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100219/74c43623/attachment.html>

From dwf at cs.toronto.edu  Fri Feb 19 16:19:52 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 19 Feb 2010 16:19:52 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<79A38618-9730-4E95-91E2-DA64F6B9BC8D@cs.toronto.edu>
	<20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<4B7EA16F.1080905@american.edu>
	<1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>
	<c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>
	<1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>
Message-ID: <E8ABAF72-40A7-4ACD-ADBA-EF9E73995BF8@cs.toronto.edu>


On 19-Feb-10, at 10:49 AM, josef.pktd at gmail.com wrote:

> He added last year the MIT license to all his code, except
> unfortunately his time series code. I will eventually target his GMM
> code, but some of the Gauss support functions are not yet available in
> python.

By GMM do you mean Gaussian mixture? I have some fairly well-tested  
code for that if you're interested (with a bunch of features like  
selective component updates, various sorts of pseudocount strategies,  
etc.

David


From josef.pktd at gmail.com  Fri Feb 19 16:43:45 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 16:43:45 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <E8ABAF72-40A7-4ACD-ADBA-EF9E73995BF8@cs.toronto.edu>
References: <c048da1c1002161114q1da85862kac727dcf74d73b92@mail.gmail.com>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<4B7EA16F.1080905@american.edu>
	<1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>
	<c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>
	<1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>
	<E8ABAF72-40A7-4ACD-ADBA-EF9E73995BF8@cs.toronto.edu>
Message-ID: <1cd32cbb1002191343u6e5494ffm73cb82a87cd74192@mail.gmail.com>

On Fri, Feb 19, 2010 at 4:19 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>
> On 19-Feb-10, at 10:49 AM, josef.pktd at gmail.com wrote:
>
>> He added last year the MIT license to all his code, except
>> unfortunately his time series code. I will eventually target his GMM
>> code, but some of the Gauss support functions are not yet available in
>> python.
>
> By GMM do you mean Gaussian mixture? I have some fairly well-tested
> code for that if you're interested (with a bunch of features like
> selective component updates, various sorts of pseudocount strategies,
> etc.

No, Generalized Method of Moments, sorry, too much econometrics context.
Generalized Method of Moments GMM is an generic estimation method
similar to maximum likelihood or matching moments that has a very wide
range of applications, but in contrast to MLE uses much less
distributional information.

It's dangerous to use acronyms because often they mean different
things in different fields.

Gaussian Mixture Models GMM are more the domain of scikits.learn with
David C's EM code.

Josef

Josef

>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From vanforeest at gmail.com  Fri Feb 19 16:50:32 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Fri, 19 Feb 2010 22:50:32 +0100
Subject: [SciPy-User] Fsolve Question
In-Reply-To: <18d53ca61002191158x65b2e09ft960361335b230cca@mail.gmail.com>
References: <18d53ca61002191158x65b2e09ft960361335b230cca@mail.gmail.com>
Message-ID: <fa510ff81002191350k7011a788pcf0ba31dcb8fface@mail.gmail.com>

Hi,

YOu could take a look at the scipy cookbook: Intersection of functions
Compute the points at which two given functions intersect.

Hope this helps.

Nicky


On 19 February 2010 20:58, Peter Cimerman?i?
<peter.cimermancic at gmail.com> wrote:
> Hi,
> I'm trying to solve a set (let's say two) of equations, where multiple
> solutions are possible. Like for example in intersections of quadratic and
> linear curve. How can I do that?
> Thank you in advance,
> Peter
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From aisaac at american.edu  Fri Feb 19 18:56:41 2010
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 19 Feb 2010 18:56:41 -0500
Subject: [SciPy-User] Fsolve Question
In-Reply-To: <fa510ff81002191350k7011a788pcf0ba31dcb8fface@mail.gmail.com>
References: <18d53ca61002191158x65b2e09ft960361335b230cca@mail.gmail.com>
	<fa510ff81002191350k7011a788pcf0ba31dcb8fface@mail.gmail.com>
Message-ID: <4B7F2539.2020309@american.edu>

> On 19 February 2010 20:58, Peter Cimerman?i? wrote:
>> I'm trying to solve a set (let's say two) of equations, where multiple
>> solutions are possible. Like for example in intersections of quadratic and
>> linear curve. How can I do that?


On 2/19/2010 4:50 PM, nicky van foreest wrote:
> YOu could take a look at the scipy cookbook: Intersection of functions
> Compute the points at which two given functions intersect.


That example actually finds a single zero.
(I.e., it is mistitled.)

fwiw,
Alan Isaac



From harald.schilly at gmail.com  Fri Feb 19 19:04:08 2010
From: harald.schilly at gmail.com (Harald Schilly)
Date: Sat, 20 Feb 2010 01:04:08 +0100
Subject: [SciPy-User] Fsolve Question
In-Reply-To: <18d53ca61002191158x65b2e09ft960361335b230cca@mail.gmail.com>
References: <18d53ca61002191158x65b2e09ft960361335b230cca@mail.gmail.com>
Message-ID: <20548feb1002191604u19d2e96frdfe6d122fe8a37e3@mail.gmail.com>

On Fri, Feb 19, 2010 at 20:58, Peter Cimerman?i?
<peter.cimermancic at gmail.com> wrote:
> Hi,
> I'm trying to solve a set (let's say two) of equations, where multiple
> solutions are possible.

You can use some kind of symbolic solve command, but that might fail
in general. You can see it numerically as a global optimization
problem: you ask for the set of all minimums for the sum of all
abs-values of all functions (an equation is a function with l.h.s -
r.h.s.) and therefore this is zero everywhere, where all equations are
fulfilled.

H


From gael.varoquaux at normalesup.org  Sat Feb 20 04:20:22 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 20 Feb 2010 10:20:22 +0100
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <E8ABAF72-40A7-4ACD-ADBA-EF9E73995BF8@cs.toronto.edu>
References: <20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<4B7EA16F.1080905@american.edu>
	<1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>
	<c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>
	<1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>
	<E8ABAF72-40A7-4ACD-ADBA-EF9E73995BF8@cs.toronto.edu>
Message-ID: <20100220092022.GA7207@phare.normalesup.org>

On Fri, Feb 19, 2010 at 04:19:52PM -0500, David Warde-Farley wrote:
> By GMM do you mean Gaussian mixture? I have some fairly well-tested  
> code for that if you're interested (with a bunch of features like  
> selective component updates, various sorts of pseudocount strategies,  
> etc.

David, 

We (scikit.learn) are interested in your code. I haven't come forward yet
and called for contribution, because we need to reorganise a bit the
layout of the project, and define uniform APIs, as the discussion showed,
but in a month, I believe that we'll be in business.

Cheers,

Ga?l


From nwagner at iam.uni-stuttgart.de  Sat Feb 20 05:05:56 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 20 Feb 2010 11:05:56 +0100
Subject: [SciPy-User] scikits.sparse
Message-ID: <web-130542846@uni-stuttgart.de>

  Hi,

I tried to install scikits.sparse.
It failed with

nwagner at linux-mogv:~/hg/scikits.sparse> python setup.py 
install --prefix=$HOME/local
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/command/sdist.py:4: 
UserWarning: Module pkg_resources was already imported 
from 
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/pkg_resources.py, 
but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/command/sdist.py:4: 
UserWarning: Module setuptools was already imported from 
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/__init__.py, 
but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/command/sdist.py:4: 
UserWarning: Module site was already imported from 
/usr/lib64/python2.6/site.pyc, but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
/home/nwagner/local/lib64/python2.6/site-packages/Pyrex/Compiler/Errors.py:17: 
DeprecationWarning: BaseException.message has been 
deprecated as of Python 2.6
   self.message = message
running install
running bdist_egg
running egg_info
writing requirements to 
scikits.sparse.egg-info/requires.txt
writing scikits.sparse.egg-info/PKG-INFO
writing namespace_packages to 
scikits.sparse.egg-info/namespace_packages.txt
writing top-level names to 
scikits.sparse.egg-info/top_level.txt
writing dependency_links to 
scikits.sparse.egg-info/dependency_links.txt
writing requirements to 
scikits.sparse.egg-info/requires.txt
writing scikits.sparse.egg-info/PKG-INFO
writing namespace_packages to 
scikits.sparse.egg-info/namespace_packages.txt
writing top-level names to 
scikits.sparse.egg-info/top_level.txt
writing dependency_links to 
scikits.sparse.egg-info/dependency_links.txt
reading manifest file 
'scikits.sparse.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*~' found 
anywhere in distribution
warning: no previously-included files matching '*.so' 
found anywhere in distribution
warning: no previously-included files matching '*.pyc' 
found anywhere in distribution
writing manifest file 
'scikits.sparse.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-2.6
creating build/lib.linux-x86_64-2.6/scikits
copying scikits/__init__.py -> 
build/lib.linux-x86_64-2.6/scikits
creating build/lib.linux-x86_64-2.6/scikits/sparse
copying scikits/sparse/__init__.py -> 
build/lib.linux-x86_64-2.6/scikits/sparse
copying scikits/sparse/test_cholmod.py -> 
build/lib.linux-x86_64-2.6/scikits/sparse
running build_ext
cythoning scikits/sparse/cholmod.pyx to 
scikits/sparse/cholmod.c

Error converting Pyrex file to C:
------------------------------------------------------------
...
     #    0xA (alpha)
     #    0xB (beta)
     #    0xC (release candidate)
     #    0xF (final)

     char[] PY_VERSION
        ^
------------------------------------------------------------

/usr/lib64/python2.6/site-packages/Cython/Includes/python_version.pxd:31:8: 
Buffer types only allowed as function local variables

Error converting Pyrex file to C:
------------------------------------------------------------
...
     #    0xB (beta)
     #    0xC (release candidate)
     #    0xF (final)

     char[] PY_VERSION
     char[] PY_PATCHLEVEL_REVISION
        ^
------------------------------------------------------------

/usr/lib64/python2.6/site-packages/Cython/Includes/python_version.pxd:32:8: 
Buffer types only allowed as function local variables


How can I fix the problem ?
pyrexc --version
/home/nwagner/local/lib64/python2.6/site-packages/Pyrex/Compiler/Errors.py:17: 
DeprecationWarning: BaseException.message has been 
deprecated as of Python 2.6
   self.message = message
Pyrex version 0.9.8.5



                    Nils


From dagss at student.matnat.uio.no  Sat Feb 20 05:22:46 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 20 Feb 2010 11:22:46 +0100
Subject: [SciPy-User] scikits.sparse
In-Reply-To: <web-130542846@uni-stuttgart.de>
References: <web-130542846@uni-stuttgart.de>
Message-ID: <427c90929b50c81849394c9d37217fc1.squirrel@webmail.uio.no>

>   Hi,
>
> I tried to install scikits.sparse.
> It failed with
..> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
>      #    0xB (beta)
>      #    0xC (release candidate)
>      #    0xF (final)
>
>      char[] PY_VERSION
>      char[] PY_PATCHLEVEL_REVISION
>         ^
> ------------------------------------------------------------
>
> /usr/lib64/python2.6/site-packages/Cython/Includes/python_version.pxd:32:8:
> Buffer types only allowed as function local variables
>
>
> How can I fix the problem ?
> pyrexc --version
> /home/nwagner/local/lib64/python2.6/site-packages/Pyrex/Compiler/Errors.py:17:
> DeprecationWarning: BaseException.message has been
> deprecated as of Python 2.6
>    self.message = message
> Pyrex version 0.9.8.5


The Cython version is what is relevant here, Pyrex is not used (regardless
of what distutils claims!). And it turns out that this error is caused by
a too old version of Cython -- upgrade your Cython and that should fix it.

Dag Sverre



From nwagner at iam.uni-stuttgart.de  Sat Feb 20 05:31:47 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 20 Feb 2010 11:31:47 +0100
Subject: [SciPy-User] scikits.sparse
In-Reply-To: <427c90929b50c81849394c9d37217fc1.squirrel@webmail.uio.no>
References: <web-130542846@uni-stuttgart.de>
	<427c90929b50c81849394c9d37217fc1.squirrel@webmail.uio.no>
Message-ID: <web-130543259@uni-stuttgart.de>

On Sat, 20 Feb 2010 11:22:46 +0100
  "Dag Sverre Seljebotn" <dagss at student.matnat.uio.no> 
wrote:
>>   Hi,
>>
>> I tried to install scikits.sparse.
>> It failed with
> ..> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>>      #    0xB (beta)
>>      #    0xC (release candidate)
>>      #    0xF (final)
>>
>>      char[] PY_VERSION
>>      char[] PY_PATCHLEVEL_REVISION
>>         ^
>> ------------------------------------------------------------
>>
>> /usr/lib64/python2.6/site-packages/Cython/Includes/python_version.pxd:32:8:
>> Buffer types only allowed as function local variables
>>
>>
>> How can I fix the problem ?
>> pyrexc --version
>> /home/nwagner/local/lib64/python2.6/site-packages/Pyrex/Compiler/Errors.py:17:
>> DeprecationWarning: BaseException.message has been
>> deprecated as of Python 2.6
>>    self.message = message
>> Pyrex version 0.9.8.5
> 
> 
> The Cython version is what is relevant here, Pyrex is 
>not used (regardless
> of what distutils claims!). And it turns out that this 
>error is caused by
> a too old version of Cython -- upgrade your Cython and 
>that should fix it.
> 
> Dag Sverre
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

I am using

cython --version
Cython version 0.10.3
  
That version is shipped with Open SuSe 11.2 which is quite 
new.

What is needed by scikits.sparse ?

Nils


From miguel.deval at gmail.com  Sat Feb 20 06:09:14 2010
From: miguel.deval at gmail.com (Miguel de Val Borro)
Date: Sat, 20 Feb 2010 12:09:14 +0100
Subject: [SciPy-User] Padded data structure
Message-ID: <20100220110914.GC10463@poincare>

Hello,

I would like to read padded output files from C structures that look
like this file: http://spec.jpl.nasa.gov/ftp/pub/catalog/catdir.cat 
The second row is a 13 character string that contains whitespaces.  I
have looked at the align option in dtype passed as a parameter to
loadtxt but could not make it work. scipy.io.read_array is depecrated
and I'm not sure it has this functionality. What other options are
there? 

Best regards,
Miguel


From hasslerjc at comcast.net  Sat Feb 20 08:25:17 2010
From: hasslerjc at comcast.net (John Hassler)
Date: Sat, 20 Feb 2010 08:25:17 -0500
Subject: [SciPy-User] Fsolve Question
In-Reply-To: <18d53ca61002191158x65b2e09ft960361335b230cca@mail.gmail.com>
References: <18d53ca61002191158x65b2e09ft960361335b230cca@mail.gmail.com>
Message-ID: <4B7FE2BD.3050705@comcast.net>

On 2/19/2010 2:58 PM, Peter Cimerman?i? wrote:
> Hi,
>
> I'm trying to solve a set (let's say two) of equations, where multiple 
> solutions are possible. Like for example in intersections of quadratic 
> and linear curve. How can I do that?
>
> Thank you in advance,
>
> Peter
>

I'm not sure I understand your question. Are you trying to figure out 
how to use fsolve in general, or how to find multiple solutions using 
fsolve?

john


From abc.mikey at googlemail.com  Sat Feb 20 11:27:39 2010
From: abc.mikey at googlemail.com (mikey)
Date: Sat, 20 Feb 2010 16:27:39 +0000
Subject: [SciPy-User] Adding a Best-Fit Line
Message-ID: <4ebfceac1002200827k12960d91jcbec81cee5d3c00c@mail.gmail.com>

Hi,

I'm trying to do a Q-Q plot for 2 sets of data (which will eventually
be uneven) against each other. I've tried my best to do the Q-Q plot
but when I try to do a best fit line it's nowhere near a best fit.
Could someone let me know how a best fit line is done?

Regards,

Mikey


import numpy as np
import matplotlib.pyplot as plt
import scipy.special as sp
from scipy import stats
idata=np.array([6,2,6,0,5,8,8,5,2,4,5,8,5,8,6,4,4,8,7,4,6,1,6,7,0,-4,3,7,5,1,4,4,5,5,4,2,5,4,4,2,5,5,0,6,3,4,2,4,2,3,4,0,4,4,2,2,7,3,8,1,4,2,2,6-1,1,5,3,3,0,8,8,7,3,5,3,4,6,0,2-5,5,6,0,4,4,3,7,6,0,8,4,3,6-2,5,4,4,5,3,6,3,3,7,1])
pidata=np.array([3,0,3,2,3,6,6,4,4,4,4,7,5,1,3,4,4,8,1,4,6,-4,5,6,-3,-2,3,5,3,1,1,4,5,2,1,0,3,4,4,2,5,4,0,6,3,3,0,3,3,2,0,0,1,0,-2,5,6,2,0,1,1,0,1,4,-1,1,1,-1,0,1,4,4,4,3,4,3,3,4,0,-1,-5,3,4,-1,4,3,1,5,3,0,5,4,4,4,-4,3,2,2,5,3,5,3,2,4,1])
pidata=
P=np.arange(0,100,0.10)
Qx=stats.mstats.mquantiles(pidata,P)
Qy=stats.mstats.mquantiles(idata,P)
plt.plot(Qx, Qy, 'o', mec='black', mfc='green')
m = np.polyfit(Qx, Qy, 1)
yfit = np.polyval(m, Qx)
plt.plot(Qx, yfit, 'b--')
x1=-4.
x2=4.
y1=x1*m[0]+m[1]
y2=x2*m[0]+m[1]
plt.plot([x1,x2], [y1,y2], 'r--')
plt.text(4, -5.8, "Slope=%f, Offset=%f" % (m[0],m[1]))
plt.grid(True)
plt.show()


From abc.mikey at googlemail.com  Sat Feb 20 12:14:02 2010
From: abc.mikey at googlemail.com (mikey)
Date: Sat, 20 Feb 2010 17:14:02 +0000
Subject: [SciPy-User] Adding a Best-Fit Line
In-Reply-To: <4ebfceac1002200827k12960d91jcbec81cee5d3c00c@mail.gmail.com>
References: <4ebfceac1002200827k12960d91jcbec81cee5d3c00c@mail.gmail.com>
Message-ID: <4ebfceac1002200914k40fb7517g893d05df69af4045@mail.gmail.com>

Never mind. Sorry about the silly post. I converted the p value to a
percentage without thinking. It works as expected when held at 1.

On 20 February 2010 16:27, mikey <abc.mikey at googlemail.com> wrote:
> Hi,
>
> I'm trying to do a Q-Q plot for 2 sets of data (which will eventually
> be uneven) against each other. I've tried my best to do the Q-Q plot
> but when I try to do a best fit line it's nowhere near a best fit.
> Could someone let me know how a best fit line is done?
>
> Regards,
>
> Mikey
>
>
> import numpy as np
> import matplotlib.pyplot as plt
> import scipy.special as sp
> from scipy import stats
> idata=np.array([6,2,6,0,5,8,8,5,2,4,5,8,5,8,6,4,4,8,7,4,6,1,6,7,0,-4,3,7,5,1,4,4,5,5,4,2,5,4,4,2,5,5,0,6,3,4,2,4,2,3,4,0,4,4,2,2,7,3,8,1,4,2,2,6-1,1,5,3,3,0,8,8,7,3,5,3,4,6,0,2-5,5,6,0,4,4,3,7,6,0,8,4,3,6-2,5,4,4,5,3,6,3,3,7,1])
> pidata=np.array([3,0,3,2,3,6,6,4,4,4,4,7,5,1,3,4,4,8,1,4,6,-4,5,6,-3,-2,3,5,3,1,1,4,5,2,1,0,3,4,4,2,5,4,0,6,3,3,0,3,3,2,0,0,1,0,-2,5,6,2,0,1,1,0,1,4,-1,1,1,-1,0,1,4,4,4,3,4,3,3,4,0,-1,-5,3,4,-1,4,3,1,5,3,0,5,4,4,4,-4,3,2,2,5,3,5,3,2,4,1])
> pidata=
> P=np.arange(0,100,0.10)
> Qx=stats.mstats.mquantiles(pidata,P)
> Qy=stats.mstats.mquantiles(idata,P)
> plt.plot(Qx, Qy, 'o', mec='black', mfc='green')
> m = np.polyfit(Qx, Qy, 1)
> yfit = np.polyval(m, Qx)
> plt.plot(Qx, yfit, 'b--')
> x1=-4.
> x2=4.
> y1=x1*m[0]+m[1]
> y2=x2*m[0]+m[1]
> plt.plot([x1,x2], [y1,y2], 'r--')
> plt.text(4, -5.8, "Slope=%f, Offset=%f" % (m[0],m[1]))
> plt.grid(True)
> plt.show()
>


From njs at pobox.com  Sat Feb 20 13:07:25 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 20 Feb 2010 10:07:25 -0800
Subject: [SciPy-User] scikits.sparse
In-Reply-To: <web-130543259@uni-stuttgart.de>
References: <web-130542846@uni-stuttgart.de>
	<427c90929b50c81849394c9d37217fc1.squirrel@webmail.uio.no>
	<web-130543259@uni-stuttgart.de>
Message-ID: <961fa2b41002201007q76965b8dl940943dd018d43d@mail.gmail.com>

On Sat, Feb 20, 2010 at 2:31 AM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> I am using
>
> cython --version
> Cython version 0.10.3
>
> That version is shipped with Open SuSe 11.2 which is quite
> new.
>
> What is needed by scikits.sparse ?

Unfortunately, Cython moves fast (but it's no harder to install than
any other Python package).

I don't know the minimum version required, but I'm using 0.12 and it
works for me. I'd just upgrade to 0.12.1, the latest stable release.

-- Nathaniel


From dwf at cs.toronto.edu  Sat Feb 20 15:25:54 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Sat, 20 Feb 2010 15:25:54 -0500
Subject: [SciPy-User] [ANN] scikit.statsmodels 0.2.0 release
In-Reply-To: <20100220092022.GA7207@phare.normalesup.org>
References: <20100218223009.GE32024@phare.normalesup.org>
	<1cd32cbb1002181434x18b6e720r78d0e10a244bde61@mail.gmail.com>
	<1cd32cbb1002181523w25419a11ob9046c54ca44dd1f@mail.gmail.com>
	<3d375d731002181528n61a66286n7b5f79815fc980d0@mail.gmail.com>
	<8F0D2093-A00A-4AE8-AD63-7511F067AA05@cs.toronto.edu>
	<4B7EA16F.1080905@american.edu>
	<1cd32cbb1002190711t1e0ee488i1134f4caf56701f0@mail.gmail.com>
	<c048da1c1002190722p7f34fafdm6fa785e09fded78f@mail.gmail.com>
	<1cd32cbb1002190749j73da4a5fvb789d2635c7392a7@mail.gmail.com>
	<E8ABAF72-40A7-4ACD-ADBA-EF9E73995BF8@cs.toronto.edu>
	<20100220092022.GA7207@phare.normalesup.org>
Message-ID: <3C025002-DB86-4004-8A09-9E79029DF84A@cs.toronto.edu>

On 20-Feb-10, at 4:20 AM, Gael Varoquaux wrote:

> On Fri, Feb 19, 2010 at 04:19:52PM -0500, David Warde-Farley wrote:
>> By GMM do you mean Gaussian mixture? I have some fairly well-tested
>> code for that if you're interested (with a bunch of features like
>> selective component updates, various sorts of pseudocount strategies,
>> etc.
>
> David,
>
> We (scikit.learn) are interested in your code. I haven't come  
> forward yet
> and called for contribution, because we need to reorganise a bit the
> layout of the project, and define uniform APIs, as the discussion  
> showed,
> but in a month, I believe that we'll be in business.


Hi Gael,

Cool, although I think David's old code in PyEM may take precedence  
(if there are users in the wild who depend on the old API). The other  
thing I have is some discrete naive Bayes code that operates on  
integer, string or object arrays (2D arrays, though making it operate  
on record arrays as well wouldn't be too hard).

David


From ariver at enthought.com  Sun Feb 21 00:37:22 2010
From: ariver at enthought.com (Aaron River)
Date: Sat, 20 Feb 2010 23:37:22 -0600
Subject: [SciPy-User] ignore:delete
Message-ID: <83ae36ae1002202137h2adddf33oaec21db8c84b132c@mail.gmail.com>

I'm testing some changes to the mail flow on scipy.org.
Please disregard this message.
No reply is necessary.

--
Aaron


From Chris.Barker at noaa.gov  Sun Feb 21 01:55:59 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Sat, 20 Feb 2010 22:55:59 -0800
Subject: [SciPy-User] Padded data structure
In-Reply-To: <20100220110914.GC10463@poincare>
References: <20100220110914.GC10463@poincare>
Message-ID: <4B80D8FF.4050907@noaa.gov>



Miguel de Val Borro wrote:
> I would like to read padded output files from C structures that look
> like this file: http://spec.jpl.nasa.gov/ftp/pub/catalog/catdir.cat 
> The second row is a 13 character string that contains whitespaces.  I
> have looked at the align option in dtype passed as a parameter to
> loadtxt but could not make it work. 

Since there is whitespace between all those columns, I don't think you 
need align.  I suspect that a dtype something like:

('i4', 'S13', 'i4', 'f8', 'f8','f8','f8','f8','f8','f8', 'i2')

will do the trick.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From miguel.deval at gmail.com  Sun Feb 21 02:45:41 2010
From: miguel.deval at gmail.com (Miguel de Val Borro)
Date: Sun, 21 Feb 2010 08:45:41 +0100
Subject: [SciPy-User] Padded data structure
In-Reply-To: <4B80D8FF.4050907@noaa.gov>
References: <20100220110914.GC10463@poincare>
 <4B80D8FF.4050907@noaa.gov>
Message-ID: <20100221074541.GA11162@poincare>

On Sat, Feb 20, 2010 at 10:55:59PM -0800, Christopher Barker wrote:
> > I would like to read padded output files from C structures that look
> > like this file: http://spec.jpl.nasa.gov/ftp/pub/catalog/catdir.cat 
> > The second row is a 13 character string that contains whitespaces.  I
> > have looked at the align option in dtype passed as a parameter to
> > loadtxt but could not make it work. 
> 
> Since there is whitespace between all those columns, I don't think you 
> need align.  I suspect that a dtype something like:
> 
> ('i4', 'S13', 'i4', 'f8', 'f8','f8','f8','f8','f8','f8', 'i2')

The problem is that the S13 string contains whitespaces, which is also
the separator between columns. Using this dtype I get the error:

ValueError: invalid literal for float(): v2,2v2,v

Miguel


From warren.weckesser at enthought.com  Sun Feb 21 03:02:09 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sun, 21 Feb 2010 02:02:09 -0600
Subject: [SciPy-User] Padded data structure
In-Reply-To: <20100221074541.GA11162@poincare>
References: <20100220110914.GC10463@poincare> <4B80D8FF.4050907@noaa.gov>
	<20100221074541.GA11162@poincare>
Message-ID: <4B80E881.5080807@enthought.com>

Miguel de Val Borro wrote:
> On Sat, Feb 20, 2010 at 10:55:59PM -0800, Christopher Barker wrote:
>   
>>> I would like to read padded output files from C structures that look
>>> like this file: http://spec.jpl.nasa.gov/ftp/pub/catalog/catdir.cat 
>>> The second row is a 13 character string that contains whitespaces.  I
>>> have looked at the align option in dtype passed as a parameter to
>>> loadtxt but could not make it work. 
>>>       
>> Since there is whitespace between all those columns, I don't think you 
>> need align.  I suspect that a dtype something like:
>>
>> ('i4', 'S13', 'i4', 'f8', 'f8','f8','f8','f8','f8','f8', 'i2')
>>     
>
> The problem is that the S13 string contains whitespaces, which is also
> the separator between columns. Using this dtype I get the error:
>
> ValueError: invalid literal for float(): v2,2v2,v
>
>   

Any chance you can upgrade your numpy version to the current trunk?  
Somewhere between version 1.3.0 and the current trunk, the 'delimiter' 
argument of genfromtxt() was enhanced to accept a list of field widths.  
The attached script shows how it can be used to read the file "catdir.dat".

Warren


> Miguel
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: read_catdir.py
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100221/a81de899/attachment.ksh>

From miguel.deval at gmail.com  Sun Feb 21 04:24:31 2010
From: miguel.deval at gmail.com (Miguel de Val Borro)
Date: Sun, 21 Feb 2010 10:24:31 +0100
Subject: [SciPy-User] Padded data structure
In-Reply-To: <4B80E881.5080807@enthought.com>
References: <20100220110914.GC10463@poincare> <4B80D8FF.4050907@noaa.gov>
	<20100221074541.GA11162@poincare> <4B80E881.5080807@enthought.com>
Message-ID: <20100221092431.GB11162@poincare>

On Sun, Feb 21, 2010 at 02:02:09AM -0600, Warren Weckesser wrote:
> Miguel de Val Borro wrote:
> >On Sat, Feb 20, 2010 at 10:55:59PM -0800, Christopher Barker wrote:
> >>>I would like to read padded output files from C structures that look
> >>>like this file:
> >>>http://spec.jpl.nasa.gov/ftp/pub/catalog/catdir.cat The second
> >>>row is a 13 character string that contains whitespaces.  I
> >>>have looked at the align option in dtype passed as a parameter to
> >>>loadtxt but could not make it work.
> >>Since there is whitespace between all those columns, I don't
> >>think you need align.  I suspect that a dtype something like:
> >>
> >>('i4', 'S13', 'i4', 'f8', 'f8','f8','f8','f8','f8','f8', 'i2')
> >
> >The problem is that the S13 string contains whitespaces, which is also
> >the separator between columns. Using this dtype I get the error:
> >
> >ValueError: invalid literal for float(): v2,2v2,v
> >
> 
> Any chance you can upgrade your numpy version to the current trunk?
> Somewhere between version 1.3.0 and the current trunk, the
> 'delimiter' argument of genfromtxt() was enhanced to accept a list
> of field widths.  The attached script shows how it can be used to
> read the file "catdir.dat".

Thank you very much for the script, it does exactly what I needed.
Actually genfromtxt accepts field widths also in numpy 1.3.0 shipped
with Ubuntu 9.10. It looks like the function is not easy to find in the
documentation, althought this may be already fixed in the repository.

Miguel


From nwagner at iam.uni-stuttgart.de  Sun Feb 21 05:00:55 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sun, 21 Feb 2010 11:00:55 +0100
Subject: [SciPy-User] scikits.sparse
In-Reply-To: <961fa2b41002201007q76965b8dl940943dd018d43d@mail.gmail.com>
References: <web-130542846@uni-stuttgart.de>
	<427c90929b50c81849394c9d37217fc1.squirrel@webmail.uio.no>
	<web-130543259@uni-stuttgart.de>
	<961fa2b41002201007q76965b8dl940943dd018d43d@mail.gmail.com>
Message-ID: <web-130558143@uni-stuttgart.de>

On Sat, 20 Feb 2010 10:07:25 -0800
  Nathaniel Smith <njs at pobox.com> wrote:
> On Sat, Feb 20, 2010 at 2:31 AM, Nils Wagner
> <nwagner at iam.uni-stuttgart.de> wrote:
>> I am using
>>
>> cython --version
>> Cython version 0.10.3
>>
>> That version is shipped with Open SuSe 11.2 which is 
>>quite
>> new.
>>
>> What is needed by scikits.sparse ?
> 
> Unfortunately, Cython moves fast (but it's no harder to 
>install than
> any other Python package).
> 
> I don't know the minimum version required, but I'm using 
>0.12 and it
> works for me. I'd just upgrade to 0.12.1, the latest 
>stable release.
> 
> -- Nathaniel
> ___________

Hi Nathanial,

I followed your advice and installed 0.12.1

Now

python setup.py install --prefix=$HOME/local

fails with

/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/command/sdist.py:4: 
UserWarning: Module pkg_resources was already imported 
from 
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/pkg_resources.py, 
but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/command/sdist.py:4: 
UserWarning: Module setuptools was already imported from 
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/__init__.py, 
but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/command/sdist.py:4: 
UserWarning: Module site was already imported from 
/usr/lib64/python2.6/site.pyc, but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
/home/nwagner/local/lib64/python2.6/site-packages/Pyrex/Compiler/Errors.py:17: 
DeprecationWarning: BaseException.message has been 
deprecated as of Python 2.6
   self.message = message
running install
running bdist_egg
running egg_info
writing requirements to 
scikits.sparse.egg-info/requires.txt
writing scikits.sparse.egg-info/PKG-INFO
writing namespace_packages to 
scikits.sparse.egg-info/namespace_packages.txt
writing top-level names to 
scikits.sparse.egg-info/top_level.txt
writing dependency_links to 
scikits.sparse.egg-info/dependency_links.txt
writing requirements to 
scikits.sparse.egg-info/requires.txt
writing scikits.sparse.egg-info/PKG-INFO
writing namespace_packages to 
scikits.sparse.egg-info/namespace_packages.txt
writing top-level names to 
scikits.sparse.egg-info/top_level.txt
writing dependency_links to 
scikits.sparse.egg-info/dependency_links.txt
reading manifest file 
'scikits.sparse.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*~' found 
anywhere in distribution
warning: no previously-included files matching '*.so' 
found anywhere in distribution
warning: no previously-included files matching '*.pyc' 
found anywhere in distribution
writing manifest file 
'scikits.sparse.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-2.6
creating build/lib.linux-x86_64-2.6/scikits
copying scikits/__init__.py -> 
build/lib.linux-x86_64-2.6/scikits
creating build/lib.linux-x86_64-2.6/scikits/sparse
copying scikits/sparse/__init__.py -> 
build/lib.linux-x86_64-2.6/scikits/sparse
copying scikits/sparse/test_cholmod.py -> 
build/lib.linux-x86_64-2.6/scikits/sparse
running build_ext
cythoning scikits/sparse/cholmod.pyx to 
scikits/sparse/cholmod.c
building 'scikits.sparse.cholmod' extension
creating build/temp.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/scikits
creating build/temp.linux-x86_64-2.6/scikits/sparse
/usr/bin/gcc -fno-strict-aliasing -DNDEBUG 
-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 
-fstack-protector -funwind-tables 
-fasynchronous-unwind-tables -g -fwrapv -fPIC 
-I/usr/include/python2.6 -c scikits/sparse/cholmod.c -o 
build/temp.linux-x86_64-2.6/scikits/sparse/cholmod.o
scikits/sparse/cholmod.c:158:31: error: 
numpy/arrayobject.h: Datei oder Verzeichnis nicht gefunden
scikits/sparse/cholmod.c:159:31: error: 
numpy/ufuncobject.h: Datei oder Verzeichnis nicht gefunden
  

"Datei oder Verzeichnis nicht gefunden" means "No such 
file or directory "

However

locate arrayobject.h

/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/include/numpy/arrayobject.h
/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/include/numpy/ndarrayobject.h
/home/nwagner/src/ScientificPython-2.9.0/Include/Scientific/arrayobject.h
/home/nwagner/svn/ironclad-read-only/stub/Include/.svn/text-base/bytearrayobject.h.svn-base
/home/nwagner/svn/ironclad-read-only/stub/Include/bytearrayobject.h
/home/nwagner/svn/numpy/numpy/core/include/numpy/.svn/prop-base/arrayobject.h.svn-base
/home/nwagner/svn/numpy/numpy/core/include/numpy/.svn/text-base/arrayobject.h.svn-base
/home/nwagner/svn/numpy/numpy/core/include/numpy/.svn/text-base/ndarrayobject.h.svn-base
/home/nwagner/svn/numpy/numpy/core/include/numpy/arrayobject.h
/home/nwagner/svn/numpy/numpy/core/include/numpy/ndarrayobject.h
/home/nwagner/svn/numpy/numpy/core/src/multiarray/.svn/text-base/arrayobject.h.svn-base
/home/nwagner/svn/numpy/numpy/core/src/multiarray/arrayobject.h
/usr/include/python2.6/bytearrayobject.h
/usr/lib64/python2.6/site-packages/numpy/core/include/numpy/arrayobject.h
/usr/lib64/python2.6/site-packages/numpy/core/include/numpy/ndarrayobject.h

How can I fix the problem ?

  Nils


From warren.weckesser at enthought.com  Sun Feb 21 10:05:59 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sun, 21 Feb 2010 09:05:59 -0600
Subject: [SciPy-User] Padded data structure
In-Reply-To: <20100221092431.GB11162@poincare>
References: <20100220110914.GC10463@poincare>
	<4B80D8FF.4050907@noaa.gov>	<20100221074541.GA11162@poincare>
	<4B80E881.5080807@enthought.com> <20100221092431.GB11162@poincare>
Message-ID: <4B814BD7.7050700@enthought.com>

Miguel de Val Borro wrote:
> On Sun, Feb 21, 2010 at 02:02:09AM -0600, Warren Weckesser wrote:
>   
>> Miguel de Val Borro wrote:
>>     
>>> On Sat, Feb 20, 2010 at 10:55:59PM -0800, Christopher Barker wrote:
>>>       
>>>>> I would like to read padded output files from C structures that look
>>>>> like this file:
>>>>> http://spec.jpl.nasa.gov/ftp/pub/catalog/catdir.cat The second
>>>>> row is a 13 character string that contains whitespaces.  I
>>>>> have looked at the align option in dtype passed as a parameter to
>>>>> loadtxt but could not make it work.
>>>>>           
>>>> Since there is whitespace between all those columns, I don't
>>>> think you need align.  I suspect that a dtype something like:
>>>>
>>>> ('i4', 'S13', 'i4', 'f8', 'f8','f8','f8','f8','f8','f8', 'i2')
>>>>         
>>> The problem is that the S13 string contains whitespaces, which is also
>>> the separator between columns. Using this dtype I get the error:
>>>
>>> ValueError: invalid literal for float(): v2,2v2,v
>>>
>>>       
>> Any chance you can upgrade your numpy version to the current trunk?
>> Somewhere between version 1.3.0 and the current trunk, the
>> 'delimiter' argument of genfromtxt() was enhanced to accept a list
>> of field widths.  The attached script shows how it can be used to
>> read the file "catdir.dat".
>>     
>
> Thank you very much for the script, it does exactly what I needed.
> Actually genfromtxt accepts field widths also in numpy 1.3.0 shipped
>   

Well, that's a nice surprise.  It works for me too in 1.3.0.  This 
feature is not mentioned in the docstring in 1.3.0, so I didn't think to 
try it.

Cheers,

Warren


> with Ubuntu 9.10. It looks like the function is not easy to find in the
> documentation, althought this may be already fixed in the repository.
>
> Miguel
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From helge.reikeras at gmail.com  Sun Feb 21 11:23:27 2010
From: helge.reikeras at gmail.com (Helge Reikeras)
Date: Sun, 21 Feb 2010 18:23:27 +0200
Subject: [SciPy-User] GMMs was Re:  [ANN] scikit.statsmodels 0.2.0 release
Message-ID: <1840ee061002210823k4508b12eqc330abeef0f2818f@mail.gmail.com>

On Fri, Feb 19, 2010 at 11:19 PM, David Warde-Farley <dwf at cs.toronto.edu>wrote:

>
> On 19-Feb-10, at 10:49 AM, josef.pktd at gmail.com wrote:
>
> > He added last year the MIT license to all his code, except
> > unfortunately his time series code. I will eventually target his GMM
> > code, but some of the Gauss support functions are not yet available in
> > python.
>
> By GMM do you mean Gaussian mixture? I have some fairly well-tested
> code for that if you're interested (with a bunch of features like
> selective component updates, various sorts of pseudocount strategies,
> etc.
>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>

David,

I am looking into Variational Bayesian Inference for simultaneous learning
of GMM model parameters and model complexity. Have you (or anyone else on
the SciPy list) ever looked at this?

Best regards,
Helge
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100221/9ba7311b/attachment.html>

From R.Springuel at umit.maine.edu  Sun Feb 21 12:01:20 2010
From: R.Springuel at umit.maine.edu (R. Padraic Springuel)
Date: Sun, 21 Feb 2010 12:01:20 -0500
Subject: [SciPy-User] New Installation
Message-ID: <4B8166E0.2010105@umit.maine.edu>

> Did you try to completely remove Python and then do a fresh install?  Whenever I
> try to do upgrades or additions I always have problems, so I like to just wipe
> everything out and install from scratch.

Originally no, but I'd tried that solution before emailing the list.

Anyway, I was just checking the numpy download site and noticed that 
there is no longer a download link for v1.4.  I figure this is because 
of some problems with that version?  So, I downloaded v1.3, installed 
it, reinstalled scipy and matplotlib and everything is working fine now.
-- 

R. Padraic Springuel
Research Assistant
Department of Physics and Astronomy
University of Maine
Bennett 309
Office Hours: Thursdays, 12-2pm


From meesters at gmx.de  Sun Feb 21 14:04:43 2010
From: meesters at gmx.de (Christian Meesters)
Date: Sun, 21 Feb 2010 20:04:43 +0100
Subject: [SciPy-User] [Swig-user] swig and OpenMP
In-Reply-To: <169ADDAA-F021-4494-912D-E0420F25846F@ient.rwth-aachen.de>
References: <1265746732.4279.6.camel@meesters>
	<169ADDAA-F021-4494-912D-E0420F25846F@ient.rwth-aachen.de>
Message-ID: <1266779083.3289.17.camel@meesters>

Hi,

Meanwhile I tried everything. I corrected the extra_compile_args,
tinkered with sys.setdlopenflags, even updated my entire system - its
not working. OpenMP and Python and gcc apparently don't work in my
hands. Perhaps Rohit is right with his comment on internal symbols - at
least I've learned a thing or two. 

Thank you all for trying to help. I'm still open for suggestions and
particularly interested when somebody finds a workaround.

Christian






From njs at pobox.com  Sun Feb 21 14:26:34 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 21 Feb 2010 11:26:34 -0800
Subject: [SciPy-User] scikits.sparse
In-Reply-To: <web-130558143@uni-stuttgart.de>
References: <web-130542846@uni-stuttgart.de>
	<427c90929b50c81849394c9d37217fc1.squirrel@webmail.uio.no>
	<web-130543259@uni-stuttgart.de>
	<961fa2b41002201007q76965b8dl940943dd018d43d@mail.gmail.com>
	<web-130558143@uni-stuttgart.de>
Message-ID: <961fa2b41002211126g21373eb7m70ec817c64cd06b4@mail.gmail.com>

On Sun, Feb 21, 2010 at 2:00 AM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> scikits/sparse/cholmod.c:158:31: error:
> numpy/arrayobject.h: Datei oder Verzeichnis nicht gefunden
> scikits/sparse/cholmod.c:159:31: error:
> numpy/ufuncobject.h: Datei oder Verzeichnis nicht gefunden

Oops, this is indeed a scikits.sparse bug. Thanks for the reminder,
actually -- it came up before, but I forgot to commit the fix for the
next release...

Anyway, the solution is to edit setup.py:
1) near the top, add the line:
  from numpy.distutils.misc_util import get_numpy_include_dirs

2) near the bottom, where it says:
              Extension("scikits.sparse.cholmod",
                        ["scikits/sparse/cholmod.pyx"],
                        libraries=["cholmod"],
                        ),
add an include_dirs line, like:
              Extension("scikits.sparse.cholmod",
                        ["scikits/sparse/cholmod.pyx"],
                        libraries=["cholmod"],
                        include_dirs=get_numpy_include_dirs(),
                        ),

This will be in the next release, of course.

-- Nathaniel


From nwagner at iam.uni-stuttgart.de  Sun Feb 21 14:32:51 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sun, 21 Feb 2010 20:32:51 +0100
Subject: [SciPy-User] scikits.sparse
In-Reply-To: <961fa2b41002211126g21373eb7m70ec817c64cd06b4@mail.gmail.com>
References: <web-130542846@uni-stuttgart.de>
	<427c90929b50c81849394c9d37217fc1.squirrel@webmail.uio.no>
	<web-130543259@uni-stuttgart.de>
	<961fa2b41002201007q76965b8dl940943dd018d43d@mail.gmail.com>
	<web-130558143@uni-stuttgart.de>
	<961fa2b41002211126g21373eb7m70ec817c64cd06b4@mail.gmail.com>
Message-ID: <web-130566157@uni-stuttgart.de>

On Sun, 21 Feb 2010 11:26:34 -0800
  Nathaniel Smith <njs at pobox.com> wrote:
> On Sun, Feb 21, 2010 at 2:00 AM, Nils Wagner
> <nwagner at iam.uni-stuttgart.de> wrote:
>> scikits/sparse/cholmod.c:158:31: error:
>> numpy/arrayobject.h: Datei oder Verzeichnis nicht 
>>gefunden
>> scikits/sparse/cholmod.c:159:31: error:
>> numpy/ufuncobject.h: Datei oder Verzeichnis nicht 
>>gefunden
> 
> Oops, this is indeed a scikits.sparse bug. Thanks for 
>the reminder,
> actually -- it came up before, but I forgot to commit 
>the fix for the
> next release...
> 
> Anyway, the solution is to edit setup.py:
> 1) near the top, add the line:
>  from numpy.distutils.misc_util import 
>get_numpy_include_dirs
> 
> 2) near the bottom, where it says:
>              Extension("scikits.sparse.cholmod",
>                        ["scikits/sparse/cholmod.pyx"],
>                        libraries=["cholmod"],
>                        ),
> add an include_dirs line, like:
>              Extension("scikits.sparse.cholmod",
>                        ["scikits/sparse/cholmod.pyx"],
>                        libraries=["cholmod"],
>                        include_dirs=get_numpy_include_dirs(),
>                        ),
> 
> This will be in the next release, of course.
> 
> -- Nathaniel

O.k.
here comes another issue

python setup.py install --prefix=$HOME/local
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/command/sdist.py:4: 
UserWarning: Module pkg_resources was already imported 
from 
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/pkg_resources.py, 
but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/command/sdist.py:4: 
UserWarning: Module setuptools was already imported from 
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/__init__.py, 
but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/setuptools/command/sdist.py:4: 
UserWarning: Module site was already imported from 
/usr/lib64/python2.6/site.pyc, but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
/home/nwagner/local/lib64/python2.6/site-packages/Pyrex/Compiler/Errors.py:17: 
DeprecationWarning: BaseException.message has been 
deprecated as of Python 2.6
   self.message = message
running install
running bdist_egg
running egg_info
writing requirements to 
scikits.sparse.egg-info/requires.txt
writing scikits.sparse.egg-info/PKG-INFO
writing namespace_packages to 
scikits.sparse.egg-info/namespace_packages.txt
writing top-level names to 
scikits.sparse.egg-info/top_level.txt
writing dependency_links to 
scikits.sparse.egg-info/dependency_links.txt
writing requirements to 
scikits.sparse.egg-info/requires.txt
writing scikits.sparse.egg-info/PKG-INFO
writing namespace_packages to 
scikits.sparse.egg-info/namespace_packages.txt
writing top-level names to 
scikits.sparse.egg-info/top_level.txt
writing dependency_links to 
scikits.sparse.egg-info/dependency_links.txt
reading manifest file 
'scikits.sparse.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*~' found 
anywhere in distribution
warning: no previously-included files matching '*.so' 
found anywhere in distribution
warning: no previously-included files matching '*.pyc' 
found anywhere in distribution
writing manifest file 
'scikits.sparse.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-2.6
creating build/lib.linux-x86_64-2.6/scikits
copying scikits/__init__.py -> 
build/lib.linux-x86_64-2.6/scikits
creating build/lib.linux-x86_64-2.6/scikits/sparse
copying scikits/sparse/__init__.py -> 
build/lib.linux-x86_64-2.6/scikits/sparse
copying scikits/sparse/test_cholmod.py -> 
build/lib.linux-x86_64-2.6/scikits/sparse
running build_ext
building 'scikits.sparse.cholmod' extension
C compiler: /usr/bin/gcc -fno-strict-aliasing -DNDEBUG 
-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 
-fstack-protector -funwind-tables 
-fasynchronous-unwind-tables -g -fwrapv -fPIC

creating build/temp.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/scikits
creating build/temp.linux-x86_64-2.6/scikits/sparse
compile options: 
'-I/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/include 
-I/usr/include/python2.6 -c'
gcc: scikits/sparse/cholmod.c
scikits/sparse/cholmod.c:160:33: error: 
suitesparse/cholmod.h: Datei oder Verzeichnis nicht 
gefunden


  locate cholmod.h
/home/nwagner/hg/scikits.sparse/doc/_build/html/cholmod.html
/home/nwagner/src/CHOLMOD/Include/cholmod.h
/home/nwagner/src/cvxopt-1.1/src/C/SuiteSparse/CHOLMOD/Include/cholmod.h
/home/nwagner/src/SuiteSparse/CHOLMOD/Include/cholmod.h
/home/nwagner/src/SuiteSparse/KLU/User/klu_cholmod.h
/home/nwagner/src/trilinos-9.0.2/packages/amesos/src/SuiteSparse/CHOLMOD/Include/amesos_cholmod.h

Any idea ?

  Cheers,
                        Nils



From warren.weckesser at enthought.com  Sun Feb 21 16:00:11 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sun, 21 Feb 2010 15:00:11 -0600
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <1cd32cbb1002141843u261a7acbx5951e0374d0b5d48@mail.gmail.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>	<1cd32cbb1001281757x14b7b581xba02f4ae21d3acfb@mail.gmail.com>	<4B6256F5.3050802@enthought.com>	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>	<c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>	<1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>	<4B71A6BD.9090002@enthought.com>	<c5b438121002110721g66eda059ve86d199f7beed0fd@mail.gmail.com>	<4B789D3D.1060506@enthought.com>
	<1cd32cbb1002141843u261a7acbx5951e0374d0b5d48@mail.gmail.com>
Message-ID: <4B819EDB.90605@enthought.com>

josef.pktd at gmail.com wrote:
> On Sun, Feb 14, 2010 at 8:02 PM, Warren Weckesser
> <warren.weckesser at enthought.com> wrote:
>   
>> I added a patch to ticket #1112.  In addition to adding the suggested
>> change to pass keyword args to odeint, I also added a new funciton,
>> impulse2(), that computes the impulse response by using odeint.  See
>> this thread for details:
>>
>>    http://mail.scipy.org/pipermail/scipy-user/2009-November/023416.html
>>     
>
> Thanks, the test examples look nice and serve also as some
> documentation for ltisys. The lsim2 test05 is interesting, I was
> convinced last year (and there were some previous threads) that ltisys
> cannot handle multi-input systems. I always got shape errors when I
> tried and without documentation it's difficult to figure out what is
> supposed to work and what not. (I will look at it more closely when I
> have more time.)
>   

I'd like to look into this, too.  I don't think it was originally 
designed for MIMO systems.

> I think the changes are reasonable including turning u and t into
> keywords instead of required arguments. Maybe Ryan or someone who is
> using this functions can comment on this.
>
> Just one improvement to the tests, assert_almost_equal takes a decimal
> argument. If you know the precision of your tests, then it would be
> useful to increase it from the default decimal=7.
>   

I suspect the default precision is fine.  To actually know the expected 
precision, one has to be able to convert the absolute and relative 
tolerances used by the ODE solver, which are *local* error control 
parameters, into a global tolerance (i.e. what is the error at 
t=f_final).  This is a nontrivial problem.


Warren

> Josef
>
>
>   
>> Warren
>>
>>
>> Ryan Krauss wrote:
>>     
>>> Yeah, **kwds is a little harder to understand and slightly less user
>>> friendly, but it is easier to maintain and less work for the patch
>>> writer.  And it is more future proof if the integrator ever changes.
>>>
>>> On Tue, Feb 9, 2010 at 12:17 PM, Warren Weckesser
>>> <warren.weckesser at enthought.com> wrote:
>>>
>>>       
>>>> josef.pktd at gmail.com wrote:
>>>>
>>>>         
>>>>> On Wed, Feb 3, 2010 at 9:00 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>>>
>>>>>
>>>>>           
>>>>>> FYI, I am quite happy with passing in an hmax value.  I basically
>>>>>> copied and pasted lsim2 from signal.ltisys and adapted it just a
>>>>>> little to make it a method of my derived class.  Then I added the hmas
>>>>>> kwarg that gets passed to odeint.
>>>>>>
>>>>>> Is there any reason not to allow the user to pass in a kwargs to lsim2
>>>>>> that gets passed to odeint?
>>>>>>
>>>>>>
>>>>>>             
>>>>> I don't see a reason why we cannot add a **kwargs, it should be
>>>>> completely backwards compatible.
>>>>> Can you file a ticket and add your adjusted version or a patch? And
>>>>> even better, add your original example as a test case?
>>>>>
>>>>>
>>>>>
>>>>>           
>>>> Josef,
>>>>
>>>> I just created ticket #1112 for this.  Unless Ryan wants to adapt his
>>>> change to lsim2, I can make a patch this week to implement the enhancement.
>>>>
>>>> Warren
>>>>
>>>>
>>>>
>>>>         
>>>>> Josef
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> On Fri, Jan 29, 2010 at 6:44 AM, Ryan Krauss <ryanlists at gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> Thanks to Warren and Josef for their time and thoughts.  I feel like I
>>>>>>> now understand the underlying problem and have some good options to
>>>>>>> solve my short term issues (I assigned the project last night and they
>>>>>>> need to be able to start working on it immediately).  I actually use a
>>>>>>> TransferFunction class that derives from ltisys.  I could override its
>>>>>>> lsim2 method to try out some of these solutions quickly and fairly
>>>>>>> easily.
>>>>>>>
>>>>>>> Ryan
>>>>>>>
>>>>>>> On Thu, Jan 28, 2010 at 10:00 PM,  <josef.pktd at gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> On Thu, Jan 28, 2010 at 10:33 PM, Warren Weckesser
>>>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> josef.pktd at gmail.com wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> On Thu, Jan 28, 2010 at 8:50 PM, Warren Weckesser
>>>>>>>>>> <warren.weckesser at enthought.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> Ryan,
>>>>>>>>>>>
>>>>>>>>>>> The problem is that the ODE solver used by lsim2 is too good. :)
>>>>>>>>>>>
>>>>>>>>>>> It uses scipy.integrate.odeint, which in turn uses the Fortran library
>>>>>>>>>>> LSODA.  Like any good solver, LSODA is an adaptive solver--it adjusts its
>>>>>>>>>>> step size to be as large as possible while keeping estimates of the error
>>>>>>>>>>> bounded.  For the problem you are solving, with initial condition 0, the
>>>>>>>>>>> exact solution is initially exactly 0.  This is such a nice smooth solution
>>>>>>>>>>> that the solver's step size quickly grows--so big, in fact, that it skips
>>>>>>>>>>> right over your pulse and never sees it.
>>>>>>>>>>>
>>>>>>>>>>> So how does it create all those intermediate points at the requested time
>>>>>>>>>>> values?  It uses interpolation between the steps that it computed to create
>>>>>>>>>>> the solution values at the times that you requested.  So using a finer grid
>>>>>>>>>>> of time values won't help.  (If lsim2 gave you a hook into the parameters
>>>>>>>>>>> passed to odeint, you could set odeint's 'hmax' to a value smaller than your
>>>>>>>>>>> pulse width, which would force the solver to see the pulse.  But there is no
>>>>>>>>>>> way to set that parameter from lsim2.)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>> It's something what I suspected. I don't know much about odeint, but
>>>>>>>>>> do you think it would be useful to let lsim2 pass through some
>>>>>>>>>> parameters to odeint?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>> Sounds useful to me.  A simple implementation is an optional keyword
>>>>>>>>> argument that is a dict of odeint arguments.   But this would almost
>>>>>>>>> certainly break if lsim2 were ever reimplemented with a different
>>>>>>>>> solver.  So perhaps it should allow a common set of ODE solver
>>>>>>>>> parameters (e.g. absolute and relative error tolerances, max and min
>>>>>>>>> step sizes, others?).
>>>>>>>>>
>>>>>>>>> Perhaps this should wait until after the ODE solver redesign that is
>>>>>>>>> occasionally discussed:
>>>>>>>>>    http://projects.scipy.org/scipy/wiki/OdeintRedesign
>>>>>>>>> Then the solver itself could be an optional argument to lsim2.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> I was just thinking of adding to the argument list a **kwds argument
>>>>>>>> that is directly passed on to whatever ODE solver is used. This should
>>>>>>>> be pretty flexible for any changes and be backwards compatible.
>>>>>>>>
>>>>>>>> I've seen and used it in a similar way for calls to optimization
>>>>>>>> routines, e.g. also optimize.curve_fit, does it. What are actually
>>>>>>>> valid keywords would depend on which function is called.
>>>>>>>>
>>>>>>>> (But I'm not a user of lsim, I'm just stealing some ideas from lti and
>>>>>>>> friends for time series analysis.)
>>>>>>>>
>>>>>>>> Josef
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> Warren
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> Josef
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> The basic problem is you are passing in a discontinuous function to a solver
>>>>>>>>>>> that expects a smooth function.  A better way to solve this problem is to
>>>>>>>>>>> explicitly account for the discontinuity. One possibility is the attached
>>>>>>>>>>> script.
>>>>>>>>>>>
>>>>>>>>>>> This is an excellent "learning opportunity" for your students on the hazards
>>>>>>>>>>> of numerical computing!
>>>>>>>>>>>
>>>>>>>>>>> Warren
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ryan Krauss wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>>>> I believe I have discovered a bug in signal.lsim2.  I believe the
>>>>>>>>>>>> short attached script illustrates the problem.  I was trying to
>>>>>>>>>>>> predict the response of a transfer function with a pure integrator:
>>>>>>>>>>>>
>>>>>>>>>>>>             g
>>>>>>>>>>>> G = -------------
>>>>>>>>>>>>         s(s+p)
>>>>>>>>>>>>
>>>>>>>>>>>> to a finite width pulse.  lsim2 seems to handle the step response just
>>>>>>>>>>>> fine, but says that the pulse response is exactly 0.0 for the entire
>>>>>>>>>>>> time of the simulation.  Obviously, this isn't the right answer.
>>>>>>>>>>>>
>>>>>>>>>>>> I am running scipy 0.7.0 and numpy 1.2.1 on Ubuntu 9.04, but I also
>>>>>>>>>>>> have the same problem on Windows running 0.7.1 and 1.4.0.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>>  ------------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> SciPy-User mailing list
>>>>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                         
>>>>>>>>>>> from pylab import *
>>>>>>>>>>> from scipy import signal
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> g = 100.0
>>>>>>>>>>> p = 15.0
>>>>>>>>>>> G = signal.ltisys.lti(g, [1,p,0])
>>>>>>>>>>>
>>>>>>>>>>> t = arange(0, 1.0, 0.002)
>>>>>>>>>>> N = len(t)
>>>>>>>>>>>
>>>>>>>>>>> # u for the whole interval (not used in lsim2, only for plotting later).
>>>>>>>>>>> amp = 50.0
>>>>>>>>>>> u = zeros(N)
>>>>>>>>>>> k1 = 50
>>>>>>>>>>> k2 = 100
>>>>>>>>>>> u[k1:k2] = amp
>>>>>>>>>>>
>>>>>>>>>>> # Create input functions for each smooth interval. (This could be simpler,
>>>>>>>>>>> since u
>>>>>>>>>>> # is constant on each interval.)
>>>>>>>>>>> a = float(k1)/N
>>>>>>>>>>> b = float(k2)/N
>>>>>>>>>>> T1 = linspace(0, a, 201)
>>>>>>>>>>> u1 = zeros_like(T1)
>>>>>>>>>>> T2 = linspace(a, b, 201)
>>>>>>>>>>> u2 = amp*ones_like(T2)
>>>>>>>>>>> T3 = linspace(b, 1.0, 201)
>>>>>>>>>>> u3 = zeros_like(T3)
>>>>>>>>>>>
>>>>>>>>>>> # Solve on each interval; use the final value of one solution as the
>>>>>>>>>>> starting
>>>>>>>>>>> # point of the next solution.
>>>>>>>>>>> # (We could skip the first calculation, since we know the solution will be
>>>>>>>>>>> 0.)
>>>>>>>>>>> (t1, y1, x1) = signal.lsim2(G,u1,T1)
>>>>>>>>>>> (t2, y2, x2) = signal.lsim2(G, u2, T2, X0=x1[-1])
>>>>>>>>>>> (t3, y3, x3) = signal.lsim2(G, u3, T3, X0=x2[-1])
>>>>>>>>>>>
>>>>>>>>>>> figure(1)
>>>>>>>>>>> clf()
>>>>>>>>>>> plot(t, u, 'k', linewidth=3)
>>>>>>>>>>> plot(t1, y1, 'y', linewidth=3)
>>>>>>>>>>> plot(t2, y2, 'b', linewidth=3)
>>>>>>>>>>> plot(t3, y3, 'g', linewidth=3)
>>>>>>>>>>>
>>>>>>>>>>> show()
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> SciPy-User mailing list
>>>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>> _______________________________________________
>>>>>>>>>> SciPy-User mailing list
>>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>> _______________________________________________
>>>>>>>>> SciPy-User mailing list
>>>>>>>>> SciPy-User at scipy.org
>>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> _______________________________________________
>>>>>>>> SciPy-User mailing list
>>>>>>>> SciPy-User at scipy.org
>>>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>>>>           
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>>         
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>       
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>     
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From warren.weckesser at enthought.com  Sun Feb 21 16:03:08 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sun, 21 Feb 2010 15:03:08 -0600
Subject: [SciPy-User] bug in signal.lsim2
In-Reply-To: <1cd32cbb1002141843u261a7acbx5951e0374d0b5d48@mail.gmail.com>
References: <c5b438121001281238r19c6bda0t7051c1d41f267b3e@mail.gmail.com>	<1cd32cbb1001281757x14b7b581xba02f4ae21d3acfb@mail.gmail.com>	<4B6256F5.3050802@enthought.com>	<1cd32cbb1001282000x332df230n3242e7e1063ee303@mail.gmail.com>	<c5b438121001290444o7a085cbdm819e4f32843babee@mail.gmail.com>	<c5b438121002030600k4fac99d7m65427f55ff97ffbf@mail.gmail.com>	<1cd32cbb1002030616x95bc910q7326673a51783c8d@mail.gmail.com>	<4B71A6BD.9090002@enthought.com>	<c5b438121002110721g66eda059ve86d199f7beed0fd@mail.gmail.com>	<4B789D3D.1060506@enthought.com>
	<1cd32cbb1002141843u261a7acbx5951e0374d0b5d48@mail.gmail.com>
Message-ID: <4B819F8C.9020206@enthought.com>

josef.pktd at gmail.com wrote:
> On Sun, Feb 14, 2010 at 8:02 PM, Warren Weckesser
> <warren.weckesser at enthought.com> wrote:
>   
>> I added a patch to ticket #1112.  In addition to adding the suggested
>> change to pass keyword args to odeint, I also added a new funciton,
>> impulse2(), that computes the impulse response by using odeint.  See
>> this thread for details:
>>
>>    http://mail.scipy.org/pipermail/scipy-user/2009-November/023416.html
>>     
>
> Thanks, the test examples look nice and serve also as some
> documentation for ltisys. The lsim2 test05 is interesting, I was
> convinced last year (and there were some previous threads) that ltisys
> cannot handle multi-input systems. I always got shape errors when I
> tried and without documentation it's difficult to figure out what is
> supposed to work and what not. (I will look at it more closely when I
> have more time.)
>   

I'd like to look into this, too.  I don't think it was originally 
designed for MIMO systems.

> I think the changes are reasonable including turning u and t into
> keywords instead of required arguments. Maybe Ryan or someone who is
> using this functions can comment on this.
>
> Just one improvement to the tests, assert_almost_equal takes a decimal
> argument. If you know the precision of your tests, then it would be
> useful to increase it from the default decimal=7.
>   

I suspect the default precision is fine.  To actually know the expected 
precision, one has to be able to convert the absolute and relative 
tolerances used by the ODE solver, which are *local* error control 
parameters, into a global tolerance (i.e. what is the error at 
t=f_final).  This is a nontrivial problem.


Warren



From gokhansever at gmail.com  Sun Feb 21 17:00:30 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Sun, 21 Feb 2010 16:00:30 -0600
Subject: [SciPy-User] ask.scipy.org
Message-ID: <49d6b3501002211400r43cbcc01y146c179e880851bf@mail.gmail.com>

Hello,

Since after Robert Kern showed http://advice.mechanicalkern.com/ on SciPy09
there are many similar initiatives that uses stackoverflow.com (SO) layout.
Some smart guys come up with this site http://stackexchange.com/ to those
who want to have a simple but a paid solution.

I don't have an intention of creating controversial discussion. It just to
my eyes SO has a very appealing and easy to use interface and it's getting
some number of posts related to scientific Python tools. I usually suggest
my friends to use the mailing lists first and SO for their questions. Some
prefer mailing lists some not. Mailing lists require more steps to get in
however SO register step is much easier due to OpenID logging.

Without belabouring further, It would be good to link R. Kern's advice site
to either ask.scipy or advice.scipy or another alternative to attract
new-comers easily. I am more in favor of the ask.scipy.org option. Thus I
can refer the people (hear I mean mostly non-programmers or
students/programmers without Python experience), simply to go
ask.scipy.orgfor their first questions instead of telling them to
search answers at many
different mediums.

What do you think?

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100221/798a5816/attachment.html>

From denis-bz-gg at t-online.de  Mon Feb 22 07:35:35 2010
From: denis-bz-gg at t-online.de (denis)
Date: Mon, 22 Feb 2010 04:35:35 -0800 (PST)
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com> 
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com> 
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com>
Message-ID: <b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com>

On Feb 19, 5:41?pm, josef.p... at gmail.com wrote:
> On Fri, Feb 19, 2010 at 11:26 AM, denis <denis-bz... at t-online.de> wrote:

> > Use "smooth" ? rbf.py just does
> > ? ?self.A = self._function(r) - eye(self.N)*self.smooth
> > and you don't know A .

That's a line from scipy/interpolate/rbf.py: it solves
    (A - smooth*I)x = b  instead of
    Ax = b
Looks to me like a hack for A singular, plus the caller doesn't know A
anyway.

I had looked at the eigenvalues too (100 points, 2d, like
test_rbf.py):
gauss     : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1.2e-10 ... 44
linear    : -1.8e+02 ... -0.027 5.6e+02
thinplate : -4.7e+03 ... -5.4e+02 0.0032

So gauss is singular => don't do that then.
(Odd that linalg.solve didn't LinAlgError though.)

> rbf is a global interpolator, and as Robert said for some cases you
> get very unstable results.
>
> with a large number of points self._function(r) ?is not very well
> behaved and you need the penalization to make it smoother. In a
> variation of your example, I printed out the eigenvalues for self.A
> and they don't look nice without penalization.
> There are still some strange things that I don't understand with the
> behavior rbf, but when I looked at it in the past, I got better
> results by using only local information, i.e. fit rbf only to a
> neighborhood of the points, although I think thinplate did pretty well
> also with a larger number of points.

Yes -- but you don't know which points are local
without some k-nearest-neighbor algorithm ? in 2d, might as well
triangulate.
Gaussian weights nearer points automatically BUT rbf gauss looks to me
singular / unusable.

cheers
  -- denis


From josef.pktd at gmail.com  Mon Feb 22 09:08:35 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 22 Feb 2010 09:08:35 -0500
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com>
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com>
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com>
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com>
Message-ID: <1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com>

On Mon, Feb 22, 2010 at 7:35 AM, denis <denis-bz-gg at t-online.de> wrote:
> On Feb 19, 5:41?pm, josef.p... at gmail.com wrote:
>> On Fri, Feb 19, 2010 at 11:26 AM, denis <denis-bz... at t-online.de> wrote:
>
>> > Use "smooth" ? rbf.py just does
>> > ? ?self.A = self._function(r) - eye(self.N)*self.smooth
>> > and you don't know A .
>
> That's a line from scipy/interpolate/rbf.py: it solves
> ? ?(A - smooth*I)x = b ?instead of
> ? ?Ax = b
> Looks to me like a hack for A singular, plus the caller doesn't know A
> anyway.

It's not a hack it's a requirement, ill-posed inverse problems need
penalization, this is just Ridge or Tychonov with a kernel matrix. A
is (nobs,nobs) and the number of features is always the same as the
number of observations that are used. (I was looking at "Kernel Ridge
Regression" and "Gaussian Process" before I realized that rbf is
essentially the same, at least for 'gauss')
I don't know anything about thinplate.

I still don't understand what you mean with "the caller doesn't know
A".  A is the internally calculated kernel matrix (if I remember
correctly.)


>
> I had looked at the eigenvalues too (100 points, 2d, like
> test_rbf.py):
> gauss ? ? : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 1.2e-10 ... 44
> linear ? ?: -1.8e+02 ... -0.027 5.6e+02
> thinplate : -4.7e+03 ... -5.4e+02 0.0032
>
> So gauss is singular => don't do that then.
> (Odd that linalg.solve didn't LinAlgError though.)
>
>> rbf is a global interpolator, and as Robert said for some cases you
>> get very unstable results.
>>
>> with a large number of points self._function(r) ?is not very well
>> behaved and you need the penalization to make it smoother. In a
>> variation of your example, I printed out the eigenvalues for self.A
>> and they don't look nice without penalization.
>> There are still some strange things that I don't understand with the
>> behavior rbf, but when I looked at it in the past, I got better
>> results by using only local information, i.e. fit rbf only to a
>> neighborhood of the points, although I think thinplate did pretty well
>> also with a larger number of points.
>
> Yes -- but you don't know which points are local
> without some k-nearest-neighbor algorithm ? in 2d, might as well
> triangulate.

I used scipy.spatial for this, or if there are not too many points use
the full dense distance matrix setting far away points to zero (kind
of). My example script should be on the mailing list.
rbf works for nd not just 1d or 2d, I think only the number of
observations is important, not the number of features.

> Gaussian weights nearer points automatically BUT rbf gauss looks to me
> singular / unusable.

unusable without large enough smooth>0 , e.g. I tried smooth=0.4 in bad cases.

Cheers,

Josef

>
> cheers
> ?-- denis
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From denis-bz-gg at t-online.de  Mon Feb 22 11:14:23 2010
From: denis-bz-gg at t-online.de (denis)
Date: Mon, 22 Feb 2010 08:14:23 -0800 (PST)
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com> 
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com> 
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com> 
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com> 
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com>
Message-ID: <43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com>

On Feb 22, 3:08?pm, josef.p... at gmail.com wrote:
> On Mon, Feb 22, 2010 at 7:35 AM, denis <denis-bz... at t-online.de> wrote:
> > On Feb 19, 5:41?pm, josef.p... at gmail.com wrote:
> >> On Fri, Feb 19, 2010 at 11:26 AM, denis <denis-bz... at t-online.de> wrote:
>
> >> > Use "smooth" ? rbf.py just does
> >> > ? ?self.A = self._function(r) - eye(self.N)*self.smooth
> >> > and you don't know A .
>
> > That's a line from scipy/interpolate/rbf.py: it solves
> > ? ?(A - smooth*I)x = b ?instead of
> > ? ?Ax = b
> > Looks to me like a hack for A singular, plus the caller doesn't know A
> > anyway.

> It's not a hack it's a requirement, ill-posed inverse problems need

OK, I must be wrong; but (sorry, I'm ignorant) how can (A - smooth)
penalize ?
For gauss the eigenvalues are >= 0, many 0, so we're shifting them
negative ??
Or is it a simple sign error, A + smooth ?

> penalization, this is just Ridge or Tychonov with a kernel matrix. A
> is (nobs,nobs) and the number of features is always the same as the
> number of observations that are used. (I was looking at "Kernel Ridge
> Regression" and "Gaussian Process" before I realized that rbf is
> essentially the same, at least for 'gauss')
> I don't know anything about thinplate.
>
> I still don't understand what you mean with "the caller doesn't know
> A". ?A is the internally calculated kernel matrix (if I remember
> correctly.)

Yes that's right; how can the caller of Rbf() give a reasonable value
of "smooth"
to solve (A - smoothI) inside Rbf, without knowing A ?  A is wildly
different for gauss, linear ... too.
Or do you just shut your eyes and try 1e-6  ?

Thanks Josef,
cheers
  -- denis


From robert.kern at gmail.com  Mon Feb 22 11:23:22 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 22 Feb 2010 10:23:22 -0600
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com> 
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com> 
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com> 
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com> 
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com> 
	<43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com>
Message-ID: <3d375d731002220823s3f737d74l7d8a0008feef5881@mail.gmail.com>

On Mon, Feb 22, 2010 at 10:14, denis <denis-bz-gg at t-online.de> wrote:
> On Feb 22, 3:08?pm, josef.p... at gmail.com wrote:

>> I still don't understand what you mean with "the caller doesn't know
>> A". ?A is the internally calculated kernel matrix (if I remember
>> correctly.)
>
> Yes that's right; how can the caller of Rbf() give a reasonable value
> of "smooth"
> to solve (A - smoothI) inside Rbf, without knowing A ? ?A is wildly
> different for gauss, linear ... too.

Ah! *That's* what you meant.

> Or do you just shut your eyes and try 1e-6 ??

Basically, yes. You can try different values while watching the
residuals if you want a more rigorous approach. But it really does
work. Try using 'multiquadric' and smooth=0.1.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Mon Feb 22 11:45:12 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 22 Feb 2010 11:45:12 -0500
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com>
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com>
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com>
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com>
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com>
	<43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com>
Message-ID: <1cd32cbb1002220845o3ad94ba8g599f25ec170c6106@mail.gmail.com>

On Mon, Feb 22, 2010 at 11:14 AM, denis <denis-bz-gg at t-online.de> wrote:
> On Feb 22, 3:08?pm, josef.p... at gmail.com wrote:
>> On Mon, Feb 22, 2010 at 7:35 AM, denis <denis-bz... at t-online.de> wrote:
>> > On Feb 19, 5:41?pm, josef.p... at gmail.com wrote:
>> >> On Fri, Feb 19, 2010 at 11:26 AM, denis <denis-bz... at t-online.de> wrote:
>>
>> >> > Use "smooth" ? rbf.py just does
>> >> > ? ?self.A = self._function(r) - eye(self.N)*self.smooth
>> >> > and you don't know A .
>>
>> > That's a line from scipy/interpolate/rbf.py: it solves
>> > ? ?(A - smooth*I)x = b ?instead of
>> > ? ?Ax = b
>> > Looks to me like a hack for A singular, plus the caller doesn't know A
>> > anyway.
>
>> It's not a hack it's a requirement, ill-posed inverse problems need
>
> OK, I must be wrong; but (sorry, I'm ignorant) how can (A - smooth)
> penalize ?
> For gauss the eigenvalues are >= 0, many 0, so we're shifting them
> negative ??
> Or is it a simple sign error, A + smooth ?

ouch, standard Ridge is A + smooth * identity_matrix to make it
positive definite.

I don't know why there is a minus. When I checked the eigenvalues, I
found it strange that there were some large *negative* eigenvalues of
A, but I didn't have time to figure this out.
Generically, A - smooth*eye would still make it invertible, although
not positive definite

I haven't looked at the sign convention in rbf, but if you figure out
what's going on, I'm very interested in an answer.

I just briefly ran an example with a negative smooth (-0.5 versus
0.5), rbf with gauss seems better, but multiquadric seems worse. If
smooth is small 1e-6, then there is not much difference.

Even with negative smooth, all except for gauss still have negative
eigenvalues. I have no idea, I only looked at the theory for gaussian
process and don't know how the other ones differ.


>
>> penalization, this is just Ridge or Tychonov with a kernel matrix. A
>> is (nobs,nobs) and the number of features is always the same as the
>> number of observations that are used. (I was looking at "Kernel Ridge
>> Regression" and "Gaussian Process" before I realized that rbf is
>> essentially the same, at least for 'gauss')
>> I don't know anything about thinplate.
>>
>> I still don't understand what you mean with "the caller doesn't know
>> A". ?A is the internally calculated kernel matrix (if I remember
>> correctly.)
>
> Yes that's right; how can the caller of Rbf() give a reasonable value
> of "smooth"
> to solve (A - smoothI) inside Rbf, without knowing A ? ?A is wildly
> different for gauss, linear ... too.
> Or do you just shut your eyes and try 1e-6 ??

That's the usual problem of bandwidth selection for non-parametric
estimation, visual inspection, cross-validation, plug-in, ... I don't
know what's recommended for rbf.

Cheers,

Josef

>
> Thanks Josef,
> cheers
> ?-- denis
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Mon Feb 22 12:27:42 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 22 Feb 2010 12:27:42 -0500
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <1cd32cbb1002220845o3ad94ba8g599f25ec170c6106@mail.gmail.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com>
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com>
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com>
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com>
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com>
	<43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com>
	<1cd32cbb1002220845o3ad94ba8g599f25ec170c6106@mail.gmail.com>
Message-ID: <1cd32cbb1002220927h229e578bhe18737d803fddc76@mail.gmail.com>

On Mon, Feb 22, 2010 at 11:45 AM,  <josef.pktd at gmail.com> wrote:
> On Mon, Feb 22, 2010 at 11:14 AM, denis <denis-bz-gg at t-online.de> wrote:
>> On Feb 22, 3:08?pm, josef.p... at gmail.com wrote:
>>> On Mon, Feb 22, 2010 at 7:35 AM, denis <denis-bz... at t-online.de> wrote:
>>> > On Feb 19, 5:41?pm, josef.p... at gmail.com wrote:
>>> >> On Fri, Feb 19, 2010 at 11:26 AM, denis <denis-bz... at t-online.de> wrote:
>>>
>>> >> > Use "smooth" ? rbf.py just does
>>> >> > ? ?self.A = self._function(r) - eye(self.N)*self.smooth
>>> >> > and you don't know A .
>>>
>>> > That's a line from scipy/interpolate/rbf.py: it solves
>>> > ? ?(A - smooth*I)x = b ?instead of
>>> > ? ?Ax = b
>>> > Looks to me like a hack for A singular, plus the caller doesn't know A
>>> > anyway.
>>
>>> It's not a hack it's a requirement, ill-posed inverse problems need
>>
>> OK, I must be wrong; but (sorry, I'm ignorant) how can (A - smooth)
>> penalize ?
>> For gauss the eigenvalues are >= 0, many 0, so we're shifting them
>> negative ??
>> Or is it a simple sign error, A + smooth ?
>
> ouch, standard Ridge is A + smooth * identity_matrix to make it
> positive definite.
>
> I don't know why there is a minus. When I checked the eigenvalues, I
> found it strange that there were some large *negative* eigenvalues of
> A, but I didn't have time to figure this out.
> Generically, A - smooth*eye would still make it invertible, although
> not positive definite
>
> I haven't looked at the sign convention in rbf, but if you figure out
> what's going on, I'm very interested in an answer.
>
> I just briefly ran an example with a negative smooth (-0.5 versus
> 0.5), rbf with gauss seems better, but multiquadric seems worse. If
> smooth is small 1e-6, then there is not much difference.
>
> Even with negative smooth, all except for gauss still have negative
> eigenvalues. I have no idea, I only looked at the theory for gaussian
> process and don't know how the other ones differ.

My intuition for gaussan might not be correct for the other rbfs,
gauss is decreasing in distance, all others are increasing in distance

>>> for func in Rbfuncs: print func, Rbf( x, y, function=func ,smooth=smooth)._function(np.arange(5))

gaussian [ 1.      0.9272  0.739   0.5063  0.2982]
linear [0 1 2 3 4]
thin-plate [  0.       0.       2.7726   9.8875  22.1807]
multiquadric [ 1.      1.0371  1.1413  1.2964  1.4866]
quintic [   0    1   32  243 1024]
cubic [ 0  1  8 27 64]

and the distance matrix itself has negative eigenvalues

Josef

>
>
>>
>>> penalization, this is just Ridge or Tychonov with a kernel matrix. A
>>> is (nobs,nobs) and the number of features is always the same as the
>>> number of observations that are used. (I was looking at "Kernel Ridge
>>> Regression" and "Gaussian Process" before I realized that rbf is
>>> essentially the same, at least for 'gauss')
>>> I don't know anything about thinplate.
>>>
>>> I still don't understand what you mean with "the caller doesn't know
>>> A". ?A is the internally calculated kernel matrix (if I remember
>>> correctly.)
>>
>> Yes that's right; how can the caller of Rbf() give a reasonable value
>> of "smooth"
>> to solve (A - smoothI) inside Rbf, without knowing A ? ?A is wildly
>> different for gauss, linear ... too.
>> Or do you just shut your eyes and try 1e-6 ??
>
> That's the usual problem of bandwidth selection for non-parametric
> estimation, visual inspection, cross-validation, plug-in, ... I don't
> know what's recommended for rbf.
>
> Cheers,
>
> Josef
>
>>
>> Thanks Josef,
>> cheers
>> ?-- denis
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From jjstickel at vcn.com  Mon Feb 22 13:00:52 2010
From: jjstickel at vcn.com (Jonathan Stickel)
Date: Mon, 22 Feb 2010 11:00:52 -0700
Subject: [SciPy-User] data smoothing by regularization
Message-ID: <4B82C654.6070407@vcn.com>

There has been some recent discussions on the list about interpolation 
and smoothing.  I have recently implemented a regularization method to 
smooth 1-dimensional (y vs. x) data.  Although the method is analogous 
to spline-based smoothing that already exists in Scipy and my code is 
currently limited to 1D, I have implemented some useful features, 
specifically generalized cross-validation for determining the 
regularization parameter and the ability to add constraints to the 
smooth result.  Would there be enough interest for me to contribute my 
code to Scipy?  How would I go about doing so?  Please know that I am 
new to the scipy community.

Thanks,
Jonathan


P.S.
For further reading, see:

http://dx.doi.org/10.1016/j.compchemeng.2009.10.007

If you do not have access to the journal, you may email me offlist.


From mohitiiit at gmail.com  Mon Feb 22 15:56:13 2010
From: mohitiiit at gmail.com (Mohit Goyal)
Date: Tue, 23 Feb 2010 02:26:13 +0530
Subject: [SciPy-User] Reg: google summer of code 2010
Message-ID: <9e78df6d1002221256g7942cc4bu95d2cf2cc45db61f@mail.gmail.com>

I am a masters student doing research in data engineering. I have done
projects involving web development and data management in python.
           I want to contribute to scipy via gsoc 2010. I am  interested in
working on data related projects in scipy like improving  data sources and
integrate it into all the numpy/scipy , adding data sets and examples to
scipy, data analysis etc.

Please suggest  me how to proceed.

Thank You.



-- 
The Game is Not Over Because i haven't Won Yet.
-Mohit Goyal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100223/3d4e6ed1/attachment.html>

From jsseabold at gmail.com  Mon Feb 22 16:07:16 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Mon, 22 Feb 2010 16:07:16 -0500
Subject: [SciPy-User] Reg: google summer of code 2010
In-Reply-To: <9e78df6d1002221256g7942cc4bu95d2cf2cc45db61f@mail.gmail.com>
References: <9e78df6d1002221256g7942cc4bu95d2cf2cc45db61f@mail.gmail.com>
Message-ID: <c048da1c1002221307y31404c62v4ac593f8d13ddb61@mail.gmail.com>

On Mon, Feb 22, 2010 at 3:56 PM, Mohit Goyal <mohitiiit at gmail.com> wrote:
> I am a masters student doing research in data engineering. I have done
> projects involving web development and data management in python.
> ?????????? I want to contribute to scipy via gsoc 2010. I am? interested in
> working on data related projects in scipy like improving? data sources and
> integrate it into all the numpy/scipy , adding data sets and examples to
> scipy, data analysis etc.
>
> Please suggest? me how to proceed.
>

Have a look at the replies to your last e-mail to the dev list on Feb.
17th here: <http://thread.gmane.org/gmane.comp.python.scientific.devel/12827>.
 These are some good suggestions.  In answer to the issues raised in
the thread, you should clarify what you're referring to, ie., would
you want to work one the machine learning scikit or the data I/O?  If
it's the datasource project, I would also search the mailing list for
discussions about this over the last year or so.

It's good to go ahead and start getting together a proposal and
application.  People on the mailing lists will be very helpful in
giving feedback and guiding you in the right direction.

One thing I would like to see is some more work on David C.'s dataset
proposal (that we have been following with a few changes for the
statsmodels scikit).  Is this still accurate of the status?
<http://www.scipy.org/scipy/scikits/wiki/MachineLearning#Datasetsconcept>

Cheers,

Skipper


From chris.jesse at flightdataservices.com  Tue Feb 23 04:35:08 2010
From: chris.jesse at flightdataservices.com (Chris Jesse)
Date: Tue, 23 Feb 2010 09:35:08 +0000 (GMT)
Subject: [SciPy-User] TimeSeries and Milliseconds for aircraft data
In-Reply-To: <3848801.16841265888539566.JavaMail.root@mohawk.flightdataservices.com>
Message-ID: <24755476.01266917706841.JavaMail.chris@jesse1c-laptop>


Hi All, 

I've got lots of timeseries data from aircraft sensors (such as altitude, temperatures, landing gear up/down, airspeed..) which I was hoping to load into a timeseries array. However, I've just read that it only supports time resolutions down to a second. Is there a solution for my data which contains samples from hundreds of different parameters at different frequencies from 8Hz to 1/64Hz over an 8 hour period? 

I need to be able to interpolate between values, mask values when samples are corrupt / outside of operational boundaries and store values as accurately to the nearest 1/16th of a second if possible. 

Your help is much appreciated! Thanks, 

Chris 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100223/49f0b07d/attachment.html>

From lesserwhirls at gmail.com  Tue Feb 23 06:08:39 2010
From: lesserwhirls at gmail.com (Sean Arms)
Date: Tue, 23 Feb 2010 05:08:39 -0600
Subject: [SciPy-User] TimeSeries and Milliseconds for aircraft data
In-Reply-To: <24755476.01266917706841.JavaMail.chris@jesse1c-laptop>
References: <3848801.16841265888539566.JavaMail.root@mohawk.flightdataservices.com>
	<24755476.01266917706841.JavaMail.chris@jesse1c-laptop>
Message-ID: <b45a83781002230308i183683aja34f3c3ce4c5b191@mail.gmail.com>

On Tue, Feb 23, 2010 at 3:35 AM, Chris Jesse
<chris.jesse at flightdataservices.com> wrote:
> Hi All,
>
> I've got lots of timeseries data from aircraft sensors (such as altitude,
> temperatures, landing gear up/down, airspeed..) which I was hoping to load
> into a timeseries array. However, I've just read that it only supports time
> resolutions down to a second. Is there a solution for my data which contains
> samples from hundreds of different parameters at different frequencies from
> 8Hz to 1/64Hz over an 8 hour period?
>
> I need to be able to interpolate between values, mask values when samples
> are corrupt / outside of operational boundaries and store values as
> accurately to the nearest 1/16th of a second if possible.
>
> Your help is much appreciated! Thanks,
>
> Chris
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>

Hi Chris,

    I analyze both 10 and 20 Hz sonic anemometer data regularly
without the use of a timeseries array (I assume you mean a TimeSeries
object from scikits.timeseries).  I've had a lot of success using
datetime objects from the standard datetime module that ships with
python to work with my time stamps, and numpy masked arrays to handle
the data.  The scikits.timeseries module does look nice though, but I
too run up against the limit on the sampling frequency.

Sean
University of Oklahoma
School of Meteorology
PhD Candidate


From peridot.faceted at gmail.com  Tue Feb 23 08:40:51 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 23 Feb 2010 08:40:51 -0500
Subject: [SciPy-User] TimeSeries and Milliseconds for aircraft data
In-Reply-To: <24755476.01266917706841.JavaMail.chris@jesse1c-laptop>
References: <3848801.16841265888539566.JavaMail.root@mohawk.flightdataservices.com>
	<24755476.01266917706841.JavaMail.chris@jesse1c-laptop>
Message-ID: <ce557a361002230540n3e33f3f5ne5c03e6143df219d@mail.gmail.com>

Hi,

For the kind of thing you're doing, is there any need for the
timeseries machinery? I'd just represent all my times in terms of
seconds since observation start, at which point you can just use the
ordinary numpy/masked array machinery.

In particular, the decision not to support milliseconds was made on
the explicit assumption that more complicated timing tasks - up to and
including pulsar timing, which deals with nanosecond measurements
spanning decades - would simply use numpy directly, rather than try to
make TimeSeries support everything anyone might ever want from it.

Anne

On 23 February 2010 04:35, Chris Jesse
<chris.jesse at flightdataservices.com> wrote:
> Hi All,
>
> I've got lots of timeseries data from aircraft sensors (such as altitude,
> temperatures, landing gear up/down, airspeed..) which I was hoping to load
> into a timeseries array. However, I've just read that it only supports time
> resolutions down to a second. Is there a solution for my data which contains
> samples from hundreds of different parameters at different frequencies from
> 8Hz to 1/64Hz over an 8 hour period?
>
> I need to be able to interpolate between values, mask values when samples
> are corrupt / outside of operational boundaries and store values as
> accurately to the nearest 1/16th of a second if possible.
>
> Your help is much appreciated! Thanks,
>
> Chris
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From gokhansever at gmail.com  Tue Feb 23 10:23:03 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Tue, 23 Feb 2010 09:23:03 -0600
Subject: [SciPy-User] TimeSeries and Milliseconds for aircraft data
In-Reply-To: <24755476.01266917706841.JavaMail.chris@jesse1c-laptop>
References: <3848801.16841265888539566.JavaMail.root@mohawk.flightdataservices.com>
	<24755476.01266917706841.JavaMail.chris@jesse1c-laptop>
Message-ID: <49d6b3501002230723x71ca6913y7aea5d2b1eecc7d0@mail.gmail.com>

On Tue, Feb 23, 2010 at 3:35 AM, Chris Jesse <
chris.jesse at flightdataservices.com> wrote:

> Hi All,
>
> I've got lots of timeseries data from aircraft sensors (such as altitude,
> temperatures, landing gear up/down, airspeed..) which I was hoping to load
> into a timeseries array. However, I've just read that it only supports time
> resolutions down to a second. Is there a solution for my data which contains
> samples from hundreds of different parameters at different frequencies from
> 8Hz to 1/64Hz over an 8 hour period?
>
> I need to be able to interpolate between values, mask values when samples
> are corrupt / outside of operational boundaries and store values as
> accurately to the nearest 1/16th of a second if possible.
>
> Your help is much appreciated! Thanks,
>
> Chris
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
Hi Chris,

Here at the University of North Dakota we work with aircraft measurements as
well. Similar to your case depends on the campaign [1] sometimes the number
of measured parameters reach over hundred. We use a QNX 4.25 based data
acquisition system called M300 where all the probe data are connected in.
M300 saves everything in a binary file which we later process using our
ADPAA package [2] It is mostly IDL coded and mainly developed by Dr. David
Delene [3] He is also the main responsible person for the aircraft
measurements/deployments in our department.

ADPAA produces files using a pre-determined ASCII data structure shown here
[4] The system supports frequencies up to 100 Hz but typically the highest
we sample is at 25 Hz. There might be exceptional cases where we might need
higher sample rates. However 1-4 Hz is sufficiently enough when sampling
clouds with cloud-aerosol microphysical instruments. (PCASP, FSSP, CCN) (I
am disregarding instruments like holographic particle detectors for the
simplicity of my response :) which we don't often get to fly with.)

Anyway after some advertisement back to your question: in our system all the
data are time-stamped with a main "Time [seconds]; UT seconds from midnight
on day aircraft flight started" value. It simplifies data analysis a lot in
my view where the Python comes into action actually. Once the ASCII files
are created (we also output to NetCDF containers but I find easy to work on
simple text files.) I read the data using NumPy and I use the MaskedArray
module to deal with the missing values in the data. I publish my code along
with data on the ccnworks [5] page where I mostly commit for my thesis using
ground and airborne measurements and scientific Python ecosystem. For
instance to read the ASCII data files (also known as NASA formatted) I use
the generic script [6] As a part of my thesis I am working on this script
[7] recently. I use the lab data [8] so far. In logn-fit.py I have a section
where I do masking on the data like:

mask02 = (pcasp.data['Time'] > 13800) & (pcasp.data['Time'] <= 14400)

This is one of the most important part of the code --to extract the right
portion of the required data. In this case I am interested in time periods
based on the supersaturation values from the DMT-CCNC instrument. The next
step will be to analyse the cloud base data. As far as I know there is no
easy solution for this. What do PCASP probe shows when the aircraft is
levelling below a cloud deck :) The only way I know is to mask manually and
most logically using the common time-stamp in all data files.

I remember Berkeley guys work on a project called NiPy but it is for
neuro-scientists mostly and not much design consideration in it for airborne
measurement scientists in mind :)

I hope extra information doesn't bother you and let me know your opinions.


[1] http://atmoswiki.aero.und.edu/atmos/citation/home
[2] http://sourceforge.net/projects/adpaa/
[3] http://aerosol.atmos.und.edu/
[4] http://aerosol.atmos.und.edu/ADPAA/fileheader.html
[5] http://code.google.com/p/ccnworks/
[6] http://code.google.com/p/ccnworks/source/browse/trunk/nasafile.py
[7]
http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/logn-fit.py
[8]
http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/20090225_034214/09_02_25_03_42_14.lab.clean

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100223/41c17e5c/attachment.html>

From stefan at sun.ac.za  Tue Feb 23 12:24:25 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 23 Feb 2010 19:24:25 +0200
Subject: [SciPy-User] SciPy2010 Call for Papers
Message-ID: <9457e7c81002230924h25acd387r9000201e5e5897bb@mail.gmail.com>

==========================
SciPy 2010 Call for Papers
==========================

SciPy 2010, the 9th Python in Science Conference, will be held from
June 28th - July 3rd, 2010 in Austin, Texas.

At this conference, novel applications and breakthroughs made in the
pursuit of science using Python are presented.  Attended by leading
figures from both academia and industry, it is an excellent
opportunity to experience the cutting edge of scientific software
development.

The conference is preceded by two days of paid tutorials, during which
community experts provide training on several scientific Python
packages.

We invite you to take part by submitting a talk abstract on the
conference website at:

http://conference.scipy.org

Talk/Paper Submission
=====================

We solicit talks and accompanying papers (either formal academic or
magazine-style articles) that discuss topics regarding scientific
computing using Python, including applications, teaching, development
and research.  Papers are included in the peer-reviewed conference
proceedings, published online.

Please note that submissions primarily aimed at the promotion of a
commercial product or service will not be considered.

Important dates for authors include:

 * 11 April: Talk abstracts due
 * 20 April: Notification of acceptance
 * 13 June: Papers due
 * 15 August: Publication of proceedings

Further detail will be made available on http://conference.scipy.org

Conference Dates
================

 * Friday, 10 May: Early registration ends
 * Monday-Tuesday, 28-29 June: Tutorials
 * Wednesday-Thursday, June 30-July 1: Conference
 * Friday-Saturday, July 2-3: Coding Sprints

Executive Committee
===================

 * Conference: Jarrod Millman & Eric Jones
 * Program: Stefan van der Walt & Ondrej Certik
 * Student Sponsorship: Travis Oliphant

For more information on Python, visit http://www.python.org.


From denis-bz-gg at t-online.de  Tue Feb 23 12:43:55 2010
From: denis-bz-gg at t-online.de (denis)
Date: Tue, 23 Feb 2010 09:43:55 -0800 (PST)
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <1cd32cbb1002220927h229e578bhe18737d803fddc76@mail.gmail.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com> 
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com> 
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com> 
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com> 
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com> 
	<43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com> 
	<1cd32cbb1002220845o3ad94ba8g599f25ec170c6106@mail.gmail.com> 
	<1cd32cbb1002220927h229e578bhe18737d803fddc76@mail.gmail.com>
Message-ID: <1c841a64-786d-4069-9f88-fcaae5a0d063@t42g2000vbt.googlegroups.com>

Robert, Josef,
  thanks much for taking the time to look at RBF some more.
Summary, correct me:
    A - smooth*I in rbf.py is a sign error (ticket ?)
    for gauss, start with A + 1e-6*I  to move eigenvalues away from 0
    others have pos/neg eigenvalues, don't need smooth.

Looking at Wikipedia Tikhonov (thanks Josef) reminded me
of lstsq_near on advice.mechanicalkern:
minimize |Ax-b| and w|x| together, i.e. Tikhonov with Gammamatrix = I.

But for RBF, why minimize |x| ?  don't we really want to minimize
    |Ax-b|2 + w2 xRx
where R is a roughness penalty ?
On a regular grid Rij ~ Laplacian smoother, not much like I.
Here I'm over my head;
any ideas for a q+d roughness matrix for scattered data ?
(just for fun -- for RBF we've reached the point of diminishing
returns).

cheers
  -- denis


From robert.kern at gmail.com  Tue Feb 23 12:46:59 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 23 Feb 2010 11:46:59 -0600
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <1c841a64-786d-4069-9f88-fcaae5a0d063@t42g2000vbt.googlegroups.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<3d375d731002180748t5c143accq98164d2a19edf471@mail.gmail.com> 
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com> 
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com> 
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com> 
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com> 
	<43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com> 
	<1cd32cbb1002220845o3ad94ba8g599f25ec170c6106@mail.gmail.com> 
	<1cd32cbb1002220927h229e578bhe18737d803fddc76@mail.gmail.com> 
	<1c841a64-786d-4069-9f88-fcaae5a0d063@t42g2000vbt.googlegroups.com>
Message-ID: <3d375d731002230946m75c7cacbk4a45e8e3db518d54@mail.gmail.com>

On Tue, Feb 23, 2010 at 11:43, denis <denis-bz-gg at t-online.de> wrote:
> Robert, Josef,
> ?thanks much for taking the time to look at RBF some more.
> Summary, correct me:
> ? ?A - smooth*I in rbf.py is a sign error (ticket ?)

Not necessarily. It seems to work well in at least some cases. Find a
reference that says otherwise if you want it changed.

> ? ?for gauss, start with A + 1e-6*I ?to move eigenvalues away from 0
> ? ?others have pos/neg eigenvalues, don't need smooth.

No. If you need a smoothed approximation to noisy data rather than
exact interpolation than you use the smooth keyword. Otherwise, you
don't.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Tue Feb 23 12:57:58 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 23 Feb 2010 12:57:58 -0500
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <3d375d731002230946m75c7cacbk4a45e8e3db518d54@mail.gmail.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<639b16b7-8c61-4249-bc7a-c825285a1397@c16g2000yqd.googlegroups.com>
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com>
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com>
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com>
	<43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com>
	<1cd32cbb1002220845o3ad94ba8g599f25ec170c6106@mail.gmail.com>
	<1cd32cbb1002220927h229e578bhe18737d803fddc76@mail.gmail.com>
	<1c841a64-786d-4069-9f88-fcaae5a0d063@t42g2000vbt.googlegroups.com>
	<3d375d731002230946m75c7cacbk4a45e8e3db518d54@mail.gmail.com>
Message-ID: <1cd32cbb1002230957w35b7119ay7270bbbc1892ce25@mail.gmail.com>

On Tue, Feb 23, 2010 at 12:46 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Feb 23, 2010 at 11:43, denis <denis-bz-gg at t-online.de> wrote:
>> Robert, Josef,
>> ?thanks much for taking the time to look at RBF some more.
>> Summary, correct me:
>> ? ?A - smooth*I in rbf.py is a sign error (ticket ?)
>
> Not necessarily. It seems to work well in at least some cases. Find a
> reference that says otherwise if you want it changed.

chapter 2 page 16, for gaussian process. As I said I don't know about
the other methods

http://docs.google.com/viewer?a=v&q=cache:qs8AaAxO6nkJ:www.gaussianprocess.org/gpml/chapters/RW2.pdf+gaussian+process+noise+Ridge&hl=en&gl=ca&pid=bl&srcid=ADGEESj4j8osT6cOIc65r3OaeAtQO_dzgZD4YxSAEkFTeRZajBcROJpJJ9zTlMSrD2OaK1iOJYgy8QqH_Nr0rNxf41faNihCdIzWyVOYxtCFIR7H8mdQZAKFoeaRkFamQlCKhp_s1FOI&sig=

www.gaussianprocess.org/gpml/chapters/RW2.pdf

Josef
>
>> ? ?for gauss, start with A + 1e-6*I ?to move eigenvalues away from 0
>> ? ?others have pos/neg eigenvalues, don't need smooth.
>
> No. If you need a smoothed approximation to noisy data rather than
> exact interpolation than you use the smooth keyword. Otherwise, you
> don't.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Tue Feb 23 12:59:56 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 23 Feb 2010 12:59:56 -0500
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <1cd32cbb1002230957w35b7119ay7270bbbc1892ce25@mail.gmail.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<1cd32cbb1002190841u64c11667x63b910c1c54081b0@mail.gmail.com>
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com>
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com>
	<43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com>
	<1cd32cbb1002220845o3ad94ba8g599f25ec170c6106@mail.gmail.com>
	<1cd32cbb1002220927h229e578bhe18737d803fddc76@mail.gmail.com>
	<1c841a64-786d-4069-9f88-fcaae5a0d063@t42g2000vbt.googlegroups.com>
	<3d375d731002230946m75c7cacbk4a45e8e3db518d54@mail.gmail.com>
	<1cd32cbb1002230957w35b7119ay7270bbbc1892ce25@mail.gmail.com>
Message-ID: <1cd32cbb1002230959u46a4da49ie6e7220d5964a381@mail.gmail.com>

On Tue, Feb 23, 2010 at 12:57 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Feb 23, 2010 at 12:46 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Tue, Feb 23, 2010 at 11:43, denis <denis-bz-gg at t-online.de> wrote:
>>> Robert, Josef,
>>> ?thanks much for taking the time to look at RBF some more.
>>> Summary, correct me:
>>> ? ?A - smooth*I in rbf.py is a sign error (ticket ?)
>>
>> Not necessarily. It seems to work well in at least some cases. Find a
>> reference that says otherwise if you want it changed.
>
> chapter 2 page 16, for gaussian process. As I said I don't know about
> the other methods
>

http://docs.google.com/viewer?a=v&q=cache:qs8AaAxO6nkJ:www.gaussianprocess.org/gpml/chapters/RW2.pdf+gaussian+process+noise+Ridge&hl=en&gl=ca&pid=bl&srcid=ADGEESj4j8osT6cOIc65r3OaeAtQO_dzgZD4YxSAEkFTeRZajBcROJpJJ9zTlMSrD2OaK1iOJYgy8QqH_Nr0rNxf41faNihCdIzWyVOYxtCFIR7H8mdQZAKFoeaRkFamQlCKhp_s1FOI&sig=AHIEtbQK35MLfnZAySw3lF-dR_mNcSaP3w

google links are very short, missed a part

>
> www.gaussianprocess.org/gpml/chapters/RW2.pdf
>
> Josef
>>
>>> ? ?for gauss, start with A + 1e-6*I ?to move eigenvalues away from 0
>>> ? ?others have pos/neg eigenvalues, don't need smooth.
>>
>> No. If you need a smoothed approximation to noisy data rather than
>> exact interpolation than you use the smooth keyword. Otherwise, you
>> don't.
>>
>> --
>> Robert Kern
>>
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>> ?-- Umberto Eco
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From bapeters at terastat.com  Tue Feb 23 16:05:21 2010
From: bapeters at terastat.com (Bruce Peterson)
Date: Tue, 23 Feb 2010 13:05:21 -0800
Subject: [SciPy-User] test
In-Reply-To: <mailman.5557.1266594142.2454.scipy-user@scipy.org>
References: <mailman.5557.1266594142.2454.scipy-user@scipy.org>
Message-ID: <20100223210542.2ED4539CB47@scipy.org>



Bruce Peterson
425 466 7344 



From andrew.collette at gmail.com  Tue Feb 23 16:52:12 2010
From: andrew.collette at gmail.com (Andrew Collette)
Date: Tue, 23 Feb 2010 13:52:12 -0800
Subject: [SciPy-User] [ANN] HDF5 for Python (h5py) 1.3.0 beta
Message-ID: <f125f5e1002231352x7547e006ha8cda585b5a0ff74@mail.gmail.com>

HDF5 for Python (h5py) 1.3.0 BETA
=================================

I'm pleased to announce that HDF5 for Python 1.3 is now available!  This
is a significant release introducing a number of new features, including
support for soft/external links as well as object and region references.

I encourage all interested HDF5/NumPy/Python users to give the beta a try
and to do your best to break it. :)  Download, documentation and contact
links are below.


What is h5py?
-------------

HDF5 for Python (h5py) is a general-purpose Python interface to the
Hierarchical Data Format library, version 5.  HDF5 is a mature scientific
software library originally developed at NCSA, designed for the fast,
flexible storage of enormous amounts of data.

>From a Python programmer's perspective, HDF5 provides a robust way to
store data, organized by name in a tree-like fashion.  You can create
datasets (arrays on disk) hundreds of gigabytes in size, and perform
random-access I/O on desired sections.  Datasets are organized in a
filesystem-like hierarchy using containers called "groups", and
accesed using the tradional POSIX /path/to/resource syntax.

In addition to providing interoperability with existing HDF5 datasets
and platforms, h5py is a convienient way to store and retrieve
arbitrary NumPy data and metadata.

HDF5 datasets and groups are presented as "array-like" and "dictionary-like"
objects in order to make best use of existing experience.  For example,
dataset I/O is done with NumPy-style slicing, and group access is via
indexing with string keys.  Standard Python exceptions (KeyError, etc) are
raised in response to underlying HDF5 errors.


New features in 1.3
-------------------

 - Full support for soft and external links

 - Full support for object and region references, in all contexts (datasets,
   attributes, etc).  Region references can be created using the standard
   NumPy slicing syntax.

 - A new get() method for HDF5 groups, which also allows the type of an
   object or link to be queried without first opening it.

 - Improved locking system which makes h5py faster in both multi-threaded and
   single-threaded applications.

 - Automatic creation of missing intermediate groups (HDF5 1.8)

 - Anonymous group and dataset creation (HDF5 1.8)

 - Option to enable cProfile support for the parts of h5py written in Cython

 - Many bug fixes and performance enhancements


Other changes
-------------

 - Old-style dictionary methods (listobjects, etc) will now issue
   DeprecationWarning, and will be removed in 1.4.

 - Dataset .value attribute is deprecated.  Use dataset[...] or dataset[()].

 - new_vlen(), get_vlen(), new_enum() and get_enum() are deprecated in favor
   of the functions h5py.special_dtype() and h5py.check_dtype(), which also
   support reference types.


Where to get it
---------------

* Main website, documentation:  http://h5py.alfven.org

* Downloads, bug tracker:       http://h5py.googlecode.com

* Mailing list (discussion and development): h5py at googlegroups.com

* Contact email: h5py at alfven.org


Requires
--------

* Linux, Mac OS-X or Windows

* Python 2.5 or 2.6

* NumPy 1.0.3 or later

* HDF5 1.6.5 or later (including 1.8); HDF5 is included with
  the Windows version.


From vincent at vincentdavis.net  Wed Feb 24 00:18:04 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Tue, 23 Feb 2010 22:18:04 -0700
Subject: [SciPy-User] List to record array problems, questions
Message-ID: <77e831101002232118gbae1671nb4cb65690be2cd2a@mail.gmail.com>

I must be missing something simple. I have a list of lists data = "[['  0',
'  0', '234.0', '24.0', ' 25'], ['  1', '  0', '22428.0', '2378.1', '
25'],......" and what to make a record array (I guess I have made a
structured array) from it but it gets screwed up or I don't get it, likely
both.
Notice that at this stage the items are strings, not numbers, and there is
whitespace not sure this matters.

Here is what is happening, notice that the number get screwed up in the
structured array (not sure who to get a record array, I thought this was but
was told it is not)
adata = numpy.array(data,numpy.float64)

>>> adata
array([[  0.00000000e+00,   0.00000000e+00,   2.34000000e+02,
          2.40000000e+01,   2.50000000e+01],
       ...,
       [  4.77000000e+02,   4.77000000e+02,   2.07000000e+02,
          4.58000000e+01,   2.50000000e+01]])

This is what I would expect except it is not a record array.

This is not what I expect. I think I have tried every iteration including
using numpy dtaypes numpy.int32 or bdata = numpy.array(data, dtype = [('x',
int),('y', int),('mean',float),('stdv',float),('npixcels',int)])
What am I missing?

bdata = numpy.array(data, [('x', int),('y',
int),('mean',float),('stdv',float),('npixcels',int)])
>>> bdata
array([[(3153952, 0, 0.0, 0.0, 0), (3153952, 0, 0.0, 0.0, 0),
        (206933603122, 0, 0.0, 0.0, 0), (808334386, 0, 0.0, 0.0, 0),
        (3486240, 0, 0.0, 0.0, 0)],
       [(3219488, 0, 0.0, 0.0, 0), (3153952, 0, 0.0, 0.0, 0),
        (13561617777439282, 0, 0.0, 0.0, 0),
        (54074581398322, 0, 0.0, 0.0, 0), (3486240, 0, 0.0, 0.0, 0)],
       [(3285024, 0, 0.0, 0.0, 0), (3153952, 0, 0.0, 0.0, 0),
        (206933931058, 0, 0.0, 0.0, 0), (925775666, 0, 0.0, 0.0, 0),
        (3486240, 0, 0.0, 0.0, 0)],
       ...,
       [(3487540, 0, 0.0, 0.0, 0), (3618612, 0, 0.0, 0.0, 0),
        (206933602866, 0, 0.0, 0.0, 0), (908996661, 0, 0.0, 0.0, 0),
        (3486240, 0, 0.0, 0.0, 0)],
       [(3553076, 0, 0.0, 0.0, 0), (3618612, 0, 0.0, 0.0, 0),
        (13561596370041137, 0, 0.0, 0.0, 0),
        (62870573495603, 0, 0.0, 0.0, 0), (3486240, 0, 0.0, 0.0, 0)],
       [(3618612, 0, 0.0, 0.0, 0), (3618612, 0, 0.0, 0.0, 0),
        (206933798962, 0, 0.0, 0.0, 0), (942552372, 0, 0.0, 0.0, 0),
        (3486240, 0, 0.0, 0.0, 0)]],

      dtype=[('x', '<i8'), ('y', '<i8'), ('mean', '<f8'), ('stdv', '<f8'),
('npixcels', '<i8')])


  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100223/a8c6a14a/attachment.html>

From stefan at sun.ac.za  Wed Feb 24 01:18:35 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 24 Feb 2010 08:18:35 +0200
Subject: [SciPy-User] List to record array problems, questions
In-Reply-To: <77e831101002232118gbae1671nb4cb65690be2cd2a@mail.gmail.com>
References: <77e831101002232118gbae1671nb4cb65690be2cd2a@mail.gmail.com>
Message-ID: <9457e7c81002232218k326b6260v208be37b711a09e8@mail.gmail.com>

Hi Vincent

On 24 February 2010 07:18, Vincent Davis <vincent at vincentdavis.net> wrote:
>
> I must be missing something simple. I have a list of lists data = "[[' ?0', ' ?0', '234.0', '24.0', ' 25'], [' ?1', ' ?0', '22428.0', '2378.1', ' 25'],......" and what to make a record array (I guess I have made a structured array) from it but it gets screwed up or I don't get it, likely both.

The constructor for structured arrays expect a list of tuples:

data_tuple = [tuple(row) for row in data]
np.array(data_tuple, dtype=[('x', int), ('y', int), ('mean', float),
('stdv', float), ('npixels', int)])

This yields

array([(0, 0, 234.0, 24.0, 25), (1, 0, 22428.0, 2378.0999999999999, 25)],
      dtype=[('x', '<i8'), ('y', '<i8'), ('mean', '<f8'), ('stdv',
'<f8'), ('npixels', '<i8')])

Regards
St?fan


From arokem at berkeley.edu  Wed Feb 24 01:32:34 2010
From: arokem at berkeley.edu (Ariel Rokem)
Date: Tue, 23 Feb 2010 22:32:34 -0800
Subject: [SciPy-User] TimeSeries and Milliseconds for aircraft data
In-Reply-To: <49d6b3501002230723x71ca6913y7aea5d2b1eecc7d0@mail.gmail.com>
References: <3848801.16841265888539566.JavaMail.root@mohawk.flightdataservices.com>
	<24755476.01266917706841.JavaMail.chris@jesse1c-laptop>
	<49d6b3501002230723x71ca6913y7aea5d2b1eecc7d0@mail.gmail.com>
Message-ID: <43958ee61002232232s469353dtcfd8730ea83c76f3@mail.gmail.com>

Hi everyone,

Gokhan is referring to this:

http://nipy.sourceforge.net/nitime/

For an exposition, see our Scipy conference proceedings paper (a pdf of
which can be found here:
http://argentum.ucbso.berkeley.edu/papers/Rokem2009Nitime.pdf).

We are still working on it. The intention of the library is to support
analysis of data from neuroscience experiments, because we are
neuroscientists, but so far, I don't think that we have made any design
decisions that would preclude other scientists from using our time-series
objects. In fact, the time-series objects we have designed support temporal
resolutions as fast as picoseconds (the representation of time is done in
int64, in order to avoid float-precision issues). It is still under
development and we have yet to make a release of this, but the code (in
development) is already available on github and the tests therein can direct
you on the possible usage:

http://github.com/fperez/nitime

Cheers,

Ariel

On Tue, Feb 23, 2010 at 7:23 AM, G?khan Sever <gokhansever at gmail.com> wrote:

>
>
> On Tue, Feb 23, 2010 at 3:35 AM, Chris Jesse <
> chris.jesse at flightdataservices.com> wrote:
>
>> Hi All,
>>
>> I've got lots of timeseries data from aircraft sensors (such as altitude,
>> temperatures, landing gear up/down, airspeed..) which I was hoping to load
>> into a timeseries array. However, I've just read that it only supports time
>> resolutions down to a second. Is there a solution for my data which contains
>> samples from hundreds of different parameters at different frequencies from
>> 8Hz to 1/64Hz over an 8 hour period?
>>
>> I need to be able to interpolate between values, mask values when samples
>> are corrupt / outside of operational boundaries and store values as
>> accurately to the nearest 1/16th of a second if possible.
>>
>> Your help is much appreciated! Thanks,
>>
>> Chris
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
> Hi Chris,
>
> Here at the University of North Dakota we work with aircraft measurements
> as well. Similar to your case depends on the campaign [1] sometimes the
> number of measured parameters reach over hundred. We use a QNX 4.25 based
> data acquisition system called M300 where all the probe data are connected
> in. M300 saves everything in a binary file which we later process using our
> ADPAA package [2] It is mostly IDL coded and mainly developed by Dr. David
> Delene [3] He is also the main responsible person for the aircraft
> measurements/deployments in our department.
>
> ADPAA produces files using a pre-determined ASCII data structure shown here
> [4] The system supports frequencies up to 100 Hz but typically the highest
> we sample is at 25 Hz. There might be exceptional cases where we might need
> higher sample rates. However 1-4 Hz is sufficiently enough when sampling
> clouds with cloud-aerosol microphysical instruments. (PCASP, FSSP, CCN) (I
> am disregarding instruments like holographic particle detectors for the
> simplicity of my response :) which we don't often get to fly with.)
>
> Anyway after some advertisement back to your question: in our system all
> the data are time-stamped with a main "Time [seconds]; UT seconds from
> midnight on day aircraft flight started" value. It simplifies data analysis
> a lot in my view where the Python comes into action actually. Once the ASCII
> files are created (we also output to NetCDF containers but I find easy to
> work on simple text files.) I read the data using NumPy and I use the
> MaskedArray module to deal with the missing values in the data. I publish my
> code along with data on the ccnworks [5] page where I mostly commit for my
> thesis using ground and airborne measurements and scientific Python
> ecosystem. For instance to read the ASCII data files (also known as NASA
> formatted) I use the generic script [6] As a part of my thesis I am working
> on this script [7] recently. I use the lab data [8] so far. In logn-fit.py I
> have a section where I do masking on the data like:
>
> mask02 = (pcasp.data['Time'] > 13800) & (pcasp.data['Time'] <= 14400)
>
> This is one of the most important part of the code --to extract the right
> portion of the required data. In this case I am interested in time periods
> based on the supersaturation values from the DMT-CCNC instrument. The next
> step will be to analyse the cloud base data. As far as I know there is no
> easy solution for this. What do PCASP probe shows when the aircraft is
> levelling below a cloud deck :) The only way I know is to mask manually and
> most logically using the common time-stamp in all data files.
>
> I remember Berkeley guys work on a project called NiPy but it is for
> neuro-scientists mostly and not much design consideration in it for airborne
> measurement scientists in mind :)
>
> I hope extra information doesn't bother you and let me know your opinions.
>
>
> [1] http://atmoswiki.aero.und.edu/atmos/citation/home
> [2] http://sourceforge.net/projects/adpaa/
> [3] http://aerosol.atmos.und.edu/
> [4] http://aerosol.atmos.und.edu/ADPAA/fileheader.html
> [5] http://code.google.com/p/ccnworks/
> [6] http://code.google.com/p/ccnworks/source/browse/trunk/nasafile.py
> [7]
> http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/logn-fit.py
> [8]
> http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/20090225_034214/09_02_25_03_42_14.lab.clean
>
> --
> G?khan
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Ariel Rokem
Helen Wills Neuroscience Institute
University of California, Berkeley
http://argentum.ucbso.berkeley.edu/ariel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100223/c44b59bf/attachment.html>

From spectre158 at gmail.com  Wed Feb 24 02:43:51 2010
From: spectre158 at gmail.com (Scott Barlow)
Date: Tue, 23 Feb 2010 23:43:51 -0800
Subject: [SciPy-User] Scipy dblquad Calculation
Message-ID: <a7e4caf91002232343h3365207eja502a1ec79bea82c@mail.gmail.com>

Hello all.  I'm writing some code and need to compute a double integral.
I'm trying to calculate this using dblquad, but I'm having an issue with the
result.  I get a result and no error messages, but the result is not as
expected.  I've check the code over and over and I see no mistakes, so I'm
asking for suggestions here (or reasons for the difference).  I realize that
the difference could be from (quadrature) estimations, but I don't think
that is it.

Python version: 2.6.4
SciPy version: 0.7.0
NumPy version: 1.3.0

Please see the code below.  This describes the equation for the bending
rigidity of an elliptical composite laminate of 16 plies under bending.
Note that "a" and "b" would be the length in inches of the major and minor
axes, and I have them both set to 2.0 here, thus this is actually a circle.
The bottom equation (Final_Result_2) is the reduced equation for a circular
tube (no integration required).  Thus when a=b, the result between the two
equations should be the same.

When run, I get the following:
21560941.3345   (Final_Result)
22852651.614    (Final_Result_2)

As mentioned before, I've checked the equations numerous times for typos,
but I haven't found any.  I'm quite sure the equation is input correctly.
I've also tried using arctan2 instead of arctan, but I get the same result.

Here is the python code:
---------------------------------------------------------------------------
-------------
import scipy
import numpy
from scipy.integrate import dblquad

Q11 = 20009576.0
Q22 = 1397653.8
Qbar11 = [Q11, Q11,  Q11,  Q11,  Q22,  Q22,  Q22,  Q22, Q22, Q22, Q22, Q22,
Q11, Q11, Q11, Q11];

a = 2.0;
b = 2.0;
thickness = 0.005;

Final_Result = 0.0;
for i in range(1, 17):
  Temp_Result = dblquad(lambda theta, t:
                        Qbar11[i-1] *
                        scipy.sin(theta)**2.0 *
                        (
                           1.0*((a**2.0 * b**2.0)/((a**2.0 -
b**2.0)*scipy.sin(theta)**2.0 + b**2.0)) +
                           t**2.0 +
                           2.0*((a**2.0 * b**2.0)/((a**2.0 -
b**2.0)*scipy.sin(theta)**2.0 + b**2.0))**(0.5)*
                           t*scipy.sin(theta +
scipy.arctan((a**2.0/b**2.0)*(1.0/scipy.tan(theta))))
                        )**(3.0/2.0),
                        thickness*(i-1),
                        thickness*(i),
                        lambda x: 0.0,
                        lambda x: 2.0*scipy.pi,
                        epsabs = 1.4899999999999999e-12,
                        epsrel = 1.4899999999999999e-12);
  Final_Result += Temp_Result[0];
print Final_Result;

ri = a;
Final_Result_2 = 0.0;
for i in range(1, 17):
  Final_Result_2 += (scipy.pi/4.0) * Qbar11[i-1] * ((ri +
thickness*(i))**4.0 - (ri + thickness*(i-1))**4.0);
print Final_Result_2;
---------------------------------------------------------------------------
-------------

Thanks for any help,
Scott
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100223/3dd53831/attachment.html>

From warren.weckesser at enthought.com  Wed Feb 24 06:16:52 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Wed, 24 Feb 2010 05:16:52 -0600
Subject: [SciPy-User] Scipy dblquad Calculation
In-Reply-To: <a7e4caf91002232343h3365207eja502a1ec79bea82c@mail.gmail.com>
References: <a7e4caf91002232343h3365207eja502a1ec79bea82c@mail.gmail.com>
Message-ID: <4B850AA4.7040505@enthought.com>


Scott Barlow wrote:
> Hello all.  I'm writing some code and need to compute a double 
> integral.  I'm trying to calculate this using dblquad, but I'm having 
> an issue with the result.  I get a result and no error messages, but 
> the result is not as expected.  I've check the code over and over and 
> I see no mistakes, so I'm asking for suggestions here (or reasons for 
> the difference).  I realize that the difference could be from 
> (quadrature) estimations, but I don't think that is it.
>
> Python version: 2.6.4
> SciPy version: 0.7.0
> NumPy version: 1.3.0
>
> Please see the code below.  This describes the equation for the 
> bending rigidity of an elliptical composite laminate of 16 plies under 
> bending.  Note that "a" and "b" would be the length in inches of the 
> major and minor axes, and I have them both set to 2.0 here, thus this 
> is actually a circle.  The bottom equation (Final_Result_2) is the 
> reduced equation for a circular tube (no integration required).  Thus 
> when a=b, the result between the two equations should be the same.
>
> When run, I get the following:
> 21560941.3345   (Final_Result)
> 22852651.614    (Final_Result_2)
>
> As mentioned before, I've checked the equations numerous times for 
> typos, but I haven't found any.  I'm quite sure the equation is input 
> correctly.  I've also tried using arctan2 instead of arctan, but I get 
> the same result.
>

Scott,

As you suspected, the problem is your use of arctan, and an appropriate 
use of arctan2 can correct it.  If I change this:

                           t*scipy.sin(theta + 
scipy.arctan((a**2.0/b**2.0)*(1.0/scipy.tan(theta))))


to this:

                           t*scipy.sin(theta + scipy.arctan2( 
a**2.0*scipy.cos(theta), b**2.0*scipy.sin(theta)) )

then the results agree.


Warren

> Here is the python code:
> ----------------------------------------------------------------------------------------
> import scipy
> import numpy
> from scipy.integrate import dblquad
>
> Q11 = 20009576.0
> Q22 = 1397653.8
> Qbar11 = [Q11, Q11,  Q11,  Q11,  Q22,  Q22,  Q22,  Q22, Q22, Q22, Q22, 
> Q22, Q11, Q11, Q11, Q11];
>
> a = 2.0;
> b = 2.0;
> thickness = 0.005;
>
> Final_Result = 0.0;
> for i in range(1, 17):
>   Temp_Result = dblquad(lambda theta, t:
>                         Qbar11[i-1] *
>                         scipy.sin(theta)**2.0 *
>                         (
>                            1.0*((a**2.0 * b**2.0)/((a**2.0 - 
> b**2.0)*scipy.sin(theta)**2.0 + b**2.0)) +
>                            t**2.0 +
>                            2.0*((a**2.0 * b**2.0)/((a**2.0 - 
> b**2.0)*scipy.sin(theta)**2.0 + b**2.0))**(0.5)*
>                            t*scipy.sin(theta + 
> scipy.arctan((a**2.0/b**2.0)*(1.0/scipy.tan(theta))))
>                         )**(3.0/2.0),
>                         thickness*(i-1),
>                         thickness*(i),
>                         lambda x: 0.0,
>                         lambda x: 2.0*scipy.pi,
>                         epsabs = 1.4899999999999999e-12,
>                         epsrel = 1.4899999999999999e-12);
>   Final_Result += Temp_Result[0];
> print Final_Result;
>
> ri = a;
> Final_Result_2 = 0.0;
> for i in range(1, 17):
>   Final_Result_2 += (scipy.pi/4.0) * Qbar11[i-1] * ((ri + 
> thickness*(i))**4.0 - (ri + thickness*(i-1))**4.0);
> print Final_Result_2;
> ----------------------------------------------------------------------------------------
>
> Thanks for any help,
> Scott
> ------------------------------------------------------------------------
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From nwagner at iam.uni-stuttgart.de  Wed Feb 24 08:12:30 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Wed, 24 Feb 2010 14:12:30 +0100
Subject: [SciPy-User] Signal to noise ratio
Message-ID: <web-130689370@uni-stuttgart.de>

Hi all,

I have two questions concerning signal processing

I have used scipy.stats.signaltonoise to compute the 
signal-to-noise ratio.
The value is 0.0447.
How can I judge it ?

How can I filter out high frequencies using scipy ?
How can I eliminate noise from the signal ?

Nils


From p.schellart at gmail.com  Wed Feb 24 09:07:42 2010
From: p.schellart at gmail.com (Pim Schellart)
Date: Wed, 24 Feb 2010 15:07:42 +0100
Subject: [SciPy-User] Suggest moving curve_fit from minpack to top level
Message-ID: <7d3dada91002240607v7d9e3cd5n94008f4ea25b2738@mail.gmail.com>

Dear Scipy users,

I would like to suggest moving the curve_fit wrapper function from the
minpack module to the top level and possibly renaming it to "fit".
So you could access it using:

import scipy as sp
sp.fit(...)

Curve fitting is one of the most important tasks in scientific
analysis and most people starting with scipy struggle to find this
(undocumented) function, especially when moving to scipy from gnuplot
which has a basic fit function that performs the same task.
What are your thoughts about this?

Kind regards,

Pim Schellart


From josef.pktd at gmail.com  Wed Feb 24 09:18:28 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 24 Feb 2010 09:18:28 -0500
Subject: [SciPy-User] Suggest moving curve_fit from minpack to top level
In-Reply-To: <7d3dada91002240607v7d9e3cd5n94008f4ea25b2738@mail.gmail.com>
References: <7d3dada91002240607v7d9e3cd5n94008f4ea25b2738@mail.gmail.com>
Message-ID: <1cd32cbb1002240618tf286f23g36fc185c18f22dea@mail.gmail.com>

On Wed, Feb 24, 2010 at 9:07 AM, Pim Schellart <p.schellart at gmail.com> wrote:
> Dear Scipy users,
>
> I would like to suggest moving the curve_fit wrapper function from the
> minpack module to the top level and possibly renaming it to "fit".
> So you could access it using:
>
> import scipy as sp
> sp.fit(...)
>
> Curve fitting is one of the most important tasks in scientific
> analysis and most people starting with scipy struggle to find this
> (undocumented) function, especially when moving to scipy from gnuplot
> which has a basic fit function that performs the same task.
> What are your thoughts about this?

I don't think it's a good idea.

I almost never import scipy directly, only

from scipy import optimize, stats
or
import scipy.linalg as splinalg

The pure scipy.* namespace is pretty empty

"fit" is a term that is too generic, polyfit, linear fit, ....

curve_fit will hopefully sufficiently documented and included in the
docs when it is included in a release of scipy.

see
>>> import this  # last item

Josef


>
> Kind regards,
>
> Pim Schellart
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From ralf.gommers at googlemail.com  Wed Feb 24 09:31:35 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Wed, 24 Feb 2010 22:31:35 +0800
Subject: [SciPy-User] Suggest moving curve_fit from minpack to top level
In-Reply-To: <1cd32cbb1002240618tf286f23g36fc185c18f22dea@mail.gmail.com>
References: <7d3dada91002240607v7d9e3cd5n94008f4ea25b2738@mail.gmail.com>
	<1cd32cbb1002240618tf286f23g36fc185c18f22dea@mail.gmail.com>
Message-ID: <dde7764a1002240631w5682aecfhea46b00ec4a72f41@mail.gmail.com>

On Wed, Feb 24, 2010 at 10:18 PM, <josef.pktd at gmail.com> wrote:

> On Wed, Feb 24, 2010 at 9:07 AM, Pim Schellart <p.schellart at gmail.com>
> wrote:
> > Curve fitting is one of the most important tasks in scientific
> > analysis and most people starting with scipy struggle to find this
> > (undocumented) function
>
> curve_fit will hopefully sufficiently documented and included in the
> docs when it is included in a release of scipy.
>
> Looks pretty well documented to me:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html
It may be helpful though if this function is pointed out in a tutorial (not
sure if that's the case or not).

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100224/ec17988e/attachment.html>

From josef.pktd at gmail.com  Wed Feb 24 09:33:12 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 24 Feb 2010 09:33:12 -0500
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <web-130689370@uni-stuttgart.de>
References: <web-130689370@uni-stuttgart.de>
Message-ID: <1cd32cbb1002240633n1a7c839ah9a214a59931e6800@mail.gmail.com>

On Wed, Feb 24, 2010 at 8:12 AM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> Hi all,
>
> I have two questions concerning signal processing
>
> I have used scipy.stats.signaltonoise to compute the
> signal-to-noise ratio.
> The value is 0.0447.
> How can I judge it ?

It's just mean over standard deviation
http://en.wikipedia.org/wiki/Signal-to-noise_ratio#Statistical_definition

I never use it, but the interpretation will depend on what your
level/mean/expected_value means.

>
> How can I filter out high frequencies using scipy ?
> How can I eliminate noise from the signal ?

(I'm no help here) There are many prefabricated filters in
scipy.signal, but I only use lfilter.

Josef

>
> Nils
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Wed Feb 24 09:48:31 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 24 Feb 2010 09:48:31 -0500
Subject: [SciPy-User] Suggest moving curve_fit from minpack to top level
In-Reply-To: <dde7764a1002240631w5682aecfhea46b00ec4a72f41@mail.gmail.com>
References: <7d3dada91002240607v7d9e3cd5n94008f4ea25b2738@mail.gmail.com>
	<1cd32cbb1002240618tf286f23g36fc185c18f22dea@mail.gmail.com>
	<dde7764a1002240631w5682aecfhea46b00ec4a72f41@mail.gmail.com>
Message-ID: <1cd32cbb1002240648r60f9d86cmd77404023d81549e@mail.gmail.com>

On Wed, Feb 24, 2010 at 9:31 AM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
>
>
> On Wed, Feb 24, 2010 at 10:18 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Wed, Feb 24, 2010 at 9:07 AM, Pim Schellart <p.schellart at gmail.com>
>> wrote:
>> > Curve fitting is one of the most important tasks in scientific
>> > analysis and most people starting with scipy struggle to find this
>> > (undocumented) function
>>
>> curve_fit will hopefully sufficiently documented and included in the
>> docs when it is included in a release of scipy.
>>
> Looks pretty well documented to me:
> http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html
> It may be helpful though if this function is pointed out in a tutorial (not
> sure if that's the case or not).

Thanks, my docs are too old and I didn't know it's already included in
the html. I checked and it is not mentioned in the optimize tutorial.

curve_fit doesn't really fit into scipy.optimize next to all the low
level optimizers.

Given the recent discussion, I think it would fit better in
scipy.interpolate which could be extended to interpolate and fitting
to noisy data, e.g. rbf, least squares splines.

Is it too late to move it?

Josef

>
> Cheers,
> Ralf
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From josef.pktd at gmail.com  Wed Feb 24 10:11:19 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 24 Feb 2010 10:11:19 -0500
Subject: [SciPy-User] noisy interpolators, smoothers and fitters
Message-ID: <1cd32cbb1002240711y47b48ae9tb62846b84af7edc2@mail.gmail.com>

Is there a need for expanding this in scipy.interpolate?

see also offer by Jonathan Stickel which looks useful with
penalization on derivatives (if my quick reading is correct).

Josef


From ivo.maljevic at gmail.com  Wed Feb 24 10:16:19 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Wed, 24 Feb 2010 10:16:19 -0500
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <1cd32cbb1002240633n1a7c839ah9a214a59931e6800@mail.gmail.com>
References: <web-130689370@uni-stuttgart.de>
	<1cd32cbb1002240633n1a7c839ah9a214a59931e6800@mail.gmail.com>
Message-ID: <826c64da1002240716p65ff7830wac050cb21d1b0322@mail.gmail.com>

One would think that you can always rely on wikipedia when it comes to math
and engineering, but it seems that is not tha case.
Josef, In the page you referenced, the SNR, or signal to noise ratio, is
defined as the ratio between the signal and noise powers. Consequently, in
terms of signals and standard deviations, it is defined as a ratio of the
average signal power and the noise variance (NOT its squre root, or standard
deviation). Or:

SNR = P_s / sigma^2

where P_s is the average signal power, and the noise variance is used to
measure the noise power. The assumption here is that the noise is a zero
mean process, otherwise variance and power wouldn't be the same thing.

Nils, your question is way too generic for anyone to help you directly. I
can only point to you that your signal to noise ratio is quite low:

>>> 10*math.log10(0.0447)
-13.496924768680636

Maybe your signal is to narrow compared to the overal band you are working
with (or you have DS spread spectrum signal?).
Anyway, you will need to figure out which filter you want to use (e.g.,
butterworth for maximally flat characteristic in the passband, etc).


On 24 February 2010 09:33, <josef.pktd at gmail.com> wrote:

> On Wed, Feb 24, 2010 at 8:12 AM, Nils Wagner
> <nwagner at iam.uni-stuttgart.de> wrote:
> > Hi all,
> >
> > I have two questions concerning signal processing
> >
> > I have used scipy.stats.signaltonoise to compute the
> > signal-to-noise ratio.
> > The value is 0.0447.
> > How can I judge it ?
>
> It's just mean over standard deviation
> http://en.wikipedia.org/wiki/Signal-to-noise_ratio#Statistical_definition
>
> I never use it, but the interpretation will depend on what your
> level/mean/expected_value means.
>
> >
> > How can I filter out high frequencies using scipy ?
> > How can I eliminate noise from the signal ?
>
> (I'm no help here) There are many prefabricated filters in
> scipy.signal, but I only use lfilter.
>
> Josef
>
> >
> > Nils
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100224/7a10a550/attachment.html>

From josef.pktd at gmail.com  Wed Feb 24 10:23:11 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 24 Feb 2010 10:23:11 -0500
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <826c64da1002240716p65ff7830wac050cb21d1b0322@mail.gmail.com>
References: <web-130689370@uni-stuttgart.de>
	<1cd32cbb1002240633n1a7c839ah9a214a59931e6800@mail.gmail.com>
	<826c64da1002240716p65ff7830wac050cb21d1b0322@mail.gmail.com>
Message-ID: <1cd32cbb1002240723j89280c1i88af661ae85fe2f8@mail.gmail.com>

On Wed, Feb 24, 2010 at 10:16 AM, Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
> One would think that you can always rely on wikipedia when it comes to math
> and engineering, but it seems that is not tha case.
> Josef, In the page you referenced, the SNR, or signal to noise ratio, is
> defined as the ratio between the signal and noise powers. Consequently, in
> terms of signals and standard deviations, it is defined as a ratio of the
> average signal power and the noise variance (NOT its squre root, or standard
> deviation). Or:
>
> SNR = P_s / sigma^2
>
> where P_s is the average signal power, and the noise variance is used to
> measure the noise power. The assumption here is that the noise is a zero
> mean process, otherwise variance and power wouldn't be the same thing.

Note: I linked to #Statistical_definition not the top of the wikipedia page
and I checked the source in scipy.stats:
    Calculates the signal-to-noise ratio, defined as the ratio between the mean
    and the standard deviation.

    m = np.mean(a, axis)
    sd = samplestd(a, axis)
    return np.where(sd == 0, 0, m/sd)

I didn't know about the different definitions until I read the
Wikipedia page, but that's what's currently in scipy.stats

Josef

>
> Nils, your question is way too generic for anyone to help you directly. I
> can only point to you that your signal to noise ratio is quite low:
>
>>>> 10*math.log10(0.0447)
> -13.496924768680636
>
> Maybe your signal is to narrow compared to the overal band you are working
> with (or you have DS spread spectrum signal?).
> Anyway, you will need to figure out which filter you want to use (e.g.,
> butterworth for maximally flat characteristic in the passband, etc).
>
>
> On 24 February 2010 09:33, <josef.pktd at gmail.com> wrote:
>>
>> On Wed, Feb 24, 2010 at 8:12 AM, Nils Wagner
>> <nwagner at iam.uni-stuttgart.de> wrote:
>> > Hi all,
>> >
>> > I have two questions concerning signal processing
>> >
>> > I have used scipy.stats.signaltonoise to compute the
>> > signal-to-noise ratio.
>> > The value is 0.0447.
>> > How can I judge it ?
>>
>> It's just mean over standard deviation
>> http://en.wikipedia.org/wiki/Signal-to-noise_ratio#Statistical_definition
>>
>> I never use it, but the interpretation will depend on what your
>> level/mean/expected_value means.
>>
>> >
>> > How can I filter out high frequencies using scipy ?
>> > How can I eliminate noise from the signal ?
>>
>> (I'm no help here) There are many prefabricated filters in
>> scipy.signal, but I only use lfilter.
>>
>> Josef
>>
>> >
>> > Nils
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From robert.kern at gmail.com  Wed Feb 24 10:34:48 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 24 Feb 2010 09:34:48 -0600
Subject: [SciPy-User] Suggest moving curve_fit from minpack to top level
In-Reply-To: <1cd32cbb1002240648r60f9d86cmd77404023d81549e@mail.gmail.com>
References: <7d3dada91002240607v7d9e3cd5n94008f4ea25b2738@mail.gmail.com> 
	<1cd32cbb1002240618tf286f23g36fc185c18f22dea@mail.gmail.com> 
	<dde7764a1002240631w5682aecfhea46b00ec4a72f41@mail.gmail.com> 
	<1cd32cbb1002240648r60f9d86cmd77404023d81549e@mail.gmail.com>
Message-ID: <3d375d731002240734n445c2424ved1d622432ca28cc@mail.gmail.com>

On Wed, Feb 24, 2010 at 08:48,  <josef.pktd at gmail.com> wrote:

> curve_fit doesn't really fit into scipy.optimize next to all the low
> level optimizers.
>
> Given the recent discussion, I think it would fit better in
> scipy.interpolate which could be extended to interpolate and fitting
> to noisy data, e.g. rbf, least squares splines.

Heh. This is precisely why scipy.odr is in scipy.odr. When I asked
where it should go, I got three responses and four opinions.
scipy.optimize and scipy.stats are both plausible places for these
routines; scipy.interpolate is plausible if you squint, but is not
really in my opinion. I would like to keep scipy.interpolate for real
interpolation methods. If those interpolation methods also have a
closely related mode for fitting noisy data, so be it, but we
shouldn't expand the scope to include all fitting methods that aren't
interpolation. Similarly, curve_fit() and leastsq() are really just
small wrappers around the underlying minimizers that do the heavy
lifting, so they ended up in scipy.optimize.

It might be time for a new top-level package scipy.fitting that brings
together all of the noisy fitting solutions and provides a nice place
for new code like Jonathan Stickel's. However, I would only want to do
that if we were providing something else along with it, like unifying
the interfaces or providing generic cross-validation routines that
would work with any fitter that had the right interface, etc. I think
that moving things around just to have a better organization is a
waste of time and is not justified by the costs of deprecating working
code and invalidating umpteen docs, tutorials, archived emails, blog
posts, Stack Overflow answers that are floating out there, being all
Googleable as outdated documentation is wont to be.

Ultimately, I think that the organization of functions into packages
is much less important than having quality documentation that answers
users' "How do I ...?" questions. I know it's much easier to move a
function into a new package than it is to write good docs, but it
doesn't solve the real problem.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ivo.maljevic at gmail.com  Wed Feb 24 10:35:23 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Wed, 24 Feb 2010 10:35:23 -0500
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <1cd32cbb1002240723j89280c1i88af661ae85fe2f8@mail.gmail.com>
References: <web-130689370@uni-stuttgart.de>
	<1cd32cbb1002240633n1a7c839ah9a214a59931e6800@mail.gmail.com>
	<826c64da1002240716p65ff7830wac050cb21d1b0322@mail.gmail.com>
	<1cd32cbb1002240723j89280c1i88af661ae85fe2f8@mail.gmail.com>
Message-ID: <826c64da1002240735k666eb608k7fcce992ce7957fa@mail.gmail.com>

Josef,
Maybe the definition you use is for some specific field I'm not familiar
with. AFIK, SNR is defined as the ratio of powers and not amplitudes, but my
background is in electrical engineering / communiation theory.

Please take a look at the following links - but this is just a couple of
links from the sea of ones you can find on the web:

http://www.scholarpedia.org/article/Signal-to-noise_ratio

where it says: Thus, the SNR equals [image: \mathsf{E}[S^2]/\sigma^2_N].

http://authors.library.caltech.edu/3763/1/CUIieeecl06a.pdf  (formula 16)

Ivo







On 24 February 2010 10:23, <josef.pktd at gmail.com> wrote:

> On Wed, Feb 24, 2010 at 10:16 AM, Ivo Maljevic <ivo.maljevic at gmail.com>
> wrote:
> > One would think that you can always rely on wikipedia when it comes to
> math
> > and engineering, but it seems that is not tha case.
> > Josef, In the page you referenced, the SNR, or signal to noise ratio, is
> > defined as the ratio between the signal and noise powers. Consequently,
> in
> > terms of signals and standard deviations, it is defined as a ratio of the
> > average signal power and the noise variance (NOT its squre root, or
> standard
> > deviation). Or:
> >
> > SNR = P_s / sigma^2
> >
> > where P_s is the average signal power, and the noise variance is used to
> > measure the noise power. The assumption here is that the noise is a zero
> > mean process, otherwise variance and power wouldn't be the same thing.
>
> Note: I linked to #Statistical_definition not the top of the wikipedia page
> and I checked the source in scipy.stats:
>    Calculates the signal-to-noise ratio, defined as the ratio between the
> mean
>    and the standard deviation.
>
>    m = np.mean(a, axis)
>    sd = samplestd(a, axis)
>    return np.where(sd == 0, 0, m/sd)
>
> I didn't know about the different definitions until I read the
> Wikipedia page, but that's what's currently in scipy.stats
>
> Josef
>
> >
> > Nils, your question is way too generic for anyone to help you directly. I
> > can only point to you that your signal to noise ratio is quite low:
> >
> >>>> 10*math.log10(0.0447)
> > -13.496924768680636
> >
> > Maybe your signal is to narrow compared to the overal band you are
> working
> > with (or you have DS spread spectrum signal?).
> > Anyway, you will need to figure out which filter you want to use (e.g.,
> > butterworth for maximally flat characteristic in the passband, etc).
> >
> >
> > On 24 February 2010 09:33, <josef.pktd at gmail.com> wrote:
> >>
> >> On Wed, Feb 24, 2010 at 8:12 AM, Nils Wagner
> >> <nwagner at iam.uni-stuttgart.de> wrote:
> >> > Hi all,
> >> >
> >> > I have two questions concerning signal processing
> >> >
> >> > I have used scipy.stats.signaltonoise to compute the
> >> > signal-to-noise ratio.
> >> > The value is 0.0447.
> >> > How can I judge it ?
> >>
> >> It's just mean over standard deviation
> >>
> http://en.wikipedia.org/wiki/Signal-to-noise_ratio#Statistical_definition
> >>
> >> I never use it, but the interpretation will depend on what your
> >> level/mean/expected_value means.
> >>
> >> >
> >> > How can I filter out high frequencies using scipy ?
> >> > How can I eliminate noise from the signal ?
> >>
> >> (I'm no help here) There are many prefabricated filters in
> >> scipy.signal, but I only use lfilter.
> >>
> >> Josef
> >>
> >> >
> >> > Nils
> >> > _______________________________________________
> >> > SciPy-User mailing list
> >> > SciPy-User at scipy.org
> >> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >> >
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100224/0ca44170/attachment.html>

From ivo.maljevic at gmail.com  Wed Feb 24 10:47:00 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Wed, 24 Feb 2010 10:47:00 -0500
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <1cd32cbb1002240723j89280c1i88af661ae85fe2f8@mail.gmail.com>
References: <web-130689370@uni-stuttgart.de>
	<1cd32cbb1002240633n1a7c839ah9a214a59931e6800@mail.gmail.com>
	<826c64da1002240716p65ff7830wac050cb21d1b0322@mail.gmail.com>
	<1cd32cbb1002240723j89280c1i88af661ae85fe2f8@mail.gmail.com>
Message-ID: <826c64da1002240747y148feae5o39a08743268b3989@mail.gmail.com>

Josef,
Just to add, while one can define the "statistical SNR" as something else
(that is, ratio E[s]/sigma_noise), what would be its used for? Astronomical
optics?

On the other hand, it doesn't really matter how it is defined in
scipy.stats. Calculation of SNR is usually a bit more involved (you need to
find the actual signal mean, and the noise variance), and I guess whoever
needs to calculate the power based SNR will not call that function by
mistake.

Ivo

On 24 February 2010 10:23, <josef.pktd at gmail.com> wrote:

> On Wed, Feb 24, 2010 at 10:16 AM, Ivo Maljevic <ivo.maljevic at gmail.com>
> wrote:
> > One would think that you can always rely on wikipedia when it comes to
> math
> > and engineering, but it seems that is not tha case.
> > Josef, In the page you referenced, the SNR, or signal to noise ratio, is
> > defined as the ratio between the signal and noise powers. Consequently,
> in
> > terms of signals and standard deviations, it is defined as a ratio of the
> > average signal power and the noise variance (NOT its squre root, or
> standard
> > deviation). Or:
> >
> > SNR = P_s / sigma^2
> >
> > where P_s is the average signal power, and the noise variance is used to
> > measure the noise power. The assumption here is that the noise is a zero
> > mean process, otherwise variance and power wouldn't be the same thing.
>
> Note: I linked to #Statistical_definition not the top of the wikipedia page
> and I checked the source in scipy.stats:
>    Calculates the signal-to-noise ratio, defined as the ratio between the
> mean
>    and the standard deviation.
>
>    m = np.mean(a, axis)
>    sd = samplestd(a, axis)
>    return np.where(sd == 0, 0, m/sd)
>
> I didn't know about the different definitions until I read the
> Wikipedia page, but that's what's currently in scipy.stats
>
> Josef
>
> >
> > Nils, your question is way too generic for anyone to help you directly. I
> > can only point to you that your signal to noise ratio is quite low:
> >
> >>>> 10*math.log10(0.0447)
> > -13.496924768680636
> >
> > Maybe your signal is to narrow compared to the overal band you are
> working
> > with (or you have DS spread spectrum signal?).
> > Anyway, you will need to figure out which filter you want to use (e.g.,
> > butterworth for maximally flat characteristic in the passband, etc).
> >
> >
> > On 24 February 2010 09:33, <josef.pktd at gmail.com> wrote:
> >>
> >> On Wed, Feb 24, 2010 at 8:12 AM, Nils Wagner
> >> <nwagner at iam.uni-stuttgart.de> wrote:
> >> > Hi all,
> >> >
> >> > I have two questions concerning signal processing
> >> >
> >> > I have used scipy.stats.signaltonoise to compute the
> >> > signal-to-noise ratio.
> >> > The value is 0.0447.
> >> > How can I judge it ?
> >>
> >> It's just mean over standard deviation
> >>
> http://en.wikipedia.org/wiki/Signal-to-noise_ratio#Statistical_definition
> >>
> >> I never use it, but the interpretation will depend on what your
> >> level/mean/expected_value means.
> >>
> >> >
> >> > How can I filter out high frequencies using scipy ?
> >> > How can I eliminate noise from the signal ?
> >>
> >> (I'm no help here) There are many prefabricated filters in
> >> scipy.signal, but I only use lfilter.
> >>
> >> Josef
> >>
> >> >
> >> > Nils
> >> > _______________________________________________
> >> > SciPy-User mailing list
> >> > SciPy-User at scipy.org
> >> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >> >
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100224/8eeada17/attachment.html>

From warren.weckesser at enthought.com  Wed Feb 24 10:51:20 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Wed, 24 Feb 2010 09:51:20 -0600
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <web-130689370@uni-stuttgart.de>
References: <web-130689370@uni-stuttgart.de>
Message-ID: <4B854AF8.3060503@enthought.com>

Nils Wagner wrote:
> Hi all,
>
> I have two questions concerning signal processing
>
> I have used scipy.stats.signaltonoise to compute the 
> signal-to-noise ratio.
> The value is 0.0447.
> How can I judge it ?
>
> How can I filter out high frequencies using scipy ?
>   

I posted an example low-pass filtering using 'butter' and 'lfilter' from 
scipy.signal here:

    http://mail.scipy.org/pipermail/scipy-user/2010-January/024032.html

Warren

> How can I eliminate noise from the signal ?
>
> Nils
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From josef.pktd at gmail.com  Wed Feb 24 11:14:32 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 24 Feb 2010 11:14:32 -0500
Subject: [SciPy-User] Suggest moving curve_fit from minpack to top level
In-Reply-To: <3d375d731002240734n445c2424ved1d622432ca28cc@mail.gmail.com>
References: <7d3dada91002240607v7d9e3cd5n94008f4ea25b2738@mail.gmail.com>
	<1cd32cbb1002240618tf286f23g36fc185c18f22dea@mail.gmail.com>
	<dde7764a1002240631w5682aecfhea46b00ec4a72f41@mail.gmail.com>
	<1cd32cbb1002240648r60f9d86cmd77404023d81549e@mail.gmail.com>
	<3d375d731002240734n445c2424ved1d622432ca28cc@mail.gmail.com>
Message-ID: <1cd32cbb1002240814q4f11e62cs9e95ecc287b3ccb0@mail.gmail.com>

On Wed, Feb 24, 2010 at 10:34 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Wed, Feb 24, 2010 at 08:48, ?<josef.pktd at gmail.com> wrote:
>
>> curve_fit doesn't really fit into scipy.optimize next to all the low
>> level optimizers.
>>
>> Given the recent discussion, I think it would fit better in
>> scipy.interpolate which could be extended to interpolate and fitting
>> to noisy data, e.g. rbf, least squares splines.
>
> Heh. This is precisely why scipy.odr is in scipy.odr. When I asked
> where it should go, I got three responses and four opinions.
> scipy.optimize and scipy.stats are both plausible places for these
> routines; scipy.interpolate is plausible if you squint, but is not
> really in my opinion. I would like to keep scipy.interpolate for real
> interpolation methods. If those interpolation methods also have a
> closely related mode for fitting noisy data, so be it, but we
> shouldn't expand the scope to include all fitting methods that aren't
> interpolation. Similarly, curve_fit() and leastsq() are really just
> small wrappers around the underlying minimizers that do the heavy
> lifting, so they ended up in scipy.optimize.
>
> It might be time for a new top-level package scipy.fitting that brings
> together all of the noisy fitting solutions and provides a nice place
> for new code like Jonathan Stickel's. However, I would only want to do
> that if we were providing something else along with it, like unifying
> the interfaces or providing generic cross-validation routines that
> would work with any fitter that had the right interface, etc. I think
> that moving things around just to have a better organization is a
> waste of time and is not justified by the costs of deprecating working
> code and invalidating umpteen docs, tutorials, archived emails, blog
> posts, Stack Overflow answers that are floating out there, being all
> Googleable as outdated documentation is wont to be.

I mainly brought moving curve_fit up now, because it hasn't been
included in any official release yet. I don't really care where the
category is located, but it would be good to have a place where this
can be expanded.

The main reason I thought of scipy.interpolate is that smoothers are
also mainly focused on fitting points in the interval of the sample
points. For general fitting and estimation there will be some overlap
with statsmodels, e.g. eventually we will get a statsmodels version of
curve_fit, but with a different focus.
Having some generic cross-validation methods available would be very
useful, e.g. also for bandwith selection in kde.

For a full new subpackage, it might be better to go through a scikits
first, but it raises the barrier and work quite a bit. Since I'm not
doing much in the "smoother" area, I don't know how large this would
become, given that there are already noisy interpolators and the
filters in signal and ndimage.

(In statsmodels, we haven't started on non-parametric methods, so I
haven't looked into cross-validation yet, and also not for other model
selection.)

>
> Ultimately, I think that the organization of functions into packages
> is much less important than having quality documentation that answers
> users' "How do I ...?" questions. I know it's much easier to move a
> function into a new package than it is to write good docs, but it
> doesn't solve the real problem.

I fully agree with this

Josef

>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Wed Feb 24 11:30:33 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 24 Feb 2010 11:30:33 -0500
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <826c64da1002240747y148feae5o39a08743268b3989@mail.gmail.com>
References: <web-130689370@uni-stuttgart.de>
	<1cd32cbb1002240633n1a7c839ah9a214a59931e6800@mail.gmail.com>
	<826c64da1002240716p65ff7830wac050cb21d1b0322@mail.gmail.com>
	<1cd32cbb1002240723j89280c1i88af661ae85fe2f8@mail.gmail.com>
	<826c64da1002240747y148feae5o39a08743268b3989@mail.gmail.com>
Message-ID: <1cd32cbb1002240830q1b80284dk88d21f646a5415d2@mail.gmail.com>

On Wed, Feb 24, 2010 at 10:47 AM, Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
> Josef,
> Just to add, while one can define the "statistical SNR" as something else
> (that is, ratio E[s]/sigma_noise), what would be its used for? Astronomical
> optics?

I never use it, so I don't know.

>
> On the other hand, it doesn't really matter how it is defined in
> scipy.stats. Calculation of SNR is usually a bit more involved (you need to
> find the actual signal mean, and the noise variance), and I guess whoever
> needs to calculate the power based SNR will not call that function by
> mistake.

I was only referring to the original question that used
scipy.stats.signaltonoise

It only has a meaning if the level is well defined. If I demean the
observations first, then signaltonoise is infinite. If the level is my
annual salary, then stats.signaltonoise is essentially the inverse of
the coefficient of variation, stats.variation.

I'm just looking up definitions, not arguing about the appropriate
definitions and it's usefulness in different fields.

Cheers,

Josef

>
> Ivo
>
> On 24 February 2010 10:23, <josef.pktd at gmail.com> wrote:
>>
>> On Wed, Feb 24, 2010 at 10:16 AM, Ivo Maljevic <ivo.maljevic at gmail.com>
>> wrote:
>> > One would think that you can always rely on wikipedia when it comes to
>> > math
>> > and engineering, but it seems that is not tha case.
>> > Josef, In the page you referenced, the SNR, or signal to noise ratio, is
>> > defined as the ratio between the signal and noise powers. Consequently,
>> > in
>> > terms of signals and standard deviations, it is defined as a ratio of
>> > the
>> > average signal power and the noise variance (NOT its squre root, or
>> > standard
>> > deviation). Or:
>> >
>> > SNR = P_s / sigma^2
>> >
>> > where P_s is the average signal power, and the noise variance is used to
>> > measure the noise power. The assumption here is that the noise is a zero
>> > mean process, otherwise variance and power wouldn't be the same thing.
>>
>> Note: I linked to #Statistical_definition not the top of the wikipedia
>> page
>> and I checked the source in scipy.stats:
>> ? ?Calculates the signal-to-noise ratio, defined as the ratio between the
>> mean
>> ? ?and the standard deviation.
>>
>> ? ?m = np.mean(a, axis)
>> ? ?sd = samplestd(a, axis)
>> ? ?return np.where(sd == 0, 0, m/sd)
>>
>> I didn't know about the different definitions until I read the
>> Wikipedia page, but that's what's currently in scipy.stats
>>
>> Josef
>>
>> >
>> > Nils, your question is way too generic for anyone to help you directly.
>> > I
>> > can only point to you that your signal to noise ratio is quite low:
>> >
>> >>>> 10*math.log10(0.0447)
>> > -13.496924768680636
>> >
>> > Maybe your signal is to narrow compared to the overal band you are
>> > working
>> > with (or you have DS spread spectrum signal?).
>> > Anyway, you will need to figure out which filter you want to use (e.g.,
>> > butterworth for maximally flat characteristic in the passband, etc).
>> >
>> >
>> > On 24 February 2010 09:33, <josef.pktd at gmail.com> wrote:
>> >>
>> >> On Wed, Feb 24, 2010 at 8:12 AM, Nils Wagner
>> >> <nwagner at iam.uni-stuttgart.de> wrote:
>> >> > Hi all,
>> >> >
>> >> > I have two questions concerning signal processing
>> >> >
>> >> > I have used scipy.stats.signaltonoise to compute the
>> >> > signal-to-noise ratio.
>> >> > The value is 0.0447.
>> >> > How can I judge it ?
>> >>
>> >> It's just mean over standard deviation
>> >>
>> >> http://en.wikipedia.org/wiki/Signal-to-noise_ratio#Statistical_definition
>> >>
>> >> I never use it, but the interpretation will depend on what your
>> >> level/mean/expected_value means.
>> >>
>> >> >
>> >> > How can I filter out high frequencies using scipy ?
>> >> > How can I eliminate noise from the signal ?
>> >>
>> >> (I'm no help here) There are many prefabricated filters in
>> >> scipy.signal, but I only use lfilter.
>> >>
>> >> Josef
>> >>
>> >> >
>> >> > Nils
>> >> > _______________________________________________
>> >> > SciPy-User mailing list
>> >> > SciPy-User at scipy.org
>> >> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >> >
>> >> _______________________________________________
>> >> SciPy-User mailing list
>> >> SciPy-User at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> >
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> >
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From gokhansever at gmail.com  Wed Feb 24 12:10:57 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Wed, 24 Feb 2010 11:10:57 -0600
Subject: [SciPy-User] TimeSeries and Milliseconds for aircraft data
In-Reply-To: <43958ee61002232232s469353dtcfd8730ea83c76f3@mail.gmail.com>
References: <3848801.16841265888539566.JavaMail.root@mohawk.flightdataservices.com>
	<24755476.01266917706841.JavaMail.chris@jesse1c-laptop>
	<49d6b3501002230723x71ca6913y7aea5d2b1eecc7d0@mail.gmail.com>
	<43958ee61002232232s469353dtcfd8730ea83c76f3@mail.gmail.com>
Message-ID: <49d6b3501002240910q1a5d5f6t4627dba3f08ab1a4@mail.gmail.com>

On Wed, Feb 24, 2010 at 12:32 AM, Ariel Rokem <arokem at berkeley.edu> wrote:

> Hi everyone,
>
> Gokhan is referring to this:
>
> http://nipy.sourceforge.net/nitime/
>
> For an exposition, see our Scipy conference proceedings paper (a pdf of
> which can be found here:
> http://argentum.ucbso.berkeley.edu/papers/Rokem2009Nitime.pdf).
>
> We are still working on it. The intention of the library is to support
> analysis of data from neuroscience experiments, because we are
> neuroscientists, but so far, I don't think that we have made any design
> decisions that would preclude other scientists from using our time-series
> objects. In fact, the time-series objects we have designed support temporal
> resolutions as fast as picoseconds (the representation of time is done in
> int64, in order to avoid float-precision issues). It is still under
> development and we have yet to make a release of this, but the code (in
> development) is already available on github and the tests therein can direct
> you on the possible usage:
>
> http://github.com/fperez/nitime
>
> Cheers,
>
> Ariel
>
>
Hi Ariel,

What kind of interface do you use to measure pico-second resolutions? Are
you talking measurements from only one instrument at a time?

In our work even at 1 Hz levels we encounter issues like time-syncing
different measurements since we interface many different instruments with
one main data acquisition unit. This is mainly due to one instrument sits
under the far edge of a wing the other one is inside the cabin sampling air
from outside. It is usually a good idea to sample fastest the system and
probes permits, in the end they would be easily averaged to a lower
acceptable resolution range.

Your job should be very hard indeed if you are dealing with a couple
different instruments at that high measurement rates.


-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100224/4f043598/attachment.html>

From arokem at berkeley.edu  Wed Feb 24 12:36:24 2010
From: arokem at berkeley.edu (Ariel Rokem)
Date: Wed, 24 Feb 2010 09:36:24 -0800
Subject: [SciPy-User] TimeSeries and Milliseconds for aircraft data
In-Reply-To: <49d6b3501002240910q1a5d5f6t4627dba3f08ab1a4@mail.gmail.com>
References: <3848801.16841265888539566.JavaMail.root@mohawk.flightdataservices.com>
	<24755476.01266917706841.JavaMail.chris@jesse1c-laptop>
	<49d6b3501002230723x71ca6913y7aea5d2b1eecc7d0@mail.gmail.com> 
	<43958ee61002232232s469353dtcfd8730ea83c76f3@mail.gmail.com> 
	<49d6b3501002240910q1a5d5f6t4627dba3f08ab1a4@mail.gmail.com>
Message-ID: <43958ee61002240936q10a59c25n42bb084b7345d5c9@mail.gmail.com>

Hi Gokhan - easy - we are not dealing with the measurement part at all, only
with the analysis of data that has already been measured, so if you don't
think a pico-second representation is appropriate for your data (and it
probably is appropriate for very few kinds of data...), then use the right
temporal resolution for your data.

Best - Ariel

On Wed, Feb 24, 2010 at 9:10 AM, G?khan Sever <gokhansever at gmail.com> wrote:

>
>
> On Wed, Feb 24, 2010 at 12:32 AM, Ariel Rokem <arokem at berkeley.edu> wrote:
>
>> Hi everyone,
>>
>> Gokhan is referring to this:
>>
>> http://nipy.sourceforge.net/nitime/
>>
>> For an exposition, see our Scipy conference proceedings paper (a pdf of
>> which can be found here:
>> http://argentum.ucbso.berkeley.edu/papers/Rokem2009Nitime.pdf).
>>
>> We are still working on it. The intention of the library is to support
>> analysis of data from neuroscience experiments, because we are
>> neuroscientists, but so far, I don't think that we have made any design
>> decisions that would preclude other scientists from using our time-series
>> objects. In fact, the time-series objects we have designed support temporal
>> resolutions as fast as picoseconds (the representation of time is done in
>> int64, in order to avoid float-precision issues). It is still under
>> development and we have yet to make a release of this, but the code (in
>> development) is already available on github and the tests therein can direct
>> you on the possible usage:
>>
>> http://github.com/fperez/nitime
>>
>> Cheers,
>>
>> Ariel
>>
>>
> Hi Ariel,
>
> What kind of interface do you use to measure pico-second resolutions? Are
> you talking measurements from only one instrument at a time?
>
> In our work even at 1 Hz levels we encounter issues like time-syncing
> different measurements since we interface many different instruments with
> one main data acquisition unit. This is mainly due to one instrument sits
> under the far edge of a wing the other one is inside the cabin sampling air
> from outside. It is usually a good idea to sample fastest the system and
> probes permits, in the end they would be easily averaged to a lower
> acceptable resolution range.
>
> Your job should be very hard indeed if you are dealing with a couple
> different instruments at that high measurement rates.
>
>
> --
> G?khan
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Ariel Rokem
Helen Wills Neuroscience Institute
University of California, Berkeley
http://argentum.ucbso.berkeley.edu/ariel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100224/3b97c6b1/attachment.html>

From vincent at vincentdavis.net  Wed Feb 24 13:12:22 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Wed, 24 Feb 2010 11:12:22 -0700
Subject: [SciPy-User] List to record array problems, questions
In-Reply-To: <9457e7c81002232218k326b6260v208be37b711a09e8@mail.gmail.com>
References: <77e831101002232118gbae1671nb4cb65690be2cd2a@mail.gmail.com>
	<9457e7c81002232218k326b6260v208be37b711a09e8@mail.gmail.com>
Message-ID: <77e831101002241012s4168cab0n550a4cdb90b5c7d7@mail.gmail.com>

@St?fan van der Walt
"the constructor for structured arrays expect a list of tuples:"
Thanks you, I am not surprised it was simple.

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


2010/2/23 St?fan van der Walt <stefan at sun.ac.za>

> Hi Vincent
>
> On 24 February 2010 07:18, Vincent Davis <vincent at vincentdavis.net> wrote:
> >
> > I must be missing something simple. I have a list of lists data = "[['
>  0', '  0', '234.0', '24.0', ' 25'], ['  1', '  0', '22428.0', '2378.1', '
> 25'],......" and what to make a record array (I guess I have made a
> structured array) from it but it gets screwed up or I don't get it, likely
> both.
>
> The constructor for structured arrays expect a list of tuples:
>
> data_tuple = [tuple(row) for row in data]
> np.array(data_tuple, dtype=[('x', int), ('y', int), ('mean', float),
> ('stdv', float), ('npixels', int)])
>
> This yields
>
> array([(0, 0, 234.0, 24.0, 25), (1, 0, 22428.0, 2378.0999999999999, 25)],
>       dtype=[('x', '<i8'), ('y', '<i8'), ('mean', '<f8'), ('stdv',
> '<f8'), ('npixels', '<i8')])
>
> Regards
> St?fan
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100224/2246f0ad/attachment.html>

From jjstickel at vcn.com  Wed Feb 24 16:20:30 2010
From: jjstickel at vcn.com (Jonathan Stickel)
Date: Wed, 24 Feb 2010 14:20:30 -0700
Subject: [SciPy-User] monic option of orthogonal polynomials
Message-ID: <4B85981E.4050008@vcn.com>

What does the "monic" option do when calling an orthogonal polynomial 
function, e.g. special.legendre?  It does not seem to be documented, or 
at least I have not found it.

Thanks,
Jonathan


From vanforeest at gmail.com  Wed Feb 24 16:25:20 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Wed, 24 Feb 2010 22:25:20 +0100
Subject: [SciPy-User] monic option of orthogonal polynomials
In-Reply-To: <4B85981E.4050008@vcn.com>
References: <4B85981E.4050008@vcn.com>
Message-ID: <fa510ff81002241325n3b07904x7b78f4fb793a9e16@mail.gmail.com>

Hi,

The coefficient of a highest order term is 1 for a monic polynomial.
Might it be this?

bye

Nicky

On 24 February 2010 22:20, Jonathan Stickel <jjstickel at vcn.com> wrote:
> What does the "monic" option do when calling an orthogonal polynomial
> function, e.g. special.legendre? ?It does not seem to be documented, or
> at least I have not found it.
>
> Thanks,
> Jonathan
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From Debabrata.Midya at services.nsw.gov.au  Wed Feb 24 18:02:45 2010
From: Debabrata.Midya at services.nsw.gov.au (Debabrata Midya)
Date: Thu, 25 Feb 2010 10:02:45 +1100
Subject: [SciPy-User] Parallel Differential Evolution
Message-ID: <4B864AC4.5860.00D0.0@services.nsw.gov.au>

Hi SciPy-Users,
 
Thanks in advance.
 
I am Deb, Statistician at NSW Department of Services, Technology and Administration, Sydney, Australia.
 
I am new to this user group as well as to Python and Parallel Python. 
 
I am interested to use Differential Evolution (DE) using Parallel Python on Windows XP. I have some experience in DE using GNU GCC compiler.
 
Is there any software for it? If yes, anyone can assist me what are the softwares I need to install and their locations please.
 
Once again, thank you very much for the time you have given.
 
I am looking forward for your reply.
 
Regards,
 
Deb
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100225/302acae5/attachment.html>

From eric at depagne.org  Wed Feb 24 18:44:26 2010
From: eric at depagne.org (eric at depagne.org)
Date: Wed, 24 Feb 2010 15:44:26 -0800
Subject: [SciPy-User] Vector comparison
Message-ID: <201002241544.26791.eric@depagne.org>

Hi all, 

I have two sets of vectors, one made of 3-dim vectors and the other one of 4-
dims vectors.

I want to find for each of my 3D vectors (let's call them the input vector), 
which one, among my 4D vectors (let's call them my models), has the 3 first 
coordinates that are the closest of my input vector, so that i can assign to 
my input vector the 4th dimension taken from the models.

I am currently maximizing the cosine of two vectors, but I was wondering if 
scipy had some more clever routines, that could for instance take into account 
measurement errors for my vectors.

Thanks a lot.

?ric.

-- 
Un clavier azerty en vaut deux
----------------------------------------------------------


From rob.clewley at gmail.com  Wed Feb 24 18:47:04 2010
From: rob.clewley at gmail.com (Rob Clewley)
Date: Wed, 24 Feb 2010 18:47:04 -0500
Subject: [SciPy-User] Parallel Differential Evolution
In-Reply-To: <4B864AC4.5860.00D0.0@services.nsw.gov.au>
References: <4B864AC4.5860.00D0.0@services.nsw.gov.au>
Message-ID: <a749952d1002241547l647a71c6t80d16f948bd8b6c1@mail.gmail.com>

Hi,

Maybe the following is a useful implementation for you:

http://cci.lbl.gov/cctbx_sources/scitbx/differential_evolution.py

According to this file you need only the scitbx library, which
unfortunately looks non-trivial to install (it seems to be related to
Boost). However, this looks like an old code (stdlib no longer exists,
you'd just write "import random" now) and all the work done with
arrays using the scitbx library (index of minimum value, mean value,
etc.) can easily be done with the much more easily-installed numpy
library from scipy.org. You probably just need that, a simple graphing
package such as matplotlib, and a working environment such as Ipython.
Binary windows installers are available for all these from a simple
google search for "install X windows" where X is any of the above
packages (they will be found at sourceforge.net). Don't forget to
match the binary installer's version to your version of Python.

So instead of importing flex you'd import numpy and then make the
following substitutions (not tested):

numpy.zeros(X) instead of flex.double(X, 0)

1000*numpy.ones(X) instead of flex.double(X, 1000)

numpy.min(X) for flex.min(X), etc.

numpy.random.uniform(size=N) for flex.random_double(N)

If you get stuck post your code back here and someone can take a look.
Hope this helps,
Rob

On Wed, Feb 24, 2010 at 6:02 PM, Debabrata Midya
<Debabrata.Midya at services.nsw.gov.au> wrote:
> Hi SciPy-Users,
>
> Thanks in advance.
>
> I am Deb, Statistician at NSW Department of Services, Technology and
> Administration, Sydney, Australia.
>
> I am new to this user group?as well as to Python and Parallel Python.
>
> I am interested to use Differential Evolution (DE) using Parallel Python on
> Windows XP. I have some experience in DE using GNU GCC compiler.
>
> Is there any software for it? If yes, anyone can assist me what are the
> softwares I need to install and their locations please.
>
> Once again, thank you very much for the time you have given.
>
> I am looking forward for your reply.
>
> Regards,
>
> Deb
>


From robert.kern at gmail.com  Wed Feb 24 18:52:18 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 24 Feb 2010 17:52:18 -0600
Subject: [SciPy-User] Parallel Differential Evolution
In-Reply-To: <4B864AC4.5860.00D0.0@services.nsw.gov.au>
References: <4B864AC4.5860.00D0.0@services.nsw.gov.au>
Message-ID: <3d375d731002241552n74c91f3es95ba83beb656da56@mail.gmail.com>

On Wed, Feb 24, 2010 at 17:02, Debabrata Midya
<Debabrata.Midya at services.nsw.gov.au> wrote:
> Hi SciPy-Users,
>
> Thanks in advance.
>
> I am Deb, Statistician at NSW Department of Services, Technology and
> Administration, Sydney, Australia.
>
> I am new to this user group?as well as to Python and Parallel Python.
>
> I am interested to use Differential Evolution (DE) using Parallel Python on
> Windows XP. I have some experience in DE using GNU GCC compiler.
>
> Is there any software for it? If yes, anyone can assist me what are the
> softwares I need to install and their locations please.

Here is some serial code for DE:

http://svn.scipy.org/svn/scipy/branches/sandbox/scipy/sandbox/rkern/diffev.py

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From vincent at vincentdavis.net  Tue Feb 23 13:04:42 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Tue, 23 Feb 2010 11:04:42 -0700
Subject: [SciPy-User] list to structured or record array
Message-ID: <77e831101002231004k1fcaeaadw3e33be6816c3fbf6@mail.gmail.com>

I must be missing something simple. I have a list of lists data = "[['  0',
'  0', '234.0', '24.0', ' 25'], ['  1', '  0', '22428.0', '2378.1', '
25'],......" and what to make a record array (I guess I have made a
structured array) from it but it gets screwed up or I don't get it, likely
both.
Notice that at this stage the items are strings, not numbers, and there is
whitespace not sure this matters.

Here is what is happening, notice that the number get screwed up in the
structured array (not sure who to get a record array, I thought this was but
was told it is not)
adata = numpy.array(data,numpy.float64)

>>> adata
array([[  0.00000000e+00,   0.00000000e+00,   2.34000000e+02,
          2.40000000e+01,   2.50000000e+01],
       ...,
       [  4.77000000e+02,   4.77000000e+02,   2.07000000e+02,
          4.58000000e+01,   2.50000000e+01]])

This is what I would expect except it is not a record array.

This is not what I expect. I think I have tried every iteration including
using numpy dtaypes numpy.int32 or bdata = numpy.array(data, dtype = [('x',
int),('y', int),('mean',float),('stdv',float),('npixcels',int)])
What am I missing?

bdata = numpy.array(data, [('x', int),('y',
int),('mean',float),('stdv',float),('npixcels',int)])
>>> bdata
array([[(3153952, 0, 0.0, 0.0, 0), (3153952, 0, 0.0, 0.0, 0),
        (206933603122, 0, 0.0, 0.0, 0), (808334386, 0, 0.0, 0.0, 0),
        (3486240, 0, 0.0, 0.0, 0)],
       [(3219488, 0, 0.0, 0.0, 0), (3153952, 0, 0.0, 0.0, 0),
        (13561617777439282, 0, 0.0, 0.0, 0),
        (54074581398322, 0, 0.0, 0.0, 0), (3486240, 0, 0.0, 0.0, 0)],
       [(3285024, 0, 0.0, 0.0, 0), (3153952, 0, 0.0, 0.0, 0),
        (206933931058, 0, 0.0, 0.0, 0), (925775666, 0, 0.0, 0.0, 0),
        (3486240, 0, 0.0, 0.0, 0)],
       ...,
       [(3487540, 0, 0.0, 0.0, 0), (3618612, 0, 0.0, 0.0, 0),
        (206933602866, 0, 0.0, 0.0, 0), (908996661, 0, 0.0, 0.0, 0),
        (3486240, 0, 0.0, 0.0, 0)],
       [(3553076, 0, 0.0, 0.0, 0), (3618612, 0, 0.0, 0.0, 0),
        (13561596370041137, 0, 0.0, 0.0, 0),
        (62870573495603, 0, 0.0, 0.0, 0), (3486240, 0, 0.0, 0.0, 0)],
       [(3618612, 0, 0.0, 0.0, 0), (3618612, 0, 0.0, 0.0, 0),
        (206933798962, 0, 0.0, 0.0, 0), (942552372, 0, 0.0, 0.0, 0),
        (3486240, 0, 0.0, 0.0, 0)]],

      dtype=[('x', '<i8'), ('y', '<i8'), ('mean', '<f8'), ('stdv', '<f8'),
('npixcels', '<i8')])


  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100223/0c13d64f/attachment.html>

From jakevdp at gmail.com  Tue Feb 23 14:14:53 2010
From: jakevdp at gmail.com (Jake VanderPlas)
Date: Tue, 23 Feb 2010 11:14:53 -0800
Subject: [SciPy-User] preconditioned conjugate gradient
Message-ID: <58df6dc21002231114p5b152ffblbfbf06ca0336b26a@mail.gmail.com>

Hello,
I'm looking for a method to solve a sparse linear equation A*x=b,
where A is a NxN symmetric scipy.sparse.LinearOperator object, and b
is a 1D numpy vector.  The obvious choice would be something like
scipy.sparse.linalg.cg.  The problem is, the condition number of A is
very large - on order of 10^26.  From a search through relevant
literature, I know that matlab's preconditioned conjugate gradient
(pcg) routine works well for the type of problem I'm dealing with.  Is
there any similar routine in scipy?
I've looked at scipy.sparse.linalg.eigen.lobpcg, which seems to be
along the lines of what I need.  I could use this to find the inverse,
but that would involve computing an NxN dense matrix of eigenvectors,
which will cause memory problems in my case.  Any help would be
appreciated!
   -Jake


From bala1486 at gmail.com  Wed Feb 24 20:05:22 2010
From: bala1486 at gmail.com (Bala Chandar)
Date: Wed, 24 Feb 2010 17:05:22 -0800 (PST)
Subject: [SciPy-User] Bala Chandar wants to stay in touch on LinkedIn
Message-ID: <1117481250.56423.1267059922021.JavaMail.app@ech3-cdn08.prod>

LinkedIn
------------
   
I'd like to add you to my professional network on LinkedIn.

- Bala Chandar

Confirm that you know Bala Chandar
https://www.linkedin.com/e/isd/1101386560/jWSr8Hwq/EML-invg_56/


 
------
(c) 2010, LinkedIn Corporation
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100224/82992044/attachment.html>

From charlesr.harris at gmail.com  Wed Feb 24 20:27:34 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 24 Feb 2010 18:27:34 -0700
Subject: [SciPy-User] preconditioned conjugate gradient
In-Reply-To: <58df6dc21002231114p5b152ffblbfbf06ca0336b26a@mail.gmail.com>
References: <58df6dc21002231114p5b152ffblbfbf06ca0336b26a@mail.gmail.com>
Message-ID: <e06186141002241727j3a787cf9ncd049a24acfb8b43@mail.gmail.com>

On Tue, Feb 23, 2010 at 12:14 PM, Jake VanderPlas <jakevdp at gmail.com> wrote:

> Hello,
> I'm looking for a method to solve a sparse linear equation A*x=b,
> where A is a NxN symmetric scipy.sparse.LinearOperator object, and b
> is a 1D numpy vector.  The obvious choice would be something like
> scipy.sparse.linalg.cg.  The problem is, the condition number of A is
> very large - on order of 10^26.  From a search through relevant
> literature, I know that matlab's preconditioned conjugate gradient
> (pcg) routine works well for the type of problem I'm dealing with.  Is
> there any similar routine in scipy?
> I've looked at scipy.sparse.linalg.eigen.lobpcg, which seems to be
> along the lines of what I need.  I could use this to find the inverse,
> but that would involve computing an NxN dense matrix of eigenvectors,
> which will cause memory problems in my case.  Any help would be
> appreciated!
>

How did you compute the condition number and why is it so large?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100224/1b7997da/attachment.html>

From Spectre158 at gmail.com  Wed Feb 24 20:48:36 2010
From: Spectre158 at gmail.com (Scott Barlow)
Date: Thu, 25 Feb 2010 01:48:36 +0000 (UTC)
Subject: [SciPy-User] Scipy dblquad Calculation
References: <a7e4caf91002232343h3365207eja502a1ec79bea82c@mail.gmail.com>
	<4B850AA4.7040505@enthought.com>
Message-ID: <loom.20100225T023514-161@post.gmane.org>

Warren Weckesser <warren.weckesser <at> enthought.com> writes:
> 
> Scott,
> 
> As you suspected, the problem is your use of arctan, and an appropriate 
> use of arctan2 can correct it.  If I change this:
> 
>                            t*scipy.sin(theta + 
> scipy.arctan((a**2.0/b**2.0)*(1.0/scipy.tan(theta))))
> 
> to this:
> 
>                            t*scipy.sin(theta + scipy.arctan2( 
> a**2.0*scipy.cos(theta), b**2.0*scipy.sin(theta)) )
> 
> then the results agree.
> 
> Warren
> 

Ah, excellent.  Thank you very much for your help Warren!  When I tried using
arctan2, I didn't think about breaking up tangent into its sine and cosine
components and instead kept it as tangent or cotangent in the numerator or
denominator, as appropriate.

Thanks,
Scott




From yosefmel at post.tau.ac.il  Thu Feb 25 03:51:00 2010
From: yosefmel at post.tau.ac.il (Yosef Meller)
Date: Thu, 25 Feb 2010 10:51:00 +0200
Subject: [SciPy-User] Parallel Differential Evolution
In-Reply-To: <4B864AC4.5860.00D0.0@services.nsw.gov.au>
References: <4B864AC4.5860.00D0.0@services.nsw.gov.au>
Message-ID: <201002251051.00551.yosefmel@post.tau.ac.il>

On ??? ????? 25 ?????? 2010 01:02:45 Debabrata Midya wrote:
> Hi SciPy-Users,
> 
> Thanks in advance.
> 
> I am Deb, Statistician at NSW Department of Services, Technology and
>  Administration, Sydney, Australia.
> 
> I am new to this user group as well as to Python and Parallel Python.
> 
> I am interested to use Differential Evolution (DE) using Parallel Python on
>  Windows XP. I have some experience in DE using GNU GCC compiler.
> 
> Is there any software for it? If yes, anyone can assist me what are the
>  softwares I need to install and their locations please.
> 
> Once again, thank you very much for the time you have given.
> 
> I am looking forward for your reply.
> 
> Regards,
> 
> Deb
> 

I once wrote thhis implementation of the epsilon-moea algorithm for 
evolutionary multiobjective optimization, in Python and Matlab:

http://github.com/yosefm/eps-moea

Could give you some hints if you need, and if you already know Matlab you can 
compare.


From pav+sp at iki.fi  Thu Feb 25 04:02:37 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Thu, 25 Feb 2010 09:02:37 +0000 (UTC)
Subject: [SciPy-User] preconditioned conjugate gradient
References: <58df6dc21002231114p5b152ffblbfbf06ca0336b26a@mail.gmail.com>
Message-ID: <hm5ebd$64h$1@dough.gmane.org>

Tue, 23 Feb 2010 11:14:53 -0800, Jake VanderPlas wrote:
> I'm looking for a method to solve a sparse linear equation A*x=b, where
> A is a NxN symmetric scipy.sparse.LinearOperator object, and b is a 1D
> numpy vector.  The obvious choice would be something like
> scipy.sparse.linalg.cg.  The problem is, the condition number of A is
> very large - on order of 10^26.  From a search through relevant
> literature, I know that matlab's preconditioned conjugate gradient (pcg)
> routine works well for the type of problem I'm dealing with.  Is there
> any similar routine in scipy?

Supply a preconditioner M for the conjugate gradients:

	scipy.sparse.linalg.cg(A, b, M=M)

Matlab's `pcg` is completely identical to this, AFAIK -- also with it you 
have to supply a suitable preconditioner yourself.

For a suitable preconditioner, you may want to look at PyAmg [1] or try 
something simpler such as Jacobi or SOR. Right now, Scipy does not have 
interfaces to incomplete LU or Cholesky, though.

.. [1] http://code.google.com/p/pyamg/

-- 
Pauli Virtanen



From denis-bz-gg at t-online.de  Thu Feb 25 05:57:10 2010
From: denis-bz-gg at t-online.de (denis)
Date: Thu, 25 Feb 2010 02:57:10 -0800 (PST)
Subject: [SciPy-User] Vector comparison
In-Reply-To: <201002241544.26791.eric@depagne.org>
References: <201002241544.26791.eric@depagne.org>
Message-ID: <5d028d97-f0ec-492d-9da0-13514e7b4a19@o30g2000yqb.googlegroups.com>

On Feb 25, 12:44?am, e... at depagne.org wrote:
> Hi all,
>
> I have two sets of vectors, one made of 3-dim vectors and the other one of 4-
> dims vectors.
>
> I want to find for each of my 3D vectors (let's call them the input vector),
> which one, among my 4D vectors (let's call them my models), has the 3 first
> coordinates that are the closest of my input vector, so that i can assign to
> my input vector the 4th dimension taken from the models.

?ric.
   does this one-liner make sense:
    def nearvec( x, vecs ):
        """ the vec nearest x, -> the pair (norm(x - v), v) """
        return min( (np.linalg.norm( x - v[:3] ), v)  for v in vecs )
# py min, not np.min

Of course change "norm" to xT.A.x if you have an A that weights the
different components,
e.g. 1 / sigmaj^2.

Numpy experts can vectorize just about anything
(a wiki for vectorization examples would be nice, may well exist
already ?)

cheers
  -- denis


From denis-bz-gg at t-online.de  Thu Feb 25 09:48:29 2010
From: denis-bz-gg at t-online.de (denis)
Date: Thu, 25 Feb 2010 06:48:29 -0800 (PST)
Subject: [SciPy-User] noisy interpolators, smoothers and fitters
In-Reply-To: <1cd32cbb1002240711y47b48ae9tb62846b84af7edc2@mail.gmail.com>
References: <1cd32cbb1002240711y47b48ae9tb62846b84af7edc2@mail.gmail.com>
Message-ID: <f2ec2b15-fef7-49ed-9f2a-3be13b9de612@d27g2000yqf.googlegroups.com>

On Feb 24, 4:11?pm, josef.p... at gmail.com wrote:
> Is there a need for expanding this in scipy.interpolate?

Good idea; I'd realy like to see a 1-page overview of "noisy
interpolators smoothers and fitters".
Also how do developers see "state of the scipy interpolate" --
where are we, where's our market, niches vs Matlab / vs R.
A couple of wishes / comments from a nonexpert:

 - need a 1-page overview: the many interpolators
in .interpolate .signals .ndimage are confusing

 - fitpack seems powerful but opaque. projects.scipy.org/scipy/ticket/
864 a year ago:
    "The documentation for class scipy.interpolate.UnivariateSpline?
is misleading, and maybe completely wrong.
    UnivariateSpline? behaves in ways that are unpredictable ..."
    no followup, tough call: we have it but.

 - avoid methods with hidden parameters and no verbose=1
 - avoid interpolate.Rbf altogether -- very noisy, cf. NR3

> see also offer by Jonathan Stickel which looks useful with
> penalization on derivatives (if my quick reading is correct).

make a scikit ?
I like to see doc *first*
but that's just my preference, grew up on paper doc.

cheers
  -- denis


From jjstickel at vcn.com  Thu Feb 25 11:11:08 2010
From: jjstickel at vcn.com (Jonathan Stickel)
Date: Thu, 25 Feb 2010 09:11:08 -0700
Subject: [SciPy-User] monic option of orthogonal polynomials
In-Reply-To: <mailman.1348.1267056370.2398.scipy-user@scipy.org>
References: <mailman.1348.1267056370.2398.scipy-user@scipy.org>
Message-ID: <4B86A11C.1060706@vcn.com>

Yep, that must be it.  Thanks!

Jonathan


On 2/24/10 scipy-user-request at scipy.org wrote:
> Date: Wed, 24 Feb 2010 22:25:20 +0100
> From: nicky van foreest <vanforeest at gmail.com>
> Subject: Re: [SciPy-User] monic option of orthogonal polynomials
> To: SciPy Users List <scipy-user at scipy.org>
> 
> Hi,
> 
> The coefficient of a highest order term is 1 for a monic polynomial.
> Might it be this?
> 
> bye
> 
> Nicky
> 
> On 24 February 2010 22:20, Jonathan Stickel <jjstickel at vcn.com> wrote:
>> > What does the "monic" option do when calling an orthogonal polynomial
>> > function, e.g. special.legendre? ?It does not seem to be documented, or
>> > at least I have not found it.
>> >
>> > Thanks,
>> > Jonathan
>> > 



From kpalamartchouk at gmail.com  Thu Feb 25 12:26:23 2010
From: kpalamartchouk at gmail.com (kpal)
Date: Thu, 25 Feb 2010 09:26:23 -0800 (PST)
Subject: [SciPy-User] putmask and NaNs
Message-ID: <3ca62c72-5cd0-4660-b459-94ed905d6eba@d2g2000yqa.googlegroups.com>

Hi, I observe a strange thing. scipy.putmask seems to treat NaN as
zero in the following piece of code:

mydata = scipy.array(((1,2,3,4,5),(1,1,-1,1,-1))).T
scipy.putmask(mydata[:,0], mydata[:,1]<0, [scipy.NaN])

At this moment I expect some of the values in mydata to be replaced by
NaNs, but they get replaced by zeros. Is this behaviour normal? Or is
it a bug?

Or is there another way to put NaNs according to a mask?

Python 2.5, scipy 0.7.0

Thanks,


From josef.pktd at gmail.com  Thu Feb 25 12:38:42 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 25 Feb 2010 12:38:42 -0500
Subject: [SciPy-User] putmask and NaNs
In-Reply-To: <3ca62c72-5cd0-4660-b459-94ed905d6eba@d2g2000yqa.googlegroups.com>
References: <3ca62c72-5cd0-4660-b459-94ed905d6eba@d2g2000yqa.googlegroups.com>
Message-ID: <1cd32cbb1002250938n17de8824ne9472be26c4012c7@mail.gmail.com>

On Thu, Feb 25, 2010 at 12:26 PM, kpal <kpalamartchouk at gmail.com> wrote:
> Hi, I observe a strange thing. scipy.putmask seems to treat NaN as
> zero in the following piece of code:
>
> mydata = scipy.array(((1,2,3,4,5),(1,1,-1,1,-1))).T
> scipy.putmask(mydata[:,0], mydata[:,1]<0, [scipy.NaN])
>
> At this moment I expect some of the values in mydata to be replaced by
> NaNs, but they get replaced by zeros. Is this behaviour normal? Or is
> it a bug?
>
> Or is there another way to put NaNs according to a mask?

It's not a problem with putmask

integer arrays cannot contain nans so they are silently cast to zero.

If you need nans than either use a float array or represent nans as
-9999 or something like this, or use masked arrays.


Josef

>
> Python 2.5, scipy 0.7.0
>
> Thanks,
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From pgmdevlist at gmail.com  Thu Feb 25 12:40:24 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 25 Feb 2010 12:40:24 -0500
Subject: [SciPy-User] putmask and NaNs
In-Reply-To: <3ca62c72-5cd0-4660-b459-94ed905d6eba@d2g2000yqa.googlegroups.com>
References: <3ca62c72-5cd0-4660-b459-94ed905d6eba@d2g2000yqa.googlegroups.com>
Message-ID: <CDF5D9AA-D607-41DA-84A2-83712D38F50A@gmail.com>

On Feb 25, 2010, at 12:26 PM, kpal wrote:
> Hi, I observe a strange thing. scipy.putmask seems to treat NaN as
> zero in the following piece of code:
> 
> mydata = scipy.array(((1,2,3,4,5),(1,1,-1,1,-1))).T
> scipy.putmask(mydata[:,0], mydata[:,1]<0, [scipy.NaN])

Let me guess, ``mydata`` has an integer dtype, right ? You can't use NaNs in that case, they are transformed into 0, as you've observed. That's a Python thing on versions lower than 2.6 (in 2.6, you get an exception).
You can:
* use masked arrays with your integer array, or
* transform your integer array into floats with .astype(float)



From dominique.orban at gmail.com  Thu Feb 25 12:53:37 2010
From: dominique.orban at gmail.com (Dominique Orban)
Date: Thu, 25 Feb 2010 17:53:37 +0000
Subject: [SciPy-User] preconditioned conjugate gradient
Message-ID: <8793ae6e1002250953h24e3e52ei737434de84f78307@mail.gmail.com>

On Thu, Feb 25, 2010 at 5:38 PM, <scipy-user-request at scipy.org> wrote:

>
> ---------- Forwarded message ----------
> From: Jake VanderPlas <jakevdp at gmail.com>
> To: scipy-user at scipy.org
> Date: Tue, 23 Feb 2010 11:14:53 -0800
> Subject: [SciPy-User] preconditioned conjugate gradient
> Hello,
> I'm looking for a method to solve a sparse linear equation A*x=b,
> where A is a NxN symmetric scipy.sparse.LinearOperator object, and b
> is a 1D numpy vector.  The obvious choice would be something like
> scipy.sparse.linalg.cg.  The problem is, the condition number of A is
> very large - on order of 10^26.  From a search through relevant
> literature, I know that matlab's preconditioned conjugate gradient
> (pcg) routine works well for the type of problem I'm dealing with.  Is
> there any similar routine in scipy?
> I've looked at scipy.sparse.linalg.eigen.lobpcg, which seems to be
> along the lines of what I need.  I could use this to find the inverse,
> but that would involve computing an NxN dense matrix of eigenvectors,
> which will cause memory problems in my case.  Any help would be
> appreciated!
>   -Jake
>

Note that your matrix A must also be (positive or negative) definite, or the
conjugate gradient method may break down. If A is indefinite, you will
probably be safer with MINRES, CRAIG or SYMMLQ. As another reader pointed
out, you still need to supply a good preconditioner. Often, the application
suggests the kind of preconditioner that works well.

-- 
Dominique
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100225/24f9c32b/attachment.html>

From jjstickel at vcn.com  Thu Feb 25 12:54:23 2010
From: jjstickel at vcn.com (Jonathan Stickel)
Date: Thu, 25 Feb 2010 10:54:23 -0700
Subject: [SciPy-User] noisy interpolators, smoothers and fitters
In-Reply-To: <mailman.1539.1267119532.2398.scipy-user@scipy.org>
References: <mailman.1539.1267119532.2398.scipy-user@scipy.org>
Message-ID: <4B86B94F.3090902@vcn.com>

On 2/25/10 scipy-user-request at scipy.org wrote:
> Date: Thu, 25 Feb 2010 06:48:29 -0800 (PST)
> From: denis <denis-bz-gg at t-online.de>
> Subject: Re: [SciPy-User] noisy interpolators, smoothers and fitters
> 
> On Feb 24, 4:11?pm, josef.p... at gmail.com wrote:
>> > see also offer by Jonathan Stickel which looks useful with
>> > penalization on derivatives (if my quick reading is correct).
> 
> make a scikit ?
> I like to see doc *first*
> but that's just my preference, grew up on paper doc.
> 

Trying to follow along, and this is the first clear suggestion for what 
to do with my code.  I was not aware of scikits when I first made my 
inquiry.  That seems like a good place for now, and I will look into how 
to submit through that interface.

Denis:  in my previous post, I included a link to a publication on the 
method.  Email me offlist if you want me to send you a copy, or if you 
want to see the docstrings for my code.

Jonathan


From josef.pktd at gmail.com  Thu Feb 25 14:07:09 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 25 Feb 2010 14:07:09 -0500
Subject: [SciPy-User] stats.distributions skew, kurtosis bugs
Message-ID: <1cd32cbb1002251107p1a482b5fj3d06697b3a9c13ba@mail.gmail.com>

I found an old script of mine to check the skew and kurtosis of the
distributions in scipy.stats with bootstrapping, and I started to
produce a nicer printout.

The attached textfile shows all distributions where the stats() method
gives results that deviate by at least by 3 bootstrap standard
deviations from the bootstrap mean. This can be because the moment is
infinite or not defined, or because there is a bug. There are bugs in
some skew and kurtosis for sure, but I don't know which. Also, this is
a random test, so some false results might show up because of the
random sampling.

Bootstrap were generated by drawing an initial sample with
rvs(size=1000) from the distribution and then run a bootstrap sampling
until the bootstrap standard deviation settles down or a maximum of
30000 samples has been reached.

It's again a warning not to rely on higher moment in
stats.distributions without checking, and if anyone finds an incorrect
calculation or can verify false positives, then I would appreciate any
help.

Josef
-------------- next part --------------

Checking scipy.stats.distributions mean, variance, kurtosis with bootstrap


tstatistic for bias
tvalue = (bootstrapmean - diststatsvalue)/bootstrapstd

1.#IO = inf
1.#QNB = nan

inf in diststats usually means unbound mean or variance
nan in diststats can mean that the corresponding statistic does not exist (integral does not have defined value)



mean
distname,                  shapes,            t-values, bootstrapmean,   diststats
     alpha            (3.5704770516650459,):     -1.#IO     0.3050        1.#IO
    cauchy                               ():     -1.#IO     2.7770        1.#IO
foldcauchy            (4.7164673455831894,):     -1.#IO     7.4451        1.#IO
halfcauchy                               ():     -1.#IO     4.0620        1.#IO
      levy                               ():     -1.#IO   939.2252        1.#IO
    levy_l                               ():     -1.#IO -3147.6643        1.#IO

variance
distname,                  shapes,            t-values, bootstrapmean,   diststats
     alpha            (3.5704770516650459,):     -1.#IO     0.0181        1.#IO
    cauchy                               ():     -1.#IO  4515.7526        1.#IO
foldcauchy            (4.7164673455831894,):     -1.#IO   303.3553        1.#IO
halfcauchy                               ():     -1.#IO   407.1620        1.#IO
  invgamma            (2.0668996136993067,):   -34.6421     1.5085      13.1320
      levy                               ():     -1.#IO 167040678.2146        1.#IO
    levy_l                               ():     -1.#IO 1340119770.2983        1.#IO
     lomax            (1.8771398388773268,):     -1.#IO     5.0821        1.#IO
    pareto             (2.621716532144454,):    -3.2486     0.8993       1.6034
  powerlaw            (1.6591133289905851,):  -349.2695     0.0660       0.8586
     rdist           (0.90000000000000002,):  -106.6559     0.5350       1.7800
tukeylambda            (3.1321477856738267,):  -291.4355     0.0255       0.3048

skew
distname,                  shapes,            t-values, bootstrapmean,   diststats
     alpha            (3.5704770516650459,):     1.#QNB     4.5995       1.#QNB
    cauchy                               ():     1.#QNB    19.2907       1.#QNB
fatiguelife                            (29,):     9.2411     3.7254       0.0409
      fisk            (3.0857548622253179,):   -30.1339     4.9220      38.7939
foldcauchy            (4.7164673455831894,):     1.#QNB     9.8484       1.#QNB
  foldnorm            (1.9521253373555869,):   -13.2886     0.1708       0.9714
   gilbrat                               ():   -20.8618     2.8363       6.1849
halfcauchy                               ():     1.#QNB    14.5184       1.#QNB
  invgamma            (2.0668996136993067,):     7.3556     5.5156      -0.4777
      levy                               ():     1.#QNB    17.6494       1.#QNB
    levy_l                               ():     1.#QNB   -14.4021       1.#QNB
loglaplace            (3.2505926592051435,):   -30.3230     2.8614      16.9237
   lognorm           (0.95368226960575331,):    -7.8588     3.3619       5.4597
     lomax            (1.8771398388773268,):     1.#QNB     7.4517       1.#QNB
    mielke       (10.4, 3.6000000000000001):    -6.3266     3.2524       7.5954
       ncf    (27, 27, 0.41578441799226107): -301743359927.9626     1.0067 40747519832.6876
    pareto             (2.621716532144454,):     1.#QNB     5.4505       1.#QNB
  powerlaw            (1.6591133289905851,):     9.7330    -0.4061      -0.9070
         t            (2.7433514990818093,):     1.#QNB    -2.1069       1.#QNB

kurtosis
distname,                  shapes,            t-values, bootstrapmean,   diststats
     alpha            (3.5704770516650459,):     1.#QNB    36.1850       1.#QNB
      burr       (10.5, 4.2999999999999998): -25059.8725     8.0754  112616.2702
    cauchy                               ():     1.#QNB   511.4940       1.#QNB
  dweibull            (2.0685080649914673,):   -58.0690    -1.0984       1.9089
      fisk            (3.0857548622253179,):    17.6460    44.9028    -224.6593
foldcauchy            (4.7164673455831894,):     1.#QNB   118.0841       1.#QNB
  foldnorm            (1.9521253373555869,):   -27.1183    -0.3591       2.7052
 genpareto           (0.10000000000000001,):    -3.0045     8.8065      14.8286
   gilbrat                               ():   -79.2412    10.0417     110.9364
halfcauchy                               ():     1.#QNB   300.8983       1.#QNB
  invgamma            (2.0668996136993067,):     3.6372    45.5398      -2.8666
      levy                               ():     1.#QNB   349.5211       1.#QNB
    levy_l                               ():     1.#QNB   228.6728       1.#QNB
loglaplace            (3.2505926592051435,):    39.7340    15.5471    -164.3326
   lognorm           (0.95368226960575331,):   -23.8260    15.3252      81.1354
     lomax            (1.8771398388773268,):     1.#QNB    81.6275       1.#QNB
    mielke       (10.4, 3.6000000000000001):    21.3019    20.6964    -149.4051
       ncf    (27, 27, 0.41578441799226107): 357335608038.1621     1.6098 -239984516632.6775
       nct        (14, 0.24045031331198066): 2064492.8697     0.5513 -409040.4071
    pareto             (2.621716532144454,):     1.#QNB    49.1351       1.#QNB
     rdist           (0.90000000000000002,):    42.8105    -1.5425      -2.5679
         t            (2.7433514990818093,):     1.#QNB    30.1689       1.#QNB
tukeylambda            (3.1321477856738267,):    38.5327    -0.8835      -2.9837

From eric at depagne.org  Thu Feb 25 14:16:02 2010
From: eric at depagne.org (=?iso-8859-1?q?=C9ric_Depagne?=)
Date: Thu, 25 Feb 2010 11:16:02 -0800
Subject: [SciPy-User] Vector comparison
In-Reply-To: <5d028d97-f0ec-492d-9da0-13514e7b4a19@o30g2000yqb.googlegroups.com>
References: <201002241544.26791.eric@depagne.org>
	<5d028d97-f0ec-492d-9da0-13514e7b4a19@o30g2000yqb.googlegroups.com>
Message-ID: <201002251116.02649.eric@depagne.org>

Hi Denis, 
> ?ric.
>    does this one-liner make sense:
>     def nearvec( x, vecs ):
>         """ the vec nearest x, -> the pair (norm(x - v), v) """
>         return min( (np.linalg.norm( x - v[:3] ), v)  for v in vecs )
> # py min, not np.min
I'll have a look at it and see how different the results are from my own 
method.
> 
> Of course change "norm" to xT.A.x if you have an A that weights the
> different components,
> e.g. 1 / sigmaj^2.
Thanks for this, I was looking desperately to find how to include the weights.

> 
> Numpy experts can vectorize just about anything
> (a wiki for vectorization examples would be nice, may well exist
> already ?)
Cheers, 
?ric.
> 
> cheers
>   -- denis
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 

-- 
Un clavier azerty en vaut deux
----------------------------------------------------------
?ric Depagne                            edepagne at lcogt.net


From tsyu80 at gmail.com  Thu Feb 25 14:26:29 2010
From: tsyu80 at gmail.com (Tony S Yu)
Date: Thu, 25 Feb 2010 14:26:29 -0500
Subject: [SciPy-User] typo in odeint docstring
Message-ID: <AE6849BD-D733-4613-B8D6-2474BB0B6128@gmail.com>

I think the shape of the solutions returned by scipy.integrate.odeint is incorrectly documented: the number of rows matches the number of time steps (in the docstring, N_rows = len(y0))

Example:

>>> from scipy import integrate
>>> import numpy as np
>>> 
>>> x0 = np.ones(5)
>>> dxdt = lambda x, t: x + 1
>>> times = np.linspace(0, 1, 10)
>>> print integrate.odeint(dxdt, x0, times).shape
(10, 5)

Cheers,
-Tony

PS: I know there's some sort of doc editor for scipy. Should I be making the change there instead of emailing the list?

Patch:

Index: scipy/integrate/odepack.py
===================================================================
--- scipy/integrate/odepack.py	(revision 6250)
+++ scipy/integrate/odepack.py	(working copy)
@@ -57,7 +57,7 @@
 
     Returns
     -------
-    y : array, shape (len(y0), len(t))
+    y : array, shape (len(t), len(y0))
         Array containing the value of y for each desired time in t,
         with the initial value y0 in the first row.



From robert.kern at gmail.com  Thu Feb 25 14:30:29 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 25 Feb 2010 13:30:29 -0600
Subject: [SciPy-User] typo in odeint docstring
In-Reply-To: <AE6849BD-D733-4613-B8D6-2474BB0B6128@gmail.com>
References: <AE6849BD-D733-4613-B8D6-2474BB0B6128@gmail.com>
Message-ID: <3d375d731002251130j68953990p2592ff7e630b4c45@mail.gmail.com>

On Thu, Feb 25, 2010 at 13:26, Tony S Yu <tsyu80 at gmail.com> wrote:

> PS: I know there's some sort of doc editor for scipy. Should I be making the change there instead of emailing the list?

http://docs.scipy.org/scipy/docs/scipy.integrate.odepack.odeint/#scipy-integrate-odeint

You will need to create an account and post here asking for write
permission (giving your chosen account name) in order to edit.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From tsyu80 at gmail.com  Thu Feb 25 14:47:15 2010
From: tsyu80 at gmail.com (Tony S Yu)
Date: Thu, 25 Feb 2010 14:47:15 -0500
Subject: [SciPy-User] typo in odeint docstring
In-Reply-To: <3d375d731002251130j68953990p2592ff7e630b4c45@mail.gmail.com>
References: <AE6849BD-D733-4613-B8D6-2474BB0B6128@gmail.com>
	<3d375d731002251130j68953990p2592ff7e630b4c45@mail.gmail.com>
Message-ID: <172B8BE8-67D2-49D7-9FBD-DD2DB385EC74@gmail.com>


On Feb 25, 2010, at 2:30 PM, Robert Kern wrote:

> On Thu, Feb 25, 2010 at 13:26, Tony S Yu <tsyu80 at gmail.com> wrote:
> 
>> PS: I know there's some sort of doc editor for scipy. Should I be making the change there instead of emailing the list?
> 
> http://docs.scipy.org/scipy/docs/scipy.integrate.odepack.odeint/#scipy-integrate-odeint
> 
> You will need to create an account and post here asking for write
> permission (giving your chosen account name) in order to edit.

I just registered as tsyu. Could I please have edit permissions? :)

Thanks,
-Tony

> -- 
> Robert Kern
> 
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From gael.varoquaux at normalesup.org  Thu Feb 25 15:07:30 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 25 Feb 2010 21:07:30 +0100
Subject: [SciPy-User] typo in odeint docstring
In-Reply-To: <172B8BE8-67D2-49D7-9FBD-DD2DB385EC74@gmail.com>
References: <AE6849BD-D733-4613-B8D6-2474BB0B6128@gmail.com>
	<3d375d731002251130j68953990p2592ff7e630b4c45@mail.gmail.com>
	<172B8BE8-67D2-49D7-9FBD-DD2DB385EC74@gmail.com>
Message-ID: <20100225200730.GA3397@phare.normalesup.org>

On Thu, Feb 25, 2010 at 02:47:15PM -0500, Tony S Yu wrote:

> On Feb 25, 2010, at 2:30 PM, Robert Kern wrote:

> > On Thu, Feb 25, 2010 at 13:26, Tony S Yu <tsyu80 at gmail.com> wrote:

> >> PS: I know there's some sort of doc editor for scipy. Should I be making the change there instead of emailing the list?

> > http://docs.scipy.org/scipy/docs/scipy.integrate.odepack.odeint/#scipy-integrate-odeint

> > You will need to create an account and post here asking for write
> > permission (giving your chosen account name) in order to edit.

> I just registered as tsyu. Could I please have edit permissions? :)

Done.

Gael


From jturner at gemini.edu  Wed Feb 24 21:34:39 2010
From: jturner at gemini.edu (James Turner)
Date: Wed, 24 Feb 2010 23:34:39 -0300
Subject: [SciPy-User] =?windows-1252?q?=5BFwd=3A_New_physicsworld=2Ecom_we?=
 =?windows-1252?q?binar_=96_Why_software_fails=2C_and_how_to_fix_it=5D?=
Message-ID: <4B85E1BF.9070404@gemini.edu>

Hi everyone (long time no post),

I thought some of you might be interested in this Web seminar on
scientific software... As far as I can tell it is open to non-members.

Cheers,

James.

-------- Original Message --------
Subject: New physicsworld.com webinar ? Why software fails, and how to fix it
Date: Thu, 25 Feb 2010 02:22:11 +0000
From: <removed to avoid spam>

We are pleased to announce the latest physicsworld.com webinar sponsored by 
Tessella.

A Crack in the Code: Why software fails in scientific research, and how to fix it

Date: Thursday 25 March 2010

Time: 3 p.m. GMT (4 p.m. Central Europe, 11 a.m. East Coast US)

Where: online - registration is FREE at
http://physicsworld.com/cws/go/webinar9

Why you should attend:

Millions of lines of code have been developed to support scientific research. 
Although an increasingly important part of almost all research projects, most of 
this software is barely fit for purpose compared with equivalent systems in the 
commercial world. The code is hard to understand and maintain, lacking 
documentation and version control, and is continually "re-invented" as the 
software developers move on to new projects.

This webinar will investigate how this situation has come about, why it is 
important to the future of research, and what can be done about it.

Reserve your free place now - follow this link 
http://physicsworld.com/cws/go/webinar9

Kind regards

Angela Gage
Marketing executive
http://physicsworld.com
*Data Protection*
The Institute of Physics (and other companies in its group, including IOP 
Publishing Limited) may like to send you further notifications like this.  If 
you would prefer not to receive these, then please reply to this e-mail with the 
word "unsubscribe" in the subject line.  We will never rent or sell your e-mail 
address to any third parties.


From wesmckinn at gmail.com  Thu Feb 25 16:19:19 2010
From: wesmckinn at gmail.com (Wes McKinney)
Date: Thu, 25 Feb 2010 16:19:19 -0500
Subject: [SciPy-User] Moving median with O(log n) updates
Message-ID: <6c476c8a1002251319s3a8b91a2r967529000a73ca87@mail.gmail.com>

At Pycon this past weekend Raymond Hettinger gave a nice talk about
leveraging Python's built-in data structures to implement various
algorithms in a clear and elegant way.

http://us.pycon.org/2010/conference/schedule/event/86/
video: http://pycon.blip.tv/file/3261295/

He posted a full implementation of a the moving median using a
skiplist here (MIT license):

http://code.activestate.com/recipes/576930-efficient-running-median-using-an-indexable-skipli/

Some time ago for pandas I had implemented moving median is a fairly
naive way (O(n * window)) using the linear-time wirth method
(quickselect would provide equivalent performance). I decided to
translate the implementation above into Cython and tailor it for
ndarray input and float data. In most cases it achieves 5-10x speedup
over the pure Python O(n * log(window)) implementation, and clearly
trouncing other implementations. Here it is:

http://code.google.com/p/pandas/source/browse/trunk/pandas/lib/src/skiplist.pyx

(the actual rolling_median is in moments.pyx)

and some timings:

In [3]: import pandas

In [4]: arr = randn(10000)

In [5]: %timeit pandas.rolling_median(arr, 1000)
1 loops, best of 3: 401 ms per loop

of course this is roughly linear in the window size:

In [11]: %time pandas.rolling_median(arr, 5000)
CPU times: user 1.07 s, sys: 0.00 s, total: 1.07 s
Wall time: 1.06 s

new version (not yet ready for primetime with NaN-handling etc.)

In [6]: import pandas.lib.tseries as ts

In [7]: %timeit ts.rolling_median(arr, 1000)
10 loops, best of 3: 120 ms per loop

Larger window size in this case has minimal effect:

In [13]: %timeit ts.rolling_median(arr, 5000)
10 loops, best of 3: 120 ms per loop

I'm pretty sure scikits.timeseries is doing something O(n * window^2)

In [8]: import scikits.timeseries.lib as l

In [9]: %time l.mov_median(arr, 1000)
CPU times: user 86.48 s, sys: 0.00 s, total: 86.48 s
Wall time: 86.48 s

I had a fun time with Cython making the implementation fairly good,
but I have the feeling that there's too much Python list access (which
is faster than accessing an object ndarray using the buffer interface
I found). If someone with better C / Cython skills wants to take a
crack at speeding it up even more please be my guest.

- Wes


From guilherme at gpfreitas.com  Thu Feb 25 20:17:05 2010
From: guilherme at gpfreitas.com (Guilherme P. de Freitas)
Date: Thu, 25 Feb 2010 17:17:05 -0800
Subject: [SciPy-User] Lagrange Multipliers in optimize.slsqp
Message-ID: <a193bbb51002251717j3ab63e73heebcf4f2e229a73e@mail.gmail.com>

Hi all,

Would it be possible for optimize.slsqp to return the Lagrange
multipliers of the constraints? I don't know Fortran, but it seems
that they are available in the original Fortran code:

http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp/slsqp_optmz.f#L581

Thanks,

Guilherme

-- 
Guilherme P. de Freitas
http://www.gpfreitas.com


From jsseabold at gmail.com  Thu Feb 25 23:08:55 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 25 Feb 2010 23:08:55 -0500
Subject: [SciPy-User] Lagrange Multipliers in optimize.slsqp
In-Reply-To: <a193bbb51002251717j3ab63e73heebcf4f2e229a73e@mail.gmail.com>
References: <a193bbb51002251717j3ab63e73heebcf4f2e229a73e@mail.gmail.com>
Message-ID: <c048da1c1002252008i124597dbl476c6e6dc95e96b0@mail.gmail.com>

On Thu, Feb 25, 2010 at 8:17 PM, Guilherme P. de Freitas
<guilherme at gpfreitas.com> wrote:
> Hi all,
>
> Would it be possible for optimize.slsqp to return the Lagrange
> multipliers of the constraints? I don't know Fortran, but it seems
> that they are available in the original Fortran code:
>
> http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp/slsqp_optmz.f#L581

That looks like the constraint of the the lsq subroutine part of
sequential least squares algorithm.

http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp/slsqp_optmz.f#L164

Skipper


From abc.mikey at googlemail.com  Fri Feb 26 08:10:09 2010
From: abc.mikey at googlemail.com (mikey)
Date: Fri, 26 Feb 2010 13:10:09 +0000
Subject: [SciPy-User] Pixel Offset example no longer working
Message-ID: <4ebfceac1002260510p2f32b1e8lb9200a046afe1dd5@mail.gmail.com>

Hi there,

Just wondering if anyone knows how to do a pixel offset in matplotlib?
I've tried running the example in
http://www.scipy.org/Cookbook/Matplotlib/Transformations#line-58 but
it appears to have been trashed by changes to the api. This is a very
useful function if you're doing discrete data plots like I am and you
want uniform clusters of points.

Thanks for your time.

Regards,


Mikey


From josef.pktd at gmail.com  Fri Feb 26 08:55:50 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 26 Feb 2010 08:55:50 -0500
Subject: [SciPy-User] Lagrange Multipliers in optimize.slsqp
In-Reply-To: <c048da1c1002252008i124597dbl476c6e6dc95e96b0@mail.gmail.com>
References: <a193bbb51002251717j3ab63e73heebcf4f2e229a73e@mail.gmail.com>
	<c048da1c1002252008i124597dbl476c6e6dc95e96b0@mail.gmail.com>
Message-ID: <1cd32cbb1002260555k3b72a50bsc2e606c9f0e68265@mail.gmail.com>

On Thu, Feb 25, 2010 at 11:08 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Thu, Feb 25, 2010 at 8:17 PM, Guilherme P. de Freitas
> <guilherme at gpfreitas.com> wrote:
>> Hi all,
>>
>> Would it be possible for optimize.slsqp to return the Lagrange
>> multipliers of the constraints? I don't know Fortran, but it seems
>> that they are available in the original Fortran code:
>>
>> http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp/slsqp_optmz.f#L581
>
> That looks like the constraint of the the lsq subroutine part of
> sequential least squares algorithm.
>
> http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp/slsqp_optmz.f#L164


here is the workspace `w` description of the main slsqp routine

http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp/slsqp_optmz.f#L144

w is available in python, but non of the content of it is used in python

http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp.py#L345

So, correctly extracting the slices from the workspace w should be
possible to get the Lagrange multipliers that were used.


Josef


> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From robert.kern at gmail.com  Fri Feb 26 10:00:20 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 26 Feb 2010 09:00:20 -0600
Subject: [SciPy-User] Pixel Offset example no longer working
In-Reply-To: <4ebfceac1002260510p2f32b1e8lb9200a046afe1dd5@mail.gmail.com>
References: <4ebfceac1002260510p2f32b1e8lb9200a046afe1dd5@mail.gmail.com>
Message-ID: <3d375d731002260700p580a50c6t482a77d4b3f96a3d@mail.gmail.com>

On Fri, Feb 26, 2010 at 07:10, mikey <abc.mikey at googlemail.com> wrote:
> Hi there,
>
> Just wondering if anyone knows how to do a pixel offset in matplotlib?
> I've tried running the example in
> http://www.scipy.org/Cookbook/Matplotlib/Transformations#line-58 but
> it appears to have been trashed by changes to the api. This is a very
> useful function if you're doing discrete data plots like I am and you
> want uniform clusters of points.

The matplotlib list is over here:

  https://lists.sourceforge.net/lists/listinfo/matplotlib-users

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jsseabold at gmail.com  Fri Feb 26 10:35:59 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Fri, 26 Feb 2010 10:35:59 -0500
Subject: [SciPy-User] Lagrange Multipliers in optimize.slsqp
In-Reply-To: <1cd32cbb1002260555k3b72a50bsc2e606c9f0e68265@mail.gmail.com>
References: <a193bbb51002251717j3ab63e73heebcf4f2e229a73e@mail.gmail.com> 
	<c048da1c1002252008i124597dbl476c6e6dc95e96b0@mail.gmail.com> 
	<1cd32cbb1002260555k3b72a50bsc2e606c9f0e68265@mail.gmail.com>
Message-ID: <c048da1c1002260735p44263cc3gc394b403f8f2100@mail.gmail.com>

On Fri, Feb 26, 2010 at 8:55 AM,  <josef.pktd at gmail.com> wrote:
> On Thu, Feb 25, 2010 at 11:08 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> On Thu, Feb 25, 2010 at 8:17 PM, Guilherme P. de Freitas
>> <guilherme at gpfreitas.com> wrote:
>>> Hi all,
>>>
>>> Would it be possible for optimize.slsqp to return the Lagrange
>>> multipliers of the constraints? I don't know Fortran, but it seems
>>> that they are available in the original Fortran code:
>>>
>>> http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp/slsqp_optmz.f#L581
>>
>> That looks like the constraint of the the lsq subroutine part of
>> sequential least squares algorithm.
>>
>> http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp/slsqp_optmz.f#L164
>
>
> here is the workspace `w` description of the main slsqp routine
>
> http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp/slsqp_optmz.f#L144
>
> w is available in python, but non of the content of it is used in python
>
> http://projects.scipy.org/scipy/browser/trunk/scipy/optimize/slsqp.py#L345
>
> So, correctly extracting the slices from the workspace w should be
> possible to get the Lagrange multipliers that were used.
>

Ah, this should be enough then to return the multipliers.  See if
it gives what you'd expect?  Then you can file an enhancement ticket
if you want the multipliers back.

Index: slsqp.py
===================================================================
--- slsqp.py    (revision 6242)
+++ slsqp.py    (working copy)
@@ -371,5 +371,6 @@
         return [list(x),
                 float(fx),
                 int(majiter),
-                int(mode),
+                list(w[:m]),
+                int(mode),
                 exit_modes[int(mode)] ]

Skipper


From abc.mikey at googlemail.com  Fri Feb 26 10:39:07 2010
From: abc.mikey at googlemail.com (mikey)
Date: Fri, 26 Feb 2010 15:39:07 +0000
Subject: [SciPy-User] Pixel Offset example no longer working
In-Reply-To: <3d375d731002260700p580a50c6t482a77d4b3f96a3d@mail.gmail.com>
References: <4ebfceac1002260510p2f32b1e8lb9200a046afe1dd5@mail.gmail.com>
	<3d375d731002260700p580a50c6t482a77d4b3f96a3d@mail.gmail.com>
Message-ID: <4ebfceac1002260739k387be815t25b5961736cbb325@mail.gmail.com>

Thanks, too used to thinking of Scipy and Matplotlib as being parts of
the same thing.

On 26 February 2010 15:00, Robert Kern <robert.kern at gmail.com> wrote:
> On Fri, Feb 26, 2010 at 07:10, mikey <abc.mikey at googlemail.com> wrote:
>> Hi there,
>>
>> Just wondering if anyone knows how to do a pixel offset in matplotlib?
>> I've tried running the example in
>> http://www.scipy.org/Cookbook/Matplotlib/Transformations#line-58 but
>> it appears to have been trashed by changes to the api. This is a very
>> useful function if you're doing discrete data plots like I am and you
>> want uniform clusters of points.
>
> The matplotlib list is over here:
>
> ?https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From nwagner at iam.uni-stuttgart.de  Fri Feb 26 14:05:08 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 26 Feb 2010 20:05:08 +0100
Subject: [SciPy-User] frequency components of a signal buried in a noisy
 time domain signal
Message-ID: <web-130795898@uni-stuttgart.de>

Hi all,

A common use of Fourier transforms is to find the 
frequency components of a signal buried in a noisy time 
domain signal.

I found a Matlab template at

http://www.mathworks.com/access/helpdesk/help/techdoc/ref/fft.shtml

Matlab has a function

nextpow2(L)

Is there a similar build-in function in numpy/scipy ?
  
I tried to convert the m-file into a pythonic form.
What is needed to obtain a similar figure of the 
single-sided amplitude spectrum using 
numpy/scipy/matplotlib ?


from numpy import sin, linspace, pi
from numpy.random import randn
from pylab import plot, show, title, xlabel, ylabel
from scipy.fft import fft

Fs = 1000. #                          Sampling frequency
T  = 1./Fs #                          Sample time
L  = 1000  #                          length of signal
t  = arange(0,L)*T
x  = 0.7*sin(2*pi*50*t)+sin(2*pi*120*t)
y  = x + 2*randn(len(t))
plot(Fs*t[:50],y[:50])
title('Signal corrupted with zero-mean random noise')
xlabel('Time (milliseconds)')
#
#
#
#NFFT = 2^nextpow2(L); # Next power of 2 from length of y

Y = fft(y,NFFT)/L
f = Fs/2*linspace(0,1,NFFT/2+1)
plot(f,2*abs(Y[:NFFT/2+1]))
title('Single-sided amplitude spectrum of y(t)')
xlabel('Frequency (Hz)')
ylabel('|Y(f)|')
show()


What can be done in case of nonequispaced data ?

http://dx.doi.org/10.1137/0914081

Thanks in advance

                     Nils


From ivo.maljevic at gmail.com  Fri Feb 26 15:56:29 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Fri, 26 Feb 2010 15:56:29 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <web-130795898@uni-stuttgart.de>
References: <web-130795898@uni-stuttgart.de>
Message-ID: <826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>

Nils,
Yes, FFT can be used to visualize the frequency content of a signal buried
in the noise, especially if it is is narrowband,
even though more advanced spectral analysis is required for advanced
applications.

I cannot help you with non-uniformly spaced samples (some sort of
interpolation comes to mind), but next power of 2 should be trivial. Without
checking for the argument type, you can implement the function like this:

def nextpow2(n):
    m_f = np.log2(n)
    m_i = np.ceil(m_f)
    return 2**m_i


Hope it helps,
Ivo

On 26 February 2010 14:05, Nils Wagner <nwagner at iam.uni-stuttgart.de> wrote:

> Hi all,
>
> A common use of Fourier transforms is to find the
> frequency components of a signal buried in a noisy time
> domain signal.
>
> I found a Matlab template at
>
> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/fft.shtml
>
> Matlab has a function
>
> nextpow2(L)
>
> Is there a similar build-in function in numpy/scipy ?
>
> I tried to convert the m-file into a pythonic form.
> What is needed to obtain a similar figure of the
> single-sided amplitude spectrum using
> numpy/scipy/matplotlib ?
>
>
> from numpy import sin, linspace, pi
> from numpy.random import randn
> from pylab import plot, show, title, xlabel, ylabel
> from scipy.fft import fft
>
> Fs = 1000. #                          Sampling frequency
> T  = 1./Fs #                          Sample time
> L  = 1000  #                          length of signal
> t  = arange(0,L)*T
> x  = 0.7*sin(2*pi*50*t)+sin(2*pi*120*t)
> y  = x + 2*randn(len(t))
> plot(Fs*t[:50],y[:50])
> title('Signal corrupted with zero-mean random noise')
> xlabel('Time (milliseconds)')
> #
> #
> #
> #NFFT = 2^nextpow2(L); # Next power of 2 from length of y
>
> Y = fft(y,NFFT)/L
> f = Fs/2*linspace(0,1,NFFT/2+1)
> plot(f,2*abs(Y[:NFFT/2+1]))
> title('Single-sided amplitude spectrum of y(t)')
> xlabel('Frequency (Hz)')
> ylabel('|Y(f)|')
> show()
>
>
> What can be done in case of nonequispaced data ?
>
> http://dx.doi.org/10.1137/0914081
>
> Thanks in advance
>
>                     Nils
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100226/fea2cd0e/attachment.html>

From nwagner at iam.uni-stuttgart.de  Fri Feb 26 16:24:34 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 26 Feb 2010 22:24:34 +0100
Subject: [SciPy-User] frequency components of a signal buried in a
 noisy	time domain signal
In-Reply-To: <826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
Message-ID: <web-130797980@uni-stuttgart.de>

On Fri, 26 Feb 2010 15:56:29 -0500
  Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
> Nils,
> Yes, FFT can be used to visualize the frequency content 
>of a signal buried
> in the noise, especially if it is is narrowband,
> even though more advanced spectral analysis is required 
>for advanced
> applications.

Thank you very much for your reply.

Are you aware of free software for advanced applications ?
What could be done in case of broadband noise ?


Nils

> 
> I cannot help you with non-uniformly spaced samples 
>(some sort of
> interpolation comes to mind), but next power of 2 should 
>be trivial. Without
> checking for the argument type, you can implement the 
>function like this:
> 
> def nextpow2(n):
>    m_f = np.log2(n)
>    m_i = np.ceil(m_f)
>    return 2**m_i
> 
> 
> Hope it helps,
> Ivo
> 
> On 26 February 2010 14:05, Nils Wagner 
><nwagner at iam.uni-stuttgart.de> wrote:
> 
>> Hi all,
>>
>> A common use of Fourier transforms is to find the
>> frequency components of a signal buried in a noisy time
>> domain signal.
>>
>> I found a Matlab template at
>>
>> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/fft.shtml
>>
>> Matlab has a function
>>
>> nextpow2(L)
>>
>> Is there a similar build-in function in numpy/scipy ?
>>
>> I tried to convert the m-file into a pythonic form.
>> What is needed to obtain a similar figure of the
>> single-sided amplitude spectrum using
>> numpy/scipy/matplotlib ?
>>
>>
>> from numpy import sin, linspace, pi
>> from numpy.random import randn
>> from pylab import plot, show, title, xlabel, ylabel
>> from scipy.fft import fft
>>
>> Fs = 1000. #                          Sampling frequency
>> T  = 1./Fs #                          Sample time
>> L  = 1000  #                          length of signal
>> t  = arange(0,L)*T
>> x  = 0.7*sin(2*pi*50*t)+sin(2*pi*120*t)
>> y  = x + 2*randn(len(t))
>> plot(Fs*t[:50],y[:50])
>> title('Signal corrupted with zero-mean random noise')
>> xlabel('Time (milliseconds)')
>> #
>> #
>> #
>> #NFFT = 2^nextpow2(L); # Next power of 2 from length of 
>>y
>>
>> Y = fft(y,NFFT)/L
>> f = Fs/2*linspace(0,1,NFFT/2+1)
>> plot(f,2*abs(Y[:NFFT/2+1]))
>> title('Single-sided amplitude spectrum of y(t)')
>> xlabel('Frequency (Hz)')
>> ylabel('|Y(f)|')
>> show()
>>
>>
>> What can be done in case of nonequispaced data ?
>>
>> http://dx.doi.org/10.1137/0914081
>>
>> Thanks in advance
>>
>>                     Nils
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>


From peridot.faceted at gmail.com  Fri Feb 26 16:32:01 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 26 Feb 2010 16:32:01 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <web-130797980@uni-stuttgart.de>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
Message-ID: <ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>

Hi,

Looking at a periodic signal buried in noise is a well-studied
problem, with many techniques for attacking it. You really need to be
a little more specific about what you want to do. For example, is your
input signal really a sinusoid, or does it have harmonic content? Are
you trying to detect a weak periodic signal or are you trying to
extract the features of a strong periodic signal? Is your signal
exactly periodic, does it have some (deterministic or random) wander,
or are you looking for the power spectrum of a broadband signal?

If your input data are non-uniformly sampled, everything becomes more
difficult (and computationally expensive), but there are solutions
(e.g. the Lomb-Scargle periodogram).

Anne

On 26 February 2010 16:24, Nils Wagner <nwagner at iam.uni-stuttgart.de> wrote:
> On Fri, 26 Feb 2010 15:56:29 -0500
>  Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
>> Nils,
>> Yes, FFT can be used to visualize the frequency content
>>of a signal buried
>> in the noise, especially if it is is narrowband,
>> even though more advanced spectral analysis is required
>>for advanced
>> applications.
>
> Thank you very much for your reply.
>
> Are you aware of free software for advanced applications ?
> What could be done in case of broadband noise ?
>
>
> Nils
>
>>
>> I cannot help you with non-uniformly spaced samples
>>(some sort of
>> interpolation comes to mind), but next power of 2 should
>>be trivial. Without
>> checking for the argument type, you can implement the
>>function like this:
>>
>> def nextpow2(n):
>>    m_f = np.log2(n)
>>    m_i = np.ceil(m_f)
>>    return 2**m_i
>>
>>
>> Hope it helps,
>> Ivo
>>
>> On 26 February 2010 14:05, Nils Wagner
>><nwagner at iam.uni-stuttgart.de> wrote:
>>
>>> Hi all,
>>>
>>> A common use of Fourier transforms is to find the
>>> frequency components of a signal buried in a noisy time
>>> domain signal.
>>>
>>> I found a Matlab template at
>>>
>>> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/fft.shtml
>>>
>>> Matlab has a function
>>>
>>> nextpow2(L)
>>>
>>> Is there a similar build-in function in numpy/scipy ?
>>>
>>> I tried to convert the m-file into a pythonic form.
>>> What is needed to obtain a similar figure of the
>>> single-sided amplitude spectrum using
>>> numpy/scipy/matplotlib ?
>>>
>>>
>>> from numpy import sin, linspace, pi
>>> from numpy.random import randn
>>> from pylab import plot, show, title, xlabel, ylabel
>>> from scipy.fft import fft
>>>
>>> Fs = 1000. #                          Sampling frequency
>>> T  = 1./Fs #                          Sample time
>>> L  = 1000  #                          length of signal
>>> t  = arange(0,L)*T
>>> x  = 0.7*sin(2*pi*50*t)+sin(2*pi*120*t)
>>> y  = x + 2*randn(len(t))
>>> plot(Fs*t[:50],y[:50])
>>> title('Signal corrupted with zero-mean random noise')
>>> xlabel('Time (milliseconds)')
>>> #
>>> #
>>> #
>>> #NFFT = 2^nextpow2(L); # Next power of 2 from length of
>>>y
>>>
>>> Y = fft(y,NFFT)/L
>>> f = Fs/2*linspace(0,1,NFFT/2+1)
>>> plot(f,2*abs(Y[:NFFT/2+1]))
>>> title('Single-sided amplitude spectrum of y(t)')
>>> xlabel('Frequency (Hz)')
>>> ylabel('|Y(f)|')
>>> show()
>>>
>>>
>>> What can be done in case of nonequispaced data ?
>>>
>>> http://dx.doi.org/10.1137/0914081
>>>
>>> Thanks in advance
>>>
>>>                     Nils
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From ivo.maljevic at gmail.com  Fri Feb 26 16:38:36 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Fri, 26 Feb 2010 16:38:36 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <web-130797980@uni-stuttgart.de>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
Message-ID: <826c64da1002261338s6420afb5k4534794a8750968f@mail.gmail.com>

Maybe you can try sigview (http://www.sigview.com/), if that is what you
need, or you can invest some time in learning more about
spectral analysis, and then write your own software that will be tied to
your particular application / requirement.

I cannot give you a short answer to your question regarding the noise
bandwidth, but I can give you a few pointers.

1. When you have sampled signal which consists of desired signal + noise,
usually it helps to know what is the bandwidth of the desired signal.
In most applications that is the case, unless you are analyzing a completely
unknown signal (military applications, SETI, etc). Then you can filter the
signal to remove as much noise as possible.

2. In real world applications, there will always be some wideband noise
(i.e., thermal noise), and you might also have some narrowband
noise/interference. White noise is the most common type of wideband noise
that you would encounter, even though it may be "coloured" noise if you are
analyzing a signal at the output of nonideal Rx filter (for example in radio
systems).

3. As long as power spectral density of the signal is higher then the power
spectral density of the noise (at least in some portions of the spectrum),
simple FFT or periodogram (do a google search) approach will suffice.

Cheers,
Ivo

On 26 February 2010 16:24, Nils Wagner <nwagner at iam.uni-stuttgart.de> wrote:

> On Fri, 26 Feb 2010 15:56:29 -0500
>  Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
> > Nils,
> > Yes, FFT can be used to visualize the frequency content
> >of a signal buried
> > in the noise, especially if it is is narrowband,
> > even though more advanced spectral analysis is required
> >for advanced
> > applications.
>
> Thank you very much for your reply.
>
> Are you aware of free software for advanced applications ?
> What could be done in case of broadband noise ?
>
>
> Nils
>
> >
> > I cannot help you with non-uniformly spaced samples
> >(some sort of
> > interpolation comes to mind), but next power of 2 should
> >be trivial. Without
> > checking for the argument type, you can implement the
> >function like this:
> >
> > def nextpow2(n):
> >    m_f = np.log2(n)
> >    m_i = np.ceil(m_f)
> >    return 2**m_i
> >
> >
> > Hope it helps,
> > Ivo
> >
> > On 26 February 2010 14:05, Nils Wagner
> ><nwagner at iam.uni-stuttgart.de> wrote:
> >
> >> Hi all,
> >>
> >> A common use of Fourier transforms is to find the
> >> frequency components of a signal buried in a noisy time
> >> domain signal.
> >>
> >> I found a Matlab template at
> >>
> >> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/fft.shtml
> >>
> >> Matlab has a function
> >>
> >> nextpow2(L)
> >>
> >> Is there a similar build-in function in numpy/scipy ?
> >>
> >> I tried to convert the m-file into a pythonic form.
> >> What is needed to obtain a similar figure of the
> >> single-sided amplitude spectrum using
> >> numpy/scipy/matplotlib ?
> >>
> >>
> >> from numpy import sin, linspace, pi
> >> from numpy.random import randn
> >> from pylab import plot, show, title, xlabel, ylabel
> >> from scipy.fft import fft
> >>
> >> Fs = 1000. #                          Sampling frequency
> >> T  = 1./Fs #                          Sample time
> >> L  = 1000  #                          length of signal
> >> t  = arange(0,L)*T
> >> x  = 0.7*sin(2*pi*50*t)+sin(2*pi*120*t)
> >> y  = x + 2*randn(len(t))
> >> plot(Fs*t[:50],y[:50])
> >> title('Signal corrupted with zero-mean random noise')
> >> xlabel('Time (milliseconds)')
> >> #
> >> #
> >> #
> >> #NFFT = 2^nextpow2(L); # Next power of 2 from length of
> >>y
> >>
> >> Y = fft(y,NFFT)/L
> >> f = Fs/2*linspace(0,1,NFFT/2+1)
> >> plot(f,2*abs(Y[:NFFT/2+1]))
> >> title('Single-sided amplitude spectrum of y(t)')
> >> xlabel('Frequency (Hz)')
> >> ylabel('|Y(f)|')
> >> show()
> >>
> >>
> >> What can be done in case of nonequispaced data ?
> >>
> >> http://dx.doi.org/10.1137/0914081
> >>
> >> Thanks in advance
> >>
> >>                     Nils
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100226/97e2ad23/attachment.html>

From ivo.maljevic at gmail.com  Fri Feb 26 22:03:08 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Fri, 26 Feb 2010 22:03:08 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <web-130795898@uni-stuttgart.de>
References: <web-130795898@uni-stuttgart.de>
Message-ID: <826c64da1002261903h455596a6pe3ccacccec95085c@mail.gmail.com>

Nils,
I modified a little bit your code. Try this version, maybe you will find it
helpful. I wouldn't bother much
with the next power of 2 unless you are going to implement something in real
time - FFT calculation
with non-power of 2 on a PC is just fine. Also, I would consider
periodograms only if the signal changes over time, and you want
to be able to track these changes over time. Otherwise, taking the whole
signal and doing one FFT may be OK.

Take a look at the code below (blue) , which is a modified version of your
code (and read the comments).

Ivo


import numpy as np
import matplotlib.pyplot as plt
from scipy.fftpack import fft, fftshift

def signal_spectrum(x, Fs):
  n = len(x)
  f = np.linspace(-Fs/2, Fs/2, n)
  X=abs(fftshift(fft(x)))  # move the upper spectrum to negative freqs
  return [f, X]

pi = np.pi
Fs = 1000. #                          Sampling frequency
T  = 1./Fs #                          Sample time
L  = 1000  #                          length of signal
t  = np.arange(0,L)*T
x  = 0.7*np.sin(2*pi*50*t)+np.sin(2*pi*120*t)
y  = x + 2*np.random.randn(len(t))
plt.figure()
plt.plot(Fs*t[:50],y[:50])
plt.title('Signal corrupted with zero-mean random noise')
plt.xlabel('Time (milliseconds)')

NFFT = 2**np.ceil(np.log2(L)) # only if you really want this

plt.figure()
Y = fft(y,NFFT)/L
f = Fs/2*np.linspace(0,1,NFFT/2+1)
plt.plot(f,2*abs(Y[:NFFT/2+1]))
plt.title('Single-sided amplitude spectrum of y(t)')
plt.xlabel('Frequency (Hz)')
plt.ylabel('|Y(f)|')

# Another way of calculating and plotting the spectrum
f,Yd = signal_spectrum(y, Fs)
plt.figure()  # you want to add this to be able to see multiple plots
plt.plot(f,Yd)
plt.title('Spectrum of the the signal')
plt.xlabel('Frequency (Hz)')
plt.ylabel('|Y(f)|')
plt.grid(True)

plt.show()

On 26 February 2010 14:05, Nils Wagner <nwagner at iam.uni-stuttgart.de> wrote:

> Hi all,
>
> A common use of Fourier transforms is to find the
> frequency components of a signal buried in a noisy time
> domain signal.
>
> I found a Matlab template at
>
> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/fft.shtml
>
> Matlab has a function
>
> nextpow2(L)
>
> Is there a similar build-in function in numpy/scipy ?
>
> I tried to convert the m-file into a pythonic form.
> What is needed to obtain a similar figure of the
> single-sided amplitude spectrum using
> numpy/scipy/matplotlib ?
>
>
> from numpy import sin, linspace, pi
> from numpy.random import randn
> from pylab import plot, show, title, xlabel, ylabel
> from scipy.fft import fft
>
> Fs = 1000. #                          Sampling frequency
> T  = 1./Fs #                          Sample time
> L  = 1000  #                          length of signal
> t  = arange(0,L)*T
> x  = 0.7*sin(2*pi*50*t)+sin(2*pi*120*t)
> y  = x + 2*randn(len(t))
> plot(Fs*t[:50],y[:50])
> title('Signal corrupted with zero-mean random noise')
> xlabel('Time (milliseconds)')
> #
> #
> #
> #NFFT = 2^nextpow2(L); # Next power of 2 from length of y
>
> Y = fft(y,NFFT)/L
> f = Fs/2*linspace(0,1,NFFT/2+1)
> plot(f,2*abs(Y[:NFFT/2+1]))
> title('Single-sided amplitude spectrum of y(t)')
> xlabel('Frequency (Hz)')
> ylabel('|Y(f)|')
> show()
>
>
> What can be done in case of nonequispaced data ?
>
> http://dx.doi.org/10.1137/0914081
>
> Thanks in advance
>
>                     Nils
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100226/30758e11/attachment.html>

From charlesr.harris at gmail.com  Sat Feb 27 03:16:49 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 27 Feb 2010 01:16:49 -0700
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <web-130795898@uni-stuttgart.de>
References: <web-130795898@uni-stuttgart.de>
Message-ID: <e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>

On Fri, Feb 26, 2010 at 12:05 PM, Nils Wagner
<nwagner at iam.uni-stuttgart.de>wrote:

> Hi all,
>
> A common use of Fourier transforms is to find the
> frequency components of a signal buried in a noisy time
> domain signal.
>
> I found a Matlab template at
>
> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/fft.shtml
>
> Matlab has a function
>
> nextpow2(L)
>
>
I use searchsorted for that.

In [5]: tab = 2**arange(30)

In [6]: tab[tab.searchsorted(10)]
Out[6]: 16

In [7]: tab[tab.searchsorted(81)]
Out[7]: 128

Is there a similar build-in function in numpy/scipy ?
>
>
It would be handy sometimes. Where would it go?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/86463a53/attachment.html>

From d.l.goldsmith at gmail.com  Sat Feb 27 03:22:14 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Sat, 27 Feb 2010 00:22:14 -0800
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
Message-ID: <45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>

On Sat, Feb 27, 2010 at 12:16 AM, Charles R Harris <
charlesr.harris at gmail.com> wrote:

>
>
> On Fri, Feb 26, 2010 at 12:05 PM, Nils Wagner <
> nwagner at iam.uni-stuttgart.de> wrote:
>
>> Hi all,
>>
>> A common use of Fourier transforms is to find the
>> frequency components of a signal buried in a noisy time
>> domain signal.
>>
>> I found a Matlab template at
>>
>> http://www.mathworks.com/access/helpdesk/help/techdoc/ref/fft.shtml
>>
>> Matlab has a function
>>
>> nextpow2(L)
>>
>>
> I use searchsorted for that.
>
> In [5]: tab = 2**arange(30)
>
> In [6]: tab[tab.searchsorted(10)]
> Out[6]: 16
>
> In [7]: tab[tab.searchsorted(81)]
> Out[7]: 128
>
> Is there a similar build-in function in numpy/scipy ?
>>
>>
> It would be handy sometimes. Where would it go?
>

Why wouldn't it go in fftpack?

DG


>
> Chuck
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/2a73e826/attachment.html>

From nwagner at iam.uni-stuttgart.de  Sat Feb 27 04:41:03 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 27 Feb 2010 10:41:03 +0100
Subject: [SciPy-User] frequency components of a signal buried in a
 noisy	time domain signal
In-Reply-To: <ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
	<ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>
Message-ID: <web-130805788@uni-stuttgart.de>

On Fri, 26 Feb 2010 16:32:01 -0500
  Anne Archibald <peridot.faceted at gmail.com> wrote:
> Hi,
> 
> Looking at a periodic signal buried in noise is a 
>well-studied
> problem, with many techniques for attacking it. You 
>really need to be
> a little more specific about what you want to do. For 
>example, is your
> input signal really a sinusoid, or does it have harmonic 
>content? Are
> you trying to detect a weak periodic signal or are you 
>trying to
> extract the features of a strong periodic signal? Is 
>your signal
> exactly periodic, does it have some (deterministic or 
>random) wander,
> or are you looking for the power spectrum of a broadband 
>signal?
> 
> If your input data are non-uniformly sampled, everything 
>becomes more
> difficult (and computationally expensive), but there are 
>solutions
> (e.g. the Lomb-Scargle periodogram).
> 
> Anne
> 
Hi Anne,

Thank you very much for your hints !

BTW, a BSD licensed code for the Lomb-Scargle periodogram 
is available at
http://www.mathworks.com/matlabcentral/fileexchange/993-lombscargle-m
http://www.mathworks.com/matlabcentral/fileexchange/20004-lomb-lomb-scargle-periodogram


I am newbie to signal processing.
Is there a good introduction that you can recommend ?
There are so many books on signal processing. It should 
cover engineering applications.
  
What makes a signal weak/strong periodic ?

The signals come from real-life application (pressure / 
acceleration data).
Do I need a filter before I apply FFT ?

What would you do if you know nothing about the origin of 
the signal ?

How can I distinguish between deterministic and random 
wander ?

Cheers,
                       Nils


From ivo.maljevic at gmail.com  Sat Feb 27 05:02:07 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Sat, 27 Feb 2010 05:02:07 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
Message-ID: <826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>

David,
Nice way of avoiding log2, but how do you determine the length of your tab
array?
Do you want to chose an arbitrary large array, or you still want to use log2
call based on an input argument to the function nexpow2()?
In other words, how would you make a generic nextpow2() function?
Ivo

On 27 February 2010 03:22, David Goldsmith <d.l.goldsmith at gmail.com> wrote:

> On Sat, Feb 27, 2010 at 12:16 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> I use searchsorted for that.
>>
>> In [5]: tab = 2**arange(30)
>>
>> In [6]: tab[tab.searchsorted(10)]
>> Out[6]: 16
>>
>> In [7]: tab[tab.searchsorted(81)]
>> Out[7]: 128
>>
>> Is there a similar build-in function in numpy/scipy ?
>>>
>>>
>> It would be handy sometimes. Where would it go?
>>
>
> Why wouldn't it go in fftpack?
>
> DG
>
>
>>
>> Chuck
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/593f1a6e/attachment.html>

From nwagner at iam.uni-stuttgart.de  Sat Feb 27 05:06:38 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 27 Feb 2010 11:06:38 +0100
Subject: [SciPy-User] frequency components of a signal buried in a
 noisy	time domain signal
In-Reply-To: <826c64da1002261903h455596a6pe3ccacccec95085c@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261903h455596a6pe3ccacccec95085c@mail.gmail.com>
Message-ID: <web-130806108@uni-stuttgart.de>

On Fri, 26 Feb 2010 22:03:08 -0500
  Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
> Nils,
> I modified a little bit your code. Try this version, 
>maybe you will find it
> helpful. I wouldn't bother much
> with the next power of 2 unless you are going to 
>implement something in real
> time - FFT calculation
> with non-power of 2 on a PC is just fine. Also, I would 
>consider
> periodograms only if the signal changes over time, and 
>you want
> to be able to track these changes over time. Otherwise, 
>taking the whole
> signal and doing one FFT may be OK.
> 
> Take a look at the code below (blue) , which is a 
>modified version of your
> code (and read the comments).
> 
> Ivo
> 
> 
> import numpy as np
> import matplotlib.pyplot as plt
> from scipy.fftpack import fft, fftshift
> 
> def signal_spectrum(x, Fs):
>  n = len(x)
>  f = np.linspace(-Fs/2, Fs/2, n)
>  X=abs(fftshift(fft(x)))  # move the upper spectrum to 
>negative freqs
>  return [f, X]
> 
> pi = np.pi
>Fs = 1000. #                          Sampling frequency
> T  = 1./Fs #                          Sample time
> L  = 1000  #                          length of signal
> t  = np.arange(0,L)*T
> x  = 0.7*np.sin(2*pi*50*t)+np.sin(2*pi*120*t)
> y  = x + 2*np.random.randn(len(t))
> plt.figure()
> plt.plot(Fs*t[:50],y[:50])
> plt.title('Signal corrupted with zero-mean random 
>noise')
> plt.xlabel('Time (milliseconds)')
> 
> NFFT = 2**np.ceil(np.log2(L)) # only if you really want 
>this
> 
> plt.figure()
> Y = fft(y,NFFT)/L
> f = Fs/2*np.linspace(0,1,NFFT/2+1)
> plt.plot(f,2*abs(Y[:NFFT/2+1]))
> plt.title('Single-sided amplitude spectrum of y(t)')
> plt.xlabel('Frequency (Hz)')
> plt.ylabel('|Y(f)|')
> 
> # Another way of calculating and plotting the spectrum
> f,Yd = signal_spectrum(y, Fs)
> plt.figure()  # you want to add this to be able to see 
>multiple plots
> plt.plot(f,Yd)
> plt.title('Spectrum of the the signal')
> plt.xlabel('Frequency (Hz)')
> plt.ylabel('|Y(f)|')
> plt.grid(True)
> 
> plt.show()
  
Hi Ivo,

Thank you very much for your hints.
My experience is, that spectrograms work well with 
synthetic signals.

What is needed to use spectrograms for real-life signals ?
I mean do I need a filter etc. ?


Nils

from scipy import linspace, vectorize, cos, pi
from pylab import plot, show, imshow, subplot, specgram, 
xlabel, ylabel, savefig
#
# Example taken from 
http://en.wikipedia.org/wiki/Short-time_Fourier_transform
#
def x(t):
     " Synthetic signal "
     if t < 5:
        return cos(2*pi*10*t)
     if t >= 5. and t < 10:
        return cos(2*pi*25*t)
     if t >=10. and t< 15:
        return cos(2*pi*50*t)
     if t >=15. and t<= 20:
        return cos(2*pi*100*t)

t = linspace(0.,20.,8001)     # sampled at 400 Hz
dt = t[1]-t[0]
NFFT = 512                    # the length of the 
windowing segments
Fs = int(1.0/dt)              # the sampling frequency
x_vec = vectorize(x)
signal = x_vec(t)

ax1=subplot(211)
plot(t,signal)
subplot(212,sharex=ax1)
Pxx, freqs, bins, im = specgram(signal, NFFT=NFFT, Fs=Fs, 
noverlap=5)
xlabel(r'Time $t$ [s]')
ylabel('Frequency $f$ [Hz]')
savefig('spectrogram',dpi=60)
show()


From ivo.maljevic at gmail.com  Sat Feb 27 05:43:57 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Sat, 27 Feb 2010 05:43:57 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <web-130805788@uni-stuttgart.de>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
	<ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>
	<web-130805788@uni-stuttgart.de>
Message-ID: <826c64da1002270243p4cc1d89fy54e58ff00bf16eb3@mail.gmail.com>

Nils,
I will not answer your questions to Anne then (I could, but you asked her :)
), but I'll give you another advice, given that you are new to signal
processing.
Whatever you do, please do not use the scipy's snr function call to
calculate the SNR in signal processing.

I'll illustrate the problem with the 'statistical SNR' in signal processing
with 3 closely related and simple examples.

1. Infinitely long sinusoidal signal with additive white noise

Let x(t)=s(t)+n(t)=A sin(2\pi f t) + n(t)

Power of the signal component can be found as P_s=1/(2T) \int_T^T s^2(t) dt
= A^2/2, and let the power of noise be P_n=var(n) = 1.

If, we chose A=100, then intuitively, we know that the SNR is large, and
indeed: SNR = P_s/ P_N = A^2/2 = 5000

On the other hand, if you use statistical definition (used in optical
astronomy, and scipy and implemented in SCIPY), you would conclude quite the
opposite:

SNR = \mu_s / signa_n = mean(s) / signa_n= 0/1 = 0, and that would be the
case regardless of what the value for A is.

2. Discrete time signal with finite duration

Let's take this same signal and sample it in a such way that we have 100
cycles of a sinusoid:


>>> f=200
>>> t=np.linspace(0,0.5,2000)
>>> s=100*np.sin(2*np.pi*f*t)

Using sampled signal definitions for powers, you get:

P_s = np.mean(s**2) = 4997.5

and you can calculate the power of the noise either as:

>>> np.var(n) = 0.97726608241925972

or

the same way as the power of the signal:

>>> np.mean(n**2) = 0.977977713

The results will slightly differ, and that has to do with biased/unbiased
estimator.

The SNR=4997.5/0.9772660=5113.756

which is pretty close to analog domain case with infinitely long signal.

Using the scipy approach, you would however conclude the following:

SNR=mean(s)/sigma_n=-5.6495516860933771e-15/0.9885676923808808=
-5.7148860210947362e-15 (which comes close to zero).

3. Same example, but let's take fractional number of cycles.

>>> t=np.linspace(0,0.5+np.pi/800,2000+int(5*np.pi)) # this is 100 cycles +
1/4 of a cycle
>>> s=100*np.sin(2*np.pi*f*t)

>>> P_s = np.mean(s**2)
>>> SNR=P_s/np.var(n)
>>> SNR
5117.8701015651177  # very close to the two results above

But, using the other definition:

>>> SNR=np.mean(s)/np.std(n)
>>> SNR
0.098933308385744489

As you can see, your SNR would be a function of the frequency of the signal
(that is, it would matter how you truncate the cycles) and it wouldn't
matter much what is the amplitude of the signal.

Just my 2 cents,

Ivo



On 27 February 2010 04:41, Nils Wagner <nwagner at iam.uni-stuttgart.de> wrote:

> On Fri, 26 Feb 2010 16:32:01 -0500
>  Anne Archibald <peridot.faceted at gmail.com> wrote:
> > Hi,
> >
> > Looking at a periodic signal buried in noise is a
> >well-studied
> > problem, with many techniques for attacking it. You
> >really need to be
> > a little more specific about what you want to do. For
> >example, is your
> > input signal really a sinusoid, or does it have harmonic
> >content? Are
> > you trying to detect a weak periodic signal or are you
> >trying to
> > extract the features of a strong periodic signal? Is
> >your signal
> > exactly periodic, does it have some (deterministic or
> >random) wander,
> > or are you looking for the power spectrum of a broadband
> >signal?
> >
> > If your input data are non-uniformly sampled, everything
> >becomes more
> > difficult (and computationally expensive), but there are
> >solutions
> > (e.g. the Lomb-Scargle periodogram).
> >
> > Anne
> >
> Hi Anne,
>
> Thank you very much for your hints !
>
> BTW, a BSD licensed code for the Lomb-Scargle periodogram
> is available at
> http://www.mathworks.com/matlabcentral/fileexchange/993-lombscargle-m
>
> http://www.mathworks.com/matlabcentral/fileexchange/20004-lomb-lomb-scargle-periodogram
>
>
> I am newbie to signal processing.
> Is there a good introduction that you can recommend ?
> There are so many books on signal processing. It should
> cover engineering applications.
>
> What makes a signal weak/strong periodic ?
>
> The signals come from real-life application (pressure /
> acceleration data).
> Do I need a filter before I apply FFT ?
>
> What would you do if you know nothing about the origin of
> the signal ?
>
> How can I distinguish between deterministic and random
> wander ?
>
> Cheers,
>                        Nils
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/5b3bd3cf/attachment.html>

From ivo.maljevic at gmail.com  Sat Feb 27 05:54:27 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Sat, 27 Feb 2010 05:54:27 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <web-130806108@uni-stuttgart.de>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261903h455596a6pe3ccacccec95085c@mail.gmail.com>
	<web-130806108@uni-stuttgart.de>
Message-ID: <826c64da1002270254n55414f3erccad71d2cc7f9804@mail.gmail.com>

> Hi Ivo,
>
> Thank you very much for your hints.
> My experience is, that spectrograms work well with
> synthetic signals.
>

What is needed to use spectrograms for real-life signals ?
>


The same way the frequency content of your synthetic signal changes  over
time, it happens with real signal (e.g., voice,
communication system's signal, etc).


I mean do I need a filter etc. ?
>
>
There are several levels of filtering:

1. You want to filter a signal even before you sample it (e.g., in comm
systems, you want to remove as much noise+interference) and focus only on
the band that interests you, plus you want to satisfy Nyquist.

2. After you have sampled your signal (if it was analog to begin with,
otherwise, skip step 1), you may do additional digital filtering, to further
reduce the impact of noise/interference, but this is all application
specific.

3. Depending on what you are doing, you might want to consider windowing
techniques in your spectral analysis, if you want to suppress the sidelobes
(fft uses rectangular window, which has high sidelobes - sinc function), or
to better identify the peaks. It all comes down to trade-offs.

In any case, to think about these problems, you need to go over signal
processing material first.

Ivo


>
> Nils
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/b01b1892/attachment.html>

From cournape at gmail.com  Sat Feb 27 07:44:30 2010
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 27 Feb 2010 21:44:30 +0900
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <web-130805788@uni-stuttgart.de>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
	<ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>
	<web-130805788@uni-stuttgart.de>
Message-ID: <5b8d13221002270444h3f1fc421l13e64a524575ea67@mail.gmail.com>

On Sat, Feb 27, 2010 at 6:41 PM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> On Fri, 26 Feb 2010 16:32:01 -0500
> ?Anne Archibald <peridot.faceted at gmail.com> wrote:
>> Hi,
>>
>> Looking at a periodic signal buried in noise is a
>>well-studied
>> problem, with many techniques for attacking it. You
>>really need to be
>> a little more specific about what you want to do. For
>>example, is your
>> input signal really a sinusoid, or does it have harmonic
>>content? Are
>> you trying to detect a weak periodic signal or are you
>>trying to
>> extract the features of a strong periodic signal? Is
>>your signal
>> exactly periodic, does it have some (deterministic or
>>random) wander,
>> or are you looking for the power spectrum of a broadband
>>signal?
>>
>> If your input data are non-uniformly sampled, everything
>>becomes more
>> difficult (and computationally expensive), but there are
>>solutions
>> (e.g. the Lomb-Scargle periodogram).
>>
>> Anne
>>
> Hi Anne,
>
> Thank you very much for your hints !
>
> BTW, a BSD licensed code for the Lomb-Scargle periodogram
> is available at
> http://www.mathworks.com/matlabcentral/fileexchange/993-lombscargle-m
> http://www.mathworks.com/matlabcentral/fileexchange/20004-lomb-lomb-scargle-periodogram
>
>
> I am newbie to signal processing.
> Is there a good introduction that you can recommend ?

It depends on what you are looking for, and the depth and time you are
willing to spend on it. A "down to earth" book is freely available
here:

http://www.dspguide.com

If you are interested in manipulating signals with noise, statistical
signal processing is where to look at, but I am not sure there is any
good book which does not start with theory there.

> There are so many books on signal processing. It should
> cover engineering applications.
>
> What makes a signal weak/strong periodic ?
>
> The signals come from real-life application (pressure /
> acceleration data).
> Do I need a filter before I apply FFT ?
>
> What would you do if you know nothing about the origin of
> the signal ?

Well, if you know really nothing, there is nothing you can do :) For
spectral estimation, there are broadly two kind of methods: parametric
and non parametric. Unfortunately, scipy itself does not have much
here - I started the talkbox scikits to implement at least what is
available in matlab and more for spectral estimation, but I am not
sure I will have time to work on it much anymore.

Basic methods, like AR modelling, are quite simple to implement by
yourself, though.

David


From peridot.faceted at gmail.com  Sat Feb 27 07:49:19 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 27 Feb 2010 07:49:19 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <web-130805788@uni-stuttgart.de>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
	<ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>
	<web-130805788@uni-stuttgart.de>
Message-ID: <ce557a361002270449p2c7ec113l8c6e51f5231d4d22@mail.gmail.com>

On 27 February 2010 04:41, Nils Wagner <nwagner at iam.uni-stuttgart.de> wrote:
> On Fri, 26 Feb 2010 16:32:01 -0500

> I am newbie to signal processing.
> Is there a good introduction that you can recommend ?
> There are so many books on signal processing. It should
> cover engineering applications.

Hmm. I don't have a good example in mind. For an overview you could
try Numerical Recipes (but keep in mind that there are free python
implementations of almost all the algorithms they describe, and there
are often better algorithms out there). Mostly I have used signal
processing in a scientific context, often somewhat different.

> What makes a signal weak/strong periodic ?

By an exactly periodic signal I mean something like sin(f*t). Such a
signal produces a very narrow peak of a characteristic shape in an
FFT, and so can be recognized even in the presence of quite strong
noise. If your signal is only quasi-periodic - perhaps the frequency
is a slowly-varying function of time - you'll have a much broader
peak, which will be much lower for the same input signal amplitude,
and hence more difficult to distinguish from noise. If your signal is
only quasi-periodic, or comes and goes in the data, you may want to do
a series of FFTs on short, overlapping pieces of the data, so you can
look at time evolution of the signal's spectral properties.

By weak versus strong, I just meant: is your signal so weak that
there's some question you'll be able to detect it at all, or is it
strong enough that you can try to extract more subtle properties?

> The signals come from real-life application (pressure /
> acceleration data).
> Do I need a filter before I apply FFT ?

The short answer is "probably not", or at least, not in the usual
sense of "filter". You will probably want to extend your data set so
that it is a power of two, just to make the FFT faster. Extending your
data set in time has the effect of interpolating the FFT, so it may
well be worth extending the data set further, to double or four times
its original length. This will broaden your peaks and affect the
statistics of the noise. (And, incidentally, if you're going to do
this, you should extend the array either by repeating the input values
or by filling the extra space with the mean of the input array to
avoid a spurious bump at low frequencies.)

Once you've got the FFT, you should know what different kinds of
signal look like. A pure sinusoid that's at one of the FFT frequencies
- which happens when there's exactly an integer number of turns in
your data - will prodduce a spike at exactly one of your FFT
frequencies. A pure sinusoid that's between two FFT frequencies will
produce a sin(x)/x response in nearby bins; in particular, a sinusoid
midway between two FFT frequencies will produce two high-power bins
that are substantially less high than the single bin would be if the
same signal were exactly at an FFT frequency; this is one reason
extending the data to interpolate the FFT can be a good idea.

There are also tricks that can be done once you've found a peak,
looking at the precise shape of the peak to determine whether the
periodic signal fills the whole of your data and if not where it is.

If you have a more broad-band signal, perhaps because it is only
approximately periodic, then you'll have an extended region of extra
power in the FFT. This can be a little tricky to recognize, if it's
not obvious from inspection; one thing to try is smoothing the power
spectrum (the squared amplitude of the FFT) so that local increases
show more prominently.

> What would you do if you know nothing about the origin of
> the signal ?

This rather depends on what I was looking for. But if I had a time
series in which I suspected some sort of periodicity, I'd start by
plotting the power spectrum and looking for peaks that stood out well
above the local average. I'd also try plotting a smoothed power
spectrum.

Also worth trying would be taking a series of overlapping FFTs each
covering part of the data; I might see a signal in some of these
individual FFTs, which would tell me where it was in the data, or I
might average all the power spectra (which is roughly equivalent to
smoothing a long-term FFT) looking for a broad-band signal.

If I expected a periodic signal that had harmonic content - a series
of pulses rather than a pure sinusoid, for example - I'd want to look
at the total power in several harmonics at once, adding the power at
f, 2f, 3f, 4f, ....

If I found an exactly periodic signal, I could investigate further by
"folding" the data, that is, adding all the data points based on the
phase at which they were taken to build up a "profile" of the signal
as a function of pulse phase.

> How can I distinguish between deterministic and random
> wander ?

Without some a priori knowledge, it's not easy, though overlapping
short-time FFTs might help. Here I'd produce a grayscale plot of power
versus frequency and time (i.e. which piece the short FFT was applied
to). If you have a nice bright signal, you might see it at the same
frequency all the time, or you might see it drift linearly up or down,
or it might vary sinusoidally in frequency over time, or it might
wander randomly, or it might just always be broad.

If you know exactly how it should be varying, say frequency varying
linearly with time (e.g. Doppler shift from a uniform acceleration) or
sinusoidally, there are special techniques to remove or detect that
modulation. Some of these techniques are done in the time domain;
others by looking at the shape of the peak (if any) in the FFT.



I should say that this is a big enough and complex enough field that
my answers reflect only my experience, which is skewed towards working
with pulsar data, in which the data tends to include very weak but
exactly periodic signals. People used to working with, say, speech
data will have very different ideas on what sort of tools are useful.

Anne


From josef.pktd at gmail.com  Sat Feb 27 08:34:00 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 27 Feb 2010 08:34:00 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <5b8d13221002270444h3f1fc421l13e64a524575ea67@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
	<ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>
	<web-130805788@uni-stuttgart.de>
	<5b8d13221002270444h3f1fc421l13e64a524575ea67@mail.gmail.com>
Message-ID: <1cd32cbb1002270534g2a2cb5c3tfc5c77240bf46b0f@mail.gmail.com>

On Sat, Feb 27, 2010 at 7:44 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Sat, Feb 27, 2010 at 6:41 PM, Nils Wagner
> <nwagner at iam.uni-stuttgart.de> wrote:
>> On Fri, 26 Feb 2010 16:32:01 -0500
>> ?Anne Archibald <peridot.faceted at gmail.com> wrote:
>>> Hi,
>>>
>>> Looking at a periodic signal buried in noise is a
>>>well-studied
>>> problem, with many techniques for attacking it. You
>>>really need to be
>>> a little more specific about what you want to do. For
>>>example, is your
>>> input signal really a sinusoid, or does it have harmonic
>>>content? Are
>>> you trying to detect a weak periodic signal or are you
>>>trying to
>>> extract the features of a strong periodic signal? Is
>>>your signal
>>> exactly periodic, does it have some (deterministic or
>>>random) wander,
>>> or are you looking for the power spectrum of a broadband
>>>signal?
>>>
>>> If your input data are non-uniformly sampled, everything
>>>becomes more
>>> difficult (and computationally expensive), but there are
>>>solutions
>>> (e.g. the Lomb-Scargle periodogram).
>>>
>>> Anne
>>>
>> Hi Anne,
>>
>> Thank you very much for your hints !
>>
>> BTW, a BSD licensed code for the Lomb-Scargle periodogram
>> is available at
>> http://www.mathworks.com/matlabcentral/fileexchange/993-lombscargle-m
>> http://www.mathworks.com/matlabcentral/fileexchange/20004-lomb-lomb-scargle-periodogram
>>
>>
>> I am newbie to signal processing.
>> Is there a good introduction that you can recommend ?
>
> It depends on what you are looking for, and the depth and time you are
> willing to spend on it. A "down to earth" book is freely available
> here:
>
> http://www.dspguide.com
>
> If you are interested in manipulating signals with noise, statistical
> signal processing is where to look at, but I am not sure there is any
> good book which does not start with theory there.
>
>> There are so many books on signal processing. It should
>> cover engineering applications.
>>
>> What makes a signal weak/strong periodic ?
>>
>> The signals come from real-life application (pressure /
>> acceleration data).
>> Do I need a filter before I apply FFT ?
>>
>> What would you do if you know nothing about the origin of
>> the signal ?
>
> Well, if you know really nothing, there is nothing you can do :) For
> spectral estimation, there are broadly two kind of methods: parametric
> and non parametric. Unfortunately, scipy itself does not have much
> here - I started the talkbox scikits to implement at least what is
> available in matlab and more for spectral estimation, but I am not
> sure I will have time to work on it much anymore.
>
> Basic methods, like AR modelling, are quite simple to implement by
> yourself, though.

nitime also has many functions for frequency domain analysis

and there are also a few functions in matplotlib

Josef

>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From ivo.maljevic at gmail.com  Sat Feb 27 08:37:27 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Sat, 27 Feb 2010 08:37:27 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <ce557a361002270449p2c7ec113l8c6e51f5231d4d22@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
	<ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>
	<web-130805788@uni-stuttgart.de>
	<ce557a361002270449p2c7ec113l8c6e51f5231d4d22@mail.gmail.com>
Message-ID: <826c64da1002270537p197584ccx6ef4f153e1b078ff@mail.gmail.com>

> > What makes a signal weak/strong periodic ?
>
> By an exactly periodic signal I mean something like sin(f*t). Such a
> signal produces a very narrow peak of a characteristic shape in an
> FFT, and so can be recognized even in the presence of quite strong
> noise. If your signal is only quasi-periodic - perhaps the frequency
> is a slowly-varying function of time - you'll have a much broader
> peak, which will be much lower for the same input signal amplitude,
> and hence more difficult to distinguish from noise. If your signal is
> only quasi-periodic, or comes and goes in the data, you may want to do
> a series of FFTs on short, overlapping pieces of the data, so you can
> look at time evolution of the signal's spectral properties.
>
>
In my opinion this is not quite so. Periodic signal, as you rightly pointed
out,
is a sinusoidal signal. Quasi-periodic signal behaves like a periodic one,
even though
it does not satisfy the periodic condition x(t) = x (t+To), where To is the
period. Best known
examples are when you add two sinusoidal signals with frequencies that are
not a fractional integer of each other.
For example: sin(2pi f t)+sin(2pi^2 f t). You would still see a "spike" in
the frequency domain, but quasi-periodicity
definitely does not relate to low frequency.

Ivo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/a2074742/attachment.html>

From jagan_cbe2003 at yahoo.co.in  Sat Feb 27 08:54:59 2010
From: jagan_cbe2003 at yahoo.co.in (jagan prabhu)
Date: Sat, 27 Feb 2010 19:24:59 +0530 (IST)
Subject: [SciPy-User] Difference between SQP and SLSQP
Message-ID: <289235.84976.qm@web95512.mail.in.yahoo.com>

Hi, 

I have a basic problem in understanding SLSQP method, what is the difference between SQP (sequential quadratic programming) and SLSQP(Sequential Least SQuares Programming)? 

please can any one explain the working of SLSQP.






      Your Mail works best with the New Yahoo Optimized IE8. Get it NOW! http://downloads.yahoo.com/in/internetexplorer/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/b00c716c/attachment.html>

From peridot.faceted at gmail.com  Sat Feb 27 09:58:17 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 27 Feb 2010 09:58:17 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <826c64da1002270537p197584ccx6ef4f153e1b078ff@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
	<ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>
	<web-130805788@uni-stuttgart.de>
	<ce557a361002270449p2c7ec113l8c6e51f5231d4d22@mail.gmail.com>
	<826c64da1002270537p197584ccx6ef4f153e1b078ff@mail.gmail.com>
Message-ID: <ce557a361002270658k70ece1f8pdc54490a27ea5d79@mail.gmail.com>

On 27 February 2010 08:37, Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
>
>> > What makes a signal weak/strong periodic ?
>>
>> By an exactly periodic signal I mean something like sin(f*t). Such a
>> signal produces a very narrow peak of a characteristic shape in an
>> FFT, and so can be recognized even in the presence of quite strong
>> noise. If your signal is only quasi-periodic - perhaps the frequency
>> is a slowly-varying function of time - you'll have a much broader
>> peak, which will be much lower for the same input signal amplitude,
>> and hence more difficult to distinguish from noise. If your signal is
>> only quasi-periodic, or comes and goes in the data, you may want to do
>> a series of FFTs on short, overlapping pieces of the data, so you can
>> look at time evolution of the signal's spectral properties.
>>
>
> In my opinion this is not quite so. Periodic signal, as you rightly pointed
> out,
> is a sinusoidal signal. Quasi-periodic signal behaves like a periodic one,
> even though
> it does not satisfy the periodic condition x(t) = x (t+To), where To is the
> period. Best known
> examples are when you add two sinusoidal signals with frequencies that are
> not a fractional integer of each other.
> For example: sin(2pi f t)+sin(2pi^2 f t). You would still see a "spike" in
> the frequency domain, but quasi-periodicity
> definitely does not relate to low frequency.

I guess we have a minor conflict of definitions. Of course the signal
you describe is not periodic, but, at least in terms of Fourier
transforms, you can treat the terms of a sum as completely independent
signals. So I would view this as simply two exactly periodic signals,
which you can indeed pull out as two separate peaks in the Fourier
transform.

What I had in mind in referring to quasi-periodic signals was
something you might get out of an oscillator with poor frequency
stability, or a not-very-good resonant cavity, or a sinusoid with some
modulation: a signal localized in frequency but not exactly periodic.
This will have a broader peak in the Fourier transform, but will still
be interesting in many contexts.

Anne


From ivo.maljevic at gmail.com  Sat Feb 27 10:20:34 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Sat, 27 Feb 2010 10:20:34 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <ce557a361002270658k70ece1f8pdc54490a27ea5d79@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002261256n45e612a3w24b0806464033acf@mail.gmail.com>
	<web-130797980@uni-stuttgart.de>
	<ce557a361002261332v1aa217achfde08d262d09b8a@mail.gmail.com>
	<web-130805788@uni-stuttgart.de>
	<ce557a361002270449p2c7ec113l8c6e51f5231d4d22@mail.gmail.com>
	<826c64da1002270537p197584ccx6ef4f153e1b078ff@mail.gmail.com>
	<ce557a361002270658k70ece1f8pdc54490a27ea5d79@mail.gmail.com>
Message-ID: <826c64da1002270720y48fb2385x20b4e24f01f3e17@mail.gmail.com>

>
> I guess we have a minor conflict of definitions. Of course the signal
> you describe is not periodic, but, at least in terms of Fourier
> transforms, you can treat the terms of a sum as completely independent
> signals. So I would view this as simply two exactly periodic signals,
> which you can indeed pull out as two separate peaks in the Fourier
> transform.


You are right. I just checked, and what I was referring to is not known as *
quasi-periodic* signal, it is known as
*almost-periodic*, and it plays a role when you want to discuss power vs.
energy signals.

Therefore, I take back my comment, as I am not familiar with the
quasi-periodic signal definition.
I am certainly familiar with phase noise / frequency error behaviour of
oscillators, but that doesn't matter.

Since you didn't recommend any book, I'll recommend a couple of books to
Nils:

1.
http://www.amazon.com/Discrete-Time-Signal-Processing-2nd-Prentice-Hall/dp/0137549202/ref=pd_sim_b_3

2.
http://www.amazon.com/Digital-Signal-Processing-Bookware-Companion/dp/0495073113/ref=pd_sim_b_2

Ivo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/9e26453d/attachment.html>

From jsseabold at gmail.com  Sat Feb 27 11:42:35 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sat, 27 Feb 2010 11:42:35 -0500
Subject: [SciPy-User] Difference between SQP and SLSQP
In-Reply-To: <289235.84976.qm@web95512.mail.in.yahoo.com>
References: <289235.84976.qm@web95512.mail.in.yahoo.com>
Message-ID: <c048da1c1002270842q58ae5bc7lbb1db648fe5aa7f6@mail.gmail.com>

On Sat, Feb 27, 2010 at 8:54 AM, jagan prabhu <jagan_cbe2003 at yahoo.co.in> wrote:
>
> Hi,
>
> I have a basic problem in understanding SLSQP method, what is the difference between SQP (sequential quadratic programming) and SLSQP(Sequential Least SQuares Programming)?
>
> please can any one explain the working of SLSQP.
>

Ungated copy of the reference in the source:
http://abs-5.me.washington.edu/dynOpt/p262-kraft.pdf

Skipper


From robert.kern at gmail.com  Sat Feb 27 13:02:59 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 27 Feb 2010 12:02:59 -0600
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com> 
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com> 
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
Message-ID: <3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>

On Sat, Feb 27, 2010 at 04:02, Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
> David,
> Nice way of avoiding log2, but how do you determine the length of your tab
> array?

The input arrays can only be so large. Even on 64-bit machines, the
table need not have more than 64 entries.

  tab = 2 ** np.arange(np.iinfo(np.intp).bits)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Sat Feb 27 13:31:32 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 27 Feb 2010 11:31:32 -0700
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
Message-ID: <e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>

On Sat, Feb 27, 2010 at 11:02 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Sat, Feb 27, 2010 at 04:02, Ivo Maljevic <ivo.maljevic at gmail.com>
> wrote:
> > David,
> > Nice way of avoiding log2, but how do you determine the length of your
> tab
> > array?
>
> The input arrays can only be so large. Even on 64-bit machines, the
> table need not have more than 64 entries.
>
>  tab = 2 ** np.arange(np.iinfo(np.intp).bits)
>
>
Or

tab = 1 << arange(iinfo(intp).bits - 1)

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/3904a9bd/attachment.html>

From ivo.maljevic at gmail.com  Sat Feb 27 13:32:31 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Sat, 27 Feb 2010 13:32:31 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
Message-ID: <826c64da1002271032t5f719f02hfd2bd3f1e828a341@mail.gmail.com>

Thanks Robert. I figured that after I sent the question, but I am not sure
if the search based solution offers any advantage in terms of speed.
Besides, and not that it matters really, but that solution restricts the
usage to max 2**32 integer result on 32 bit machines, whereas log based one
doesn't:

>>> def nextpow2(n):
...   return 2**(np.ceil(np.log2(n)))
...
>>> np.iinfo(np.intp).bits
32
>>> n=2**32+300.5
>>> n
4294967596.5
>>> nextpow2(n)
8589934592.0

On 27 February 2010 13:02, Robert Kern <robert.kern at gmail.com> wrote:

> On Sat, Feb 27, 2010 at 04:02, Ivo Maljevic <ivo.maljevic at gmail.com>
> wrote:
> > David,
> > Nice way of avoiding log2, but how do you determine the length of your
> tab
> > array?
>
> The input arrays can only be so large. Even on 64-bit machines, the
> table need not have more than 64 entries.
>
>  tab = 2 ** np.arange(np.iinfo(np.intp).bits)
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/067aece4/attachment.html>

From robert.kern at gmail.com  Sat Feb 27 13:39:20 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 27 Feb 2010 12:39:20 -0600
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <826c64da1002271032t5f719f02hfd2bd3f1e828a341@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com> 
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com> 
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com> 
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com> 
	<826c64da1002271032t5f719f02hfd2bd3f1e828a341@mail.gmail.com>
Message-ID: <3d375d731002271039v5bce7afer131be4d8375477eb@mail.gmail.com>

On Sat, Feb 27, 2010 at 12:32, Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
> Thanks Robert. I figured that after I sent the question, but I am not sure
> if the search based solution offers any advantage in terms of speed.
> Besides, and not that it matters really, but that solution restricts the
> usage to max 2**32 integer result on 32 bit machines, whereas log based one
> doesn't:

The point of nextpow2() is to find a good size array for doing the
FFT. It is not a general purpose computational routine. On 32-bit
machines, you cannot have an array larger than 2**32.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Sat Feb 27 13:41:04 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 27 Feb 2010 13:41:04 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
Message-ID: <1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>

On Sat, Feb 27, 2010 at 1:31 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sat, Feb 27, 2010 at 11:02 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Sat, Feb 27, 2010 at 04:02, Ivo Maljevic <ivo.maljevic at gmail.com>
>> wrote:
>> > David,
>> > Nice way of avoiding log2, but how do you determine the length of your
>> > tab
>> > array?
>>
>> The input arrays can only be so large. Even on 64-bit machines, the
>> table need not have more than 64 entries.
>>
>> ?tab = 2 ** np.arange(np.iinfo(np.intp).bits)
>>
>
> Or
>
> tab = 1 << arange(iinfo(intp).bits - 1)

I think I prefer a readable solution to this for calls that are not in
an inner loop.

scipy.signal.fftconvolve uses the same as Ivo's solution

<<  means it's much larger than 1 ?

Josef


>
> Chuck
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From robert.kern at gmail.com  Sat Feb 27 13:45:09 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 27 Feb 2010 12:45:09 -0600
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com> 
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com> 
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com> 
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com> 
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com> 
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
Message-ID: <3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>

On Sat, Feb 27, 2010 at 12:41,  <josef.pktd at gmail.com> wrote:
> On Sat, Feb 27, 2010 at 1:31 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>> On Sat, Feb 27, 2010 at 11:02 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>>
>>> On Sat, Feb 27, 2010 at 04:02, Ivo Maljevic <ivo.maljevic at gmail.com>
>>> wrote:
>>> > David,
>>> > Nice way of avoiding log2, but how do you determine the length of your
>>> > tab
>>> > array?
>>>
>>> The input arrays can only be so large. Even on 64-bit machines, the
>>> table need not have more than 64 entries.
>>>
>>> ?tab = 2 ** np.arange(np.iinfo(np.intp).bits)
>>>
>>
>> Or
>>
>> tab = 1 << arange(iinfo(intp).bits - 1)
>
> I think I prefer a readable solution to this for calls that are not in
> an inner loop.
>
> scipy.signal.fftconvolve uses the same as Ivo's solution
>
> << ?means it's much larger than 1 ?

It's a bitshift operator. Basically "x << y" means "x * (2 ** y)" for
integer arguments.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Sat Feb 27 13:57:46 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 27 Feb 2010 11:57:46 -0700
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <826c64da1002271032t5f719f02hfd2bd3f1e828a341@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
	<826c64da1002271032t5f719f02hfd2bd3f1e828a341@mail.gmail.com>
Message-ID: <e06186141002271057x530c2196h8aecabd7fe83837d@mail.gmail.com>

On Sat, Feb 27, 2010 at 11:32 AM, Ivo Maljevic <ivo.maljevic at gmail.com>wrote:

> Thanks Robert. I figured that after I sent the question, but I am not sure
> if the search based solution offers any advantage in terms of speed.
> Besides, and not that it matters really, but that solution restricts the
> usage to max 2**32 integer result on 32 bit machines, whereas log based one
> doesn't:
>
> >>> def nextpow2(n):
> ...   return 2**(np.ceil(np.log2(n)))
> ...
> >>> np.iinfo(np.intp).bits
> 32
> >>> n=2**32+300.5
> >>> n
> 4294967596.5
> >>> nextpow2(n)
> 8589934592.0
>
>
For (ieee) floats

In [17]: def nextpow2(x) : return np.ldexp(1., np.frexp(x)[1])
   ....:

In [18]: nextpow2(2**32+300.5)
Out[18]: 8589934592.0


Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/576bb669/attachment.html>

From ivo.maljevic at gmail.com  Sat Feb 27 13:58:23 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Sat, 27 Feb 2010 13:58:23 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
	<3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
Message-ID: <826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>

I am sorry guys, but the solution I've proposed, and the one that was
proposed by the scipy camp would cause confusion for those who switch from
Matlab, and I just checked that.
If you are going to use the same name as the Matlab one, than this function
should return the m such that 2**m >= abs(n), and should not return 2^m.

Whatever approach you guys want to use (and I believe that especially given
what the function should return), do not return 2**m, but just m instead.
For example:

>>> def nextpow2(n):
...   return np.ceil(np.log2(n))
...
>>> m=nextpow2(1000)
>>> m
10.0
>>> 2**m
1024.0


On 27 February 2010 13:45, Robert Kern <robert.kern at gmail.com> wrote:

> On Sat, Feb 27, 2010 at 12:41,  <josef.pktd at gmail.com> wrote:
> > On Sat, Feb 27, 2010 at 1:31 PM, Charles R Harris
> > <charlesr.harris at gmail.com> wrote:
> >>
> >> On Sat, Feb 27, 2010 at 11:02 AM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>>
> >>> On Sat, Feb 27, 2010 at 04:02, Ivo Maljevic <ivo.maljevic at gmail.com>
> >>> wrote:
> >>> > David,
> >>> > Nice way of avoiding log2, but how do you determine the length of
> your
> >>> > tab
> >>> > array?
> >>>
> >>> The input arrays can only be so large. Even on 64-bit machines, the
> >>> table need not have more than 64 entries.
> >>>
> >>>  tab = 2 ** np.arange(np.iinfo(np.intp).bits)
> >>>
> >>
> >> Or
> >>
> >> tab = 1 << arange(iinfo(intp).bits - 1)
> >
> > I think I prefer a readable solution to this for calls that are not in
> > an inner loop.
> >
> > scipy.signal.fftconvolve uses the same as Ivo's solution
> >
> > <<  means it's much larger than 1 ?
>
> It's a bitshift operator. Basically "x << y" means "x * (2 ** y)" for
> integer arguments.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/02d97bbc/attachment.html>

From charlesr.harris at gmail.com  Sat Feb 27 14:03:02 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 27 Feb 2010 12:03:02 -0700
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
	<3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
	<826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
Message-ID: <e06186141002271103i1da029c0x6fee6d9aea99bef3@mail.gmail.com>

On Sat, Feb 27, 2010 at 11:58 AM, Ivo Maljevic <ivo.maljevic at gmail.com>wrote:

> I am sorry guys, but the solution I've proposed, and the one that was
> proposed by the scipy camp would cause confusion for those who switch from
> Matlab, and I just checked that.
> If you are going to use the same name as the Matlab one, than this function
> should return the m such that 2**m >= abs(n), and should not return 2^m.
>
> Whatever approach you guys want to use (and I believe that especially given
> what the function should return), do not return 2**m, but just m instead.
> For example:
>

For the table approach, it is the index returned by searchsorted. For floats
it is np.frexp(x)[1].

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/a2eede0c/attachment.html>

From ivo.maljevic at gmail.com  Sat Feb 27 14:03:07 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Sat, 27 Feb 2010 14:03:07 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
	<3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
	<826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
Message-ID: <826c64da1002271103q50f6785ei44c52beb72a26593@mail.gmail.com>

Another suggestion, and this one also makes the parallel to Matlab. Make
sure it works for vectors:

>>> a=np.array([7,11,250])
>>> nextpow2(a)
array([ 3.,  4.,  8.])



On 27 February 2010 13:58, Ivo Maljevic <ivo.maljevic at gmail.com> wrote:

> I am sorry guys, but the solution I've proposed, and the one that was
> proposed by the scipy camp would cause confusion for those who switch from
> Matlab, and I just checked that.
> If you are going to use the same name as the Matlab one, than this function
> should return the m such that 2**m >= abs(n), and should not return 2^m.
>
> Whatever approach you guys want to use (and I believe that especially given
> what the function should return), do not return 2**m, but just m instead.
> For example:
>
> >>> def nextpow2(n):
> ...   return np.ceil(np.log2(n))
> ...
> >>> m=nextpow2(1000)
> >>> m
> 10.0
> >>> 2**m
> 1024.0
>
>
>
> On 27 February 2010 13:45, Robert Kern <robert.kern at gmail.com> wrote:
>
>> On Sat, Feb 27, 2010 at 12:41,  <josef.pktd at gmail.com> wrote:
>> > On Sat, Feb 27, 2010 at 1:31 PM, Charles R Harris
>> > <charlesr.harris at gmail.com> wrote:
>> >>
>> >> On Sat, Feb 27, 2010 at 11:02 AM, Robert Kern <robert.kern at gmail.com>
>> wrote:
>> >>>
>> >>> On Sat, Feb 27, 2010 at 04:02, Ivo Maljevic <ivo.maljevic at gmail.com>
>> >>> wrote:
>> >>> > David,
>> >>> > Nice way of avoiding log2, but how do you determine the length of
>> your
>> >>> > tab
>> >>> > array?
>> >>>
>> >>> The input arrays can only be so large. Even on 64-bit machines, the
>> >>> table need not have more than 64 entries.
>> >>>
>> >>>  tab = 2 ** np.arange(np.iinfo(np.intp).bits)
>> >>>
>> >>
>> >> Or
>> >>
>> >> tab = 1 << arange(iinfo(intp).bits - 1)
>> >
>> > I think I prefer a readable solution to this for calls that are not in
>> > an inner loop.
>> >
>> > scipy.signal.fftconvolve uses the same as Ivo's solution
>> >
>> > <<  means it's much larger than 1 ?
>>
>> It's a bitshift operator. Basically "x << y" means "x * (2 ** y)" for
>> integer arguments.
>>
>> --
>> Robert Kern
>>
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>>  -- Umberto Eco
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/caec07bc/attachment.html>

From charlesr.harris at gmail.com  Sat Feb 27 14:07:58 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 27 Feb 2010 12:07:58 -0700
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <826c64da1002271103q50f6785ei44c52beb72a26593@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002270016r2e0fefe2m25894a4527e651bd@mail.gmail.com>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
	<3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
	<826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
	<826c64da1002271103q50f6785ei44c52beb72a26593@mail.gmail.com>
Message-ID: <e06186141002271107k66aa562bg5f5fad1f1edb6021@mail.gmail.com>

On Sat, Feb 27, 2010 at 12:03 PM, Ivo Maljevic <ivo.maljevic at gmail.com>wrote:

> Another suggestion, and this one also makes the parallel to Matlab. Make
> sure it works for vectors:
>
> >>> a=np.array([7,11,250])
> >>> nextpow2(a)
> array([ 3.,  4.,  8.])
>
>
>
In [27]: tab.searchsorted([7,11,250])
Out[27]: array([3, 4, 8])

In [28]: np.frexp([7,11,250])[1]
Out[28]: array([3, 4, 8], dtype=int32)

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/a73f3b94/attachment.html>

From josef.pktd at gmail.com  Sat Feb 27 14:16:25 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 27 Feb 2010 14:16:25 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <e06186141002271107k66aa562bg5f5fad1f1edb6021@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<45d1ab481002270022o1f215a1aj3fd09c82d5fd924a@mail.gmail.com>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
	<3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
	<826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
	<826c64da1002271103q50f6785ei44c52beb72a26593@mail.gmail.com>
	<e06186141002271107k66aa562bg5f5fad1f1edb6021@mail.gmail.com>
Message-ID: <1cd32cbb1002271116m551e7136v500cac8369b3ba5a@mail.gmail.com>

On Sat, Feb 27, 2010 at 2:07 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sat, Feb 27, 2010 at 12:03 PM, Ivo Maljevic <ivo.maljevic at gmail.com>
> wrote:
>>
>> Another suggestion, and this one also makes the parallel to Matlab. Make
>> sure it works for vectors:
>>
>> >>> a=np.array([7,11,250])
>> >>> nextpow2(a)
>> array([ 3.,? 4.,? 8.])
>>
>
> In [27]: tab.searchsorted([7,11,250])
> Out[27]: array([3, 4, 8])
>
> In [28]: np.frexp([7,11,250])[1]
> Out[28]: array([3, 4, 8], dtype=int32)

It looks like frexp is doubling the array if n is already an integer power

>>> np.frexp(2**np.arange(5))
(array([ 0.5,  0.5,  0.5,  0.5,  0.5]), array([1, 2, 3, 4, 5]))
>>> np.arange(5)
array([0, 1, 2, 3, 4])
>>> np.ceil(np.log2(2**np.arange(5)))
array([ 0.,  1.,  2.,  3.,  4.])


Josef

>
> <snip>
>
> Chuck
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From charlesr.harris at gmail.com  Sat Feb 27 14:59:18 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 27 Feb 2010 12:59:18 -0700
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <1cd32cbb1002271116m551e7136v500cac8369b3ba5a@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<826c64da1002270202j34b402d6o9ab5e22dc0f55c79@mail.gmail.com>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
	<3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
	<826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
	<826c64da1002271103q50f6785ei44c52beb72a26593@mail.gmail.com>
	<e06186141002271107k66aa562bg5f5fad1f1edb6021@mail.gmail.com>
	<1cd32cbb1002271116m551e7136v500cac8369b3ba5a@mail.gmail.com>
Message-ID: <e06186141002271159sb07aaffj2603153f4e9fd869@mail.gmail.com>

On Sat, Feb 27, 2010 at 12:16 PM, <josef.pktd at gmail.com> wrote:

> On Sat, Feb 27, 2010 at 2:07 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > On Sat, Feb 27, 2010 at 12:03 PM, Ivo Maljevic <ivo.maljevic at gmail.com>
> > wrote:
> >>
> >> Another suggestion, and this one also makes the parallel to Matlab. Make
> >> sure it works for vectors:
> >>
> >> >>> a=np.array([7,11,250])
> >> >>> nextpow2(a)
> >> array([ 3.,  4.,  8.])
> >>
> >
> > In [27]: tab.searchsorted([7,11,250])
> > Out[27]: array([3, 4, 8])
> >
> > In [28]: np.frexp([7,11,250])[1]
> > Out[28]: array([3, 4, 8], dtype=int32)
>
> It looks like frexp is doubling the array if n is already an integer power
>
> >>> np.frexp(2**np.arange(5))
> (array([ 0.5,  0.5,  0.5,  0.5,  0.5]), array([1, 2, 3, 4, 5]))
> >>> np.arange(5)
> array([0, 1, 2, 3, 4])
> >>> np.ceil(np.log2(2**np.arange(5)))
> array([ 0.,  1.,  2.,  3.,  4.])
>
>
Sure enough ;) Looks like the mantissa need to be checked for .5.

In [22]: def nextpow2(x): m, e = np.frexp(np.abs(x)); return e - (m == .5)
   ....:

In [23]: nextpow2(.5)
Out[23]: -1

In [24]: nextpow2(.25)
Out[24]: -2

In [25]: nextpow2(1)
Out[25]: 0

In [26]: nextpow2(2)
Out[26]: 1

Which brings up the question: is nextpow2(.5) 0 or -1?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/c2c6a936/attachment.html>

From ivo.maljevic at gmail.com  Sat Feb 27 16:14:06 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Sat, 27 Feb 2010 16:14:06 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <e06186141002271159sb07aaffj2603153f4e9fd869@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<3d375d731002271002k323b56ebg37f15c69b414f43f@mail.gmail.com>
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
	<3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
	<826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
	<826c64da1002271103q50f6785ei44c52beb72a26593@mail.gmail.com>
	<e06186141002271107k66aa562bg5f5fad1f1edb6021@mail.gmail.com>
	<1cd32cbb1002271116m551e7136v500cac8369b3ba5a@mail.gmail.com>
	<e06186141002271159sb07aaffj2603153f4e9fd869@mail.gmail.com>
Message-ID: <826c64da1002271314o36e677d6rb1d164fbdf8a6e35@mail.gmail.com>

-1

Which brings up the question: is nextpow2(.5) 0 or -1?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/cf993e2f/attachment.html>

From josef.pktd at gmail.com  Sat Feb 27 16:38:06 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 27 Feb 2010 16:38:06 -0500
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <826c64da1002271314o36e677d6rb1d164fbdf8a6e35@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<e06186141002271031x4d245be6s45868c271c171def@mail.gmail.com>
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
	<3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
	<826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
	<826c64da1002271103q50f6785ei44c52beb72a26593@mail.gmail.com>
	<e06186141002271107k66aa562bg5f5fad1f1edb6021@mail.gmail.com>
	<1cd32cbb1002271116m551e7136v500cac8369b3ba5a@mail.gmail.com>
	<e06186141002271159sb07aaffj2603153f4e9fd869@mail.gmail.com>
	<826c64da1002271314o36e677d6rb1d164fbdf8a6e35@mail.gmail.com>
Message-ID: <1cd32cbb1002271338w3690bb2aq1c39470f314a89a8@mail.gmail.com>

On Sat, Feb 27, 2010 at 4:14 PM, Ivo Maljevic <ivo.maljevic at gmail.com> wrote:
> -1
>
> Which brings up the question: is nextpow2(.5) 0 or -1?
>

frexp has a discontinuity at zero, same as matlab
(or is this a computer science definition)

matlab:
>> nextpow2(0.5)
ans =
    -1
>> nextpow2(0.25)
ans =
    -2
>>
>> nextpow2(0.125)
ans =
    -3
>> nextpow2(1e-300)
ans =
  -996
>> nextpow2(0)
ans =
     0

Josef

> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From charlesr.harris at gmail.com  Sat Feb 27 16:47:09 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 27 Feb 2010 14:47:09 -0700
Subject: [SciPy-User] frequency components of a signal buried in a noisy
	time domain signal
In-Reply-To: <1cd32cbb1002271338w3690bb2aq1c39470f314a89a8@mail.gmail.com>
References: <web-130795898@uni-stuttgart.de>
	<1cd32cbb1002271041u180c9ab6gf34573952359a500@mail.gmail.com>
	<3d375d731002271045x45dbb6d8m4bd7b9897014c5cb@mail.gmail.com>
	<826c64da1002271058x30fe64eci8339a245a7629f4d@mail.gmail.com>
	<826c64da1002271103q50f6785ei44c52beb72a26593@mail.gmail.com>
	<e06186141002271107k66aa562bg5f5fad1f1edb6021@mail.gmail.com>
	<1cd32cbb1002271116m551e7136v500cac8369b3ba5a@mail.gmail.com>
	<e06186141002271159sb07aaffj2603153f4e9fd869@mail.gmail.com>
	<826c64da1002271314o36e677d6rb1d164fbdf8a6e35@mail.gmail.com>
	<1cd32cbb1002271338w3690bb2aq1c39470f314a89a8@mail.gmail.com>
Message-ID: <e06186141002271347n39d19e1er64910be8b9db8808@mail.gmail.com>

On Sat, Feb 27, 2010 at 2:38 PM, <josef.pktd at gmail.com> wrote:

> On Sat, Feb 27, 2010 at 4:14 PM, Ivo Maljevic <ivo.maljevic at gmail.com>
> wrote:
> > -1
> >
> > Which brings up the question: is nextpow2(.5) 0 or -1?
> >
>
> frexp has a discontinuity at zero, same as matlab
> (or is this a computer science definition)
>
> matlab:
> >> nextpow2(0.5)
> ans =
>    -1
> >> nextpow2(0.25)
> ans =
>    -2
> >>
> >> nextpow2(0.125)
> ans =
>    -3
> >> nextpow2(1e-300)
> ans =
>  -996
> >> nextpow2(0)
> ans =
>     0
>
>
Yeah, zero is special, I was thinking -inf, but it's not an integer.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100227/30618b10/attachment.html>