From ralf.gommers at googlemail.com  Thu Apr  1 05:37:56 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Thu, 1 Apr 2010 17:37:56 +0800
Subject: [SciPy-User] New official SciPy release ?
In-Reply-To: <4BB360AC.80809@gmail.com>
References: <4BB32112.7080708@gmail.com>
	<n2y1cd32cbb1003310416k486e6ea4kbbdf698748e11fc5@mail.gmail.com>
	<4BB360AC.80809@gmail.com>
Message-ID: <h2rdde7764a1004010237h42487cd4g500684765e198f38@mail.gmail.com>

Hi Axel,

On Wed, Mar 31, 2010 at 10:48 PM, Axel Kowald <axel.kowald at gmail.com> wrote:

> Hello,
>
> I wonder when there will be a new official SciPy release ?
> Documentation is already available for 0.8 but the current release is 0.7.1
> I'm not really able to build SciPy from the sources for Windows and so I
> wanted to hear if there are any planned release dates ?
>
> Good question. There is no date set for the 0.8 release yet. There will be
an 0.7.2 release candidate soon, but that will not have any new features
compared with 0.7.1 except for that it should work correctly with numpy
1.4.1.

For scipy 0.8, I would think before the SciPy 2010 conference (last week of
June). It would be good to hear what scipy developers think.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100401/dc56d29d/attachment.html>

From david at silveregg.co.jp  Thu Apr  1 05:47:13 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Thu, 01 Apr 2010 18:47:13 +0900
Subject: [SciPy-User] New official SciPy release ?
In-Reply-To: <h2rdde7764a1004010237h42487cd4g500684765e198f38@mail.gmail.com>
References: <4BB32112.7080708@gmail.com>
	<n2y1cd32cbb1003310416k486e6ea4kbbdf698748e11fc5@mail.gmail.com>
	<4BB360AC.80809@gmail.com>
	<h2rdde7764a1004010237h42487cd4g500684765e198f38@mail.gmail.com>
Message-ID: <4BB46BA1.7000906@silveregg.co.jp>

Ralf Gommers wrote:
> Hi Axel,
> 
> On Wed, Mar 31, 2010 at 10:48 PM, Axel Kowald <axel.kowald at gmail.com> wrote:
> 
>> Hello,
>>
>> I wonder when there will be a new official SciPy release ?
>> Documentation is already available for 0.8 but the current release is 0.7.1
>> I'm not really able to build SciPy from the sources for Windows and so I
>> wanted to hear if there are any planned release dates ?
>>
>> Good question. There is no date set for the 0.8 release yet. There will be
> an 0.7.2 release candidate soon, but that will not have any new features
> compared with 0.7.1 except for that it should work correctly with numpy
> 1.4.1.

Is there something holding back 0.7.2 ? We should really try to release 
it (with numpy 1.4.1) ASAP, and I don't think much needs to be done.

cheers,

David


From ralf.gommers at googlemail.com  Thu Apr  1 06:01:10 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Thu, 1 Apr 2010 18:01:10 +0800
Subject: [SciPy-User] New official SciPy release ?
In-Reply-To: <4BB46BA1.7000906@silveregg.co.jp>
References: <4BB32112.7080708@gmail.com>
	<n2y1cd32cbb1003310416k486e6ea4kbbdf698748e11fc5@mail.gmail.com>
	<4BB360AC.80809@gmail.com>
	<h2rdde7764a1004010237h42487cd4g500684765e198f38@mail.gmail.com>
	<4BB46BA1.7000906@silveregg.co.jp>
Message-ID: <n2qdde7764a1004010301h6b05f08cre2c70058c1f029c@mail.gmail.com>

On Thu, Apr 1, 2010 at 5:47 PM, David Cournapeau <david at silveregg.co.jp>wrote:

>
> Is there something holding back 0.7.2 ? We should really try to release
> it (with numpy 1.4.1) ASAP, and I don't think much needs to be done.
>
> Mainly my binary-building skills I'm afraid. I have some numpy 1.4.1
binaries ready, but scipy is proving to be a lot harder. I may try to pick
your brain again this weekend.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100401/2df40265/attachment.html>

From guyer at nist.gov  Thu Apr  1 15:56:22 2010
From: guyer at nist.gov (Jonathan Guyer)
Date: Thu, 1 Apr 2010 15:56:22 -0400
Subject: [SciPy-User] ANN: FiPy 2.1
Message-ID: <12EB0FA7-A45C-4320-9D62-81CBE3C38C87@nist.gov>

We are please to announce the release of FiPy 2.1.

    http://www.ctcms.nist.gov/fipy

The relatively small change in version number belies significant  
advances
in FiPy capabilities. This release did not receive a "full" version
increment because it is completely (er... almost...) compatible with  
older
scripts.

The significant changes since version 2.0.2 are:

- FiPy can use Trilinos for solving in parallel.

- We have switched from MayaVi 1 to Mayavi 2. This Viewer is an
   independent process that allows interaction with the display while a
   simulation is running.

- Documentation has been switched to Sphinx, allowing the entire
   manual to be available on the web and for our documentation to link  
to
   the documentation for packages such as numpy, scipy, matplotlib,  
and for
   Python itself.



========================================================================

FiPy is an object oriented, partial differential equation (PDE) solver,
written in Python, based on a standard finite volume (FV) approach. The
framework has been developed in the Metallurgy Division and Center for
Theoretical and Computational Materials Science (CTCMS), in the  
Materials
Science and Engineering Laboratory (MSEL) at the National Institute of
Standards and Technology (NIST).

The solution of coupled sets of PDEs is ubiquitous to the numerical
simulation of science problems. Numerous PDE solvers exist, using a  
variety
of languages and numerical approaches. Many are proprietary, expensive  
and
difficult to customize. As a result, scientists spend considerable
resources repeatedly developing limited tools for specific problems. Our
approach, combining the FV method and Python, provides a tool that is
extensible, powerful and freely available. A significant advantage to
Python is the existing suite of tools for array calculations, sparse
matrices and data rendering.

The FiPy framework includes terms for transient diffusion, convection  
and
standard sources, enabling the solution of arbitrary combinations of
coupled elliptic, hyperbolic and parabolic PDEs. Currently implemented
models include phase field treatments of polycrystalline, dendritic, and
electrochemical phase transformations as well as a level set treatment  
of
the electrodeposition process.



From tonyyu at MIT.EDU  Fri Apr  2 15:13:19 2010
From: tonyyu at MIT.EDU (Tony S Yu)
Date: Fri, 2 Apr 2010 15:13:19 -0400
Subject: [SciPy-User] Fancy indexing bug in sparse.lil_matrix?
Message-ID: <E408FC2C-489E-4AF0-9B6E-435DB3B6AB9D@mit.edu>

The scipy tutorial suggests that lil_matrix supports fancy indexing, but that doesn't seem to be completely true:

>>> from scipy import sparse
>>>
>>> N = 5
>>> A_lil = sparse.eye(N, N, format='lil')
>>> idx = range(N)
>>> print A_lil[idx, idx].todense()
[[ 1.  1.  1.  1.  1.]]
>>> A_lil[idx, idx] = 5
>>> print A_lil.todense()
[[ 5.  5.  5.  5.  5.]
 [ 5.  5.  5.  5.  5.]
 [ 5.  5.  5.  5.  5.]
 [ 5.  5.  5.  5.  5.]
 [ 5.  5.  5.  5.  5.]]

In the above code, getting the main diagonal works as expected, but setting the main diagonal produces unexpected results. Is this a known issue?

This issue doesn't seem to be the same as Ticket #1064 (plus, I believe that issue is fixed in my version of scipy).

-Tony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100402/35ae5651/attachment.html>

From mailinglists at xgm.de  Sat Apr  3 06:02:40 2010
From: mailinglists at xgm.de (Florian Lindner)
Date: Sat, 3 Apr 2010 12:02:40 +0200
Subject: [SciPy-User] Use of numpy functions instead builtins
Message-ID: <201004031202.40236.mailinglists@xgm.de>

Hello,
I've code that have a lot longer formulas like this on:

tan_theta = ( 2 * (1/tan(sa))*(Ma**2*sin(sa)**2-1)) / ( 2+Ma**2 * (self.f_prop.kappa+1 - 2 * sin(sa)**2) )

all variables are scalars so it works fine with the stuff I get from the math package. Since I"m using scipy anyway in my app I wonder if there is any advantage using the numpy math functions (sin, cos, ..., exp, power). Is there any advantage using them over the python bultins?

Thanks,

Florian 


From peridot.faceted at gmail.com  Sat Apr  3 07:15:23 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 3 Apr 2010 07:15:23 -0400
Subject: [SciPy-User] Use of numpy functions instead builtins
In-Reply-To: <201004031202.40236.mailinglists@xgm.de>
References: <201004031202.40236.mailinglists@xgm.de>
Message-ID: <v2vce557a361004030415we802a198me6e09aa2d6c6acbf@mail.gmail.com>

On 3 April 2010 06:02, Florian Lindner <mailinglists at xgm.de> wrote:
> Hello,
> I've code that have a lot longer formulas like this on:
>
> tan_theta = ( 2 * (1/tan(sa))*(Ma**2*sin(sa)**2-1)) / ( 2+Ma**2 * (self.f_prop.kappa+1 - 2 * sin(sa)**2) )
>
> all variables are scalars so it works fine with the stuff I get from the math package. Since I"m using scipy anyway in my app I wonder if there is any advantage using the numpy math functions (sin, cos, ..., exp, power). Is there any advantage using them over the python bultins?

The short answer is yes, you should use the numpy versions.

The chief advantage, for you, is that your functions will work on
whole arrays of numbers:
In [1]: np.exp([0,1,2,3])
Out[1]: array([  1.        ,   2.71828183,   7.3890561 ,  20.08553692])

Beyond the simple convenience of not having to write loops, this is
usually much faster than looping over the elements, and for simple
scalar formulas, you get it more or less for free.

There may be other advantages to using the functions in scipy, in
particular; for example, scipy's cosine function works on complex
numbers, and scipy provides a square root that returns complex numbers
when given negative inputs. The numpy functions may also be cleaner
about handling infinities and not-a-number inputs (that is, failing
gracefully when roundoff error becomes overwhelming). But really, the
vectorization is the key advantage.

Anne

> Thanks,
>
> Florian
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From warren.weckesser at enthought.com  Sat Apr  3 07:32:39 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sat, 03 Apr 2010 06:32:39 -0500
Subject: [SciPy-User] Use of numpy functions instead builtins
In-Reply-To: <v2vce557a361004030415we802a198me6e09aa2d6c6acbf@mail.gmail.com>
References: <201004031202.40236.mailinglists@xgm.de>
	<v2vce557a361004030415we802a198me6e09aa2d6c6acbf@mail.gmail.com>
Message-ID: <4BB72757.7050607@enthought.com>

Anne Archibald wrote:
> On 3 April 2010 06:02, Florian Lindner <mailinglists at xgm.de> wrote:
>   
>> Hello,
>> I've code that have a lot longer formulas like this on:
>>
>> tan_theta = ( 2 * (1/tan(sa))*(Ma**2*sin(sa)**2-1)) / ( 2+Ma**2 * (self.f_prop.kappa+1 - 2 * sin(sa)**2) )
>>
>> all variables are scalars so it works fine with the stuff I get from the math package. Since I"m using scipy anyway in my app I wonder if there is any advantage using the numpy math functions (sin, cos, ..., exp, power). Is there any advantage using them over the python bultins?
>>     
>
> The short answer is yes, you should use the numpy versions.
>
> The chief advantage, for you, is that your functions will work on
> whole arrays of numbers:
> In [1]: np.exp([0,1,2,3])
> Out[1]: array([  1.        ,   2.71828183,   7.3890561 ,  20.08553692])
>
> Beyond the simple convenience of not having to write loops, this is
> usually much faster than looping over the elements, and for simple
> scalar formulas, you get it more or less for free.
>
> There may be other advantages to using the functions in scipy, in
> particular; for example, scipy's cosine function works on complex
> numbers, and scipy provides a square root that returns complex numbers
> when given negative inputs. The numpy functions may also be cleaner
> about handling infinities and not-a-number inputs (that is, failing
> gracefully when roundoff error becomes overwhelming). But really, the
> vectorization is the key advantage.
>
>   

On the other hand, the numpy versions are *much* slower, so if all the 
variables are scalars and performance is an issue, you might prefer the 
standard math functions.

Warren


> Anne
>
>   
>> Thanks,
>>
>> Florian
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>     
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From peridot.faceted at gmail.com  Sat Apr  3 08:39:12 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 3 Apr 2010 08:39:12 -0400
Subject: [SciPy-User] Use of numpy functions instead builtins
In-Reply-To: <4BB72757.7050607@enthought.com>
References: <201004031202.40236.mailinglists@xgm.de>
	<v2vce557a361004030415we802a198me6e09aa2d6c6acbf@mail.gmail.com>
	<4BB72757.7050607@enthought.com>
Message-ID: <m2xce557a361004030539tfdf22088ic8ab7358401d1da1@mail.gmail.com>

On 3 April 2010 07:32, Warren Weckesser <warren.weckesser at enthought.com> wrote:
> Anne Archibald wrote:
>> On 3 April 2010 06:02, Florian Lindner <mailinglists at xgm.de> wrote:
>>
>>> Hello,
>>> I've code that have a lot longer formulas like this on:
>>>
>>> tan_theta = ( 2 * (1/tan(sa))*(Ma**2*sin(sa)**2-1)) / ( 2+Ma**2 * (self.f_prop.kappa+1 - 2 * sin(sa)**2) )
>>>
>>> all variables are scalars so it works fine with the stuff I get from the math package. Since I"m using scipy anyway in my app I wonder if there is any advantage using the numpy math functions (sin, cos, ..., exp, power). Is there any advantage using them over the python bultins?
>>>
>>
>> The short answer is yes, you should use the numpy versions.
>>
>> The chief advantage, for you, is that your functions will work on
>> whole arrays of numbers:
>> In [1]: np.exp([0,1,2,3])
>> Out[1]: array([ ?1. ? ? ? ?, ? 2.71828183, ? 7.3890561 , ?20.08553692])
>>
>> Beyond the simple convenience of not having to write loops, this is
>> usually much faster than looping over the elements, and for simple
>> scalar formulas, you get it more or less for free.
>>
>> There may be other advantages to using the functions in scipy, in
>> particular; for example, scipy's cosine function works on complex
>> numbers, and scipy provides a square root that returns complex numbers
>> when given negative inputs. The numpy functions may also be cleaner
>> about handling infinities and not-a-number inputs (that is, failing
>> gracefully when roundoff error becomes overwhelming). But really, the
>> vectorization is the key advantage.
>>
>>
>
> On the other hand, the numpy versions are *much* slower, so if all the
> variables are scalars and performance is an issue, you might prefer the
> standard math functions.

Just to be clear: if you're calling the functions on scalars, the
numpy functions are slower than the python library functions. This is
because they do a certain amount of marshaling and type conversion. In
the very common case that you want to apply the same function to many
data points, if you pass those data points as an array to a numpy
function - or to your function that uses numpy functions - you will
not only avoid having to write a loop, the code will run much faster.

Anne

> Warren
>
>
>> Anne
>>
>>
>>> Thanks,
>>>
>>> Florian
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From cr.anil at gmail.com  Sat Apr  3 08:39:29 2010
From: cr.anil at gmail.com (Anil C R)
Date: Sat, 3 Apr 2010 18:09:29 +0530
Subject: [SciPy-User] label2rgb equivalent
Message-ID: <n2h8cf7db601004030539kd6810ddeg8c6dcbe1d04e2b0a@mail.gmail.com>

Hi all,

Is there an equivalent to the Matlab function label2rgb in scipy or numpy ?

Regards,
Anil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100403/b3274909/attachment.html>

From paul.anton.letnes at gmail.com  Sat Apr  3 10:57:31 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Sat, 3 Apr 2010 07:57:31 -0700
Subject: [SciPy-User] Use of numpy functions instead builtins
In-Reply-To: <201004031202.40236.mailinglists@xgm.de>
References: <201004031202.40236.mailinglists@xgm.de>
Message-ID: <D1A24A02-C630-4766-B036-0B5CFB40CA1E@gmail.com>



On 3. apr. 2010, at 03.02, Florian Lindner wrote:

> Hello,
> I've code that have a lot longer formulas like this on:
> 
> tan_theta = ( 2 * (1/tan(sa))*(Ma**2*sin(sa)**2-1)) / ( 2+Ma**2 * (self.f_prop.kappa+1 - 2 * sin(sa)**2) )
> 
> all variables are scalars so it works fine with the stuff I get from the math package. Since I"m using scipy anyway in my app I wonder if there is any advantage using the numpy math functions (sin, cos, ..., exp, power). Is there any advantage using them over the python bultins?
> 

Hi,

I'm guessing this is not usually a problem, but I've also come across bugs in the python cmath library, so I don't trust it anymore. I'm not sure how often it is a problem, but it's damn annoying if you keep debugging and it's the library functions that are wrong.

Oh, and yes, the corresponding numpy functions were right.

cheers,
Paul.

From pav+sp at iki.fi  Sat Apr  3 11:16:08 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Sat, 3 Apr 2010 15:16:08 +0000 (UTC)
Subject: [SciPy-User] Use of numpy functions instead builtins
References: <201004031202.40236.mailinglists@xgm.de>
	<D1A24A02-C630-4766-B036-0B5CFB40CA1E@gmail.com>
Message-ID: <hp7m3o$6la$1@dough.gmane.org>

Sat, 03 Apr 2010 07:57:31 -0700, Paul Anton Letnes wrote:
[clip]
> I'm guessing this is not usually a problem, but I've also come across
> bugs in the python cmath library, so I don't trust it anymore. I'm not
> sure how often it is a problem, but it's damn annoying if you keep
> debugging and it's the library functions that are wrong.
> 
> Oh, and yes, the corresponding numpy functions were right.

I remember the cmath module was revamped in Python 2.6. Before that there 
was indeed a bug at least in asinh.

-- 
Pauli Virtanen



From zachary.pincus at yale.edu  Sat Apr  3 11:39:50 2010
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Sat, 3 Apr 2010 11:39:50 -0400
Subject: [SciPy-User] label2rgb equivalent
In-Reply-To: <n2h8cf7db601004030539kd6810ddeg8c6dcbe1d04e2b0a@mail.gmail.com>
References: <n2h8cf7db601004030539kd6810ddeg8c6dcbe1d04e2b0a@mail.gmail.com>
Message-ID: <961D9C63-AA72-4A06-BEEF-8B6C35053C70@yale.edu>

>
> Hi all,
>
> Is there an equivalent to the Matlab function label2rgb in scipy or  
> numpy ?
>
> Regards,
> Anil


matplotlib might have such a function, but numpy fancy indexing makes  
this almost a one-liner:

Let "labels" be your label array (of arbitrary shape); assume "colors"  
is an Nx3 array of rgb triples, where N > labels.max().

Then:
colors[labels]

gives you an array of the same shape as labels except that it has an  
extra dimension at the end containing the RGB triples. For example:
labels = numpy.random.random_integers(10, size=(10,10))
colors = numpy.arange(75).reshape((25,3))
labeled = colors[labels]
labeled.shape # (10,10,3)

you could use numpy.rollaxis to change which dimension the RGB data is  
in:
numpy.rollaxis(labeled, 2).shape # (3, 10, 10)

you could also have colors be a structured array...

As far as making the "colors" array, there are many strategies. You  
could use random integers, or make a colormap (matplotlib has some  
tools, or you could roll your own easily enough.

Zach


From aisaac at american.edu  Sat Apr  3 11:40:52 2010
From: aisaac at american.edu (Alan G Isaac)
Date: Sat, 03 Apr 2010 11:40:52 -0400
Subject: [SciPy-User] Use of numpy functions instead builtins
In-Reply-To: <hp7m3o$6la$1@dough.gmane.org>
References: <201004031202.40236.mailinglists@xgm.de>	<D1A24A02-C630-4766-B036-0B5CFB40CA1E@gmail.com>
	<hp7m3o$6la$1@dough.gmane.org>
Message-ID: <4BB76184.6060203@american.edu>

On 4/3/2010 11:16 AM, Pauli Virtanen wrote:
> I remember the cmath module was revamped in Python 2.6. Before that there
> was indeed a bug at least in asinh.


http://bugs.python.org/issue1381

The background story is a telling one, iirc:
code accepted in 2001 without accompanying tests.

Alan Isaac




From harald.schilly at gmail.com  Sat Apr  3 14:36:47 2010
From: harald.schilly at gmail.com (Harald Schilly)
Date: Sat, 3 Apr 2010 20:36:47 +0200
Subject: [SciPy-User] label2rgb equivalent
In-Reply-To: <961D9C63-AA72-4A06-BEEF-8B6C35053C70@yale.edu>
References: <n2h8cf7db601004030539kd6810ddeg8c6dcbe1d04e2b0a@mail.gmail.com> 
	<961D9C63-AA72-4A06-BEEF-8B6C35053C70@yale.edu>
Message-ID: <m2r20548feb1004031136n21ed6729wb86b6d505d131018@mail.gmail.com>

On Sat, Apr 3, 2010 at 17:39, Zachary Pincus <zachary.pincus at yale.edu> wrote:
> matplotlib might have such a function,

matplotlib.cm generates scalable color maps. i.e. there is a set of
functions and you tell it how many steps should be generated.

http://matplotlib.sourceforge.net/api/cm_api.html

i.e. something like:

matplotlib.cm.datad.keys() lists you all registered map functions and
matplotlib.cm.get_cmap(...) gives you the colormap.

h


From josef.pktd at gmail.com  Sat Apr  3 14:48:02 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 3 Apr 2010 14:48:02 -0400
Subject: [SciPy-User] stats.kstwobign
Message-ID: <n2t1cd32cbb1004031148r4222bab2v203339deb21205c3@mail.gmail.com>

Just a curious piece of information I just found:

I was looking at the asymptotic distributions for tests for structural
changes in regression with unknown break point, and found that it is
available in scipy

stats.kstwobign is the distribution of the supremum of  the absolute
value of a standard  Brownian Bridge

>>> stats.kstwobign.isf([0.01,0.05,0.1])
array([ 1.62762361,  1.35809864,  1.22384787])

gives the critical values.

I think this might be useful for quite a few supremum tests.

Josef


From dkdropbox at gmail.com  Mon Apr  5 14:26:44 2010
From: dkdropbox at gmail.com (David Kim)
Date: Mon, 5 Apr 2010 14:26:44 -0400
Subject: [SciPy-User] scikits.timeseries installation
In-Reply-To: <s2if5014d111004030620gd3b006adl44181f518690e8b9@mail.gmail.com>
References: <s2if5014d111004030620gd3b006adl44181f518690e8b9@mail.gmail.com>
Message-ID: <i2kf5014d111004051126pdb7f70d8k7287cd4b3f1b2dd7@mail.gmail.com>

Hello,

I am trying to install scikits.timeseries on Leopard 10.6.3. Running into
some problems and unfortunately I lack the experience to know where to look.
If anyone can point me in the right direction, i'd appreciate it. I've
included error messages from attempting easy_install, pip install, python
setup.py install, and an attempt to import scikits.timeseries into ipython
(as it looks like the egg made it into site packages). Thanks.

easy_install yields the following error:

Sat Apr 03 DK ~$ easy_install scikits.timeseries
Searching for scikits.timeseries
Reading http://pypi.python.org/simple/scikits.timeseries/
Reading http://pytseries.sourceforge.net
Reading https://sourceforge.net/project/showfiles.php?group_id=239604
No local packages or download links found for scikits.timeseries
error: Could not find suitable distribution for
Requirement.parse('scikits.timeseries')

pip install yields the following error:

Sat Apr 03 DK ~$ pip install scikits.timeseries
Downloading/unpacking scikits.timeseries
Could not find any downloads that satisfy the requirement scikits.timeseries
No distributions at all found for scikits.timeseries
Storing complete log in ./pip-log.txt

python setup.py install from tar yields the following (long):

Sat Apr 03 DK scikits.timeseries-0.91.3$ python setup.py install
running install
running bdist_egg
running egg_info
running build_src
build_src
building extension "scikits.timeseries.cseries" sources
building data_files sources
build_src: building npy-pkg config files
writing requirements to scikits.timeseries.egg-info/requires.txt
writing scikits.timeseries.egg-info/PKG-INFO
writing namespace_packages to
scikits.timeseries.egg-info/namespace_packages.txt
writing top-level names to scikits.timeseries.egg-info/top_level.txt
writing dependency_links to scikits.timeseries.egg-info/dependency_links.txt
reading manifest file 'scikits.timeseries.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching
'scikits/timeseries/doc/source/generated'
no previously-included directories found matching
'scikits/timeseries/doc/build'
writing manifest file 'scikits.timeseries.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.5-i386/egg
running install_lib
running build_py
creating build
creating build/lib.macosx-10.5-i386-2.6
creating build/lib.macosx-10.5-i386-2.6/scikits
copying ./scikits/__init__.py -> build/lib.macosx-10.5-i386-2.6/scikits
creating build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/__init__.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/_preview.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/_tools.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/const.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/extras.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/parser.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/setup.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/tdates.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/trecords.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/tseries.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
copying scikits/timeseries/version.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries
creating build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib
copying scikits/timeseries/lib/__init__.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib
copying scikits/timeseries/lib/avcf.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib
copying scikits/timeseries/lib/interpolate.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib
copying scikits/timeseries/lib/moving_funcs.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib
copying scikits/timeseries/lib/plotlib.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib
copying scikits/timeseries/lib/reportlib.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib
copying scikits/timeseries/lib/tstables.py ->
build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
building 'scikits.timeseries.cseries' extension
compiling C sources
C compiler: gcc -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3
-arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk
-I/Library/Frameworks/Python.framework/Versions/6.1/include

creating build/temp.macosx-10.5-i386-2.6
creating build/temp.macosx-10.5-i386-2.6/scikits
creating build/temp.macosx-10.5-i386-2.6/scikits/timeseries
creating build/temp.macosx-10.5-i386-2.6/scikits/timeseries/src
compile options:
'-I/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/site-packages/numpy/core/include/numpy
-Iscikits/timeseries/include
-I/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/site-packages/numpy/core/include
-I/Library/Frameworks/Python.framework/Versions/6.1/include/python2.6 -c'
gcc: scikits/timeseries/src/c_dates.c
gcc: scikits/timeseries/src/cseries.c
gcc: scikits/timeseries/src/c_lib.c
gcc: scikits/timeseries/src/c_tseries.c
gcc -g -L/usr/local/lib
-L/Library/Frameworks/Python.framework/Versions/6.1/lib -bundle -undefined
dynamic_lookup -arch i386
build/temp.macosx-10.5-i386-2.6/scikits/timeseries/src/c_lib.o
build/temp.macosx-10.5-i386-2.6/scikits/timeseries/src/c_dates.o
build/temp.macosx-10.5-i386-2.6/scikits/timeseries/src/c_tseries.o
build/temp.macosx-10.5-i386-2.6/scikits/timeseries/src/cseries.o -o
build/lib.macosx-10.5-i386-2.6/scikits/timeseries/cseries.so
running scons
creating build/bdist.macosx-10.5-i386
creating build/bdist.macosx-10.5-i386/egg
creating build/bdist.macosx-10.5-i386/egg/scikits
copying build/lib.macosx-10.5-i386-2.6/scikits/__init__.py ->
build/bdist.macosx-10.5-i386/egg/scikits
creating build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/__init__.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/_preview.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/_tools.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/const.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/cseries.so ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/extras.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
creating build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib/__init__.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib/avcf.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib/interpolate.py
-> build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib
copying
build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib/moving_funcs.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib/plotlib.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib/reportlib.py
-> build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/lib/tstables.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/parser.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/setup.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/tdates.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/trecords.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/tseries.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
copying build/lib.macosx-10.5-i386-2.6/scikits/timeseries/version.py ->
build/bdist.macosx-10.5-i386/egg/scikits/timeseries
byte-compiling build/bdist.macosx-10.5-i386/egg/scikits/__init__.py to
__init__.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/__init__.py to
__init__.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/_preview.py to
_preview.pyc
byte-compiling build/bdist.macosx-10.5-i386/egg/scikits/timeseries/_tools.py
to _tools.pyc
byte-compiling build/bdist.macosx-10.5-i386/egg/scikits/timeseries/const.py
to const.pyc
byte-compiling build/bdist.macosx-10.5-i386/egg/scikits/timeseries/extras.py
to extras.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib/__init__.py to
__init__.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib/avcf.py to avcf.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib/interpolate.py to
interpolate.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib/moving_funcs.py to
moving_funcs.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib/plotlib.py to
plotlib.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib/reportlib.py to
reportlib.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/lib/tstables.py to
tstables.pyc
byte-compiling build/bdist.macosx-10.5-i386/egg/scikits/timeseries/parser.py
to parser.pyc
byte-compiling build/bdist.macosx-10.5-i386/egg/scikits/timeseries/setup.py
to setup.pyc
byte-compiling build/bdist.macosx-10.5-i386/egg/scikits/timeseries/tdates.py
to tdates.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/trecords.py to
trecords.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/tseries.py to
tseries.pyc
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/version.py to
version.pyc
creating stub loader for scikits/timeseries/cseries.so
byte-compiling
build/bdist.macosx-10.5-i386/egg/scikits/timeseries/cseries.py to
cseries.pyc
installing package data to build/bdist.macosx-10.5-i386/egg
running install_data
running install_clib
customize UnixCCompiler
creating build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying scikits.timeseries.egg-info/PKG-INFO ->
build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying scikits.timeseries.egg-info/SOURCES.txt ->
build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying scikits.timeseries.egg-info/dependency_links.txt ->
build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying scikits.timeseries.egg-info/namespace_packages.txt ->
build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying scikits.timeseries.egg-info/requires.txt ->
build/bdist.macosx-10.5-i386/egg/EGG-INFO
copying scikits.timeseries.egg-info/top_level.txt ->
build/bdist.macosx-10.5-i386/egg/EGG-INFO
writing build/bdist.macosx-10.5-i386/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
scikits.timeseries.version: module references __file__
creating dist
creating 'dist/scikits.timeseries-0.91.3-py2.6-macosx-10.5-i386.egg' and
adding 'build/bdist.macosx-10.5-i386/egg' to it
removing 'build/bdist.macosx-10.5-i386/egg' (and everything under it)
Processing scikits.timeseries-0.91.3-py2.6-macosx-10.5-i386.egg
creating
/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/site-packages/scikits.timeseries-0.91.3-py2.6-macosx-10.5-i386.egg
Extracting scikits.timeseries-0.91.3-py2.6-macosx-10.5-i386.egg to
/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/site-packages
Adding scikits.timeseries 0.91.3 to easy-install.pth file

Installed
/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/site-packages/scikits.timeseries-0.91.3-py2.6-macosx-10.5-i386.egg
Processing dependencies for scikits.timeseries==0.91.3
Searching for numpy==1.4.0
Best match: numpy 1.4.0
Adding numpy 1.4.0 to easy-install.pth file
No files installed.

Using
/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/site-packages
Finished processing dependencies for scikits.timeseries==0.91.3

I get the following traceback when attempting to import into ipython:

In [1]: import scikits.timeseries as ts
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/Library/Frameworks/Python.framework/Versions/6.1/LOCAL-REPO/scikits.timeseries-0.91.3/<ipython
console> in <module>()

/Library/Frameworks/Python.framework/Versions/6.1/LOCAL-REPO/scikits.timeseries-0.91.3/scikits/timeseries/__init__.py
in <module>()
     11 __date__     = '$Date: 2009-08-23 12:43:40 -0400 (Sun, 23 Aug 2009)
$'
     12
---> 13 import const
     14 import tdates
     15 from tdates import *

/Library/Frameworks/Python.framework/Versions/6.1/LOCAL-REPO/scikits.timeseries-0.91.3/scikits/timeseries/const.py
in <module>()
     77 __date__     = '$Date: 2008-06-18 21:42:59 -0400 (Wed, 18 Jun 2008)
$'
     78
---> 79 from cseries import freq_constants
     80
     81 """add constants in cseries.freq_constants dictionary to global
namespace

ImportError: No module named cseries

In [2]: import numpy as np

In [3]: import numpy.ma as ma

In [4]: import datetime

In [5]: import scikits.timeseries as ts
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/Library/Frameworks/Python.framework/Versions/6.1/LOCAL-REPO/scikits.timeseries-0.91.3/<ipython
console> in <module>()

/Library/Frameworks/Python.framework/Versions/6.1/LOCAL-REPO/scikits.timeseries-0.91.3/scikits/timeseries/__init__.py
in <module>()
     11 __date__     = '$Date: 2009-08-23 12:43:40 -0400 (Sun, 23 Aug 2009)
$'
     12
---> 13 import const
     14 import tdates
     15 from tdates import *

/Library/Frameworks/Python.framework/Versions/6.1/LOCAL-REPO/scikits.timeseries-0.91.3/scikits/timeseries/const.py
in <module>()
     77 __date__     = '$Date: 2008-06-18 21:42:59 -0400 (Wed, 18 Jun 2008)
$'
     78
---> 79 from cseries import freq_constants
     80
     81 """add constants in cseries.freq_constants dictionary to global
namespace

ImportError: No module named cseries








-- 
David Kim
notestoself.posterous.com

"I hear and I forget. I see and I remember. I do and I understand." --
 Confucius
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100405/2ad727be/attachment.html>

From pgmdevlist at gmail.com  Mon Apr  5 14:46:08 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 5 Apr 2010 14:46:08 -0400
Subject: [SciPy-User] scikits.timeseries installation
In-Reply-To: <i2kf5014d111004051126pdb7f70d8k7287cd4b3f1b2dd7@mail.gmail.com>
References: <s2if5014d111004030620gd3b006adl44181f518690e8b9@mail.gmail.com>
	<i2kf5014d111004051126pdb7f70d8k7287cd4b3f1b2dd7@mail.gmail.com>
Message-ID: <9A7DB437-1E00-4FE4-BF0B-63A7E5CB1181@gmail.com>

On Apr 5, 2010, at 2:26 PM, David Kim wrote:
> 
> 
> Hello,
> 
> I am trying to install scikits.timeseries on Leopard 10.6.3. 

Please follow the instructions on this page
http://pytseries.sourceforge.net/installation.html

I don't think we have an easy_install/pip option yet. I'd recommend downloading the SVN sources and running 
python setup.py install
in the directory where you downloaded the sources.
Please don't hesitate to contact me if you have any problem with the compilation.

I note that you're trying to install the package directory into /Library/... I'm not sure it's the best option. Try to install the package locally, using for example:
python setup.py install --user
That way, you install the package in your home directory.
Alternatively, you can use a virtual environment 
http://www.doughellmann.com/projects/virtualenvwrapper/
(That's what I'm using on my 10.6.3 machine)




From yosefmel at post.tau.ac.il  Tue Apr  6 05:22:38 2010
From: yosefmel at post.tau.ac.il (Yosef Meller)
Date: Tue, 6 Apr 2010 12:22:38 +0300
Subject: [SciPy-User] Is kKahan summation used in numpy.mean?
Message-ID: <201004061222.38879.yosefmel@post.tau.ac.il>

I seem to be getting a much better accuracy from numpy.mean then from a loop 
of a += val; a /= num_vals. I've seen some discussion on mailing lists about 
using Kahan summation for mean(), so I suspect this is the reason, but I can't 
for the life of me find if it was finally implemented or not. Anybody knows?

Also, if you know where it is in the numpy source tree, please tell - I 
searched it the whole morning.

Thanks,
Yosef.


From yosefmel at post.tau.ac.il  Tue Apr  6 08:26:03 2010
From: yosefmel at post.tau.ac.il (Yosef Meller)
Date: Tue, 6 Apr 2010 15:26:03 +0300
Subject: [SciPy-User] Is kKahan summation used in numpy.mean?
In-Reply-To: <201004061222.38879.yosefmel@post.tau.ac.il>
References: <201004061222.38879.yosefmel@post.tau.ac.il>
Message-ID: <201004061526.03736.yosefmel@post.tau.ac.il>

On ??? ????? 06 ????? 2010 12:22:38 you wrote:
> I seem to be getting a much better accuracy from numpy.mean then from a
>  loop of a += val; a /= num_vals. I've seen some discussion on mailing
>  lists about using Kahan summation for mean(), so I suspect this is the
>  reason, but I can't for the life of me find if it was finally implemented
>  or not. Anybody knows?
> 
> Also, if you know where it is in the numpy source tree, please tell - I
> searched it the whole morning.
> 
> Thanks,
> Yosef.
> 

Never mind, I had a sneaky bug somewhere else that caused this. 


From cournape at gmail.com  Tue Apr  6 10:45:35 2010
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 6 Apr 2010 23:45:35 +0900
Subject: [SciPy-User] Is kKahan summation used in numpy.mean?
In-Reply-To: <201004061222.38879.yosefmel@post.tau.ac.il>
References: <201004061222.38879.yosefmel@post.tau.ac.il>
Message-ID: <w2k5b8d13221004060745la7f00c80ra241a35d0d38e868@mail.gmail.com>

On Tue, Apr 6, 2010 at 6:22 PM, Yosef Meller <yosefmel at post.tau.ac.il> wrote:
>
> Also, if you know where it is in the numpy source tree, please tell - I
> searched it the whole morning.

Look at PyArray_Mean code - it is the obvious implementation (using
reduce + normalization).

David


From mailinglists at xgm.de  Tue Apr  6 12:21:26 2010
From: mailinglists at xgm.de (Florian Lindner)
Date: Tue, 6 Apr 2010 18:21:26 +0200
Subject: [SciPy-User] Error in constants documentation?
Message-ID: <201004061821.26121.mailinglists@xgm.de>

Hello,

http://docs.scipy.org/doc/scipy-0.7.x/reference/constants.html chapter Force says:

dyn 	one dyne in watts
lbf 	one pound force in watts
kgf 	one kilogram force in watts

doesn't the watts actually have to newtons as the unit of force?

Regards,

Florian


From warren.weckesser at enthought.com  Tue Apr  6 12:37:58 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Tue, 06 Apr 2010 11:37:58 -0500
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <201004061821.26121.mailinglists@xgm.de>
References: <201004061821.26121.mailinglists@xgm.de>
Message-ID: <4BBB6366.3000507@enthought.com>

Florian Lindner wrote:
> Hello,
>
> http://docs.scipy.org/doc/scipy-0.7.x/reference/constants.html chapter Force says:
>
> dyn 	one dyne in watts
> lbf 	one pound force in watts
> kgf 	one kilogram force in watts
>
> doesn't the watts actually have to newtons as the unit of force?
>
>   

There is a ticket with the same question here:
    http://projects.scipy.org/scipy/ticket/1001


Warren

> Regards,
>
> Florian
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From warren.weckesser at enthought.com  Tue Apr  6 12:58:15 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Tue, 06 Apr 2010 11:58:15 -0500
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <4BBB6366.3000507@enthought.com>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com>
Message-ID: <4BBB6827.6080307@enthought.com>

Warren Weckesser wrote:
> Florian Lindner wrote:
>   
>> Hello,
>>
>> http://docs.scipy.org/doc/scipy-0.7.x/reference/constants.html chapter Force says:
>>
>> dyn 	one dyne in watts
>> lbf 	one pound force in watts
>> kgf 	one kilogram force in watts
>>
>> doesn't the watts actually have to newtons as the unit of force?
>>
>>   
>>     
>
> There is a ticket with the same question here:
>     http://projects.scipy.org/scipy/ticket/1001
>
>
> Warren
>   

Fixed in trunk by r6310.  Thanks for pointing out the mistake.


Warren



From disappearedng at gmail.com  Tue Apr  6 16:04:28 2010
From: disappearedng at gmail.com (disappearedng)
Date: Wed, 7 Apr 2010 04:04:28 +0800
Subject: [SciPy-User] Hcluster Negative Distance Error?
Message-ID: <g2m15dd54ec1004061304rf07374c7p58945e4df8c76fc5@mail.gmail.com>

Dear Everyone,

I have a input file which are all floating point numbers to 4 decimal place.
i.e. 13359    0.0000    0.0000    0.0001    0.0001    0.0002    0.0003
0.0007    ... (the first is the id).
My class uses the loadVectorsFromFile method which multiplies it by 10000
and then int() these numbers. On top of that, I also loop through each
vector to ensure that there are no
negative values inside. However, when I perform _hclustering, I am
continually seeing the error, "Linkage `Z` contains negative values".

I seriously think this is a bug because: 1) I checked my values, 2) the
values are no where small enough or big enough to approach the limits of the
floating point numbers and 3) the
formula that I used to derive the values in the file uses absolute value (my
input is DEFINITELY right).

Can someone enligten me as to why I am seeing this weird error? What is
going on that is causing this negative distance error?

=====

    def loadVectorsFromFile(self, limit, loc, assertAllPositive=True,
inflate=True):
        """Inflate to prevent "negative" distance, we use 4 decimal points,
so *10000
        """
        vectors = {}
        self.winfo("Each vector is set to have %d limit in length" % limit)
        with open( loc ) as inf:
            for line in filter(None, inf.read().split('\n')):
                l = line.split('\t')
                if limit:
                    scores = map(float, l[1:limit+1])
                else:
                    scores = map(float, l[1:])

                if inflate:
                    vectors[ l[0]] = map( lambda x: int(x*10000),
scores)     #int might save space
                else:
                    vectors[ l[0]] = scores

        if assertAllPositive:
            #Assert that it has no negative value
            for dirID, l in vectors.iteritems():
                if reduce(operator.or_, map( lambda x: x < 0, l)):
                    self.werror( "Vector %s has negative values!" % dirID)
        return vectors

    def main( self, inputDir, outputDir, limit=0,
            inFname="data.vectors.all",
mappingFname='all.id.features.group.intermediate'):
        """
        Loads vector from a file and start clustering
        INPUT
            vectors is { featureID: tfidfVector (list), }
        """
        IDFeatureDic = loadIdFeatureGroupDicFromIntermediate(
pjoin(self.configDir, mappingFname))
        if not os.path.exists(outputDir):
            os.makedirs(outputDir)

        vectors = self.loadVectorsFromFile( limit, pjoin( inputDir,
inFname))
        for threshold in map( lambda x:float(x)/30, range(20,30)):
            clusters = self._hclustering(threshold, vectors)
            if clusters:
                outputLoc = pjoin(outputDir, "threshold.%s.result" %
str(threshold))
                with open(outputLoc, 'w') as outf:
                    for clusterNo, cluster in clusters.iteritems():
                        outf.write('%s\n' % str(clusterNo))
                        for featureID in cluster:
                            feature, group = IDFeatureDic[featureID]
                            outline = "%s\t%s\n" % (feature, group)
                            outf.write(outline.encode('utf-8'))
                        outf.write("\n")
            else:
                continue

    def _hclustering(self, threshold, vectors):
        """function which you should call to vary the threshold
        vectors:    { featureID:    [ tfidf scores, tfidf score, .. ]
        """
        clusters = defaultdict(list)
        if len(vectors) > 1:
            try:
                results = hierarchy.fclusterdata( vectors.values(),
threshold, metric='cosine')
            except ValueError, e:
                self.werror("_hclustering: %s" % str(e))
                return False

            for i, featureID in enumerate( vectors.keys()):
                clusters[results[i]].append( featureID)
            return clusters
        else:
            return False
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100407/625a2b2e/attachment.html>

From amg at iri.columbia.edu  Tue Apr  6 16:08:53 2010
From: amg at iri.columbia.edu (Arthur M. Greene)
Date: Tue, 06 Apr 2010 16:08:53 -0400
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <4BBB6366.3000507@enthought.com>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com>
Message-ID: <4BBB94D5.6020608@iri.columbia.edu>

It would seem that there is some confusion, in the 
constants.html, between force and mass...

Strictly speaking, kg is a unit of mass, Newton a unit 
of force. Weight is force, not mass: A gold brick 
floating in interstellar space is weightless but still 
massive.  Pounds and kilograms can be equated, but only 
in some specified gravitational field (like at the 
surface of the earth, where we usually weigh things). 
So mass is the more fundamental quantity, since it does 
not depend on gravity for its value. In Imperial units 
(feet, pounds) the unit of mass is the slug: 
http://en.wikipedia.org/wiki/Slug_(mass). This is 
absent from the constants page.

Regards,

AMG

Warren Weckesser wrote:
> Florian Lindner wrote:
>> Hello,
>>
>> http://docs.scipy.org/doc/scipy-0.7.x/reference/constants.html chapter Force says:
>>
>> dyn 	one dyne in watts
>> lbf 	one pound force in watts
>> kgf 	one kilogram force in watts
>>
>> doesn't the watts actually have to newtons as the unit of force?
>>
>>   
> 
> There is a ticket with the same question here:
>     http://projects.scipy.org/scipy/ticket/1001
> 
> 
> Warren
> 
>> Regards,
>>
>> Florian
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>   
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


-------------- next part --------------
A non-text attachment was scrubbed...
Name: amg.vcf
Type: text/x-vcard
Size: 174 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100406/6100d4ad/attachment.vcf>

From josef.pktd at gmail.com  Tue Apr  6 21:41:32 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 6 Apr 2010 21:41:32 -0400
Subject: [SciPy-User] gsoc 2010
Message-ID: <v2k1cd32cbb1004061841s6ac46f08y7b58a1816b45f11e@mail.gmail.com>

Just a reminder

April 9: 19:00 UTC  Student application deadline.

Josef


From yosefmel at post.tau.ac.il  Wed Apr  7 02:36:43 2010
From: yosefmel at post.tau.ac.il (Yosef Meller)
Date: Wed, 7 Apr 2010 09:36:43 +0300
Subject: [SciPy-User] Is kKahan summation used in numpy.mean?
In-Reply-To: <w2k5b8d13221004060745la7f00c80ra241a35d0d38e868@mail.gmail.com>
References: <201004061222.38879.yosefmel@post.tau.ac.il>
	<w2k5b8d13221004060745la7f00c80ra241a35d0d38e868@mail.gmail.com>
Message-ID: <201004070936.43611.yosefmel@post.tau.ac.il>

On ??? ????? 06 ????? 2010 17:45:35 David Cournapeau wrote:
> On Tue, Apr 6, 2010 at 6:22 PM, Yosef Meller <yosefmel at post.tau.ac.il> 
wrote:
> > Also, if you know where it is in the numpy source tree, please tell - I
> > searched it the whole morning.
> 
> Look at PyArray_Mean code - it is the obvious implementation (using
> reduce + normalization).

Thanks. 


From j.reid at mail.cryst.bbk.ac.uk  Wed Apr  7 09:08:53 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Wed, 07 Apr 2010 14:08:53 +0100
Subject: [SciPy-User] Scipy test failure
Message-ID: <hpi055$ttl$1@dough.gmane.org>

Hi,

My scipy tests fail although I seem to have built numpy correctly using 
a similar configuration. I'm on OpenSuse with gcc 4.2.3 and I'm using 
the svn versions of numpy and scipy (2.0.0dev8324 and 0.8.0dev6310). I'm 
using UMFpack and GotoBlas. My python was compiled with a pre-release 
version of gcc 4.1.2

python2.5 -c "import numpy; numpy.test()"

gives me 4 known failures.


python2.5 -c "import scipy; scipy.test()"

gives me a lot of errors (see below) and then a segmentation fault. I've 
checked to see if I'm using different fortran compilers. As far as I can 
tell I'm just using gfortran.

What can I do to track down the problem?

Thanks in advance,
John.



Here's the scipy test output:
/usr/local/lib/python2.5/site-packages/scipy/linsolve/__init__.py:4: 
DeprecationWarning: scipy.linsolve has moved to scipy.sparse.linalg.dsolve
   warn('scipy.linsolve has moved to scipy.sparse.linalg.dsolve', 
DeprecationWarning)
/usr/local/lib/python2.5/site-packages/scipy/splinalg/__init__.py:3: 
DeprecationWarning: scipy.splinalg has moved to scipy.sparse.linalg
   warn('scipy.splinalg has moved to scipy.sparse.linalg', 
DeprecationWarning)
............................................................................................................................................................................................................................................................................../usr/local/lib/python2.5/site-packages/scipy/interpolate/fitpack2.py:512: 
UserWarning:
The coefficients of the spline returned have been computed as the
minimal norm least-squares solution of a (numerically) rank deficient
system (deficiency=7). If deficiency is large, the results may be
inaccurate. Deficiency may strongly depend on the value of eps.
   warnings.warn(message)
...../usr/local/lib/python2.5/site-packages/scipy/interpolate/fitpack2.py:453: 
UserWarning:
The required storage space exceeds the available storage space: nxest
or nyest too small, or s too small.
The weighted least-squares spline corresponds to the current set of
knots.
   warnings.warn(message)
...........................................K..K............................................................Warning: 
divide by zero encountered in log
Warning: invalid value encountered in multiply
Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
.Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
.Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
.........Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
Warning: divide by zero encountered in log
Warning: invalid value encountered in multiply
.................................../usr/local/lib/python2.5/site-packages/scipy/io/matlab/mio.py:190: 
FutureWarning: Using oned_as default value ('column') This will change 
to 'row' in future versions
   oned_as=oned_as)
........................./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:435: 
FutureWarning: Using oned_as default value ('column') This will change 
to 'row' in future versions
   mfw = MatFile5Writer(StringIO())
....../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:450: 
FutureWarning: Using oned_as default value ('column') This will change 
to 'row' in future versions
   wtr = MatFile5Writer(sio)
../usr/local/lib/python2.5/site-packages/scipy/io/matlab/mio.py:99: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   return MatFile5Reader(byte_stream, **kwargs)
.............................../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:633: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(stream)
./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:637: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(stream, squeeze_me=True)
../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:641: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(stream, byte_order=boc.native_code)
./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:645: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(stream, byte_order=boc.swapped_code)
../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:652: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(stream)
./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:654: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(stream, chars_as_strings=False)
../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:664: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(file(estring_fname))
./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:674: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(stream)
./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:679: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(stream)
../usr/local/lib/python2.5/site-packages/scipy/io/matlab/mio4.py:338: 
DeprecationWarning: Matlab 4 files only support <=2 dimensions; future 
versions of scipy will raise an error when trying to write >2D arrays to 
matlab 4 format files
   DeprecationWarning,
./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:699: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(file(func_eg))
./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:703: 
FutureWarning: Using oned_as default value ('column') This will change 
to 'row' in future versions
   wtr = MatFile5Writer(stream)
./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:709: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(file(double_eg), mat_dtype=False)
./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:712: 
FutureWarning: Using struct_as_record default value (False) This will 
change to True in future versions
   rdr = MatFile5Reader(file(double_eg), mat_dtype=True)
..............................................................................................................................Warning: 
1000000 bytes requested, 20 bytes read.
./usr/local/lib/python2.5/site-packages/numpy/lib/utils.py:139: 
DeprecationWarning: `write_array` is deprecated!

This function is replaced by numpy.savetxt which allows the same 
functionality
through a different syntax.

   warnings.warn(depdoc, DeprecationWarning)
/usr/local/lib/python2.5/site-packages/numpy/lib/utils.py:139: 
DeprecationWarning: `read_array` is deprecated!

The functionality of read_array is in numpy.loadtxt which allows the same
functionality using different syntax.

   warnings.warn(depdoc, DeprecationWarning)
...........................................Exception 
exceptions.AttributeError: "'netcdf_file' object has no attribute 
'mode'" in <bound method netcdf_file.close of 
<scipy.io.netcdf.netcdf_file object at 0xb33ae4cc>> ignored
............/usr/local/lib/python2.5/site-packages/numpy/lib/utils.py:139: 
DeprecationWarning: `npfile` is deprecated!

You can achieve the same effect as using npfile using numpy.save and
numpy.load.

You can use memory-mapped arrays and data-types to map out a
file format for direct manipulation in NumPy.

   warnings.warn(depdoc, DeprecationWarning)
...../usr/local/lib/python2.5/warnings.py:41: ComplexWarning: Casting 
complex values to real discards the imaginary part
   lineno = caller.f_lineno
..../usr/local/lib/python2.5/site-packages/scipy/io/wavfile.py:20: 
WavFileWarning: Unfamiliar format bytes
   warnings.warn("Unfamiliar format bytes", WavFileWarning)
/usr/local/lib/python2.5/site-packages/scipy/io/wavfile.py:92: 
WavFileWarning: chunk not understood
   warnings.warn("chunk not understood", WavFileWarning)
................................Running unit tests for scipy
NumPy version 2.0.0.dev8324
NumPy is installed in /usr/local/lib/python2.5/site-packages/numpy
SciPy version 0.8.0.dev6310
SciPy is installed in /usr/local/lib/python2.5/site-packages/scipy
Python version 2.5.2 (r252:60911, May  9 2008, 15:14:37) [GCC 4.1.2 
20070115 (prerelease) (SUSE Linux)]
nose version 0.11.3
[ 91  61  11  79 119] [ 91  61  11  79 119]
[[ 93   3]
  [120  56]
  [125  96]
  [ 97  80]
  [  6  44]] [[ 93   3]
  [120  56]
  [125  96]
  [ 97  80]
  [  6  44]]
[[ 71  96  99  65 103]
  [ 93 115  29  91  10]
  [ 46  42  28  31  48]
  [ 80   0  20  49 109]
  [ 70  24 103 106  26]] [[ 71  96  99  65 103]
  [ 93 115  29  91  10]
  [ 46  42  28  31  48]
  [ 80   0  20  49 109]
  [ 70  24 103 106  26]]
[16 56 93 14 56] [16 56 93 14 56]
[[113 115]
  [101  29]
  [103  23]
  [ 58  23]
  [ 91 124]] [[113 115]
  [101  29]
  [103  23]
  [ 58  23]
  [ 91 124]]
[[ 45  23  14  49  28]
  [ 67 123 104  97  94]
  [120  32  76   0  90]
  [113  54  42  19 100]
  [  0   0  12   4  39]] [[ 45  23  14  49  28]
  [ 67 123 104  97  94]
  [120  32  76   0  90]
  [113  54  42  19 100]
  [  0   0  12   4  39]]
[  2   8  59  69 113] [  2   8  59  69 113]
[[ 67  72]
  [126  26]
  [ 33  67]
  [ 70  80]
  [ 64 124]] [[ 67  72]
  [126  26]
  [ 33  67]
  [ 70  80]
  [ 64 124]]
[[ 41  76  45  74  19]
  [ 29  75  79  22 120]
  [ 52  69  67  24  25]
  [109  45  91 114  28]
  [117  52  11  33  76]] [[ 41  76  45  74  19]
  [ 29  75  79  22 120]
  [ 52  69  67  24  25]
  [109  45  91 114  28]
  [117  52  11  33  76]]
[25 29 28 97 82] [25 29 28 97 82]
[[ 83 117]
  [ 69 104]
  [ 78  26]
  [ 16  15]
  [ 14  82]] [[ 83 117]
  [ 69 104]
  [ 78  26]
  [ 16  15]
  [ 14  82]]
[[ 82 126  74  74   2]
  [ 68 126  92 125  51]
  [ 98 101  32 119 103]
  [126 103  12   1  56]
  [ 30 106  69 101 117]] [[ 82 126  74  74   2]
  [ 68 126  92 125  51]
  [ 98 101  32 119 103]
  [126 103  12   1  56]
  [ 30 106  69 101 117]]
[116  54  10  78  55] [116  54  10  78  55]
[[116  82]
  [120  82]
  [  0  64]
  [ 93 123]
  [  0  77]] [[116  82]
  [120  82]
  [  0  64]
  [ 93 123]
  [  0  77]]
[[ 93  70  56   5  71]
  [ 59  43  82  83  37]
  [ 29  43  70  58  32]
  [ 38 114   4 122  36]
  [ 74 125  16  90 109]] [[ 93  70  56   5  71]
  [ 59  43  82  83  37]
  [ 29  43  70  58  32]
  [ 38 114   4 122  36]
  [ 74 125  16  90 109]]
[77  1 40 61 89] [77  1 40 61 89]
[[ 86   8]
  [ 40  25]
  [ 68  30]
  [ 44 118]
  [104  76]] [[ 86   8]
  [ 40  25]
  [ 68  30]
  [ 44 118]
  [104  76]]
[[ 49  29  39  90 101]
  [ 79  48  18  96 116]
  [ 60  96 112 110  12]
  [ 21  59  83 105 111]
  [ 57  93 106  57  93]] [[ 49  29  39  90 101]
  [ 79  48  18  96 116]
  [ 60  96 112 110  12]
  [ 21  59  83 105 111]
  [ 57  93 106  57  93]]
[ 51 115  71 117  30] [ 51 115  71 117  30]
[[ 86 122]
  [  9   8]
  [ 43  51]
  [ 28  54]
  [ 15  52]] [[ 86 122]
  [  9   8]
  [ 43  51]
  [ 28  54]
  [ 15  52]]
[[  3  97  88  67  15]
  [ 85 125   4   2  64]
  [ 41  86 107  51  58]
  [ 12  21 101  59  66]
  [  5  36 101  69  47]] [[  3  97  88  67  15]
  [ 85 125   4   2  64]
  [ 41  86 107  51  58]
  [ 12  21 101  59  66]
  [  5  36 101  69  47]]
[ 48 104  34  86 124] [ 48 104  34  86 124]
[[ 34  29]
  [115 123]
  [ 42  48]
  [ 31  71]
  [100  48]] [[ 34  29]
  [115 123]
  [ 42  48]
  [ 31  71]
  [100  48]]
[[ 30  64  76  98   8]
  [ 76  92  98  45 109]
  [ 29  75  65  17 105]
  [ 89 114  69  95 101]
  [102  71 116  16  76]] [[ 30  64  76  98   8]
  [ 76  92  98  45 109]
  [ 29  75  65  17 105]
  [ 89 114  69  95 101]
  [102  71 116  16  76]]
[ 46 101  45 102  37] [ 46 101  45 102  37]
[[ 98  68]
  [ 16  18]
  [ 69  52]
  [114  35]
  [124  20]] [[ 98  68]
  [ 16  18]
  [ 69  52]
  [114  35]
  [124  20]]
[[ 58  35  74  91  14]
  [ 23 121  33  40  65]
  [123 124  82   9  16]
  [107  25 122  84 102]
  [110  54  91  94  48]] [[ 58  35  74  91  14]
  [ 23 121  33  40  65]
  [123 124  82   9  16]
  [107  25 122  84 102]
  [110  54  91  94  48]]
[109  71  15  76   4] [109  71  15  76   4]
[[ 39  11]
  [  7  75]
  [ 70  37]
  [113   2]
  [ 17  80]] [[ 39  11]
  [  7  75]
  [ 70  37]
  [113   2]
  [ 17  80]]
[[113  13  88  46  69]
  [  2 125 113  49  51]
  [ 83  29  30  95 103]
  [126 110  12  89  22]
  [ 92  49  77  21  82]] [[113  13  88  46 
.................................69]
  [  2 125 113  49  51]
  [ 83  29  30  95 103]
  [126 110  12  89  22]
  [ 92  49  77  21  82]]
[ 30 123 117   9  43] [ 30 123 117   9  43]
[[ 43  73]
  [113  27]
  [ 93 118]
  [104   8]
  [ 13  78]] [[ 43  73]
  [113  27]
  [ 93 118]
  [104   8]
  [ 13  78]]
[[ 76  45  48  24  25]
  [ 12  98  81   5  76]
  [ 79  47  64   6 107]
  [120  56 121  43   0]
  [ 29   1 106   0  86]] [[ 76  45  48  24  25]
  [ 12  98  81   5  76]
  [ 79  47  64   6 107]
  [120  56 121  43   0]
  [ 29   1 106   0  86]]
[ 55  83 106 125  98] [ 55  83 106 125  98]
[[ 54  80]
  [109  65]
  [105 126]
  [ 41  59]
  [110  40]] [[ 54  80]
  [109  65]
  [105 126]
  [ 41  59]
  [110  40]]
[[ 19   7 114  32   0]
  [ 58  18  21  66  67]
  [ 37  63  66  86   9]
  [ 94  65  87  51  65]
  [  6  44  29  54  27]] [[ 19   7 114  32   0]
  [ 58  18  21  66  67]
  [ 37  63  66  86   9]
  [ 94  65  87  51  65]
  [  6  44  29  54  27]]
[64 84 78 28 94] [64 84 78 28 94]
[[ 71  20]
  [102  87]
  [ 68  68]
  [111 123]
  [ 72  78]] [[ 71  20]
  [102  87]
  [ 68  68]
  [111 123]
  [ 72  78]]
[[ 80  45 106 126  59]
  [110  61  18 100  19]
  [120 107  68  16  68]
  [110  41  97   5 115]
  [ 88 101  97 101 114]] [[ 80  45 106 126  59]
  [110  61  18 100  19]
  [120 107  68  16  68]
  [110  41  97   5 115]
  [ 88 101  97 101 114]]
[40 32  3 11 41] [40 32  3 11 41]
[[ 13  24]
  [ 18 105]
  [ 53  63]
  [ 58 127]
  [103  70]] [[ 13  24]
  [ 18 105]
  [ 53  63]
  [ 58 127]
  [103  70]]
[[  2  40   4  53  41]
  [ 42  83   0  96  16]
  [ 88 117 104  88  92]
  [ 62  94  50  47  91]
  [ 54  85 112  82  10]] [[  2  40   4  53  41]
  [ 42  83   0  96  16]
  [ 88 117 104  88  92]
  [ 62  94  50  47  91]
  [ 54  85 112  82  10]]
[ 11  16 108 113 121] [ 11  16 108 113 121]
[[  2 113]
  [ 57 120]
  [ 19  52]
  [ 28  64]
  [ 14   1]] [[  2 113]
  [ 57 120]
  [ 19  52]
  [ 28  64]
  [ 14   1]]
[[ 16  93  24  63  73]
  [ 53  70 101  82 101]
  [ 43  68   6  56  27]
  [109  56   3  43  12]
  [105  80  92  18  46]] [[ 16  93  24  63  73]
  [ 53  70 101  82 101]
  [ 43  68   6  56  27]
  [109  56   3  43  12]
  [105  80  92  18  46]]
[ 23 105  19  92  15] [ 23 105  19  92  15]
[[40 91]
  [11 97]
  [ 5 27]
  [42 25]
  [97 85]] [[40 91]
  [11 97]
  [ 5 27]
  [42 25]
  [97 85]]
[[ 54  53 122  37  28]
  [126  24 117  76  64]
  [ 17  82 127  31 120]
  [115  17 112  10 126]
  [ 95  98  83  67   7]] [[ 54  53 122  37  28]
  [126  24 117  76  64]
  [ 17  82 127  31 120]
  [115  17 112  10 126]
  [ 95  98  83  67   7]]
[ 11 124  95 125  36] [ 11 124  95 125  36]
[[122  55]
  [115  64]
  [ 22  45]
  [ 11  83]
  [ 38  39]] [[122  55]
  [115  64]
  [ 22  45]
  [ 11  83]
  [ 38  39]]
[[ 85 116  54 113 116]
  [113 122  57  66 127]
  [  0  96  85  95  82]
  [ 99 119   6  78 104]
  [ 59  71  61 102 117]] [[ 85 116  54 113 116]
  [113 122  57  66 127]
  [  0  96  85  95  82]
  [ 99 119   6  78 104]
  [ 59  71  61 102 117]]
[101 115 123  35  53] [101 115 123  35  53]
[[112 121]
  [ 70  36]
  [ 79   0]
  [  3  30]
  [  3  36]] [[112 121]
  [ 70  36]
  [ 79   0]
  [  3  30]
  [  3  36]]
[[100  27  78  19  85]
  [ 77  63  81  56  28]
  [ 31  16  17  53 125]
  [ 82  10  43 108  20]
  [  7 104   5  76  87]] [[100  27  78  19  85]
  [ 77  63  81  56  28]
  [ 31  16  17  53 125]
  [ 82  10  43 108  20]
  [  7 104   5  76  87]]
[123  72  63  52  18] [123  72  63  52  18]
[[ 43  37]
  [ 97  39]
  [123  58]
  [ 62  26]
  [125  25]] [[ 43  37]
  [ 97  39]
  [123  58]
  [ 62  26]
  [125  25]]
[[ 48 119  16 116  69]
  [119  89  13 113  79]
  [ 39 110  43 100  73]
  [  7  36  80  45 100]
  [125  73  39  27 110]] [[ 48 119  16 116  69]
  [119  89  13 113  79]
  [ 39 110  43 100  73]
  [  7  36  80  45 100]
  [125  73  39  27 110]]
[ 26  37  82 102   4] [ 26  37  82 102   4]
[[116 121]
  [ 48  48]
  [ 96  76]
  [  0 103]
  [102 125]] [[116 121]
  [ 48  48]
  [ 96  76]
  [  0 103]
  [102 125]]
[[103  90  43 106  88]
  [ 84  49   2  19 107]
  [ 12  75 102  25  21]
  [ 92  27 113 107  77]
  [ 89 127   3  83  83]] [[103  90  43 106  88]
  [ 84  49   2  19 107]
  [ 12  75 102  25  21]
  [ 92  27 113 107  77]
  [ 89 127   3  83  83]]
[114   8  87   8   8] [114   8  87   8   8]
[[ 38 124]
  [ 26   7]
  [ 63 110]
  [ 20  75]
  [ 50  37]] [[ 38 124]
  [ 26   7]
  [ 63 110]
  [ 20  75]
  [ 50  37]]
[[ 
......................E.....FF................................................................................................................................SSSSSS......SSSSSS...FFFSSSS........................................................EE.......F.......Warning: 
invalid value encountered in divide
.....Warning: invalid value encountered in divide
Warning: invalid value encountered in divide
............................................/usr/local/lib/python2.5/site-packages/scipy/linalg/decomp.py:1180: 
DeprecationWarning: qr econ argument will be removed after scipy 0.7. 
The economy transform will then be available through the mode='economic' 
argument.
   "the mode='economic' argument.", DeprecationWarning)
........................



From bsouthey at gmail.com  Wed Apr  7 09:58:28 2010
From: bsouthey at gmail.com (Bruce Southey)
Date: Wed, 07 Apr 2010 08:58:28 -0500
Subject: [SciPy-User] Scipy test failure
In-Reply-To: <hpi055$ttl$1@dough.gmane.org>
References: <hpi055$ttl$1@dough.gmane.org>
Message-ID: <4BBC8F84.7010703@gmail.com>

On 04/07/2010 08:08 AM, John Reid wrote:
> Hi,
>
> My scipy tests fail although I seem to have built numpy correctly using
> a similar configuration. I'm on OpenSuse with gcc 4.2.3 and I'm using
> the svn versions of numpy and scipy (2.0.0dev8324 and 0.8.0dev6310). I'm
> using UMFpack and GotoBlas. My python was compiled with a pre-release
> version of gcc 4.1.2
>
> python2.5 -c "import numpy; numpy.test()"
>
> gives me 4 known failures.
>
>
> python2.5 -c "import scipy; scipy.test()"
>
> gives me a lot of errors (see below) and then a segmentation fault. I've
> checked to see if I'm using different fortran compilers. As far as I can
> tell I'm just using gfortran.
>
> What can I do to track down the problem?
>
> Thanks in advance,
> John.
>
>
>
> Here's the scipy test output:
> /usr/local/lib/python2.5/site-packages/scipy/linsolve/__init__.py:4:
> DeprecationWarning: scipy.linsolve has moved to scipy.sparse.linalg.dsolve
>     warn('scipy.linsolve has moved to scipy.sparse.linalg.dsolve',
> DeprecationWarning)
> /usr/local/lib/python2.5/site-packages/scipy/splinalg/__init__.py:3:
> DeprecationWarning: scipy.splinalg has moved to scipy.sparse.linalg
>     warn('scipy.splinalg has moved to scipy.sparse.linalg',
> DeprecationWarning)
> ............................................................................................................................................................................................................................................................................../usr/local/lib/python2.5/site-packages/scipy/interpolate/fitpack2.py:512:
> UserWarning:
> The coefficients of the spline returned have been computed as the
> minimal norm least-squares solution of a (numerically) rank deficient
> system (deficiency=7). If deficiency is large, the results may be
> inaccurate. Deficiency may strongly depend on the value of eps.
>     warnings.warn(message)
> ...../usr/local/lib/python2.5/site-packages/scipy/interpolate/fitpack2.py:453:
> UserWarning:
> The required storage space exceeds the available storage space: nxest
> or nyest too small, or s too small.
> The weighted least-squares spline corresponds to the current set of
> knots.
>     warnings.warn(message)
> ...........................................K..K............................................................Warning:
> divide by zero encountered in log
> Warning: invalid value encountered in multiply
> Warning: divide by zero encountered in log
> Warning: invalid value encountered in multiply
> Warning: divide by zero encountered in log
> Warning: invalid value encountered in multiply
> .Warning: divide by zero encountered in log
> Warning: invalid value encountered in multiply
> Warning: divide by zero encountered in log
> Warning: invalid value encountered in multiply
> .Warning: divide by zero encountered in log
> Warning: invalid value encountered in multiply
> Warning: divide by zero encountered in log
> Warning: invalid value encountered in multiply
> .........Warning: divide by zero encountered in log
> Warning: invalid value encountered in multiply
> Warning: divide by zero encountered in log
> Warning: invalid value encountered in multiply
> .................................../usr/local/lib/python2.5/site-packages/scipy/io/matlab/mio.py:190:
> FutureWarning: Using oned_as default value ('column') This will change
> to 'row' in future versions
>     oned_as=oned_as)
> ........................./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:435:
> FutureWarning: Using oned_as default value ('column') This will change
> to 'row' in future versions
>     mfw = MatFile5Writer(StringIO())
> ....../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:450:
> FutureWarning: Using oned_as default value ('column') This will change
> to 'row' in future versions
>     wtr = MatFile5Writer(sio)
> ../usr/local/lib/python2.5/site-packages/scipy/io/matlab/mio.py:99:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     return MatFile5Reader(byte_stream, **kwargs)
> .............................../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:633:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(stream)
> ./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:637:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(stream, squeeze_me=True)
> ../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:641:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(stream, byte_order=boc.native_code)
> ./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:645:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(stream, byte_order=boc.swapped_code)
> ../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:652:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(stream)
> ./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:654:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(stream, chars_as_strings=False)
> ../usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:664:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(file(estring_fname))
> ./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:674:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(stream)
> ./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:679:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(stream)
> ../usr/local/lib/python2.5/site-packages/scipy/io/matlab/mio4.py:338:
> DeprecationWarning: Matlab 4 files only support<=2 dimensions; future
> versions of scipy will raise an error when trying to write>2D arrays to
> matlab 4 format files
>     DeprecationWarning,
> ./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:699:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(file(func_eg))
> ./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:703:
> FutureWarning: Using oned_as default value ('column') This will change
> to 'row' in future versions
>     wtr = MatFile5Writer(stream)
> ./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:709:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(file(double_eg), mat_dtype=False)
> ./usr/local/lib/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py:712:
> FutureWarning: Using struct_as_record default value (False) This will
> change to True in future versions
>     rdr = MatFile5Reader(file(double_eg), mat_dtype=True)
> ..............................................................................................................................Warning:
> 1000000 bytes requested, 20 bytes read.
> ./usr/local/lib/python2.5/site-packages/numpy/lib/utils.py:139:
> DeprecationWarning: `write_array` is deprecated!
>
> This function is replaced by numpy.savetxt which allows the same
> functionality
> through a different syntax.
>
>     warnings.warn(depdoc, DeprecationWarning)
> /usr/local/lib/python2.5/site-packages/numpy/lib/utils.py:139:
> DeprecationWarning: `read_array` is deprecated!
>
> The functionality of read_array is in numpy.loadtxt which allows the same
> functionality using different syntax.
>
>     warnings.warn(depdoc, DeprecationWarning)
> ...........................................Exception
> exceptions.AttributeError: "'netcdf_file' object has no attribute
> 'mode'" in<bound method netcdf_file.close of
> <scipy.io.netcdf.netcdf_file object at 0xb33ae4cc>>  ignored
> ............/usr/local/lib/python2.5/site-packages/numpy/lib/utils.py:139:
> DeprecationWarning: `npfile` is deprecated!
>
> You can achieve the same effect as using npfile using numpy.save and
> numpy.load.
>
> You can use memory-mapped arrays and data-types to map out a
> file format for direct manipulation in NumPy.
>
>     warnings.warn(depdoc, DeprecationWarning)
> ...../usr/local/lib/python2.5/warnings.py:41: ComplexWarning: Casting
> complex values to real discards the imaginary part
>     lineno = caller.f_lineno
> ..../usr/local/lib/python2.5/site-packages/scipy/io/wavfile.py:20:
> WavFileWarning: Unfamiliar format bytes
>     warnings.warn("Unfamiliar format bytes", WavFileWarning)
> /usr/local/lib/python2.5/site-packages/scipy/io/wavfile.py:92:
> WavFileWarning: chunk not understood
>     warnings.warn("chunk not understood", WavFileWarning)
> ................................Running unit tests for scipy
> NumPy version 2.0.0.dev8324
> NumPy is installed in /usr/local/lib/python2.5/site-packages/numpy
> SciPy version 0.8.0.dev6310
> SciPy is installed in /usr/local/lib/python2.5/site-packages/scipy
> Python version 2.5.2 (r252:60911, May  9 2008, 15:14:37) [GCC 4.1.2
> 20070115 (prerelease) (SUSE Linux)]
> nose version 0.11.3
> [ 91  61  11  79 119] [ 91  61  11  79 119]
> [[ 93   3]
>    [120  56]
>    [125  96]
>    [ 97  80]
>    [  6  44]] [[ 93   3]
>    [120  56]
>    [125  96]
>    [ 97  80]
>    [  6  44]]
> [[ 71  96  99  65 103]
>    [ 93 115  29  91  10]
>    [ 46  42  28  31  48]
>    [ 80   0  20  49 109]
>    [ 70  24 103 106  26]] [[ 71  96  99  65 103]
>    [ 93 115  29  91  10]
>    [ 46  42  28  31  48]
>    [ 80   0  20  49 109]
>    [ 70  24 103 106  26]]
> [16 56 93 14 56] [16 56 93 14 56]
> [[113 115]
>    [101  29]
>    [103  23]
>    [ 58  23]
>    [ 91 124]] [[113 115]
>    [101  29]
>    [103  23]
>    [ 58  23]
>    [ 91 124]]
> [[ 45  23  14  49  28]
>    [ 67 123 104  97  94]
>    [120  32  76   0  90]
>    [113  54  42  19 100]
>    [  0   0  12   4  39]] [[ 45  23  14  49  28]
>    [ 67 123 104  97  94]
>    [120  32  76   0  90]
>    [113  54  42  19 100]
>    [  0   0  12   4  39]]
> [  2   8  59  69 113] [  2   8  59  69 113]
> [[ 67  72]
>    [126  26]
>    [ 33  67]
>    [ 70  80]
>    [ 64 124]] [[ 67  72]
>    [126  26]
>    [ 33  67]
>    [ 70  80]
>    [ 64 124]]
> [[ 41  76  45  74  19]
>    [ 29  75  79  22 120]
>    [ 52  69  67  24  25]
>    [109  45  91 114  28]
>    [117  52  11  33  76]] [[ 41  76  45  74  19]
>    [ 29  75  79  22 120]
>    [ 52  69  67  24  25]
>    [109  45  91 114  28]
>    [117  52  11  33  76]]
> [25 29 28 97 82] [25 29 28 97 82]
> [[ 83 117]
>    [ 69 104]
>    [ 78  26]
>    [ 16  15]
>    [ 14  82]] [[ 83 117]
>    [ 69 104]
>    [ 78  26]
>    [ 16  15]
>    [ 14  82]]
> [[ 82 126  74  74   2]
>    [ 68 126  92 125  51]
>    [ 98 101  32 119 103]
>    [126 103  12   1  56]
>    [ 30 106  69 101 117]] [[ 82 126  74  74   2]
>    [ 68 126  92 125  51]
>    [ 98 101  32 119 103]
>    [126 103  12   1  56]
>    [ 30 106  69 101 117]]
> [116  54  10  78  55] [116  54  10  78  55]
> [[116  82]
>    [120  82]
>    [  0  64]
>    [ 93 123]
>    [  0  77]] [[116  82]
>    [120  82]
>    [  0  64]
>    [ 93 123]
>    [  0  77]]
> [[ 93  70  56   5  71]
>    [ 59  43  82  83  37]
>    [ 29  43  70  58  32]
>    [ 38 114   4 122  36]
>    [ 74 125  16  90 109]] [[ 93  70  56   5  71]
>    [ 59  43  82  83  37]
>    [ 29  43  70  58  32]
>    [ 38 114   4 122  36]
>    [ 74 125  16  90 109]]
> [77  1 40 61 89] [77  1 40 61 89]
> [[ 86   8]
>    [ 40  25]
>    [ 68  30]
>    [ 44 118]
>    [104  76]] [[ 86   8]
>    [ 40  25]
>    [ 68  30]
>    [ 44 118]
>    [104  76]]
> [[ 49  29  39  90 101]
>    [ 79  48  18  96 116]
>    [ 60  96 112 110  12]
>    [ 21  59  83 105 111]
>    [ 57  93 106  57  93]] [[ 49  29  39  90 101]
>    [ 79  48  18  96 116]
>    [ 60  96 112 110  12]
>    [ 21  59  83 105 111]
>    [ 57  93 106  57  93]]
> [ 51 115  71 117  30] [ 51 115  71 117  30]
> [[ 86 122]
>    [  9   8]
>    [ 43  51]
>    [ 28  54]
>    [ 15  52]] [[ 86 122]
>    [  9   8]
>    [ 43  51]
>    [ 28  54]
>    [ 15  52]]
> [[  3  97  88  67  15]
>    [ 85 125   4   2  64]
>    [ 41  86 107  51  58]
>    [ 12  21 101  59  66]
>    [  5  36 101  69  47]] [[  3  97  88  67  15]
>    [ 85 125   4   2  64]
>    [ 41  86 107  51  58]
>    [ 12  21 101  59  66]
>    [  5  36 101  69  47]]
> [ 48 104  34  86 124] [ 48 104  34  86 124]
> [[ 34  29]
>    [115 123]
>    [ 42  48]
>    [ 31  71]
>    [100  48]] [[ 34  29]
>    [115 123]
>    [ 42  48]
>    [ 31  71]
>    [100  48]]
> [[ 30  64  76  98   8]
>    [ 76  92  98  45 109]
>    [ 29  75  65  17 105]
>    [ 89 114  69  95 101]
>    [102  71 116  16  76]] [[ 30  64  76  98   8]
>    [ 76  92  98  45 109]
>    [ 29  75  65  17 105]
>    [ 89 114  69  95 101]
>    [102  71 116  16  76]]
> [ 46 101  45 102  37] [ 46 101  45 102  37]
> [[ 98  68]
>    [ 16  18]
>    [ 69  52]
>    [114  35]
>    [124  20]] [[ 98  68]
>    [ 16  18]
>    [ 69  52]
>    [114  35]
>    [124  20]]
> [[ 58  35  74  91  14]
>    [ 23 121  33  40  65]
>    [123 124  82   9  16]
>    [107  25 122  84 102]
>    [110  54  91  94  48]] [[ 58  35  74  91  14]
>    [ 23 121  33  40  65]
>    [123 124  82   9  16]
>    [107  25 122  84 102]
>    [110  54  91  94  48]]
> [109  71  15  76   4] [109  71  15  76   4]
> [[ 39  11]
>    [  7  75]
>    [ 70  37]
>    [113   2]
>    [ 17  80]] [[ 39  11]
>    [  7  75]
>    [ 70  37]
>    [113   2]
>    [ 17  80]]
> [[113  13  88  46  69]
>    [  2 125 113  49  51]
>    [ 83  29  30  95 103]
>    [126 110  12  89  22]
>    [ 92  49  77  21  82]] [[113  13  88  46
> .................................69]
>    [  2 125 113  49  51]
>    [ 83  29  30  95 103]
>    [126 110  12  89  22]
>    [ 92  49  77  21  82]]
> [ 30 123 117   9  43] [ 30 123 117   9  43]
> [[ 43  73]
>    [113  27]
>    [ 93 118]
>    [104   8]
>    [ 13  78]] [[ 43  73]
>    [113  27]
>    [ 93 118]
>    [104   8]
>    [ 13  78]]
> [[ 76  45  48  24  25]
>    [ 12  98  81   5  76]
>    [ 79  47  64   6 107]
>    [120  56 121  43   0]
>    [ 29   1 106   0  86]] [[ 76  45  48  24  25]
>    [ 12  98  81   5  76]
>    [ 79  47  64   6 107]
>    [120  56 121  43   0]
>    [ 29   1 106   0  86]]
> [ 55  83 106 125  98] [ 55  83 106 125  98]
> [[ 54  80]
>    [109  65]
>    [105 126]
>    [ 41  59]
>    [110  40]] [[ 54  80]
>    [109  65]
>    [105 126]
>    [ 41  59]
>    [110  40]]
> [[ 19   7 114  32   0]
>    [ 58  18  21  66  67]
>    [ 37  63  66  86   9]
>    [ 94  65  87  51  65]
>    [  6  44  29  54  27]] [[ 19   7 114  32   0]
>    [ 58  18  21  66  67]
>    [ 37  63  66  86   9]
>    [ 94  65  87  51  65]
>    [  6  44  29  54  27]]
> [64 84 78 28 94] [64 84 78 28 94]
> [[ 71  20]
>    [102  87]
>    [ 68  68]
>    [111 123]
>    [ 72  78]] [[ 71  20]
>    [102  87]
>    [ 68  68]
>    [111 123]
>    [ 72  78]]
> [[ 80  45 106 126  59]
>    [110  61  18 100  19]
>    [120 107  68  16  68]
>    [110  41  97   5 115]
>    [ 88 101  97 101 114]] [[ 80  45 106 126  59]
>    [110  61  18 100  19]
>    [120 107  68  16  68]
>    [110  41  97   5 115]
>    [ 88 101  97 101 114]]
> [40 32  3 11 41] [40 32  3 11 41]
> [[ 13  24]
>    [ 18 105]
>    [ 53  63]
>    [ 58 127]
>    [103  70]] [[ 13  24]
>    [ 18 105]
>    [ 53  63]
>    [ 58 127]
>    [103  70]]
> [[  2  40   4  53  41]
>    [ 42  83   0  96  16]
>    [ 88 117 104  88  92]
>    [ 62  94  50  47  91]
>    [ 54  85 112  82  10]] [[  2  40   4  53  41]
>    [ 42  83   0  96  16]
>    [ 88 117 104  88  92]
>    [ 62  94  50  47  91]
>    [ 54  85 112  82  10]]
> [ 11  16 108 113 121] [ 11  16 108 113 121]
> [[  2 113]
>    [ 57 120]
>    [ 19  52]
>    [ 28  64]
>    [ 14   1]] [[  2 113]
>    [ 57 120]
>    [ 19  52]
>    [ 28  64]
>    [ 14   1]]
> [[ 16  93  24  63  73]
>    [ 53  70 101  82 101]
>    [ 43  68   6  56  27]
>    [109  56   3  43  12]
>    [105  80  92  18  46]] [[ 16  93  24  63  73]
>    [ 53  70 101  82 101]
>    [ 43  68   6  56  27]
>    [109  56   3  43  12]
>    [105  80  92  18  46]]
> [ 23 105  19  92  15] [ 23 105  19  92  15]
> [[40 91]
>    [11 97]
>    [ 5 27]
>    [42 25]
>    [97 85]] [[40 91]
>    [11 97]
>    [ 5 27]
>    [42 25]
>    [97 85]]
> [[ 54  53 122  37  28]
>    [126  24 117  76  64]
>    [ 17  82 127  31 120]
>    [115  17 112  10 126]
>    [ 95  98  83  67   7]] [[ 54  53 122  37  28]
>    [126  24 117  76  64]
>    [ 17  82 127  31 120]
>    [115  17 112  10 126]
>    [ 95  98  83  67   7]]
> [ 11 124  95 125  36] [ 11 124  95 125  36]
> [[122  55]
>    [115  64]
>    [ 22  45]
>    [ 11  83]
>    [ 38  39]] [[122  55]
>    [115  64]
>    [ 22  45]
>    [ 11  83]
>    [ 38  39]]
> [[ 85 116  54 113 116]
>    [113 122  57  66 127]
>    [  0  96  85  95  82]
>    [ 99 119   6  78 104]
>    [ 59  71  61 102 117]] [[ 85 116  54 113 116]
>    [113 122  57  66 127]
>    [  0  96  85  95  82]
>    [ 99 119   6  78 104]
>    [ 59  71  61 102 117]]
> [101 115 123  35  53] [101 115 123  35  53]
> [[112 121]
>    [ 70  36]
>    [ 79   0]
>    [  3  30]
>    [  3  36]] [[112 121]
>    [ 70  36]
>    [ 79   0]
>    [  3  30]
>    [  3  36]]
> [[100  27  78  19  85]
>    [ 77  63  81  56  28]
>    [ 31  16  17  53 125]
>    [ 82  10  43 108  20]
>    [  7 104   5  76  87]] [[100  27  78  19  85]
>    [ 77  63  81  56  28]
>    [ 31  16  17  53 125]
>    [ 82  10  43 108  20]
>    [  7 104   5  76  87]]
> [123  72  63  52  18] [123  72  63  52  18]
> [[ 43  37]
>    [ 97  39]
>    [123  58]
>    [ 62  26]
>    [125  25]] [[ 43  37]
>    [ 97  39]
>    [123  58]
>    [ 62  26]
>    [125  25]]
> [[ 48 119  16 116  69]
>    [119  89  13 113  79]
>    [ 39 110  43 100  73]
>    [  7  36  80  45 100]
>    [125  73  39  27 110]] [[ 48 119  16 116  69]
>    [119  89  13 113  79]
>    [ 39 110  43 100  73]
>    [  7  36  80  45 100]
>    [125  73  39  27 110]]
> [ 26  37  82 102   4] [ 26  37  82 102   4]
> [[116 121]
>    [ 48  48]
>    [ 96  76]
>    [  0 103]
>    [102 125]] [[116 121]
>    [ 48  48]
>    [ 96  76]
>    [  0 103]
>    [102 125]]
> [[103  90  43 106  88]
>    [ 84  49   2  19 107]
>    [ 12  75 102  25  21]
>    [ 92  27 113 107  77]
>    [ 89 127   3  83  83]] [[103  90  43 106  88]
>    [ 84  49   2  19 107]
>    [ 12  75 102  25  21]
>    [ 92  27 113 107  77]
>    [ 89 127   3  83  83]]
> [114   8  87   8   8] [114   8  87   8   8]
> [[ 38 124]
>    [ 26   7]
>    [ 63 110]
>    [ 20  75]
>    [ 50  37]] [[ 38 124]
>    [ 26   7]
>    [ 63 110]
>    [ 20  75]
>    [ 50  37]]
> [[
> ......................E.....FF................................................................................................................................SSSSSS......SSSSSS...FFFSSSS........................................................EE.......F.......Warning:
> invalid value encountered in divide
> .....Warning: invalid value encountered in divide
> Warning: invalid value encountered in divide
> ............................................/usr/local/lib/python2.5/site-packages/scipy/linalg/decomp.py:1180:
> DeprecationWarning: qr econ argument will be removed after scipy 0.7.
> The economy transform will then be available through the mode='economic'
> argument.
>     "the mode='economic' argument.", DeprecationWarning)
> ........................
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>    
First run tests in verbose mode that will at least identify the test. I 
would guess that it is the eigenvalue tests especially given the usage 
of GotoBlas.


$ python2.5 -c "import scipy; scipy.test(verbose=10)"

 From one of Robert's instructions:

$ gdb python2.5
...
(gdb) run
Starting program ...
... # Possibly another (gdb) prompt:
(gdb) continue  #<- Type this.
Python 2.6.2 ...


>>> >>>  import numpy  #<- Type this and do whatever else is necessary to reproduce the crash.
>>>        
... (gdb) bt # <- Type this. .... # <- Copy-paste these results here.

Bruce


From j.reid at mail.cryst.bbk.ac.uk  Wed Apr  7 10:13:40 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Wed, 07 Apr 2010 15:13:40 +0100
Subject: [SciPy-User] Scipy test failure
In-Reply-To: <4BBC8F84.7010703@gmail.com>
References: <hpi055$ttl$1@dough.gmane.org> <4BBC8F84.7010703@gmail.com>
Message-ID: <hpi3ul$dsj$1@dough.gmane.org>



Bruce Southey wrote:
> First run tests in verbose mode that will at least identify the test. I 
> would guess that it is the eigenvalue tests especially given the usage 
> of GotoBlas.
> 
> 
> $ python2.5 -c "import scipy; scipy.test(verbose=10)"

These are the lines in the output that contain "ERROR":

test_dot (test_blas.TestFBLAS1Simple) ... ERROR
test_complex_dotc (test_blas.TestFBLAS1Simple) ... ERROR
test_complex_dotu (test_blas.TestFBLAS1Simple) ... ERROR




These are the lines that contain FAIL:

test_interpolate.TestInterp1D.test_nd_zero_spline ... KNOWNFAIL: 
zero-order splines fail for the last point
Check the actual implementation of zero-order spline interpolation. ... 
KNOWNFAIL: zero-order splines fail for the last point
test_gemm (test_blas.TestFBLAS3Simple) ... FAIL
test_gemm2 (test_blas.TestFBLAS3Simple) ... FAIL
test_ssygv_1 (test_gesv.TestSygv) ... FAIL
test_ssygv_2 (test_gesv.TestSygv) ... FAIL
test_ssygv_3 (test_gesv.TestSygv) ... FAIL
test_gemm (test_blas.TestFBLAS3Simple) ... FAIL


The last few lines of the output are:
test_simple_underdet (test_decomp.TestSVDVals) ... ok
test_simple_underdet_complex (test_decomp.TestSVDVals) ... ok
test_simple (test_decomp.TestSchur) ... ok
test_decomp.test_eigh('ordinary', 6, 'f', True, True, True, None) ... ok
test_decomp.test_eigh('general ', 6, 'f', True, True, True, None) ...


which I guess confirms your suspicion it is to do with eignevalues.

> 
>  From one of Robert's instructions:
> 
> $ gdb python2.5
> ...
> (gdb) run
> Starting program ...
> ... # Possibly another (gdb) prompt:
> (gdb) continue  #<- Type this.
> Python 2.6.2 ...
> 
> 
>>>>>>>  import numpy  #<- Type this and do whatever else is necessary to reproduce the crash.
>>>>        
> ... (gdb) bt # <- Type this. .... # <- Copy-paste these results here.

I'll try and see if I can do this



From j.reid at mail.cryst.bbk.ac.uk  Wed Apr  7 10:47:18 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Wed, 07 Apr 2010 15:47:18 +0100
Subject: [SciPy-User] Scipy test failure
In-Reply-To: <4BBC8F84.7010703@gmail.com>
References: <hpi055$ttl$1@dough.gmane.org> <4BBC8F84.7010703@gmail.com>
Message-ID: <hpi5tm$lur$1@dough.gmane.org>



Bruce Southey wrote:
> First run tests in verbose mode that will at least identify the test. I 
> would guess that it is the eigenvalue tests especially given the usage 
> of GotoBlas.

Are there known issues between scipy and GotoBlas? I've just tried using 
GotoBlas2 but I'm getting a similar set of errors.




From j.reid at mail.cryst.bbk.ac.uk  Wed Apr  7 10:59:10 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Wed, 07 Apr 2010 15:59:10 +0100
Subject: [SciPy-User] Scipy test failure
In-Reply-To: <4BBC8F84.7010703@gmail.com>
References: <hpi055$ttl$1@dough.gmane.org> <4BBC8F84.7010703@gmail.com>
Message-ID: <hpi6ju$oqg$1@dough.gmane.org>



Bruce Southey wrote:
>  From one of Robert's instructions:
> 
> $ gdb python2.5
> ...
> (gdb) run
> Starting program ...
> ... # Possibly another (gdb) prompt:
> (gdb) continue  #<- Type this.
> Python 2.6.2 ...
> 
> 
>>>>>>>  import numpy  #<- Type this and do whatever else is necessary to reproduce the crash.
>>>>        
> ... (gdb) bt # <- Type this. .... # <- Copy-paste these results here.
> 

Here's the stack trace. Since moving to GotoBlas2, the segmentation 
violation seems to be in test_decomp.test_lapack_misaligned(<function 
cholesky at 0xb4a022cc>, ...

Any ideas?

test_decomp.test_lapack_misaligned(<function svd at 0xb4a02224>, 
(array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.], ... ok
test_decomp.test_lapack_misaligned(<function svd at 0xb4a02224>, 
(array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9], ... ok
test_decomp.test_lapack_misaligned(<function svd at 0xb4a02224>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function svdvals at 0xb4a0225c>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function svdvals at 0xb4a0225c>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function cholesky at 0xb4a022cc>, 
(array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.], ...
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1209674064 (LWP 20248)]
0xb44167f0 in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
(gdb) bt
#0  0xb44167f0 in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
#1  0xb4887cdc in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
#2  0x00000003 in ?? ()
#3  0x09070f7c in ?? ()
#4  0xb40ec6e3 in ddot_ () from /usr/local/lib/libgoto2_opteronp-r1.13.so
#5  0xb19297f2 in dpotf2_ () from 
/usr/local/lib/python2.5/site-packages/scipy/linalg/flapack.so
#6  0xb192a173 in dpotrf_ () from 
/usr/local/lib/python2.5/site-packages/scipy/linalg/flapack.so
#7  0xb1881338 in f2py_rout_flapack_dpotrf (capi_self=0xb49f24d0, 
capi_args=0xaf2699cc, capi_keywds=0xaf2803e4, f2py_func=0xb1929d70 
<dpotrf_>)
     at 
build/src.linux-i686-2.5/build/src.linux-i686-2.5/scipy/linalg/flapackmodule.c:13235
#8  0xb1899951 in fortran_call (fp=0x2, arg=0xaf2699cc, kw=0xaf2803e4) 
at build/src.linux-i686-2.5/fortranobject.c:346
#9  0x0805a427 in PyObject_Call (func=0x2, arg=0xaf2699cc, 
kw=0xaf2803e4) at Objects/abstract.c:1861
#10 0x080c114a in PyEval_EvalFrameEx (f=0x903879c, throwflag=0) at 
Python/ceval.c:3784
#11 0x080c7265 in PyEval_EvalCodeEx (co=0xb49fc068, globals=0xb49f7f0c, 
locals=0x0, args=0xaf269978, argcount=1, kws=0x9029868, kwcount=1, 
defs=0xb4a005f8, defcount=2,
     closure=0x0) at Python/ceval.c:2836
#12 0x0810e456 in function_call (func=0xb4a022cc, arg=0xaf26996c, 
kw=0xaf28013c) at Objects/funcobject.c:517
#13 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf26996c, 
kw=0xaf28013c) at Objects/abstract.c:1861
#14 0x080c4168 in PyEval_EvalFrameEx (f=0x8ffda8c, throwflag=0) at 
Python/ceval.c:3853
#15 0x080c7265 in PyEval_EvalCodeEx (co=0xaf2999b0, globals=0xafe209bc, 
locals=0x0, args=0xafde1768, argcount=3, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
#16 0x0810e381 in function_call (func=0xafd5acdc, arg=0xafde175c, 
kw=0x0) at Objects/funcobject.c:517
#17 0x0805a427 in PyObject_Call (func=0x2, arg=0xafde175c, kw=0x0) at 
Objects/abstract.c:1861
#18 0x080c4168 in PyEval_EvalFrameEx (f=0x8d736dc, throwflag=0) at 
Python/ceval.c:3853
#19 0x080c5d9c in PyEval_EvalFrameEx (f=0x8fc187c, throwflag=0) at 
Python/ceval.c:3659
#20 0x080c7265 in PyEval_EvalCodeEx (co=0xb7af8c80, globals=0xb7bbe934, 
locals=0x0, args=0xafd467d8, argcount=2, kws=0x9017618, kwcount=0, 
defs=0xb7b03e18, defcount=1,
     closure=0x0) at Python/ceval.c:2836
#21 0x0810e456 in function_call (func=0xb7afefb4, arg=0xafd467cc, 
kw=0xaf2659bc) at Objects/funcobject.c:517
#22 0x0805a427 in PyObject_Call (func=0x2, arg=0xafd467cc, 
kw=0xaf2659bc) at Objects/abstract.c:1861
#23 0x080c4168 in PyEval_EvalFrameEx (f=0x8fc1714, throwflag=0) at 
Python/ceval.c:3853
#24 0x080c7265 in PyEval_EvalCodeEx (co=0xb7af8cc8, globals=0xb7bbe934, 
locals=0x0, args=0xafd43b98, argcount=2, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
#25 0x0810e381 in function_call (func=0xb7b0602c, arg=0xafd43b8c, 
kw=0x0) at Objects/funcobject.c:517
#26 0x0805a427 in PyObject_Call (func=0x2, arg=0xafd43b8c, kw=0x0) at 
Objects/abstract.c:1861
#27 0x08060557 in instancemethod_call (func=0xb7b0602c, arg=0xafd43b8c, 
kw=0x0) at Objects/classobject.c:2519
#28 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf26992c, kw=0x0) at 
Objects/abstract.c:1861
#29 0x0809a290 in slot_tp_call (self=0xaf2694ec, args=0xaf26992c, 
kwds=0x0) at Objects/typeobject.c:4714
#30 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf26992c, kw=0x0) at 
Objects/abstract.c:1861
#31 0x080c114a in PyEval_EvalFrameEx (f=0x8fc15ac, throwflag=0) at 
Python/ceval.c:3784
#32 0x080c5d9c in PyEval_EvalFrameEx (f=0x8fc142c, throwflag=0) at 
Python/ceval.c:3659
#33 0x080c7265 in PyEval_EvalCodeEx (co=0xb737cec0, globals=0xb7594934, 
locals=0x0, args=0xafd6bf78, argcount=2, kws=0x8f37308, kwcount=0, 
defs=0x0, defcount=0, closure=0x0)
     at Python/ceval.c:2836
#34 0x0810e456 in function_call (func=0xb751dbfc, arg=0xafd6bf6c, 
kw=0xaf26579c) at Objects/funcobject.c:517
#35 0x0805a427 in PyObject_Call (func=0x2, arg=0xafd6bf6c, 
kw=0xaf26579c) at Objects/abstract.c:1861
#36 0x080c4168 in PyEval_EvalFrameEx (f=0x8fc12c4, throwflag=0) at 
Python/ceval.c:3853
#37 0x080c7265 in PyEval_EvalCodeEx (co=0xb737cb18, globals=0xb7594934, 
locals=0x0, args=0xafddfb78, argcount=2, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
---Type <return> to continue, or q <return> to quit---
#38 0x0810e381 in function_call (func=0xb751da04, arg=0xafddfb6c, 
kw=0x0) at Objects/funcobject.c:517
#39 0x0805a427 in PyObject_Call (func=0x2, arg=0xafddfb6c, kw=0x0) at 
Objects/abstract.c:1861
#40 0x08060557 in instancemethod_call (func=0xb751da04, arg=0xafddfb6c, 
kw=0x0) at Objects/classobject.c:2519
#41 0x0805a427 in PyObject_Call (func=0x2, arg=0xafe3caec, kw=0x0) at 
Objects/abstract.c:1861
#42 0x0809a290 in slot_tp_call (self=0xaf1dfcac, args=0xafe3caec, 
kwds=0x0) at Objects/typeobject.c:4714
#43 0x0805a427 in PyObject_Call (func=0x2, arg=0xafe3caec, kw=0x0) at 
Objects/abstract.c:1861
#44 0x080c114a in PyEval_EvalFrameEx (f=0x903802c, throwflag=0) at 
Python/ceval.c:3784
#45 0x080c7265 in PyEval_EvalCodeEx (co=0xb75242f0, globals=0xb759468c, 
locals=0x0, args=0xafd46758, argcount=2, kws=0x904a9f0, kwcount=0, 
defs=0x0, defcount=0, closure=0x0)
     at Python/ceval.c:2836
#46 0x0810e456 in function_call (func=0xb752e72c, arg=0xafd4674c, 
kw=0xaf26546c) at Objects/funcobject.c:517
#47 0x0805a427 in PyObject_Call (func=0x2, arg=0xafd4674c, 
kw=0xaf26546c) at Objects/abstract.c:1861
#48 0x080c4168 in PyEval_EvalFrameEx (f=0x90421e4, throwflag=0) at 
Python/ceval.c:3853
#49 0x080c7265 in PyEval_EvalCodeEx (co=0xb75241d0, globals=0xb759468c, 
locals=0x0, args=0xafd3c8f8, argcount=2, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
#50 0x0810e381 in function_call (func=0xb752e684, arg=0xafd3c8ec, 
kw=0x0) at Objects/funcobject.c:517
#51 0x0805a427 in PyObject_Call (func=0x2, arg=0xafd3c8ec, kw=0x0) at 
Objects/abstract.c:1861
#52 0x08060557 in instancemethod_call (func=0xb752e684, arg=0xafd3c8ec, 
kw=0x0) at Objects/classobject.c:2519
#53 0x0805a427 in PyObject_Call (func=0x2, arg=0xafe2d32c, kw=0x0) at 
Objects/abstract.c:1861
#54 0x0809a290 in slot_tp_call (self=0xafe2dc4c, args=0xafe2d32c, 
kwds=0x0) at Objects/typeobject.c:4714
#55 0x0805a427 in PyObject_Call (func=0x2, arg=0xafe2d32c, kw=0x0) at 
Objects/abstract.c:1861
#56 0x080c114a in PyEval_EvalFrameEx (f=0x8fea97c, throwflag=0) at 
Python/ceval.c:3784
#57 0x080c7265 in PyEval_EvalCodeEx (co=0xb75242f0, globals=0xb759468c, 
locals=0x0, args=0xafddf538, argcount=2, kws=0x8fb9020, kwcount=0, 
defs=0x0, defcount=0, closure=0x0)
     at Python/ceval.c:2836
#58 0x0810e456 in function_call (func=0xb752e72c, arg=0xafddf52c, 
kw=0xafe1ad74) at Objects/funcobject.c:517
#59 0x0805a427 in PyObject_Call (func=0x2, arg=0xafddf52c, 
kw=0xafe1ad74) at Objects/abstract.c:1861
#60 0x080c4168 in PyEval_EvalFrameEx (f=0x8fccd84, throwflag=0) at 
Python/ceval.c:3853
#61 0x080c7265 in PyEval_EvalCodeEx (co=0xb75241d0, globals=0xb759468c, 
locals=0x0, args=0xafd462d8, argcount=2, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
#62 0x0810e381 in function_call (func=0xb752e684, arg=0xafd462cc, 
kw=0x0) at Objects/funcobject.c:517
#63 0x0805a427 in PyObject_Call (func=0x2, arg=0xafd462cc, kw=0x0) at 
Objects/abstract.c:1861
#64 0x08060557 in instancemethod_call (func=0xb752e684, arg=0xafd462cc, 
kw=0x0) at Objects/classobject.c:2519
#65 0x0805a427 in PyObject_Call (func=0x2, arg=0xafe2d1cc, kw=0x0) at 
Objects/abstract.c:1861
#66 0x0809a290 in slot_tp_call (self=0xafe2dccc, args=0xafe2d1cc, 
kwds=0x0) at Objects/typeobject.c:4714
#67 0x0805a427 in PyObject_Call (func=0x2, arg=0xafe2d1cc, kw=0x0) at 
Objects/abstract.c:1861
#68 0x080c114a in PyEval_EvalFrameEx (f=0x8f8e88c, throwflag=0) at 
Python/ceval.c:3784
#69 0x080c7265 in PyEval_EvalCodeEx (co=0xb75242f0, globals=0xb759468c, 
locals=0x0, args=0xafeaba78, argcount=2, kws=0x8ef0d48, kwcount=0, 
defs=0x0, defcount=0, closure=0x0)
     at Python/ceval.c:2836
#70 0x0810e456 in function_call (func=0xb752e72c, arg=0xafeaba6c, 
kw=0xafe36c64) at Objects/funcobject.c:517
#71 0x0805a427 in PyObject_Call (func=0x2, arg=0xafeaba6c, 
kw=0xafe36c64) at Objects/abstract.c:1861
#72 0x080c4168 in PyEval_EvalFrameEx (f=0x8f4793c, throwflag=0) at 
Python/ceval.c:3853
#73 0x080c7265 in PyEval_EvalCodeEx (co=0xb75241d0, globals=0xb759468c, 
locals=0x0, args=0xafeab438, argcount=2, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
#74 0x0810e381 in function_call (func=0xb752e684, arg=0xafeab42c, 
kw=0x0) at Objects/funcobject.c:517
#75 0x0805a427 in PyObject_Call (func=0x2, arg=0xafeab42c, kw=0x0) at 
Objects/abstract.c:1861
#76 0x08060557 in instancemethod_call (func=0xb752e684, arg=0xafeab42c, 
kw=0x0) at Objects/classobject.c:2519
---Type <return> to continue, or q <return> to quit---
#77 0x0805a427 in PyObject_Call (func=0x2, arg=0xafeb210c, kw=0x0) at 
Objects/abstract.c:1861
#78 0x0809a290 in slot_tp_call (self=0xb36ab34c, args=0xafeb210c, 
kwds=0x0) at Objects/typeobject.c:4714
#79 0x0805a427 in PyObject_Call (func=0x2, arg=0xafeb210c, kw=0x0) at 
Objects/abstract.c:1861
#80 0x080c114a in PyEval_EvalFrameEx (f=0x8fc6524, throwflag=0) at 
Python/ceval.c:3784
#81 0x080c7265 in PyEval_EvalCodeEx (co=0xb75242f0, globals=0xb759468c, 
locals=0x0, args=0xafe9dad8, argcount=2, kws=0x8c89b38, kwcount=0, 
defs=0x0, defcount=0, closure=0x0)
     at Python/ceval.c:2836
#82 0x0810e456 in function_call (func=0xb752e72c, arg=0xafe9dacc, 
kw=0xafe36acc) at Objects/funcobject.c:517
#83 0x0805a427 in PyObject_Call (func=0x2, arg=0xafe9dacc, 
kw=0xafe36acc) at Objects/abstract.c:1861
#84 0x080c4168 in PyEval_EvalFrameEx (f=0x8fc70cc, throwflag=0) at 
Python/ceval.c:3853
#85 0x080c7265 in PyEval_EvalCodeEx (co=0xb75241d0, globals=0xb759468c, 
locals=0x0, args=0xafe80e58, argcount=2, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
#86 0x0810e381 in function_call (func=0xb752e684, arg=0xafe80e4c, 
kw=0x0) at Objects/funcobject.c:517
#87 0x0805a427 in PyObject_Call (func=0x2, arg=0xafe80e4c, kw=0x0) at 
Objects/abstract.c:1861
#88 0x08060557 in instancemethod_call (func=0xb752e684, arg=0xafe80e4c, 
kw=0x0) at Objects/classobject.c:2519
#89 0x0805a427 in PyObject_Call (func=0x2, arg=0xafffb36c, kw=0x0) at 
Objects/abstract.c:1861
#90 0x0809a290 in slot_tp_call (self=0xb4a078cc, args=0xafffb36c, 
kwds=0x0) at Objects/typeobject.c:4714
#91 0x0805a427 in PyObject_Call (func=0x2, arg=0xafffb36c, kw=0x0) at 
Objects/abstract.c:1861
#92 0x080c114a in PyEval_EvalFrameEx (f=0x8d4107c, throwflag=0) at 
Python/ceval.c:3784
#93 0x080c7265 in PyEval_EvalCodeEx (co=0xb75242f0, globals=0xb759468c, 
locals=0x0, args=0xafeea838, argcount=2, kws=0x8cbaf00, kwcount=0, 
defs=0x0, defcount=0, closure=0x0)
     at Python/ceval.c:2836
#94 0x0810e456 in function_call (func=0xb752e72c, arg=0xafeea82c, 
kw=0xafe36a44) at Objects/funcobject.c:517
#95 0x0805a427 in PyObject_Call (func=0x2, arg=0xafeea82c, 
kw=0xafe36a44) at Objects/abstract.c:1861
#96 0x080c4168 in PyEval_EvalFrameEx (f=0x8d40f14, throwflag=0) at 
Python/ceval.c:3853
#97 0x080c7265 in PyEval_EvalCodeEx (co=0xb75241d0, globals=0xb759468c, 
locals=0x0, args=0xb00a83f8, argcount=2, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
#98 0x0810e381 in function_call (func=0xb752e684, arg=0xb00a83ec, 
kw=0x0) at Objects/funcobject.c:517
#99 0x0805a427 in PyObject_Call (func=0x2, arg=0xb00a83ec, kw=0x0) at 
Objects/abstract.c:1861
#100 0x08060557 in instancemethod_call (func=0xb752e684, arg=0xb00a83ec, 
kw=0x0) at Objects/classobject.c:2519
#101 0x0805a427 in PyObject_Call (func=0x2, arg=0xb004224c, kw=0x0) at 
Objects/abstract.c:1861
#102 0x0809a290 in slot_tp_call (self=0xb743d9cc, args=0xb004224c, 
kwds=0x0) at Objects/typeobject.c:4714
#103 0x0805a427 in PyObject_Call (func=0x2, arg=0xb004224c, kw=0x0) at 
Objects/abstract.c:1861
#104 0x080c114a in PyEval_EvalFrameEx (f=0x8d3ff14, throwflag=0) at 
Python/ceval.c:3784
#105 0x080c7265 in PyEval_EvalCodeEx (co=0xb75242f0, globals=0xb759468c, 
locals=0x0, args=0xb0066e38, argcount=2, kws=0x8bab7f0, kwcount=0, 
defs=0x0, defcount=0, closure=0x0)
     at Python/ceval.c:2836
#106 0x0810e456 in function_call (func=0xb752e72c, arg=0xb0066e2c, 
kw=0xafe368ac) at Objects/funcobject.c:517
#107 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0066e2c, 
kw=0xafe368ac) at Objects/abstract.c:1861
#108 0x080c4168 in PyEval_EvalFrameEx (f=0x8c7b174, throwflag=0) at 
Python/ceval.c:3853
#109 0x080c7265 in PyEval_EvalCodeEx (co=0xb75241d0, globals=0xb759468c, 
locals=0x0, args=0xb743b9b8, argcount=2, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
#110 0x0810e381 in function_call (func=0xb752e684, arg=0xb743b9ac, 
kw=0x0) at Objects/funcobject.c:517
#111 0x0805a427 in PyObject_Call (func=0x2, arg=0xb743b9ac, kw=0x0) at 
Objects/abstract.c:1861
#112 0x08060557 in instancemethod_call (func=0xb752e684, arg=0xb743b9ac, 
kw=0x0) at Objects/classobject.c:2519
#113 0x0805a427 in PyObject_Call (func=0x2, arg=0xb003cd2c, kw=0x0) at 
Objects/abstract.c:1861
#114 0x0809a290 in slot_tp_call (self=0xb743bfcc, args=0xb003cd2c, 
kwds=0x0) at Objects/typeobject.c:4714
#115 0x0805a427 in PyObject_Call (func=0x2, arg=0xb003cd2c, kw=0x0) at 
Objects/abstract.c:1861
---Type <return> to continue, or q <return> to quit---
#116 0x080c114a in PyEval_EvalFrameEx (f=0x8d3e6fc, throwflag=0) at 
Python/ceval.c:3784
#117 0x080c5d9c in PyEval_EvalFrameEx (f=0x85507ec, throwflag=0) at 
Python/ceval.c:3659
#118 0x080c5d9c in PyEval_EvalFrameEx (f=0x852cfbc, throwflag=0) at 
Python/ceval.c:3659
#119 0x080c7265 in PyEval_EvalCodeEx (co=0xb7b01188, globals=0xb7bbe934, 
locals=0x0, args=0xb75061f8, argcount=1, kws=0x850f698, kwcount=5, 
defs=0xb7b051b8, defcount=5,
     closure=0x0) at Python/ceval.c:2836
#120 0x0810e456 in function_call (func=0xb7b06b54, arg=0xb75061ec, 
kw=0xb7436a44) at Objects/funcobject.c:517
#121 0x0805a427 in PyObject_Call (func=0x2, arg=0xb75061ec, 
kw=0xb7436a44) at Objects/abstract.c:1861
#122 0x08060557 in instancemethod_call (func=0xb7b06b54, arg=0xb75061ec, 
kw=0xb7436a44) at Objects/classobject.c:2519
#123 0x0805a427 in PyObject_Call (func=0x2, arg=0xb75061ec, 
kw=0xb7436a44) at Objects/abstract.c:1861
#124 0x080c4168 in PyEval_EvalFrameEx (f=0x84c5db4, throwflag=0) at 
Python/ceval.c:3853
#125 0x080c7265 in PyEval_EvalCodeEx (co=0xb76ac6e0, globals=0xb76b368c, 
locals=0x0, args=0xb7db1ab8, argcount=1, kws=0x8524070, kwcount=3, 
defs=0xb752a350, defcount=11,
     closure=0x0) at Python/ceval.c:2836
#126 0x0810e456 in function_call (func=0xb7531374, arg=0xb7db1aac, 
kw=0xb743679c) at Objects/funcobject.c:517
#127 0x0805a427 in PyObject_Call (func=0x2, arg=0xb7db1aac, 
kw=0xb743679c) at Objects/abstract.c:1861
#128 0x08060557 in instancemethod_call (func=0xb7531374, arg=0xb7db1aac, 
kw=0xb743679c) at Objects/classobject.c:2519
#129 0x0805a427 in PyObject_Call (func=0x2, arg=0xb7e1d02c, 
kw=0xb743679c) at Objects/abstract.c:1861
#130 0x08099f1b in slot_tp_init (self=0xb742c4ec, args=0xb7e1d02c, 
kwds=0xb743679c) at Objects/typeobject.c:4943
#131 0x0809c9a3 in type_call (type=0x85386a4, args=0xb7e1d02c, 
kwds=0xb743679c) at Objects/typeobject.c:436
#132 0x0805a427 in PyObject_Call (func=0x2, arg=0xb7e1d02c, 
kw=0xb743679c) at Objects/abstract.c:1861
#133 0x080c114a in PyEval_EvalFrameEx (f=0x8414e34, throwflag=0) at 
Python/ceval.c:3784
#134 0x080c7265 in PyEval_EvalCodeEx (co=0xb7b13380, globals=0xb7b0868c, 
locals=0x0, args=0x8202b24, argcount=1, kws=0x8202b28, kwcount=1, 
defs=0xb7b15458, defcount=5,
     closure=0x0) at Python/ceval.c:2836
#135 0x080c55f5 in PyEval_EvalFrameEx (f=0x82029ec, throwflag=0) at 
Python/ceval.c:3669
#136 0x080c7265 in PyEval_EvalCodeEx (co=0xb7da86e0, globals=0xb7e36acc, 
locals=0xb7e36acc, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, 
defcount=0, closure=0x0)
     at Python/ceval.c:2836
#137 0x080c72d7 in PyEval_EvalCode (co=0xb7da86e0, globals=0xb7e36acc, 
locals=0xb7e36acc) at Python/ceval.c:494
#138 0x080e66e3 in PyRun_InteractiveOneFlags (fp=0xb7f7c440, 
filename=0x811edc4 "<stdin>", flags=0xbfad3ea8) at Python/pythonrun.c:1273
#139 0x080e6916 in PyRun_InteractiveLoopFlags (fp=0xb7f7c440, 
filename=0x811edc4 "<stdin>", flags=0xbfad3ea8) at Python/pythonrun.c:723
#140 0x080e6a32 in PyRun_AnyFileExFlags (fp=0xb7f7c440, 
filename=0x811edc4 "<stdin>", closeit=0, flags=0xbfad3ea8) at 
Python/pythonrun.c:692
#141 0x08056e3a in Py_Main (argc=0, argv=0xbfad3f74) at Modules/main.c:523
#142 0x08056382 in main (argc=Cannot access memory at address 0x2
) at ./Modules/python.c:23



From mailinglists at xgm.de  Wed Apr  7 13:02:49 2010
From: mailinglists at xgm.de (Florian Lindner)
Date: Wed, 7 Apr 2010 19:02:49 +0200
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <4BBB94D5.6020608@iri.columbia.edu>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com>
	<4BBB94D5.6020608@iri.columbia.edu>
Message-ID: <201004071902.49240.mailinglists@xgm.de>

Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
> It would seem that there is some confusion, in the 
> constants.html, between force and mass...

Beside the wrong unit which is fixed now I don't see any confusion.

> Strictly speaking, kg is a unit of mass, Newton a unit 
> of force. Weight is force, not mass: A gold brick 
> floating in interstellar space is weightless but still 
> massive.  Pounds and kilograms can be equated, but only 
> in some specified gravitational field (like at the 
> surface of the earth, where we usually weigh things).

This is true for pounds-force and kilograms. Pounds-mass and kilograms could be equated in any context. Pound itself is ambigous.

> So mass is the more fundamental quantity, since it does 
> not depend on gravity for its value. In Imperial units 
> (feet, pounds) the unit of mass is the slug: 
> http://en.wikipedia.org/wiki/Slug_(mass). This is 
> absent from the constants page.

Mmmh.. never heard of it though I read quite some English language aerospace engineering literature. However I'm using SI units. I think pounds-mass is more widely used as a imperial unit of mass.

Regards,

Florian

> 
> Regards,
> 
> AMG
> 
> Warren Weckesser wrote:
> > Florian Lindner wrote:
> >> Hello,
> >>
> >> http://docs.scipy.org/doc/scipy-0.7.x/reference/constants.html chapter Force says:
> >>
> >> dyn 	one dyne in watts
> >> lbf 	one pound force in watts
> >> kgf 	one kilogram force in watts
> >>
> >> doesn't the watts actually have to newtons as the unit of force?
> >>
> >>   
> > 
> > There is a ticket with the same question here:
> >     http://projects.scipy.org/scipy/ticket/1001
> > 
> > 
> > Warren
> > 
> >> Regards,
> >>
> >> Florian
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>   
> > 
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 
> 


From pav+sp at iki.fi  Wed Apr  7 13:15:09 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Wed, 7 Apr 2010 17:15:09 +0000 (UTC)
Subject: [SciPy-User] Error in constants documentation?
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com> <4BBB94D5.6020608@iri.columbia.edu>
	<201004071902.49240.mailinglists@xgm.de>
Message-ID: <hpieit$i1b$2@dough.gmane.org>

Wed, 07 Apr 2010 19:02:49 +0200, Florian Lindner wrote:

> Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
>> It would seem that there is some confusion, in the constants.html,
>> between force and mass...
> 
> Beside the wrong unit which is fixed now I don't see any confusion.

One of the headings should, strictly speaking, say "Mass" instead of 
"Weight".

-- 
Pauli Virtanen



From charlesr.harris at gmail.com  Wed Apr  7 13:52:36 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 7 Apr 2010 11:52:36 -0600
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <201004071902.49240.mailinglists@xgm.de>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com> <4BBB94D5.6020608@iri.columbia.edu>
	<201004071902.49240.mailinglists@xgm.de>
Message-ID: <k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com>

On Wed, Apr 7, 2010 at 11:02 AM, Florian Lindner <mailinglists at xgm.de>wrote:

> Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
> > It would seem that there is some confusion, in the
> > constants.html, between force and mass...
>
> Beside the wrong unit which is fixed now I don't see any confusion.
>
> > Strictly speaking, kg is a unit of mass, Newton a unit
> > of force. Weight is force, not mass: A gold brick
> > floating in interstellar space is weightless but still
> > massive.  Pounds and kilograms can be equated, but only
> > in some specified gravitational field (like at the
> > surface of the earth, where we usually weigh things).
>
> This is true for pounds-force and kilograms. Pounds-mass and kilograms
> could be equated in any context. Pound itself is ambigous.
>
> > So mass is the more fundamental quantity, since it does
> > not depend on gravity for its value. In Imperial units
> > (feet, pounds) the unit of mass is the slug:
> > http://en.wikipedia.org/wiki/Slug_(mass)<http://en.wikipedia.org/wiki/Slug_%28mass%29>.
> This is
> > absent from the constants page.
>
> Mmmh.. never heard of it though I read quite some English language
> aerospace engineering literature. However I'm using SI units. I think
> pounds-mass is more widely used as a imperial unit of mass.
>
>
I recall slug being used in amateur rocketry books 50 years ago or so. But
SI units are definitely simpler.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100407/58f40756/attachment.html>

From d.l.goldsmith at gmail.com  Wed Apr  7 14:32:03 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Wed, 7 Apr 2010 11:32:03 -0700
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com> <4BBB94D5.6020608@iri.columbia.edu>
	<201004071902.49240.mailinglists@xgm.de>
	<k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com>
Message-ID: <i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com>

On Wed, Apr 7, 2010 at 10:52 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Wed, Apr 7, 2010 at 11:02 AM, Florian Lindner <mailinglists at xgm.de>wrote:
>
>> Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
>> > It would seem that there is some confusion, in the
>> > constants.html, between force and mass...
>>
>> Beside the wrong unit which is fixed now I don't see any confusion.
>>
>> > Strictly speaking, kg is a unit of mass, Newton a unit
>> > of force. Weight is force, not mass: A gold brick
>> > floating in interstellar space is weightless but still
>> > massive.  Pounds and kilograms can be equated, but only
>> > in some specified gravitational field (like at the
>> > surface of the earth, where we usually weigh things).
>>
>> This is true for pounds-force and kilograms. Pounds-mass and kilograms
>> could be equated in any context. Pound itself is ambigous.
>>
>> > So mass is the more fundamental quantity, since it does
>> > not depend on gravity for its value. In Imperial units
>> > (feet, pounds) the unit of mass is the slug:
>> > http://en.wikipedia.org/wiki/Slug_(mass)<http://en.wikipedia.org/wiki/Slug_%28mass%29>.
>> This is
>> > absent from the constants page.
>>
>> Mmmh.. never heard of it though I read quite some English language
>> aerospace engineering literature. However I'm using SI units. I think
>> pounds-mass is more widely used as a imperial unit of mass.
>>
>>
> I recall slug being used in amateur rocketry books 50 years ago or so. But
> SI units are definitely simpler.
>
> Chuck
>

OK, since Charles opened the door: what about taking the bold,
forward-looking step of not supporting "Imperial" units at all?  (I say
"good riddance.")

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100407/6fe2b51e/attachment.html>

From charlesr.harris at gmail.com  Wed Apr  7 14:40:53 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 7 Apr 2010 12:40:53 -0600
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com> <4BBB94D5.6020608@iri.columbia.edu>
	<201004071902.49240.mailinglists@xgm.de>
	<k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com>
	<i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com>
Message-ID: <n2ue06186141004071140ja4a67827k4db1d68fbea0de81@mail.gmail.com>

On Wed, Apr 7, 2010 at 12:32 PM, David Goldsmith <d.l.goldsmith at gmail.com>wrote:

> On Wed, Apr 7, 2010 at 10:52 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Wed, Apr 7, 2010 at 11:02 AM, Florian Lindner <mailinglists at xgm.de>wrote:
>>
>>> Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
>>> > It would seem that there is some confusion, in the
>>> > constants.html, between force and mass...
>>>
>>> Beside the wrong unit which is fixed now I don't see any confusion.
>>>
>>> > Strictly speaking, kg is a unit of mass, Newton a unit
>>> > of force. Weight is force, not mass: A gold brick
>>> > floating in interstellar space is weightless but still
>>> > massive.  Pounds and kilograms can be equated, but only
>>> > in some specified gravitational field (like at the
>>> > surface of the earth, where we usually weigh things).
>>>
>>> This is true for pounds-force and kilograms. Pounds-mass and kilograms
>>> could be equated in any context. Pound itself is ambigous.
>>>
>>> > So mass is the more fundamental quantity, since it does
>>> > not depend on gravity for its value. In Imperial units
>>> > (feet, pounds) the unit of mass is the slug:
>>> > http://en.wikipedia.org/wiki/Slug_(mass)<http://en.wikipedia.org/wiki/Slug_%28mass%29>.
>>> This is
>>> > absent from the constants page.
>>>
>>> Mmmh.. never heard of it though I read quite some English language
>>> aerospace engineering literature. However I'm using SI units. I think
>>> pounds-mass is more widely used as a imperial unit of mass.
>>>
>>>
>> I recall slug being used in amateur rocketry books 50 years ago or so. But
>> SI units are definitely simpler.
>>
>> Chuck
>>
>
> OK, since Charles opened the door: what about taking the bold,
> forward-looking step of not supporting "Imperial" units at all?  (I say
> "good riddance.")
>
>
Well, what if someone wants to know what a slug is in SI units? It's not as
if it is a big problem to support, and the more conversions the better,
IMHO. The point of having these things down in code is that one doesn't have
to go looking when the unit turns up somewhere.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100407/ed797291/attachment.html>

From matthew.brett at gmail.com  Wed Apr  7 14:45:03 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 7 Apr 2010 14:45:03 -0400
Subject: [SciPy-User] scipy.stats.invnorm - rename to invgauss?
Message-ID: <n2k1e2af89e1004071145pe067ee26j4c9c0b684a4afdc7@mail.gmail.com>

Hi,

I was caught in the same trap as another innocent, in thinking that
scipy.stats.invnorm was the inverse of the normal distribution:

http://bytes.com/topic/python/answers/478142-scipy-numpy-inverse-cumulative-normal

In my desire to improve the docstring, I started looking; after a
while I found this:

http://en.wikipedia.org/wiki/Inverse_Gaussian_distribution

- and that seems to fit the invnorm code.

A Google search for 'inverse normal distribution' does not include the
page above in the first 20 hits, but does find the inverse of the
normal distribution, and this:

http://en.wikipedia.org/wiki/Normal-inverse_Gaussian_distribution

I wonder whether it would be good to rename scipy.stats.invnorm to
scipy.stats.invgauss ?  Maybe with a deprecation warning to invnorm?

Best,

Matthew


From d.l.goldsmith at gmail.com  Wed Apr  7 14:57:29 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Wed, 7 Apr 2010 11:57:29 -0700
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <n2ue06186141004071140ja4a67827k4db1d68fbea0de81@mail.gmail.com>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com> <4BBB94D5.6020608@iri.columbia.edu>
	<201004071902.49240.mailinglists@xgm.de>
	<k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com>
	<i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com>
	<n2ue06186141004071140ja4a67827k4db1d68fbea0de81@mail.gmail.com>
Message-ID: <l2o45d1ab481004071157m3f879bbevf4d6c7a3be66d6fa@mail.gmail.com>

On Wed, Apr 7, 2010 at 11:40 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

> On Wed, Apr 7, 2010 at 12:32 PM, David Goldsmith <d.l.goldsmith at gmail.com>wrote:
>
>> On Wed, Apr 7, 2010 at 10:52 AM, Charles R Harris <
>> charlesr.harris at gmail.com> wrote:
>>
>>> On Wed, Apr 7, 2010 at 11:02 AM, Florian Lindner <mailinglists at xgm.de>wrote:
>>>
>>>> Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
>>>> > It would seem that there is some confusion, in the
>>>> > constants.html, between force and mass...
>>>>
>>>> Beside the wrong unit which is fixed now I don't see any confusion.
>>>>
>>>> > Strictly speaking, kg is a unit of mass, Newton a unit
>>>> > of force. Weight is force, not mass: A gold brick
>>>> > floating in interstellar space is weightless but still
>>>> > massive.  Pounds and kilograms can be equated, but only
>>>> > in some specified gravitational field (like at the
>>>> > surface of the earth, where we usually weigh things).
>>>>
>>>> This is true for pounds-force and kilograms. Pounds-mass and kilograms
>>>> could be equated in any context. Pound itself is ambigous.
>>>>
>>>> > So mass is the more fundamental quantity, since it does
>>>> > not depend on gravity for its value. In Imperial units
>>>> > (feet, pounds) the unit of mass is the slug:
>>>> > http://en.wikipedia.org/wiki/Slug_(mass)<http://en.wikipedia.org/wiki/Slug_%28mass%29>.
>>>> This is
>>>> > absent from the constants page.
>>>>
>>>> Mmmh.. never heard of it though I read quite some English language
>>>> aerospace engineering literature. However I'm using SI units. I think
>>>> pounds-mass is more widely used as a imperial unit of mass.
>>>>
>>>>
>>> I recall slug being used in amateur rocketry books 50 years ago or so.
>>> But SI units are definitely simpler.
>>>
>>> Chuck
>>>
>>
>> OK, since Charles opened the door: what about taking the bold,
>> forward-looking step of not supporting "Imperial" units at all?  (I say
>> "good riddance.")
>>
>>
> Well, what if someone wants to know what a slug is in SI units? It's not as
> if it is a big problem to support,
>

I think as time drags on it will become a bother to support; if someone
needs Imperial support, they can roll their own.

DG


> and the more conversions the better, IMHO. The point of having these things
> down in code is that one doesn't have to go looking when the unit turns up
> somewhere.
>
> Chuck
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Mathematician: noun, someone who disavows certainty when their uncertainty
set is non-empty, even if that set has measure zero.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100407/15c30a27/attachment.html>

From robert.kern at gmail.com  Wed Apr  7 15:03:27 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 7 Apr 2010 14:03:27 -0500
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <l2o45d1ab481004071157m3f879bbevf4d6c7a3be66d6fa@mail.gmail.com>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com> 
	<4BBB94D5.6020608@iri.columbia.edu>
	<201004071902.49240.mailinglists@xgm.de> 
	<k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com> 
	<i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com> 
	<n2ue06186141004071140ja4a67827k4db1d68fbea0de81@mail.gmail.com> 
	<l2o45d1ab481004071157m3f879bbevf4d6c7a3be66d6fa@mail.gmail.com>
Message-ID: <i2y3d375d731004071203p9ff6e4a0w34472792013eb3cf@mail.gmail.com>

On Wed, Apr 7, 2010 at 13:57, David Goldsmith <d.l.goldsmith at gmail.com> wrote:
> On Wed, Apr 7, 2010 at 11:40 AM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>> On Wed, Apr 7, 2010 at 12:32 PM, David Goldsmith <d.l.goldsmith at gmail.com>
>> wrote:
>>>
>>> On Wed, Apr 7, 2010 at 10:52 AM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>>>
>>>> On Wed, Apr 7, 2010 at 11:02 AM, Florian Lindner <mailinglists at xgm.de>
>>>> wrote:
>>>>>
>>>>> Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
>>>>> > It would seem that there is some confusion, in the
>>>>> > constants.html, between force and mass...
>>>>>
>>>>> Beside the wrong unit which is fixed now I don't see any confusion.
>>>>>
>>>>> > Strictly speaking, kg is a unit of mass, Newton a unit
>>>>> > of force. Weight is force, not mass: A gold brick
>>>>> > floating in interstellar space is weightless but still
>>>>> > massive. ?Pounds and kilograms can be equated, but only
>>>>> > in some specified gravitational field (like at the
>>>>> > surface of the earth, where we usually weigh things).
>>>>>
>>>>> This is true for pounds-force and kilograms. Pounds-mass and kilograms
>>>>> could be equated in any context. Pound itself is ambigous.
>>>>>
>>>>> > So mass is the more fundamental quantity, since it does
>>>>> > not depend on gravity for its value. In Imperial units
>>>>> > (feet, pounds) the unit of mass is the slug:
>>>>> > http://en.wikipedia.org/wiki/Slug_(mass). This is
>>>>> > absent from the constants page.
>>>>>
>>>>> Mmmh.. never heard of it though I read quite some English language
>>>>> aerospace engineering literature. However I'm using SI units. I think
>>>>> pounds-mass is more widely used as a imperial unit of mass.
>>>>>
>>>>
>>>> I recall slug being used in amateur rocketry books 50 years ago or so.
>>>> But SI units are definitely simpler.
>>>>
>>>> Chuck
>>>
>>>
>>> OK, since Charles opened the door: what about taking the bold,
>>> forward-looking step of not supporting "Imperial" units at all?? (I say
>>> "good riddance.")
>>>
>>
>> Well, what if someone wants to know what a slug is in SI units? It's not
>> as if it is a big problem to support,
>
> I think as time drags on it will become a bother to support; if someone
> needs Imperial support, they can roll their own.

They're constants. The amount of support they need is smaller than the
smallest function in our codebase.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Wed Apr  7 15:05:43 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 7 Apr 2010 15:05:43 -0400
Subject: [SciPy-User] scipy.stats.invnorm - rename to invgauss?
In-Reply-To: <n2k1e2af89e1004071145pe067ee26j4c9c0b684a4afdc7@mail.gmail.com>
References: <n2k1e2af89e1004071145pe067ee26j4c9c0b684a4afdc7@mail.gmail.com>
Message-ID: <r2v1cd32cbb1004071205w9d683b09n1d9cbc69ca2be686@mail.gmail.com>

On Wed, Apr 7, 2010 at 2:45 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> I was caught in the same trap as another innocent, in thinking that
> scipy.stats.invnorm was the inverse of the normal distribution:
>
> http://bytes.com/topic/python/answers/478142-scipy-numpy-inverse-cumulative-normal
>
> In my desire to improve the docstring, I started looking; after a
> while I found this:
>
> http://en.wikipedia.org/wiki/Inverse_Gaussian_distribution
>
> - and that seems to fit the invnorm code.
>
> A Google search for 'inverse normal distribution' does not include the
> page above in the first 20 hits, but does find the inverse of the
> normal distribution, and this:
>
> http://en.wikipedia.org/wiki/Normal-inverse_Gaussian_distribution
>
> I wonder whether it would be good to rename scipy.stats.invnorm to
> scipy.stats.invgauss ? ?Maybe with a deprecation warning to invnorm?

in general I wouldn't mind, but for naming consistency I prefer the current name

distributions created by some transformation of the normal distribution:

>>> [i for i in dir(stats) if 'norm' in i and isinstance(getattr(stats,i), stats.distributions.rv_generic)]
['foldnorm', 'halfnorm', 'invnorm', 'lognorm', 'norm', 'powerlognorm',
'powernorm', 'truncnorm']

I don't think the confusion persists very long, invnorm is a
distribution, the inverse function of the cdf of the normal
distribution is a function or in our case a method (ppf).

Josef


>
> Best,
>
> Matthew
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Wed Apr  7 15:11:02 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 7 Apr 2010 15:11:02 -0400
Subject: [SciPy-User] scipy.stats.invnorm - rename to invgauss?
In-Reply-To: <r2v1cd32cbb1004071205w9d683b09n1d9cbc69ca2be686@mail.gmail.com>
References: <n2k1e2af89e1004071145pe067ee26j4c9c0b684a4afdc7@mail.gmail.com>
	<r2v1cd32cbb1004071205w9d683b09n1d9cbc69ca2be686@mail.gmail.com>
Message-ID: <t2h1cd32cbb1004071211i6336ff55rf61b3bba437b63b0@mail.gmail.com>

On Wed, Apr 7, 2010 at 3:05 PM,  <josef.pktd at gmail.com> wrote:
> On Wed, Apr 7, 2010 at 2:45 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> I was caught in the same trap as another innocent, in thinking that
>> scipy.stats.invnorm was the inverse of the normal distribution:
>>
>> http://bytes.com/topic/python/answers/478142-scipy-numpy-inverse-cumulative-normal
>>
>> In my desire to improve the docstring, I started looking; after a
>> while I found this:
>>
>> http://en.wikipedia.org/wiki/Inverse_Gaussian_distribution
>>
>> - and that seems to fit the invnorm code.
>>
>> A Google search for 'inverse normal distribution' does not include the
>> page above in the first 20 hits, but does find the inverse of the
>> normal distribution, and this:
>>
>> http://en.wikipedia.org/wiki/Normal-inverse_Gaussian_distribution
>>
>> I wonder whether it would be good to rename scipy.stats.invnorm to
>> scipy.stats.invgauss ? ?Maybe with a deprecation warning to invnorm?
>
> in general I wouldn't mind, but for naming consistency I prefer the current name
>
> distributions created by some transformation of the normal distribution:
>
>>>> [i for i in dir(stats) if 'norm' in i and isinstance(getattr(stats,i), stats.distributions.rv_generic)]
> ['foldnorm', 'halfnorm', 'invnorm', 'lognorm', 'norm', 'powerlognorm',
> 'powernorm', 'truncnorm']
>
> I don't think the confusion persists very long, invnorm is a
> distribution, the inverse function of the cdf of the normal
> distribution is a function or in our case a method (ppf).

But from http://en.wikipedia.org/wiki/Inverse_Gaussian_distribution
it looks like it's not a inverse transformation. In that case a
different name might reduce some confusion.

I will look at what invnorm really is, (I don't know)

Josef

>
> Josef
>
>
>>
>> Best,
>>
>> Matthew
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From d.l.goldsmith at gmail.com  Wed Apr  7 15:13:04 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Wed, 7 Apr 2010 12:13:04 -0700
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <i2y3d375d731004071203p9ff6e4a0w34472792013eb3cf@mail.gmail.com>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com> <4BBB94D5.6020608@iri.columbia.edu>
	<201004071902.49240.mailinglists@xgm.de>
	<k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com>
	<i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com>
	<n2ue06186141004071140ja4a67827k4db1d68fbea0de81@mail.gmail.com>
	<l2o45d1ab481004071157m3f879bbevf4d6c7a3be66d6fa@mail.gmail.com>
	<i2y3d375d731004071203p9ff6e4a0w34472792013eb3cf@mail.gmail.com>
Message-ID: <i2g45d1ab481004071213ye70347fbobc75ec6432e50690@mail.gmail.com>

Oh wohl, I tried.

DG

On Wed, Apr 7, 2010 at 12:03 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Wed, Apr 7, 2010 at 13:57, David Goldsmith <d.l.goldsmith at gmail.com>
> wrote:
> > On Wed, Apr 7, 2010 at 11:40 AM, Charles R Harris
> > <charlesr.harris at gmail.com> wrote:
> >>
> >> On Wed, Apr 7, 2010 at 12:32 PM, David Goldsmith <
> d.l.goldsmith at gmail.com>
> >> wrote:
> >>>
> >>> On Wed, Apr 7, 2010 at 10:52 AM, Charles R Harris
> >>> <charlesr.harris at gmail.com> wrote:
> >>>>
> >>>> On Wed, Apr 7, 2010 at 11:02 AM, Florian Lindner <mailinglists at xgm.de
> >
> >>>> wrote:
> >>>>>
> >>>>> Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
> >>>>> > It would seem that there is some confusion, in the
> >>>>> > constants.html, between force and mass...
> >>>>>
> >>>>> Beside the wrong unit which is fixed now I don't see any confusion.
> >>>>>
> >>>>> > Strictly speaking, kg is a unit of mass, Newton a unit
> >>>>> > of force. Weight is force, not mass: A gold brick
> >>>>> > floating in interstellar space is weightless but still
> >>>>> > massive.  Pounds and kilograms can be equated, but only
> >>>>> > in some specified gravitational field (like at the
> >>>>> > surface of the earth, where we usually weigh things).
> >>>>>
> >>>>> This is true for pounds-force and kilograms. Pounds-mass and
> kilograms
> >>>>> could be equated in any context. Pound itself is ambigous.
> >>>>>
> >>>>> > So mass is the more fundamental quantity, since it does
> >>>>> > not depend on gravity for its value. In Imperial units
> >>>>> > (feet, pounds) the unit of mass is the slug:
> >>>>> > http://en.wikipedia.org/wiki/Slug_(mass)<http://en.wikipedia.org/wiki/Slug_%28mass%29>.
> This is
> >>>>> > absent from the constants page.
> >>>>>
> >>>>> Mmmh.. never heard of it though I read quite some English language
> >>>>> aerospace engineering literature. However I'm using SI units. I think
> >>>>> pounds-mass is more widely used as a imperial unit of mass.
> >>>>>
> >>>>
> >>>> I recall slug being used in amateur rocketry books 50 years ago or so.
> >>>> But SI units are definitely simpler.
> >>>>
> >>>> Chuck
> >>>
> >>>
> >>> OK, since Charles opened the door: what about taking the bold,
> >>> forward-looking step of not supporting "Imperial" units at all?  (I say
> >>> "good riddance.")
> >>>
> >>
> >> Well, what if someone wants to know what a slug is in SI units? It's not
> >> as if it is a big problem to support,
> >
> > I think as time drags on it will become a bother to support; if someone
> > needs Imperial support, they can roll their own.
>
> They're constants. The amount of support they need is smaller than the
> smallest function in our codebase.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Mathematician: noun, someone who disavows certainty when their uncertainty
set is non-empty, even if that set has measure zero.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100407/330ceee5/attachment.html>

From hasslerjc at comcast.net  Wed Apr  7 18:14:58 2010
From: hasslerjc at comcast.net (John Hassler)
Date: Wed, 07 Apr 2010 18:14:58 -0400
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <n2ue06186141004071140ja4a67827k4db1d68fbea0de81@mail.gmail.com>
References: <201004061821.26121.mailinglists@xgm.de>	<4BBB6366.3000507@enthought.com>
	<4BBB94D5.6020608@iri.columbia.edu>	<201004071902.49240.mailinglists@xgm.de>	<k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com>	<i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com>
	<n2ue06186141004071140ja4a67827k4db1d68fbea0de81@mail.gmail.com>
Message-ID: <4BBD03E2.9090207@comcast.net>

An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100407/4dc794ba/attachment.html>

From jsseabold at gmail.com  Thu Apr  8 00:52:10 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 8 Apr 2010 00:52:10 -0400
Subject: [SciPy-User] statsmodels gsoc application for comments
Message-ID: <v2ic048da1c1004072152m471ffdedm894a2ad0ed5c35ae@mail.gmail.com>

All,

Was hoping anyone who is interested could give some last minute
comments on my google summer of code application (deadline is Friday).
 Given the focus on Python 3 transition, it's a shot in the dark, but
I'd like to continue working on the project.

Note that the schedule is in order of importance for the developers
and that most if not all of the goals are part of my work and have
been begun in some form or another but need more focused attention.

Cheers,

Skipper

Title
------
Development of the SciPy SciKit Statsmodels

Summary
--------------
Statsmodels is a pure Python package for statistical and econometric modeling.
It began as part of the SciPy project, was spun off into NiPy [0], and then
picked up last year as my GSoC 2009 project.  I intend to make statsmodels ready
for the jump to Python 3, focus on the lingering design issues to ensure the
package is mature and generally usable for all, and to extend the types of
estimators available.


Background
------------------
Last summer I successfully complete my first Google Summer of Code with the
statsmodels package for the SciPy project under the Python Software
Foundation.
Since the end of GSoC 2009, my mentor and I have been able to release our
software as a standalone package while continuing to work on the code [1,2].
The project and its aim have garnered some attention, and a group of users,
particularly in Finance, Economics, and the Social Sciences more generally,
are coming together to work on related projects in Python.  In addition to
discussions on the scipy mailing list, we are seeing some traffic on our
pystatsmodels mailing list [3].  Pandas is a package focused on quantitative
finance that leverages statsmodels, and Wes McKinney has presented on pandas at
PyCon 2010 [4] and will lead a special topic presentation at SciPy conference
2010 [5].  Projects such as larry and tabular are working on data structures
and their analysis geared towards statistics [6,7].  Further, the scipy.stats
sprint has currently garnered the most votes for the SciPy conference [8].
Given that the project is approaching a critical mass, I would like the
opportunity to go forward towards its main aim as a statistical/econometric
library for Python users while also getting the project ready for the Python 3
transition.  Only continued efforts will see to it that Python 3 becomes a
serious option for statistical data analysis for a larger group of users.


Project Schedule
----------------
pre-GSoC: Organize code in the sandbox so that "low hanging fruit" is very
          easy to pick.  Search for appropriately licensed code that might
          be helpful to translate for the project.

Week 1 - 2 (May 24 - June 6)
           Work on design issues, particularly, the model wrapper idea [9,10]
           so that statsmodels remains a useful library for other projects that
           implement different data structures rather than just numpy arrays.
           Streamline the maximum likelihood framework including the Model
           needs (ie., analytic gradients, hessians; automatic differentiation,
           numerical differentations ideas).  Generic maximum likelihood
           framework.  Design of generic bootstrap methods and generic
           post-estimation testing framework.  Internal variable name handling
           and summary functions for output using SimpleTable in the sandbox.

Week 2 - 4 (June 7 - June 20)
           Work on time series models.  Bring together and test all helper
           functions, including solvers and algorithsm.  Finish VARMA and
           GARCH models, Hodrick-Prescott filter and Kalman filter.

Week 4 - 6 (June 21 - July 4)
           Work on systems of equations models and mainly design.
           Work on panel data models.  Finish including random-effects.
           Dynamic Panel estimators.  Error specification tests.  Implicit in
           this is a general instrumental variables framework.

Week 6 - 8 (July 5 - July 18)
           Nonparametric estimators. Nonparametric regression and univariate
           kernel density estimators with bandwidth cross-validation.
           Multivariate density estimators.

Week 8 - 10 (July 19 - August 1)
           Information theoretic measures.  Generalized maximum entropy.
           Refactor of scipy.maxentropy to be more general.

Week 11 - (August 2 - August 8)
          Catch up week for any work not finished according to deadlines above.

Week 12 - (August 9 - August 16)
          Polish and tie up any remaining loose ends in the code, making sure
          test coverage is good, call signatures and inheritance structures are
          consistent, and all TODOs and NOTES are covered.  Make sure there are
          no remaining sphinx/documentation build issues.


About Me
---------------
I am finishing my second year as a PhD student in Economics at American
University in Washington, DC.  Since last summer, I have continued to study
statistics and econometrics including special topics in microeconometrics,
time series and macroeconometrics, and information theory and entropy
econometrics.  I try to do as much of my work in Python as possible, when this
is not an option I work in R, Octave/Matlab, and other commercial statistical
software (mainly Stata, eViews, SAS, NLogit/Limdep). Working with these other
packages definitely helps inform design decisions on the statsmodels package
and reinforces my belief that Python should be the scientific programming and
scripting language of choice for practicing researchers!


[1] http://statsmodels.sourceforge.net/
[2] https://code.launchpad.net/statsmodels
[3] http://groups.google.ca/group/pystatsmodels
[4] http://us.pycon.org/2010/conference/schedule/event/50/
[5] http://conference.scipy.org/scipy2010/papers.html
[6] http://larry.sourceforge.net/
[7] http://www.parsemydata.com/tabular/
[8] http://conference.scipy.org/scipy2010/sprints.html
[9] http://groups.google.ca/group/pystatsmodels/browse_thread/thread/72267d8e784a318b/
[10] http://groups.google.ca/group/pystatsmodels/browse_thread/thread/a47a84be6a41c45e/


From j.reid at mail.cryst.bbk.ac.uk  Thu Apr  8 08:38:11 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Thu, 08 Apr 2010 13:38:11 +0100
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
Message-ID: <hpkinj$pmp$1@dough.gmane.org>

Hi,

I'm trying to build numpy and scipy from the latest svn source. I have 
followed the instructions in site.cfg.example to configure my own 
site.cfg and I'm using the same site.cfg for both numpy and scipy. numpy 
builds and installs fine and the tests pass. scipy won't build and 
throws out lots of warnings about the BLAS libraries and source. I have 
GotoBLAS2 installed in /usr/local/lib but it doesn't seem to find it.

Here is my site.cfg:
# Defaults
# ========
# The settings given here will apply to all other sections if not 
overridden.
# This is a good place to add general library and include directories like
# /usr/local/{lib,include}
#
[DEFAULT]
library_dirs = 
/usr/local/lib:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib:/home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
include_dirs = 
/usr/local/include:/home/john/local/src/SuiteSparse-3.4.0/AMD/Include:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Include

# Optimized BLAS and LAPACK
# -------------------------
# Use the blas_opt and lapack_opt sections to give any settings that are
# required to link against your chosen BLAS and LAPACK, including the 
regular
# FORTRAN reference BLAS and also ATLAS. Some other sections still exist for
# linking against certain optimized libraries (e.g. [atlas], 
[lapack_atlas]),
# however, they are now deprecated and should not be used.
#
# These are typical configurations for ATLAS (assuming that the library and
# include directories have already been set in [DEFAULT]; the include 
directory
# is important for the BLAS C interface):
#
[blas_opt]
libraries = goto2_penrynp-r1.13
#
[lapack_opt]
libraries = goto2_penrynp-r1.13
#
# If your ATLAS was compiled with pthreads, the names of the libraries 
might be
# different:
#
#[blas_opt]
#libraries = ptf77blas, ptcblas, atlas
#
#[lapack_opt]
#libraries = lapack, ptf77blas, ptcblas, atlas

# UMFPACK
# -------
# The UMFPACK library is used to factor large sparse matrices. It, in turn,
# depends on the AMD library for reordering the matrices for better 
performance.
# Note that the AMD library has nothing to do with AMD (Advanced Micro 
Devices),
# the CPU company.
#
#   http://www.cise.ufl.edu/research/sparse/umfpack/
#   http://www.cise.ufl.edu/research/sparse/amd/
#
#[amd]
amd_libs = amd
#
#[umfpack]
umfpack_libs = umfpack





and here are the errors from "python2.5 setup.py build" for scipy:

Warning: No configuration returned, assuming unavailable.blas_opt_info:
blas_mkl_info:
   libraries mkl,vml,guide not found in /usr/local/lib
   libraries mkl,vml,guide not found in 
/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib
   libraries mkl,vml,guide not found in 
/home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
   NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
   libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
   libraries ptf77blas,ptcblas,atlas not found in 
/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib
   libraries ptf77blas,ptcblas,atlas not found in 
/home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
   NOT AVAILABLE

atlas_blas_info:
   libraries f77blas,cblas,atlas not found in /usr/local/lib
   libraries f77blas,cblas,atlas not found in 
/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib
   libraries f77blas,cblas,atlas not found in 
/home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
   NOT AVAILABLE

/usr/lib/python2.5/site-packages/numpy/distutils/system_info.py:1399: 
UserWarning:
     Atlas (http://math-atlas.sourceforge.net/) libraries not found.
     Directories to search for the libraries can be specified in the
     numpy/distutils/site.cfg file (section [atlas]) or by setting
     the ATLAS environment variable.
   warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
   libraries blas not found in /usr/local/lib
   libraries blas not found in 
/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib
   libraries blas not found in 
/home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
   NOT AVAILABLE

/usr/lib/python2.5/site-packages/numpy/distutils/system_info.py:1408: 
UserWarning:
     Blas (http://www.netlib.org/blas/) libraries not found.
     Directories to search for the libraries can be specified in the
     numpy/distutils/site.cfg file (section [blas]) or by setting
     the BLAS environment variable.
   warnings.warn(BlasNotFoundError.__doc__)
blas_src_info:
   NOT AVAILABLE

/usr/lib/python2.5/site-packages/numpy/distutils/system_info.py:1411: 
UserWarning:
     Blas (http://www.netlib.org/blas/) sources not found.
     Directories to search for the sources can be specified in the
     numpy/distutils/site.cfg file (section [blas_src]) or by setting
     the BLAS_SRC environment variable.
   warnings.warn(BlasSrcNotFoundError.__doc__)
Traceback (most recent call last):
   File "setup.py", line 160, in <module>
     setup_package()
   File "setup.py", line 152, in setup_package
     configuration=configuration )
   File "/usr/lib/python2.5/site-packages/numpy/distutils/core.py", line 
152, in setup
     config = configuration()
   File "setup.py", line 118, in configuration
     config.add_subpackage('scipy')
   File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py", 
line 972, in add_subpackage
     caller_level = 2)
   File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py", 
line 941, in get_subpackage
     caller_level = caller_level + 1)
   File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py", 
line 878, in _get_configuration_from_setup_py
     config = setup_module.configuration(*args)
   File "scipy/setup.py", line 8, in configuration
     config.add_subpackage('integrate')
   File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py", 
line 972, in add_subpackage
     caller_level = 2)
   File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py", 
line 941, in get_subpackage
     caller_level = caller_level + 1)
   File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py", 
line 878, in _get_configuration_from_setup_py
     config = setup_module.configuration(*args)
   File "scipy/integrate/setup.py", line 10, in configuration
     blas_opt = get_info('blas_opt',notfound_action=2)
   File 
"/usr/lib/python2.5/site-packages/numpy/distutils/system_info.py", line 
310, in get_info
     return cl().get_info(notfound_action)
   File 
"/usr/lib/python2.5/site-packages/numpy/distutils/system_info.py", line 
461, in get_info
     raise self.notfounderror(self.notfounderror.__doc__)
numpy.distutils.system_info.BlasNotFoundError:
     Blas (http://www.netlib.org/blas/) libraries not found.
     Directories to search for the libraries can be specified in the
     numpy/distutils/site.cfg file (section [blas]) or by setting
     the BLAS environment variable.


Can anyone tell me what I'm doing wrong?

Thanks,
John.



From kmichael.aye at googlemail.com  Thu Apr  8 12:11:50 2010
From: kmichael.aye at googlemail.com (Michael Aye)
Date: Thu, 8 Apr 2010 09:11:50 -0700 (PDT)
Subject: [SciPy-User] my own hist_equal (Vectorization gods required! ;)
Message-ID: <4fca2d46-0967-48eb-9909-387926e02e01@r18g2000yqd.googlegroups.com>

Hi all,

I am implementing my own histogram equalization, as the few lines of
code that were mentioned before in this forum didn't really work for
me and I wanted to really understand what is going on there as well
anyway.

So here is what I got so far, main effort was to wrap my brain around
the double lookup table I needed to find the right values to replace
and the new values, based on the transformed cdf.
So, it seems to work fine, but the remaining loop of course disturbs
me in terms of efficiency and I wondered if some of you vectorization
gods could have a look if there's something possible.
Especially when I have 16 bit images, there are a lot of different
luminances I have to loop through, so it's not the fastest.
Apart from that I think it can't be optimized further much (I think?)

def hist_equal(data):
    # range is +2 to have the highest luminance to get into correct
bin
    bins = numpy.arange(data.min(), data.max() + 2)
    # first the histogram of luminances
    h, bins = numpy.histogram(data, bins=bins)
    # now get the cdf
    cdf = h.cumsum()
    # now get the unique luminance values
    uniques = numpy.unique(data)
    # this to not look it up again everytime in the numpy array
    nPixel = data.size
    cdfmin = cdf.min()
    # don't change the incoming data
    nData = data.copy()
    for lum in uniques:
        # algorithm taken from wikipedia entry for histogram
equalization
        # always scaling to 256 luminance values
        nData[numpy.where(data == lum)] = \
            numpy.round((cdf[numpy.where(bins == lum)] - cdfmin) *
255 / nPixel)
    return nData

I am thinking that this might even be useful to implement somewhere
(unless it is already and I missed it??)

Best regards,
Michael


From ighalp at gmail.com  Thu Apr  8 12:21:48 2010
From: ighalp at gmail.com (igor Halperin)
Date: Thu, 8 Apr 2010 12:21:48 -0400
Subject: [SciPy-User] Interpolation of function on a line vs 2D interpolation
Message-ID: <n2kea11c68c1004080921p474c38b3hd7e04e7c21b9d49b@mail.gmail.com>

Hi,

I am a bit stuck with the following interpolation problem. I have to compute
a function
z = f(x,y) whose arguments are realizations of components of a
two-dimensional random  variable (X,Y), obtained by Monte Carlo. As each
Monte Carlo simulation gives me a pair (x(t), y(t)) where t = 0,...,N-1
enumerates Monte Carlo runs, I can equivalently say that my function f is
defined on a discretized line t = 0,..., N-1 in a 2D space. Now, as
calculation of function f is costly but the function itself is smooth, I
want to calculate it on a sparse subset of simulated 2D points (x(t),y(t)),
and then interpolate for the rest of points.  (my sparse set includes
scenarios providing extreme values of x,y to avoid the need to
extrapolate).

Though this problem could be viewed as a 2D interpolation and solved using
function interpolate.bisplrep, in practice I run out of memory with this
method as the number N of 2D points is about 100,000, and I get memory error
when trying to produce matrices of dimension 100,000x100,000 needed for
interpolate.bisplrep using function meshgrid. However, in view of what is
said above,
the problem can alternatively be viewed as interpolation of a function
defined on a line. I was unable to figure out how/whether I could use
function interpolate.splprep to this end.

Any help on this would be greatly appreciated.

Igor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100408/c24be9ba/attachment.html>

From robert.kern at gmail.com  Thu Apr  8 12:34:12 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 8 Apr 2010 11:34:12 -0500
Subject: [SciPy-User] Interpolation of function on a line vs 2D
	interpolation
In-Reply-To: <n2kea11c68c1004080921p474c38b3hd7e04e7c21b9d49b@mail.gmail.com>
References: <n2kea11c68c1004080921p474c38b3hd7e04e7c21b9d49b@mail.gmail.com>
Message-ID: <h2u3d375d731004080934v229dc0fdh151f17a45f9a0b82@mail.gmail.com>

On Thu, Apr 8, 2010 at 11:21, igor Halperin <ighalp at gmail.com> wrote:
> Hi,
> I am a bit stuck with the following interpolation problem. I have to compute
> a function
> z = f(x,y)?whose arguments are realizations of components of a
> two-dimensional random ?variable (X,Y), obtained by Monte Carlo. As each
> Monte Carlo simulation gives me a pair (x(t), y(t)) where t = 0,...,N-1
> enumerates Monte Carlo runs, I can equivalently say that my function f is
> defined on a discretized line t = 0,..., N-1 in a 2D space. Now, as
> calculation of function f is costly but the function itself is smooth, I
> want to calculate it on a sparse?subset of simulated 2D points (x(t),y(t)),
> and then interpolate for the rest of points. ?(my sparse set includes
> scenarios providing extreme values of x,y to avoid the need to
> extrapolate).
> Though this problem could be viewed as a 2D interpolation and solved using
> function?interpolate.bisplrep, in practice I run out of memory with this
> method as the number N of 2D points is about 100,000, and I get memory error
> when trying to produce matrices of dimension 100,000x100,000 needed for
> interpolate.bisplrep?using function meshgrid.

Why would you use meshgrid? Can you show us the code you are trying?
It might be easier to see the problems that way.

> However, in view of what is
> said above,
> the problem can alternatively be viewed as interpolation of a function
> defined on a line.

No, not really. You will not get sensible results if you try that.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From cr.anil at gmail.com  Thu Apr  8 13:23:57 2010
From: cr.anil at gmail.com (Anil C R)
Date: Thu, 8 Apr 2010 22:53:57 +0530
Subject: [SciPy-User] equivalent of ind2sub
Message-ID: <y2l8cf7db601004081023kd971bd2cr42a1a0c316756cb2@mail.gmail.com>

is there any function which is similar to matlab function ind2sub?

Anil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100408/65dc7396/attachment.html>

From robert.kern at gmail.com  Thu Apr  8 13:46:51 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 8 Apr 2010 12:46:51 -0500
Subject: [SciPy-User] equivalent of ind2sub
In-Reply-To: <y2l8cf7db601004081023kd971bd2cr42a1a0c316756cb2@mail.gmail.com>
References: <y2l8cf7db601004081023kd971bd2cr42a1a0c316756cb2@mail.gmail.com>
Message-ID: <n2w3d375d731004081046k6f8e3cdfrdcd1f9e98f1acdfe@mail.gmail.com>

On Thu, Apr 8, 2010 at 12:23, Anil C R <cr.anil at gmail.com> wrote:
> is there any function which is similar to matlab function ind2sub?

Not everyone here uses Matlab. When asking for an equivalent, please
describe what the Matlab function does or provide a link to the
documentation. Thanks.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From kwgoodman at gmail.com  Thu Apr  8 13:48:09 2010
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 8 Apr 2010 10:48:09 -0700
Subject: [SciPy-User] equivalent of ind2sub
In-Reply-To: <y2l8cf7db601004081023kd971bd2cr42a1a0c316756cb2@mail.gmail.com>
References: <y2l8cf7db601004081023kd971bd2cr42a1a0c316756cb2@mail.gmail.com>
Message-ID: <p2kf4f93d421004081048kc5fbb72fyca9b6522d820dee@mail.gmail.com>

On Thu, Apr 8, 2010 at 10:23 AM, Anil C R <cr.anil at gmail.com> wrote:
> is there any function which is similar to matlab function ind2sub?

Is it enough to be able to use 1d indexing of a 2d array? You can do that with:

>> x
array([[0, 1, 2],
       [3, 4, 5]])
>> x.flat[0]
   0
>> x.flat[1]
   1
>> x.flat[5]
   5


From j.reid at mail.cryst.bbk.ac.uk  Thu Apr  8 14:17:43 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Thu, 08 Apr 2010 19:17:43 +0100
Subject: [SciPy-User] Where can I get some help building scipy?
Message-ID: <hpl6k7$bio$1@dough.gmane.org>

Is there a different mailing list for building/installing scipy?



From vanforeest at gmail.com  Thu Apr  8 14:23:26 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Thu, 8 Apr 2010 20:23:26 +0200
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <4BBD03E2.9090207@comcast.net>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com> <4BBB94D5.6020608@iri.columbia.edu>
	<201004071902.49240.mailinglists@xgm.de>
	<k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com>
	<i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com>
	<n2ue06186141004071140ja4a67827k4db1d68fbea0de81@mail.gmail.com>
	<4BBD03E2.9090207@comcast.net>
Message-ID: <g2ufa510ff81004081123l7091cbb5x74b4ed286d7894bb@mail.gmail.com>

Dear John,

I once read that the Beagle (the Mars spacecraft ) crashed because
some engineer forgot to convert inches to metres, or the other way
around. Since people use python at nasa, it might even be an
improvement to remove the English units rather than to keep them in
:-)

bye

NIcky


On 8 April 2010 00:14, John Hassler <hasslerjc at comcast.net> wrote:
>
> On 4/7/2010 2:40 PM, Charles R Harris wrote:
>
> On Wed, Apr 7, 2010 at 12:32 PM, David Goldsmith <d.l.goldsmith at gmail.com>
> wrote:
>>
>> On Wed, Apr 7, 2010 at 10:52 AM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>>
>>>
>>> On Wed, Apr 7, 2010 at 11:02 AM, Florian Lindner <mailinglists at xgm.de>
>>> wrote:
>>>>
>>>> Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
>>>> > It would seem that there is some confusion, in the
>>>> > constants.html, between force and mass...
>>>>
>>>> Beside the wrong unit which is fixed now I don't see any confusion.
>>>>
>>>> > Strictly speaking, kg is a unit of mass, Newton a unit
>>>> > of force. Weight is force, not mass: A gold brick
>>>> > floating in interstellar space is weightless but still
>>>> > massive. ?Pounds and kilograms can be equated, but only
>>>> > in some specified gravitational field (like at the
>>>> > surface of the earth, where we usually weigh things).
>>>>
>>>> This is true for pounds-force and kilograms. Pounds-mass and kilograms
>>>> could be equated in any context. Pound itself is ambigous.
>>>>
>>>> > So mass is the more fundamental quantity, since it does
>>>> > not depend on gravity for its value. In Imperial units
>>>> > (feet, pounds) the unit of mass is the slug:
>>>> > http://en.wikipedia.org/wiki/Slug_(mass). This is
>>>> > absent from the constants page.
>>>>
>>>> Mmmh.. never heard of it though I read quite some English language
>>>> aerospace engineering literature. However I'm using SI units. I think
>>>> pounds-mass is more widely used as a imperial unit of mass.
>>>>
>>>
>>> I recall slug being used in amateur rocketry books 50 years ago or so.
>>> But SI units are definitely simpler.
>>>
>>> Chuck
>>
>>
>> OK, since Charles opened the door: what about taking the bold,
>> forward-looking step of not supporting "Imperial" units at all?? (I say
>> "good riddance.")
>>
>
> Well, what if someone wants to know what a slug is in SI units? It's not as
> if it is a big problem to support, and the more conversions the better,
> IMHO. The point of having these things down in code is that one doesn't have
> to go looking when the unit turns up somewhere.
>
> Chuck
>
> I think you're always going to have to (in effect) look up the units ... or
> at least understand what they are doing.
>
> Back when I was a productive member of society, I used to teach this stuff.
> English units of mass and force are very difficult, and here's why.
> Someone, back in the dim dark reaches of history, before even I can
> remember, decided that one pound mass (lbm) should WEIGH one pound force
> (lbf).? Now, Newton said that F = M*A ... force = mass times acceleration.
> The natural unit for acceleration in English units is foot/sec**2, so we
> would naturally expect that:
> l lbf = 1 ft/sec**2 * 1 lbm.?? BUT .... weight is the force caused by the
> acceleration of gravity, and 1 lbm WEIGHS 1 lbf, so:
> 1 lbf = (32.174 ft/sec**2)*1 lbm.
>
> There seems to be a little problem here.? (Metric units _define_ force in
> terms of mass and acceleration, so this difficulty doesn't arise.? English
> units try to fix more variables than there are available degrees of
> freedom.)? There are two alternative ways to fix it.? One is to choose a
> more rational system of units.? The other is to use a fudge factor.? English
> engineers, naturally, chose the latter.
>
> The most common "fix" in Chemical Engineering is to define a fudge factor,
> the "gravitational constant," gc, such that:
> force (lbf) = acceleration (ft/sec**2)*mass(lbm)/gc.? So gc = 32.174 ft
> lbm/(lbf sec**2)
>
> (Another "fix" is to use mass in terms of "slugs," where a "slug" is the
> mass which is accelerated by 1 ft/sec**2 by a force of 1 lbf.? So a slug =
> 32.174 lbm.? This doesn't seem to be used much any more.)
>
> The units problem is a real stumbling block for engineering students
> whenever we use English units (which is most of the time, in American
> industry).? Some equations require gc, and some don't, and it isn't always
> obvious which is which.? Sometimes "lbm" is hidden.? For example, viscosity
> "contains" lbm ... except of course, if we use kinematic viscosity, which
> doesn't ... so equations containing viscosity need to have gc, except when
> they don't.
>
> And, of course, Murphy's law applies.
>
> john
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From josef.pktd at gmail.com  Thu Apr  8 14:29:11 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 8 Apr 2010 14:29:11 -0400
Subject: [SciPy-User] Where can I get some help building scipy?
In-Reply-To: <hpl6k7$bio$1@dough.gmane.org>
References: <hpl6k7$bio$1@dough.gmane.org>
Message-ID: <v2v1cd32cbb1004081129mfac309b6r9122ada27a3ab30c@mail.gmail.com>

On Thu, Apr 8, 2010 at 2:17 PM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
> Is there a different mailing list for building/installing scipy?

Here is the right place to ask, but because there are not many people
that know install issues across platforms, you might not get an answer
very fast if they are all busy.

Josef

>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Thu Apr  8 14:36:07 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 8 Apr 2010 14:36:07 -0400
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <hpkinj$pmp$1@dough.gmane.org>
References: <hpkinj$pmp$1@dough.gmane.org>
Message-ID: <v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>

On Thu, Apr 8, 2010 at 8:38 AM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
> Hi,
>
> I'm trying to build numpy and scipy from the latest svn source. I have
> followed the instructions in site.cfg.example to configure my own
> site.cfg and I'm using the same site.cfg for both numpy and scipy. numpy
> builds and installs fine and the tests pass. scipy won't build and
> throws out lots of warnings about the BLAS libraries and source. I have
> GotoBLAS2 installed in /usr/local/lib but it doesn't seem to find it.
>
> Here is my site.cfg:
> # Defaults
> # ========
> # The settings given here will apply to all other sections if not
> overridden.
> # This is a good place to add general library and include directories like
> # /usr/local/{lib,include}
> #
> [DEFAULT]
> library_dirs =
> /usr/local/lib:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib:/home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
> include_dirs =
> /usr/local/include:/home/john/local/src/SuiteSparse-3.4.0/AMD/Include:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Include

I don't know Linux nor gotblas, but I think that here you are telling
numpy distutils that all libraries and include are under AMD and
UMFPACK
So, it will look for the Blas libraries in these directories.
If your blas libraries are somewhere else, you need to specify the
correct library and include_dirs

(in my site.cfg on Windows XP I left these commented out, because
everything is in python lib)

Josef

>
> # Optimized BLAS and LAPACK
> # -------------------------
> # Use the blas_opt and lapack_opt sections to give any settings that are
> # required to link against your chosen BLAS and LAPACK, including the
> regular
> # FORTRAN reference BLAS and also ATLAS. Some other sections still exist for
> # linking against certain optimized libraries (e.g. [atlas],
> [lapack_atlas]),
> # however, they are now deprecated and should not be used.
> #
> # These are typical configurations for ATLAS (assuming that the library and
> # include directories have already been set in [DEFAULT]; the include
> directory
> # is important for the BLAS C interface):
> #
> [blas_opt]
> libraries = goto2_penrynp-r1.13
> #
> [lapack_opt]
> libraries = goto2_penrynp-r1.13
> #
> # If your ATLAS was compiled with pthreads, the names of the libraries
> might be
> # different:
> #
> #[blas_opt]
> #libraries = ptf77blas, ptcblas, atlas
> #
> #[lapack_opt]
> #libraries = lapack, ptf77blas, ptcblas, atlas
>
> # UMFPACK
> # -------
> # The UMFPACK library is used to factor large sparse matrices. It, in turn,
> # depends on the AMD library for reordering the matrices for better
> performance.
> # Note that the AMD library has nothing to do with AMD (Advanced Micro
> Devices),
> # the CPU company.
> #
> # ? http://www.cise.ufl.edu/research/sparse/umfpack/
> # ? http://www.cise.ufl.edu/research/sparse/amd/
> #
> #[amd]
> amd_libs = amd
> #
> #[umfpack]
> umfpack_libs = umfpack
>
>
>
>
>
> and here are the errors from "python2.5 setup.py build" for scipy:
>
> Warning: No configuration returned, assuming unavailable.blas_opt_info:
> blas_mkl_info:
> ? libraries mkl,vml,guide not found in /usr/local/lib
> ? libraries mkl,vml,guide not found in
> /home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib
> ? libraries mkl,vml,guide not found in
> /home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
> ? NOT AVAILABLE
>
> atlas_blas_threads_info:
> Setting PTATLAS=ATLAS
> ? libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
> ? libraries ptf77blas,ptcblas,atlas not found in
> /home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib
> ? libraries ptf77blas,ptcblas,atlas not found in
> /home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
> ? NOT AVAILABLE
>
> atlas_blas_info:
> ? libraries f77blas,cblas,atlas not found in /usr/local/lib
> ? libraries f77blas,cblas,atlas not found in
> /home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib
> ? libraries f77blas,cblas,atlas not found in
> /home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
> ? NOT AVAILABLE
>
> /usr/lib/python2.5/site-packages/numpy/distutils/system_info.py:1399:
> UserWarning:
> ? ? Atlas (http://math-atlas.sourceforge.net/) libraries not found.
> ? ? Directories to search for the libraries can be specified in the
> ? ? numpy/distutils/site.cfg file (section [atlas]) or by setting
> ? ? the ATLAS environment variable.
> ? warnings.warn(AtlasNotFoundError.__doc__)
> blas_info:
> ? libraries blas not found in /usr/local/lib
> ? libraries blas not found in
> /home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib
> ? libraries blas not found in
> /home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
> ? NOT AVAILABLE
>
> /usr/lib/python2.5/site-packages/numpy/distutils/system_info.py:1408:
> UserWarning:
> ? ? Blas (http://www.netlib.org/blas/) libraries not found.
> ? ? Directories to search for the libraries can be specified in the
> ? ? numpy/distutils/site.cfg file (section [blas]) or by setting
> ? ? the BLAS environment variable.
> ? warnings.warn(BlasNotFoundError.__doc__)
> blas_src_info:
> ? NOT AVAILABLE
>
> /usr/lib/python2.5/site-packages/numpy/distutils/system_info.py:1411:
> UserWarning:
> ? ? Blas (http://www.netlib.org/blas/) sources not found.
> ? ? Directories to search for the sources can be specified in the
> ? ? numpy/distutils/site.cfg file (section [blas_src]) or by setting
> ? ? the BLAS_SRC environment variable.
> ? warnings.warn(BlasSrcNotFoundError.__doc__)
> Traceback (most recent call last):
> ? File "setup.py", line 160, in <module>
> ? ? setup_package()
> ? File "setup.py", line 152, in setup_package
> ? ? configuration=configuration )
> ? File "/usr/lib/python2.5/site-packages/numpy/distutils/core.py", line
> 152, in setup
> ? ? config = configuration()
> ? File "setup.py", line 118, in configuration
> ? ? config.add_subpackage('scipy')
> ? File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py",
> line 972, in add_subpackage
> ? ? caller_level = 2)
> ? File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py",
> line 941, in get_subpackage
> ? ? caller_level = caller_level + 1)
> ? File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py",
> line 878, in _get_configuration_from_setup_py
> ? ? config = setup_module.configuration(*args)
> ? File "scipy/setup.py", line 8, in configuration
> ? ? config.add_subpackage('integrate')
> ? File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py",
> line 972, in add_subpackage
> ? ? caller_level = 2)
> ? File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py",
> line 941, in get_subpackage
> ? ? caller_level = caller_level + 1)
> ? File "/usr/lib/python2.5/site-packages/numpy/distutils/misc_util.py",
> line 878, in _get_configuration_from_setup_py
> ? ? config = setup_module.configuration(*args)
> ? File "scipy/integrate/setup.py", line 10, in configuration
> ? ? blas_opt = get_info('blas_opt',notfound_action=2)
> ? File
> "/usr/lib/python2.5/site-packages/numpy/distutils/system_info.py", line
> 310, in get_info
> ? ? return cl().get_info(notfound_action)
> ? File
> "/usr/lib/python2.5/site-packages/numpy/distutils/system_info.py", line
> 461, in get_info
> ? ? raise self.notfounderror(self.notfounderror.__doc__)
> numpy.distutils.system_info.BlasNotFoundError:
> ? ? Blas (http://www.netlib.org/blas/) libraries not found.
> ? ? Directories to search for the libraries can be specified in the
> ? ? numpy/distutils/site.cfg file (section [blas]) or by setting
> ? ? the BLAS environment variable.
>
>
> Can anyone tell me what I'm doing wrong?
>
> Thanks,
> John.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From peridot.faceted at gmail.com  Thu Apr  8 14:38:09 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 8 Apr 2010 14:38:09 -0400
Subject: [SciPy-User] Error in constants documentation?
In-Reply-To: <i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com>
References: <201004061821.26121.mailinglists@xgm.de>
	<4BBB6366.3000507@enthought.com> 
	<4BBB94D5.6020608@iri.columbia.edu>
	<201004071902.49240.mailinglists@xgm.de> 
	<k2ye06186141004071052u3b6077fbk6c4fb4d9980c1688@mail.gmail.com> 
	<i2x45d1ab481004071132u5b1238b7y57bd58e33df1d3bc@mail.gmail.com>
Message-ID: <p2jce557a361004081138o6ad396f9t6babf93f720ab085@mail.gmail.com>

On 7 April 2010 14:32, David Goldsmith <d.l.goldsmith at gmail.com> wrote:
> On Wed, Apr 7, 2010 at 10:52 AM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>>
>> On Wed, Apr 7, 2010 at 11:02 AM, Florian Lindner <mailinglists at xgm.de>
>> wrote:
>>>
>>> Am Dienstag, 6. April 2010 22:08:53 schrieb Arthur M. Greene:
>>> > It would seem that there is some confusion, in the
>>> > constants.html, between force and mass...
>>>
>>> Beside the wrong unit which is fixed now I don't see any confusion.
>>>
>>> > Strictly speaking, kg is a unit of mass, Newton a unit
>>> > of force. Weight is force, not mass: A gold brick
>>> > floating in interstellar space is weightless but still
>>> > massive.  Pounds and kilograms can be equated, but only
>>> > in some specified gravitational field (like at the
>>> > surface of the earth, where we usually weigh things).
>>>
>>> This is true for pounds-force and kilograms. Pounds-mass and kilograms
>>> could be equated in any context. Pound itself is ambigous.
>>>
>>> > So mass is the more fundamental quantity, since it does
>>> > not depend on gravity for its value. In Imperial units
>>> > (feet, pounds) the unit of mass is the slug:
>>> > http://en.wikipedia.org/wiki/Slug_(mass). This is
>>> > absent from the constants page.
>>>
>>> Mmmh.. never heard of it though I read quite some English language
>>> aerospace engineering literature. However I'm using SI units. I think
>>> pounds-mass is more widely used as a imperial unit of mass.
>>>
>>
>> I recall slug being used in amateur rocketry books 50 years ago or so. But
>> SI units are definitely simpler.
>>
>> Chuck
>
>
> OK, since Charles opened the door: what about taking the bold,
> forward-looking step of not supporting "Imperial" units at all?  (I say
> "good riddance.")

No, no! The point of providing these constants is so that users can
easily convert everything to SI (or cgs) from the horrible units they
are presented with by some third party. If we don't provide easy
conversions, they'll stick to whatever random system of units they're
stuck with.

Anne

> DG
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From j.reid at mail.cryst.bbk.ac.uk  Thu Apr  8 15:16:19 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Thu, 08 Apr 2010 20:16:19 +0100
Subject: [SciPy-User] Where can I get some help building scipy?
In-Reply-To: <v2v1cd32cbb1004081129mfac309b6r9122ada27a3ab30c@mail.gmail.com>
References: <hpl6k7$bio$1@dough.gmane.org>
	<v2v1cd32cbb1004081129mfac309b6r9122ada27a3ab30c@mail.gmail.com>
Message-ID: <hpla23$pgc$1@dough.gmane.org>

josef.pktd at gmail.com wrote:
> On Thu, Apr 8, 2010 at 2:17 PM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
>> Is there a different mailing list for building/installing scipy?
> 
> Here is the right place to ask, but because there are not many people
> that know install issues across platforms, you might not get an answer
> very fast if they are all busy.
> 
Ok thanks.



From j.reid at mail.cryst.bbk.ac.uk  Thu Apr  8 15:18:59 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Thu, 08 Apr 2010 20:18:59 +0100
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>
References: <hpkinj$pmp$1@dough.gmane.org>
	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>
Message-ID: <hpla73$pgc$2@dough.gmane.org>

josef.pktd at gmail.com wrote:
> On Thu, Apr 8, 2010 at 8:38 AM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
>> Hi,
>>
>> I'm trying to build numpy and scipy from the latest svn source. I have
>> followed the instructions in site.cfg.example to configure my own
>> site.cfg and I'm using the same site.cfg for both numpy and scipy. numpy
>> builds and installs fine and the tests pass. scipy won't build and
>> throws out lots of warnings about the BLAS libraries and source. I have
>> GotoBLAS2 installed in /usr/local/lib but it doesn't seem to find it.
>>
>> Here is my site.cfg:
>> # Defaults
>> # ========
>> # The settings given here will apply to all other sections if not
>> overridden.
>> # This is a good place to add general library and include directories like
>> # /usr/local/{lib,include}
>> #
>> [DEFAULT]
>> library_dirs =
>> /usr/local/lib:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib:/home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
>> include_dirs =
>> /usr/local/include:/home/john/local/src/SuiteSparse-3.4.0/AMD/Include:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Include
> 
> I don't know Linux nor gotblas, but I think that here you are telling
> numpy distutils that all libraries and include are under AMD and
> UMFPACK

I do have /usr/local/lib at the front of these library lists. That is 
where the libgoto.a and libgoto.so are stored. How else should I tell it 
where these libraries are?

Thanks,
John.



From e.antero.tammi at gmail.com  Thu Apr  8 15:33:53 2010
From: e.antero.tammi at gmail.com (eat)
Date: Thu, 8 Apr 2010 19:33:53 +0000 (UTC)
Subject: [SciPy-User] equivalent of ind2sub
References: <y2l8cf7db601004081023kd971bd2cr42a1a0c316756cb2@mail.gmail.com>
Message-ID: <loom.20100408T213020-652@post.gmane.org>

Anil C R <cr.anil <at> gmail.com> writes:
> 
> 
> is there any function which is similar to matlab function ind2sub?
> 
> Anil
> 
Hi,

Would you like to give some use case(s) of ind2sub?


Regards,
eat
> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 






From josef.pktd at gmail.com  Thu Apr  8 16:04:10 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 8 Apr 2010 16:04:10 -0400
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <hpla73$pgc$2@dough.gmane.org>
References: <hpkinj$pmp$1@dough.gmane.org>
	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>
	<hpla73$pgc$2@dough.gmane.org>
Message-ID: <m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>

On Thu, Apr 8, 2010 at 3:18 PM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
> josef.pktd at gmail.com wrote:
>> On Thu, Apr 8, 2010 at 8:38 AM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
>>> Hi,
>>>
>>> I'm trying to build numpy and scipy from the latest svn source. I have
>>> followed the instructions in site.cfg.example to configure my own
>>> site.cfg and I'm using the same site.cfg for both numpy and scipy. numpy
>>> builds and installs fine and the tests pass. scipy won't build and
>>> throws out lots of warnings about the BLAS libraries and source. I have
>>> GotoBLAS2 installed in /usr/local/lib but it doesn't seem to find it.
>>>
>>> Here is my site.cfg:
>>> # Defaults
>>> # ========
>>> # The settings given here will apply to all other sections if not
>>> overridden.
>>> # This is a good place to add general library and include directories like
>>> # /usr/local/{lib,include}
>>> #
>>> [DEFAULT]
>>> library_dirs =
>>> /usr/local/lib:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib:/home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
>>> include_dirs =
>>> /usr/local/include:/home/john/local/src/SuiteSparse-3.4.0/AMD/Include:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Include
>>
>> I don't know Linux nor gotblas, but I think that here you are telling
>> numpy distutils that all libraries and include are under AMD and
>> UMFPACK
>
> I do have /usr/local/lib at the front of these library lists. That is
> where the libgoto.a and libgoto.so are stored. How else should I tell it
> where these libraries are?

My mistake, I didn't know : works as separator
the only other thing, I can think of, is whether it looks for the
correct filenames, is goto2_penrynp-r1.13
the actual filename?

my impression is that
#[blas_opt]
#libraries = ptf77blas, ptcblas, atlas
#
#[lapack_opt]
#libraries = lapack, ptf77blas, ptcblas, atlas

refer to actual filenames of the libraries (maybe minus prefix and extension).

But I'm usually debugging by trial and error (and I'm not too bad on
it on Windows), so the guess in your case might be as bad as my first
guess.

Josef

>
> Thanks,
> John.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From david_baddeley at yahoo.com.au  Thu Apr  8 18:59:22 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Thu, 8 Apr 2010 15:59:22 -0700 (PDT)
Subject: [SciPy-User] my own hist_equal (Vectorization gods required! ; )
In-Reply-To: <4fca2d46-0967-48eb-9909-387926e02e01@r18g2000yqd.googlegroups.com>
References: <4fca2d46-0967-48eb-9909-387926e02e01@r18g2000yqd.googlegroups.com>
Message-ID: <808235.46205.qm@web33002.mail.mud.yahoo.com>

Hi Micheal,

I think the folowing should also work (& is probably faster):

def hist_equal(data):
    I = data.ravel().argsort()
    return numpy.linspace(0, 255, I.size)[I].reshape(data.shape)

this works because you can effectively create the cdf by plotting the sorted data on the x axis, with something going linearly form 0 to 1 on the y (rather than using histograms and cumsums). The downside of this approach is that intensity values which are the same in the input data are not going to be exactly the same in the output (they'll be spread over the interval between one value and the next highest). For practical usage (particularly seeing as histogram equalisation is a horribly non-linear operation to start with), this might well be a problem you can live with. 

If this isn't acceptable, you should be able to speed your code up a bit by doing the following: 
1) - precomputing round((cdf - cdfmin)*255/nPixels)
2) - getting rid of numpy.where (which tends to be slow) and just using boolean/fancy indexing (fast) eg:
nData[data == lum]

David

----- Original Message ----
From: Michael Aye <kmichael.aye at googlemail.com>
To: scipy-user at scipy.org
Sent: Fri, 9 April, 2010 4:11:50 AM
Subject: [SciPy-User] my own hist_equal (Vectorization gods required! ;)

Hi all,

I am implementing my own histogram equalization, as the few lines of
code that were mentioned before in this forum didn't really work for
me and I wanted to really understand what is going on there as well
anyway.

So here is what I got so far, main effort was to wrap my brain around
the double lookup table I needed to find the right values to replace
and the new values, based on the transformed cdf.
So, it seems to work fine, but the remaining loop of course disturbs
me in terms of efficiency and I wondered if some of you vectorization
gods could have a look if there's something possible.
Especially when I have 16 bit images, there are a lot of different
luminances I have to loop through, so it's not the fastest.
Apart from that I think it can't be optimized further much (I think?)

def hist_equal(data):
    # range is +2 to have the highest luminance to get into correct
bin
    bins = numpy.arange(data.min(), data.max() + 2)
    # first the histogram of luminances
    h, bins = numpy.histogram(data, bins=bins)
    # now get the cdf
    cdf = h.cumsum()
    # now get the unique luminance values
    uniques = numpy.unique(data)
    # this to not look it up again everytime in the numpy array
    nPixel = data.size
    cdfmin = cdf.min()
    # don't change the incoming data
    nData = data.copy()
    for lum in uniques:
        # algorithm taken from wikipedia entry for histogram
equalization
        # always scaling to 256 luminance values
        nData[numpy.where(data == lum)] = \
            numpy.round((cdf[numpy.where(bins == lum)] - cdfmin) *
255 / nPixel)
    return nData

I am thinking that this might even be useful to implement somewhere
(unless it is already and I missed it??)

Best regards,
Michael
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From wesmckinn at gmail.com  Thu Apr  8 19:37:54 2010
From: wesmckinn at gmail.com (Wes McKinney)
Date: Thu, 8 Apr 2010 19:37:54 -0400
Subject: [SciPy-User] my own hist_equal (Vectorization gods required! ; )
In-Reply-To: <808235.46205.qm@web33002.mail.mud.yahoo.com>
References: <4fca2d46-0967-48eb-9909-387926e02e01@r18g2000yqd.googlegroups.com>
	<808235.46205.qm@web33002.mail.mud.yahoo.com>
Message-ID: <z2h6c476c8a1004081637v15bd4232ob749de5191b6317a@mail.gmail.com>

On Thu, Apr 8, 2010 at 6:59 PM, David Baddeley
<david_baddeley at yahoo.com.au> wrote:
> Hi Micheal,
>
> I think the folowing should also work (& is probably faster):
>
> def hist_equal(data):
> ? ?I = data.ravel().argsort()
> ? ?return numpy.linspace(0, 255, I.size)[I].reshape(data.shape)
>
> this works because you can effectively create the cdf by plotting the sorted data on the x axis, with something going linearly form 0 to 1 on the y (rather than using histograms and cumsums). The downside of this approach is that intensity values which are the same in the input data are not going to be exactly the same in the output (they'll be spread over the interval between one value and the next highest). For practical usage (particularly seeing as histogram equalisation is a horribly non-linear operation to start with), this might well be a problem you can live with.
>
> If this isn't acceptable, you should be able to speed your code up a bit by doing the following:
> 1) - precomputing round((cdf - cdfmin)*255/nPixels)
> 2) - getting rid of numpy.where (which tends to be slow) and just using boolean/fancy indexing (fast) eg:
> nData[data == lum]
>
> David
>
> ----- Original Message ----
> From: Michael Aye <kmichael.aye at googlemail.com>
> To: scipy-user at scipy.org
> Sent: Fri, 9 April, 2010 4:11:50 AM
> Subject: [SciPy-User] my own hist_equal (Vectorization gods required! ;)
>
> Hi all,
>
> I am implementing my own histogram equalization, as the few lines of
> code that were mentioned before in this forum didn't really work for
> me and I wanted to really understand what is going on there as well
> anyway.
>
> So here is what I got so far, main effort was to wrap my brain around
> the double lookup table I needed to find the right values to replace
> and the new values, based on the transformed cdf.
> So, it seems to work fine, but the remaining loop of course disturbs
> me in terms of efficiency and I wondered if some of you vectorization
> gods could have a look if there's something possible.
> Especially when I have 16 bit images, there are a lot of different
> luminances I have to loop through, so it's not the fastest.
> Apart from that I think it can't be optimized further much (I think?)
>
> def hist_equal(data):
> ? ?# range is +2 to have the highest luminance to get into correct
> bin
> ? ?bins = numpy.arange(data.min(), data.max() + 2)
> ? ?# first the histogram of luminances
> ? ?h, bins = numpy.histogram(data, bins=bins)
> ? ?# now get the cdf
> ? ?cdf = h.cumsum()
> ? ?# now get the unique luminance values
> ? ?uniques = numpy.unique(data)
> ? ?# this to not look it up again everytime in the numpy array
> ? ?nPixel = data.size
> ? ?cdfmin = cdf.min()
> ? ?# don't change the incoming data
> ? ?nData = data.copy()
> ? ?for lum in uniques:
> ? ? ? ?# algorithm taken from wikipedia entry for histogram
> equalization
> ? ? ? ?# always scaling to 256 luminance values
> ? ? ? ?nData[numpy.where(data == lum)] = \
> ? ? ? ? ? ?numpy.round((cdf[numpy.where(bins == lum)] - cdfmin) *
> 255 / nPixel)
> ? ?return nData
>
> I am thinking that this might even be useful to implement somewhere
> (unless it is already and I missed it??)
>
> Best regards,
> Michael
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>

In my experience, for setting values using a boolean mask like the
above, numpy.putmask is significantly faster than fancy indexing. This
is also true of numpy.take versus fancy indexing (but I don't know if
that can be used here)

- Wes


From bioseasoul at gmail.com  Fri Apr  9 00:15:37 2010
From: bioseasoul at gmail.com (seasoul)
Date: Fri, 9 Apr 2010 12:15:37 +0800
Subject: [SciPy-User] How to find the intersections between two lines
Message-ID: <y2i7869ca2b1004082115y9328babdke4d98f698248aa0e@mail.gmail.com>

This is my first post in the mailing list.
I want to find the intersections between two lines. One line is an arbitrary
straight line, the other is a fitting line created by a interpolation
procedure.
Below is a simple code:


x = linspace(0,10,20)
y = rand(20)*10
plt.plot(x,y,'o')
f = interpolate.interp1d(x,y,kind=5)
xx = linspace(0,10,100)
yy = f(xx)
plt.plot(xx,yy,'r-')
y2 = x
plt.plot(x,y2,'b-')
def func(x):
   out = [x[1]-x[0]]
   out.append(x[1]-f(x[0]))
   return out
fsolve(func,[1,1])

The fitting line is a polynomial curve, the straight line is y=x. This
function can only find one of the intersection points,not all.

I tried to use x[:,1] and x[:,2], and hope in the function func, the
operation is element-by-element, but some errors came out.
Any help is appreciated.

The output figure:

 http://picasaweb.google.com/lh/photo/vJZRP6tAmmhpIh4J2BG0-g?feat=directlink
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100409/c28e748a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 4.png
Type: image/png
Size: 33435 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100409/c28e748a/attachment.png>

From charlesr.harris at gmail.com  Fri Apr  9 01:06:15 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 8 Apr 2010 23:06:15 -0600
Subject: [SciPy-User] How to find the intersections between two lines
In-Reply-To: <y2i7869ca2b1004082115y9328babdke4d98f698248aa0e@mail.gmail.com>
References: <y2i7869ca2b1004082115y9328babdke4d98f698248aa0e@mail.gmail.com>
Message-ID: <y2oe06186141004082206o1d0433acs74e85447b68153fa@mail.gmail.com>

On Thu, Apr 8, 2010 at 10:15 PM, seasoul <bioseasoul at gmail.com> wrote:

>
> This is my first post in the mailing list.
> I want to find the intersections between two lines. One line is an
> arbitrary straight line, the other is a fitting line created by a
> interpolation procedure.
> Below is a simple code:
>
>
> x = linspace(0,10,20)
> y = rand(20)*10
> plt.plot(x,y,'o')
> f = interpolate.interp1d(x,y,kind=5)
> xx = linspace(0,10,100)
> yy = f(xx)
> plt.plot(xx,yy,'r-')
> y2 = x
> plt.plot(x,y2,'b-')
> def func(x):
>    out = [x[1]-x[0]]
>    out.append(x[1]-f(x[0]))
>    return out
> fsolve(func,[1,1])
>
> The fitting line is a polynomial curve, the straight line is y=x. This
> function can only find one of the intersection points,not all.
>
>

This is spline interpolation, not polynomial. So I think you are stuck with
sampling the difference of the two curves on a grid, finding adjacent pairs
of sample points at which the function has opposite sign, and then calling
one of the 1d zero finders, brentq for instance.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100408/e65f41c9/attachment.html>

From ighalp at gmail.com  Fri Apr  9 02:03:40 2010
From: ighalp at gmail.com (igor Halperin)
Date: Fri, 9 Apr 2010 02:03:40 -0400
Subject: [SciPy-User] Interpolation of function on a line vs 2D
	interpolation
Message-ID: <h2mea11c68c1004082303n16615285w870d2ce6279016c3@mail.gmail.com>

Hi Robert,

On your questions:

> Why would you use meshgrid? Can you show us the code you are trying?
> It might be easier to see the problems that way.

My bad. I glanced through examples in the cookbook and thought that
interpolate.bisplev needs to pass matrices rather than 1D arrays, which is
not the case, hence was my confusion.

> No, not really. You will not get sensible results if you try that.

You are right. Even though I do formally have a function defined on a line,
this line does not have a structure, and thus I would not get sensible
results in this way.
I realized it after submitting the original question. My attempt to
reformulate it as a line interpolation problem was due to my
misinterpretation of the usage of function
interpolate.bisplev per above.

Looks like all I needed to do were these two standard lines of code:

tck = interpolate.bisplrep(sValsOnReducedSet[:,0],
                                    sValsOnReducedSet[:,1],
                                    optValsOnReducedSet,s=0)

optValsOnFullSet =
interpolate.bisplev(sValsOnFullSet[:,0],sValsOnFullSet[:,1],tck)


Thanks a lot for your help!
Igor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100409/58925bb5/attachment.html>

From bioseasoul at gmail.com  Fri Apr  9 03:23:03 2010
From: bioseasoul at gmail.com (seasoul)
Date: Fri, 9 Apr 2010 00:23:03 -0700 (PDT)
Subject: [SciPy-User] [SciPy-user] How to find the intersections between
	two lines
In-Reply-To: <y2oe06186141004082206o1d0433acs74e85447b68153fa@mail.gmail.com>
References: <y2i7869ca2b1004082115y9328babdke4d98f698248aa0e@mail.gmail.com>
	<y2oe06186141004082206o1d0433acs74e85447b68153fa@mail.gmail.com>
Message-ID: <28188548.post@talk.nabble.com>


A nice idea. Thank you Charles. I will give it a try.

Charles R Harris wrote:
> 
> On Thu, Apr 8, 2010 at 10:15 PM, seasoul <bioseasoul at gmail.com> wrote:
> 
>>
>> This is my first post in the mailing list.
>> I want to find the intersections between two lines. One line is an
>> arbitrary straight line, the other is a fitting line created by a
>> interpolation procedure.
>> Below is a simple code:
>>
>>
>> x = linspace(0,10,20)
>> y = rand(20)*10
>> plt.plot(x,y,'o')
>> f = interpolate.interp1d(x,y,kind=5)
>> xx = linspace(0,10,100)
>> yy = f(xx)
>> plt.plot(xx,yy,'r-')
>> y2 = x
>> plt.plot(x,y2,'b-')
>> def func(x):
>>    out = [x[1]-x[0]]
>>    out.append(x[1]-f(x[0]))
>>    return out
>> fsolve(func,[1,1])
>>
>> The fitting line is a polynomial curve, the straight line is y=x. This
>> function can only find one of the intersection points,not all.
>>
>>
> 
> This is spline interpolation, not polynomial. So I think you are stuck
> with
> sampling the difference of the two curves on a grid, finding adjacent
> pairs
> of sample points at which the function has opposite sign, and then calling
> one of the 1d zero finders, brentq for instance.
> 
> Chuck
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 

-- 
View this message in context: http://old.nabble.com/How-to-find-the-intersections-between-two-lines-tp28187442p28188548.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From j.reid at mail.cryst.bbk.ac.uk  Fri Apr  9 04:04:27 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Fri, 09 Apr 2010 09:04:27 +0100
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>
References: <hpkinj$pmp$1@dough.gmane.org>	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>	<hpla73$pgc$2@dough.gmane.org>
	<m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>
Message-ID: <hpmn2c$vdo$1@dough.gmane.org>

josef.pktd at gmail.com wrote:
> On Thu, Apr 8, 2010 at 3:18 PM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
>> josef.pktd at gmail.com wrote:
>>> On Thu, Apr 8, 2010 at 8:38 AM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
>>>> Hi,
>>>>
>>>> I'm trying to build numpy and scipy from the latest svn source. I have
>>>> followed the instructions in site.cfg.example to configure my own
>>>> site.cfg and I'm using the same site.cfg for both numpy and scipy. numpy
>>>> builds and installs fine and the tests pass. scipy won't build and
>>>> throws out lots of warnings about the BLAS libraries and source. I have
>>>> GotoBLAS2 installed in /usr/local/lib but it doesn't seem to find it.
>>>>
>>>> Here is my site.cfg:
>>>> # Defaults
>>>> # ========
>>>> # The settings given here will apply to all other sections if not
>>>> overridden.
>>>> # This is a good place to add general library and include directories like
>>>> # /usr/local/{lib,include}
>>>> #
>>>> [DEFAULT]
>>>> library_dirs =
>>>> /usr/local/lib:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Lib:/home/john/local/src/SuiteSparse-3.4.0/AMD/Lib
>>>> include_dirs =
>>>> /usr/local/include:/home/john/local/src/SuiteSparse-3.4.0/AMD/Include:/home/john/local/src/SuiteSparse-3.4.0/UMFPACK/Include
>>> I don't know Linux nor gotblas, but I think that here you are telling
>>> numpy distutils that all libraries and include are under AMD and
>>> UMFPACK
>> I do have /usr/local/lib at the front of these library lists. That is
>> where the libgoto.a and libgoto.so are stored. How else should I tell it
>> where these libraries are?
> 
> My mistake, I didn't know : works as separator
> the only other thing, I can think of, is whether it looks for the
> correct filenames, is goto2_penrynp-r1.13
> the actual filename?
Yes. Well I have both libgoto2_penrynp-r1.13.a and 
libgoto2_penrynp-r1.13.so in /usr/local/lib

> 
> my impression is that
> #[blas_opt]
> #libraries = ptf77blas, ptcblas, atlas
> #
> #[lapack_opt]
> #libraries = lapack, ptf77blas, ptcblas, atlas
> 
> refer to actual filenames of the libraries (maybe minus prefix and extension).
> 
> But I'm usually debugging by trial and error (and I'm not too bad on
> it on Windows), so the guess in your case might be as bad as my first
> guess.
That's what I find frustrating : python setup.py produces all these 
warnings and errors but they don't seem to relate directly to the 
sections in site.cfg. As far as I can tell from the documentation on the 
website, I've done the right thing. Unfortunately its not working and 
the error messages aren't constructive enough to help me fix it.



From david at silveregg.co.jp  Fri Apr  9 05:32:49 2010
From: david at silveregg.co.jp (David)
Date: Fri, 09 Apr 2010 18:32:49 +0900
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <hpmn2c$vdo$1@dough.gmane.org>
References: <hpkinj$pmp$1@dough.gmane.org>
	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>
	<hpla73$pgc$2@dough.gmane.org>
	<m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>
	<hpmn2c$vdo$1@dough.gmane.org>
Message-ID: <4BBEF441.1060301@silveregg.co.jp>

On 04/09/2010 05:04 PM, John Reid wrote:

> That's what I find frustrating : python setup.py produces all these
> warnings and errors but they don't seem to relate directly to the
> sections in site.cfg.

Don't look at the documentation or the output, they are useless at best, 
when not outright confusing. You have to look at the sources to 
understand what's going on,

cheers,

David


From j.reid at mail.cryst.bbk.ac.uk  Fri Apr  9 05:47:36 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Fri, 09 Apr 2010 10:47:36 +0100
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <4BBEF441.1060301@silveregg.co.jp>
References: <hpkinj$pmp$1@dough.gmane.org>	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>	<hpla73$pgc$2@dough.gmane.org>	<m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>	<hpmn2c$vdo$1@dough.gmane.org>
	<4BBEF441.1060301@silveregg.co.jp>
Message-ID: <hpmt3o$m4v$1@dough.gmane.org>

David wrote:
> On 04/09/2010 05:04 PM, John Reid wrote:
> 
>> That's what I find frustrating : python setup.py produces all these
>> warnings and errors but they don't seem to relate directly to the
>> sections in site.cfg.
> 
> Don't look at the documentation or the output, they are useless at best, 
> when not outright confusing. You have to look at the sources to 
> understand what's going on,

I tried that as well. I don't know the internals of numpy.distutils so I 
found that frustrating too.



From baker.alexander at gmail.com  Fri Apr  9 07:15:08 2010
From: baker.alexander at gmail.com (alexander baker)
Date: Fri, 9 Apr 2010 12:15:08 +0100
Subject: [SciPy-User] [SciPy-user] How to find the intersections between
	two lines
In-Reply-To: <28188548.post@talk.nabble.com>
References: <y2i7869ca2b1004082115y9328babdke4d98f698248aa0e@mail.gmail.com>
	<y2oe06186141004082206o1d0433acs74e85447b68153fa@mail.gmail.com>
	<28188548.post@talk.nabble.com>
Message-ID: <h2r270620221004090415r221e803jfa0e19b99a9cc24a@mail.gmail.com>

bit rushed but seems to do the trick!

Alex

import numpy as np
from scipy import interpolate
from scipy import optimize
import pylab

x = np.linspace(0,10,20)
y = np.random.random(20)*10
f = interpolate.interp1d(x,y,kind=5)
xx = np.linspace(0,10,100)
pylab.plot(xx,f(xx),'r-',x,x,'b-')

g = [(s,f(s)-s) for s in xx]
h = np.array([(i,k,l) for i, (k,l) in enumerate(g)])
for i,j in enumerate(h):
        if i < 98:
                if (h[i][2] > 0 and h[i+1][2] < 0) or (h[i][2] < 0 and
h[i+1][2] > 0):
                        print optimize.brentq(lambda x: f(x)-x, h[i][1],
h[i+1][1])


Mobile: 07788 872118
Blog: www.alexfb.com

--
All science is either physics or stamp collecting.


On 9 April 2010 08:23, seasoul <bioseasoul at gmail.com> wrote:

>
> A nice idea. Thank you Charles. I will give it a try.
>
> Charles R Harris wrote:
> >
> > On Thu, Apr 8, 2010 at 10:15 PM, seasoul <bioseasoul at gmail.com> wrote:
> >
> >>
> >> This is my first post in the mailing list.
> >> I want to find the intersections between two lines. One line is an
> >> arbitrary straight line, the other is a fitting line created by a
> >> interpolation procedure.
> >> Below is a simple code:
> >>
> >>
> >> x = linspace(0,10,20)
> >> y = rand(20)*10
> >> plt.plot(x,y,'o')
> >> f = interpolate.interp1d(x,y,kind=5)
> >> xx = linspace(0,10,100)
> >> yy = f(xx)
> >> plt.plot(xx,yy,'r-')
> >> y2 = x
> >> plt.plot(x,y2,'b-')
> >> def func(x):
> >>    out = [x[1]-x[0]]
> >>    out.append(x[1]-f(x[0]))
> >>    return out
> >> fsolve(func,[1,1])
> >>
> >> The fitting line is a polynomial curve, the straight line is y=x. This
> >> function can only find one of the intersection points,not all.
> >>
> >>
> >
> > This is spline interpolation, not polynomial. So I think you are stuck
> > with
> > sampling the difference of the two curves on a grid, finding adjacent
> > pairs
> > of sample points at which the function has opposite sign, and then
> calling
> > one of the 1d zero finders, brentq for instance.
> >
> > Chuck
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/How-to-find-the-intersections-between-two-lines-tp28187442p28188548.html
> Sent from the Scipy-User mailing list archive at Nabble.com.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100409/1a590bbe/attachment.html>

From cr.anil at gmail.com  Fri Apr  9 07:37:43 2010
From: cr.anil at gmail.com (Anil C R)
Date: Fri, 9 Apr 2010 17:07:43 +0530
Subject: [SciPy-User] equivalent of ind2sub
In-Reply-To: <loom.20100408T213020-652@post.gmane.org>
References: <y2l8cf7db601004081023kd971bd2cr42a1a0c316756cb2@mail.gmail.com> 
	<loom.20100408T213020-652@post.gmane.org>
Message-ID: <u2z8cf7db601004090437g52f7873arefa740557a00152f@mail.gmail.com>

hey guys, I wrote my own function which does what ind2sub does:

############################CODE###################################
def ind2sub( dim, shap, lst ):
    '''
    I1, I2, I3, ..., Idim = ind2sub(dim, shap, lst)
    Input:
        dim - number of dimensions
        shap - shape of the array
        lst - list of indicies
    Output:
        list of subscripts
    r = array([[0,1,0,0],[0,0,1,1],[1,1,1,0],[1,0,0,1]])
    l = find(r==1)
    [cols,rows] = ind2sub(2, r.shape, l)
    # The element in coulmn cols[i], and row rows[i] is 1.
    '''
    if len(shap) <= dim:
        shap = shap + tuple(zeros((dim - len(shap),)))
    else:
        shap = shap[0:dim-1] + (prod(shap[(dim-1):]),)

    n = len(shap)
    k = array([1] + cumprod(shap[0:(n-1)]).tolist())

    argout = [zeros((len(lst),))]*n

    for i in xrange(n-1,-1,-1):
        vi = (lst)%k[i]
        vj = (lst-vi)/k[i]
        argout[i] = vj
        lst = vi
    return argout
##########################END CODE#################################

Anil


On Fri, Apr 9, 2010 at 1:03 AM, eat <e.antero.tammi at gmail.com> wrote:

> Anil C R <cr.anil <at> gmail.com> writes:
> >
> >
> > is there any function which is similar to matlab function ind2sub?
> >
> > Anil
> >
> Hi,
>
> Would you like to give some use case(s) of ind2sub?
>
>
> Regards,
> eat
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User <at> scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100409/f7e5751e/attachment.html>

From amcmorl at gmail.com  Fri Apr  9 08:21:36 2010
From: amcmorl at gmail.com (Angus McMorland)
Date: Fri, 9 Apr 2010 13:21:36 +0100
Subject: [SciPy-User] equivalent of ind2sub
In-Reply-To: <y2l8cf7db601004081023kd971bd2cr42a1a0c316756cb2@mail.gmail.com>
References: <y2l8cf7db601004081023kd971bd2cr42a1a0c316756cb2@mail.gmail.com>
Message-ID: <z2te85f13c41004090521ofb423eazc4fd50b6dc640981@mail.gmail.com>

On 8 April 2010 18:23, Anil C R <cr.anil at gmail.com> wrote:

> is there any function which is similar to matlab function ind2sub?


numpy.unravel_index

docstring:

 Convert a flat index into an index tuple for an array of given shape.

    Parameters
    ----------
    x : int
        Flattened index.
    dims : shape tuple
        Input shape.

    Notes
    -----
    In the Examples section, since ``arr.flat[x] == arr.max()`` it may be
    easier to use flattened indexing than to re-map the index to a tuple.

    Examples
    --------
    >>> arr = np.arange(20).reshape(5,4)
    >>> arr
    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11],
           [12, 13, 14, 15],
           [16, 17, 18, 19]])
    >>> x = arr.argmax()
    >>> x
    19
    >>> dims = arr.shape
    >>> idx = np.unravel_index(x, dims)
    >>> idx
    (4, 3)
    >>> arr[idx] == arr.max()
    True


-- 
AJC McMorland
Post-doctoral research fellow
Neurobiology, University of Pittsburgh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100409/63d46ddc/attachment.html>

From j.reid at mail.cryst.bbk.ac.uk  Fri Apr  9 09:18:53 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Fri, 09 Apr 2010 14:18:53 +0100
Subject: [SciPy-User] How to link against custom BLAS?
Message-ID: <hpn9ft$5hf$1@dough.gmane.org>

Perhaps I can put this another way.

If I have a library libmyblasimplementation.so in directory 
/the/directory/where/I/store/my/libraries, what do I put in my site.cfg 
to get numpy and scipy to use it? I've been told the scipy build 
documentation isn't the best place to find this out.

Also, can I share my site.cfg between numpy and scipy?

Am I right in thinking that whatever I put in the numpy site.cfg is 
copied into the installation /usr/lib/python2.5/numpy/... somewhere and 
when I come to build scipy, these values are used?

I'm on Ubuntu and OpenSuse.

Thanks,
John.



From bruce.labitt at autoliv.com  Fri Apr  9 13:50:28 2010
From: bruce.labitt at autoliv.com (bruce.labitt at autoliv.com)
Date: Fri, 9 Apr 2010 13:50:28 -0400
Subject: [SciPy-User] pythonxy scipy error win32
Message-ID: <OF417F824B.F88541E8-ON85257700.00594B9A-85257700.0062306D@autoliv.int>

Not sure how to diagnose this.  Looking to the list for some guidance.

Running scipy/numpy routines that use leastsq and svd.  The code runs fine 
(to completion - and generates plots) under Ubuntu9.10 x86-64. 

I am also testing the routines using Pythonxy 2.6.2.0 bundle running on 
WinXP 32 bit. 
Installed plugins: CDT 6.0.1, cython 0.11.3, ETS 3.3.0, IPython 0.10.0, 
OpenGL 3.0.1a3, PIL 1.1.6, Photran 5.0.0, PyQt4 4.5.4, PyWavelets 0.1.6, 
Pydev 1.5.0, Qwt5 5.2.0, SciTE 2.01, WinMerge 2.12.4, console 2.0.144, itk 
3.16, jinja2 2.2.1, matplotlib 0.99.1, mdp 2.5, mingw 4.4.0, nose 0.11.1, 
numexpr 1.3.1, numpy 1.3.0, opencv 1.1pre1a, pp 1.5.7, psyco 1.6, py2exe 
0.6.9, pygments 1.1.1, pylint 0.18.1, pyreadline 1.5, py2win32 2.14, scipy 
0.7.1, spyder 1.0.1, swig 1.3.40, vpython 5.13, vtk 5.4.2, winpdb 1.4.6, 
wxPython 2.8.10.1, xlutils 1.4.1, xy 1.1.0

I am running in an IPython (mlab) session.  The routine runs a while, 
(outputting to the console) then the last message I get is: 

 ** On entry to ZGELSS parameter number 12 had an illegal value

Then the IPython session is terminated with no additional error messages. 
The IPython window merely reads: C:\Documents and 
Settings\me\.xy\startups>

Google reveals that ZGELSS is an interface to LAPACK.

If I run a Spyder session and execute from there, values are printed to 
the console then the Spyder application terminates and the entire Spyder 
Window vanishes.

Is this a Scipy Win32 issue or Pythonxy Win32 issue, neither, both ??? In, 
other words, which list do I post to?

TIA,
Bruce



******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an <br>electronic signature and/or legally binding agreement in the absence of an <br>express statement or Autoliv policy and/or procedure to the contrary.<br>This E-mail and any attachments hereto are Autoliv property and may contain legally <br>privileged, confidential and/or proprietary information.<br>The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way <br>disseminating any material contained within this E-mail without prior written <br>permission from the author. If you receive this E-mail in error, please <br>immediately notify the author and delete this E-mail.  Autoliv disclaims all <br>responsibility and liability for the consequences of any person who fails to <br>abide by the terms herein. <br>
******************************

From kmichael.aye at googlemail.com  Fri Apr  9 14:14:46 2010
From: kmichael.aye at googlemail.com (Michael Aye)
Date: Fri, 9 Apr 2010 11:14:46 -0700 (PDT)
Subject: [SciPy-User] my own hist_equal (Vectorization gods required! ; )
In-Reply-To: <808235.46205.qm@web33002.mail.mud.yahoo.com>
References: <4fca2d46-0967-48eb-9909-387926e02e01@r18g2000yqd.googlegroups.com>
	<808235.46205.qm@web33002.mail.mud.yahoo.com>
Message-ID: <04612cb2-f3d1-4b33-ae23-34e6a691c5e8@z6g2000yqz.googlegroups.com>

Hi David,

thanks for your input!

>
> def hist_equal(data):
> ? ? I = data.ravel().argsort()
> ? ? return numpy.linspace(0, 255, I.size)[I].reshape(data.shape)
>
I quickly pasted it into my code, but something must be wrong here, as
the resulting image is totally messed up
(see
http://dl.dropbox.com/u/139035/FC21A0000000_08239230753F1C.IMG.equal.bad.png
where it should look more like
http://dl.dropbox.com/u/139035/FC21A0000000_08239230753F1C.IMG.equal.png
)

> If this isn't acceptable, you should be able to speed your code up a bit by doing the following:
> 1) - precomputing round((cdf - cdfmin)*255/nPixels)
> 2) - getting rid of numpy.where (which tends to be slow) and just using boolean/fancy indexing (fast) eg:
> nData[data == lum]
>
tried these and they left the speed roughly similar, but nevertheless
important that you reminded me of fancy indexing, thanks! ;)

Wes's numpy.putmask however sped up things a bit at least, maybe 2 to
3 times. (Thanks Wes!)
But shouldn't the mapping to each luminance value be 'puttable' as
well without this darn loop? ;)

Have a good weekend!
Best regards,
Michael


From matthew.brett at gmail.com  Fri Apr  9 15:38:04 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 9 Apr 2010 15:38:04 -0400
Subject: [SciPy-User] scipy.stats.invnorm - rename to invgauss?
In-Reply-To: <t2h1cd32cbb1004071211i6336ff55rf61b3bba437b63b0@mail.gmail.com>
References: <n2k1e2af89e1004071145pe067ee26j4c9c0b684a4afdc7@mail.gmail.com>
	<r2v1cd32cbb1004071205w9d683b09n1d9cbc69ca2be686@mail.gmail.com>
	<t2h1cd32cbb1004071211i6336ff55rf61b3bba437b63b0@mail.gmail.com>
Message-ID: <k2k1e2af89e1004091238q963f1f6fsd41537c25c3ce627@mail.gmail.com>

Hi,

> But from http://en.wikipedia.org/wiki/Inverse_Gaussian_distribution
> it looks like it's not a inverse transformation. In that case a
> different name might reduce some confusion.
>
> I will look at what invnorm really is, (I don't know)

My impression is that the accepted name for this distribution is
'inverse Gaussian' or 'Wald' - but I don't think anyone else calls it
the 'inverse normal' - I am happy to be corrected if I am wrong.

Given that 'inverse normal' confused both of us, in different ways, do
you agree that a switch would be good?

See you,

Matthew


From josef.pktd at gmail.com  Fri Apr  9 17:55:07 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 9 Apr 2010 17:55:07 -0400
Subject: [SciPy-User] scipy.stats.invnorm - rename to invgauss?
In-Reply-To: <k2k1e2af89e1004091238q963f1f6fsd41537c25c3ce627@mail.gmail.com>
References: <n2k1e2af89e1004071145pe067ee26j4c9c0b684a4afdc7@mail.gmail.com>
	<r2v1cd32cbb1004071205w9d683b09n1d9cbc69ca2be686@mail.gmail.com>
	<t2h1cd32cbb1004071211i6336ff55rf61b3bba437b63b0@mail.gmail.com>
	<k2k1e2af89e1004091238q963f1f6fsd41537c25c3ce627@mail.gmail.com>
Message-ID: <p2j1cd32cbb1004091455h99ffbfcfhdf87bc822f422381@mail.gmail.com>

On Fri, Apr 9, 2010 at 3:38 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
>> But from http://en.wikipedia.org/wiki/Inverse_Gaussian_distribution
>> it looks like it's not a inverse transformation. In that case a
>> different name might reduce some confusion.
>>
>> I will look at what invnorm really is, (I don't know)
>
> My impression is that the accepted name for this distribution is
> 'inverse Gaussian' or 'Wald' - but I don't think anyone else calls it
> the 'inverse normal' - I am happy to be corrected if I am wrong.
>
> Given that 'inverse normal' confused both of us, in different ways, do
> you agree that a switch would be good?

I agree with the switch, I forgot to reply

I had checked Johnson, Kotz and Balakrishnan, and they have 3 or 4
parameterizations of the Inverse Gaussian, one of them is Wald, I
haven't checked yet if scale is a missing parameter in their version.

According to JKB, Tweedie called it the inverse gaussian because the
cumulants of IG and normal are (somehow) inversely related. But it is
not a transformation as e.g. lognorm.
(IG as abbreviation is confusing because I have seen IG for inverse
gamma or something like this.)

Numpy.random has this distribution available as wald, invnorm.rvs =
np.random.wald
My impression is InverseGaussian is the main name, in indices I saw
"Wald (see Inverse Gaussian)".

So the main question is whether to call it invgauss or wald, assuming
nobody objects to a renaming to clarify this.
I still want to compare parameterizations, but I'm in favor of renaming it.

Josef

>
> See you,
>
> Matthew
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From nico.schloemer at gmail.com  Fri Apr  9 19:54:51 2010
From: nico.schloemer at gmail.com (=?ISO-8859-1?Q?Nico_Schl=F6mer?=)
Date: Sat, 10 Apr 2010 01:54:51 +0200
Subject: [SciPy-User] SciPy 0.7.2 RC x download?
Message-ID: <q2nd933725a1004091654wdce88e3fk7b30649caac610db@mail.gmail.com>

Hi,

I understand that the missing curve_fit has been re-added in SciPy
0.7.2 (RC 1), but I can't find the release candidate for download
anywhere.
It'd be great if someone could point me to it.

Cheers,
Nico


From ralf.gommers at googlemail.com  Fri Apr  9 23:08:35 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 10 Apr 2010 11:08:35 +0800
Subject: [SciPy-User] SciPy 0.7.2 RC x download?
In-Reply-To: <q2nd933725a1004091654wdce88e3fk7b30649caac610db@mail.gmail.com>
References: <q2nd933725a1004091654wdce88e3fk7b30649caac610db@mail.gmail.com>
Message-ID: <x2pdde7764a1004092008rf1097c1bt27fbd7894f8cdb92@mail.gmail.com>

On Sat, Apr 10, 2010 at 7:54 AM, Nico Schl?mer <nico.schloemer at gmail.com>wrote:

> Hi,
>
> I understand that the missing curve_fit has been re-added in SciPy
> 0.7.2 (RC 1), but I can't find the release candidate for download
> anywhere.
> It'd be great if someone could point me to it.
>
>
Windows binaries:
http://www.filefactory.com/file/b0ghb9e/n/numpy-1.4.1rc1-win32-superpack-python2.6.exe
http://filefactory.com/file/b0ghce6/n/scipy-0.7.2rc1
-win32-superpack-python2.6.exe

OS X binaries:
http://www.filefactory.com/file/b0gh5h1/n/numpy-1.4.1rc1-py2.6-python.org.dmg
http://www.filefactory.com/file/b0gh638/n/scipy-0.7.2rc1
-py2.6-python.org.dmg


0.7.2 is functionally the same as 0.7.1, curve_fit is not present. In trunk
it should be. Where did you hear otherwise?

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100410/b4e449b5/attachment.html>

From bioseasoul at gmail.com  Sat Apr 10 01:54:26 2010
From: bioseasoul at gmail.com (seasoul)
Date: Fri, 9 Apr 2010 22:54:26 -0700 (PDT)
Subject: [SciPy-User] [SciPy-user] How to find the intersections between
	two lines
In-Reply-To: <h2r270620221004090415r221e803jfa0e19b99a9cc24a@mail.gmail.com>
References: <y2i7869ca2b1004082115y9328babdke4d98f698248aa0e@mail.gmail.com>
	<y2oe06186141004082206o1d0433acs74e85447b68153fa@mail.gmail.com>
	<28188548.post@talk.nabble.com>
	<h2r270620221004090415r221e803jfa0e19b99a9cc24a@mail.gmail.com>
Message-ID: <28200155.post@talk.nabble.com>


Really really works.


Alexander Baker wrote:
> 
> bit rushed but seems to do the trick!
> 
> Alex
> 
> import numpy as np
> from scipy import interpolate
> from scipy import optimize
> import pylab
> 
> x = np.linspace(0,10,20)
> y = np.random.random(20)*10
> f = interpolate.interp1d(x,y,kind=5)
> xx = np.linspace(0,10,100)
> pylab.plot(xx,f(xx),'r-',x,x,'b-')
> 
> g = [(s,f(s)-s) for s in xx]
> h = np.array([(i,k,l) for i, (k,l) in enumerate(g)])
> for i,j in enumerate(h):
>         if i < 98:
>                 if (h[i][2] > 0 and h[i+1][2] < 0) or (h[i][2] < 0 and
> h[i+1][2] > 0):
>                         print optimize.brentq(lambda x: f(x)-x, h[i][1],
> h[i+1][1])
> 
> 
> Mobile: 07788 872118
> Blog: www.alexfb.com
> 
> --
> All science is either physics or stamp collecting.
> 
> 
> On 9 April 2010 08:23, seasoul <bioseasoul at gmail.com> wrote:
> 
>>
>> A nice idea. Thank you Charles. I will give it a try.
>>
>> Charles R Harris wrote:
>> >
>> > On Thu, Apr 8, 2010 at 10:15 PM, seasoul <bioseasoul at gmail.com> wrote:
>> >
>> >>
>> >> This is my first post in the mailing list.
>> >> I want to find the intersections between two lines. One line is an
>> >> arbitrary straight line, the other is a fitting line created by a
>> >> interpolation procedure.
>> >> Below is a simple code:
>> >>
>> >>
>> >> x = linspace(0,10,20)
>> >> y = rand(20)*10
>> >> plt.plot(x,y,'o')
>> >> f = interpolate.interp1d(x,y,kind=5)
>> >> xx = linspace(0,10,100)
>> >> yy = f(xx)
>> >> plt.plot(xx,yy,'r-')
>> >> y2 = x
>> >> plt.plot(x,y2,'b-')
>> >> def func(x):
>> >>    out = [x[1]-x[0]]
>> >>    out.append(x[1]-f(x[0]))
>> >>    return out
>> >> fsolve(func,[1,1])
>> >>
>> >> The fitting line is a polynomial curve, the straight line is y=x. This
>> >> function can only find one of the intersection points,not all.
>> >>
>> >>
>> >
>> > This is spline interpolation, not polynomial. So I think you are stuck
>> > with
>> > sampling the difference of the two curves on a grid, finding adjacent
>> > pairs
>> > of sample points at which the function has opposite sign, and then
>> calling
>> > one of the 1d zero finders, brentq for instance.
>> >
>> > Chuck
>> >
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/How-to-find-the-intersections-between-two-lines-tp28187442p28188548.html
>> Sent from the Scipy-User mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 
:clap::clap:
-- 
View this message in context: http://old.nabble.com/How-to-find-the-intersections-between-two-lines-tp28187442p28200155.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From nico.schloemer at gmail.com  Sat Apr 10 03:32:43 2010
From: nico.schloemer at gmail.com (=?ISO-8859-1?Q?Nico_Schl=F6mer?=)
Date: Sat, 10 Apr 2010 09:32:43 +0200
Subject: [SciPy-User] SciPy 0.7.2 RC x download?
In-Reply-To: <x2pdde7764a1004092008rf1097c1bt27fbd7894f8cdb92@mail.gmail.com>
References: <q2nd933725a1004091654wdce88e3fk7b30649caac610db@mail.gmail.com>
	<x2pdde7764a1004092008rf1097c1bt27fbd7894f8cdb92@mail.gmail.com>
Message-ID: <s2id933725a1004100032wc6a9d53fyd40a5f6b788b26fa@mail.gmail.com>

> Where did you hear otherwise?
Oh, I was just implicitly assuming it, considering the fact that the bug

    http://projects.scipy.org/scipy/ticket/1030

has been fixed half a *year a ago. Hm.

I'd need the sources, anyway (runnning Gentoo here).

Cheers,
Nico




On Sat, Apr 10, 2010 at 5:08 AM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
>
>
> On Sat, Apr 10, 2010 at 7:54 AM, Nico Schl?mer <nico.schloemer at gmail.com>
> wrote:
>>
>> Hi,
>>
>> I understand that the missing curve_fit has been re-added in SciPy
>> 0.7.2 (RC 1), but I can't find the release candidate for download
>> anywhere.
>> It'd be great if someone could point me to it.
>>
>
> Windows binaries:
> http://www.filefactory.com/file/b0ghb9e/n/numpy-1.4.1rc1-win32-superpack-python2.6.exe
> http://filefactory.com/file/b0ghce6/n/scipy-0.7.2rc1-win32-superpack-python2.6.exe
>
> OS X binaries:
> http://www.filefactory.com/file/b0gh5h1/n/numpy-1.4.1rc1-py2.6-python.org.dmg
> http://www.filefactory.com/file/b0gh638/n/scipy-0.7.2rc1-py2.6-python.org.dmg
>
>
> 0.7.2 is functionally the same as 0.7.1, curve_fit is not present. In trunk
> it should be. Where did you hear otherwise?
>
> Cheers,
> Ralf
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From ralf.gommers at googlemail.com  Sat Apr 10 09:51:02 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 10 Apr 2010 21:51:02 +0800
Subject: [SciPy-User] SciPy 0.7.2 RC x download?
In-Reply-To: <s2id933725a1004100032wc6a9d53fyd40a5f6b788b26fa@mail.gmail.com>
References: <q2nd933725a1004091654wdce88e3fk7b30649caac610db@mail.gmail.com>
	<x2pdde7764a1004092008rf1097c1bt27fbd7894f8cdb92@mail.gmail.com>
	<s2id933725a1004100032wc6a9d53fyd40a5f6b788b26fa@mail.gmail.com>
Message-ID: <v2idde7764a1004100651q821c32dbsd93f925956161563@mail.gmail.com>

On Sat, Apr 10, 2010 at 3:32 PM, Nico Schl?mer <nico.schloemer at gmail.com>wrote:

> > Where did you hear otherwise?
> Oh, I was just implicitly assuming it, considering the fact that the bug
>
>    http://projects.scipy.org/scipy/ticket/1030
>
> has been fixed half a *year a ago. Hm.
>
> I'd need the sources, anyway (runnning Gentoo here).
>
> The 0.7.x branch was started in January 2009, the only reason for another
release from this branch is to solve the binary compatibility issue with
numpy 1.4.x. If you're compiling from source anyway I'd just use current
trunk if I were you.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100410/9dac81ec/attachment.html>

From nico.schloemer at gmail.com  Sat Apr 10 11:59:14 2010
From: nico.schloemer at gmail.com (=?ISO-8859-1?Q?Nico_Schl=F6mer?=)
Date: Sat, 10 Apr 2010 17:59:14 +0200
Subject: [SciPy-User] SciPy 0.7.2 RC x download?
In-Reply-To: <v2idde7764a1004100651q821c32dbsd93f925956161563@mail.gmail.com>
References: <q2nd933725a1004091654wdce88e3fk7b30649caac610db@mail.gmail.com>
	<x2pdde7764a1004092008rf1097c1bt27fbd7894f8cdb92@mail.gmail.com>
	<s2id933725a1004100032wc6a9d53fyd40a5f6b788b26fa@mail.gmail.com>
	<v2idde7764a1004100651q821c32dbsd93f925956161563@mail.gmail.com>
Message-ID: <i2md933725a1004100859qd31da4dbj228e7c662f4bc90f@mail.gmail.com>

> numpy 1.4.x. If you're compiling from source anyway I'd just use current
> trunk if I were you.

Alright, I guess that's what I'll be doing.
Cheers!
Nico



On Sat, Apr 10, 2010 at 3:51 PM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
>
>
> On Sat, Apr 10, 2010 at 3:32 PM, Nico Schl?mer <nico.schloemer at gmail.com>
> wrote:
>>
>> > Where did you hear otherwise?
>> Oh, I was just implicitly assuming it, considering the fact that the bug
>>
>> ? ?http://projects.scipy.org/scipy/ticket/1030
>>
>> has been fixed half a *year a ago. Hm.
>>
>> I'd need the sources, anyway (runnning Gentoo here).
>>
> The 0.7.x branch was started in January 2009, the only reason for another
> release from this branch is to solve the binary compatibility issue with
> numpy 1.4.x. If you're compiling from source anyway I'd just use current
> trunk if I were you.
>
> Cheers,
> Ralf
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From stephane.campinas at deri.org  Sat Apr 10 12:42:05 2010
From: stephane.campinas at deri.org (guts)
Date: Sat, 10 Apr 2010 09:42:05 -0700 (PDT)
Subject: [SciPy-User] [SciPy-user] Anova and the level of significance
Message-ID: <28194677.post@talk.nabble.com>


Hello,

I used the #f_oneway method from scipy.stats to compute the f_value and the
critical value. To test it, i used the groups given in this book [1], which
are:

                                                               Alternatives
Measurements                                1               2         3
1                                                0.0972      0.1382    
0.7966
2                                                0.0971      0.1432    
0.5300
3                                                0.0969      0.1382    
0.5152
4                                                0.1954      0.1730    
0.6675
5                                                0.0974      0.1383    
0.5298

The F-value calculated is:          66.4     and the critical value is    
3:89 (with a level of significance of 0.05).
Those returned by the f_oneway method are, respectively:       66.37 and 
3.2462326729e-07.

I don't understand why the critical value is so much smaller. I wanted to
get the level significance from the method
but didn't find how. Do you have an explanation for this number?

Thanks for the help!

[1]: Measuring computer performance: a practitioner's guide, By David J.
Lilja

-----
Stephane Campinas
-- 
View this message in context: http://old.nabble.com/Anova-and-the-level-of-significance-tp28194677p28194677.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From josef.pktd at gmail.com  Sat Apr 10 13:29:11 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 10 Apr 2010 13:29:11 -0400
Subject: [SciPy-User] [SciPy-user] Anova and the level of significance
In-Reply-To: <28194677.post@talk.nabble.com>
References: <28194677.post@talk.nabble.com>
Message-ID: <i2p1cd32cbb1004101029ve11eef8ci3052b8af95186d91@mail.gmail.com>

On Sat, Apr 10, 2010 at 12:42 PM, guts <stephane.campinas at deri.org> wrote:
>
> Hello,
>
> I used the #f_oneway method from scipy.stats to compute the f_value and the
> critical value. To test it, i used the groups given in this book [1], which
> are:
>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Alternatives
> Measurements ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?1 ? ? ? ? ? ? ? 2 ? ? ? ? 3
> 1 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?0.0972 ? ? ?0.1382
> 0.7966
> 2 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?0.0971 ? ? ?0.1432
> 0.5300
> 3 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?0.0969 ? ? ?0.1382
> 0.5152
> 4 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?0.1954 ? ? ?0.1730
> 0.6675
> 5 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?0.0974 ? ? ?0.1383
> 0.5298
>
> The F-value calculated is: ? ? ? ? ?66.4 ? ? and the critical value is
> 3:89 (with a level of significance of 0.05).
> Those returned by the f_oneway method are, respectively: ? ? ? 66.37 and
> 3.2462326729e-07.
>
> I don't understand why the critical value is so much smaller. I wanted to
> get the level significance from the method
> but didn't find how. Do you have an explanation for this number?

It took me a bit of time to figure out the degrees of freedom for your example

3.89 is the f-value  at a significance level of 0.05
>>> stats.f.sf(3.89, 2, 15-3)
0.049857414449464656

For this dataset, the f-value is much higher than the 5% critical
value, this means that the p-value, i.e. the probability that the f
value of 66.37 would be observed under the null hypothesis is tiny

>>> stats.f.sf(66.37, 2, 15-3)
3.2475516470133395e-007

So the null hypothesis of identical means (no differences across
groups) can be rejected with a very high confidence level.

>>> x.mean(0)
array([ 0.1168 ,  0.14618,  0.60782])

I hope this helps,
(I have verified f_oneway against the NIST benchmark cases, and the
results are correct except for very badly scaled examples.)

Josef


>
> Thanks for the help!
>
> [1]: Measuring computer performance: a practitioner's guide, By David J.
> Lilja
>
> -----
> Stephane Campinas
> --
> View this message in context: http://old.nabble.com/Anova-and-the-level-of-significance-tp28194677p28194677.html
> Sent from the Scipy-User mailing list archive at Nabble.com.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From devicerandom at gmail.com  Sat Apr 10 13:55:17 2010
From: devicerandom at gmail.com (ms)
Date: Sat, 10 Apr 2010 19:55:17 +0200
Subject: [SciPy-User] ODR and full output puzzle
Message-ID: <4BC0BB85.8060707@gmail.com>

Hi,

I am using ODR for a program and I now would like to have the optional
attributes in the output from the fit, as described here (In particular,
I'd like to have the sum of squares error):

http://www.scipy.org/doc/api_docs/SciPy.odr.odrpack.Output.html

The documentation there says such parameters:

" [are] Present if odr() was run with "full_output=1"."

Well, problem is that nothing in the documentation (no examples,
nothing) mentions running odr(), that help(scipy.odr.odr) returns
nothing except a terse list of parameters, that no other documentation
is also available online about this function and in fact I do my fits
since years with ODR without having ever run odr(). All I can find on
the function is the bottom lines of this page:

http://docs.scipy.org/doc/scipy/reference/odr.html

which doesn't help a lot, as you can see.

Does anyone know what does this function do and how to obtain the full
output?

Thanks a lot!
Massimo


From robert.kern at gmail.com  Sat Apr 10 18:57:20 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 10 Apr 2010 17:57:20 -0500
Subject: [SciPy-User] ODR and full output puzzle
In-Reply-To: <4BC0BB85.8060707@gmail.com>
References: <4BC0BB85.8060707@gmail.com>
Message-ID: <l2s3d375d731004101557nc57250bexac55cec8cdacbac1@mail.gmail.com>

On Sat, Apr 10, 2010 at 12:55, ms <devicerandom at gmail.com> wrote:
> Hi,
>
> I am using ODR for a program and I now would like to have the optional
> attributes in the output from the fit, as described here (In particular,
> I'd like to have the sum of squares error):
>
> http://www.scipy.org/doc/api_docs/SciPy.odr.odrpack.Output.html
>
> The documentation there says such parameters:
>
> " [are] Present if odr() was run with "full_output=1"."
>
> Well, problem is that nothing in the documentation (no examples,
> nothing) mentions running odr(), that help(scipy.odr.odr) returns
> nothing except a terse list of parameters, that no other documentation
> is also available online about this function and in fact I do my fits
> since years with ODR without having ever run odr(). All I can find on
> the function is the bottom lines of this page:
>
> http://docs.scipy.org/doc/scipy/reference/odr.html
>
> which doesn't help a lot, as you can see.
>
> Does anyone know what does this function do and how to obtain the full
> output?

Hmm. The docstring is in an older style, so it doesn't seem to be
parsed correctly by the web doc app. Anyways, all of the attributes
are fully described in the docstring:

class Output(object):
    """ The Output class stores the output of an ODR run.

    Takes one argument for initialization: the return value from the
    function odr().

    Attributes
    ----------
      beta -- estimated parameter values [beta.shape == (q,)]

      sd_beta -- standard errors of the estimated parameters
                 [sd_beta.shape == (p,)]

      cov_beta -- covariance matrix of the estimated parameters
                  [cov_beta.shape == (p, p)]

    Optional Attributes
    -------------------
    Present if odr() was run with "full_output=1".

      delta -- array of estimated errors in input variables
               [delta.shape == data.x.shape]

      eps -- array of estimated errors in response variables
             [eps.shape == data.y.shape]

      xplus -- array of x + delta [xplus.shape == data.x.shape]

      y -- array of y = fcn(x + delta) [y.shape == data.y.shape]

      res_var -- residual variance [scalar]

      sum_sqare -- sum of squares error [scalar]

      sum_square_delta -- sum of squares of delta error [scalar]

      sum_square_eps -- sum of squares of eps error [scalar]

      inv_condnum -- inverse condition number [scalar] (cf. ODRPACK UG p. 77)

      rel_error -- relative error in function values computed within
fcn [scalar]

      work -- final work array [array]

      work_ind -- indices into work for drawing out values [dictionary]
                  (cf. ODRPACK UG p. 83)

      info -- reason for returning (as output by ODRPACK) [integer]
              (cf. ODRPACK UG p. 38)

      stopreason -- "info" interpreted into English [list of strings]
    """

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Sat Apr 10 19:03:02 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 10 Apr 2010 18:03:02 -0500
Subject: [SciPy-User] ODR and full output puzzle
In-Reply-To: <4BC0BB85.8060707@gmail.com>
References: <4BC0BB85.8060707@gmail.com>
Message-ID: <u2r3d375d731004101603z66f1789ch68d3f4b6542a32a7@mail.gmail.com>

On Sat, Apr 10, 2010 at 12:55, ms <devicerandom at gmail.com> wrote:
> Hi,
>
> I am using ODR for a program and I now would like to have the optional
> attributes in the output from the fit, as described here (In particular,
> I'd like to have the sum of squares error):
>
> http://www.scipy.org/doc/api_docs/SciPy.odr.odrpack.Output.html
>
> The documentation there says such parameters:
>
> " [are] Present if odr() was run with "full_output=1"."
>
> Well, problem is that nothing in the documentation (no examples,
> nothing) mentions running odr(), that help(scipy.odr.odr) returns
> nothing except a terse list of parameters, that no other documentation
> is also available online about this function and in fact I do my fits
> since years with ODR without having ever run odr(). All I can find on
> the function is the bottom lines of this page:
>
> http://docs.scipy.org/doc/scipy/reference/odr.html
>
> which doesn't help a lot, as you can see.
>
> Does anyone know what does this function do and how to obtain the full
> output?

Sorry, the answer to your actual question is that the Output instance
with all of the attributes is output by the ODR.run() method. You
don't need to change anything or use the odr() function. ODR.run()
will call odr() with full_output=1.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From apalomba at austin.rr.com  Sat Apr 10 19:34:42 2010
From: apalomba at austin.rr.com (Anthony Palomba)
Date: Sat, 10 Apr 2010 18:34:42 -0500
Subject: [SciPy-User] visualizing scipy...
Message-ID: <s2o675c26741004101634qf478adfau9eded1fc7b5764d5@mail.gmail.com>

Hey folks, I am new to scipy, I would like to explore computational geometry

with scipy. To do so, I need a way to visualize my experiments. What is
the best thing to use to visualize 2d and 3d data in scipy?



Thanks,
Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100410/2e2f3b3f/attachment.html>

From cournape at gmail.com  Sun Apr 11 02:25:46 2010
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 11 Apr 2010 15:25:46 +0900
Subject: [SciPy-User] visualizing scipy...
In-Reply-To: <s2o675c26741004101634qf478adfau9eded1fc7b5764d5@mail.gmail.com>
References: <s2o675c26741004101634qf478adfau9eded1fc7b5764d5@mail.gmail.com>
Message-ID: <s2y5b8d13221004102325o903b421ay69e935f9fa0edfcd@mail.gmail.com>

On Sun, Apr 11, 2010 at 8:34 AM, Anthony Palomba <apalomba at austin.rr.com> wrote:
> Hey folks, I am new to scipy, I would like to explore computational geometry
> with scipy. To do so, I need a way to visualize my experiments. What is
> the best thing to use to visualize 2d and 3d data in scipy?

Scipy itself purposely doesn't offer any graphical capabilities - it
relies on 3rd party packages. The most used package is matplotlib, for
publication quality graphics - it does mostly 2d, with a bit of 3d. If
you know matlab, matplotlib gives you a similar feature set.

For more involved 3d, there is mayavi. Other options such are chaco
are available, with a somewhat different focus than matplotlib.

David


From devicerandom at gmail.com  Sun Apr 11 05:08:16 2010
From: devicerandom at gmail.com (ms)
Date: Sun, 11 Apr 2010 11:08:16 +0200
Subject: [SciPy-User] ODR and full output puzzle
In-Reply-To: <u2r3d375d731004101603z66f1789ch68d3f4b6542a32a7@mail.gmail.com>
References: <4BC0BB85.8060707@gmail.com>
	<u2r3d375d731004101603z66f1789ch68d3f4b6542a32a7@mail.gmail.com>
Message-ID: <4BC19180.40200@gmail.com>

On 04/11/10 01:03, Robert Kern wrote:
> On Sat, Apr 10, 2010 at 12:55, ms <devicerandom at gmail.com> wrote:
>> Does anyone know what does this function do and how to obtain the full
>> output?
> 
> Sorry, the answer to your actual question is that the Output instance
> with all of the attributes is output by the ODR.run() method. You
> don't need to change anything or use the odr() function. ODR.run()
> will call odr() with full_output=1.
> 

Ouch, thanks! Can this be added to the documentation (in the Output
class documentation?). It is just a little thing but it would help
clarifying it. :)

thanks a lot!
m.


From spons at utm.csic.es  Sun Apr 11 13:50:49 2010
From: spons at utm.csic.es (Sergi Pons Freixes)
Date: Sun, 11 Apr 2010 19:50:49 +0200
Subject: [SciPy-User] Joining 1D Timeseries to create a nD Timeseries
Message-ID: <y2h2d88eb101004111050h555af1d2r92a37744d2db31cc@mail.gmail.com>

Is there any elegant and fast way to join multiple unidimensional
timeseries to create a new one? E.g:

ts1 = timeseries 1D with param1
ts2 = timeseries 2D with param2
(ts1 and ts2 have same Date array)

newts = join_ts(ts1, ts2) ---> newts= timeseries 2D with param2, param2

Or can this only be accomplished by creating a new timeseries, and
manually setting the date array and all the data at the creation time?

Regards,
Sergi

-- 
Sergi Pons Freixes
Ph.D. Student
Marine Technology Unit
Centre Mediterrani d'Investigacions Marines i Ambientals (CMIMA-CSIC)
Pg. Mar?tim Barceloneta, 37-49
E-08003 Barcelona (Spain)
Ph.  +34 93 230 95 00 (ext. 1510)
Fax. +34 93 230 95 55
spons at utm.csic.es
http://www.utm.csic.es


From pgmdevlist at gmail.com  Sun Apr 11 14:17:12 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 11 Apr 2010 14:17:12 -0400
Subject: [SciPy-User] Joining 1D Timeseries to create a nD Timeseries
In-Reply-To: <y2h2d88eb101004111050h555af1d2r92a37744d2db31cc@mail.gmail.com>
References: <y2h2d88eb101004111050h555af1d2r92a37744d2db31cc@mail.gmail.com>
Message-ID: <CF790699-EFF6-44C8-AB91-8092AB3C83A1@gmail.com>

On Apr 11, 2010, at 1:50 PM, Sergi Pons Freixes wrote:
> Is there any elegant and fast way to join multiple unidimensional
> timeseries to create a new one? E.g:
> 
> ts1 = timeseries 1D with param1
> ts2 = timeseries 2D with param2
> (ts1 and ts2 have same Date array)
> 
> newts = join_ts(ts1, ts2) ---> newts= timeseries 2D with param2, param2

ts.stack should do the trick.

> 
> Or can this only be accomplished by creating a new timeseries, and
> manually setting the date array and all the data at the creation time?

Or you could try
ts.time_series(np.column_stack(*yourseries),dates=your_first_series.dates)
provided that you checked your series have the same dates and that there's no missing values. If the second condition is not met, use np.ma.column_stack instead (so that you don't lose the mask).





From spons at utm.csic.es  Sun Apr 11 17:34:57 2010
From: spons at utm.csic.es (Sergi Pons Freixes)
Date: Sun, 11 Apr 2010 23:34:57 +0200
Subject: [SciPy-User] Joining 1D Timeseries to create a nD Timeseries
In-Reply-To: <CF790699-EFF6-44C8-AB91-8092AB3C83A1@gmail.com>
References: <y2h2d88eb101004111050h555af1d2r92a37744d2db31cc@mail.gmail.com>
	<CF790699-EFF6-44C8-AB91-8092AB3C83A1@gmail.com>
Message-ID: <n2n2d88eb101004111434o411399b0if5229713f74ed3e2@mail.gmail.com>

On Sun, Apr 11, 2010 at 8:17 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> On Apr 11, 2010, at 1:50 PM, Sergi Pons Freixes wrote:
>> Is there any elegant and fast way to join multiple unidimensional
>> timeseries to create a new one? E.g:
>>
>> ts1 = timeseries 1D with param1
>> ts2 = timeseries 2D with param2
>> (ts1 and ts2 have same Date array)
>>
>> newts = join_ts(ts1, ts2) ---> newts= timeseries 2D with param2, param2
>
> ts.stack should do the trick.

Thank you Pierre. It works when both timeseries have the same shape
(as I stated).

>
>>
>> Or can this only be accomplished by creating a new timeseries, and
>> manually setting the date array and all the data at the creation time?
>
> Or you could try
> ts.time_series(np.column_stack(*yourseries),dates=your_first_series.dates)
> provided that you checked your series have the same dates and that there's no missing values. If the second condition is not met, use np.ma.column_stack instead (so that you don't lose the mask).

I prefer this approach, because it allows stacking timeseries with
different number of variables. In fact, my objective is to build a
multidimensional timeseries, stacking only a unidimensional timeseries
at a time.

The issue, now, is that if the timeseries dtypes have variable names
(e.g:, dtypes like ('foo', 'int32') instead of just 'int32'), the
np.column_stack crashes with a TypeError: expected a readable buffer
object. Maybe I should approach the building of the multidimensional
timeseries on a different way.


From stef.mientki at gmail.com  Sun Apr 11 18:22:08 2010
From: stef.mientki at gmail.com (Stef Mientki)
Date: Mon, 12 Apr 2010 00:22:08 +0200
Subject: [SciPy-User] visualizing scipy...
In-Reply-To: <s2o675c26741004101634qf478adfau9eded1fc7b5764d5@mail.gmail.com>
References: <s2o675c26741004101634qf478adfau9eded1fc7b5764d5@mail.gmail.com>
Message-ID: <4BC24B90.3080501@gmail.com>

On 11-04-2010 01:34, Anthony Palomba wrote:
> Hey folks, I am new to scipy, I would like to explore computational
> geometry
> with scipy. To do so, I need a way to visualize my experiments. What is
> the best thing to use to visualize 2d and 3d data in scipy?
>
If you want interactive 3D, take a look at VPython.
Here is quick and dirty demo, where I used VPython for some geometry demo.
http://mientki.ruhosting.nl/movies/geo_demo2.html

cheers,
Stef


From patrickmarshwx at gmail.com  Mon Apr 12 00:04:59 2010
From: patrickmarshwx at gmail.com (Patrick Marsh)
Date: Sun, 11 Apr 2010 23:04:59 -0500
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <hpmt3o$m4v$1@dough.gmane.org>
References: <hpkinj$pmp$1@dough.gmane.org>
	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>
	<hpla73$pgc$2@dough.gmane.org>
	<m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>
	<hpmn2c$vdo$1@dough.gmane.org> <4BBEF441.1060301@silveregg.co.jp> 
	<hpmt3o$m4v$1@dough.gmane.org>
Message-ID: <g2ge9962b831004112104k6b194f3fo3938d6c152d5f3d6@mail.gmail.com>

I know that my experience has been that I cannot pass multiple directories
to the library setting.  When I had everything is in a single directory, I
could get the setup script to find everything correctly, but the moment I
put multiple directories ( ':' as a separator) nothing was found.  The only
difference is I was using MKL for BLAS.

My simple hack was to use symbolic links to ensure everything could be found
in the same directory.

Patrick




On Fri, Apr 9, 2010 at 4:47 AM, John Reid <j.reid at mail.cryst.bbk.ac.uk>wrote:

> David wrote:
> > On 04/09/2010 05:04 PM, John Reid wrote:
> >
> >> That's what I find frustrating : python setup.py produces all these
> >> warnings and errors but they don't seem to relate directly to the
> >> sections in site.cfg.
> >
> > Don't look at the documentation or the output, they are useless at best,
> > when not outright confusing. You have to look at the sources to
> > understand what's going on,
>
> I tried that as well. I don't know the internals of numpy.distutils so I
> found that frustrating too.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Patrick Marsh
Ph.D. Student / NSSL Liaison to the HWT
School of Meteorology / University of Oklahoma
Cooperative Institute for Mesoscale Meteorological Studies
National Severe Storms Laboratory
http://www.patricktmarsh.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100411/aeaa4fe8/attachment.html>

From pmtri80 at gmail.com  Mon Apr 12 03:22:07 2010
From: pmtri80 at gmail.com (Minh-Tri Pham)
Date: Mon, 12 Apr 2010 08:22:07 +0100
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <g2ge9962b831004112104k6b194f3fo3938d6c152d5f3d6@mail.gmail.com>
References: <hpkinj$pmp$1@dough.gmane.org>	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>	<hpla73$pgc$2@dough.gmane.org>	<m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>	<hpmn2c$vdo$1@dough.gmane.org>
	<4BBEF441.1060301@silveregg.co.jp> <hpmt3o$m4v$1@dough.gmane.org>
	<g2ge9962b831004112104k6b194f3fo3938d6c152d5f3d6@mail.gmail.com>
Message-ID: <4BC2CA1F.6060206@gmail.com>

Hi everyone,

I've been using scipy for 4 years but I haven't posted a single message.
Sorry. :)

The following is my site.cfg configuration that works with GotoBLAS2 (at
least on my end):

[DEFAULT]
library_dirs = /usr/local/lib
include_dirs = /usr/local/include

[blas]
libraries = goto2_core2p-r1.13
language = fortran

[lapack]
libraries = goto2_core2p-r1.13
language = fortran

The trick is to use [blas] and [lapack] instead of [blas_opt] and [lapack_opt].

Best regards,
Minh-Tri

Patrick Marsh wrote:
> I know that my experience has been that I cannot pass multiple directories
> to the library setting.  When I had everything is in a single directory, I
> could get the setup script to find everything correctly, but the moment I
> put multiple directories ( ':' as a separator) nothing was found.  The only
> difference is I was using MKL for BLAS.
> 
> My simple hack was to use symbolic links to ensure everything could be found
> in the same directory.
> 
> Patrick
> 
> 
> 
> 
> On Fri, Apr 9, 2010 at 4:47 AM, John Reid <j.reid at mail.cryst.bbk.ac.uk>wrote:
> 
>> David wrote:
>>> On 04/09/2010 05:04 PM, John Reid wrote:
>>>
>>>> That's what I find frustrating : python setup.py produces all these
>>>> warnings and errors but they don't seem to relate directly to the
>>>> sections in site.cfg.
>>> Don't look at the documentation or the output, they are useless at best,
>>> when not outright confusing. You have to look at the sources to
>>> understand what's going on,
>> I tried that as well. I don't know the internals of numpy.distutils so I
>> found that frustrating too.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-- 
Dr. Minh-Tri Pham
Research Fellow
Surrey Space Centre, University of Surrey, Guildford, GU2 7XH, UK
Mob: +44 (0) 778 774 1089 | Tel: +44 (0) 148 368 2224


From apalomba at austin.rr.com  Mon Apr 12 10:58:25 2010
From: apalomba at austin.rr.com (Anthony Palomba)
Date: Mon, 12 Apr 2010 09:58:25 -0500
Subject: [SciPy-User] visualizing scipy...
In-Reply-To: <4BC24B90.3080501@gmail.com>
References: <s2o675c26741004101634qf478adfau9eded1fc7b5764d5@mail.gmail.com>
	<4BC24B90.3080501@gmail.com>
Message-ID: <z2p675c26741004120758s66239353ua3ebb5e8d392a1d9@mail.gmail.com>

Hey folks, thanks for the suggestions.
I will check them out.


-ap




On Sun, Apr 11, 2010 at 5:22 PM, Stef Mientki <stef.mientki at gmail.com>wrote:

> On 11-04-2010 01:34, Anthony Palomba wrote:
> > Hey folks, I am new to scipy, I would like to explore computational
> > geometry
> > with scipy. To do so, I need a way to visualize my experiments. What is
> > the best thing to use to visualize 2d and 3d data in scipy?
> >
> If you want interactive 3D, take a look at VPython.
> Here is quick and dirty demo, where I used VPython for some geometry demo.
> http://mientki.ruhosting.nl/movies/geo_demo2.html
>
> cheers,
> Stef
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100412/aacdfe11/attachment.html>

From j.reid at mail.cryst.bbk.ac.uk  Mon Apr 12 11:12:20 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Mon, 12 Apr 2010 16:12:20 +0100
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <4BC2CA1F.6060206@gmail.com>
References: <hpkinj$pmp$1@dough.gmane.org>	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>	<hpla73$pgc$2@dough.gmane.org>	<m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>	<hpmn2c$vdo$1@dough.gmane.org>	<4BBEF441.1060301@silveregg.co.jp>
	<hpmt3o$m4v$1@dough.gmane.org>	<g2ge9962b831004112104k6b194f3fo3938d6c152d5f3d6@mail.gmail.com>
	<4BC2CA1F.6060206@gmail.com>
Message-ID: <hpvd8l$q1n$1@dough.gmane.org>

Thanks for the help Patrick, Minh-Tri, Josef and David.

I can get scipy compiled but its still failing the tests. When I run 
with verbose=10 I can identify the test it is running:

<snip>....
Check that complex objects don't need to be completely aligned ... ok
test_decomp.test_lapack_misaligned(<function eig at 0xb38f7d14>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function eigvals at 0xb38f7df4>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function lu at 0xb38f7f44>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function lu_factor at 0xb38f7ed4>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function lu_solve at 0xb390b33c>, 
((array([[ -1.64308767e+289,   3.52371971e-294,   1.73394741e-255, ... ok
test_decomp.test_lapack_misaligned(<function solve at 0xb390b3ac>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ERROR
test_decomp.test_lapack_misaligned(<function svd at 0xb38f7f7c>, 
(array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.], ... ok
test_decomp.test_lapack_misaligned(<function svd at 0xb38f7f7c>, 
(array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9], ... ok
test_decomp.test_lapack_misaligned(<function svd at 0xb38f7f7c>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function svdvals at 0xb38f7fb4>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function svdvals at 0xb38f7fb4>, 
(array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
test_decomp.test_lapack_misaligned(<function cholesky at 0xb390b064>, 
(array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.], ...
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1210173776 (LWP 12789)]
0xb44a67f0 in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so




I get the following stacktrace from gdb:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1210566992 (LWP 12757)]
0xb44467f0 in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
(gdb) bt
#0  0xb44467f0 in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
#1  0xb48b7cdc in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
#2  0x00000002 in ?? ()
#3  0x09014af4 in ?? ()
#4  0xb411c6e3 in ddot_ () from /usr/local/lib/libgoto2_opteronp-r1.13.so
#5  0xb19827f2 in dpotf2_ () from 
/usr/local/lib/python2.5/site-packages/scipy/linalg/flapack.so
#6  0xb1983173 in dpotrf_ () from 
/usr/local/lib/python2.5/site-packages/scipy/linalg/flapack.so
#7  0xb18da338 in f2py_rout_flapack_dpotrf (capi_self=0xb409a488, 
capi_args=0xaf27bb6c,
     capi_keywds=0xaf28ea44, f2py_func=0xb1982d70 <dpotrf_>)
     at 
build/src.linux-i686-2.5/build/src.linux-i686-2.5/scipy/linalg/flapackmodule.c:13235
#8  0xb18f2951 in fortran_call (fp=0x2, arg=0xaf27bb6c, kw=0xaf28ea44)
     at build/src.linux-i686-2.5/fortranobject.c:346
#9  0x0805a427 in PyObject_Call (func=0x2, arg=0xaf27bb6c, 
kw=0xaf28ea44) at Objects/abstract.c:1861
#10 0x080c114a in PyEval_EvalFrameEx (f=0x8ff934c, throwflag=0) at 
Python/ceval.c:3784
#11 0x080c7265 in PyEval_EvalCodeEx (co=0xb4097c80, globals=0xb40a0b54, 
locals=0x0, args=0xaf27bab8,
     argcount=1, kws=0x8fbadd8, kwcount=1, defs=0xb40a6ef8, defcount=2, 
closure=0x0) at Python/ceval.c:2836
#12 0x0810e456 in function_call (func=0xb40ac064, arg=0xaf27baac, 
kw=0xaf28e68c)
     at Objects/funcobject.c:517
#13 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf27baac, 
kw=0xaf28e68c) at Objects/abstract.c:1861
#14 0x080c4168 in PyEval_EvalFrameEx (f=0x900ebc4, throwflag=0) at 
Python/ceval.c:3853
#15 0x080c7265 in PyEval_EvalCodeEx (co=0xb0dd6140, globals=0xb0dc2cec, 
locals=0x0, args=0xb0d01268,
     argcount=3, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#16 0x0810e381 in function_call (func=0xafcb24c4, arg=0xb0d0125c, 
kw=0x0) at Objects/funcobject.c:517
#17 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0d0125c, kw=0x0) at 
Objects/abstract.c:1861
#18 0x080c4168 in PyEval_EvalFrameEx (f=0x8d06514, throwflag=0) at 
Python/ceval.c:3853
#19 0x080c5d9c in PyEval_EvalFrameEx (f=0x8f612bc, throwflag=0) at 
Python/ceval.c:3659
#20 0x080c7265 in PyEval_EvalCodeEx (co=0xb7ae3920, globals=0xb7ae257c, 
locals=0x0, args=0xafc9f998,
     argcount=2, kws=0x8d17938, kwcount=0, defs=0xb7af0798, defcount=1, 
closure=0x0) at Python/ceval.c:2836
#21 0x0810e456 in function_call (func=0xb7aedbfc, arg=0xafc9f98c, 
kw=0xaf2b2f0c)
     at Objects/funcobject.c:517
#22 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9f98c, 
kw=0xaf2b2f0c) at Objects/abstract.c:1861
#23 0x080c4168 in PyEval_EvalFrameEx (f=0x8f61154, throwflag=0) at 
Python/ceval.c:3853
#24 0x080c7265 in PyEval_EvalCodeEx (co=0xb7ae3968, globals=0xb7ae257c, 
locals=0x0, args=0xafc9eef8,
     argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#25 0x0810e381 in function_call (func=0xb7aedc34, arg=0xafc9eeec, 
kw=0x0) at Objects/funcobject.c:517
#26 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9eeec, kw=0x0) at 
Objects/abstract.c:1861
#27 0x08060557 in instancemethod_call (func=0xb7aedc34, arg=0xafc9eeec, 
kw=0x0)
     at Objects/classobject.c:2519
#28 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf27b84c, kw=0x0) at 
Objects/abstract.c:1861
#29 0x0809a290 in slot_tp_call (self=0xaf27b5ec, args=0xaf27b84c, 
kwds=0x0) at Objects/typeobject.c:4714
#30 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf27b84c, kw=0x0) at 
Objects/abstract.c:1861
#31 0x080c114a in PyEval_EvalFrameEx (f=0x8f60fec, throwflag=0) at 
Python/ceval.c:3784
#32 0x080c5d9c in PyEval_EvalFrameEx (f=0x8f60e6c, throwflag=0) at 
Python/ceval.c:3659
#33 0x080c7265 in PyEval_EvalCodeEx (co=0xb736ab60, globals=0xb738a13c, 
locals=0x0, args=0xafcc96b8,
     argcount=2, kws=0x8ec68e0, kwcount=0, defs=0x0, defcount=0, 
closure=0x0) at Python/ceval.c:2836
#34 0x0810e456 in function_call (func=0xb7381994, arg=0xafcc96ac, 
kw=0xaf2b2dfc)
     at Objects/funcobject.c:517
#35 0x0805a427 in PyObject_Call (func=0x2, arg=0xafcc96ac, 
kw=0xaf2b2dfc) at Objects/abstract.c:1861
#36 0x080c4168 in PyEval_EvalFrameEx (f=0x8f60d04, throwflag=0) at 
Python/ceval.c:3853
#37 0x080c7265 in PyEval_EvalCodeEx (co=0xb736a7b8, globals=0xb738a13c, 
locals=0x0, args=0xafc9e298,
---Type <return> to continue, or q <return> to quit---
     argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#38 0x0810e381 in function_call (func=0xb738179c, arg=0xafc9e28c, 
kw=0x0) at Objects/funcobject.c:517
#39 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9e28c, kw=0x0) at 
Objects/abstract.c:1861
#40 0x08060557 in instancemethod_call (func=0xb738179c, arg=0xafc9e28c, 
kw=0x0)
     at Objects/classobject.c:2519
#41 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3d0cb8c, kw=0x0) at 
Objects/abstract.c:1861
#42 0x0809a290 in slot_tp_call (self=0xaf2b1e2c, args=0xb3d0cb8c, 
kwds=0x0) at Objects/typeobject.c:4714
#43 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3d0cb8c, kw=0x0) at 
Objects/abstract.c:1861
#44 0x080c114a in PyEval_EvalFrameEx (f=0x8ff72cc, throwflag=0) at 
Python/ceval.c:3784
#45 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
locals=0x0, args=0xaf3f3438,
     argcount=2, kws=0x8edd848, kwcount=0, defs=0x0, defcount=0, 
closure=0x0) at Python/ceval.c:2836
#46 0x0810e456 in function_call (func=0xb73944c4, arg=0xaf3f342c, 
kw=0xaf2b2934)
     at Objects/funcobject.c:517
#47 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf3f342c, 
kw=0xaf2b2934) at Objects/abstract.c:1861
#48 0x080c4168 in PyEval_EvalFrameEx (f=0x8fcc724, throwflag=0) at 
Python/ceval.c:3853
#49 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
locals=0x0, args=0xafc9ef18,
     argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#50 0x0810e381 in function_call (func=0xb739441c, arg=0xafc9ef0c, 
kw=0x0) at Objects/funcobject.c:517
#51 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9ef0c, kw=0x0) at 
Objects/abstract.c:1861
#52 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xafc9ef0c, 
kw=0x0)
     at Objects/classobject.c:2519
#53 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0dc644c, kw=0x0) at 
Objects/abstract.c:1861
#54 0x0809a290 in slot_tp_call (self=0xb0dc6dac, args=0xb0dc644c, 
kwds=0x0) at Objects/typeobject.c:4714
#55 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0dc644c, kw=0x0) at 
Objects/abstract.c:1861
#56 0x080c114a in PyEval_EvalFrameEx (f=0x8f76034, throwflag=0) at 
Python/ceval.c:3784
#57 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
locals=0x0, args=0xafc9fc78,
     argcount=2, kws=0x8d17910, kwcount=0, defs=0x0, defcount=0, 
closure=0x0) at Python/ceval.c:2836
#58 0x0810e456 in function_call (func=0xb73944c4, arg=0xafc9fc6c, 
kw=0xb0dc03e4)
     at Objects/funcobject.c:517
#59 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9fc6c, 
kw=0xb0dc03e4) at Objects/abstract.c:1861
#60 0x080c4168 in PyEval_EvalFrameEx (f=0x8f89d0c, throwflag=0) at 
Python/ceval.c:3853
#61 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
locals=0x0, args=0xafc9fc58,
     argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#62 0x0810e381 in function_call (func=0xb739441c, arg=0xafc9fc4c, 
kw=0x0) at Objects/funcobject.c:517
#63 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9fc4c, kw=0x0) at 
Objects/abstract.c:1861
#64 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xafc9fc4c, 
kw=0x0)
     at Objects/classobject.c:2519
#65 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0dc62ec, kw=0x0) at 
Objects/abstract.c:1861
#66 0x0809a290 in slot_tp_call (self=0xb0dc6e2c, args=0xb0dc62ec, 
kwds=0x0) at Objects/typeobject.c:4714
#67 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0dc62ec, kw=0x0) at 
Objects/abstract.c:1861
#68 0x080c114a in PyEval_EvalFrameEx (f=0x8f1f26c, throwflag=0) at 
Python/ceval.c:3784
#69 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
locals=0x0, args=0xb0e92138,
     argcount=2, kws=0x8e6e080, kwcount=0, defs=0x0, defcount=0, 
closure=0x0) at Python/ceval.c:2836
#70 0x0810e456 in function_call (func=0xb73944c4, arg=0xb0e9212c, 
kw=0xaf4552d4)
     at Objects/funcobject.c:517
#71 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0e9212c, 
kw=0xaf4552d4) at Objects/abstract.c:1861
#72 0x080c4168 in PyEval_EvalFrameEx (f=0x8d7ecbc, throwflag=0) at 
Python/ceval.c:3853
#73 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
locals=0x0, args=0xb0e7fa18,
     argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#74 0x0810e381 in function_call (func=0xb739441c, arg=0xb0e7fa0c, 
kw=0x0) at Objects/funcobject.c:517
---Type <return> to continue, or q <return> to quit---
#75 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0e7fa0c, kw=0x0) at 
Objects/abstract.c:1861
#76 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xb0e7fa0c, 
kw=0x0)
     at Objects/classobject.c:2519
#77 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0cf21cc, kw=0x0) at 
Objects/abstract.c:1861
#78 0x0809a290 in slot_tp_call (self=0xb3e95c4c, args=0xb0cf21cc, 
kwds=0x0) at Objects/typeobject.c:4714
#79 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0cf21cc, kw=0x0) at 
Objects/abstract.c:1861
#80 0x080c114a in PyEval_EvalFrameEx (f=0x8f6bacc, throwflag=0) at 
Python/ceval.c:3784
#81 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
locals=0x0, args=0xb0e7fa38,
     argcount=2, kws=0x8b765d0, kwcount=0, defs=0x0, defcount=0, 
closure=0x0) at Python/ceval.c:2836
#82 0x0810e456 in function_call (func=0xb73944c4, arg=0xb0e7fa2c, 
kw=0xafcbaf0c)
     at Objects/funcobject.c:517
#83 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0e7fa2c, 
kw=0xafcbaf0c) at Objects/abstract.c:1861
#84 0x080c4168 in PyEval_EvalFrameEx (f=0x8f5e7ac, throwflag=0) at 
Python/ceval.c:3853
#85 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
locals=0x0, args=0xb0e68e78,
     argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#86 0x0810e381 in function_call (func=0xb739441c, arg=0xb0e68e6c, 
kw=0x0) at Objects/funcobject.c:517
#87 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0e68e6c, kw=0x0) at 
Objects/abstract.c:1861
#88 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xb0e68e6c, 
kw=0x0)
     at Objects/classobject.c:2519
#89 0x0805a427 in PyObject_Call (func=0x2, arg=0xb11aa06c, kw=0x0) at 
Objects/abstract.c:1861
#90 0x0809a290 in slot_tp_call (self=0xb40addcc, args=0xb11aa06c, 
kwds=0x0) at Objects/typeobject.c:4714
#91 0x0805a427 in PyObject_Call (func=0x2, arg=0xb11aa06c, kw=0x0) at 
Objects/abstract.c:1861
#92 0x080c114a in PyEval_EvalFrameEx (f=0x8ce4d1c, throwflag=0) at 
Python/ceval.c:3784
#93 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
locals=0x0, args=0xb0ed1858,
     argcount=2, kws=0x89f4c90, kwcount=0, defs=0x0, defcount=0, 
closure=0x0) at Python/ceval.c:2836
#94 0x0810e456 in function_call (func=0xb73944c4, arg=0xb0ed184c, 
kw=0xb0e9c4f4)
     at Objects/funcobject.c:517
#95 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0ed184c, 
kw=0xb0e9c4f4) at Objects/abstract.c:1861
#96 0x080c4168 in PyEval_EvalFrameEx (f=0x8ce4bb4, throwflag=0) at 
Python/ceval.c:3853
#97 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
locals=0x0, args=0xb11db6f8,
     argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#98 0x0810e381 in function_call (func=0xb739441c, arg=0xb11db6ec, 
kw=0x0) at Objects/funcobject.c:517
#99 0x0805a427 in PyObject_Call (func=0x2, arg=0xb11db6ec, kw=0x0) at 
Objects/abstract.c:1861
#100 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xb11db6ec, 
kw=0x0)
     at Objects/classobject.c:2519
#101 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3ec55ac, kw=0x0) at 
Objects/abstract.c:1861
#102 0x0809a290 in slot_tp_call (self=0xb741c98c, args=0xb3ec55ac, 
kwds=0x0) at Objects/typeobject.c:4714
#103 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3ec55ac, kw=0x0) at 
Objects/abstract.c:1861
#104 0x080c114a in PyEval_EvalFrameEx (f=0x8ce4434, throwflag=0) at 
Python/ceval.c:3784
#105 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
locals=0x0, args=0xb3d347f8,
     argcount=2, kws=0x84e8b60, kwcount=0, defs=0x0, defcount=0, 
closure=0x0) at Python/ceval.c:2836
#106 0x0810e456 in function_call (func=0xb73944c4, arg=0xb3d347ec, 
kw=0xb0e9c2d4)
     at Objects/funcobject.c:517
#107 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3d347ec, 
kw=0xb0e9c2d4) at Objects/abstract.c:1861
#108 0x080c4168 in PyEval_EvalFrameEx (f=0x8aef9ac, throwflag=0) at 
Python/ceval.c:3853
#109 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
locals=0x0, args=0xb741c378,
     argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#110 0x0810e381 in function_call (func=0xb739441c, arg=0xb741c36c, 
kw=0x0) at Objects/funcobject.c:517
#111 0x0805a427 in PyObject_Call (func=0x2, arg=0xb741c36c, kw=0x0) at 
Objects/abstract.c:1861
#112 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xb741c36c, 
kw=0x0)
---Type <return> to continue, or q <return> to quit---
     at Objects/classobject.c:2519
#113 0x0805a427 in PyObject_Call (func=0x2, arg=0xb740dfac, kw=0x0) at 
Objects/abstract.c:1861
#114 0x0809a290 in slot_tp_call (self=0xb741ce2c, args=0xb740dfac, 
kwds=0x0) at Objects/typeobject.c:4714
#115 0x0805a427 in PyObject_Call (func=0x2, arg=0xb740dfac, kw=0x0) at 
Objects/abstract.c:1861
#116 0x080c114a in PyEval_EvalFrameEx (f=0x8c30d54, throwflag=0) at 
Python/ceval.c:3784
#117 0x080c5d9c in PyEval_EvalFrameEx (f=0x85274c4, throwflag=0) at 
Python/ceval.c:3659
#118 0x080c5d9c in PyEval_EvalFrameEx (f=0x8505154, throwflag=0) at 
Python/ceval.c:3659
#119 0x080c7265 in PyEval_EvalCodeEx (co=0xb7ae8de8, globals=0xb7ae257c, 
locals=0x0, args=0xb74c2b98,
     argcount=1, kws=0x84355c8, kwcount=5, defs=0xb7baee48, defcount=5, 
closure=0x0) at Python/ceval.c:2836
#120 0x0810e456 in function_call (func=0xb7af279c, arg=0xb74c2b8c, 
kw=0xb741668c)
     at Objects/funcobject.c:517
#121 0x0805a427 in PyObject_Call (func=0x2, arg=0xb74c2b8c, 
kw=0xb741668c) at Objects/abstract.c:1861
#122 0x08060557 in instancemethod_call (func=0xb7af279c, arg=0xb74c2b8c, 
kw=0xb741668c)
     at Objects/classobject.c:2519
#123 0x0805a427 in PyObject_Call (func=0x2, arg=0xb74c2b8c, 
kw=0xb741668c) at Objects/abstract.c:1861
#124 0x080c4168 in PyEval_EvalFrameEx (f=0x84901b4, throwflag=0) at 
Python/ceval.c:3853
#125 0x080c7265 in PyEval_EvalCodeEx (co=0xb76b32f0, globals=0xb76b924c, 
locals=0x0, args=0xb7cd82d8,
     argcount=1, kws=0x84d71a0, kwcount=3, defs=0xb738dfb0, defcount=11, 
closure=0x0) at Python/ceval.c:2836
#126 0x0810e456 in function_call (func=0xb739310c, arg=0xb7cd82cc, 
kw=0xb74163e4)
     at Objects/funcobject.c:517
#127 0x0805a427 in PyObject_Call (func=0x2, arg=0xb7cd82cc, 
kw=0xb74163e4) at Objects/abstract.c:1861
#128 0x08060557 in instancemethod_call (func=0xb739310c, arg=0xb7cd82cc, 
kw=0xb74163e4)
     at Objects/classobject.c:2519
#129 0x0805a427 in PyObject_Call (func=0x2, arg=0xb7d4302c, 
kw=0xb74163e4) at Objects/abstract.c:1861
#130 0x08099f1b in slot_tp_init (self=0xb741048c, args=0xb7d4302c, 
kwds=0xb74163e4)
     at Objects/typeobject.c:4943
#131 0x0809c9a3 in type_call (type=0x8510464, args=0xb7d4302c, 
kwds=0xb74163e4) at Objects/typeobject.c:436
#132 0x0805a427 in PyObject_Call (func=0x2, arg=0xb7d4302c, 
kw=0xb74163e4) at Objects/abstract.c:1861
#133 0x080c114a in PyEval_EvalFrameEx (f=0x83e2ff4, throwflag=0) at 
Python/ceval.c:3784
#134 0x080c7265 in PyEval_EvalCodeEx (co=0xb7b01020, globals=0xb7afb2d4, 
locals=0x0, args=0x81d5a5c,
     argcount=1, kws=0x81d5a60, kwcount=0, defs=0xb7b02128, defcount=5, 
closure=0x0) at Python/ceval.c:2836
#135 0x080c55f5 in PyEval_EvalFrameEx (f=0x81d5924, throwflag=0) at 
Python/ceval.c:3669
#136 0x080c7265 in PyEval_EvalCodeEx (co=0xb7d08d10, globals=0xb7d5cacc, 
locals=0xb7d5cacc, args=0x0,
     argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
at Python/ceval.c:2836
#137 0x080c72d7 in PyEval_EvalCode (co=0xb7d08d10, globals=0xb7d5cacc, 
locals=0xb7d5cacc)
     at Python/ceval.c:494
#138 0x080e6cd2 in PyRun_StringFlags (
     str=0x8166008 "import scipy as M; print M.__version__, M.__file__; 
M.test()\n", start=257,
     globals=0xb7d5cacc, locals=0xb7d5cacc, flags=0xbf9b4d68) at 
Python/pythonrun.c:1273
#139 0x080e6d95 in PyRun_SimpleStringFlags (
     command=0x8166008 "import scipy as M; print M.__version__, 
M.__file__; M.test()\n", flags=0xbf9b4d68)
     at Python/pythonrun.c:900
#140 0x0805687e in Py_Main (argc=1, argv=0xbf9b4e34) at Modules/main.c:512
#141 0x08056382 in main (argc=Cannot access memory at address 0x2
) at ./Modules/python.c:23


I'm not sure how to debug this. GotoBLAS2 passes some tests when I make 
it. However when I run its lapack-test target it fails. Can anyone here 
tell me anything I can do to check if the problem is with my build of 
scipy or GotoBLAS?

Thanks,
John.



From warren.weckesser at enthought.com  Mon Apr 12 13:02:14 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Mon, 12 Apr 2010 12:02:14 -0500
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <hpvd8l$q1n$1@dough.gmane.org>
References: <hpkinj$pmp$1@dough.gmane.org>	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>	<hpla73$pgc$2@dough.gmane.org>	<m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>	<hpmn2c$vdo$1@dough.gmane.org>	<4BBEF441.1060301@silveregg.co.jp>	<hpmt3o$m4v$1@dough.gmane.org>	<g2ge9962b831004112104k6b194f3fo3938d6c152d5f3d6@mail.gmail.com>	<4BC2CA1F.6060206@gmail.com>
	<hpvd8l$q1n$1@dough.gmane.org>
Message-ID: <4BC35216.8060100@enthought.com>

John Reid wrote:
> Thanks for the help Patrick, Minh-Tri, Josef and David.
>
> I can get scipy compiled but its still failing the tests. When I run 
> with verbose=10 I can identify the test it is running:
>
> <snip>....
> Check that complex objects don't need to be completely aligned ... ok
> test_decomp.test_lapack_misaligned(<function eig at 0xb38f7d14>, 
> (array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
> test_decomp.test_lapack_misaligned(<function eigvals at 0xb38f7df4>, 
> (array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
> test_decomp.test_lapack_misaligned(<function lu at 0xb38f7f44>, 
> (array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
> test_decomp.test_lapack_misaligned(<function lu_factor at 0xb38f7ed4>, 
> (array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
> test_decomp.test_lapack_misaligned(<function lu_solve at 0xb390b33c>, 
> ((array([[ -1.64308767e+289,   3.52371971e-294,   1.73394741e-255, ... ok
> test_decomp.test_lapack_misaligned(<function solve at 0xb390b3ac>, 
> (array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ERROR
>   

I don't know about the other errors, but I just created a ticket for the 
error that occurs in the alignment test of the 'solve' function in linalg:

    http://projects.scipy.org/scipy/ticket/1152

Warren

> test_decomp.test_lapack_misaligned(<function svd at 0xb38f7f7c>, 
> (array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.], ... ok
> test_decomp.test_lapack_misaligned(<function svd at 0xb38f7f7c>, 
> (array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9], ... ok
> test_decomp.test_lapack_misaligned(<function svd at 0xb38f7f7c>, 
> (array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
> test_decomp.test_lapack_misaligned(<function svdvals at 0xb38f7fb4>, 
> (array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
> test_decomp.test_lapack_misaligned(<function svdvals at 0xb38f7fb4>, 
> (array([[  1.73394741e-255,   8.18880997e-217,   4.02522535e-178, ... ok
> test_decomp.test_lapack_misaligned(<function cholesky at 0xb390b064>, 
> (array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.], ...
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread -1210173776 (LWP 12789)]
> 0xb44a67f0 in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
>
>
>
>
> I get the following stacktrace from gdb:
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread -1210566992 (LWP 12757)]
> 0xb44467f0 in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
> (gdb) bt
> #0  0xb44467f0 in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
> #1  0xb48b7cdc in chroot () from /usr/local/lib/libgoto2_opteronp-r1.13.so
> #2  0x00000002 in ?? ()
> #3  0x09014af4 in ?? ()
> #4  0xb411c6e3 in ddot_ () from /usr/local/lib/libgoto2_opteronp-r1.13.so
> #5  0xb19827f2 in dpotf2_ () from 
> /usr/local/lib/python2.5/site-packages/scipy/linalg/flapack.so
> #6  0xb1983173 in dpotrf_ () from 
> /usr/local/lib/python2.5/site-packages/scipy/linalg/flapack.so
> #7  0xb18da338 in f2py_rout_flapack_dpotrf (capi_self=0xb409a488, 
> capi_args=0xaf27bb6c,
>      capi_keywds=0xaf28ea44, f2py_func=0xb1982d70 <dpotrf_>)
>      at 
> build/src.linux-i686-2.5/build/src.linux-i686-2.5/scipy/linalg/flapackmodule.c:13235
> #8  0xb18f2951 in fortran_call (fp=0x2, arg=0xaf27bb6c, kw=0xaf28ea44)
>      at build/src.linux-i686-2.5/fortranobject.c:346
> #9  0x0805a427 in PyObject_Call (func=0x2, arg=0xaf27bb6c, 
> kw=0xaf28ea44) at Objects/abstract.c:1861
> #10 0x080c114a in PyEval_EvalFrameEx (f=0x8ff934c, throwflag=0) at 
> Python/ceval.c:3784
> #11 0x080c7265 in PyEval_EvalCodeEx (co=0xb4097c80, globals=0xb40a0b54, 
> locals=0x0, args=0xaf27bab8,
>      argcount=1, kws=0x8fbadd8, kwcount=1, defs=0xb40a6ef8, defcount=2, 
> closure=0x0) at Python/ceval.c:2836
> #12 0x0810e456 in function_call (func=0xb40ac064, arg=0xaf27baac, 
> kw=0xaf28e68c)
>      at Objects/funcobject.c:517
> #13 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf27baac, 
> kw=0xaf28e68c) at Objects/abstract.c:1861
> #14 0x080c4168 in PyEval_EvalFrameEx (f=0x900ebc4, throwflag=0) at 
> Python/ceval.c:3853
> #15 0x080c7265 in PyEval_EvalCodeEx (co=0xb0dd6140, globals=0xb0dc2cec, 
> locals=0x0, args=0xb0d01268,
>      argcount=3, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #16 0x0810e381 in function_call (func=0xafcb24c4, arg=0xb0d0125c, 
> kw=0x0) at Objects/funcobject.c:517
> #17 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0d0125c, kw=0x0) at 
> Objects/abstract.c:1861
> #18 0x080c4168 in PyEval_EvalFrameEx (f=0x8d06514, throwflag=0) at 
> Python/ceval.c:3853
> #19 0x080c5d9c in PyEval_EvalFrameEx (f=0x8f612bc, throwflag=0) at 
> Python/ceval.c:3659
> #20 0x080c7265 in PyEval_EvalCodeEx (co=0xb7ae3920, globals=0xb7ae257c, 
> locals=0x0, args=0xafc9f998,
>      argcount=2, kws=0x8d17938, kwcount=0, defs=0xb7af0798, defcount=1, 
> closure=0x0) at Python/ceval.c:2836
> #21 0x0810e456 in function_call (func=0xb7aedbfc, arg=0xafc9f98c, 
> kw=0xaf2b2f0c)
>      at Objects/funcobject.c:517
> #22 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9f98c, 
> kw=0xaf2b2f0c) at Objects/abstract.c:1861
> #23 0x080c4168 in PyEval_EvalFrameEx (f=0x8f61154, throwflag=0) at 
> Python/ceval.c:3853
> #24 0x080c7265 in PyEval_EvalCodeEx (co=0xb7ae3968, globals=0xb7ae257c, 
> locals=0x0, args=0xafc9eef8,
>      argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #25 0x0810e381 in function_call (func=0xb7aedc34, arg=0xafc9eeec, 
> kw=0x0) at Objects/funcobject.c:517
> #26 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9eeec, kw=0x0) at 
> Objects/abstract.c:1861
> #27 0x08060557 in instancemethod_call (func=0xb7aedc34, arg=0xafc9eeec, 
> kw=0x0)
>      at Objects/classobject.c:2519
> #28 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf27b84c, kw=0x0) at 
> Objects/abstract.c:1861
> #29 0x0809a290 in slot_tp_call (self=0xaf27b5ec, args=0xaf27b84c, 
> kwds=0x0) at Objects/typeobject.c:4714
> #30 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf27b84c, kw=0x0) at 
> Objects/abstract.c:1861
> #31 0x080c114a in PyEval_EvalFrameEx (f=0x8f60fec, throwflag=0) at 
> Python/ceval.c:3784
> #32 0x080c5d9c in PyEval_EvalFrameEx (f=0x8f60e6c, throwflag=0) at 
> Python/ceval.c:3659
> #33 0x080c7265 in PyEval_EvalCodeEx (co=0xb736ab60, globals=0xb738a13c, 
> locals=0x0, args=0xafcc96b8,
>      argcount=2, kws=0x8ec68e0, kwcount=0, defs=0x0, defcount=0, 
> closure=0x0) at Python/ceval.c:2836
> #34 0x0810e456 in function_call (func=0xb7381994, arg=0xafcc96ac, 
> kw=0xaf2b2dfc)
>      at Objects/funcobject.c:517
> #35 0x0805a427 in PyObject_Call (func=0x2, arg=0xafcc96ac, 
> kw=0xaf2b2dfc) at Objects/abstract.c:1861
> #36 0x080c4168 in PyEval_EvalFrameEx (f=0x8f60d04, throwflag=0) at 
> Python/ceval.c:3853
> #37 0x080c7265 in PyEval_EvalCodeEx (co=0xb736a7b8, globals=0xb738a13c, 
> locals=0x0, args=0xafc9e298,
> ---Type <return> to continue, or q <return> to quit---
>      argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #38 0x0810e381 in function_call (func=0xb738179c, arg=0xafc9e28c, 
> kw=0x0) at Objects/funcobject.c:517
> #39 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9e28c, kw=0x0) at 
> Objects/abstract.c:1861
> #40 0x08060557 in instancemethod_call (func=0xb738179c, arg=0xafc9e28c, 
> kw=0x0)
>      at Objects/classobject.c:2519
> #41 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3d0cb8c, kw=0x0) at 
> Objects/abstract.c:1861
> #42 0x0809a290 in slot_tp_call (self=0xaf2b1e2c, args=0xb3d0cb8c, 
> kwds=0x0) at Objects/typeobject.c:4714
> #43 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3d0cb8c, kw=0x0) at 
> Objects/abstract.c:1861
> #44 0x080c114a in PyEval_EvalFrameEx (f=0x8ff72cc, throwflag=0) at 
> Python/ceval.c:3784
> #45 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
> locals=0x0, args=0xaf3f3438,
>      argcount=2, kws=0x8edd848, kwcount=0, defs=0x0, defcount=0, 
> closure=0x0) at Python/ceval.c:2836
> #46 0x0810e456 in function_call (func=0xb73944c4, arg=0xaf3f342c, 
> kw=0xaf2b2934)
>      at Objects/funcobject.c:517
> #47 0x0805a427 in PyObject_Call (func=0x2, arg=0xaf3f342c, 
> kw=0xaf2b2934) at Objects/abstract.c:1861
> #48 0x080c4168 in PyEval_EvalFrameEx (f=0x8fcc724, throwflag=0) at 
> Python/ceval.c:3853
> #49 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
> locals=0x0, args=0xafc9ef18,
>      argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #50 0x0810e381 in function_call (func=0xb739441c, arg=0xafc9ef0c, 
> kw=0x0) at Objects/funcobject.c:517
> #51 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9ef0c, kw=0x0) at 
> Objects/abstract.c:1861
> #52 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xafc9ef0c, 
> kw=0x0)
>      at Objects/classobject.c:2519
> #53 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0dc644c, kw=0x0) at 
> Objects/abstract.c:1861
> #54 0x0809a290 in slot_tp_call (self=0xb0dc6dac, args=0xb0dc644c, 
> kwds=0x0) at Objects/typeobject.c:4714
> #55 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0dc644c, kw=0x0) at 
> Objects/abstract.c:1861
> #56 0x080c114a in PyEval_EvalFrameEx (f=0x8f76034, throwflag=0) at 
> Python/ceval.c:3784
> #57 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
> locals=0x0, args=0xafc9fc78,
>      argcount=2, kws=0x8d17910, kwcount=0, defs=0x0, defcount=0, 
> closure=0x0) at Python/ceval.c:2836
> #58 0x0810e456 in function_call (func=0xb73944c4, arg=0xafc9fc6c, 
> kw=0xb0dc03e4)
>      at Objects/funcobject.c:517
> #59 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9fc6c, 
> kw=0xb0dc03e4) at Objects/abstract.c:1861
> #60 0x080c4168 in PyEval_EvalFrameEx (f=0x8f89d0c, throwflag=0) at 
> Python/ceval.c:3853
> #61 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
> locals=0x0, args=0xafc9fc58,
>      argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #62 0x0810e381 in function_call (func=0xb739441c, arg=0xafc9fc4c, 
> kw=0x0) at Objects/funcobject.c:517
> #63 0x0805a427 in PyObject_Call (func=0x2, arg=0xafc9fc4c, kw=0x0) at 
> Objects/abstract.c:1861
> #64 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xafc9fc4c, 
> kw=0x0)
>      at Objects/classobject.c:2519
> #65 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0dc62ec, kw=0x0) at 
> Objects/abstract.c:1861
> #66 0x0809a290 in slot_tp_call (self=0xb0dc6e2c, args=0xb0dc62ec, 
> kwds=0x0) at Objects/typeobject.c:4714
> #67 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0dc62ec, kw=0x0) at 
> Objects/abstract.c:1861
> #68 0x080c114a in PyEval_EvalFrameEx (f=0x8f1f26c, throwflag=0) at 
> Python/ceval.c:3784
> #69 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
> locals=0x0, args=0xb0e92138,
>      argcount=2, kws=0x8e6e080, kwcount=0, defs=0x0, defcount=0, 
> closure=0x0) at Python/ceval.c:2836
> #70 0x0810e456 in function_call (func=0xb73944c4, arg=0xb0e9212c, 
> kw=0xaf4552d4)
>      at Objects/funcobject.c:517
> #71 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0e9212c, 
> kw=0xaf4552d4) at Objects/abstract.c:1861
> #72 0x080c4168 in PyEval_EvalFrameEx (f=0x8d7ecbc, throwflag=0) at 
> Python/ceval.c:3853
> #73 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
> locals=0x0, args=0xb0e7fa18,
>      argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #74 0x0810e381 in function_call (func=0xb739441c, arg=0xb0e7fa0c, 
> kw=0x0) at Objects/funcobject.c:517
> ---Type <return> to continue, or q <return> to quit---
> #75 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0e7fa0c, kw=0x0) at 
> Objects/abstract.c:1861
> #76 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xb0e7fa0c, 
> kw=0x0)
>      at Objects/classobject.c:2519
> #77 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0cf21cc, kw=0x0) at 
> Objects/abstract.c:1861
> #78 0x0809a290 in slot_tp_call (self=0xb3e95c4c, args=0xb0cf21cc, 
> kwds=0x0) at Objects/typeobject.c:4714
> #79 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0cf21cc, kw=0x0) at 
> Objects/abstract.c:1861
> #80 0x080c114a in PyEval_EvalFrameEx (f=0x8f6bacc, throwflag=0) at 
> Python/ceval.c:3784
> #81 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
> locals=0x0, args=0xb0e7fa38,
>      argcount=2, kws=0x8b765d0, kwcount=0, defs=0x0, defcount=0, 
> closure=0x0) at Python/ceval.c:2836
> #82 0x0810e456 in function_call (func=0xb73944c4, arg=0xb0e7fa2c, 
> kw=0xafcbaf0c)
>      at Objects/funcobject.c:517
> #83 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0e7fa2c, 
> kw=0xafcbaf0c) at Objects/abstract.c:1861
> #84 0x080c4168 in PyEval_EvalFrameEx (f=0x8f5e7ac, throwflag=0) at 
> Python/ceval.c:3853
> #85 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
> locals=0x0, args=0xb0e68e78,
>      argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #86 0x0810e381 in function_call (func=0xb739441c, arg=0xb0e68e6c, 
> kw=0x0) at Objects/funcobject.c:517
> #87 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0e68e6c, kw=0x0) at 
> Objects/abstract.c:1861
> #88 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xb0e68e6c, 
> kw=0x0)
>      at Objects/classobject.c:2519
> #89 0x0805a427 in PyObject_Call (func=0x2, arg=0xb11aa06c, kw=0x0) at 
> Objects/abstract.c:1861
> #90 0x0809a290 in slot_tp_call (self=0xb40addcc, args=0xb11aa06c, 
> kwds=0x0) at Objects/typeobject.c:4714
> #91 0x0805a427 in PyObject_Call (func=0x2, arg=0xb11aa06c, kw=0x0) at 
> Objects/abstract.c:1861
> #92 0x080c114a in PyEval_EvalFrameEx (f=0x8ce4d1c, throwflag=0) at 
> Python/ceval.c:3784
> #93 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
> locals=0x0, args=0xb0ed1858,
>      argcount=2, kws=0x89f4c90, kwcount=0, defs=0x0, defcount=0, 
> closure=0x0) at Python/ceval.c:2836
> #94 0x0810e456 in function_call (func=0xb73944c4, arg=0xb0ed184c, 
> kw=0xb0e9c4f4)
>      at Objects/funcobject.c:517
> #95 0x0805a427 in PyObject_Call (func=0x2, arg=0xb0ed184c, 
> kw=0xb0e9c4f4) at Objects/abstract.c:1861
> #96 0x080c4168 in PyEval_EvalFrameEx (f=0x8ce4bb4, throwflag=0) at 
> Python/ceval.c:3853
> #97 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
> locals=0x0, args=0xb11db6f8,
>      argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #98 0x0810e381 in function_call (func=0xb739441c, arg=0xb11db6ec, 
> kw=0x0) at Objects/funcobject.c:517
> #99 0x0805a427 in PyObject_Call (func=0x2, arg=0xb11db6ec, kw=0x0) at 
> Objects/abstract.c:1861
> #100 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xb11db6ec, 
> kw=0x0)
>      at Objects/classobject.c:2519
> #101 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3ec55ac, kw=0x0) at 
> Objects/abstract.c:1861
> #102 0x0809a290 in slot_tp_call (self=0xb741c98c, args=0xb3ec55ac, 
> kwds=0x0) at Objects/typeobject.c:4714
> #103 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3ec55ac, kw=0x0) at 
> Objects/abstract.c:1861
> #104 0x080c114a in PyEval_EvalFrameEx (f=0x8ce4434, throwflag=0) at 
> Python/ceval.c:3784
> #105 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383f50, globals=0xb737bbdc, 
> locals=0x0, args=0xb3d347f8,
>      argcount=2, kws=0x84e8b60, kwcount=0, defs=0x0, defcount=0, 
> closure=0x0) at Python/ceval.c:2836
> #106 0x0810e456 in function_call (func=0xb73944c4, arg=0xb3d347ec, 
> kw=0xb0e9c2d4)
>      at Objects/funcobject.c:517
> #107 0x0805a427 in PyObject_Call (func=0x2, arg=0xb3d347ec, 
> kw=0xb0e9c2d4) at Objects/abstract.c:1861
> #108 0x080c4168 in PyEval_EvalFrameEx (f=0x8aef9ac, throwflag=0) at 
> Python/ceval.c:3853
> #109 0x080c7265 in PyEval_EvalCodeEx (co=0xb7383e30, globals=0xb737bbdc, 
> locals=0x0, args=0xb741c378,
>      argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #110 0x0810e381 in function_call (func=0xb739441c, arg=0xb741c36c, 
> kw=0x0) at Objects/funcobject.c:517
> #111 0x0805a427 in PyObject_Call (func=0x2, arg=0xb741c36c, kw=0x0) at 
> Objects/abstract.c:1861
> #112 0x08060557 in instancemethod_call (func=0xb739441c, arg=0xb741c36c, 
> kw=0x0)
> ---Type <return> to continue, or q <return> to quit---
>      at Objects/classobject.c:2519
> #113 0x0805a427 in PyObject_Call (func=0x2, arg=0xb740dfac, kw=0x0) at 
> Objects/abstract.c:1861
> #114 0x0809a290 in slot_tp_call (self=0xb741ce2c, args=0xb740dfac, 
> kwds=0x0) at Objects/typeobject.c:4714
> #115 0x0805a427 in PyObject_Call (func=0x2, arg=0xb740dfac, kw=0x0) at 
> Objects/abstract.c:1861
> #116 0x080c114a in PyEval_EvalFrameEx (f=0x8c30d54, throwflag=0) at 
> Python/ceval.c:3784
> #117 0x080c5d9c in PyEval_EvalFrameEx (f=0x85274c4, throwflag=0) at 
> Python/ceval.c:3659
> #118 0x080c5d9c in PyEval_EvalFrameEx (f=0x8505154, throwflag=0) at 
> Python/ceval.c:3659
> #119 0x080c7265 in PyEval_EvalCodeEx (co=0xb7ae8de8, globals=0xb7ae257c, 
> locals=0x0, args=0xb74c2b98,
>      argcount=1, kws=0x84355c8, kwcount=5, defs=0xb7baee48, defcount=5, 
> closure=0x0) at Python/ceval.c:2836
> #120 0x0810e456 in function_call (func=0xb7af279c, arg=0xb74c2b8c, 
> kw=0xb741668c)
>      at Objects/funcobject.c:517
> #121 0x0805a427 in PyObject_Call (func=0x2, arg=0xb74c2b8c, 
> kw=0xb741668c) at Objects/abstract.c:1861
> #122 0x08060557 in instancemethod_call (func=0xb7af279c, arg=0xb74c2b8c, 
> kw=0xb741668c)
>      at Objects/classobject.c:2519
> #123 0x0805a427 in PyObject_Call (func=0x2, arg=0xb74c2b8c, 
> kw=0xb741668c) at Objects/abstract.c:1861
> #124 0x080c4168 in PyEval_EvalFrameEx (f=0x84901b4, throwflag=0) at 
> Python/ceval.c:3853
> #125 0x080c7265 in PyEval_EvalCodeEx (co=0xb76b32f0, globals=0xb76b924c, 
> locals=0x0, args=0xb7cd82d8,
>      argcount=1, kws=0x84d71a0, kwcount=3, defs=0xb738dfb0, defcount=11, 
> closure=0x0) at Python/ceval.c:2836
> #126 0x0810e456 in function_call (func=0xb739310c, arg=0xb7cd82cc, 
> kw=0xb74163e4)
>      at Objects/funcobject.c:517
> #127 0x0805a427 in PyObject_Call (func=0x2, arg=0xb7cd82cc, 
> kw=0xb74163e4) at Objects/abstract.c:1861
> #128 0x08060557 in instancemethod_call (func=0xb739310c, arg=0xb7cd82cc, 
> kw=0xb74163e4)
>      at Objects/classobject.c:2519
> #129 0x0805a427 in PyObject_Call (func=0x2, arg=0xb7d4302c, 
> kw=0xb74163e4) at Objects/abstract.c:1861
> #130 0x08099f1b in slot_tp_init (self=0xb741048c, args=0xb7d4302c, 
> kwds=0xb74163e4)
>      at Objects/typeobject.c:4943
> #131 0x0809c9a3 in type_call (type=0x8510464, args=0xb7d4302c, 
> kwds=0xb74163e4) at Objects/typeobject.c:436
> #132 0x0805a427 in PyObject_Call (func=0x2, arg=0xb7d4302c, 
> kw=0xb74163e4) at Objects/abstract.c:1861
> #133 0x080c114a in PyEval_EvalFrameEx (f=0x83e2ff4, throwflag=0) at 
> Python/ceval.c:3784
> #134 0x080c7265 in PyEval_EvalCodeEx (co=0xb7b01020, globals=0xb7afb2d4, 
> locals=0x0, args=0x81d5a5c,
>      argcount=1, kws=0x81d5a60, kwcount=0, defs=0xb7b02128, defcount=5, 
> closure=0x0) at Python/ceval.c:2836
> #135 0x080c55f5 in PyEval_EvalFrameEx (f=0x81d5924, throwflag=0) at 
> Python/ceval.c:3669
> #136 0x080c7265 in PyEval_EvalCodeEx (co=0xb7d08d10, globals=0xb7d5cacc, 
> locals=0xb7d5cacc, args=0x0,
>      argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) 
> at Python/ceval.c:2836
> #137 0x080c72d7 in PyEval_EvalCode (co=0xb7d08d10, globals=0xb7d5cacc, 
> locals=0xb7d5cacc)
>      at Python/ceval.c:494
> #138 0x080e6cd2 in PyRun_StringFlags (
>      str=0x8166008 "import scipy as M; print M.__version__, M.__file__; 
> M.test()\n", start=257,
>      globals=0xb7d5cacc, locals=0xb7d5cacc, flags=0xbf9b4d68) at 
> Python/pythonrun.c:1273
> #139 0x080e6d95 in PyRun_SimpleStringFlags (
>      command=0x8166008 "import scipy as M; print M.__version__, 
> M.__file__; M.test()\n", flags=0xbf9b4d68)
>      at Python/pythonrun.c:900
> #140 0x0805687e in Py_Main (argc=1, argv=0xbf9b4e34) at Modules/main.c:512
> #141 0x08056382 in main (argc=Cannot access memory at address 0x2
> ) at ./Modules/python.c:23
>
>
> I'm not sure how to debug this. GotoBLAS2 passes some tests when I make 
> it. However when I run its lapack-test target it fails. Can anyone here 
> tell me anything I can do to check if the problem is with my build of 
> scipy or GotoBLAS?
>
> Thanks,
> John.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From gokhansever at gmail.com  Mon Apr 12 13:14:51 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Mon, 12 Apr 2010 12:14:51 -0500
Subject: [SciPy-User] Fitting tools
Message-ID: <p2u49d6b3501004121014p382c1712k72ba68fb91bececd@mail.gmail.com>

Hello,

Are there any compact curve/distribution fitting tools around similar to the
commercial TableCurve 2D [1] and EasyFit [2] products?

[1] http://www.sigmaplot.com/products/tablecurve2d/tablecurve2d.php
[2] http://www.mathwave.com/easyfit-distribution-fitting.html

What would you suggest as alternatives in SciPy and open-source?

Thanks.

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100412/7b9cb14c/attachment.html>

From josef.pktd at gmail.com  Mon Apr 12 13:34:07 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 12 Apr 2010 13:34:07 -0400
Subject: [SciPy-User] Fitting tools
In-Reply-To: <p2u49d6b3501004121014p382c1712k72ba68fb91bececd@mail.gmail.com>
References: <p2u49d6b3501004121014p382c1712k72ba68fb91bececd@mail.gmail.com>
Message-ID: <w2k1cd32cbb1004121034q58fd3a9seea52b1a1505e399@mail.gmail.com>

On Mon, Apr 12, 2010 at 1:14 PM, G?khan Sever <gokhansever at gmail.com> wrote:
> Hello,
>
> Are there any compact curve/distribution fitting tools around similar to the
> commercial TableCurve 2D [1] and EasyFit [2] products?
>
> [1] http://www.sigmaplot.com/products/tablecurve2d/tablecurve2d.php

I think astrogui, or astropysics might have something like this,
Travis showcased it briefly in the stats webinar. There is also an
online curve fitting (zunzun ?) with (I think) BSD licensed source
code that might be useful.

> [2] http://www.mathwave.com/easyfit-distribution-fitting.html

I looked at easyfit a year or more ago, and wrote a script that tried
to do similar fitting for all distributions in scipy.stats. It worked
ok for distributions with unlimited support, but there are still the
problems with estimating distributions with a finite bound in the
support.
I used kstest as distance measure, but never rewrote Anderson-Darling
to function as generic distance measure as (if I remember correctly)
easyfit does.

Josef

>
> What would you suggest as alternatives in SciPy and open-source?

>
> Thanks.
>
> --
> G?khan
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From jorgesmbox-ml at yahoo.es  Mon Apr 12 12:00:24 2010
From: jorgesmbox-ml at yahoo.es (Jorge Scandaliaris)
Date: Mon, 12 Apr 2010 16:00:24 +0000 (UTC)
Subject: [SciPy-User] =?utf-8?q?Question_about_gaussian=5Fkde?=
Message-ID: <loom.20100412T174309-768@post.gmane.org>

Hi,
I am using gaussian_kde mainly to visualize the distribution of some 2D
measurements. It works ok but there's something I don't understand. When I
evaluate the estimated pdf, the peaks have values larger than one. If I use
integrate_box(), however, the results seem correct. How should I normalize the
values obtained from evaluate? Dividing by the number of datapoints?

Thanks,

Jorge



From josef.pktd at gmail.com  Mon Apr 12 13:48:01 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 12 Apr 2010 13:48:01 -0400
Subject: [SciPy-User] Question about gaussian_kde
In-Reply-To: <loom.20100412T174309-768@post.gmane.org>
References: <loom.20100412T174309-768@post.gmane.org>
Message-ID: <z2o1cd32cbb1004121048u256f6d35t70e641634b6d6fc9@mail.gmail.com>

On Mon, Apr 12, 2010 at 12:00 PM, Jorge Scandaliaris
<jorgesmbox-ml at yahoo.es> wrote:
> Hi,
> I am using gaussian_kde mainly to visualize the distribution of some 2D
> measurements. It works ok but there's something I don't understand. When I
> evaluate the estimated pdf, the peaks have values larger than one. If I use
> integrate_box(), however, the results seem correct. How should I normalize the
> values obtained from evaluate? Dividing by the number of datapoints?

pdf is a density function on continuous support, there is no
requirement that it has any bounded value.
It only needs to integrate to one, and be non-negative

Only the pmf of discrete distributions is required to  have values
smaller than one.

Josef

>
> Thanks,
>
> Jorge
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From robert.kern at gmail.com  Mon Apr 12 13:48:01 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 12 Apr 2010 12:48:01 -0500
Subject: [SciPy-User] Question about gaussian_kde
In-Reply-To: <loom.20100412T174309-768@post.gmane.org>
References: <loom.20100412T174309-768@post.gmane.org>
Message-ID: <o2k3d375d731004121048h657c6f69l1965d4253427c909@mail.gmail.com>

On Mon, Apr 12, 2010 at 11:00, Jorge Scandaliaris
<jorgesmbox-ml at yahoo.es> wrote:
> Hi,
> I am using gaussian_kde mainly to visualize the distribution of some 2D
> measurements. It works ok but there's something I don't understand. When I
> evaluate the estimated pdf, the peaks have values larger than one. If I use
> integrate_box(), however, the results seem correct. How should I normalize the
> values obtained from evaluate? Dividing by the number of datapoints?

The values are correct. Remember that this is a probability *density*.
All that means is that the integral over the domain is equal to 1. The
value at any point is not a probability itself; it just needs to be
non-negative. For example:

In [1]: from scipy import stats

In [2]: stats.norm.pdf(0.0, scale=0.01)
Out[2]: 39.894228040143268


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gokhansever at gmail.com  Mon Apr 12 15:54:15 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Mon, 12 Apr 2010 14:54:15 -0500
Subject: [SciPy-User] Fitting tools
In-Reply-To: <w2k1cd32cbb1004121034q58fd3a9seea52b1a1505e399@mail.gmail.com>
References: <p2u49d6b3501004121014p382c1712k72ba68fb91bececd@mail.gmail.com>
	<w2k1cd32cbb1004121034q58fd3a9seea52b1a1505e399@mail.gmail.com>
Message-ID: <v2l49d6b3501004121254ic2929a9fn9364bf6e03f3030b@mail.gmail.com>

On Mon, Apr 12, 2010 at 12:34 PM, <josef.pktd at gmail.com> wrote:

> On Mon, Apr 12, 2010 at 1:14 PM, G?khan Sever <gokhansever at gmail.com>
> wrote:
> > Hello,
> >
> > Are there any compact curve/distribution fitting tools around similar to
> the
> > commercial TableCurve 2D [1] and EasyFit [2] products?
> >
> > [1] http://www.sigmaplot.com/products/tablecurve2d/tablecurve2d.php
>
> I think astrogui, or astropysics might have something like this,
> Travis showcased it briefly in the stats webinar. There is also an
> online curve fitting (zunzun ?) with (I think) BSD licensed source
> code that might be useful.
>
> > [2] http://www.mathwave.com/easyfit-distribution-fitting.html
>
> I looked at easyfit a year or more ago, and wrote a script that tried
> to do similar fitting for all distributions in scipy.stats. It worked
> ok for distributions with unlimited support, but there are still the
> problems with estimating distributions with a finite bound in the
> support.
> I used kstest as distance measure, but never rewrote Anderson-Darling
> to function as generic distance measure as (if I remember correctly)
> easyfit does.
>
> Josef
>

Thanks for the input Josef.

I have known astrogui from Travis' webinar as well. However I have never
been able to run it. Maybe Erik could make some comments to fix the problem:

# Using bzr branch lp:astropysics

python
Python 2.6.2 (r262:71600, Jan 25 2010, 18:46:45)
[GCC 4.4.2 20091222 (Red Hat 4.4.2-20)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from astropysics import gui
/home/gsever/Desktop/python-repo/ETS_3.3.1/Enable_3.2.1/enthought/kiva/fonttools/font_manager.py:1064:
UserWarning: Could not match sans-serif, normal, normal.  Returning
/usr/share/fonts/khmeros/KhmerOS.ttf
  warnings.warn('Could not match %s, %s, %s.  Returning %s' % (name, style,
variant, self.defaultFont))
/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/io.py:34:
UserWarning: pyfits not found - all FITS-related IO will not work
  warn('pyfits not found - all FITS-related IO will not work')
/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/io.py:314:
UserWarning: vo.table not found - VOTable processing limited to
VOTableReader class
  warn('vo.table not found - VOTable processing limited to VOTableReader
class')
/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py:2742:
UserWarning: SExtractor not found on system - phot.SExtractor class will not
function
  warn('SExtractor not found on system - phot.SExtractor class will not
function')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/gui/__init__.py",
line 28, in <module>
    from spectarget import spec_target
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/gui/spectarget.py",
line 18, in <module>
    from ..phot import CMDAnalyzer
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py",
line 3463, in <module>
    Spectrum._rgbsensitivity = __generateRGBSensTuple()
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py",
line 3455, in __generateRGBSensTuple
    from .models import BlackbodyModel
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/__init__.py",
line 73, in <module>
    from core import *
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/core.py",
line 2247, in <module>
    class FunctionModel2DScalar(FunctionModel,InputCoordinateTransformer):
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/core.py",
line 2253, in FunctionModel2DScalar
    from ..coords import cartesian_to_polar,polar_to_cartesian
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
line 2447, in <module>
    solsysobjs['sun'] = Sun()
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
line 2267, in __init__
    EphemerisObject.__init__(self,name='Sol',jd0=KeplerianOrbit.jd2000-0.5)
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
line 1791, in __init__
    self.jd0 = EphemerisObject.jd2000 if jd0 is None else jd0
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
line 1799, in _setJd0
    from .obstools import calendar_to_jd
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/obstools.py",
line 1580, in <module>
    sites['uciobs'] =
Site(33.63614044191056,-117.83079922199249,80,'PST','UC Irvine Observatory')
  File
"/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/obstools.py",
line 498, in __init__
    raise ValueError('unrecognized time zone string '+tz)
ValueError: unrecognized time zone string PST


-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100412/a953d7b9/attachment.html>

From jorgesmbox-ml at yahoo.es  Mon Apr 12 17:34:23 2010
From: jorgesmbox-ml at yahoo.es (Jorge Scandaliaris)
Date: Mon, 12 Apr 2010 21:34:23 +0000 (UTC)
Subject: [SciPy-User] =?utf-8?q?Question_about_gaussian=5Fkde?=
References: <loom.20100412T174309-768@post.gmane.org>
Message-ID: <loom.20100412T232201-879@post.gmane.org>

Oops! Thanks Josef and Robert, I was thinking about a pmf when actually using a
pdf. 

Jorge





From jlconlin at gmail.com  Mon Apr 12 18:55:31 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Mon, 12 Apr 2010 16:55:31 -0600
Subject: [SciPy-User] Can I get the "rings" of a 2D array?
Message-ID: <q2w2588da421004121555h38bf8bc5lb74abfb4cffb4a22@mail.gmail.com>

I need to get the "rings" of a 2D array; i.e. the first ring would be
the first and last column in addition to the first and last row, the
second ring would be the second and second-from-last row and the
second and second-from-last column---except for what is not in the
first ring.  Is there a way to slice an array to get this data?

Thanks,
Jeremy


From charlesr.harris at gmail.com  Mon Apr 12 19:16:10 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 12 Apr 2010 17:16:10 -0600
Subject: [SciPy-User] Can I get the "rings" of a 2D array?
In-Reply-To: <q2w2588da421004121555h38bf8bc5lb74abfb4cffb4a22@mail.gmail.com>
References: <q2w2588da421004121555h38bf8bc5lb74abfb4cffb4a22@mail.gmail.com>
Message-ID: <k2ue06186141004121616kd2fd3cfdv734ce3855522af9e@mail.gmail.com>

On Mon, Apr 12, 2010 at 4:55 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:

> I need to get the "rings" of a 2D array; i.e. the first ring would be
> the first and last column in addition to the first and last row, the
> second ring would be the second and second-from-last row and the
> second and second-from-last column---except for what is not in the
> first ring.  Is there a way to slice an array to get this data?
>
>
So will the rings be different sizes or all the same size with the other
elements zeroed?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100412/9ebd51c9/attachment.html>

From bblais at bryant.edu  Mon Apr 12 19:27:49 2010
From: bblais at bryant.edu (Brian Blais)
Date: Mon, 12 Apr 2010 19:27:49 -0400
Subject: [SciPy-User] keyboard interrupt problem with odeint
Message-ID: <55743F95-45D2-4E0B-9BDE-71EB8A98CCC6@bryant.edu>

Hello,

I have a busy loop with odeint that I want to allow the user to stop,  
and then have the program return gracefully.  Unfortunately, when I  
try to catch the KeyboardInterrupt exception, it doesn't exit or  
catch it.  I paired the program down to the following silly example  
to reproduce the problem:

#==========
from scipy.integrate import odeint
from numpy import *


def damped_osc(u,t,b): #defines the system of odes
     x=u[0]
     v=u[1]
     return(v,-x-b*v) #the derivatives of u

t = arange(0,20,0.001)
u0 = array([1,0])
b=0.4
try:
     while True:
         u=odeint(damped_osc,u0,t,args=(b,)) #b is in tuple, needs comma
except KeyboardInterrupt:
     print "here!"
#==========

hitting control-C, I get the following printed to the screen:

error: Error occured while calling the Python function named damped_osc

but the thing keeps going!


Is there a way I can consistently stop it?


		thanks,

			Brian Blais

-- 
Brian Blais
bblais at bryant.edu
http://web.bryant.edu/~bblais
http://bblais.blogspot.com/



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100412/d5a95acc/attachment.html>

From jlconlin at gmail.com  Mon Apr 12 22:23:00 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Mon, 12 Apr 2010 20:23:00 -0600
Subject: [SciPy-User] Can I get the "rings" of a 2D array?
In-Reply-To: <k2ue06186141004121616kd2fd3cfdv734ce3855522af9e@mail.gmail.com>
References: <q2w2588da421004121555h38bf8bc5lb74abfb4cffb4a22@mail.gmail.com>
	<k2ue06186141004121616kd2fd3cfdv734ce3855522af9e@mail.gmail.com>
Message-ID: <t2u2588da421004121923z8e0c103co5ed37bcba6b8a4f0@mail.gmail.com>

On Mon, Apr 12, 2010 at 5:16 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Mon, Apr 12, 2010 at 4:55 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>
>> I need to get the "rings" of a 2D array; i.e. the first ring would be
>> the first and last column in addition to the first and last row, the
>> second ring would be the second and second-from-last row and the
>> second and second-from-last column---except for what is not in the
>> first ring. ?Is there a way to slice an array to get this data?
>>
>
> So will the rings be different sizes or all the same size with the other
> elements zeroed?

Good question.  The better option would be to keep the size of the
array the same and zero all the other elements.

Jeremy


From josef.pktd at gmail.com  Mon Apr 12 23:19:02 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 12 Apr 2010 23:19:02 -0400
Subject: [SciPy-User] Can I get the "rings" of a 2D array?
In-Reply-To: <t2u2588da421004121923z8e0c103co5ed37bcba6b8a4f0@mail.gmail.com>
References: <q2w2588da421004121555h38bf8bc5lb74abfb4cffb4a22@mail.gmail.com>
	<k2ue06186141004121616kd2fd3cfdv734ce3855522af9e@mail.gmail.com>
	<t2u2588da421004121923z8e0c103co5ed37bcba6b8a4f0@mail.gmail.com>
Message-ID: <s2u1cd32cbb1004122019wa3821b69taed50714b6260072@mail.gmail.com>

On Mon, Apr 12, 2010 at 10:23 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
> On Mon, Apr 12, 2010 at 5:16 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>>
>> On Mon, Apr 12, 2010 at 4:55 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>>
>>> I need to get the "rings" of a 2D array; i.e. the first ring would be
>>> the first and last column in addition to the first and last row, the
>>> second ring would be the second and second-from-last row and the
>>> second and second-from-last column---except for what is not in the
>>> first ring. ?Is there a way to slice an array to get this data?
>>>
>>
>> So will the rings be different sizes or all the same size with the other
>> elements zeroed?
>
> Good question. ?The better option would be to keep the size of the
> array the same and zero all the other elements.

The following seems to work for building the index arrays for the rings,
tested on only 2 examples


import numpy as np
a = np.arange(30).reshape(6,5)
n = np.array(a.shape)
from collections import defaultdict
rings = defaultdict(list)
for i in np.ndindex(*n):
    #print i,
    imin = min(i)
    imax = min(n-i-1)
    #print imin,imax
    rings[min(imin,imax)].append(i)

for r in rings:
    print r
    print a[np.array(rings[r])[:,0],np.array(rings[r])[:,1]]


0
[ 0  1  2  3  4  5  9 10 14 15 19 20 24 25 26 27 28 29]
1
[ 6  7  8 11 13 16 18 21 22 23]
2
[12 17]
>>> a
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24],
       [25, 26, 27, 28, 29]])

Josef


>
> Jeremy
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From jlconlin at gmail.com  Mon Apr 12 23:35:55 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Mon, 12 Apr 2010 21:35:55 -0600
Subject: [SciPy-User] Can I get the "rings" of a 2D array?
In-Reply-To: <s2u1cd32cbb1004122019wa3821b69taed50714b6260072@mail.gmail.com>
References: <q2w2588da421004121555h38bf8bc5lb74abfb4cffb4a22@mail.gmail.com>
	<k2ue06186141004121616kd2fd3cfdv734ce3855522af9e@mail.gmail.com>
	<t2u2588da421004121923z8e0c103co5ed37bcba6b8a4f0@mail.gmail.com>
	<s2u1cd32cbb1004122019wa3821b69taed50714b6260072@mail.gmail.com>
Message-ID: <z2v2588da421004122035ga8bd9959ne12f6fee919985b@mail.gmail.com>

On Mon, Apr 12, 2010 at 9:19 PM,  <josef.pktd at gmail.com> wrote:
> On Mon, Apr 12, 2010 at 10:23 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>> On Mon, Apr 12, 2010 at 5:16 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>>
>>>
>>> On Mon, Apr 12, 2010 at 4:55 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>>>
>>>> I need to get the "rings" of a 2D array; i.e. the first ring would be
>>>> the first and last column in addition to the first and last row, the
>>>> second ring would be the second and second-from-last row and the
>>>> second and second-from-last column---except for what is not in the
>>>> first ring. ?Is there a way to slice an array to get this data?
>>>>
>>>
>>> So will the rings be different sizes or all the same size with the other
>>> elements zeroed?
>>
>> Good question. ?The better option would be to keep the size of the
>> array the same and zero all the other elements.
>
> The following seems to work for building the index arrays for the rings,
> tested on only 2 examples
>
>
> import numpy as np
> a = np.arange(30).reshape(6,5)
> n = np.array(a.shape)
> from collections import defaultdict
> rings = defaultdict(list)
> for i in np.ndindex(*n):
> ? ?#print i,
> ? ?imin = min(i)
> ? ?imax = min(n-i-1)
> ? ?#print imin,imax
> ? ?rings[min(imin,imax)].append(i)
>
> for r in rings:
> ? ?print r
> ? ?print a[np.array(rings[r])[:,0],np.array(rings[r])[:,1]]
>
>
> 0
> [ 0 ?1 ?2 ?3 ?4 ?5 ?9 10 14 15 19 20 24 25 26 27 28 29]
> 1
> [ 6 ?7 ?8 11 13 16 18 21 22 23]
> 2
> [12 17]
>>>> a
> array([[ 0, ?1, ?2, ?3, ?4],
> ? ? ? [ 5, ?6, ?7, ?8, ?9],
> ? ? ? [10, 11, 12, 13, 14],
> ? ? ? [15, 16, 17, 18, 19],
> ? ? ? [20, 21, 22, 23, 24],
> ? ? ? [25, 26, 27, 28, 29]])
>
> Josef
>

Thanks.  I just might be able to put that to work.

Jeremy


From erik.tollerud at gmail.com  Tue Apr 13 02:13:57 2010
From: erik.tollerud at gmail.com (Erik Tollerud)
Date: Mon, 12 Apr 2010 23:13:57 -0700
Subject: [SciPy-User] Fitting tools
In-Reply-To: <v2l49d6b3501004121254ic2929a9fn9364bf6e03f3030b@mail.gmail.com>
References: <p2u49d6b3501004121014p382c1712k72ba68fb91bececd@mail.gmail.com> 
	<w2k1cd32cbb1004121034q58fd3a9seea52b1a1505e399@mail.gmail.com> 
	<v2l49d6b3501004121254ic2929a9fn9364bf6e03f3030b@mail.gmail.com>
Message-ID: <h2zfbab3e0d1004122313tbba74bber11a870119a537be@mail.gmail.com>

Ok, I just committed a patch that I think should fix the particular
problem you were encountering - let me know if it does not, or if you
encounter some other problem trying to run the fitgui.

Also, if you there are some models/distributions you want to use that
are not already included in the builtin models (rather likely, as most
of them are astronomy-related aside from the basics like gaussian,
linear, polynomial, and so on) it's straightforward to add new ones in
your own code and register them so that the gui will display them as
options.  If it's a fairly standard distribution, feel free to submit
it to me and I can add it to the astropysics builtins, as well.


On Mon, Apr 12, 2010 at 12:54 PM, G?khan Sever <gokhansever at gmail.com> wrote:
>
>
> On Mon, Apr 12, 2010 at 12:34 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Mon, Apr 12, 2010 at 1:14 PM, G?khan Sever <gokhansever at gmail.com>
>> wrote:
>> > Hello,
>> >
>> > Are there any compact curve/distribution fitting tools around similar to
>> > the
>> > commercial TableCurve 2D [1] and EasyFit [2] products?
>> >
>> > [1] http://www.sigmaplot.com/products/tablecurve2d/tablecurve2d.php
>>
>> I think astrogui, or astropysics might have something like this,
>> Travis showcased it briefly in the stats webinar. There is also an
>> online curve fitting (zunzun ?) with (I think) BSD licensed source
>> code that might be useful.
>>
>> > [2] http://www.mathwave.com/easyfit-distribution-fitting.html
>>
>> I looked at easyfit a year or more ago, and wrote a script that tried
>> to do similar fitting for all distributions in scipy.stats. It worked
>> ok for distributions with unlimited support, but there are still the
>> problems with estimating distributions with a finite bound in the
>> support.
>> I used kstest as distance measure, but never rewrote Anderson-Darling
>> to function as generic distance measure as (if I remember correctly)
>> easyfit does.
>>
>> Josef
>
> Thanks for the input Josef.
>
> I have known astrogui from Travis' webinar as well. However I have never
> been able to run it. Maybe Erik could make some comments to fix the problem:
>
> # Using bzr branch lp:astropysics
>
> python
> Python 2.6.2 (r262:71600, Jan 25 2010, 18:46:45)
> [GCC 4.4.2 20091222 (Red Hat 4.4.2-20)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> from astropysics import gui
> /home/gsever/Desktop/python-repo/ETS_3.3.1/Enable_3.2.1/enthought/kiva/fonttools/font_manager.py:1064:
> UserWarning: Could not match sans-serif, normal, normal.? Returning
> /usr/share/fonts/khmeros/KhmerOS.ttf
> ? warnings.warn('Could not match %s, %s, %s.? Returning %s' % (name, style,
> variant, self.defaultFont))
> /usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/io.py:34:
> UserWarning: pyfits not found - all FITS-related IO will not work
> ? warn('pyfits not found - all FITS-related IO will not work')
> /usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/io.py:314:
> UserWarning: vo.table not found - VOTable processing limited to
> VOTableReader class
> ? warn('vo.table not found - VOTable processing limited to VOTableReader
> class')
> /usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py:2742:
> UserWarning: SExtractor not found on system - phot.SExtractor class will not
> function
> ? warn('SExtractor not found on system - phot.SExtractor class will not
> function')
> Traceback (most recent call last):
> ? File "<stdin>", line 1, in <module>
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/gui/__init__.py",
> line 28, in <module>
> ??? from spectarget import spec_target
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/gui/spectarget.py",
> line 18, in <module>
> ??? from ..phot import CMDAnalyzer
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py",
> line 3463, in <module>
> ??? Spectrum._rgbsensitivity = __generateRGBSensTuple()
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py",
> line 3455, in __generateRGBSensTuple
> ??? from .models import BlackbodyModel
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/__init__.py",
> line 73, in <module>
> ??? from core import *
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/core.py",
> line 2247, in <module>
> ??? class FunctionModel2DScalar(FunctionModel,InputCoordinateTransformer):
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/core.py",
> line 2253, in FunctionModel2DScalar
> ??? from ..coords import cartesian_to_polar,polar_to_cartesian
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
> line 2447, in <module>
> ??? solsysobjs['sun'] = Sun()
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
> line 2267, in __init__
> ??? EphemerisObject.__init__(self,name='Sol',jd0=KeplerianOrbit.jd2000-0.5)
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
> line 1791, in __init__
> ??? self.jd0 = EphemerisObject.jd2000 if jd0 is None else jd0
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
> line 1799, in _setJd0
> ??? from .obstools import calendar_to_jd
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/obstools.py",
> line 1580, in <module>
> ??? sites['uciobs'] =
> Site(33.63614044191056,-117.83079922199249,80,'PST','UC Irvine Observatory')
> ? File
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/obstools.py",
> line 498, in __init__
> ??? raise ValueError('unrecognized time zone string '+tz)
> ValueError: unrecognized time zone string PST
>
>
> --
> G?khan
>


From j.reid at mail.cryst.bbk.ac.uk  Tue Apr 13 04:02:43 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Tue, 13 Apr 2010 09:02:43 +0100
Subject: [SciPy-User] site.cfg problems : scipy can't find BLAS
In-Reply-To: <hpvd8l$q1n$1@dough.gmane.org>
References: <hpkinj$pmp$1@dough.gmane.org>	<v2k1cd32cbb1004081136qde4e3039p59adce6937edbf82@mail.gmail.com>	<hpla73$pgc$2@dough.gmane.org>	<m2v1cd32cbb1004081304ya41d7b33s1d05588775ead2b0@mail.gmail.com>	<hpmn2c$vdo$1@dough.gmane.org>	<4BBEF441.1060301@silveregg.co.jp>	<hpmt3o$m4v$1@dough.gmane.org>	<g2ge9962b831004112104k6b194f3fo3938d6c152d5f3d6@mail.gmail.com>	<4BC2CA1F.6060206@gmail.com>
	<hpvd8l$q1n$1@dough.gmane.org>
Message-ID: <hq18f3$k8p$1@dough.gmane.org>

John Reid wrote:
> Thanks for the help Patrick, Minh-Tri, Josef and David.
> 
> I can get scipy compiled but its still failing the tests. When I run 
> with verbose=10 I can identify the test it is running:
> 
I just thought I'd let everyone know I've worked around these problems 
with the following steps:

- I've gone back to numpy 1.3.0 and scipy 0.7.1
- I follow the instructions for Ubuntu at 
http://www.scipy.org/Installing_SciPy/Linux#head-1c4018a51422706809ee96a4db03ca0669f5f6d1
but I replace the building ATLAS step by building GotoBLAS2 and a netlib 
LAPACK that uses it


My new site.cfg is:
[DEFAULT]
library_dirs = /usr/local/src/scipy-stack/lib
include_dirs = /usr/local/src/scipy-stack/include

[blas]
libraries = goto2
language = fortran

[lapack]
libraries = lapack goto2
language = fortran

[amd]
amd_libs = amd

[umfpack]
umfpack_libs = umfpack

[fftw]
libraries = fftw3


I hope that helps anyone else who has similar problems. Thanks all for 
the help.

John.



From kuiper at jpl.nasa.gov  Tue Apr 13 09:20:58 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Tue, 13 Apr 2010 06:20:58 -0700
Subject: [SciPy-User] unpacking binary data from a C structure
Message-ID: <4BC46FBA.6070505@jpl.nasa.gov>

Dear list,

here's something I find very strange.  I have a C structure defined as:

typedef struct
{
  unsigned short spcid;         /* station id - 10, 40, 60, 21 */
  unsigned short vsrid;         /* vsr1a, vsr1b ... from enum */     
  unsigned short chanid;        /* subchannel id 0,1,2,3 */
  unsigned short bps;           /* number of bits per sample - 1, 2, 4, 
8, or
                                   16 */
  unsigned long  srate;         /* number of samples per second in 
kilo-samples
                                   per second */
  unsigned short error;         /* hw err flag, dma error or num_samples 
error,
                                   0 ==> no errors */
  unsigned short year;          /* time tag - year */
  unsigned short doy;           /* time tag - day of year */
  unsigned long  sec;           /* time tag - second of day */
  double         freq;          /* in Hz */
  unsigned long  orate;         /* number of statistics samples per 
second */
  unsigned short nsubchan;      /* number of output sub chans */
}
stats_hdr_t;

The python module struct unpack expected format is 'HHHH L HHH L d L H'
Here's a real header structure as it appears at the head of a file:

  0000000  000d  0001  0006  0008
  0000010  4240  000f  0000  0000
  0000020  0000  07da  0064  4730
  0000030  0001  0000  0000  0000
  0000040  d800  d31d  421d  03e8
  0000048  0000  0000  0000  0002

Decoded as unsigned shorts:

  0000000    13     1     6     8
  0000010 16960    15     0     0
  0000020     0  2010   100 18224
  0000030     1     0     0     0
  0000040 55296 54045 16925  1000
  0000050     0     0     0     2

Matching these to the stats_hdr_t with 'unpack' notation:

  0000000     H     H     H     H
  0000010    L1    L2     H     ?
  0000020     ?     H     H    L1
  0000030    L2     ?     ?    D1
  0000040    D2    D3    D4    L1
  0000050    L2     ?     ?     H

So the actual format is 'HHHH L H xxxx HH L xxxx d L xxxx H'
What are all the mystery 4-byte blanks?  This works:

buf = fd.read(50)
header = unpack_from('=4H LH2x 2x2HL4xdL4xH',buf)

Since unpacking binary data must be a fairly common activity in 
scientific circles. I hope you will have some suggestions.

Tom



From kuiper at jpl.nasa.gov  Tue Apr 13 09:28:55 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Tue, 13 Apr 2010 06:28:55 -0700
Subject: [SciPy-User] unpacking binary data from a C structure
Message-ID: <4BC47197.2090805@jpl.nasa.gov>

I should probably have mentioned how I got the buffer:

  buf = fd.read(50)
  header = unpack_from('=4H LH2x 2x2HL4xdL4xH',buf)

Also, the system is Debian Linux so gcc was the compiler that made the 
program that wrote the binary file.

Tom


From robert.kern at gmail.com  Tue Apr 13 10:41:12 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 13 Apr 2010 09:41:12 -0500
Subject: [SciPy-User] unpacking binary data from a C structure
In-Reply-To: <4BC46FBA.6070505@jpl.nasa.gov>
References: <4BC46FBA.6070505@jpl.nasa.gov>
Message-ID: <t2r3d375d731004130741lb7da8bb3m78cfab05d35a32d@mail.gmail.com>

On Tue, Apr 13, 2010 at 08:20, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
> Dear list,
>
> here's something I find very strange. ?I have a C structure defined as:
>
> typedef struct
> {
> ?unsigned short spcid; ? ? ? ? /* station id - 10, 40, 60, 21 */
> ?unsigned short vsrid; ? ? ? ? /* vsr1a, vsr1b ... from enum */
> ?unsigned short chanid; ? ? ? ?/* subchannel id 0,1,2,3 */
> ?unsigned short bps; ? ? ? ? ? /* number of bits per sample - 1, 2, 4,
> 8, or
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 16 */
> ?unsigned long ?srate; ? ? ? ? /* number of samples per second in
> kilo-samples
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? per second */
> ?unsigned short error; ? ? ? ? /* hw err flag, dma error or num_samples
> error,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0 ==> no errors */
> ?unsigned short year; ? ? ? ? ?/* time tag - year */
> ?unsigned short doy; ? ? ? ? ? /* time tag - day of year */
> ?unsigned long ?sec; ? ? ? ? ? /* time tag - second of day */
> ?double ? ? ? ? freq; ? ? ? ? ?/* in Hz */
> ?unsigned long ?orate; ? ? ? ? /* number of statistics samples per
> second */
> ?unsigned short nsubchan; ? ? ?/* number of output sub chans */
> }
> stats_hdr_t;
>
> The python module struct unpack expected format is 'HHHH L HHH L d L H'
> Here's a real header structure as it appears at the head of a file:
>
> ?0000000 ?000d ?0001 ?0006 ?0008
> ?0000010 ?4240 ?000f ?0000 ?0000
> ?0000020 ?0000 ?07da ?0064 ?4730
> ?0000030 ?0001 ?0000 ?0000 ?0000
> ?0000040 ?d800 ?d31d ?421d ?03e8
> ?0000048 ?0000 ?0000 ?0000 ?0002
>
> Decoded as unsigned shorts:
>
> ?0000000 ? ?13 ? ? 1 ? ? 6 ? ? 8
> ?0000010 16960 ? ?15 ? ? 0 ? ? 0
> ?0000020 ? ? 0 ?2010 ? 100 18224
> ?0000030 ? ? 1 ? ? 0 ? ? 0 ? ? 0
> ?0000040 55296 54045 16925 ?1000
> ?0000050 ? ? 0 ? ? 0 ? ? 0 ? ? 2
>
> Matching these to the stats_hdr_t with 'unpack' notation:
>
> ?0000000 ? ? H ? ? H ? ? H ? ? H
> ?0000010 ? ?L1 ? ?L2 ? ? H ? ? ?
> ?0000020 ? ? ? ? ? H ? ? H ? ?L1
> ?0000030 ? ?L2 ? ? ? ? ? ? ? ?D1
> ?0000040 ? ?D2 ? ?D3 ? ?D4 ? ?L1
> ?0000050 ? ?L2 ? ? ? ? ? ? ? ? H
>
> So the actual format is 'HHHH L H xxxx HH L xxxx d L xxxx H'
> What are all the mystery 4-byte blanks? ?This works:
>
> buf = fd.read(50)
> header = unpack_from('=4H LH2x 2x2HL4xdL4xH',buf)
>
> Since unpacking binary data must be a fairly common activity in
> scientific circles. I hope you will have some suggestions.

C compilers can insert padding into the memory layout of a structure
in order to align certain members to certain boundaries, particularly
doubles. This behavior is compiler- and platform-specific.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From sransom at nrao.edu  Tue Apr 13 10:57:38 2010
From: sransom at nrao.edu (Scott Ransom)
Date: Tue, 13 Apr 2010 10:57:38 -0400
Subject: [SciPy-User] unpacking binary data from a C structure
In-Reply-To: <t2r3d375d731004130741lb7da8bb3m78cfab05d35a32d@mail.gmail.com>
References: <4BC46FBA.6070505@jpl.nasa.gov>
	<t2r3d375d731004130741lb7da8bb3m78cfab05d35a32d@mail.gmail.com>
Message-ID: <201004131057.38774.sransom@nrao.edu>

On Tuesday 13 April 2010 10:41:12 am Robert Kern wrote:
> On Tue, Apr 13, 2010 at 08:20, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
> > Dear list,
> >
> > here's something I find very strange.  I have a C structure defined
> > as:
> >
> > typedef struct
> > {
> >  unsigned short spcid;         /* station id - 10, 40, 60, 21 */
> >  unsigned short vsrid;         /* vsr1a, vsr1b ... from enum */
> >  unsigned short chanid;        /* subchannel id 0,1,2,3 */
> >  unsigned short bps;           /* number of bits per sample - 1, 2,
> > 4, 8, or
> >                                   16 */
> >  unsigned long  srate;         /* number of samples per second in
> > kilo-samples
> >                                   per second */
> >  unsigned short error;         /* hw err flag, dma error or
> > num_samples error,
> >                                   0 ==> no errors */
> >  unsigned short year;          /* time tag - year */
> >  unsigned short doy;           /* time tag - day of year */
> >  unsigned long  sec;           /* time tag - second of day */
> >  double         freq;          /* in Hz */
> >  unsigned long  orate;         /* number of statistics samples per
> > second */
> >  unsigned short nsubchan;      /* number of output sub chans */
> > }
> > stats_hdr_t;
> >
> > The python module struct unpack expected format is 'HHHH L HHH L d
> > L H' Here's a real header structure as it appears at the head of a
> > file:
> >
> >  0000000  000d  0001  0006  0008
> >  0000010  4240  000f  0000  0000
> >  0000020  0000  07da  0064  4730
> >  0000030  0001  0000  0000  0000
> >  0000040  d800  d31d  421d  03e8
> >  0000048  0000  0000  0000  0002
> >
> > Decoded as unsigned shorts:
> >
> >  0000000    13     1     6     8
> >  0000010 16960    15     0     0
> >  0000020     0  2010   100 18224
> >  0000030     1     0     0     0
> >  0000040 55296 54045 16925  1000
> >  0000050     0     0     0     2
> >
> > Matching these to the stats_hdr_t with 'unpack' notation:
> >
> >  0000000     H     H     H     H
> >  0000010    L1    L2     H     ?
> >  0000020     ?     H     H    L1
> >  0000030    L2     ?     ?    D1
> >  0000040    D2    D3    D4    L1
> >  0000050    L2     ?     ?     H
> >
> > So the actual format is 'HHHH L H xxxx HH L xxxx d L xxxx H'
> > What are all the mystery 4-byte blanks?  This works:
> >
> > buf = fd.read(50)
> > header = unpack_from('=4H LH2x 2x2HL4xdL4xH',buf)
> >
> > Since unpacking binary data must be a fairly common activity in
> > scientific circles. I hope you will have some suggestions.
> 
> C compilers can insert padding into the memory layout of a structure
> in order to align certain members to certain boundaries, particularly
> doubles. This behavior is compiler- and platform-specific.

If the exact order of the elements in the structure is not important, 
you can mitigate against this (but not prevent it entirely) by putting 
common types of structure elements together, with the largest ones 
first.  In your case, that would mean grouping all the longs first and 
then putting all the shorts together second.

Scott


-- 
Scott M. Ransom            Address:  NRAO
Phone:  (434) 296-0320               520 Edgemont Rd.
email:  sransom at nrao.edu             Charlottesville, VA 22903 USA
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989


From jason-sage at creativetrax.com  Tue Apr 13 11:06:24 2010
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 13 Apr 2010 10:06:24 -0500
Subject: [SciPy-User] unpacking binary data from a C structure
In-Reply-To: <4BC46FBA.6070505@jpl.nasa.gov>
References: <4BC46FBA.6070505@jpl.nasa.gov>
Message-ID: <4BC48870.70404@creativetrax.com>

On 04/13/2010 08:20 AM, Tom Kuiper wrote:
> So the actual format is 'HHHH L H xxxx HH L xxxx d L xxxx H'
> What are all the mystery 4-byte blanks?  This works:
>
> buf = fd.read(50)
> header = unpack_from('=4H LH2x 2x2HL4xdL4xH',buf)
>    

If you are using gcc, this might be relevant:

http://sig9.com/articles/gcc-packed-structures

Thanks,

Jason


--
Jason Grout



From charlesr.harris at gmail.com  Tue Apr 13 11:27:55 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 13 Apr 2010 09:27:55 -0600
Subject: [SciPy-User] unpacking binary data from a C structure
In-Reply-To: <4BC46FBA.6070505@jpl.nasa.gov>
References: <4BC46FBA.6070505@jpl.nasa.gov>
Message-ID: <y2le06186141004130827zf9f568b6rc96627279e9e2519@mail.gmail.com>

On Tue, Apr 13, 2010 at 7:20 AM, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:

> Dear list,
>
> here's something I find very strange.  I have a C structure defined as:
>
> typedef struct
> {
>  unsigned short spcid;         /* station id - 10, 40, 60, 21 */
>  unsigned short vsrid;         /* vsr1a, vsr1b ... from enum */
>  unsigned short chanid;        /* subchannel id 0,1,2,3 */
>  unsigned short bps;           /* number of bits per sample - 1, 2, 4,
> 8, or
>                                   16 */
>  unsigned long  srate;         /* number of samples per second in
> kilo-samples
>                                   per second */
>  unsigned short error;         /* hw err flag, dma error or num_samples
> error,
>                                   0 ==> no errors */
>  unsigned short year;          /* time tag - year */
>  unsigned short doy;           /* time tag - day of year */
>  unsigned long  sec;           /* time tag - second of day */
>  double         freq;          /* in Hz */
>  unsigned long  orate;         /* number of statistics samples per
> second */
>  unsigned short nsubchan;      /* number of output sub chans */
> }
> stats_hdr_t;
>
> The python module struct unpack expected format is 'HHHH L HHH L d L H'
> Here's a real header structure as it appears at the head of a file:
>
>  0000000  000d  0001  0006  0008
>  0000010  4240  000f  0000  0000
>  0000020  0000  07da  0064  4730
>  0000030  0001  0000  0000  0000
>  0000040  d800  d31d  421d  03e8
>  0000048  0000  0000  0000  0002
>
> Decoded as unsigned shorts:
>
>  0000000    13     1     6     8
>  0000010 16960    15     0     0
>  0000020     0  2010   100 18224
>  0000030     1     0     0     0
>  0000040 55296 54045 16925  1000
>  0000050     0     0     0     2
>
> Matching these to the stats_hdr_t with 'unpack' notation:
>
>  0000000     H     H     H     H
>  0000010    L1    L2     H     ?
>  0000020     ?     H     H    L1
>  0000030    L2     ?     ?    D1
>  0000040    D2    D3    D4    L1
>  0000050    L2     ?     ?     H
>
> So the actual format is 'HHHH L H xxxx HH L xxxx d L xxxx H'
> What are all the mystery 4-byte blanks?  This works:
>
> buf = fd.read(50)
> header = unpack_from('=4H LH2x 2x2HL4xdL4xH',buf)
>
> Since unpacking binary data must be a fairly common activity in
> scientific circles. I hope you will have some suggestions.
>
>
I presume you didn't produce the data, but as a rule of thumb c structures
should not be used to write out binary data, as the binary layout of the
data won't be portable. Text, netcdf, hdf5, or some other standard data
format is preferable, with text being perhaps the most portable. That said,
lots of old data collection programs write out c structures, and no doubt
newer programs do so also.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100413/646d2a60/attachment.html>

From kuiper at jpl.nasa.gov  Tue Apr 13 11:39:21 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Tue, 13 Apr 2010 08:39:21 -0700
Subject: [SciPy-User] SciPy-User Digest, Vol 80, Issue 22
In-Reply-To: <mailman.226.1271169703.28903.scipy-user@scipy.org>
References: <mailman.226.1271169703.28903.scipy-user@scipy.org>
Message-ID: <4BC49029.3060201@jpl.nasa.gov>

Thanks for the tip Robert. 

On my 32-bit laptop:
In [11]: calcsize('HHHH L HHH L d L H')
Out[11]: 38

On a 64-bit machine:
In [3]: calcsize('HHHH L HHH L d L H')
Out[3]: 50

Regards,

Tom
> Date: Tue, 13 Apr 2010 09:41:12 -0500
> From: Robert Kern <robert.kern at gmail.com>
> C compilers can insert padding into the memory layout of a structure
> in order to align certain members to certain boundaries, particularly
> doubles. This behavior is compiler- and platform-specific.
>   



From sransom at nrao.edu  Tue Apr 13 11:45:51 2010
From: sransom at nrao.edu (Scott Ransom)
Date: Tue, 13 Apr 2010 11:45:51 -0400
Subject: [SciPy-User] SciPy-User Digest, Vol 80, Issue 22
In-Reply-To: <4BC49029.3060201@jpl.nasa.gov>
References: <mailman.226.1271169703.28903.scipy-user@scipy.org>
	<4BC49029.3060201@jpl.nasa.gov>
Message-ID: <201004131145.51056.sransom@nrao.edu>

On Tuesday 13 April 2010 11:39:21 am Tom Kuiper wrote:
> Thanks for the tip Robert.
> 
> On my 32-bit laptop:
> In [11]: calcsize('HHHH L HHH L d L H')
> Out[11]: 38
> 
> On a 64-bit machine:
> In [3]: calcsize('HHHH L HHH L d L H')
> Out[3]: 50

Hi Tom,

That's probably not because of the padding.  It is likely due to the 
fact that longs are 4 bytes on 32-bit machines and 8 bytes on 64-bit 
machines.

Scott


-- 
Scott M. Ransom            Address:  NRAO
Phone:  (434) 296-0320               520 Edgemont Rd.
email:  sransom at nrao.edu             Charlottesville, VA 22903 USA
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989


From peridot.faceted at gmail.com  Tue Apr 13 11:55:53 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 13 Apr 2010 11:55:53 -0400
Subject: [SciPy-User] unpacking binary data from a C structure
In-Reply-To: <y2le06186141004130827zf9f568b6rc96627279e9e2519@mail.gmail.com>
References: <4BC46FBA.6070505@jpl.nasa.gov>
	<y2le06186141004130827zf9f568b6rc96627279e9e2519@mail.gmail.com>
Message-ID: <i2xce557a361004130855rabadb16rc9b732506effc175@mail.gmail.com>

On 13 April 2010 11:27, Charles R Harris <charlesr.harris at gmail.com> wrote:
>
>
> On Tue, Apr 13, 2010 at 7:20 AM, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
>>
>> Dear list,
>>
>> here's something I find very strange.  I have a C structure defined as:
>>
>> typedef struct
>> {
>>  unsigned short spcid;         /* station id - 10, 40, 60, 21 */
>>  unsigned short vsrid;         /* vsr1a, vsr1b ... from enum */
>>  unsigned short chanid;        /* subchannel id 0,1,2,3 */
>>  unsigned short bps;           /* number of bits per sample - 1, 2, 4,
>> 8, or
>>                                   16 */
>>  unsigned long  srate;         /* number of samples per second in
>> kilo-samples
>>                                   per second */
>>  unsigned short error;         /* hw err flag, dma error or num_samples
>> error,
>>                                   0 ==> no errors */
>>  unsigned short year;          /* time tag - year */
>>  unsigned short doy;           /* time tag - day of year */
>>  unsigned long  sec;           /* time tag - second of day */
>>  double         freq;          /* in Hz */
>>  unsigned long  orate;         /* number of statistics samples per
>> second */
>>  unsigned short nsubchan;      /* number of output sub chans */
>> }
>> stats_hdr_t;
>>
>> The python module struct unpack expected format is 'HHHH L HHH L d L H'
>> Here's a real header structure as it appears at the head of a file:
>>
>>  0000000  000d  0001  0006  0008
>>  0000010  4240  000f  0000  0000
>>  0000020  0000  07da  0064  4730
>>  0000030  0001  0000  0000  0000
>>  0000040  d800  d31d  421d  03e8
>>  0000048  0000  0000  0000  0002
>>
>> Decoded as unsigned shorts:
>>
>>  0000000    13     1     6     8
>>  0000010 16960    15     0     0
>>  0000020     0  2010   100 18224
>>  0000030     1     0     0     0
>>  0000040 55296 54045 16925  1000
>>  0000050     0     0     0     2
>>
>> Matching these to the stats_hdr_t with 'unpack' notation:
>>
>>  0000000     H     H     H     H
>>  0000010    L1    L2     H     ?
>>  0000020     ?     H     H    L1
>>  0000030    L2     ?     ?    D1
>>  0000040    D2    D3    D4    L1
>>  0000050    L2     ?     ?     H
>>
>> So the actual format is 'HHHH L H xxxx HH L xxxx d L xxxx H'
>> What are all the mystery 4-byte blanks?  This works:
>>
>> buf = fd.read(50)
>> header = unpack_from('=4H LH2x 2x2HL4xdL4xH',buf)
>>
>> Since unpacking binary data must be a fairly common activity in
>> scientific circles. I hope you will have some suggestions.
>>
>
> I presume you didn't produce the data, but as a rule of thumb c structures
> should not be used to write out binary data, as the binary layout of the
> data won't be portable. Text, netcdf, hdf5, or some other standard data
> format is preferable, with text being perhaps the most portable. That said,
> lots of old data collection programs write out c structures, and no doubt
> newer programs do so also.

There's also a FORTRAN binary format which one program I have to cope
with uses; the exact layout of those data files depends on the
compiler (g77 vs. gfortran) as well as the hardware. I'd also add FITS
to the list of self-describing portable binary formats that python
supports well.


Anne

> Chuck
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From charlesr.harris at gmail.com  Tue Apr 13 11:57:26 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 13 Apr 2010 09:57:26 -0600
Subject: [SciPy-User] SciPy-User Digest, Vol 80, Issue 22
In-Reply-To: <201004131145.51056.sransom@nrao.edu>
References: <mailman.226.1271169703.28903.scipy-user@scipy.org>
	<4BC49029.3060201@jpl.nasa.gov> <201004131145.51056.sransom@nrao.edu>
Message-ID: <q2je06186141004130857nea9a2ba4s65cf4f7fdc157ffc@mail.gmail.com>

On Tue, Apr 13, 2010 at 9:45 AM, Scott Ransom <sransom at nrao.edu> wrote:

> On Tuesday 13 April 2010 11:39:21 am Tom Kuiper wrote:
> > Thanks for the tip Robert.
> >
> > On my 32-bit laptop:
> > In [11]: calcsize('HHHH L HHH L d L H')
> > Out[11]: 38
> >
> > On a 64-bit machine:
> > In [3]: calcsize('HHHH L HHH L d L H')
> > Out[3]: 50
>
> Hi Tom,
>
> That's probably not because of the padding.  It is likely due to the
> fact that longs are 4 bytes on 32-bit machines and 8 bytes on 64-bit
> machines.
>
>
Unless you are using MSVC on 64 bit windows, in which case they are still 32
bits, IIRC.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100413/87823aaf/attachment.html>

From kuiper at jpl.nasa.gov  Tue Apr 13 12:22:12 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Tue, 13 Apr 2010 09:22:12 -0700
Subject: [SciPy-User] SciPy-User Digest, Vol 80, Issue 22
In-Reply-To: <201004131145.51056.sransom@nrao.edu>
References: <mailman.226.1271169703.28903.scipy-user@scipy.org>
	<4BC49029.3060201@jpl.nasa.gov>
	<201004131145.51056.sransom@nrao.edu>
Message-ID: <4BC49A34.4020007@jpl.nasa.gov>

Scott Ransom wrote:
> On Tuesday 13 April 2010 11:39:21 am Tom Kuiper wrote:
>   
>> Thanks for the tip Robert.
>>
>> On my 32-bit laptop:
>> In [11]: calcsize('HHHH L HHH L d L H')
>> Out[11]: 38
>>
>> On a 64-bit machine:
>> In [3]: calcsize('HHHH L HHH L d L H')
>> Out[3]: 50
>>     
>
> That's probably not because of the padding.  It is likely due to the 
> fact that longs are 4 bytes on 32-bit machines and 8 bytes on 64-bit 
> machines.
>
> Scott
Hi Scott,

I thought of that.  4xL would be the same as Q.  But I'm confused that 
the 'pads' are not always adjacent to the longs.

This works on my 32-bit machine.  (The file was written by the 64-bit 
machine.)

header = unpack_from('=4H L H 4x 2H L 4x d L 4x H',buf)

If I changed all the longs to long longs on my 32-bit machine, I don't 
think it would solve the problem.

Regards

Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100413/d31c4c70/attachment.html>

From sransom at nrao.edu  Tue Apr 13 13:13:04 2010
From: sransom at nrao.edu (Scott Ransom)
Date: Tue, 13 Apr 2010 13:13:04 -0400
Subject: [SciPy-User] SciPy-User Digest, Vol 80, Issue 22
In-Reply-To: <4BC49A34.4020007@jpl.nasa.gov>
References: <mailman.226.1271169703.28903.scipy-user@scipy.org>
	<201004131145.51056.sransom@nrao.edu>
	<4BC49A34.4020007@jpl.nasa.gov>
Message-ID: <201004131313.04930.sransom@nrao.edu>


On Tuesday 13 April 2010 12:22:12 pm Tom Kuiper wrote:
> Scott Ransom wrote:
> > On Tuesday 13 April 2010 11:39:21 am Tom Kuiper wrote:
> >> Thanks for the tip Robert.
> >>
> >> On my 32-bit laptop:
> >> In [11]: calcsize('HHHH L HHH L d L H')
> >> Out[11]: 38
> >>
> >> On a 64-bit machine:
> >> In [3]: calcsize('HHHH L HHH L d L H')
> >> Out[3]: 50
> >
> > That's probably not because of the padding.  It is likely due to the
> > fact that longs are 4 bytes on 32-bit machines and 8 bytes on 64-bit
> > machines.
> >
> > Scott
> 
> Hi Scott,
> 
> I thought of that.  4xL would be the same as Q.  But I'm confused that
> the 'pads' are not always adjacent to the longs.
> 
> This works on my 32-bit machine.  (The file was written by the 64-bit
> machine.)
> 
> header = unpack_from('=4H L H 4x 2H L 4x d L 4x H',buf)
> 
> If I changed all the longs to long longs on my 32-bit machine, I don't
> think it would solve the problem.
> 
> Regards
> 
> Tom

I think most padding is actually done next to the shorts.  That is because 
the compiler tries to make all of the data align on 4- or more typically 
8-byte boundaries.  Therefore, if you rearrange and put all the longs 
first, they will naturally be aligned (esp on 64-bit platforms) and the 
first short afterwards will also be aligned automatically.  If you have an 
odd number of shorts, though, there will likely be padding at the end of 
the shorts if you have another datatype afterwards.

Scott



-- 
Scott M. Ransom            Address:  NRAO
Phone:  (434) 296-0320               520 Edgemont Rd.
email:  sransom at nrao.edu             Charlottesville, VA 22903 USA
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989


From itsdho at ucla.edu  Tue Apr 13 14:52:35 2010
From: itsdho at ucla.edu (David Ho)
Date: Tue, 13 Apr 2010 11:52:35 -0700
Subject: [SciPy-User] How do I use vonmises.fit()?
In-Reply-To: <1cd32cbb1003301351q3a865878u1c4775c68a8547e3@mail.gmail.com>
References: <469ba0f41003301210x5bfb6bd7w364123db34f1f54a@mail.gmail.com> 
	<1cd32cbb1003301334q6b990ff5yc31482d4e1e28a87@mail.gmail.com> 
	<1cd32cbb1003301351q3a865878u1c4775c68a8547e3@mail.gmail.com>
Message-ID: <z2y469ba0f41004131152udbc25bdctea2c509d721b25b7@mail.gmail.com>

Setting scipy.stats.distributions.vonmises.a and
scipy.stats.distributions.vonmises.b seems to work well; thanks for your
help!
For the record, here's what I ended up doing.
I had to manually shift the data so that the mean was 0, perform the
fitting, and then shift everything back.
(Actually, I originally expected vonmises.fit() to do something like that in
the first place.)

>>> import matplotlib.pyplot as mp
>>> import scipy.stats
>>> scipy.stats.distributions.vonmises.a = -numpy.pi
>>> scipy.stats.distributions.vonmises.b = numpy.pi
>>>
>>> def fit_vonmises(spike_phases):
>>>
>>>     # the values in spike_phases originally lie between 0 and 2pi.
>>>     # we want the values in shifted_spike_phases to lie between -pi and
pi, with the mean at 0.
>>>     mean_spike_phase = scipy.stats.circmean(spike_phases)
>>>     shifted_spike_phases = numpy.angle(numpy.exp(1j*(spike_phases -
mean_spike_phase)))
>>>     # that's just the "circular difference" between spike_phases and
mean_spike_phase.
>>>
>>>     # plot spike_phases and shifted_spike_phases.
>>>     mp.figure()
>>>     mp.subplot(211)
>>>     mp.hist(spike_phases, numpy.linspace(0, 2*numpy.pi, 37),
facecolor='b')
>>>     mp.axvline(mean_spike_phase, color='r')
>>>     mp.axis(xmin=-numpy.pi, xmax=2*numpy.pi)
>>>     mp.subplot(212)
>>>     mp.hist(shifted_spike_phases, numpy.linspace(-numpy.pi, numpy.pi,
37), facecolor='g')
>>>     mp.axvline(0, color='r')
>>>     mp.axis(xmin=-numpy.pi, xmax=2*numpy.pi)
>>>
>>>     # fit a von mises distribution to spike_phases_shifted.
>>>     pars = scipy.stats.distributions.vonmises.fit(shifted_spike_phases)
>>>     xs = numpy.linspace(-numpy.pi, numpy.pi, 361)
>>>     ys = scipy.stats.distributions.vonmises.pdf(xs, pars[0],
loc=pars[1], scale=pars[2])
>>>
>>>     # plot the fitted distribution on top of shifted_spike_phases.
>>>     mp.figure()
>>>     mp.hist(shifted_spike_phases, numpy.linspace(-numpy.pi, numpy.pi,
37), facecolor='g')
>>>     mp.axis(xmin=-numpy.pi, xmax=numpy.pi)
>>>     mp.twinx()
>>>     mp.axis(xmin=-numpy.pi, xmax=numpy.pi)
>>>     mp.plot(xs, ys, 'r') # but now, the indices of ys match up with xs,
which goes from -pi to pi.
>>>
>>>     # since the original data goes from 0 to 2pi, we have to shift
everything back by mean_spike_phase.
>>>     shifted_xs = numpy.mod(xs + mean_spike_phase, 2*numpy.pi) # use
mod() to shift around a circle.
>>>     shifted_indices = numpy.argsort(shifted_xs)
>>>     shifted_ys = ys[shifted_indices]
>>>
>>>     # now, plot the fitted distribution on top of the original
spike_phases.
>>>     mp.figure()
>>>     mp.hist(spike_phases, numpy.linspace(0, 2*numpy.pi, 37),
facecolor='b')
>>>     mp.axis(xmin=0, xmax=2*numpy.pi)
>>>     mp.twinx()
>>>     mp.plot(shifted_xs[shifted_indices], shifted_ys, 'r')
>>>     mp.axis(xmin=0, xmax=2*numpy.pi)
>>>
>>>     return (pars[0], mean_spike_phase, pars[2])
>>>
>>> spike_phases = numpy.mod(scipy.stats.distributions.vonmises.rvs(1.23,
loc=5.67, scale=1, size=1000), 2*numpy.pi)
>>> fit_vonmises(spike_phases)

It still seems a little bit hacky, so let me know if you can think of a
better way. =)

--David Ho


On Tue, Mar 30, 2010 at 1:51 PM, <josef.pktd at gmail.com> wrote:
>
> On Tue, Mar 30, 2010 at 4:34 PM,  <josef.pktd at gmail.com> wrote:
> > On Tue, Mar 30, 2010 at 3:10 PM, David Ho <itsdho at ucla.edu> wrote:
> >> Hi all!
> >>
> >> I want to fit some data with a Von Mises distribution, and get the mean
and
> >> "kappa" parameters for that distribution.
> >> I'm a little confused about scipy.stats.distributions.vonmises.fit().
> >>
> >> Just as a test, I tried this:
> >>
> >>>>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000) # the first
> >>>>> argument appears to be "kappa", and the second argument is the mean.
> >>>>> scipy.stats.distributions.vonmises.fit(vm_rvs)
> >> array([  1.17643696e-01,   3.38956854e-03,   1.27331662e-27])
> >>
> >> I got an array of 3 values, none of which seem to be equal to the mean
or
> >> kappa of the distribution.
> >> I looked around in some of the docstrings, but I couldn't find any
clear
> >> documentation of what these values are supposed to correspond to.
> >>
> >> I would've expected vonmises.fit() to return something like:
> >> array([  1.2,   2.3])
> >>
> >> What am I doing wrong?
> >> Thanks for your help,
> >
> > When I did I check of the fit method for all distributions, then
> > vonmises most of the time didn't produce any useful results,
> > especially estimated scale of almost zero.
> >
> > The problem is that vonmises doesn't have a well defined pdf on the real
line
> >
> >>>> import matplotlib.pyplot as plt
> >>>> plt.plot(scipy.stats.vonmises.pdf(np.linspace(-10,10),1.2, loc=2.3))
> >>>> plt.show()
> >
> > Since vonmises is intended for circular data, and I don't know much
> > about circular statistics (except for what Anne explained on the
> > mailing list), I never tried to get it to work like the other
> > distributions.
> >
> > It might be possible to patch vonmises to work on the real line but I
> > never tried.
>
> Here is a way that seems to work
> * set bounds a,b
> * use mean of random variable as starting value to start inside the
> support of the distribution
>
> >>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000)
> >>> stats.vonmises.a = -np.pi
> >>> stats.vonmises.b = np.pi
> >>> stats.vonmises.fit(vm_rvs,1,loc=vm_rvs.mean(),scale=1)
> array([ 1.19767227,  2.30077064,  0.99916372])
>
> Josef
>
>
> >
> > Josef
> >
> >>
> >> --David Ho
> >>
> >> PS: I also don't fully understand the "scale" and "loc" parameters for
all
> >> of the continuous random variables.
> >> Does "loc" always correspond to the mean, and "scale" always correspond
to
> >> the square root of the variance, or something else?
> >
> > loc=0, scale=1 is the standard distribution, the loc and scale
> > parameters transform the distribution of the transformation x~ =
> > (x-loc)/scale
> >
> > If the standard distribution has mean or variance different from 0, 1
> > then the transformed distribution has mean and variance different from
> > loc, scale.
> > the stats method shows the implied variance scale:
> >
> >>>> scipy.stats.vonmises.stats(1.2, loc=2.3, scale=2)
> > (array(2.2999999999999998), array(5.4900668161122423))
> >>>> scipy.stats.vonmises.stats(1.2, loc=0, scale=1)
> > (array(1.5832118030775403e-017), array(1.3725167040280606))
> >
> >
> >>
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>
> >>
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100413/cc458367/attachment.html>

From josef.pktd at gmail.com  Tue Apr 13 15:13:18 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 13 Apr 2010 15:13:18 -0400
Subject: [SciPy-User] How do I use vonmises.fit()?
In-Reply-To: <z2y469ba0f41004131152udbc25bdctea2c509d721b25b7@mail.gmail.com>
References: <469ba0f41003301210x5bfb6bd7w364123db34f1f54a@mail.gmail.com>
	<1cd32cbb1003301334q6b990ff5yc31482d4e1e28a87@mail.gmail.com>
	<1cd32cbb1003301351q3a865878u1c4775c68a8547e3@mail.gmail.com>
	<z2y469ba0f41004131152udbc25bdctea2c509d721b25b7@mail.gmail.com>
Message-ID: <i2y1cd32cbb1004131213mad00ff38o61ed49f300fdaccc@mail.gmail.com>

On Tue, Apr 13, 2010 at 2:52 PM, David Ho <itsdho at ucla.edu> wrote:
> Setting scipy.stats.distributions.vonmises.a and
> scipy.stats.distributions.vonmises.b seems to work well; thanks for your
> help!
> For the record, here's what I ended up doing.
> I had to manually shift the data so that the mean was 0, perform the
> fitting, and then shift everything back.
> (Actually, I originally expected vonmises.fit() to do something like that in
> the first place.)
>
>>>> import matplotlib.pyplot as mp
>>>> import scipy.stats
>>>> scipy.stats.distributions.vonmises.a = -numpy.pi
>>>> scipy.stats.distributions.vonmises.b = numpy.pi
>>>>
>>>> def fit_vonmises(spike_phases):
>>>>
>>>> ??? # the values in spike_phases originally lie between 0 and 2pi.
>>>> ??? # we want the values in shifted_spike_phases to lie between -pi and
>>>> pi, with the mean at 0.
>>>> ??? mean_spike_phase = scipy.stats.circmean(spike_phases)
>>>> ??? shifted_spike_phases = numpy.angle(numpy.exp(1j*(spike_phases -
>>>> mean_spike_phase)))
>>>> ??? # that's just the "circular difference" between spike_phases and
>>>> mean_spike_phase.
>>>>
>>>> ??? # plot spike_phases and shifted_spike_phases.
>>>> ??? mp.figure()
>>>> ??? mp.subplot(211)
>>>> ??? mp.hist(spike_phases, numpy.linspace(0, 2*numpy.pi, 37),
>>>> facecolor='b')
>>>> ??? mp.axvline(mean_spike_phase, color='r')
>>>> ??? mp.axis(xmin=-numpy.pi, xmax=2*numpy.pi)
>>>> ??? mp.subplot(212)
>>>> ??? mp.hist(shifted_spike_phases, numpy.linspace(-numpy.pi, numpy.pi,
>>>> 37), facecolor='g')
>>>> ??? mp.axvline(0, color='r')
>>>> ??? mp.axis(xmin=-numpy.pi, xmax=2*numpy.pi)
>>>>
>>>> ??? # fit a von mises distribution to spike_phases_shifted.
>>>> ??? pars = scipy.stats.distributions.vonmises.fit(shifted_spike_phases)
>>>> ??? xs = numpy.linspace(-numpy.pi, numpy.pi, 361)
>>>> ??? ys = scipy.stats.distributions.vonmises.pdf(xs, pars[0],
>>>> loc=pars[1], scale=pars[2])
>>>>
>>>> ??? # plot the fitted distribution on top of shifted_spike_phases.
>>>> ??? mp.figure()
>>>> ??? mp.hist(shifted_spike_phases, numpy.linspace(-numpy.pi, numpy.pi,
>>>> 37), facecolor='g')
>>>> ??? mp.axis(xmin=-numpy.pi, xmax=numpy.pi)
>>>> ??? mp.twinx()
>>>> ??? mp.axis(xmin=-numpy.pi, xmax=numpy.pi)
>>>> ??? mp.plot(xs, ys, 'r') # but now, the indices of ys match up with xs,
>>>> which goes from -pi to pi.
>>>>
>>>> ??? # since the original data goes from 0 to 2pi, we have to shift
>>>> everything back by mean_spike_phase.
>>>> ??? shifted_xs = numpy.mod(xs + mean_spike_phase, 2*numpy.pi) # use
>>>> mod() to shift around a circle.
>>>> ??? shifted_indices = numpy.argsort(shifted_xs)
>>>> ??? shifted_ys = ys[shifted_indices]
>>>>
>>>> ??? # now, plot the fitted distribution on top of the original
>>>> spike_phases.
>>>> ??? mp.figure()
>>>> ??? mp.hist(spike_phases, numpy.linspace(0, 2*numpy.pi, 37),
>>>> facecolor='b')
>>>> ??? mp.axis(xmin=0, xmax=2*numpy.pi)
>>>> ??? mp.twinx()
>>>> ??? mp.plot(shifted_xs[shifted_indices], shifted_ys, 'r')
>>>> ??? mp.axis(xmin=0, xmax=2*numpy.pi)
>>>>
>>>> ??? return (pars[0], mean_spike_phase, pars[2])
>>>>
>>>> spike_phases = numpy.mod(scipy.stats.distributions.vonmises.rvs(1.23,
>>>> loc=5.67, scale=1, size=1000), 2*numpy.pi)
>>>> fit_vonmises(spike_phases)
>
> It still seems a little bit hacky, so let me know if you can think of a
> better way. =)

Just a quick reply, I haven't read the script very carefully

loc is shifting the location, scale is extending the support which
will be 2*pi*scale.

in the way I did it originally with starting values
stats.vonmises.fit(vm_rvs,1,loc=vm_rvs.mean(),scale=1)
the estimation is supposed to do the location change automatically. I
think your shifting is essentially the same as what the distributions
do internally with loc.

In the current fit version, loc (and with it the support of the
distribution) and scale are always estimated. In some cases this is
not desired.
You are transforming the original data to fit into the standard
distribution with loc=0, scale=1 . Do you get reasonable estimates for
loc and scale in this case?
If not, then there is another patch or enhanced fit function that
could take loc and scale as fixed.

I will look at some details in your function later, especially I'm
curious how the circular statistics works.

Thanks for the example, maybe I can stop ignoring vonmises.

Josef

>
> --David Ho
>
>
> On Tue, Mar 30, 2010 at 1:51 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Tue, Mar 30, 2010 at 4:34 PM, ?<josef.pktd at gmail.com> wrote:
>> > On Tue, Mar 30, 2010 at 3:10 PM, David Ho <itsdho at ucla.edu> wrote:
>> >> Hi all!
>> >>
>> >> I want to fit some data with a Von Mises distribution, and get the mean
>> >> and
>> >> "kappa" parameters for that distribution.
>> >> I'm a little confused about scipy.stats.distributions.vonmises.fit().
>> >>
>> >> Just as a test, I tried this:
>> >>
>> >>>>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000) # the first
>> >>>>> argument appears to be "kappa", and the second argument is the mean.
>> >>>>> scipy.stats.distributions.vonmises.fit(vm_rvs)
>> >> array([? 1.17643696e-01,?? 3.38956854e-03,?? 1.27331662e-27])
>> >>
>> >> I got an array of 3 values, none of which seem to be equal to the mean
>> >> or
>> >> kappa of the distribution.
>> >> I looked around in some of the docstrings, but I couldn't find any
>> >> clear
>> >> documentation of what these values are supposed to correspond to.
>> >>
>> >> I would've expected vonmises.fit() to return something like:
>> >> array([? 1.2,?? 2.3])
>> >>
>> >> What am I doing wrong?
>> >> Thanks for your help,
>> >
>> > When I did I check of the fit method for all distributions, then
>> > vonmises most of the time didn't produce any useful results,
>> > especially estimated scale of almost zero.
>> >
>> > The problem is that vonmises doesn't have a well defined pdf on the real
>> > line
>> >
>> >>>> import matplotlib.pyplot as plt
>> >>>> plt.plot(scipy.stats.vonmises.pdf(np.linspace(-10,10),1.2, loc=2.3))
>> >>>> plt.show()
>> >
>> > Since vonmises is intended for circular data, and I don't know much
>> > about circular statistics (except for what Anne explained on the
>> > mailing list), I never tried to get it to work like the other
>> > distributions.
>> >
>> > It might be possible to patch vonmises to work on the real line but I
>> > never tried.
>>
>> Here is a way that seems to work
>> * set bounds a,b
>> * use mean of random variable as starting value to start inside the
>> support of the distribution
>>
>> >>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000)
>> >>> stats.vonmises.a = -np.pi
>> >>> stats.vonmises.b = np.pi
>> >>> stats.vonmises.fit(vm_rvs,1,loc=vm_rvs.mean(),scale=1)
>> array([ 1.19767227, ?2.30077064, ?0.99916372])
>>
>> Josef
>>
>>
>> >
>> > Josef
>> >
>> >>
>> >> --David Ho
>> >>
>> >> PS: I also don't fully understand the "scale" and "loc" parameters for
>> >> all
>> >> of the continuous random variables.
>> >> Does "loc" always correspond to the mean, and "scale" always correspond
>> >> to
>> >> the square root of the variance, or something else?
>> >
>> > loc=0, scale=1 is the standard distribution, the loc and scale
>> > parameters transform the distribution of the transformation x~ =
>> > (x-loc)/scale
>> >
>> > If the standard distribution has mean or variance different from 0, 1
>> > then the transformed distribution has mean and variance different from
>> > loc, scale.
>> > the stats method shows the implied variance scale:
>> >
>> >>>> scipy.stats.vonmises.stats(1.2, loc=2.3, scale=2)
>> > (array(2.2999999999999998), array(5.4900668161122423))
>> >>>> scipy.stats.vonmises.stats(1.2, loc=0, scale=1)
>> > (array(1.5832118030775403e-017), array(1.3725167040280606))
>> >
>> >
>> >>
>> >> _______________________________________________
>> >> SciPy-User mailing list
>> >> SciPy-User at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/scipy-user
>> >>
>> >>
>> >
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From kuiper at jpl.nasa.gov  Tue Apr 13 15:24:06 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Tue, 13 Apr 2010 12:24:06 -0700
Subject: [SciPy-User] unpacking binary data from a C structure
In-Reply-To: <mailman.240.1271175771.28903.scipy-user@scipy.org>
References: <mailman.240.1271175771.28903.scipy-user@scipy.org>
Message-ID: <4BC4C4D6.1010109@jpl.nasa.gov>

scipy-user-request at scipy.org wrote:
> Send SciPy-User mailing list submissions to
>         scipy-user at scipy.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mail.scipy.org/mailman/listinfo/scipy-user
> or, via email, send a message with subject or body 'help' to
>         scipy-user-request at scipy.org
>
> You can reach the person managing the list at
>         scipy-user-owner at scipy.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of SciPy-User digest..."
>
>
> Today's Topics:
>
>    1. Re: unpacking binary data from a C structure (Scott Ransom)
>    2. Re: unpacking binary data from a C structure (Jason Grout)
>    3. Re: unpacking binary data from a C structure (Charles R Harris)
>    4. Re: SciPy-User Digest, Vol 80, Issue 22 (Tom Kuiper)
>    5. Re: SciPy-User Digest, Vol 80, Issue 22 (Scott Ransom)
>    6. Re: unpacking binary data from a C structure (Anne Archibald)
>    7. Re: SciPy-User Digest, Vol 80, Issue 22 (Charles R Harris)
>    8. Re: SciPy-User Digest, Vol 80, Issue 22 (Tom Kuiper)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 13 Apr 2010 10:57:38 -0400
> From: Scott Ransom <sransom at nrao.edu>
> Subject: Re: [SciPy-User] unpacking binary data from a C structure
> To: scipy-user at scipy.org
> Message-ID: <201004131057.38774.sransom at nrao.edu>
> Content-Type: Text/Plain;  charset="utf-8"
>
> On Tuesday 13 April 2010 10:41:12 am Robert Kern wrote:
>   
>> On Tue, Apr 13, 2010 at 08:20, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
>>     
>>> Dear list,
>>>
>>> here's something I find very strange.  I have a C structure defined
>>> as:
>>>
>>> typedef struct
>>> {
>>>  unsigned short spcid;         /* station id - 10, 40, 60, 21 */
>>>  unsigned short vsrid;         /* vsr1a, vsr1b ... from enum */
>>>  unsigned short chanid;        /* subchannel id 0,1,2,3 */
>>>  unsigned short bps;           /* number of bits per sample - 1, 2,
>>> 4, 8, or
>>>                                   16 */
>>>  unsigned long  srate;         /* number of samples per second in
>>> kilo-samples
>>>                                   per second */
>>>  unsigned short error;         /* hw err flag, dma error or
>>> num_samples error,
>>>                                   0 ==> no errors */
>>>  unsigned short year;          /* time tag - year */
>>>  unsigned short doy;           /* time tag - day of year */
>>>  unsigned long  sec;           /* time tag - second of day */
>>>  double         freq;          /* in Hz */
>>>  unsigned long  orate;         /* number of statistics samples per
>>> second */
>>>  unsigned short nsubchan;      /* number of output sub chans */
>>> }
>>> stats_hdr_t;
>>>
>>> The python module struct unpack expected format is 'HHHH L HHH L d
>>> L H' Here's a real header structure as it appears at the head of a
>>> file:
>>>
>>>  0000000  000d  0001  0006  0008
>>>  0000010  4240  000f  0000  0000
>>>  0000020  0000  07da  0064  4730
>>>  0000030  0001  0000  0000  0000
>>>  0000040  d800  d31d  421d  03e8
>>>  0000048  0000  0000  0000  0002
>>>
>>> Decoded as unsigned shorts:
>>>
>>>  0000000    13     1     6     8
>>>  0000010 16960    15     0     0
>>>  0000020     0  2010   100 18224
>>>  0000030     1     0     0     0
>>>  0000040 55296 54045 16925  1000
>>>  0000050     0     0     0     2
>>>
>>> Matching these to the stats_hdr_t with 'unpack' notation:
>>>
>>>  0000000     H     H     H     H
>>>  0000010    L1    L2     H     ?
>>>  0000020     ?     H     H    L1
>>>  0000030    L2     ?     ?    D1
>>>  0000040    D2    D3    D4    L1
>>>  0000050    L2     ?     ?     H
>>>
>>> So the actual format is 'HHHH L H xxxx HH L xxxx d L xxxx H'
>>> What are all the mystery 4-byte blanks?  This works:
>>>
>>> buf = fd.read(50)
>>> header = unpack_from('=4H LH2x 2x2HL4xdL4xH',buf)
>>>
>>> Since unpacking binary data must be a fairly common activity in
>>> scientific circles. I hope you will have some suggestions.
>>>       
>> C compilers can insert padding into the memory layout of a structure
>> in order to align certain members to certain boundaries, particularly
>> doubles. This behavior is compiler- and platform-specific.
>>     
>
> If the exact order of the elements in the structure is not important,
> you can mitigate against this (but not prevent it entirely) by putting
> common types of structure elements together, with the largest ones
> first.  In your case, that would mean grouping all the longs first and
> then putting all the shorts together second.
>
> Scott
>
>
> --
> Scott M. Ransom            Address:  NRAO
> Phone:  (434) 296-0320               520 Edgemont Rd.
> email:  sransom at nrao.edu             Charlottesville, VA 22903 USA
> GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 13 Apr 2010 10:06:24 -0500
> From: Jason Grout <jason-sage at creativetrax.com>
>   
...
> If you are using gcc, this might be relevant:
>
> http://sig9.com/articles/gcc-packed-structures
>   
I've bookmarked that.  Thanks.
> Date: Tue, 13 Apr 2010 09:27:55 -0600
> From: Charles R Harris <charlesr.harris at gmail.com>
>   
....
> I presume you didn't produce the data,
A colleague did.  I'm encouraging him to re-compile as per the above.
>  but as a rule of thumb c structures
> should not be used to write out binary data, as the binary layout of the
> data won't be portable. Text, netcdf, hdf5, or some other standard data
> format is preferable, with text being perhaps the most portable. That said,
> lots of old data collection programs write out c structures, and no doubt
> newer programs do so also.
>
> Chuck
>   
....
> Date: Tue, 13 Apr 2010 11:55:53 -0400
> From: Anne Archibald <peridot.faceted at gmail.com>
>   
...
> There's also a FORTRAN binary format which one program I have to cope
> with uses; the exact layout of those data files depends on the
> compiler (g77 vs. gfortran) as well as the hardware. I'd also add FITS
> to the list of self-describing portable binary formats that python
> supports well.
>   
Writing the data in FITS was on my TODO list.  Unfortunately,  the 
amount of data to be written is not known beforehand and is very large. 
5-6 GB is not atypical.  I was going to write an intermediate file and, 
then once I know how much data I have, convert it to FITS.  I was doing 
Python to have some sneak peeks at subsets of the data.
> Date: Tue, 13 Apr 2010 09:57:26 -0600
> From: Charles R Harris <charlesr.harris at gmail.com>
>   
...
>> That's probably not because of the padding.  It is likely due to the
>> fact that longs are 4 bytes on 32-bit machines and 8 bytes on 64-bit
>> machines.
>>     
> Unless you are using MSVC on 64 bit windows,
Never!
> in which case they are still 32 bits, IIRC.
>   
Thanks for the warning.

Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100413/149b7a59/attachment.html>

From josef.pktd at gmail.com  Tue Apr 13 15:59:32 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 13 Apr 2010 15:59:32 -0400
Subject: [SciPy-User] How do I use vonmises.fit()?
In-Reply-To: <i2y1cd32cbb1004131213mad00ff38o61ed49f300fdaccc@mail.gmail.com>
References: <469ba0f41003301210x5bfb6bd7w364123db34f1f54a@mail.gmail.com>
	<1cd32cbb1003301334q6b990ff5yc31482d4e1e28a87@mail.gmail.com>
	<1cd32cbb1003301351q3a865878u1c4775c68a8547e3@mail.gmail.com>
	<z2y469ba0f41004131152udbc25bdctea2c509d721b25b7@mail.gmail.com>
	<i2y1cd32cbb1004131213mad00ff38o61ed49f300fdaccc@mail.gmail.com>
Message-ID: <v2u1cd32cbb1004131259t26a0ac82ud7fb9551d7084614@mail.gmail.com>

On Tue, Apr 13, 2010 at 3:13 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Apr 13, 2010 at 2:52 PM, David Ho <itsdho at ucla.edu> wrote:
>> Setting scipy.stats.distributions.vonmises.a and
>> scipy.stats.distributions.vonmises.b seems to work well; thanks for your
>> help!
>> For the record, here's what I ended up doing.
>> I had to manually shift the data so that the mean was 0, perform the
>> fitting, and then shift everything back.
>> (Actually, I originally expected vonmises.fit() to do something like that in
>> the first place.)
>>
>>>>> import matplotlib.pyplot as mp
>>>>> import scipy.stats
>>>>> scipy.stats.distributions.vonmises.a = -numpy.pi
>>>>> scipy.stats.distributions.vonmises.b = numpy.pi
>>>>>
>>>>> def fit_vonmises(spike_phases):
>>>>>
>>>>> ??? # the values in spike_phases originally lie between 0 and 2pi.
>>>>> ??? # we want the values in shifted_spike_phases to lie between -pi and
>>>>> pi, with the mean at 0.
>>>>> ??? mean_spike_phase = scipy.stats.circmean(spike_phases)
>>>>> ??? shifted_spike_phases = numpy.angle(numpy.exp(1j*(spike_phases -
>>>>> mean_spike_phase)))
>>>>> ??? # that's just the "circular difference" between spike_phases and
>>>>> mean_spike_phase.
>>>>>
>>>>> ??? # plot spike_phases and shifted_spike_phases.
>>>>> ??? mp.figure()
>>>>> ??? mp.subplot(211)
>>>>> ??? mp.hist(spike_phases, numpy.linspace(0, 2*numpy.pi, 37),
>>>>> facecolor='b')
>>>>> ??? mp.axvline(mean_spike_phase, color='r')
>>>>> ??? mp.axis(xmin=-numpy.pi, xmax=2*numpy.pi)
>>>>> ??? mp.subplot(212)
>>>>> ??? mp.hist(shifted_spike_phases, numpy.linspace(-numpy.pi, numpy.pi,
>>>>> 37), facecolor='g')
>>>>> ??? mp.axvline(0, color='r')
>>>>> ??? mp.axis(xmin=-numpy.pi, xmax=2*numpy.pi)
>>>>>
>>>>> ??? # fit a von mises distribution to spike_phases_shifted.
>>>>> ??? pars = scipy.stats.distributions.vonmises.fit(shifted_spike_phases)
>>>>> ??? xs = numpy.linspace(-numpy.pi, numpy.pi, 361)
>>>>> ??? ys = scipy.stats.distributions.vonmises.pdf(xs, pars[0],
>>>>> loc=pars[1], scale=pars[2])
>>>>>
>>>>> ??? # plot the fitted distribution on top of shifted_spike_phases.
>>>>> ??? mp.figure()
>>>>> ??? mp.hist(shifted_spike_phases, numpy.linspace(-numpy.pi, numpy.pi,
>>>>> 37), facecolor='g')
>>>>> ??? mp.axis(xmin=-numpy.pi, xmax=numpy.pi)
>>>>> ??? mp.twinx()
>>>>> ??? mp.axis(xmin=-numpy.pi, xmax=numpy.pi)
>>>>> ??? mp.plot(xs, ys, 'r') # but now, the indices of ys match up with xs,
>>>>> which goes from -pi to pi.
>>>>>
>>>>> ??? # since the original data goes from 0 to 2pi, we have to shift
>>>>> everything back by mean_spike_phase.
>>>>> ??? shifted_xs = numpy.mod(xs + mean_spike_phase, 2*numpy.pi) # use
>>>>> mod() to shift around a circle.
>>>>> ??? shifted_indices = numpy.argsort(shifted_xs)
>>>>> ??? shifted_ys = ys[shifted_indices]
>>>>>
>>>>> ??? # now, plot the fitted distribution on top of the original
>>>>> spike_phases.
>>>>> ??? mp.figure()
>>>>> ??? mp.hist(spike_phases, numpy.linspace(0, 2*numpy.pi, 37),
>>>>> facecolor='b')
>>>>> ??? mp.axis(xmin=0, xmax=2*numpy.pi)
>>>>> ??? mp.twinx()
>>>>> ??? mp.plot(shifted_xs[shifted_indices], shifted_ys, 'r')
>>>>> ??? mp.axis(xmin=0, xmax=2*numpy.pi)
>>>>>
>>>>> ??? return (pars[0], mean_spike_phase, pars[2])
>>>>>
>>>>> spike_phases = numpy.mod(scipy.stats.distributions.vonmises.rvs(1.23,
>>>>> loc=5.67, scale=1, size=1000), 2*numpy.pi)
>>>>> fit_vonmises(spike_phases)
>>
>> It still seems a little bit hacky, so let me know if you can think of a
>> better way. =)
>
> Just a quick reply, I haven't read the script very carefully
>
> loc is shifting the location, scale is extending the support which
> will be 2*pi*scale.
>
> in the way I did it originally with starting values
> stats.vonmises.fit(vm_rvs,1,loc=vm_rvs.mean(),scale=1)
> the estimation is supposed to do the location change automatically. I
> think your shifting is essentially the same as what the distributions
> do internally with loc.

No, I don't think this is correct, you are using circular
transformation to shift your original data, and I don't think the
distribution would be able to do this correctly.

The results in the example look good.

All other ideas, I have, might also not be correct because I don't
have any intuition for circular statistics.

Josef


>
> In the current fit version, loc (and with it the support of the
> distribution) and scale are always estimated. In some cases this is
> not desired.
> You are transforming the original data to fit into the standard
> distribution with loc=0, scale=1 . Do you get reasonable estimates for
> loc and scale in this case?
> If not, then there is another patch or enhanced fit function that
> could take loc and scale as fixed.
>
> I will look at some details in your function later, especially I'm
> curious how the circular statistics works.
>
> Thanks for the example, maybe I can stop ignoring vonmises.
>
> Josef
>
>>
>> --David Ho
>>
>>
>> On Tue, Mar 30, 2010 at 1:51 PM, <josef.pktd at gmail.com> wrote:
>>>
>>> On Tue, Mar 30, 2010 at 4:34 PM, ?<josef.pktd at gmail.com> wrote:
>>> > On Tue, Mar 30, 2010 at 3:10 PM, David Ho <itsdho at ucla.edu> wrote:
>>> >> Hi all!
>>> >>
>>> >> I want to fit some data with a Von Mises distribution, and get the mean
>>> >> and
>>> >> "kappa" parameters for that distribution.
>>> >> I'm a little confused about scipy.stats.distributions.vonmises.fit().
>>> >>
>>> >> Just as a test, I tried this:
>>> >>
>>> >>>>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000) # the first
>>> >>>>> argument appears to be "kappa", and the second argument is the mean.
>>> >>>>> scipy.stats.distributions.vonmises.fit(vm_rvs)
>>> >> array([? 1.17643696e-01,?? 3.38956854e-03,?? 1.27331662e-27])
>>> >>
>>> >> I got an array of 3 values, none of which seem to be equal to the mean
>>> >> or
>>> >> kappa of the distribution.
>>> >> I looked around in some of the docstrings, but I couldn't find any
>>> >> clear
>>> >> documentation of what these values are supposed to correspond to.
>>> >>
>>> >> I would've expected vonmises.fit() to return something like:
>>> >> array([? 1.2,?? 2.3])
>>> >>
>>> >> What am I doing wrong?
>>> >> Thanks for your help,
>>> >
>>> > When I did I check of the fit method for all distributions, then
>>> > vonmises most of the time didn't produce any useful results,
>>> > especially estimated scale of almost zero.
>>> >
>>> > The problem is that vonmises doesn't have a well defined pdf on the real
>>> > line
>>> >
>>> >>>> import matplotlib.pyplot as plt
>>> >>>> plt.plot(scipy.stats.vonmises.pdf(np.linspace(-10,10),1.2, loc=2.3))
>>> >>>> plt.show()
>>> >
>>> > Since vonmises is intended for circular data, and I don't know much
>>> > about circular statistics (except for what Anne explained on the
>>> > mailing list), I never tried to get it to work like the other
>>> > distributions.
>>> >
>>> > It might be possible to patch vonmises to work on the real line but I
>>> > never tried.
>>>
>>> Here is a way that seems to work
>>> * set bounds a,b
>>> * use mean of random variable as starting value to start inside the
>>> support of the distribution
>>>
>>> >>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000)
>>> >>> stats.vonmises.a = -np.pi
>>> >>> stats.vonmises.b = np.pi
>>> >>> stats.vonmises.fit(vm_rvs,1,loc=vm_rvs.mean(),scale=1)
>>> array([ 1.19767227, ?2.30077064, ?0.99916372])
>>>
>>> Josef
>>>
>>>
>>> >
>>> > Josef
>>> >
>>> >>
>>> >> --David Ho
>>> >>
>>> >> PS: I also don't fully understand the "scale" and "loc" parameters for
>>> >> all
>>> >> of the continuous random variables.
>>> >> Does "loc" always correspond to the mean, and "scale" always correspond
>>> >> to
>>> >> the square root of the variance, or something else?
>>> >
>>> > loc=0, scale=1 is the standard distribution, the loc and scale
>>> > parameters transform the distribution of the transformation x~ =
>>> > (x-loc)/scale
>>> >
>>> > If the standard distribution has mean or variance different from 0, 1
>>> > then the transformed distribution has mean and variance different from
>>> > loc, scale.
>>> > the stats method shows the implied variance scale:
>>> >
>>> >>>> scipy.stats.vonmises.stats(1.2, loc=2.3, scale=2)
>>> > (array(2.2999999999999998), array(5.4900668161122423))
>>> >>>> scipy.stats.vonmises.stats(1.2, loc=0, scale=1)
>>> > (array(1.5832118030775403e-017), array(1.3725167040280606))
>>> >
>>> >
>>> >>
>>> >> _______________________________________________
>>> >> SciPy-User mailing list
>>> >> SciPy-User at scipy.org
>>> >> http://mail.scipy.org/mailman/listinfo/scipy-user
>>> >>
>>> >>
>>> >
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>


From kuiper at jpl.nasa.gov  Tue Apr 13 16:18:24 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Tue, 13 Apr 2010 13:18:24 -0700
Subject: [SciPy-User] SciPy-User Digest, Vol 80, Issue 24
In-Reply-To: <mailman.248.1271186004.28903.scipy-user@scipy.org>
References: <mailman.248.1271186004.28903.scipy-user@scipy.org>
Message-ID: <4BC4D190.1030606@jpl.nasa.gov>


> Date: Tue, 13 Apr 2010 13:13:04 -0400
> From: Scott Ransom <sransom at nrao.edu>
>   
...
> On Tuesday 13 April 2010 12:22:12 pm Tom Kuiper wrote:
>   
>> Scott Ransom wrote:
>>     
>>> On Tuesday 13 April 2010 11:39:21 am Tom Kuiper wrote:
>>>       
>>>> Thanks for the tip Robert.
>>>>
>>>> On my 32-bit laptop:
>>>> In [11]: calcsize('HHHH L HHH L d L H')
>>>> Out[11]: 38
>>>>
>>>> On a 64-bit machine:
>>>> In [3]: calcsize('HHHH L HHH L d L H')
>>>> Out[3]: 50
>>>>         
>>> That's probably not because of the padding.  It is likely due to the
>>> fact that longs are 4 bytes on 32-bit machines and 8 bytes on 64-bit
>>> machines.
>>>       
>> I thought of that.  4xL would be the same as Q.  But I'm confused that
>> the 'pads' are not always adjacent to the longs.
>>
>> This works on my 32-bit machine.  (The file was written by the 64-bit
>> machine.)
>>
>> header = unpack_from('=4H L H 4x 2H L 4x d L 4x H',buf)
>>
>> If I changed all the longs to long longs on my 32-bit machine, I don't
>> think it would solve the problem.
>>     
> I think most padding is actually done next to the shorts.  That is because
> the compiler tries to make all of the data align on 4- or more typically
> 8-byte boundaries.  Therefore, if you rearrange and put all the longs
> first, they will naturally be aligned (esp on 64-bit platforms) and the
> first short afterwards will also be aligned automatically.  If you have an
> odd number of shorts, though, there will likely be padding at the end of
> the shorts if you have another datatype afterwards.
>   
You were right the first time Scott and I should just have tried it:

  buf = fd.read(50)
  header = unpack_from('=4H Q HHH Q d Q H',buf)

SPC 13 VSR 1 channel 6 b/s 8
1000000 samps/sec
error flag 0
date/time 2010 100 83760
Frequency: 32024000000.0
Averages/sec: 1000
2 subchannels

All the right numbers without padding!

Thanks

Tom



From amenity at enthought.com  Tue Apr 13 16:30:46 2010
From: amenity at enthought.com (Amenity Applewhite)
Date: Tue, 13 Apr 2010 15:30:46 -0500
Subject: [SciPy-User] SciPy 2010 News: Specialized track deadline extended
Message-ID: <6AB8EBE6-52E1-4EC3-A9FE-76446829ABB0@enthought.com>


Have you been meaning to prepare an abstract to submit for a SciPy 2010
specialized track (http://conference.scipy.org/scipy2010/papers.html#tracks 
)?
Didn't find the time? Well you're in luck.
This weekend, we had technical issues with the email submissions for the
specialized tracks. In light of the inconvenience, we've decided to  
extend
the deadline an additional two weeks until Sunday, April 25th.

If you have an abstract ready for one of the four specialized tracks,  
please
use the links below to submit it to the program chair. If you previously
submitted one and didn't receive confirmation that we received it, it  
would
be a great idea to submit it again to ensure we get it.

   * Biomedical/bioinformatics chaired by Glen Otero, Dell
     submit/contact: 2010bioinformatics at scipy.org

   * Financial analysis chaired by Wes McKinney, AQR Capital Management
     submit/contact: 2010finance at scipy.org

   * Geophysics chaired by Alan Jackson, Shell
     submit/contact: 2010geophysics at scipy.org

   * Parallel processing & cloud computing co-chaired by Ken Elkabany,
     PiCloud & Brian Granger, CalPoly
     submit/contact: 2010parallel at scipy.org


Main Conference Submissions
Submissions for the main SciPy 2010 conference closed Sunday. Thanks to
everyone who submitted. We'll announce the accepted talks Tuesday April
20th.

Student Sponsorships
If you're an academic and contribute to SciPy or related projects,  
make sure
to apply for one of our student sponsorships. The deadline to apply is
April 18th. We are also accepting nominations.
http://conference.scipy.org/scipy2010/student.html

Don't forget to register...
Registrations are coming in pretty steadily now. Remember that to get  
early
registration prices you need to [8]register before May 10th!
https://conference.scipy.org/scipy2010/registration.html

The SciPy 2010 Team
@SciPy2010 on Twitter

--
Amenity Applewhite
Enthought, Inc.
Scientific Computing Solutions
www.enthought.com









-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100413/7ca88cf1/attachment.html>

From itsdho at ucla.edu  Wed Apr 14 00:09:47 2010
From: itsdho at ucla.edu (David Ho)
Date: Tue, 13 Apr 2010 21:09:47 -0700
Subject: [SciPy-User] Can I "fix" scale=1 when fitting distributions?
Message-ID: <k2p469ba0f41004132109tb47e72bdjbef3d5aa303beeb7@mail.gmail.com>

Actually, being able to "fix" scale=1 would be very useful for me.

The reason I'm trying to fit a von mises distribution to my data is to find
"kappa", a measure of the concentration of the data.

On wikipedia, I see that the von mises distribution only really has 2
parameters: mu (the mean), and kappa (1/kappa being analogous to the
variance).
When I use vonmises.fit(), though, I get 3 parameters: kappa, mu, and a
"scale parameter", respectively.
However, I don't think the scale parameter for the von mises distribution is
really independent of kappa, is it? (Am I understanding this correctly?)
(Using the normal distribution as a comparison, I think the "scale
parameter" for a normal distribution certainly isn't independent of sigma,
and norm.fit() only returns mu and sigma. This makes a lot more sense to
me.)

I'm basically trying to use kappa as a measure of the "width" of the
distribution, so the extra degree of freedom introduced by the scale
parameter is problematic for me. For two distributions that are
superficially very similar in "width", I might get wildly varying values of
kappa.

For example, I once got fitted values of kappa=1, and kappa=400 for two
distributions that looked very similar in width.
I thought I must have been doing something wrong... until I saw the scale
parameters were around 1 and 19, respectively.
Plotting the fitted distributions:
>>> xs = numpy.linspace(-numpy.pi, numpy.pi, 361)
>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 1, loc=0, scale=1))
>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 400, loc=0,
scale=19))

What I'd really like to do is "force" scale=1 when I perform the fitting.
(In fact, it would be nice to even force the mean of the fitted distribution
to a precalculated value, as well. I really only want one degree of freedom
for the fitting algorithm -- I just want it to explore different values of
kappa.)

Is there any way to do this?
Thanks for your help,

--David Ho


On Tue, Apr 13, 2010 at 3:13 PM,  <josef.pktd at gmail.com> wrote:
>  > In the current fit version, loc (and with it the support of the
> > distribution) and scale are always estimated. In some cases this is
> > not desired.
> > You are transforming the original data to fit into the standard
> > distribution with loc=0, scale=1 . Do you get reasonable estimates for
> > loc and scale in this case?
> > If not, then there is another patch or enhanced fit function that
> > could take loc and scale as fixed.
> >
> > I will look at some details in your function later, especially I'm
> > curious how the circular statistics works.
> >
> > Thanks for the example, maybe I can stop ignoring vonmises.
> >
> > Josef
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100413/4b1587eb/attachment.html>

From josef.pktd at gmail.com  Wed Apr 14 00:52:36 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 14 Apr 2010 00:52:36 -0400
Subject: [SciPy-User] Can I "fix" scale=1 when fitting distributions?
In-Reply-To: <k2p469ba0f41004132109tb47e72bdjbef3d5aa303beeb7@mail.gmail.com>
References: <k2p469ba0f41004132109tb47e72bdjbef3d5aa303beeb7@mail.gmail.com>
Message-ID: <i2w1cd32cbb1004132152z43676e96xb8043459b25e9cd6@mail.gmail.com>

On Wed, Apr 14, 2010 at 12:09 AM, David Ho <itsdho at ucla.edu> wrote:
> Actually, being able to "fix" scale=1 would be very useful for me.
>
> The reason I'm trying to fit a von mises distribution to my data is to find
> "kappa", a measure of the concentration of the data.
>
> On wikipedia, I see that the von mises distribution only really has 2
> parameters: mu (the mean), and kappa (1/kappa being analogous to the
> variance).
> When I use vonmises.fit(), though, I get 3 parameters: kappa, mu, and a
> "scale parameter", respectively.
> However, I don't think the scale parameter for the von mises distribution is
> really independent of kappa, is it? (Am I understanding this correctly?)
> (Using the normal distribution as a comparison, I think the "scale
> parameter" for a normal distribution certainly isn't independent of sigma,
> and norm.fit() only returns mu and sigma. This makes a lot more sense to
> me.)

for normal  distribution loc=mean and scale  = sqrt(variance)  and has
no shape parameter.

essentially all distributions are transformed  y = (x-mean)/scale
where x is the original data, and y is the standardized data, the
actual _pdf, _cdf, ... are for the standard version of the
distribution,
it's a generic framework, but doesn't make sense in all cases or
applications mainly when we want to fix the support of the
distribution

>
> I'm basically trying to use kappa as a measure of the "width" of the
> distribution, so the extra degree of freedom introduced by the scale
> parameter is problematic for me. For two distributions that are
> superficially very similar in "width", I might get wildly varying values of
> kappa.
>
> For example, I once got fitted values of kappa=1, and kappa=400 for two
> distributions that looked very similar in width.
> I thought I must have been doing something wrong... until I saw the scale
> parameters were around 1 and 19, respectively.
> Plotting the fitted distributions:
>>>> xs = numpy.linspace(-numpy.pi, numpy.pi, 361)
>>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 1, loc=0, scale=1))
>>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 400, loc=0,
>>>> scale=19))

one idea is to check the implied moments, eg. vonmises.stats(1, loc=0,
scale=1, moment='mvsk'))
to see whether the implied mean, variance, skew, kurtosis are similar
in two different parameterizations.
Warning skew, kurtosis is incorrect for a few distributions, I don't
know about vonmises.

>
> What I'd really like to do is "force" scale=1 when I perform the fitting.
> (In fact, it would be nice to even force the mean of the fitted distribution
> to a precalculated value, as well. I really only want one degree of freedom
> for the fitting algorithm -- I just want it to explore different values of
> kappa.)
>
> Is there any way to do this?

It's not yet possible in scipy, but easy to write, essentially copy
the existing fit and nnlf methods and fix loc and scale and call the
optimization with only one argument.

I have a more than one year old ticket:
http://projects.scipy.org/scipy/ticket/832

It might be possible that it works immediately if you monkey patch
rv_continuous  by adding
nnlf_fr and fit_fr
in
http://mail.scipy.org/pipermail/scipy-user/2009-February/019968.html

scipy.stats.rv_continuous.nnlf_fr = nnlfr
scipy.stats.rv_continuous.fit_fr = fit_fr

and call
vonmises.fit_fr(data, frozen=[np.nan, 0.0, 1.0])

It's worth a try, but if you only need vonmises it might also be easy
to use scipy.optimize.fmin on a function that wraps vonmises._nnlf and
fixes loc, scale.
To give you an idea, I didn't check what the correct function arguments are ::

def mynnlf(kappa):
    loc = 0
    scale = 1
    return vonmises.nnlf(kappa, loc, scale)

scipy.optimize.fmin(mynnlf, startingvalueforkappa)

I could check this for your case at the end of the week.

If you have an opinion for a good generic API for fit_semifrozen, let me know.
I hope this helps, and please keep me informed. I would like to make
at least a cookbook recipe out of it.

Josef


> Thanks for your help,
>
> --David Ho
>
>
>> On Tue, Apr 13, 2010 at 3:13 PM, ?<josef.pktd at gmail.com> wrote:
>> > In the current fit version, loc (and with it the support of the
>> > distribution) and scale are always estimated. In some cases this is
>> > not desired.
>> > You are transforming the original data to fit into the standard
>> > distribution with loc=0, scale=1 . Do you get reasonable estimates for
>> > loc and scale in this case?
>> > If not, then there is another patch or enhanced fit function that
>> > could take loc and scale as fixed.
>> >
>> > I will look at some details in your function later, especially I'm
>> > curious how the circular statistics works.
>> >
>> > Thanks for the example, maybe I can stop ignoring vonmises.
>> >
>> > Josef
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From josef.pktd at gmail.com  Wed Apr 14 00:56:26 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 14 Apr 2010 00:56:26 -0400
Subject: [SciPy-User] Can I "fix" scale=1 when fitting distributions?
In-Reply-To: <i2w1cd32cbb1004132152z43676e96xb8043459b25e9cd6@mail.gmail.com>
References: <k2p469ba0f41004132109tb47e72bdjbef3d5aa303beeb7@mail.gmail.com>
	<i2w1cd32cbb1004132152z43676e96xb8043459b25e9cd6@mail.gmail.com>
Message-ID: <s2s1cd32cbb1004132156l78d4461ax6dde01800336878a@mail.gmail.com>

On Wed, Apr 14, 2010 at 12:52 AM,  <josef.pktd at gmail.com> wrote:
> On Wed, Apr 14, 2010 at 12:09 AM, David Ho <itsdho at ucla.edu> wrote:
>> Actually, being able to "fix" scale=1 would be very useful for me.
>>
>> The reason I'm trying to fit a von mises distribution to my data is to find
>> "kappa", a measure of the concentration of the data.
>>
>> On wikipedia, I see that the von mises distribution only really has 2
>> parameters: mu (the mean), and kappa (1/kappa being analogous to the
>> variance).
>> When I use vonmises.fit(), though, I get 3 parameters: kappa, mu, and a
>> "scale parameter", respectively.
>> However, I don't think the scale parameter for the von mises distribution is
>> really independent of kappa, is it? (Am I understanding this correctly?)
>> (Using the normal distribution as a comparison, I think the "scale
>> parameter" for a normal distribution certainly isn't independent of sigma,
>> and norm.fit() only returns mu and sigma. This makes a lot more sense to
>> me.)
>
> for normal ?distribution loc=mean and scale ?= sqrt(variance) ?and has
> no shape parameter.
>
> essentially all distributions are transformed ?y = (x-mean)/scale
> where x is the original data, and y is the standardized data, the
> actual _pdf, _cdf, ... are for the standard version of the
> distribution,
> it's a generic framework, but doesn't make sense in all cases or
> applications mainly when we want to fix the support of the
> distribution
>
>>
>> I'm basically trying to use kappa as a measure of the "width" of the
>> distribution, so the extra degree of freedom introduced by the scale
>> parameter is problematic for me. For two distributions that are
>> superficially very similar in "width", I might get wildly varying values of
>> kappa.
>>
>> For example, I once got fitted values of kappa=1, and kappa=400 for two
>> distributions that looked very similar in width.
>> I thought I must have been doing something wrong... until I saw the scale
>> parameters were around 1 and 19, respectively.
>> Plotting the fitted distributions:
>>>>> xs = numpy.linspace(-numpy.pi, numpy.pi, 361)
>>>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 1, loc=0, scale=1))
>>>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 400, loc=0,
>>>>> scale=19))
>
> one idea is to check the implied moments, eg. vonmises.stats(1, loc=0,
> scale=1, moment='mvsk'))
> to see whether the implied mean, variance, skew, kurtosis are similar
> in two different parameterizations.
> Warning skew, kurtosis is incorrect for a few distributions, I don't
> know about vonmises.
>
>>
>> What I'd really like to do is "force" scale=1 when I perform the fitting.
>> (In fact, it would be nice to even force the mean of the fitted distribution
>> to a precalculated value, as well. I really only want one degree of freedom
>> for the fitting algorithm -- I just want it to explore different values of
>> kappa.)
>>
>> Is there any way to do this?
>
> It's not yet possible in scipy, but easy to write, essentially copy
> the existing fit and nnlf methods and fix loc and scale and call the
> optimization with only one argument.
>
> I have a more than one year old ticket:
> http://projects.scipy.org/scipy/ticket/832
>
> It might be possible that it works immediately if you monkey patch
> rv_continuous ?by adding
> nnlf_fr and fit_fr
> in
> http://mail.scipy.org/pipermail/scipy-user/2009-February/019968.html
>
> scipy.stats.rv_continuous.nnlf_fr = nnlfr
> scipy.stats.rv_continuous.fit_fr = fit_fr

correction
this should instead refer to
scipy.stats.distributions.rv_continuous

Josef
>
> and call
> vonmises.fit_fr(data, frozen=[np.nan, 0.0, 1.0])
>
> It's worth a try, but if you only need vonmises it might also be easy
> to use scipy.optimize.fmin on a function that wraps vonmises._nnlf and
> fixes loc, scale.
> To give you an idea, I didn't check what the correct function arguments are ::
>
> def mynnlf(kappa):
> ? ?loc = 0
> ? ?scale = 1
> ? ?return vonmises.nnlf(kappa, loc, scale)
>
> scipy.optimize.fmin(mynnlf, startingvalueforkappa)
>
> I could check this for your case at the end of the week.
>
> If you have an opinion for a good generic API for fit_semifrozen, let me know.
> I hope this helps, and please keep me informed. I would like to make
> at least a cookbook recipe out of it.
>
> Josef
>
>
>> Thanks for your help,
>>
>> --David Ho
>>
>>
>>> On Tue, Apr 13, 2010 at 3:13 PM, ?<josef.pktd at gmail.com> wrote:
>>> > In the current fit version, loc (and with it the support of the
>>> > distribution) and scale are always estimated. In some cases this is
>>> > not desired.
>>> > You are transforming the original data to fit into the standard
>>> > distribution with loc=0, scale=1 . Do you get reasonable estimates for
>>> > loc and scale in this case?
>>> > If not, then there is another patch or enhanced fit function that
>>> > could take loc and scale as fixed.
>>> >
>>> > I will look at some details in your function later, especially I'm
>>> > curious how the circular statistics works.
>>> >
>>> > Thanks for the example, maybe I can stop ignoring vonmises.
>>> >
>>> > Josef
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>


From gokhansever at gmail.com  Wed Apr 14 02:47:26 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Wed, 14 Apr 2010 01:47:26 -0500
Subject: [SciPy-User] Fitting tools
In-Reply-To: <h2zfbab3e0d1004122313tbba74bber11a870119a537be@mail.gmail.com>
References: <p2u49d6b3501004121014p382c1712k72ba68fb91bececd@mail.gmail.com>
	<w2k1cd32cbb1004121034q58fd3a9seea52b1a1505e399@mail.gmail.com>
	<v2l49d6b3501004121254ic2929a9fn9364bf6e03f3030b@mail.gmail.com>
	<h2zfbab3e0d1004122313tbba74bber11a870119a537be@mail.gmail.com>
Message-ID: <m2r49d6b3501004132347h60bbcfajbebd48dab4165972@mail.gmail.com>

Erik,

Thanks for the fix. It works now. Finally I was able to launch the graphical
interface as well. Is it possible to get parameter estimates using the GUI?

On Tue, Apr 13, 2010 at 1:13 AM, Erik Tollerud <erik.tollerud at gmail.com>wrote:

> Ok, I just committed a patch that I think should fix the particular
> problem you were encountering - let me know if it does not, or if you
> encounter some other problem trying to run the fitgui.
>
> Also, if you there are some models/distributions you want to use that
> are not already included in the builtin models (rather likely, as most
> of them are astronomy-related aside from the basics like gaussian,
> linear, polynomial, and so on) it's straightforward to add new ones in
> your own code and register them so that the gui will display them as
> options.  If it's a fairly standard distribution, feel free to submit
> it to me and I can add it to the astropysics builtins, as well.
>
>
> On Mon, Apr 12, 2010 at 12:54 PM, G?khan Sever <gokhansever at gmail.com>
> wrote:
> >
> >
> > On Mon, Apr 12, 2010 at 12:34 PM, <josef.pktd at gmail.com> wrote:
> >>
> >> On Mon, Apr 12, 2010 at 1:14 PM, G?khan Sever <gokhansever at gmail.com>
> >> wrote:
> >> > Hello,
> >> >
> >> > Are there any compact curve/distribution fitting tools around similar
> to
> >> > the
> >> > commercial TableCurve 2D [1] and EasyFit [2] products?
> >> >
> >> > [1] http://www.sigmaplot.com/products/tablecurve2d/tablecurve2d.php
> >>
> >> I think astrogui, or astropysics might have something like this,
> >> Travis showcased it briefly in the stats webinar. There is also an
> >> online curve fitting (zunzun ?) with (I think) BSD licensed source
> >> code that might be useful.
> >>
> >> > [2] http://www.mathwave.com/easyfit-distribution-fitting.html
> >>
> >> I looked at easyfit a year or more ago, and wrote a script that tried
> >> to do similar fitting for all distributions in scipy.stats. It worked
> >> ok for distributions with unlimited support, but there are still the
> >> problems with estimating distributions with a finite bound in the
> >> support.
> >> I used kstest as distance measure, but never rewrote Anderson-Darling
> >> to function as generic distance measure as (if I remember correctly)
> >> easyfit does.
> >>
> >> Josef
> >
> > Thanks for the input Josef.
> >
> > I have known astrogui from Travis' webinar as well. However I have never
> > been able to run it. Maybe Erik could make some comments to fix the
> problem:
> >
> > # Using bzr branch lp:astropysics
> >
> > python
> > Python 2.6.2 (r262:71600, Jan 25 2010, 18:46:45)
> > [GCC 4.4.2 20091222 (Red Hat 4.4.2-20)] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> from astropysics import gui
> >
> /home/gsever/Desktop/python-repo/ETS_3.3.1/Enable_3.2.1/enthought/kiva/fonttools/font_manager.py:1064:
> > UserWarning: Could not match sans-serif, normal, normal.  Returning
> > /usr/share/fonts/khmeros/KhmerOS.ttf
> >   warnings.warn('Could not match %s, %s, %s.  Returning %s' % (name,
> style,
> > variant, self.defaultFont))
> >
> /usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/io.py:34:
> > UserWarning: pyfits not found - all FITS-related IO will not work
> >   warn('pyfits not found - all FITS-related IO will not work')
> >
> /usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/io.py:314:
> > UserWarning: vo.table not found - VOTable processing limited to
> > VOTableReader class
> >   warn('vo.table not found - VOTable processing limited to VOTableReader
> > class')
> >
> /usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py:2742:
> > UserWarning: SExtractor not found on system - phot.SExtractor class will
> not
> > function
> >   warn('SExtractor not found on system - phot.SExtractor class will not
> > function')
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/gui/__init__.py",
> > line 28, in <module>
> >     from spectarget import spec_target
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/gui/spectarget.py",
> > line 18, in <module>
> >     from ..phot import CMDAnalyzer
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py",
> > line 3463, in <module>
> >     Spectrum._rgbsensitivity = __generateRGBSensTuple()
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py",
> > line 3455, in __generateRGBSensTuple
> >     from .models import BlackbodyModel
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/__init__.py",
> > line 73, in <module>
> >     from core import *
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/core.py",
> > line 2247, in <module>
> >     class
> FunctionModel2DScalar(FunctionModel,InputCoordinateTransformer):
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/core.py",
> > line 2253, in FunctionModel2DScalar
> >     from ..coords import cartesian_to_polar,polar_to_cartesian
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
> > line 2447, in <module>
> >     solsysobjs['sun'] = Sun()
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
> > line 2267, in __init__
> >
> EphemerisObject.__init__(self,name='Sol',jd0=KeplerianOrbit.jd2000-0.5)
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
> > line 1791, in __init__
> >     self.jd0 = EphemerisObject.jd2000 if jd0 is None else jd0
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
> > line 1799, in _setJd0
> >     from .obstools import calendar_to_jd
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/obstools.py",
> > line 1580, in <module>
> >     sites['uciobs'] =
> > Site(33.63614044191056,-117.83079922199249,80,'PST','UC Irvine
> Observatory')
> >   File
> >
> "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/obstools.py",
> > line 498, in __init__
> >     raise ValueError('unrecognized time zone string '+tz)
> > ValueError: unrecognized time zone string PST
> >
> >
> > --
> > G?khan
> >
>



-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100414/87bcfb73/attachment.html>

From kuiper at jpl.nasa.gov  Wed Apr 14 11:53:31 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Wed, 14 Apr 2010 08:53:31 -0700
Subject: [SciPy-User] Python seek argument type
Message-ID: <4BC5E4FB.3030505@jpl.nasa.gov>

Having disgraced myself with a not-to-bright question yesterday, let me 
try again, still on the topic of reading binary files.

While trying to access the last record in the file I got:

     98     new_pos = pos+76*(index-current_index)
     99     print "Moving file index from",pos,"to",new_pos
--> 100     fd.seek(new_pos,0)
    101   buf = fd.read(76)
    102   data = unpack_from("=LLL dddd dddd", buf)

OverflowError: long int too large to convert to int

The argument in question, new_pos, has a value of 2,252,639,972, which 
is slightly too large to be a Python signed int:
math.log(2252639972,2) = 31.068969607366373
and, of course, a Python int is a C long on a 32-bit machine, which is 
the type specified for the first argument of the underlying C fseek command.

Now a 2.1 GB file is not large by modern standards.  I imagine someone 
must have come up with a way to position past the 2 billionth byte in a 
file.  Does anyone know what it is?

Thanks and regards

Tom


From robert.kern at gmail.com  Wed Apr 14 11:55:56 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 14 Apr 2010 10:55:56 -0500
Subject: [SciPy-User] Python seek argument type
In-Reply-To: <4BC5E4FB.3030505@jpl.nasa.gov>
References: <4BC5E4FB.3030505@jpl.nasa.gov>
Message-ID: <o2o3d375d731004140855v163dc970qe712fc9ba67fd37d@mail.gmail.com>

On Wed, Apr 14, 2010 at 10:53, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
> Having disgraced myself with a not-to-bright question yesterday, let me
> try again, still on the topic of reading binary files.
>
> While trying to access the last record in the file I got:
>
> ? ? 98 ? ? new_pos = pos+76*(index-current_index)
> ? ? 99 ? ? print "Moving file index from",pos,"to",new_pos
> --> 100 ? ? fd.seek(new_pos,0)
> ? ?101 ? buf = fd.read(76)
> ? ?102 ? data = unpack_from("=LLL dddd dddd", buf)
>
> OverflowError: long int too large to convert to int
>
> The argument in question, new_pos, has a value of 2,252,639,972, which
> is slightly too large to be a Python signed int:
> math.log(2252639972,2) = 31.068969607366373
> and, of course, a Python int is a C long on a 32-bit machine, which is
> the type specified for the first argument of the underlying C fseek command.
>
> Now a 2.1 GB file is not large by modern standards. ?I imagine someone
> must have come up with a way to position past the 2 billionth byte in a
> file. ?Does anyone know what it is?

What version of Python are you using? Python 2.4 definitely had
problems like this, but I believe that 2.5 resolved them.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jgomezdans at gmail.com  Wed Apr 14 12:22:05 2010
From: jgomezdans at gmail.com (Jose Gomez-Dans)
Date: Wed, 14 Apr 2010 17:22:05 +0100
Subject: [SciPy-User] Efficient 3d smoothing/interpolation
Message-ID: <x2l91d218431004140922vf13dce5elfdfec46565063016@mail.gmail.com>

Hi,

A two part question , I guess! I have a set of images (2d datasets), and I
want to interpolate from one timestep to the next for each pixel when the
data is invalid. Since I can get invalid data on the edges (missing first
few time steps), I want to be able to set these values to the closest (in
time) valid values for those pixels.

Let's say I have a 3D dataset: a 2d (2000x2000) dataset with 180 time
frames. Let's call it A
>>> print A.shape
(180, 2000, 2000)

Some of the values in the dataset will be missing (flagged by a known
value). I would like to interpolate over them (and maybe do some smoothing,
but that's easy with splines) in time. Additionally, since the ends of my
time series for a given "pixel" may be missing, I want to set these
endpoints to the closest valid value in the time series (this is required to
be able to interpolate over these points). My first attempt uses loops, but
is obviously quite slow, and is quite probably the ugliest piece of code
I've seen in a long while! So please bear with me for a minute ;)

t_axis = numpy.arange(A.shape[0])
for ix in xrange(2000):
    for iy in xrange(2000):
        # get the time series for the current pixel
        t_series = A[:, ix, iy ]
        # Locate valid samples
        valid = t_series <> INVALID_VALUE
        # if the edges are invalid, fill in with the closest valid value.
        if not valid [0]:
            t_series[0] = t_series[valid][0]
            valid[0] = True
        if not valid [-1]:
            t_series[-1] = t_series[valid][-1]
            valid[-1] = True
        # do 1D interpolation on t_series, returning the interpolated values
where valid isn't True. Use eg scipy.interpolate.Univariate.Spline
        sp = scipy.interpolate.UnivariateSpline ( t_axis[valid],
t_series[valid] )
        it_series = numpy.where ( valid, t_series, sp ( t_axis ) )
        # Pack the data back into a 3D array

I don't know whether standard scipy (or ndimage) has stuff to solve this
problem in a reasonable way. Any ideas on how to improve on this much
welcomed!

Thanks!
Jose
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100414/b67cf964/attachment.html>

From kuiper at jpl.nasa.gov  Wed Apr 14 13:10:04 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Wed, 14 Apr 2010 10:10:04 -0700
Subject: [SciPy-User] Python seek argument type
In-Reply-To: <mailman.9.1271264404.31094.scipy-user@scipy.org>
References: <mailman.9.1271264404.31094.scipy-user@scipy.org>
Message-ID: <4BC5F6EC.7070900@jpl.nasa.gov>


> Date: Wed, 14 Apr 2010 10:55:56 -0500
> From: Robert Kern <robert.kern at gmail.com>
> Subject: Re: [SciPy-User] Python seek argument type
> To: SciPy Users List <scipy-user at scipy.org>
>
> On Wed, Apr 14, 2010 at 10:53, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
>   
>> Having disgraced myself with a not-to-bright question yesterday, let me
>> try again, still on the topic of reading binary files.
>>
>> While trying to access the last record in the file I got:
>>
>> ? ? 98 ? ? new_pos = pos+76*(index-current_index)
>> ? ? 99 ? ? print "Moving file index from",pos,"to",new_pos
>> --> 100 ? ? fd.seek(new_pos,0)
>> ? ?101 ? buf = fd.read(76)
>> ? ?102 ? data = unpack_from("=LLL dddd dddd", buf)
>>
>> OverflowError: long int too large to convert to int
>>
>> The argument in question, new_pos, has a value of 2,252,639,972, which
>> is slightly too large to be a Python signed int:
>> math.log(2252639972,2) = 31.068969607366373
>> and, of course, a Python int is a C long on a 32-bit machine, which is
>> the type specified for the first argument of the underlying C fseek command.
>>
>> Now a 2.1 GB file is not large by modern standards. ?I imagine someone
>> must have come up with a way to position past the 2 billionth byte in a
>> file. ?Does anyone know what it is?
>>     
>
> What version of Python are you using? Python 2.4 definitely had
> problems like this, but I believe that 2.5 resolved them.
>   
Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14)

Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100414/b0360f04/attachment.html>

From hnry2k at hotmail.com  Wed Apr 14 13:25:18 2010
From: hnry2k at hotmail.com (=?iso-8859-1?B?Sm9yZ2UgRS4gtFNhbmNoZXogU2FuY2hleg==?=)
Date: Wed, 14 Apr 2010 12:25:18 -0500
Subject: [SciPy-User] Python seek argument type
In-Reply-To: <4BC5F6EC.7070900@jpl.nasa.gov>
References: <mailman.9.1271264404.31094.scipy-user@scipy.org>,
	<4BC5F6EC.7070900@jpl.nasa.gov>
Message-ID: <COL104-W41CA147C81704787109E4A9B100@phx.gbl>


As far as I understand the biggest long int in C you can manage in a 32bit machine is 2147483647, in order to work with bigger numbers than this you had to use the gmp library (general multiple precision) which  also has a python module gmpy, you can have more information from http://gmpy.sourceforge.net/.

Best Regards
Jorge

Date: Wed, 14 Apr 2010 10:10:04 -0700
From: kuiper at jpl.nasa.gov
To: scipy-user at scipy.org
Subject: Re: [SciPy-User] Python seek argument type






  
  





  Date: Wed, 14 Apr 2010 10:55:56 -0500
From: Robert Kern <robert.kern at gmail.com>
Subject: Re: [SciPy-User] Python seek argument type
To: SciPy Users List <scipy-user at scipy.org>

On Wed, Apr 14, 2010 at 10:53, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
  
  
    Having disgraced myself with a not-to-bright question yesterday, let me
try again, still on the topic of reading binary files.

While trying to access the last record in the file I got:

? ? 98 ? ? new_pos = pos+76*(index-current_index)
? ? 99 ? ? print "Moving file index from",pos,"to",new_pos
--> 100 ? ? fd.seek(new_pos,0)
? ?101 ? buf = fd.read(76)
? ?102 ? data = unpack_from("=LLL dddd dddd", buf)

OverflowError: long int too large to convert to int

The argument in question, new_pos, has a value of 2,252,639,972, which
is slightly too large to be a Python signed int:
math.log(2252639972,2) = 31.068969607366373
and, of course, a Python int is a C long on a 32-bit machine, which is
the type specified for the first argument of the underlying C fseek command.

Now a 2.1 GB file is not large by modern standards. ?I imagine someone
must have come up with a way to position past the 2 billionth byte in a
file. ?Does anyone know what it is?
    
  
  What version of Python are you using? Python 2.4 definitely had
problems like this, but I believe that 2.5 resolved them.
  

Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14)



Tom
 		 	   		  
_________________________________________________________________
Enciende tu hotness con Hotmail
www.hotmailhotness.com.mx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100414/60ad3625/attachment.html>

From njs at pobox.com  Wed Apr 14 13:54:33 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 14 Apr 2010 10:54:33 -0700
Subject: [SciPy-User] Python seek argument type
In-Reply-To: <4BC5E4FB.3030505@jpl.nasa.gov>
References: <4BC5E4FB.3030505@jpl.nasa.gov>
Message-ID: <h2o961fa2b41004141054m6772a27bi215bff472320b67a@mail.gmail.com>

On Wed, Apr 14, 2010 at 8:53 AM, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
> The argument in question, new_pos, has a value of 2,252,639,972, which
> is slightly too large to be a Python signed int:
> math.log(2252639972,2) = 31.068969607366373
> and, of course, a Python int is a C long on a 32-bit machine, which is
> the type specified for the first argument of the underlying C fseek command.
>
> Now a 2.1 GB file is not large by modern standards. ?I imagine someone
> must have come up with a way to position past the 2 billionth byte in a
> file. ?Does anyone know what it is?

The whole traditional POSIX filesystem API has a 64-bit version -- on
Linux at least, you use syscalls that take "off_t" instead of "int".
For fseek the corresponding syscall is fseeko. And then you compile
with -D_FILE_OFFSET_BITS=64.

Python really should be taking care of all that mess for you, though.
Since Python runs on both systems with and without support for large
files, you do have to make sure that it's using the right code. Is
your Python built by you, or does it come from some distributor? If
it's built by you, you might want to double-check how you compiled
it...? Or check if any bugs were fixed in this area in a later Python
point release? Since this problem doesn't have anything to do with
scipy, you might find more knowledgeable help on python-list.

I can confirm that with Ubuntu's Python 2.5.4, fd.seek(2252639972, 0)
works fine.

-- Nathaniel


From erik.tollerud at gmail.com  Wed Apr 14 14:13:30 2010
From: erik.tollerud at gmail.com (Erik Tollerud)
Date: Wed, 14 Apr 2010 11:13:30 -0700
Subject: [SciPy-User] Fitting tools
In-Reply-To: <m2r49d6b3501004132347h60bbcfajbebd48dab4165972@mail.gmail.com>
References: <p2u49d6b3501004121014p382c1712k72ba68fb91bececd@mail.gmail.com> 
	<w2k1cd32cbb1004121034q58fd3a9seea52b1a1505e399@mail.gmail.com> 
	<v2l49d6b3501004121254ic2929a9fn9364bf6e03f3030b@mail.gmail.com> 
	<h2zfbab3e0d1004122313tbba74bber11a870119a537be@mail.gmail.com> 
	<m2r49d6b3501004132347h60bbcfajbebd48dab4165972@mail.gmail.com>
Message-ID: <n2rfbab3e0d1004141113r36b33368yd4fc3b82f22a7fd3@mail.gmail.com>

I'm not sure exactly what you mean... the best-fit values for the
parameters should appear in the bottom panel of the fitgui dialog, is
that what you're hoping to see?  Or do you mean extract them from the
GUI back into an interactive ipython session?  If the latter, just do

mod = fit_data(xdata,ydata,...)
mod.pardict

and mod will be the output model, with mod.pardict a dictionary of the
parameters and their values.  Or if you want the gui non-modal, do

fg = FitGui(...)
fg.edit_traits()
mod = fg.getModelObject()

On Tue, Apr 13, 2010 at 11:47 PM, G?khan Sever <gokhansever at gmail.com> wrote:
> Erik,
>
> Thanks for the fix. It works now. Finally I was able to launch the graphical
> interface as well. Is it possible to get parameter estimates using the GUI?
>
> On Tue, Apr 13, 2010 at 1:13 AM, Erik Tollerud <erik.tollerud at gmail.com>
> wrote:
>>
>> Ok, I just committed a patch that I think should fix the particular
>> problem you were encountering - let me know if it does not, or if you
>> encounter some other problem trying to run the fitgui.
>>
>> Also, if you there are some models/distributions you want to use that
>> are not already included in the builtin models (rather likely, as most
>> of them are astronomy-related aside from the basics like gaussian,
>> linear, polynomial, and so on) it's straightforward to add new ones in
>> your own code and register them so that the gui will display them as
>> options. ?If it's a fairly standard distribution, feel free to submit
>> it to me and I can add it to the astropysics builtins, as well.
>>
>>
>> On Mon, Apr 12, 2010 at 12:54 PM, G?khan Sever <gokhansever at gmail.com>
>> wrote:
>> >
>> >
>> > On Mon, Apr 12, 2010 at 12:34 PM, <josef.pktd at gmail.com> wrote:
>> >>
>> >> On Mon, Apr 12, 2010 at 1:14 PM, G?khan Sever <gokhansever at gmail.com>
>> >> wrote:
>> >> > Hello,
>> >> >
>> >> > Are there any compact curve/distribution fitting tools around similar
>> >> > to
>> >> > the
>> >> > commercial TableCurve 2D [1] and EasyFit [2] products?
>> >> >
>> >> > [1] http://www.sigmaplot.com/products/tablecurve2d/tablecurve2d.php
>> >>
>> >> I think astrogui, or astropysics might have something like this,
>> >> Travis showcased it briefly in the stats webinar. There is also an
>> >> online curve fitting (zunzun ?) with (I think) BSD licensed source
>> >> code that might be useful.
>> >>
>> >> > [2] http://www.mathwave.com/easyfit-distribution-fitting.html
>> >>
>> >> I looked at easyfit a year or more ago, and wrote a script that tried
>> >> to do similar fitting for all distributions in scipy.stats. It worked
>> >> ok for distributions with unlimited support, but there are still the
>> >> problems with estimating distributions with a finite bound in the
>> >> support.
>> >> I used kstest as distance measure, but never rewrote Anderson-Darling
>> >> to function as generic distance measure as (if I remember correctly)
>> >> easyfit does.
>> >>
>> >> Josef
>> >
>> > Thanks for the input Josef.
>> >
>> > I have known astrogui from Travis' webinar as well. However I have never
>> > been able to run it. Maybe Erik could make some comments to fix the
>> > problem:
>> >
>> > # Using bzr branch lp:astropysics
>> >
>> > python
>> > Python 2.6.2 (r262:71600, Jan 25 2010, 18:46:45)
>> > [GCC 4.4.2 20091222 (Red Hat 4.4.2-20)] on linux2
>> > Type "help", "copyright", "credits" or "license" for more information.
>> >>>> from astropysics import gui
>> >
>> > /home/gsever/Desktop/python-repo/ETS_3.3.1/Enable_3.2.1/enthought/kiva/fonttools/font_manager.py:1064:
>> > UserWarning: Could not match sans-serif, normal, normal.? Returning
>> > /usr/share/fonts/khmeros/KhmerOS.ttf
>> > ? warnings.warn('Could not match %s, %s, %s.? Returning %s' % (name,
>> > style,
>> > variant, self.defaultFont))
>> >
>> > /usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/io.py:34:
>> > UserWarning: pyfits not found - all FITS-related IO will not work
>> > ? warn('pyfits not found - all FITS-related IO will not work')
>> >
>> > /usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/io.py:314:
>> > UserWarning: vo.table not found - VOTable processing limited to
>> > VOTableReader class
>> > ? warn('vo.table not found - VOTable processing limited to VOTableReader
>> > class')
>> >
>> > /usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py:2742:
>> > UserWarning: SExtractor not found on system - phot.SExtractor class will
>> > not
>> > function
>> > ? warn('SExtractor not found on system - phot.SExtractor class will not
>> > function')
>> > Traceback (most recent call last):
>> > ? File "<stdin>", line 1, in <module>
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/gui/__init__.py",
>> > line 28, in <module>
>> > ??? from spectarget import spec_target
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/gui/spectarget.py",
>> > line 18, in <module>
>> > ??? from ..phot import CMDAnalyzer
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py",
>> > line 3463, in <module>
>> > ??? Spectrum._rgbsensitivity = __generateRGBSensTuple()
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/phot.py",
>> > line 3455, in __generateRGBSensTuple
>> > ??? from .models import BlackbodyModel
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/__init__.py",
>> > line 73, in <module>
>> > ??? from core import *
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/core.py",
>> > line 2247, in <module>
>> > ??? class
>> > FunctionModel2DScalar(FunctionModel,InputCoordinateTransformer):
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/models/core.py",
>> > line 2253, in FunctionModel2DScalar
>> > ??? from ..coords import cartesian_to_polar,polar_to_cartesian
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
>> > line 2447, in <module>
>> > ??? solsysobjs['sun'] = Sun()
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
>> > line 2267, in __init__
>> >
>> > EphemerisObject.__init__(self,name='Sol',jd0=KeplerianOrbit.jd2000-0.5)
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
>> > line 1791, in __init__
>> > ??? self.jd0 = EphemerisObject.jd2000 if jd0 is None else jd0
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/coords.py",
>> > line 1799, in _setJd0
>> > ??? from .obstools import calendar_to_jd
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/obstools.py",
>> > line 1580, in <module>
>> > ??? sites['uciobs'] =
>> > Site(33.63614044191056,-117.83079922199249,80,'PST','UC Irvine
>> > Observatory')
>> > ? File
>> >
>> > "/usr/lib/python2.6/site-packages/Astropysics-0.1.dev-py2.6.egg/astropysics/obstools.py",
>> > line 498, in __init__
>> > ??? raise ValueError('unrecognized time zone string '+tz)
>> > ValueError: unrecognized time zone string PST
>> >
>> >
>> > --
>> > G?khan
>> >
>
>
>
> --
> G?khan
>


From abiel.x.reinhart at jpmchase.com  Wed Apr 14 14:53:14 2010
From: abiel.x.reinhart at jpmchase.com (Abiel X Reinhart)
Date: Wed, 14 Apr 2010 14:53:14 -0400
Subject: [SciPy-User] scikits.timeseries: setting values outside of existing
	series range
Message-ID: <DF11712721890A4880B3FA1B046893720CF79E8C74@EMARC105VS01.exchad.jpmchase.net>

What is the best way to set particular indices of a TimeSeries to new values, where the indices in question are not already contained in the TimeSeries? For example, I might have an annual series running from 2000 to 2005, and I want to insert a lone value for 2009.

Thanks.

Abiel


This communication is for informational purposes only. It is not
intended as an offer or solicitation for the purchase or sale of
any financial instrument or as an official confirmation of any
transaction. All market prices, data and other information are not
warranted as to completeness or accuracy and are subject to change
without notice. Any comments or statements made herein do not
necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
and affiliates.

This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by JPMorgan Chase &
Co., its subsidiaries and affiliates, as applicable, for any loss
or damage arising in any way from its use. If you received this
transmission in error, please immediately contact the sender and
destroy the material in its entirety, whether in electronic or hard
copy format. Thank you.

Please refer to http://www.jpmorgan.com/pages/disclosures for
disclosures relating to European legal entities.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100414/f3bd3eca/attachment.html>

From pgmdevlist at gmail.com  Wed Apr 14 15:58:54 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 14 Apr 2010 15:58:54 -0400
Subject: [SciPy-User] scikits.timeseries: setting values outside of
	existing series range
In-Reply-To: <DF11712721890A4880B3FA1B046893720CF79E8C74@EMARC105VS01.exchad.jpmchase.net>
References: <DF11712721890A4880B3FA1B046893720CF79E8C74@EMARC105VS01.exchad.jpmchase.net>
Message-ID: <6A90B18B-9CCB-4DB3-A2E6-E705A6772CD8@gmail.com>

On Apr 14, 2010, at 2:53 PM, Abiel X Reinhart wrote:
> 
> What is the best way to set particular indices of a TimeSeries to new values, where the indices in question are not already contained in the TimeSeries? For example, I might have an annual series running from 2000 to 2005, and I want to insert a lone value for 2009.  

I'm not sure I see what you mean, could you provide an actual example ?
If you want to add a single value to an existing series, create a new 1-element series for this value and use ts.concatenate to join the two.
If you want to change both the date and value of a particular entry, things get tricky: you could modify the value by simple indexing (like a[-1]) but you'd have to modify explicitly the date as well. I would recommend caution in that case, as you could break things rather easily. It's better to create a new series.
HIH, but don't hesitate to ask if you need more info
Cheers
P.

From gokhansever at gmail.com  Wed Apr 14 18:37:03 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Wed, 14 Apr 2010 17:37:03 -0500
Subject: [SciPy-User] Vectorized newton function
Message-ID: <o2v49d6b3501004141537wdf78861ez75edf7a0cf229778@mail.gmail.com>

Hello,

How can I make scipy.optimize.newton function accepting a numpy.array as its
x0 parameter?

File:             /usr/lib/python2.6/site-packages/scipy/optimize/minpack.py
Definition:       newton(func, x0, fprime=None, args=(), tol=1.48e-08,
maxiter=50)

x0 : float
        An initial estimate of the zero that should be somewhere near the
        actual zero.

(Without using a for-loop)

Thanks.

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100414/7ad69feb/attachment.html>

From kuiper at jpl.nasa.gov  Wed Apr 14 19:11:16 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Wed, 14 Apr 2010 16:11:16 -0700
Subject: [SciPy-User] SciPy-User Digest, Vol 80, Issue 28
In-Reply-To: <mailman.376.1271271248.28903.scipy-user@scipy.org>
References: <mailman.376.1271271248.28903.scipy-user@scipy.org>
Message-ID: <4BC64B94.4090405@jpl.nasa.gov>


> Date: Wed, 14 Apr 2010 12:25:18 -0500
> From: Jorge E. ?Sanchez Sanchez         <hnry2k at hotmail.com>
> Subject: Re: [SciPy-User] Python seek argument type
>
> As far as I understand the biggest long int in C you can manage in a 32bit machine is 2147483647, in order to work with bigger numbers than this you had to use the gmp library (general multiple precision) which  also has a python module gmpy, you can have more information from http://gmpy.sourceforge.net/.
>   
Thank you.  I just downloaded the Debian package for the Python module.
> Date: Wed, 14 Apr 2010 10:54:33 -0700
> From: Nathaniel Smith <njs at pobox.com>
> Subject: Re: [SciPy-User] Python seek argument type
>
> The whole traditional POSIX filesystem API has a 64-bit version -- on
> Linux at least, you use syscalls that take "off_t" instead of "int".
> For fseek the corresponding syscall is fseeko. And then you compile
> with -D_FILE_OFFSET_BITS=64.
>
> Python really should be taking care of all that mess for you, though.
> Since Python runs on both systems with and without support for large
> files, you do have to make sure that it's using the right code. Is
> your Python built by you, or does it come from some distributor?
It's what in the current Debian 'stable' (lenny) distribution.
> Or check if any bugs were fixed in this area in a later Python
> point release? Since this problem doesn't have anything to do with
> scipy, you might find more knowledgeable help on python-list.
>   
You are right but I didn't think I was dealing with a Python bug until 
now.  I'll check it.
> I can confirm that with Ubuntu's Python 2.5.4, fd.seek(2252639972, 0)
> works fine.
>   
So I assume that a bug was filed and fixed.  I will reflect overnight on 
the merits of upgrading to the 'testing' distribution.

Thanks for all the advice.  At least this time I don't feel so dumb.

Tom


From robert.kern at gmail.com  Wed Apr 14 19:12:54 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 14 Apr 2010 18:12:54 -0500
Subject: [SciPy-User] SciPy-User Digest, Vol 80, Issue 28
In-Reply-To: <4BC64B94.4090405@jpl.nasa.gov>
References: <mailman.376.1271271248.28903.scipy-user@scipy.org> 
	<4BC64B94.4090405@jpl.nasa.gov>
Message-ID: <w2i3d375d731004141612t8503e7c3x2144aba4b0a31bed@mail.gmail.com>

On Wed, Apr 14, 2010 at 18:11, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
>
>> Date: Wed, 14 Apr 2010 12:25:18 -0500
>> From: Jorge E. ?Sanchez Sanchez ? ? ? ? <hnry2k at hotmail.com>
>> Subject: Re: [SciPy-User] Python seek argument type
>>
>> As far as I understand the biggest long int in C you can manage in a 32bit machine is 2147483647, in order to work with bigger numbers than this you had to use the gmp library (general multiple precision) which ?also has a python module gmpy, you can have more information from http://gmpy.sourceforge.net/.
>>
> Thank you. ?I just downloaded the Debian package for the Python module.

This won't help the problem you reported, of course.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From peridot.faceted at gmail.com  Wed Apr 14 19:23:21 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 14 Apr 2010 19:23:21 -0400
Subject: [SciPy-User] Vectorized newton function
In-Reply-To: <o2v49d6b3501004141537wdf78861ez75edf7a0cf229778@mail.gmail.com>
References: <o2v49d6b3501004141537wdf78861ez75edf7a0cf229778@mail.gmail.com>
Message-ID: <l2zce557a361004141623k9cd32132j1bc63cc346c0bcea@mail.gmail.com>

On 14 April 2010 18:37, G?khan Sever <gokhansever at gmail.com> wrote:
> Hello,
>
> How can I make scipy.optimize.newton function accepting a numpy.array as its
> x0 parameter?

The short answer is, you can't. Just use a loop.

The basic problem is that this function is a univariate optimizer, so
that it adjusts the scalar x0 as needed to (try to) find a zero of
func. If func is a multivariate function, you need a multivariate root
finder like fsolve. If what you want is instead to run one-dimensional
root-finding with many different starting points, well, I'll first
point out that the dependence of the result of newton on the starting
point is complicated (in fact fractal) and nasty, so you're going to
get weird hard-to-predict results unless you carefully start each x0
near a zero. If at all possible I recommend using something like
brentq/brenth that takes a "bracket" instead of a single point. That
said, if you're looking for lots of roots, there's not really any way
around looping in python to call newton many times, but this isn't
your efficiency problem anyway: the python function you provide as
"func" will be called many times for each call of newton, and that's
where the python overhead will matter. If your goal is just to avoid
writing a for loop, you can consider using vectorize() to create the
python loop for you.

You could argue that newton should allow "parallel" root-finding by
passing an array of xs into func and the derivative and getting an
array back, but since newton is an adaptive method, this would be very
messy code.

Anne

> File:             /usr/lib/python2.6/site-packages/scipy/optimize/minpack.py
> Definition:       newton(func, x0, fprime=None, args=(), tol=1.48e-08,
> maxiter=50)
>
> x0 : float
>         An initial estimate of the zero that should be somewhere near the
>         actual zero.
>
> (Without using a for-loop)
>
> Thanks.
>
> --
> G?khan
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From gokhansever at gmail.com  Wed Apr 14 20:15:40 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Wed, 14 Apr 2010 19:15:40 -0500
Subject: [SciPy-User] Vectorized newton function
In-Reply-To: <l2zce557a361004141623k9cd32132j1bc63cc346c0bcea@mail.gmail.com>
References: <o2v49d6b3501004141537wdf78861ez75edf7a0cf229778@mail.gmail.com>
	<l2zce557a361004141623k9cd32132j1bc63cc346c0bcea@mail.gmail.com>
Message-ID: <o2n49d6b3501004141715ice4b6720hf292ed88677f99b5@mail.gmail.com>

Thanks Anne for suggesting "fsolve" and rest of the useful information.
Below is how my function is read:

def myfunc(myarray):
       term1 = compute1_array1
       term2 = compute3_array2
       term3 = compute3_array3
       return term1+term2+term3

result_arr = fsolve(myfunc, myarray)

IDL handles univariate and multivariate cases using one fun newton function.
I get almost equivalent results using fsolve to IDL's newton.

On Wed, Apr 14, 2010 at 6:23 PM, Anne Archibald
<peridot.faceted at gmail.com>wrote:

> On 14 April 2010 18:37, G?khan Sever <gokhansever at gmail.com> wrote:
> > Hello,
> >
> > How can I make scipy.optimize.newton function accepting a numpy.array as
> its
> > x0 parameter?
>
> The short answer is, you can't. Just use a loop.
>
> The basic problem is that this function is a univariate optimizer, so
> that it adjusts the scalar x0 as needed to (try to) find a zero of
> func. If func is a multivariate function, you need a multivariate root
> finder like fsolve. If what you want is instead to run one-dimensional
> root-finding with many different starting points, well, I'll first
> point out that the dependence of the result of newton on the starting
> point is complicated (in fact fractal) and nasty, so you're going to
> get weird hard-to-predict results unless you carefully start each x0
> near a zero. If at all possible I recommend using something like
> brentq/brenth that takes a "bracket" instead of a single point. That
> said, if you're looking for lots of roots, there's not really any way
> around looping in python to call newton many times, but this isn't
> your efficiency problem anyway: the python function you provide as
> "func" will be called many times for each call of newton, and that's
> where the python overhead will matter. If your goal is just to avoid
> writing a for loop, you can consider using vectorize() to create the
> python loop for you.
>
> You could argue that newton should allow "parallel" root-finding by
> passing an array of xs into func and the derivative and getting an
> array back, but since newton is an adaptive method, this would be very
> messy code.
>
> Anne
>
> > File:
> /usr/lib/python2.6/site-packages/scipy/optimize/minpack.py
> > Definition:       newton(func, x0, fprime=None, args=(), tol=1.48e-08,
> > maxiter=50)
> >
> > x0 : float
> >         An initial estimate of the zero that should be somewhere near the
> >         actual zero.
> >
> > (Without using a for-loop)
> >
> > Thanks.
> >
> > --
> > G?khan
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100414/af89e704/attachment.html>

From charlesr.harris at gmail.com  Wed Apr 14 20:33:57 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 14 Apr 2010 18:33:57 -0600
Subject: [SciPy-User] Vectorized newton function
In-Reply-To: <o2n49d6b3501004141715ice4b6720hf292ed88677f99b5@mail.gmail.com>
References: <o2v49d6b3501004141537wdf78861ez75edf7a0cf229778@mail.gmail.com>
	<l2zce557a361004141623k9cd32132j1bc63cc346c0bcea@mail.gmail.com>
	<o2n49d6b3501004141715ice4b6720hf292ed88677f99b5@mail.gmail.com>
Message-ID: <q2ye06186141004141733r58518ff2k4216023319be5f81@mail.gmail.com>

On Wed, Apr 14, 2010 at 6:15 PM, G?khan Sever <gokhansever at gmail.com> wrote:

> Thanks Anne for suggesting "fsolve" and rest of the useful information.
> Below is how my function is read:
>
> def myfunc(myarray):
>        term1 = compute1_array1
>        term2 = compute3_array2
>        term3 = compute3_array3
>        return term1+term2+term3
>
> result_arr = fsolve(myfunc, myarray)
>
> IDL handles univariate and multivariate cases using one fun newton
> function. I get almost equivalent results using fsolve to IDL's newton.
>
>
>
The fsolve function is essentially multidimensional newton, IIRC.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100414/66aa884d/attachment.html>

From robert.kern at gmail.com  Wed Apr 14 22:26:51 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 14 Apr 2010 21:26:51 -0500
Subject: [SciPy-User] Vectorized newton function
In-Reply-To: <q2ye06186141004141733r58518ff2k4216023319be5f81@mail.gmail.com>
References: <o2v49d6b3501004141537wdf78861ez75edf7a0cf229778@mail.gmail.com> 
	<l2zce557a361004141623k9cd32132j1bc63cc346c0bcea@mail.gmail.com> 
	<o2n49d6b3501004141715ice4b6720hf292ed88677f99b5@mail.gmail.com> 
	<q2ye06186141004141733r58518ff2k4216023319be5f81@mail.gmail.com>
Message-ID: <z2g3d375d731004141926mab58fd84n7b909d8c08df905@mail.gmail.com>

On Wed, Apr 14, 2010 at 19:33, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
> On Wed, Apr 14, 2010 at 6:15 PM, G?khan Sever <gokhansever at gmail.com> wrote:
>>
>> Thanks Anne for suggesting "fsolve" and rest of the useful information.
>> Below is how my function is read:
>>
>> def myfunc(myarray):
>> ?????? term1 = compute1_array1
>> ?????? term2 = compute3_array2
>> ?????? term3 = compute3_array3
>> ?????? return term1+term2+term3
>>
>> result_arr = fsolve(myfunc, myarray)
>>
>> IDL handles univariate and multivariate cases using one fun newton
>> function. I get almost equivalent results using fsolve to IDL's newton.
>>
>
> The fsolve function is essentially multidimensional newton, IIRC.

Yes, but it's an N**2 waste when the equations aren't coupled.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From gokhansever at gmail.com  Thu Apr 15 10:35:37 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Thu, 15 Apr 2010 09:35:37 -0500
Subject: [SciPy-User] Vectorized newton function
In-Reply-To: <z2g3d375d731004141926mab58fd84n7b909d8c08df905@mail.gmail.com>
References: <o2v49d6b3501004141537wdf78861ez75edf7a0cf229778@mail.gmail.com>
	<l2zce557a361004141623k9cd32132j1bc63cc346c0bcea@mail.gmail.com>
	<o2n49d6b3501004141715ice4b6720hf292ed88677f99b5@mail.gmail.com>
	<q2ye06186141004141733r58518ff2k4216023319be5f81@mail.gmail.com>
	<z2g3d375d731004141926mab58fd84n7b909d8c08df905@mail.gmail.com>
Message-ID: <k2j49d6b3501004150735q78d2f7fjdc588d0460339506@mail.gmail.com>

On Wed, Apr 14, 2010 at 9:26 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Wed, Apr 14, 2010 at 19:33, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> > On Wed, Apr 14, 2010 at 6:15 PM, G?khan Sever <gokhansever at gmail.com>
> wrote:
> >>
> >> Thanks Anne for suggesting "fsolve" and rest of the useful information.
> >> Below is how my function is read:
> >>
> >> def myfunc(myarray):
> >>        term1 = compute1_array1
> >>        term2 = compute3_array2
> >>        term3 = compute3_array3
> >>        return term1+term2+term3
> >>
> >> result_arr = fsolve(myfunc, myarray)
> >>
> >> IDL handles univariate and multivariate cases using one fun newton
> >> function. I get almost equivalent results using fsolve to IDL's newton.
> >>
> >
> > The fsolve function is essentially multidimensional newton, IIRC.
>
> Yes, but it's an N**2 waste when the equations aren't coupled.
>

Robert,

Could you give an example what you mean here?

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100415/0fb34f48/attachment.html>

From josef.pktd at gmail.com  Thu Apr 15 10:39:57 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 15 Apr 2010 10:39:57 -0400
Subject: [SciPy-User] Vectorized newton function
In-Reply-To: <k2j49d6b3501004150735q78d2f7fjdc588d0460339506@mail.gmail.com>
References: <o2v49d6b3501004141537wdf78861ez75edf7a0cf229778@mail.gmail.com>
	<l2zce557a361004141623k9cd32132j1bc63cc346c0bcea@mail.gmail.com>
	<o2n49d6b3501004141715ice4b6720hf292ed88677f99b5@mail.gmail.com>
	<q2ye06186141004141733r58518ff2k4216023319be5f81@mail.gmail.com>
	<z2g3d375d731004141926mab58fd84n7b909d8c08df905@mail.gmail.com>
	<k2j49d6b3501004150735q78d2f7fjdc588d0460339506@mail.gmail.com>
Message-ID: <z2i1cd32cbb1004150739xfaa3646fw7d342d6d3a12d68a@mail.gmail.com>

On Thu, Apr 15, 2010 at 10:35 AM, G?khan Sever <gokhansever at gmail.com> wrote:
>
>
> On Wed, Apr 14, 2010 at 9:26 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Wed, Apr 14, 2010 at 19:33, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>> >
>> > On Wed, Apr 14, 2010 at 6:15 PM, G?khan Sever <gokhansever at gmail.com>
>> > wrote:
>> >>
>> >> Thanks Anne for suggesting "fsolve" and rest of the useful information.
>> >> Below is how my function is read:
>> >>
>> >> def myfunc(myarray):
>> >> ?????? term1 = compute1_array1
>> >> ?????? term2 = compute3_array2
>> >> ?????? term3 = compute3_array3
>> >> ?????? return term1+term2+term3
>> >>
>> >> result_arr = fsolve(myfunc, myarray)
>> >>
>> >> IDL handles univariate and multivariate cases using one fun newton
>> >> function. I get almost equivalent results using fsolve to IDL's newton.
>> >>
>> >
>> > The fsolve function is essentially multidimensional newton, IIRC.
>>
>> Yes, but it's an N**2 waste when the equations aren't coupled.
>
> Robert,
>
> Could you give an example what you mean here?

For example the Hessian in an uncoupled system is a diagonal matrix
but without restriction, the solver needs to calculate all n*(n-1)/2
terms, including those that are zero but the method doesn't know this.
Other operations also work on the full Hessian.

Josef

> --
> G?khan
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From josef.pktd at gmail.com  Thu Apr 15 11:34:28 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 15 Apr 2010 11:34:28 -0400
Subject: [SciPy-User] Can I "fix" scale=1 when fitting distributions?
In-Reply-To: <s2s1cd32cbb1004132156l78d4461ax6dde01800336878a@mail.gmail.com>
References: <k2p469ba0f41004132109tb47e72bdjbef3d5aa303beeb7@mail.gmail.com>
	<i2w1cd32cbb1004132152z43676e96xb8043459b25e9cd6@mail.gmail.com>
	<s2s1cd32cbb1004132156l78d4461ax6dde01800336878a@mail.gmail.com>
Message-ID: <t2o1cd32cbb1004150834s84ae4a5ck3478605a6471590a@mail.gmail.com>

On Wed, Apr 14, 2010 at 12:56 AM,  <josef.pktd at gmail.com> wrote:
> On Wed, Apr 14, 2010 at 12:52 AM, ?<josef.pktd at gmail.com> wrote:
>> On Wed, Apr 14, 2010 at 12:09 AM, David Ho <itsdho at ucla.edu> wrote:
>>> Actually, being able to "fix" scale=1 would be very useful for me.
>>>
>>> The reason I'm trying to fit a von mises distribution to my data is to find
>>> "kappa", a measure of the concentration of the data.
>>>
>>> On wikipedia, I see that the von mises distribution only really has 2
>>> parameters: mu (the mean), and kappa (1/kappa being analogous to the
>>> variance).
>>> When I use vonmises.fit(), though, I get 3 parameters: kappa, mu, and a
>>> "scale parameter", respectively.
>>> However, I don't think the scale parameter for the von mises distribution is
>>> really independent of kappa, is it? (Am I understanding this correctly?)
>>> (Using the normal distribution as a comparison, I think the "scale
>>> parameter" for a normal distribution certainly isn't independent of sigma,
>>> and norm.fit() only returns mu and sigma. This makes a lot more sense to
>>> me.)
>>
>> for normal ?distribution loc=mean and scale ?= sqrt(variance) ?and has
>> no shape parameter.
>>
>> essentially all distributions are transformed ?y = (x-mean)/scale
>> where x is the original data, and y is the standardized data, the
>> actual _pdf, _cdf, ... are for the standard version of the
>> distribution,
>> it's a generic framework, but doesn't make sense in all cases or
>> applications mainly when we want to fix the support of the
>> distribution
>>
>>>
>>> I'm basically trying to use kappa as a measure of the "width" of the
>>> distribution, so the extra degree of freedom introduced by the scale
>>> parameter is problematic for me. For two distributions that are
>>> superficially very similar in "width", I might get wildly varying values of
>>> kappa.
>>>
>>> For example, I once got fitted values of kappa=1, and kappa=400 for two
>>> distributions that looked very similar in width.
>>> I thought I must have been doing something wrong... until I saw the scale
>>> parameters were around 1 and 19, respectively.
>>> Plotting the fitted distributions:
>>>>>> xs = numpy.linspace(-numpy.pi, numpy.pi, 361)
>>>>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 1, loc=0, scale=1))
>>>>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 400, loc=0,
>>>>>> scale=19))
>>
>> one idea is to check the implied moments, eg. vonmises.stats(1, loc=0,
>> scale=1, moment='mvsk'))
>> to see whether the implied mean, variance, skew, kurtosis are similar
>> in two different parameterizations.
>> Warning skew, kurtosis is incorrect for a few distributions, I don't
>> know about vonmises.
>>
>>>
>>> What I'd really like to do is "force" scale=1 when I perform the fitting.
>>> (In fact, it would be nice to even force the mean of the fitted distribution
>>> to a precalculated value, as well. I really only want one degree of freedom
>>> for the fitting algorithm -- I just want it to explore different values of
>>> kappa.)
>>>
>>> Is there any way to do this?
>>
>> It's not yet possible in scipy, but easy to write, essentially copy
>> the existing fit and nnlf methods and fix loc and scale and call the
>> optimization with only one argument.
>>
>> I have a more than one year old ticket:
>> http://projects.scipy.org/scipy/ticket/832
>>
>> It might be possible that it works immediately if you monkey patch
>> rv_continuous ?by adding
>> nnlf_fr and fit_fr
>> in
>> http://mail.scipy.org/pipermail/scipy-user/2009-February/019968.html
>>
>> scipy.stats.rv_continuous.nnlf_fr = nnlfr
>> scipy.stats.rv_continuous.fit_fr = fit_fr
>
> correction
> this should instead refer to
> scipy.stats.distributions.rv_continuous

It looks like it works, checked only on a few examples, see attachment

Josef

>
> Josef
>>
>> and call
>> vonmises.fit_fr(data, frozen=[np.nan, 0.0, 1.0])
>>
>> It's worth a try, but if you only need vonmises it might also be easy
>> to use scipy.optimize.fmin on a function that wraps vonmises._nnlf and
>> fixes loc, scale.
>> To give you an idea, I didn't check what the correct function arguments are ::
>>
>> def mynnlf(kappa):
>> ? ?loc = 0
>> ? ?scale = 1
>> ? ?return vonmises.nnlf(kappa, loc, scale)
>>
>> scipy.optimize.fmin(mynnlf, startingvalueforkappa)
>>
>> I could check this for your case at the end of the week.
>>
>> If you have an opinion for a good generic API for fit_semifrozen, let me know.
>> I hope this helps, and please keep me informed. I would like to make
>> at least a cookbook recipe out of it.
>>
>> Josef
>>
>>
>>> Thanks for your help,
>>>
>>> --David Ho
>>>
>>>
>>>> On Tue, Apr 13, 2010 at 3:13 PM, ?<josef.pktd at gmail.com> wrote:
>>>> > In the current fit version, loc (and with it the support of the
>>>> > distribution) and scale are always estimated. In some cases this is
>>>> > not desired.
>>>> > You are transforming the original data to fit into the standard
>>>> > distribution with loc=0, scale=1 . Do you get reasonable estimates for
>>>> > loc and scale in this case?
>>>> > If not, then there is another patch or enhanced fit function that
>>>> > could take loc and scale as fixed.
>>>> >
>>>> > I will look at some details in your function later, especially I'm
>>>> > curious how the circular statistics works.
>>>> >
>>>> > Thanks for the example, maybe I can stop ignoring vonmises.
>>>> >
>>>> > Josef
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>
-------------- next part --------------
'''patching scipy to fit distributions with some fixed/frozen parameters

'''

#import numpy
import numpy as np
#import matplotlib.pyplot as mp
#import scipy.stats
from scipy import stats, optimize


########## patching scipy

stats.distributions.vonmises.a = -np.pi
stats.distributions.vonmises.b = np.pi



def _fitstart(self, x):
    # method of moment estimator as starting values, not verified
    # with literature
    loc = np.min([x.min(),0])
    a = 4/stats.skew(x)**2
    scale = np.std(x) / np.sqrt(a)
    return (a, loc, scale)

def nnlf_fr(self, thetash, x, frmask):
    # new frozen version
    # - sum (log pdf(x, theta),axis=0)
    #   where theta are the parameters (including loc and scale)
    #
    try:
        if frmask != None:
            theta = frmask.copy()
            theta[np.isnan(frmask)] = thetash
        else:
            theta = thetash
        loc = theta[-2]
        scale = theta[-1]
        args = tuple(theta[:-2])
    except IndexError:
        raise ValueError, "Not enough input arguments."
    if not self._argcheck(*args) or scale <= 0:
        return np.inf
    x = np.array((x-loc) / scale)
    cond0 = (x <= self.a) | (x >= self.b)
    if (np.any(cond0)):
        return np.inf
    else:
        N = len(x)
        #raise ValueError
        return self._nnlf(x, *args) + N*np.log(scale)

def fit_fr(self, data, *args, **kwds):
    loc0, scale0 = map(kwds.get, ['loc', 'scale'],[0.0, 1.0])
    Narg = len(args)
    
    if Narg == 0 and hasattr(self, '_fitstart'):
        x0 = self._fitstart(data)
    elif Narg > self.numargs:
        raise ValueError, "Too many input arguments."
    else:
        args += (1.0,)*(self.numargs-Narg)
        # location and scale are at the end
        x0 = args + (loc0, scale0)
    
    if 'frozen' in kwds:
        frmask = np.array(kwds['frozen'])
        if len(frmask) != self.numargs+2:
            raise ValueError, "Incorrect number of frozen arguments."
        else:
            # keep starting values for not frozen parameters
            x0  = np.array(x0)[np.isnan(frmask)]
    else:
        frmask = None
        
    #print x0
    #print frmask
    return optimize.fmin(self.nnlf_fr, x0,
                args=(np.ravel(data), frmask), disp=0)

stats.distributions.rv_continuous.fit_fr = fit_fr
stats.distributions.rv_continuous.nnlf_fr = nnlf_fr

########## end patching scipy

for nobs in [20000, 1000, 100]:
    x = stats.vonmises.rvs(1.23, loc=0, scale=1, size=nobs)
    print '\nnobs:', nobs
    print 'true parameter'
    print '1.23, loc=0, scale=1'
    print 'unconstraint'
    print stats.vonmises.fit(x)
    print stats.vonmises.fit_fr(x, frozen=[np.nan, np.nan, np.nan])
    print 'with fixed loc and scale'
    print stats.vonmises.fit_fr(x, frozen=[np.nan, 0.0, 1.0])


distr = stats.gamma
arg, loc, scale = 2.5, 0., 20.

for nobs in [20000, 1000, 100]:
    x = distr.rvs(arg, loc=loc, scale=scale, size=nobs)
    print '\nnobs:', nobs
    print 'true parameter'
    print '%f, loc=%f, scale=%f' % (arg, loc, scale)
    print 'unconstraint'
    print distr.fit(x)
    print distr.fit_fr(x, frozen=[np.nan, np.nan, np.nan])
    print 'with fixed loc and scale'
    print distr.fit_fr(x, frozen=[np.nan, 0.0, 1.0])
    print 'with fixed loc'
    print distr.fit_fr(x, frozen=[np.nan, 0.0, np.nan])

From georges.schutz at internet.lu  Thu Apr 15 12:07:55 2010
From: georges.schutz at internet.lu (Georges Schutz)
Date: Thu, 15 Apr 2010 18:07:55 +0200
Subject: [SciPy-User] scikits.timeseries: cmov_average() strange result for
	dtype int
Message-ID: <hq7dl8$2jn$1@dough.gmane.org>

Hello all
first thank you all for your great work and the really useful scikits 
extensions.

While analysing the behaviour of the lib.moving_funcs of 
scikits.timeseries I encountered a strange but explainable behaviour 
using the cmov_average() method in the case of integer data.

cmov_average() should do a "centralized" moving average on a 
time-series. This seams also OK for float data but in the case of 
integer data the behaviour is not as one could expect form numerical 
point of view.

Here two examples to illustrate what I mean:
1) TS = [-- 4 4 4 0 0 0 --]
as Integer: cmov_average(TS,3) -> [-- 4 4 2 1 0 0 --]
as float: cmov_average(TS,3)   -> [-- 4.0 4.0 2.667 1.333 0.0 0.0 --]
I would have expected form the signal / numerical point of view for the 
integer result -> [-- 4 4 3 1 0 0 --] which is symmetric like the 
original signal.

2) TS = [-- 1 1 1 0 0 0 --]
as Integer: cmov_average(TS,3) -> [-- 1 1 0 0 0 0 --]
as float: cmov_average(TS,3)   -> [-- 1.0 1.0 0.667 0.333 0.0 0.0 --]
My expectation would have been -> [-- 1 1 1 0 0 0 --]
In this case the central moving average is simply shifting the step to 
the left by one timestamp.

The reason of this behaviour is the floor rounding done by default if 
using integer division. If the round() method where used the result 
where as I would have expected.

Is there a possibility to change this default behaviour knowing that the 
functions cmov_average() and others are implemented in C.

Best Regards
Georges Schutz

--- Here the used code ---
import numpy as np
import scikits.timeseries as ts
from scikits.timeseries.lib.moving_funcs import cmov_average

ISeries = ts.time_series(np.array([3]*4+[0]*4), #Change 3 to 1 for ex. 2
                          start_date=ts.Date(freq='D',
                                             string='2010-01-01'),
                          dtype=int)
FSeries = ts.time_series(ISeries, dtype=float)
print cmov_average(ISeries, 3)
print cmov_average(FSeries, 3)





From itsdho at ucla.edu  Thu Apr 15 12:53:36 2010
From: itsdho at ucla.edu (David Ho)
Date: Thu, 15 Apr 2010 09:53:36 -0700
Subject: [SciPy-User] Is there an easy way to plot a log-frequency
	spectrogram?
Message-ID: <q2l469ba0f41004150953g2da08d02u404b0dbecee85c61@mail.gmail.com>

Quick question: matplotlib's specgram() by default plots a spectrogram with
a linear y-axis (frequency).
Is there an easy way to plot a spectrogram using a logarithmic frequency
axis?

Thanks!

--David Ho
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100415/88f89615/attachment.html>

From robert.kern at gmail.com  Thu Apr 15 12:54:58 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 15 Apr 2010 11:54:58 -0500
Subject: [SciPy-User] Is there an easy way to plot a log-frequency
	spectrogram?
In-Reply-To: <q2l469ba0f41004150953g2da08d02u404b0dbecee85c61@mail.gmail.com>
References: <q2l469ba0f41004150953g2da08d02u404b0dbecee85c61@mail.gmail.com>
Message-ID: <h2i3d375d731004150954m4ed6c415t6fa2ecbfcdd50519@mail.gmail.com>

On Thu, Apr 15, 2010 at 11:53, David Ho <itsdho at ucla.edu> wrote:
> Quick question: matplotlib's specgram() by default plots a spectrogram with
> a linear y-axis (frequency).
> Is there an easy way to plot a spectrogram using a logarithmic frequency
> axis?

The matplotlib list is over here:

https://lists.sourceforge.net/lists/listinfo/matplotlib-users

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Dharhas.Pothina at twdb.state.tx.us  Thu Apr 15 14:54:58 2010
From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina)
Date: Thu, 15 Apr 2010 13:54:58 -0500
Subject: [SciPy-User] line/edge detection and output of coordinates.
Message-ID: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>

Hi,

I'm trying to do some line/edge detection in python. I've done some googling and found some mailing list archives that talked about ways to do edge detection but they seem to mainly return an image with the edges highlighted. What I need is x,y coordinates of the pixels that make up the lines. I've attached an image that shows a dark band on a light background. The final output I need would be line definitions in terms of a series of x,y coordinates for the upper light to dark interface and the lower dark to light interface of the band.

Any pointers on packages to use or ways to do this are highly appreciated.

Thanks,

- dharhas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: example_small.jpg
Type: image/jpeg
Size: 35045 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100415/bd67be42/attachment.jpg>

From david_baddeley at yahoo.com.au  Thu Apr 15 16:21:24 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Thu, 15 Apr 2010 13:21:24 -0700 (PDT)
Subject: [SciPy-User] line/edge detection and output of coordinates.
In-Reply-To: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
References: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
Message-ID: <934944.95035.qm@web33007.mail.mud.yahoo.com>

Hi Dharhas,

most traditional edge detection algorithms use some kind of gradient based filter - your image has two features which suggest that this approach might not be ideal. First it is noisy (which means that any gradient image will be even noisier) and second, you actually have relatively good contrast as to what is part of the object and what is not, and might not need gradient based edge detection per se. I'd take the following approach:

- low pass filter with scipy.ndimage.gaussian_filter to reduce the noise
- threshold the image to get a binary mask of your band (eg mask = image > threshold)
- potentially use a morphological operation such as binary closing (scipy.ndimage.binary_closing) or hole filling to tidy up the mask

from this mask you have a number of options to get the edges:
A - do a binary dilation and then subtract the original mask - this should give you a mask with all the edge pixels (both top and bottom). Applying ndimage.label to this mask might allow you to extract masks of both edges separately.

B - as you want to find the top and bottom edges (rather than edges which can fold back on themselves), you could take the difference between the mask and a copy of the mask which has been shifted vertically by one pixel (eg mask[:-1,:] - mask[1:,:]). This should give you an image in which the top edge pixels have a value of 1, the bottom edge has a value of -1, and all other pixels are zero. 

once you have an image in which the pixels of each edge have different values, you can find the coordinates using numpy.where.

cheers,
David



----- Original Message ----
From: Dharhas Pothina <Dharhas.Pothina at twdb.state.tx.us>
To: SciPy at yahoo.com
Sent: Fri, 16 April, 2010 6:54:58 AM
Subject: [SciPy-User] line/edge detection and output of coordinates.

Hi,

I'm trying to do some line/edge detection in python. I've done some googling and found some mailing list archives that talked about ways to do edge detection but they seem to mainly return an image with the edges highlighted. What I need is x,y coordinates of the pixels that make up the lines. I've attached an image that shows a dark band on a light background. The final output I need would be line definitions in terms of a series of x,y coordinates for the upper light to dark interface and the lower dark to light interface of the band.

Any pointers on packages to use or ways to do this are highly appreciated.

Thanks,

- dharhas


      


From schlesin at cshl.edu  Thu Apr 15 17:29:27 2010
From: schlesin at cshl.edu (Felix Schlesinger)
Date: Thu, 15 Apr 2010 17:29:27 -0400
Subject: [SciPy-User] Sparse vector
Message-ID: <j2y7317d50c1004151429j5141589dr4bab16e09bd55ae9@mail.gmail.com>

Hello,

I was wondering what peoples recommendations and thoughts are on
sparse vectors, i.e. long vectors where most entries are 0.
1-D numpy arrays waste a lot of memory in that case. Python
defaultdicts still use more memory then should be needed (since they
store python objects) and do not work well for numpy math operations
and slicing. Scipy.sparse has several implementations for sparse 2D
matrices which could be used for vectors, but that does not seem ideal
for clarity, efficiency and function broadcasting. Is there something
else out there or am I maybe missing a simple way to do it
efficiently?
In my particular case the vectors would be write-once, read-often and
maybe about 1% filled with integers. They are small enough to fit into
memory in dense form one at a time during construction.

Thanks
  Felix


From timmichelsen at gmx-topmail.de  Thu Apr 15 17:38:53 2010
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Thu, 15 Apr 2010 23:38:53 +0200
Subject: [SciPy-User] line/edge detection and output of coordinates.
In-Reply-To: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
References: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
Message-ID: <hq811d$h6v$1@dough.gmane.org>

> I'm trying to do some line/edge detection in python. I've done some
> googling and found some mailing list archives that talked about ways
> to do edge detection but they seem to mainly return an image with the
> edges highlighted. What I need is x,y coordinates of the pixels that
> make up the lines. I
I'd go for a GIS application (GRASS, gvSIG, etc.):
1) import
2) convert raster to vector
3) export



From Chris.Barker at noaa.gov  Thu Apr 15 17:42:09 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 15 Apr 2010 14:42:09 -0700
Subject: [SciPy-User] line/edge detection and output of coordinates.
In-Reply-To: <hq811d$h6v$1@dough.gmane.org>
References: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
	<hq811d$h6v$1@dough.gmane.org>
Message-ID: <4BC78831.8070904@noaa.gov>

Tim Michelsen wrote:
>> I'm trying to do some line/edge detection in python. I've done some
>> googling and found some mailing list archives that talked about ways
>> to do edge detection but they seem to mainly return an image with the
>> edges highlighted. What I need is x,y coordinates of the pixels that
>> make up the lines. I
> I'd go for a GIS application (GRASS, gvSIG, etc.):

or the GDAL command line utilities.


Another option is to look at drawing programs -- InkScape has something 
that would probably work.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From peridot.faceted at gmail.com  Thu Apr 15 18:18:02 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 15 Apr 2010 18:18:02 -0400
Subject: [SciPy-User] Sparse vector
In-Reply-To: <j2y7317d50c1004151429j5141589dr4bab16e09bd55ae9@mail.gmail.com>
References: <j2y7317d50c1004151429j5141589dr4bab16e09bd55ae9@mail.gmail.com>
Message-ID: <s2sce557a361004151518zab4b8f24oba2ee12d991a1c10@mail.gmail.com>

On 15 April 2010 17:29, Felix Schlesinger <schlesin at cshl.edu> wrote:
> Hello,
>
> I was wondering what peoples recommendations and thoughts are on
> sparse vectors, i.e. long vectors where most entries are 0.
> 1-D numpy arrays waste a lot of memory in that case. Python
> defaultdicts still use more memory then should be needed (since they
> store python objects) and do not work well for numpy math operations
> and slicing. Scipy.sparse has several implementations for sparse 2D
> matrices which could be used for vectors, but that does not seem ideal
> for clarity, efficiency and function broadcasting. Is there something
> else out there or am I maybe missing a simple way to do it
> efficiently?
> In my particular case the vectors would be write-once, read-often and
> maybe about 1% filled with integers. They are small enough to fit into
> memory in dense form one at a time during construction.

The short answer is, no, there's no support for such a thing in numpy/scipy.

There's no way to make such a thing under-the-hood compatible with
numpy arrays, since they require evenly-strided memory. And scipy's
sparse matrices are built on the assumption that an n by n matrix will
have at least O(n) nonzero elements, so you are going to have to watch
carefully what you do with your sparse vectors. That said, dok
matrices should be all right (though no more efficient than
defaultdicts) and one of csr/csc matrices will be efficient, depending
on whether you view your vectors as row or column matrices.

I occasionally wonder whether a generalized sparse ndarray object
would be useful. You'd want it to specify the value for empty elements
so that things like boolean arrays could be done this way, and I think
a dictionary of keys approach would be the way to go. In any case,
nothing exists now.

Anne

> Thanks
> ?Felix
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Thu Apr 15 18:29:47 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 15 Apr 2010 18:29:47 -0400
Subject: [SciPy-User] Sparse vector
In-Reply-To: <s2sce557a361004151518zab4b8f24oba2ee12d991a1c10@mail.gmail.com>
References: <j2y7317d50c1004151429j5141589dr4bab16e09bd55ae9@mail.gmail.com>
	<s2sce557a361004151518zab4b8f24oba2ee12d991a1c10@mail.gmail.com>
Message-ID: <i2n1cd32cbb1004151529x9df9d1cdv1ef2c4019eea9551@mail.gmail.com>

On Thu, Apr 15, 2010 at 6:18 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> On 15 April 2010 17:29, Felix Schlesinger <schlesin at cshl.edu> wrote:
>> Hello,
>>
>> I was wondering what peoples recommendations and thoughts are on
>> sparse vectors, i.e. long vectors where most entries are 0.
>> 1-D numpy arrays waste a lot of memory in that case. Python
>> defaultdicts still use more memory then should be needed (since they
>> store python objects) and do not work well for numpy math operations
>> and slicing. Scipy.sparse has several implementations for sparse 2D
>> matrices which could be used for vectors, but that does not seem ideal
>> for clarity, efficiency and function broadcasting. Is there something
>> else out there or am I maybe missing a simple way to do it
>> efficiently?
>> In my particular case the vectors would be write-once, read-often and
>> maybe about 1% filled with integers. They are small enough to fit into
>> memory in dense form one at a time during construction.
>
> The short answer is, no, there's no support for such a thing in numpy/scipy.
>
> There's no way to make such a thing under-the-hood compatible with
> numpy arrays, since they require evenly-strided memory. And scipy's
> sparse matrices are built on the assumption that an n by n matrix will
> have at least O(n) nonzero elements, so you are going to have to watch
> carefully what you do with your sparse vectors. That said, dok
> matrices should be all right (though no more efficient than
> defaultdicts) and one of csr/csc matrices will be efficient, depending
> on whether you view your vectors as row or column matrices.
>
> I occasionally wonder whether a generalized sparse ndarray object
> would be useful. You'd want it to specify the value for empty elements
> so that things like boolean arrays could be done this way, and I think
> a dictionary of keys approach would be the way to go. In any case,
> nothing exists now.

for some applications, I keep just the nonzero values in an array and
the index separate, which allows easy back and forth conversion to
dense.
But I don't really use it as substitute for sparse, just so that I
have a convenient representation e.g. for optimization and for easier
input.

Josef

>
> Anne
>
>> Thanks
>> ?Felix
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Fri Apr 16 02:24:27 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 16 Apr 2010 02:24:27 -0400
Subject: [SciPy-User] Can I "fix" scale=1 when fitting distributions?
In-Reply-To: <t2o1cd32cbb1004150834s84ae4a5ck3478605a6471590a@mail.gmail.com>
References: <k2p469ba0f41004132109tb47e72bdjbef3d5aa303beeb7@mail.gmail.com>
	<i2w1cd32cbb1004132152z43676e96xb8043459b25e9cd6@mail.gmail.com>
	<s2s1cd32cbb1004132156l78d4461ax6dde01800336878a@mail.gmail.com>
	<t2o1cd32cbb1004150834s84ae4a5ck3478605a6471590a@mail.gmail.com>
Message-ID: <r2h1cd32cbb1004152324m4e101129o395b2ea99c1633a1@mail.gmail.com>

On Thu, Apr 15, 2010 at 11:34 AM,  <josef.pktd at gmail.com> wrote:
> On Wed, Apr 14, 2010 at 12:56 AM, ?<josef.pktd at gmail.com> wrote:
>> On Wed, Apr 14, 2010 at 12:52 AM, ?<josef.pktd at gmail.com> wrote:
>>> On Wed, Apr 14, 2010 at 12:09 AM, David Ho <itsdho at ucla.edu> wrote:
>>>> Actually, being able to "fix" scale=1 would be very useful for me.
>>>>
>>>> The reason I'm trying to fit a von mises distribution to my data is to find
>>>> "kappa", a measure of the concentration of the data.
>>>>
>>>> On wikipedia, I see that the von mises distribution only really has 2
>>>> parameters: mu (the mean), and kappa (1/kappa being analogous to the
>>>> variance).
>>>> When I use vonmises.fit(), though, I get 3 parameters: kappa, mu, and a
>>>> "scale parameter", respectively.
>>>> However, I don't think the scale parameter for the von mises distribution is
>>>> really independent of kappa, is it? (Am I understanding this correctly?)
>>>> (Using the normal distribution as a comparison, I think the "scale
>>>> parameter" for a normal distribution certainly isn't independent of sigma,
>>>> and norm.fit() only returns mu and sigma. This makes a lot more sense to
>>>> me.)
>>>
>>> for normal ?distribution loc=mean and scale ?= sqrt(variance) ?and has
>>> no shape parameter.
>>>
>>> essentially all distributions are transformed ?y = (x-mean)/scale
>>> where x is the original data, and y is the standardized data, the
>>> actual _pdf, _cdf, ... are for the standard version of the
>>> distribution,
>>> it's a generic framework, but doesn't make sense in all cases or
>>> applications mainly when we want to fix the support of the
>>> distribution
>>>
>>>>
>>>> I'm basically trying to use kappa as a measure of the "width" of the
>>>> distribution, so the extra degree of freedom introduced by the scale
>>>> parameter is problematic for me. For two distributions that are
>>>> superficially very similar in "width", I might get wildly varying values of
>>>> kappa.
>>>>
>>>> For example, I once got fitted values of kappa=1, and kappa=400 for two
>>>> distributions that looked very similar in width.
>>>> I thought I must have been doing something wrong... until I saw the scale
>>>> parameters were around 1 and 19, respectively.
>>>> Plotting the fitted distributions:
>>>>>>> xs = numpy.linspace(-numpy.pi, numpy.pi, 361)
>>>>>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 1, loc=0, scale=1))
>>>>>>> plot(xs, scipy.stats.distributions.vonmises.pdf(xs, 400, loc=0,
>>>>>>> scale=19))
>>>
>>> one idea is to check the implied moments, eg. vonmises.stats(1, loc=0,
>>> scale=1, moment='mvsk'))
>>> to see whether the implied mean, variance, skew, kurtosis are similar
>>> in two different parameterizations.
>>> Warning skew, kurtosis is incorrect for a few distributions, I don't
>>> know about vonmises.
>>>
>>>>
>>>> What I'd really like to do is "force" scale=1 when I perform the fitting.
>>>> (In fact, it would be nice to even force the mean of the fitted distribution
>>>> to a precalculated value, as well. I really only want one degree of freedom
>>>> for the fitting algorithm -- I just want it to explore different values of
>>>> kappa.)
>>>>
>>>> Is there any way to do this?
>>>
>>> It's not yet possible in scipy, but easy to write, essentially copy
>>> the existing fit and nnlf methods and fix loc and scale and call the
>>> optimization with only one argument.
>>>
>>> I have a more than one year old ticket:
>>> http://projects.scipy.org/scipy/ticket/832
>>>
>>> It might be possible that it works immediately if you monkey patch
>>> rv_continuous ?by adding
>>> nnlf_fr and fit_fr
>>> in
>>> http://mail.scipy.org/pipermail/scipy-user/2009-February/019968.html
>>>
>>> scipy.stats.rv_continuous.nnlf_fr = nnlfr
>>> scipy.stats.rv_continuous.fit_fr = fit_fr
>>
>> correction
>> this should instead refer to
>> scipy.stats.distributions.rv_continuous
>
> It looks like it works, checked only on a few examples, see attachment

An updated version:
I added a quickly written Bootstrap and MonteCarlo check to the examples.
looks pretty good overall.

It's a bit slow with 5000 bootstrap/montecarlo replications.

results for 1 parameter vonmises:

Bootstrap
true parameter value
1.5
MLE estimate of parameters using sample (nobs=500)
[ 1.46689453]
simple bootstrap distribution of parameter estimate (nrepl=5000)
mean = 1.470008, bias=-0.029992
median 1.46650390625
var and std 0.0082988915425 0.0910982521375
mse, rmse 0.00919842753983 0.0959084331007
bootstrap confidence interval (90% coverage)
1.328515625 1.62685546875
bootstrap confidence interval (90% coverage) normal approximation
1.32016444394 1.61985102481
Kolmogorov-Smirnov test for normality of bootstrap distribution
 - estimated parameters, p-values not really correct
(0.018543534773093451, 0.064215366586253375)

MonteCarlo     (replace bootstrap by montecarlo in text)
true parameter value
1.5
MLE estimate of parameters using sample (nobs=500)
[ 1.46689453]
simple bootstrap distribution of parameter estimate (nrepl=5000)
mean = 1.503128, bias=0.003128
median 1.50078125
var and std 0.00797727984433 0.089315619263
mse, rmse 0.00798706268883 0.0893703680692
bootstrap confidence interval (90% coverage)
1.3638671875 1.654296875
bootstrap confidence interval (90% coverage) normal approximation
1.35621663362 1.65003887419
Kolmogorov-Smirnov test for normality of bootstrap distribution
 - estimated parameters, p-values not really correct
(0.02053833535547045, 0.029449209724884552)
>>>

Josef




>
> Josef
>
>>
>> Josef
>>>
>>> and call
>>> vonmises.fit_fr(data, frozen=[np.nan, 0.0, 1.0])
>>>
>>> It's worth a try, but if you only need vonmises it might also be easy
>>> to use scipy.optimize.fmin on a function that wraps vonmises._nnlf and
>>> fixes loc, scale.
>>> To give you an idea, I didn't check what the correct function arguments are ::
>>>
>>> def mynnlf(kappa):
>>> ? ?loc = 0
>>> ? ?scale = 1
>>> ? ?return vonmises.nnlf(kappa, loc, scale)
>>>
>>> scipy.optimize.fmin(mynnlf, startingvalueforkappa)
>>>
>>> I could check this for your case at the end of the week.
>>>
>>> If you have an opinion for a good generic API for fit_semifrozen, let me know.
>>> I hope this helps, and please keep me informed. I would like to make
>>> at least a cookbook recipe out of it.
>>>
>>> Josef
>>>
>>>
>>>> Thanks for your help,
>>>>
>>>> --David Ho
>>>>
>>>>
>>>>> On Tue, Apr 13, 2010 at 3:13 PM, ?<josef.pktd at gmail.com> wrote:
>>>>> > In the current fit version, loc (and with it the support of the
>>>>> > distribution) and scale are always estimated. In some cases this is
>>>>> > not desired.
>>>>> > You are transforming the original data to fit into the standard
>>>>> > distribution with loc=0, scale=1 . Do you get reasonable estimates for
>>>>> > loc and scale in this case?
>>>>> > If not, then there is another patch or enhanced fit function that
>>>>> > could take loc and scale as fixed.
>>>>> >
>>>>> > I will look at some details in your function later, especially I'm
>>>>> > curious how the circular statistics works.
>>>>> >
>>>>> > Thanks for the example, maybe I can stop ignoring vonmises.
>>>>> >
>>>>> > Josef
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
'''patching scipy to fit distributions with some fixed/frozen parameters

'''

#import numpy
import numpy as np
#import matplotlib.pyplot as mp
#import scipy.stats
from scipy import stats, optimize


########## patching scipy

stats.distributions.vonmises.a = -np.pi
stats.distributions.vonmises.b = np.pi



def _fitstart(self, x):
    # method of moment estimator as starting values, not verified
    # with literature
    loc = np.min([x.min(),0])
    a = 4/stats.skew(x)**2
    scale = np.std(x) / np.sqrt(a)
    return (a, loc, scale)

def nnlf_fr(self, thetash, x, frmask):
    # new frozen version
    # - sum (log pdf(x, theta),axis=0)
    #   where theta are the parameters (including loc and scale)
    #
    try:
        if frmask != None:
            theta = frmask.copy()
            theta[np.isnan(frmask)] = thetash
        else:
            theta = thetash
        loc = theta[-2]
        scale = theta[-1]
        args = tuple(theta[:-2])
    except IndexError:
        raise ValueError, "Not enough input arguments."
    if not self._argcheck(*args) or scale <= 0:
        return np.inf
    x = np.array((x-loc) / scale)
    cond0 = (x <= self.a) | (x >= self.b)
    if (np.any(cond0)):
        return np.inf
    else:
        N = len(x)
        #raise ValueError
        return self._nnlf(x, *args) + N*np.log(scale)

def fit_fr(self, data, *args, **kwds):
    loc0, scale0 = map(kwds.get, ['loc', 'scale'],[0.0, 1.0])
    Narg = len(args)
    
    if Narg == 0 and hasattr(self, '_fitstart'):
        x0 = self._fitstart(data)
    elif Narg > self.numargs:
        raise ValueError, "Too many input arguments."
    else:
        args += (1.0,)*(self.numargs-Narg)
        # location and scale are at the end
        x0 = args + (loc0, scale0)
    
    if 'frozen' in kwds:
        frmask = np.array(kwds['frozen'])
        if len(frmask) != self.numargs+2:
            raise ValueError, "Incorrect number of frozen arguments."
        else:
            # keep starting values for not frozen parameters
            x0  = np.array(x0)[np.isnan(frmask)]
    else:
        frmask = None
        
    #print x0
    #print frmask
    return optimize.fmin(self.nnlf_fr, x0,
                args=(np.ravel(data), frmask), disp=0)

stats.distributions.rv_continuous.fit_fr = fit_fr
stats.distributions.rv_continuous.nnlf_fr = nnlf_fr

########## end patching scipy

for nobs in []:#[20000, 1000, 100]:
    x = stats.vonmises.rvs(1.23, loc=0, scale=1, size=nobs)
    print '\nnobs:', nobs
    print 'true parameter'
    print '1.23, loc=0, scale=1'
    print 'unconstraint'
    print stats.vonmises.fit(x)
    print stats.vonmises.fit_fr(x, frozen=[np.nan, np.nan, np.nan])
    print 'with fixed loc and scale'
    print stats.vonmises.fit_fr(x, frozen=[np.nan, 0.0, 1.0])


distr = stats.gamma
arg, loc, scale = 2.5, 0., 20.

for nobs in []:#[20000, 1000, 100]:
    x = distr.rvs(arg, loc=loc, scale=scale, size=nobs)
    print '\nnobs:', nobs
    print 'true parameter'
    print '%f, loc=%f, scale=%f' % (arg, loc, scale)
    print 'unconstraint'
    print distr.fit(x)
    print distr.fit_fr(x, frozen=[np.nan, np.nan, np.nan])
    print 'with fixed loc and scale'
    print distr.fit_fr(x, frozen=[np.nan, 0.0, 1.0])
    print 'with fixed loc'
    print distr.fit_fr(x, frozen=[np.nan, 0.0, np.nan])


def distfitbootstrap(sample, distr, nrepl=100):
    nobs = len(sample)
    res = np.zeros(nrepl)
    for ii in xrange(nrepl):
        rvsind = np.random.randint(nobs, size=nobs)
        x = sample[rvsind]
        res[ii] = distr.fit_fr(x, frozen=[np.nan, 0.0, 1.0])
    return res

def distfitmc(sample, distr, nrepl=100, distkwds={}):
    arg = distkwds.pop('arg')
    nobs = len(sample)
    res = np.zeros(nrepl)
    for ii in xrange(nrepl):
        x = distr.rvs(arg, size=nobs, **distkwds)
        res[ii] = distr.fit_fr(x, frozen=[np.nan, 0.0, 1.0])
    return res

def printresults(sample, arg, bres):
    print 'true parameter value'
    print arg
    print 'MLE estimate of parameters using sample (nobs=%d)'% (nobs)
    print distr.fit_fr(sample, frozen=[np.nan, 0.0, 1.0])
    print 'simple bootstrap distribution of parameter estimate (nrepl=%d)'% (nrepl)
    print 'mean = %f, bias=%f' % (bres.mean(0), bres.mean(0)-arg)
    print 'median', np.median(bres, axis=0)
    print 'var and std', bres.var(0), np.sqrt(bres.var(0))
    bmse = ((bres - arg)**2).mean(0)
    print 'mse, rmse', bmse, np.sqrt(bmse)
    bressorted = np.sort(bres)
    print 'bootstrap confidence interval (90% coverage)'
    print bressorted[np.floor(nrepl*0.05)], bressorted[np.floor(nrepl*0.95)]
    print 'bootstrap confidence interval (90% coverage) normal approximation'
    print stats.norm.ppf(0.05, loc=bres.mean(), scale=bres.std()),
    print stats.norm.isf(0.05, loc=bres.mean(), scale=bres.std())
    print 'Kolmogorov-Smirnov test for normality of bootstrap distribution'
    print ' - estimated parameters, p-values not really correct'
    print stats.kstest(bres, 'norm', (bres.mean(), bres.std()))

ex = ['gamma', 'vonmises'][1]

if ex == 'gamma':
    distr = stats.gamma
    arg, loc, scale = 2.5, 0., 1
elif ex == 'vonmises':
    distr = stats.vonmises
    arg, loc, scale = 1.5, 0., 1
else:
    raise ValueError('wrong example')

nobs = 500
nrepl = 5000
sample = distr.rvs(arg, loc=loc, scale=scale, size=nobs)

print 'Bootstrap'
bres = distfitbootstrap(sample, distr, nrepl=nrepl )
printresults(sample, arg, bres)

print '\nMonteCarlo'
mcres = distfitmc(sample, distr, nrepl=nrepl,
                        distkwds=dict(arg=arg, loc=loc, scale=scale) )
printresults(sample, arg, mcres)

From t.charrett at cranfield.ac.uk  Fri Apr 16 03:40:50 2010
From: t.charrett at cranfield.ac.uk (Charrett, Thomas)
Date: Fri, 16 Apr 2010 08:40:50 +0100
Subject: [SciPy-User] line/edge detection and output of coordinates.
In-Reply-To: <mailman.486.1271357727.28903.scipy-user@scipy.org>
References: <mailman.486.1271357727.28903.scipy-user@scipy.org>
Message-ID: <CB2B2AE46366814B817A6E0B01EE7B3704D4FD9A80@CCEXCHSTORE-1.central.cranfield.ac.uk>

You could try Radon / Hough transforms - which give you the gradient intercept of the lines.

Tom

-----Original Message-----
From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of scipy-user-request at scipy.org
Sent: 15 April 2010 19:55
To: scipy-user at scipy.org
Subject: SciPy-User Digest, Vol 80, Issue 31

Send SciPy-User mailing list submissions to
	scipy-user at scipy.org

To subscribe or unsubscribe via the World Wide Web, visit
	http://mail.scipy.org/mailman/listinfo/scipy-user
or, via email, send a message with subject or body 'help' to
	scipy-user-request at scipy.org

You can reach the person managing the list at
	scipy-user-owner at scipy.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of SciPy-User digest..."


Today's Topics:

   1. line/edge detection and output of coordinates. (Dharhas Pothina)


----------------------------------------------------------------------

Message: 1
Date: Thu, 15 Apr 2010 13:54:58 -0500
From: "Dharhas Pothina" <Dharhas.Pothina at twdb.state.tx.us>
Subject: [SciPy-User] line/edge detection and output of coordinates.
To: <SciPy Users List <scipy-user at scipy.org>
Message-ID: <4BC71AB3.63BA.009B.0 at twdb.state.tx.us>
Content-Type: text/plain; charset="us-ascii"

Hi,

I'm trying to do some line/edge detection in python. I've done some googling and found some mailing list archives that talked about ways to do edge detection but they seem to mainly return an image with the edges highlighted. What I need is x,y coordinates of the pixels that make up the lines. I've attached an image that shows a dark band on a light background. The final output I need would be line definitions in terms of a series of x,y coordinates for the upper light to dark interface and the lower dark to light interface of the band.

Any pointers on packages to use or ways to do this are highly appreciated.

Thanks,

- dharhas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: example_small.jpg
Type: image/jpeg
Size: 35045 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/scipy-user/attachments/20100415/bd67be42/attachment.jpg 

------------------------------

_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user


End of SciPy-User Digest, Vol 80, Issue 31
******************************************


From georges.schutz at internet.lu  Fri Apr 16 05:35:37 2010
From: georges.schutz at internet.lu (Georges Schutz)
Date: Fri, 16 Apr 2010 11:35:37 +0200
Subject: [SciPy-User] scikits.timeseries: How to define frequency of 15
	minutes
Message-ID: <hq9b1d$282$1@dough.gmane.org>

I have a data logger giving me a value all 15 minutes. I would like to 
use DateArray from scikits.timeseries to handle this time series.
I do not see how to define a frequency that is out of the specified 
constants. Is there a possibility?

If I use the guess_freq() on that time data it returns 8000 ('T') which 
is a minutely frequency. Using this the has_missing_dates() gave me a 
True what is obvious as it checks for a resolution of one minute.

The situation is the same for other timeseries that I have with a 
frequency of 30 seconds or 2 hours, 6 hours, ...

As I am new to the use of scikits.timeseries and scipy in general so I 
may have missed an obvious way to do this.

Thanks for your help

Georges Schutz



From Dharhas.Pothina at twdb.state.tx.us  Fri Apr 16 08:25:39 2010
From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina)
Date: Fri, 16 Apr 2010 07:25:39 -0500
Subject: [SciPy-User] line/edge detection and output of	coordinates.
In-Reply-To: <4BC78831.8070904@noaa.gov>
References: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
	<hq811d$h6v$1@dough.gmane.org> <4BC78831.8070904@noaa.gov>
Message-ID: <4BC810F7.63BA.009B.0@twdb.state.tx.us>

Tim/Chris,

So for the GIS/GDAL suggestion are you talking about exporting the lines once I've used another technique to highlight the edges? Or will the Raster 2 Vector functions also help with the edge detection.

- dharhas

>>> Christopher Barker <Chris.Barker at noaa.gov> 4/15/2010 4:42 PM >>>
Tim Michelsen wrote:
>> I'm trying to do some line/edge detection in python. I've done some
>> googling and found some mailing list archives that talked about ways
>> to do edge detection but they seem to mainly return an image with the
>> edges highlighted. What I need is x,y coordinates of the pixels that
>> make up the lines. I
> I'd go for a GIS application (GRASS, gvSIG, etc.):

or the GDAL command line utilities.


Another option is to look at drawing programs -- InkScape has something 
that would probably work.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov 
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user



From Dharhas.Pothina at twdb.state.tx.us  Fri Apr 16 08:26:50 2010
From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina)
Date: Fri, 16 Apr 2010 07:26:50 -0500
Subject: [SciPy-User] line/edge detection and output of	coordinates.
In-Reply-To: <CB2B2AE46366814B817A6E0B01EE7B3704D4FD9A80@CCEXCHSTORE-1.central.cranfield.ac.uk>
References: <mailman.486.1271357727.28903.scipy-user@scipy.org>
	<CB2B2AE46366814B817A6E0B01EE7B3704D4FD9A80@CCEXCHSTORE-1.central.cranfield.ac.uk>
Message-ID: <4BC8113E.63BA.009B.0@twdb.state.tx.us>

Tom,

I found the Radon/Hough transforms while googling but all the examples I found were for detecting straight lines. Do they work for curves?

thanks

- dharhas

>>> "Charrett, Thomas" <t.charrett at cranfield.ac.uk> 4/16/2010 2:40 AM >>>
You could try Radon / Hough transforms - which give you the gradient intercept of the lines.

Tom

-----Original Message-----
From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of scipy-user-request at scipy.org 
Sent: 15 April 2010 19:55
To: scipy-user at scipy.org 
Subject: SciPy-User Digest, Vol 80, Issue 31

Send SciPy-User mailing list submissions to
	scipy-user at scipy.org 

To subscribe or unsubscribe via the World Wide Web, visit
	http://mail.scipy.org/mailman/listinfo/scipy-user 
or, via email, send a message with subject or body 'help' to
	scipy-user-request at scipy.org 

You can reach the person managing the list at
	scipy-user-owner at scipy.org 

When replying, please edit your Subject line so it is more specific
than "Re: Contents of SciPy-User digest..."


Today's Topics:

   1. line/edge detection and output of coordinates. (Dharhas Pothina)


----------------------------------------------------------------------

Message: 1
Date: Thu, 15 Apr 2010 13:54:58 -0500
From: "Dharhas Pothina" <Dharhas.Pothina at twdb.state.tx.us>
Subject: [SciPy-User] line/edge detection and output of coordinates.
To: <SciPy Users List <scipy-user at scipy.org>
Message-ID: <4BC71AB3.63BA.009B.0 at twdb.state.tx.us>
Content-Type: text/plain; charset="us-ascii"

Hi,

I'm trying to do some line/edge detection in python. I've done some googling and found some mailing list archives that talked about ways to do edge detection but they seem to mainly return an image with the edges highlighted. What I need is x,y coordinates of the pixels that make up the lines. I've attached an image that shows a dark band on a light background. The final output I need would be line definitions in terms of a series of x,y coordinates for the upper light to dark interface and the lower dark to light interface of the band.

Any pointers on packages to use or ways to do this are highly appreciated.

Thanks,

- dharhas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: example_small.jpg
Type: image/jpeg
Size: 35045 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/scipy-user/attachments/20100415/bd67be42/attachment.jpg 

------------------------------

_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user 


End of SciPy-User Digest, Vol 80, Issue 31
******************************************
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user



From Dharhas.Pothina at twdb.state.tx.us  Fri Apr 16 08:33:26 2010
From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina)
Date: Fri, 16 Apr 2010 07:33:26 -0500
Subject: [SciPy-User] scikits.timeseries: How to define frequency	of
	15minutes
In-Reply-To: <hq9b1d$282$1@dough.gmane.org>
References: <hq9b1d$282$1@dough.gmane.org>
Message-ID: <4BC812CA.63BA.009B.0@twdb.state.tx.us>

Hi George,

For the 15 minute data you can use the minutely frequency and for the 30 second frequency you can use the second frequency, for the 2hr, 6hr use the hourly frequency. That is what I do when analyzing data from data loggers. Or if you are pulling in multiple datasets and want to treat them all the same just use the highest frequency you expect to see for all of them (in this case seconds)

You will end up with a timeseries with missing dates which is fine for most work. Some of the timeseries toolkit functions however choke on this and you have to use the fill_missing_dates function to fill them. This will insert the missing dates at the frequency you chose earlier and mask those values. It makes the timeseries a lot larger because of all the masked values but will work for any analysis you are doing.

- dharhas 

>>> Georges Schutz <georges.schutz at internet.lu> 4/16/2010 4:35 AM >>>
I have a data logger giving me a value all 15 minutes. I would like to 
use DateArray from scikits.timeseries to handle this time series.
I do not see how to define a frequency that is out of the specified 
constants. Is there a possibility?

If I use the guess_freq() on that time data it returns 8000 ('T') which 
is a minutely frequency. Using this the has_missing_dates() gave me a 
True what is obvious as it checks for a resolution of one minute.

The situation is the same for other timeseries that I have with a 
frequency of 30 seconds or 2 hours, 6 hours, ...

As I am new to the use of scikits.timeseries and scipy in general so I 
may have missed an obvious way to do this.

Thanks for your help

Georges Schutz

_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user



From georges.schutz at internet.lu  Fri Apr 16 09:30:27 2010
From: georges.schutz at internet.lu (Georges Schutz)
Date: Fri, 16 Apr 2010 15:30:27 +0200
Subject: [SciPy-User] scikits.timeseries: How to define frequency of
	15minutes
In-Reply-To: <4BC812CA.63BA.009B.0@twdb.state.tx.us>
References: <hq9b1d$282$1@dough.gmane.org>
	<4BC812CA.63BA.009B.0@twdb.state.tx.us>
Message-ID: <4BC86673.1080600@internet.lu>

Thanks Dharhas,

This is a way to process, I checked it out rapidly and it seams OK at 
least for my tests.
But if I got it right it removes the possibility to do basic sampling 
error checking based on missing_dates(). I will have to do this by other 
means. I had a look at the "steps" attribute of a DateArray which will 
give me a list that I can filter to find out if there are missing dates 
within a series.
Is a user defined frequency not possible?

Georges Schutz

On 16/04/2010 14:33, Dharhas Pothina wrote:
> Hi George,
>
> For the 15 minute data you can use the minutely frequency and for the 30 second frequency you can use the second frequency, for the 2hr, 6hr use the hourly frequency. That is what I do when analyzing data from data loggers. Or if you are pulling in multiple datasets and want to treat them all the same just use the highest frequency you expect to see for all of them (in this case seconds)
>
> You will end up with a timeseries with missing dates which is fine for most work. Some of the timeseries toolkit functions however choke on this and you have to use the fill_missing_dates function to fill them. This will insert the missing dates at the frequency you chose earlier and mask those values. It makes the timeseries a lot larger because of all the masked values but will work for any analysis you are doing.
>
> - dharhas
>
>>>> Georges Schutz<georges.schutz at internet.lu>  4/16/2010 4:35 AM>>>
> I have a data logger giving me a value all 15 minutes. I would like to
> use DateArray from scikits.timeseries to handle this time series.
> I do not see how to define a frequency that is out of the specified
> constants. Is there a possibility?
>
> If I use the guess_freq() on that time data it returns 8000 ('T') which
> is a minutely frequency. Using this the has_missing_dates() gave me a
> True what is obvious as it checks for a resolution of one minute.
>
> The situation is the same for other timeseries that I have with a
> frequency of 30 seconds or 2 hours, 6 hours, ...
>
> As I am new to the use of scikits.timeseries and scipy in general so I
> may have missed an obvious way to do this.
>
> Thanks for your help
>
> Georges Schutz
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user




From Dharhas.Pothina at twdb.state.tx.us  Fri Apr 16 10:58:52 2010
From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina)
Date: Fri, 16 Apr 2010 09:58:52 -0500
Subject: [SciPy-User] scikits.timeseries: How to define
	frequency	of15minutes
In-Reply-To: <4BC86673.1080600@internet.lu>
References: <hq9b1d$282$1@dough.gmane.org>
	<4BC812CA.63BA.009B.0@twdb.state.tx.us> <4BC86673.1080600@internet.lu>
Message-ID: <4BC834E1.63BA.009B.0@twdb.state.tx.us>


I had this question about custom frequencies when I first started using the timeseries package about a year or so ago and I think the answer was that you couldn't do custom frequencies because the underlying code is in C and a custom frequency would involve modifying that code. 

You could probably write a function that takes your custom frequency and loops through the dates in the timeseries and checks if they exist and returns false if there are any missing. I'm not sure if anything like that already exists.

- dharhas

>>> Georges Schutz <georges.schutz at internet.lu> 4/16/2010 8:30 AM >>>
Thanks Dharhas,

This is a way to process, I checked it out rapidly and it seams OK at 
least for my tests.
But if I got it right it removes the possibility to do basic sampling 
error checking based on missing_dates(). I will have to do this by other 
means. I had a look at the "steps" attribute of a DateArray which will 
give me a list that I can filter to find out if there are missing dates 
within a series.
Is a user defined frequency not possible?

Georges Schutz

On 16/04/2010 14:33, Dharhas Pothina wrote:
> Hi George,
>
> For the 15 minute data you can use the minutely frequency and for the 30 second frequency you can use the second frequency, for the 2hr, 6hr use the hourly frequency. That is what I do when analyzing data from data loggers. Or if you are pulling in multiple datasets and want to treat them all the same just use the highest frequency you expect to see for all of them (in this case seconds)
>
> You will end up with a timeseries with missing dates which is fine for most work. Some of the timeseries toolkit functions however choke on this and you have to use the fill_missing_dates function to fill them. This will insert the missing dates at the frequency you chose earlier and mask those values. It makes the timeseries a lot larger because of all the masked values but will work for any analysis you are doing.
>
> - dharhas
>
>>>> Georges Schutz<georges.schutz at internet.lu>  4/16/2010 4:35 AM>>>
> I have a data logger giving me a value all 15 minutes. I would like to
> use DateArray from scikits.timeseries to handle this time series.
> I do not see how to define a frequency that is out of the specified
> constants. Is there a possibility?
>
> If I use the guess_freq() on that time data it returns 8000 ('T') which
> is a minutely frequency. Using this the has_missing_dates() gave me a
> True what is obvious as it checks for a resolution of one minute.
>
> The situation is the same for other timeseries that I have with a
> frequency of 30 seconds or 2 hours, 6 hours, ...
>
> As I am new to the use of scikits.timeseries and scipy in general so I
> may have missed an obvious way to do this.
>
> Thanks for your help
>
> Georges Schutz
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org 
> http://mail.scipy.org/mailman/listinfo/scipy-user 


_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user



From Chris.Barker at noaa.gov  Fri Apr 16 12:15:50 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 16 Apr 2010 09:15:50 -0700
Subject: [SciPy-User] line/edge detection and output of	coordinates.
In-Reply-To: <4BC810F7.63BA.009B.0@twdb.state.tx.us>
References: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
	<hq811d$h6v$1@dough.gmane.org> <4BC78831.8070904@noaa.gov>
	<4BC810F7.63BA.009B.0@twdb.state.tx.us>
Message-ID: <4BC88D36.5040004@noaa.gov>

Dharhas Pothina wrote:
> So for the GIS/GDAL suggestion are you talking about exporting the
> lines once I've used another technique to highlight the edges? Or
> will the Raster 2 Vector functions also help with the edge detection.

In GIS, the a common problem is create contour lines from a digital 
elevation model. In your case, you could think of the darkness of the 
pixels as an "elevation", and thus contouring it would create a vector 
outline of the dark pixels.

Another problem in GIS is taking a rater where each pixel value 
represents some scalr quantity, like land cover type, and to turn that 
into vector polygons -- that might actually be closer to what you need 
than contouring.

One trick is that your data are noisy -- hopefully appropriate smoothing 
will take care of that.

Now that I think about it, there are other contouring tools in python, 
one in Matplotlib, for instance, that might work.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From Dharhas.Pothina at twdb.state.tx.us  Fri Apr 16 12:27:04 2010
From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina)
Date: Fri, 16 Apr 2010 11:27:04 -0500
Subject: [SciPy-User] line/edge detection and output	of	coordinates.
In-Reply-To: <4BC88D36.5040004@noaa.gov>
References: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
	<hq811d$h6v$1@dough.gmane.org> <4BC78831.8070904@noaa.gov>
	<4BC810F7.63BA.009B.0@twdb.state.tx.us> <4BC88D36.5040004@noaa.gov>
Message-ID: <4BC8498E.63BA.009B.0@twdb.state.tx.us>


I like this idea. I'm more familiar with GDAL than with the scipy image processing tools. I think I'll try the suggestion of using a low pass filter to remove noise followed by contouring using the raster 2 vector tools in GDAL.

- dharhas

>>> Christopher Barker <Chris.Barker at noaa.gov> 4/16/2010 11:15 AM >>>
Dharhas Pothina wrote:
> So for the GIS/GDAL suggestion are you talking about exporting the
> lines once I've used another technique to highlight the edges? Or
> will the Raster 2 Vector functions also help with the edge detection.

In GIS, the a common problem is create contour lines from a digital 
elevation model. In your case, you could think of the darkness of the 
pixels as an "elevation", and thus contouring it would create a vector 
outline of the dark pixels.

Another problem in GIS is taking a rater where each pixel value 
represents some scalr quantity, like land cover type, and to turn that 
into vector polygons -- that might actually be closer to what you need 
than contouring.

One trick is that your data are noisy -- hopefully appropriate smoothing 
will take care of that.

Now that I think about it, there are other contouring tools in python, 
one in Matplotlib, for instance, that might work.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov 
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user



From georges.schutz at internet.lu  Fri Apr 16 12:42:41 2010
From: georges.schutz at internet.lu (Georges Schutz)
Date: Fri, 16 Apr 2010 18:42:41 +0200
Subject: [SciPy-User] scikits.timeseries: How to define frequency of
	15minutes
In-Reply-To: <4BC812CA.63BA.009B.0@twdb.state.tx.us>
References: <hq9b1d$282$1@dough.gmane.org>
	<4BC812CA.63BA.009B.0@twdb.state.tx.us>
Message-ID: <4BC89381.3070806@internet.lu>

Hi again

I have an other situation where I think custom frequencies would be of 
interest. I often have the situation that data from different sources 
(different sampling - frequencies) are used in one analysis method and 
often, at leased until know, we used 2h mean values over all de datasets 
to feed the models.

The timeseries.convert() method using np.ma.mean is a very convenient 
way of producing coherent data but restricted to the frequencies that 
exist in the defined list.

I suppose there are other such methods where a custom frequency would be 
of interest.

Georges Schutz


On 16/04/2010 14:33, Dharhas Pothina wrote:
> Hi George,
>
> For the 15 minute data you can use the minutely frequency and for the 30 second frequency you can use the second frequency, for the 2hr, 6hr use the hourly frequency. That is what I do when analyzing data from data loggers. Or if you are pulling in multiple datasets and want to treat them all the same just use the highest frequency you expect to see for all of them (in this case seconds)
>
> You will end up with a timeseries with missing dates which is fine for most work. Some of the timeseries toolkit functions however choke on this and you have to use the fill_missing_dates function to fill them. This will insert the missing dates at the frequency you chose earlier and mask those values. It makes the timeseries a lot larger because of all the masked values but will work for any analysis you are doing.
>
> - dharhas
>
>>>> Georges Schutz<georges.schutz at internet.lu>  4/16/2010 4:35 AM>>>
> I have a data logger giving me a value all 15 minutes. I would like to
> use DateArray from scikits.timeseries to handle this time series.
> I do not see how to define a frequency that is out of the specified
> constants. Is there a possibility?
>
> If I use the guess_freq() on that time data it returns 8000 ('T') which
> is a minutely frequency. Using this the has_missing_dates() gave me a
> True what is obvious as it checks for a resolution of one minute.
>
> The situation is the same for other timeseries that I have with a
> frequency of 30 seconds or 2 hours, 6 hours, ...
>
> As I am new to the use of scikits.timeseries and scipy in general so I
> may have missed an obvious way to do this.
>
> Thanks for your help
>
> Georges Schutz
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user




From georges.schutz at internet.lu  Fri Apr 16 12:57:36 2010
From: georges.schutz at internet.lu (Georges Schutz)
Date: Fri, 16 Apr 2010 18:57:36 +0200
Subject: [SciPy-User] scikits.timeseries: How to define frequency
	of15minutes
In-Reply-To: <4BC834E1.63BA.009B.0@twdb.state.tx.us>
References: <hq9b1d$282$1@dough.gmane.org>	<4BC812CA.63BA.009B.0@twdb.state.tx.us>
	<4BC86673.1080600@internet.lu>
	<4BC834E1.63BA.009B.0@twdb.state.tx.us>
Message-ID: <4BC89700.6010500@internet.lu>

I know that the underlying code is in C and that this is also the reason 
of the fast runtime of this python modules what I appreciate of cause. 
My initial question was if there exist a method to realize custom 
frequencies which seems not being the case. Know the question is is 
nobody else needing such a feature or is it somewhere on a todo list or 
even someone already implemented it in a one extension.

Concerning the sampling error check I have a way of doing it currently 
but as I discover this extension, which I considder very helpful, I try 
to use the existing thinks as much as possible to avoid redoing things 
that already exists and that are mostly tested by a lot more people than 
if I do it by my own.

Thanks for your help and ideas.

Georges Schutz

On 16/04/2010 16:58, Dharhas Pothina wrote:
>
> I had this question about custom frequencies when I first started using the timeseries package about a year or so ago and I think the answer was that you couldn't do custom frequencies because the underlying code is in C and a custom frequency would involve modifying that code.
>
> You could probably write a function that takes your custom frequency and loops through the dates in the timeseries and checks if they exist and returns false if there are any missing. I'm not sure if anything like that already exists.
>
> - dharhas
>
>>>> Georges Schutz<georges.schutz at internet.lu>  4/16/2010 8:30 AM>>>
> Thanks Dharhas,
>
> This is a way to process, I checked it out rapidly and it seams OK at
> least for my tests.
> But if I got it right it removes the possibility to do basic sampling
> error checking based on missing_dates(). I will have to do this by other
> means. I had a look at the "steps" attribute of a DateArray which will
> give me a list that I can filter to find out if there are missing dates
> within a series.
> Is a user defined frequency not possible?
>
> Georges Schutz
>
> On 16/04/2010 14:33, Dharhas Pothina wrote:
>> Hi George,
>>
>> For the 15 minute data you can use the minutely frequency and for the 30 second frequency you can use the second frequency, for the 2hr, 6hr use the hourly frequency. That is what I do when analyzing data from data loggers. Or if you are pulling in multiple datasets and want to treat them all the same just use the highest frequency you expect to see for all of them (in this case seconds)
>>
>> You will end up with a timeseries with missing dates which is fine for most work. Some of the timeseries toolkit functions however choke on this and you have to use the fill_missing_dates function to fill them. This will insert the missing dates at the frequency you chose earlier and mask those values. It makes the timeseries a lot larger because of all the masked values but will work for any analysis you are doing.
>>
>> - dharhas
>>
>>>>> Georges Schutz<georges.schutz at internet.lu>   4/16/2010 4:35 AM>>>
>> I have a data logger giving me a value all 15 minutes. I would like to
>> use DateArray from scikits.timeseries to handle this time series.
>> I do not see how to define a frequency that is out of the specified
>> constants. Is there a possibility?
>>
>> If I use the guess_freq() on that time data it returns 8000 ('T') which
>> is a minutely frequency. Using this the has_missing_dates() gave me a
>> True what is obvious as it checks for a resolution of one minute.
>>
>> The situation is the same for other timeseries that I have with a
>> frequency of 30 seconds or 2 hours, 6 hours, ...
>>
>> As I am new to the use of scikits.timeseries and scipy in general so I
>> may have missed an obvious way to do this.
>>
>> Thanks for your help
>>
>> Georges Schutz
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user




From mattknox.ca at gmail.com  Fri Apr 16 14:07:14 2010
From: mattknox.ca at gmail.com (Matt Knox)
Date: Fri, 16 Apr 2010 18:07:14 +0000 (UTC)
Subject: [SciPy-User]
	=?utf-8?q?scikits=2Etimeseries=3A_How_to_define_freq?=
	=?utf-8?q?uency=09of15minutes?=
References: <hq9b1d$282$1@dough.gmane.org>	<4BC812CA.63BA.009B.0@twdb.state.tx.us>
	<4BC86673.1080600@internet.lu>
	<4BC834E1.63BA.009B.0@twdb.state.tx.us>
	<4BC89700.6010500@internet.lu>
Message-ID: <loom.20100416T200049-441@post.gmane.org>

Georges Schutz <georges.schutz <at> internet.lu> writes:

> 
> I know that the underlying code is in C and that this is also the reason 
> of the fast runtime of this python modules what I appreciate of cause. 
> My initial question was if there exist a method to realize custom 
> frequencies which seems not being the case. Know the question is is 
> nobody else needing such a feature or is it somewhere on a todo list or 
> even someone already implemented it in a one extension.

This isn't on my to-do list, or Pierre's either so far as I know. Although I
certainly agree it is a useful feature. As far as major new features, I am
kind of waiting to see what transpires with the numpy datetime data type when
it matures a bit and then maybe see how we might integrate that into the time
series package, at which point it would also be useful to revisit this feature.

Of course, if a motivated individual put together a rock solid proposal for how
to incorporate this with the existing module and a patch as well, I'd be more
than happy to include it. But it obviously does require some C programming,
which significantly reduces the number of people that could put together such
a patch.

- Matt




From pgmdevlist at gmail.com  Fri Apr 16 15:23:41 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 16 Apr 2010 15:23:41 -0400
Subject: [SciPy-User] scikits.timeseries: How to define
	frequency	of15minutes
In-Reply-To: <loom.20100416T200049-441@post.gmane.org>
References: <hq9b1d$282$1@dough.gmane.org>	<4BC812CA.63BA.009B.0@twdb.state.tx.us>
	<4BC86673.1080600@internet.lu>
	<4BC834E1.63BA.009B.0@twdb.state.tx.us>
	<4BC89700.6010500@internet.lu>
	<loom.20100416T200049-441@post.gmane.org>
Message-ID: <56D1BFD8-120D-4055-9100-70E85A15EC3A@gmail.com>

On Apr 16, 2010, at 2:07 PM, Matt Knox wrote:
> Georges Schutz <georges.schutz <at> internet.lu> writes:
> 
>> 
>> I know that the underlying code is in C and that this is also the reason 
>> of the fast runtime of this python modules what I appreciate of cause. 
>> My initial question was if there exist a method to realize custom 
>> frequencies which seems not being the case. Know the question is is 
>> nobody else needing such a feature or is it somewhere on a todo list or 
>> even someone already implemented it in a one extension.
> 
> This isn't on my to-do list, or Pierre's either so far as I know. Although I
> certainly agree it is a useful feature. As far as major new features, I am
> kind of waiting to see what transpires with the numpy datetime data type when
> it matures a bit and then maybe see how we might integrate that into the time
> series package, at which point it would also be useful to revisit this feature.


I agree with Matt: the new numpy dtypes may be a good lead, but it's still far too early to tell how easy it'll be to merge it w/ scikits.timeseries.
Keep in mind that the package was initially designed to provide some convenience while analyzing data at frequencies of a day or lower, because that was what Matt and I had to deal with. However, like Matt, I recognize that having the possibility to define specific frequencies would be great. Unfortunately, we are at this point stuck with the existing frequencies.
There are a couple of ways to handle 15-min frequencies, though. As suggested before, you could use a minutely frequency. However, you'd end up with 93% of missing data, which can be a bit wasteful. A second possibility is to use a hour frequency with a (n,4) series, with n the number of hours. In turn, this might be a problem if you want to convert the series to another frequency (nD series are not recognized by .convert), but that could get you started.
Of course, that'll depend on what you want to do.
Let us know how it goes
P.




From georges.schutz at internet.lu  Fri Apr 16 17:17:06 2010
From: georges.schutz at internet.lu (Georges Schutz)
Date: Fri, 16 Apr 2010 23:17:06 +0200
Subject: [SciPy-User] scikits.timeseries: How to define
	frequency	of15minutes
References: <hq9b1d$282$1@dough.gmane.org>
	<4BC812CA.63BA.009B.0@twdb.state.tx.us>
	<4BC86673.1080600@internet.lu>
	<4BC834E1.63BA.009B.0@twdb.state.tx.us>
	<4BC89700.6010500@internet.lu>
	<loom.20100416T200049-441@post.gmane.org>
	<56D1BFD8-120D-4055-9100-70E85A15EC3A@gmail.com>
Message-ID: <hqakdu$529$1@dough.gmane.org>

OK I got that, and I thank you for your clear words stating 
the current situation. I have not yet analyzed the 
consequences of working with missing data. But I also think 
that 93% of missing data is some how waste of memory 
especially with large datasets which is often the case using  
time-series.

I suppose I have to little background in C programming and my 
experiences in C are too long ago to be able to propose an 
useful C-based patch for that issue.

I will experience with the current code and will follow 
closely the evolution of scikits.timeseries.

Keep doing what you are doing it is really helpful.

Georges Schutz 

Pierre GM wrote:

> On Apr 16, 2010, at 2:07 PM, Matt Knox wrote:
>> Georges Schutz <georges.schutz <at> internet.lu> writes:
>> 
>>> 
>>> I know that the underlying code is in C and that this is 
also the reason
>>> of the fast runtime of this python modules what I 
appreciate of cause.
>>> My initial question was if there exist a method to realize 
custom
>>> frequencies which seems not being the case. Know the 
question is is
>>> nobody else needing such a feature or is it somewhere on a 
todo list or
>>> even someone already implemented it in a one extension.
>> 
>> This isn't on my to-do list, or Pierre's either so far as I 
know.
>> Although I certainly agree it is a useful feature. As far 
as major new
>> features, I am kind of waiting to see what transpires with 
the numpy
>> datetime data type when it matures a bit and then maybe see 
how we might
>> integrate that into the time series package, at which point 
it would also
>> be useful to revisit this feature.
> 
> 
> I agree with Matt: the new numpy dtypes may be a good lead, 
but it's still
> far too early to tell how easy it'll be to merge it w/ 
scikits.timeseries.
> Keep in mind that the package was initially designed to 
provide some
> convenience while analyzing data at frequencies of a day or 
lower, because
> that was what Matt and I had to deal with. However, like 
Matt, I recognize
> that having the possibility to define specific frequencies 
would be great.
> Unfortunately, we are at this point stuck with the existing 
frequencies.
> There are a couple of ways to handle 15-min frequencies, 
though. As
> suggested before, you could use a minutely frequency. 
However, you'd end
> up with 93% of missing data, which can be a bit wasteful. A 
second
> possibility is to use a hour frequency with a (n,4) series, 
with n the
> number of hours. In turn, this might be a problem if you 
want to convert
> the series to another frequency (nD series are not 
recognized by
> .convert), but that could get you started. Of course, 
that'll depend on
> what you want to do. Let us know how it goes P.




From lists at hilboll.de  Sat Apr 17 13:28:11 2010
From: lists at hilboll.de (Andreas)
Date: Sat, 17 Apr 2010 19:28:11 +0200
Subject: [SciPy-User] scipy.misc.imrotate() for float datatype
Message-ID: <4BC9EFAB.6070106@hilboll.de>

Hi there,

I have a 2d array of float values. Now, I would like to use an 
imrotate() function on these data, like the one provided by 
scipy.misc.imrotate.pilutil.imrotate(). However, this function 
automatically converts my data to uint8.

Is there any way to 'rotate' my 2d data in an easy way, without having 
to code everything myself? Performance is not an issue.

Thanks for your help,

Andreas.


From stefan at sun.ac.za  Sun Apr 18 11:17:09 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 18 Apr 2010 17:17:09 +0200
Subject: [SciPy-User] scipy.misc.imrotate() for float datatype
In-Reply-To: <4BC9EFAB.6070106@hilboll.de>
References: <4BC9EFAB.6070106@hilboll.de>
Message-ID: <u2o9457e7c81004180817s85c38d2bgaba8b7dfc00274d6@mail.gmail.com>

Hi Andreas

On 17 April 2010 19:28, Andreas <lists at hilboll.de> wrote:
> I have a 2d array of float values. Now, I would like to use an
> imrotate() function on these data, like the one provided by
> scipy.misc.imrotate.pilutil.imrotate(). However, this function
> automatically converts my data to uint8.
>
> Is there any way to 'rotate' my 2d data in an easy way, without having
> to code everything myself? Performance is not an issue.

We've got code to do this in scikits.image (http://scikits.appspot.com/image).

See http://stefanv.github.com/scikits.image/api/scikits.image.transform.html#homography

Regards
St?fan


From j.reid at mail.cryst.bbk.ac.uk  Mon Apr 19 09:49:17 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Mon, 19 Apr 2010 14:49:17 +0100
Subject: [SciPy-User] scipy.stats.beta does not handle small parameters
Message-ID: <hqhn0t$gfi$1@dough.gmane.org>

I've been using scipy.stats.beta and I've noticed it returns NaN when 
the parameters are small. I've tried using the same parameters in R code 
and it handles them just fine. R also handles parameters up to 17 orders 
of magnitude smaller. Is there any documentation on which parameter 
ranges are acceptable? Can I expect similar results with other 
distributions? Should I file a bug report?

Here's the code I used:
import scipy.stats as S, numpy as N
from rpy2.robjects import r

alpha, beta = 0.0710, 0.4222
for i in xrange(20):
     x_from_scipy = S.beta.rvs(alpha, beta)
     x_from_R = r.rbeta(1, alpha, beta)
     print 'Alpha=%.2e; Beta=%.2e; scipy.stats.beta.rvs=%.2e; 
R.rbeta=%.2e' % (alpha, beta, x_from_scipy, x_from_R[0])
     alpha /= 10.
     beta /= 10.

and the output from it:
Alpha=7.10e-02; Beta=4.22e-01; scipy.stats.beta.rvs=2.75e-11; 
R.rbeta=6.60e-02
Alpha=7.10e-03; Beta=4.22e-02; scipy.stats.beta.rvs=3.73e-84; 
R.rbeta=4.50e-124
Alpha=7.10e-04; Beta=4.22e-03; scipy.stats.beta.rvs=1.00e+00; 
R.rbeta=1.00e+00
Alpha=7.10e-05; Beta=4.22e-04; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-313
Alpha=7.10e-06; Beta=4.22e-05; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
Alpha=7.10e-07; Beta=4.22e-06; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-315
Alpha=7.10e-08; Beta=4.22e-07; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-316
Alpha=7.10e-09; Beta=4.22e-08; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-317
Alpha=7.10e-10; Beta=4.22e-09; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-318
Alpha=7.10e-11; Beta=4.22e-10; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-319
Alpha=7.10e-12; Beta=4.22e-11; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-320
Alpha=7.10e-13; Beta=4.22e-12; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-321
Alpha=7.10e-14; Beta=4.22e-13; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-322
Alpha=7.10e-15; Beta=4.22e-14; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
Alpha=7.10e-16; Beta=4.22e-15; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
Alpha=7.10e-17; Beta=4.22e-16; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
Alpha=7.10e-18; Beta=4.22e-17; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
Alpha=7.10e-19; Beta=4.22e-18; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
Alpha=7.10e-20; Beta=4.22e-19; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
Alpha=7.10e-21; Beta=4.22e-20; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00



From Nikolaus at rath.org  Mon Apr 19 17:10:21 2010
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Mon, 19 Apr 2010 17:10:21 -0400
Subject: [SciPy-User] matplotlib: ticks at 0 and 2pi
Message-ID: <87bpdf5ete.fsf@inspiron.ap.columbia.edu>

Hello,

I'm trying to plot something from 0 to 2pi:

    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.set_title('Radial Magnetic Field')
    ax.set_ylabel(r'Poloidal Angle $\theta$')
    ax.set_xlabel(r'Toroidal Angle $\phi$')

    ax.set_xticks([0, 2 * math.pi])
    ax.set_xticklabels(['0', r'$2\pi$'])
    ax.set_yticklabels([r'$-\pi$', r'$\pi$'])
    ax.set_yticks([-math.pi, math.pi])

    ax.set_xlim(xmin=0, xmax=2 * math.pi)
    ax.set_ylim(ymin= -math.pi, ymax=math.pi)

But unfortunately the ticks for x=2pi and y=pi are not shown. They do
show up if I move them a tiny bit:

    ax.set_xticks([0, 2 * math.pi * 0.98])
    ax.set_yticks([-math.pi, math.pi * 0.98])
    
But obviously this is ugly.

Is there a way to show the labels without moving the to the wrong
position and without extending the axis limits (since that introduces a
white border)?

Best,
    
   -Nikolaus

-- 
 ?Time flies like an arrow, fruit flies like a Banana.?

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C



From josef.pktd at gmail.com  Tue Apr 20 09:52:02 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 20 Apr 2010 09:52:02 -0400
Subject: [SciPy-User] scipy.stats.beta does not handle small parameters
In-Reply-To: <hqhn0t$gfi$1@dough.gmane.org>
References: <hqhn0t$gfi$1@dough.gmane.org>
Message-ID: <y2z1cd32cbb1004200652w48a8a69el5b352242d7d542dd@mail.gmail.com>

On Mon, Apr 19, 2010 at 9:49 AM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
> I've been using scipy.stats.beta and I've noticed it returns NaN when
> the parameters are small. I've tried using the same parameters in R code
> and it handles them just fine. R also handles parameters up to 17 orders
> of magnitude smaller. Is there any documentation on which parameter
> ranges are acceptable? Can I expect similar results with other
> distributions? Should I file a bug report?

General answer:
Many distributions have numerical problems in the boundary range of
the valid distribution parameters, and there are no systematic tests
for these. In the last two years, we improved the numerical precision
in many of these cases. However, in other cases, working around
numerical precision problems is too much effort, too difficult or
would require too many special cases, so it's not really worth it for
the "common" usage.
Filing a bug report is always good, (and even better with a patch), in
the worst case it adds to documenting the limitation of the current
implementation.

In the beta.rvs case the relevant code is in numpy.random
    def _rvs(self, a, b):
        return mtrand.beta(a,b,self._size)

which means digging through the pyrex or c code to find the source of
the nans. Most of the other code in distributions.beta is from
scipy.special (either c or fortran code).

I just tried your first example (the following is also intended as
documentation for myself)

>>> alpha, beta
(7.1e-008, 4.2199999999999999e-007)

This is very close to a two point distribution, cdf in scipy and R aggree
>>> stats.beta.cdf(1e-8, alpha, beta)
0.85598265330617773
>>> r.pbeta(1e-8, alpha, beta)[0]
0.85598265330617762
>>> stats.beta.sf(1-1e-8, alpha, beta)
0.14401510767081682
>>> 1-r.pbeta(1-1e-8, alpha, beta)[0]
0.14401510767081682
>>> stats.beta.cdf(1e-8, alpha, beta) + stats.beta.sf(1-1e-8, alpha, beta)
0.99999776097699455

quantile/ppf: R and scipy disagree on what almost zero

>>> r.qbeta(0.5, alpha, beta)[0]
1.0547236079445126e-230
>>> stats.beta.ppf(0.5, alpha, beta)
4.9406564584124654e-324
>>> r.qbeta(0.85, alpha, beta)[0]
1.7763568394002505e-015
>>> stats.beta.ppf(0.85, alpha, beta)
0.0

pdf looks ok
>>> r.dbeta(1e-8, alpha, beta)[0]
6.0774768992486097
>>> stats.beta.pdf(1e-8, alpha, beta)
6.0774768992486035
>>> r.dbeta(1-1e-12, alpha, beta)[0]
60775.483679644421
>>> stats.beta.pdf(1-1e-12, alpha, beta)
60775.483679644392

ppf cdf roundtripping looks ok in one direction

>>> stats.beta.ppf(stats.beta.cdf(0.5, alpha, beta), alpha, beta)
0.49999999974425491
>>> r.qbeta(r.pbeta(0.5, alpha, beta)[0], alpha, beta)[0]
0.50000000006892631

roundtripping doesn't work in R nor in scipy in the other direction
>>> r.qbeta(r.qbeta(0.5, alpha, beta)[0], alpha, beta)[0]
4.0301331108853522e-308
>>> stats.beta.cdf(stats.beta.ppf(0.5, alpha, beta), alpha, beta)
0.85593853078304538

#check roundtripping formula is correct
>>> stats.norm.cdf(stats.norm.ppf(0.5, alpha, beta), alpha, beta)
0.5
>>> stats.norm.cdf(stats.norm.ppf(0.25, alpha, beta), alpha, beta)
0.25

rvs in R looks ok
>>> rrvs = r.rbeta(1000, alpha, beta)
>>> rrvsp = [rrvs[i] for i in range(1000)]
>>> (np.array(rrvsp)<0.5).mean()
0.85499999999999998
>>> (np.array(rrvsp)>0.5).mean()
0.14499999999999999

but not in numpy
>>> np.random.beta(alpha, beta, size=4)
array([ NaN,  NaN,  NaN,  NaN])

So it might be worth a look into numpy.random.beta to see what causes the nan.
beta.ppf and qbeta both have problems in the neighborhood of zero (and one ?)

Josef



> Here's the code I used:
> import scipy.stats as S, numpy as N
> from rpy2.robjects import r
>
> alpha, beta = 0.0710, 0.4222
> for i in xrange(20):
> ? ? x_from_scipy = S.beta.rvs(alpha, beta)
> ? ? x_from_R = r.rbeta(1, alpha, beta)
> ? ? print 'Alpha=%.2e; Beta=%.2e; scipy.stats.beta.rvs=%.2e;
> R.rbeta=%.2e' % (alpha, beta, x_from_scipy, x_from_R[0])
> ? ? alpha /= 10.
> ? ? beta /= 10.
>
> and the output from it:
> Alpha=7.10e-02; Beta=4.22e-01; scipy.stats.beta.rvs=2.75e-11;
> R.rbeta=6.60e-02
> Alpha=7.10e-03; Beta=4.22e-02; scipy.stats.beta.rvs=3.73e-84;
> R.rbeta=4.50e-124
> Alpha=7.10e-04; Beta=4.22e-03; scipy.stats.beta.rvs=1.00e+00;
> R.rbeta=1.00e+00
> Alpha=7.10e-05; Beta=4.22e-04; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-313
> Alpha=7.10e-06; Beta=4.22e-05; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
> Alpha=7.10e-07; Beta=4.22e-06; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-315
> Alpha=7.10e-08; Beta=4.22e-07; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-316
> Alpha=7.10e-09; Beta=4.22e-08; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-317
> Alpha=7.10e-10; Beta=4.22e-09; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-318
> Alpha=7.10e-11; Beta=4.22e-10; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-319
> Alpha=7.10e-12; Beta=4.22e-11; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-320
> Alpha=7.10e-13; Beta=4.22e-12; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-321
> Alpha=7.10e-14; Beta=4.22e-13; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-322
> Alpha=7.10e-15; Beta=4.22e-14; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
> Alpha=7.10e-16; Beta=4.22e-15; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
> Alpha=7.10e-17; Beta=4.22e-16; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
> Alpha=7.10e-18; Beta=4.22e-17; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
> Alpha=7.10e-19; Beta=4.22e-18; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
> Alpha=7.10e-20; Beta=4.22e-19; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
> Alpha=7.10e-21; Beta=4.22e-20; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From robert.kern at gmail.com  Tue Apr 20 11:14:56 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 20 Apr 2010 10:14:56 -0500
Subject: [SciPy-User] matplotlib: ticks at 0 and 2pi
In-Reply-To: <87bpdf5ete.fsf@inspiron.ap.columbia.edu>
References: <87bpdf5ete.fsf@inspiron.ap.columbia.edu>
Message-ID: <p2v3d375d731004200814rbdc39cbcq79d75b11b3983cd1@mail.gmail.com>

The matplotlib mailing list is over here:

  https://lists.sourceforge.net/lists/listinfo/matplotlib-users

On Mon, Apr 19, 2010 at 16:10, Nikolaus Rath <Nikolaus at rath.org> wrote:
> Hello,
>
> I'm trying to plot something from 0 to 2pi:
>
> ? ?fig = plt.figure()
> ? ?ax = fig.add_subplot(111)
> ? ?ax.set_title('Radial Magnetic Field')
> ? ?ax.set_ylabel(r'Poloidal Angle $\theta$')
> ? ?ax.set_xlabel(r'Toroidal Angle $\phi$')
>
> ? ?ax.set_xticks([0, 2 * math.pi])
> ? ?ax.set_xticklabels(['0', r'$2\pi$'])
> ? ?ax.set_yticklabels([r'$-\pi$', r'$\pi$'])
> ? ?ax.set_yticks([-math.pi, math.pi])
>
> ? ?ax.set_xlim(xmin=0, xmax=2 * math.pi)
> ? ?ax.set_ylim(ymin= -math.pi, ymax=math.pi)
>
> But unfortunately the ticks for x=2pi and y=pi are not shown. They do
> show up if I move them a tiny bit:
>
> ? ?ax.set_xticks([0, 2 * math.pi * 0.98])
> ? ?ax.set_yticks([-math.pi, math.pi * 0.98])
>
> But obviously this is ugly.
>
> Is there a way to show the labels without moving the to the wrong
> position and without extending the axis limits (since that introduces a
> white border)?
>
> Best,
>
> ? -Nikolaus
>
> --
> ??Time flies like an arrow, fruit flies like a Banana.?
>
> ?PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 ?02CF A9AD B7F8 AE4E 425C
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Dharhas.Pothina at twdb.state.tx.us  Tue Apr 20 13:42:40 2010
From: Dharhas.Pothina at twdb.state.tx.us (Dharhas Pothina)
Date: Tue, 20 Apr 2010 12:42:40 -0500
Subject: [SciPy-User] line/edge detection and output of	coordinates.
References: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
	<934944.95035.qm@web33007.mail.mud.yahoo.com>
	<4BC858F3.63BA.009B.0@twdb.state.tx.us>
	<727959.33128.qm@web33002.mail.mud.yahoo.com>
	<4BCD8F2B.63BA.009B.0@twdb.state.tx.us>
Message-ID: <4BCDA140.63BA.009B.0@twdb.state.tx.us>

David & Others,

Using the method similarto 'B' outlined by David below, I managed to get to an array of x,y coordinates that define the edge. I've attached the image to this email. As you can see,  I still have noise that needs to be removed. Any suggestions on how to do this?

thanks,

- dharhas




>>> David Baddeley <david_baddeley at yahoo.com.au> 4/19/2010 5:51 AM >>>
Hi Dharhas,

I think the best approach would probably be trial and error - I'd start with a sigma of around 2-3 (all the other parameters can be left as their defaults ) and either decrease of increase it depending on whether you're blurring too much, or not enough. You might also want to take a look at ndimage.median_filter (again I'd experiment, this time with the size parameter), as there is a chance that this will let you smooth more without loosing edge definition.

cheers,
David


----- Original Message ----
From: Dharhas Pothina <Dharhas.Pothina at twdb.state.tx.us>
To: David Baddeley <david_baddeley at yahoo.com.au>
Sent: Sat, 17 April, 2010 5:32:45 AM
Subject: Re: [SciPy-User] line/edge detection and output of coordinates.

David,

Can you give any guidance (or point me to documentation) on how to specify parameters for scipy.ndimage.gaussian_filter to remove the noise in my image? I don't have enough background in this area for the docstring to be useful.

thanks,

- dharhas


>>> David Baddeley <david_baddeley at yahoo.com.au> 4/15/2010 3:21 PM >>>
Hi Dharhas,

most traditional edge detection algorithms use some kind of gradient based filter - your image has two features which suggest that this approach might not be ideal. First it is noisy (which means that any gradient image will be even noisier) and second, you actually have relatively good contrast as to what is part of the object and what is not, and might not need gradient based edge detection per se. I'd take the following approach:

- low pass filter with scipy.ndimage.gaussian_filter to reduce the noise
- threshold the image to get a binary mask of your band (eg mask = image > threshold)
- potentially use a morphological operation such as binary closing (scipy.ndimage.binary_closing) or hole filling to tidy up the mask

from this mask you have a number of options to get the edges:
A - do a binary dilation and then subtract the original mask - this should give you a mask with all the edge pixels (both top and bottom). Applying ndimage.label to this mask might allow you to extract masks of both edges separately.

B - as you want to find the top and bottom edges (rather than edges which can fold back on themselves), you could take the difference between the mask and a copy of the mask which has been shifted vertically by one pixel (eg mask[:-1,:] - mask[1:,:]). This should give you an image in which the top edge pixels have a value of 1, the bottom edge has a value of -1, and all other pixels are zero. 

once you have an image in which the pixels of each edge have different values, you can find the coordinates using numpy.where.

cheers,
David



----- Original Message ----
From: Dharhas Pothina <Dharhas.Pothina at twdb.state.tx.us>
To: SciPy at yahoo.com 
Sent: Fri, 16 April, 2010 6:54:58 AM
Subject: [SciPy-User] line/edge detection and output of coordinates.

Hi,

I'm trying to do some line/edge detection in python. I've done some googling and found some mailing list archives that talked about ways to do edge detection but they seem to mainly return an image with the edges highlighted. What I need is x,y coordinates of the pixels that make up the lines. I've attached an image that shows a dark band on a light background. The final output I need would be line definitions in terms of a series of x,y coordinates for the upper light to dark interface and the lower dark to light interface of the band.

Any pointers on packages to use or ways to do this are highly appreciated.

Thanks,

- dharhas


      
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user 



-------------- next part --------------
A non-text attachment was scrubbed...
Name: test1_small.jpg
Type: image/jpeg
Size: 12304 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100420/e8d4af9b/attachment.jpg>

From david_baddeley at yahoo.com.au  Tue Apr 20 17:43:06 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Tue, 20 Apr 2010 14:43:06 -0700 (PDT)
Subject: [SciPy-User] line/edge detection and output of coordinates.
In-Reply-To: <4BCD8F2B.63BA.009B.0@twdb.state.tx.us>
References: <4BC71AB3.63BA.009B.0@twdb.state.tx.us>
	<934944.95035.qm@web33007.mail.mud.yahoo.com>
	<4BC858F3.63BA.009B.0@twdb.state.tx.us>
	<727959.33128.qm@web33002.mail.mud.yahoo.com>
	<4BCD8F2B.63BA.009B.0@twdb.state.tx.us>
Message-ID: <43991.61076.qm@web33001.mail.mud.yahoo.com>

Hi Dharhas,

it's a bit hard to tell without playing with the data, but I suspect that the best point to get rid of the noise would be in the thresholded image, before you take the difference between the two shifted copies. I suspect that a ndimage.binary_fill_holes, potentially preceeded by a binary_closing might remove some of the clutter you're getting off the lower edge of the band. I'd then try running ndimage.label on the resulting image, which should give you connected regions. You could then throw away all the connected regions smaller than a certain cuttoff using logic such as:

#label connected regions
labels, numObjects = scipy.ndimage.label(thresholded_image) 

#find the number of pixels in each object (alternatively use ndimage.measure)
nPixels, bins = scipy.histogram(labels, scipy.arange(numObjects) + 1)

newMask = scipy.zeros(labels.shape)

#loop over objects
for i in range(numObjects):
    #and add back into mask if larger than the cutoff
    if nPixels[i] > sizeCutoff:
        newMask += (labels == (i+1))

hope this helps,
David


----- Original Message ----
From: Dharhas Pothina <Dharhas.Pothina at twdb.state.tx.us>
To: David Baddeley <david_baddeley at yahoo.com.au>
Sent: Wed, 21 April, 2010 4:25:30 AM
Subject: Re: [SciPy-User] line/edge detection and output of coordinates.

David & Others,

Using the method similarto 'B' outlined by David below, I managed to get to an array of x,y coordinates that define the edge. I've attached the image to this email. As you can see,  I still have noise that needs to be removed. Any suggestions on how to do this?

thanks,

- dharhas




>>> David Baddeley <david_baddeley at yahoo.com.au> 4/19/2010 5:51 AM >>>
Hi Dharhas,

I think the best approach would probably be trial and error - I'd start with a sigma of around 2-3 (all the other parameters can be left as their defaults ) and either decrease of increase it depending on whether you're blurring too much, or not enough. You might also want to take a look at ndimage.median_filter (again I'd experiment, this time with the size parameter), as there is a chance that this will let you smooth more without loosing edge definition.

cheers,
David


----- Original Message ----
From: Dharhas Pothina <Dharhas.Pothina at twdb.state.tx.us>
To: David Baddeley <david_baddeley at yahoo.com.au>
Sent: Sat, 17 April, 2010 5:32:45 AM
Subject: Re: [SciPy-User] line/edge detection and output of coordinates.

David,

Can you give any guidance (or point me to documentation) on how to specify parameters for scipy.ndimage.gaussian_filter to remove the noise in my image? I don't have enough background in this area for the docstring to be useful.

thanks,

- dharhas


>>> David Baddeley <david_baddeley at yahoo.com.au> 4/15/2010 3:21 PM >>>
Hi Dharhas,

most traditional edge detection algorithms use some kind of gradient based filter - your image has two features which suggest that this approach might not be ideal. First it is noisy (which means that any gradient image will be even noisier) and second, you actually have relatively good contrast as to what is part of the object and what is not, and might not need gradient based edge detection per se. I'd take the following approach:

- low pass filter with scipy.ndimage.gaussian_filter to reduce the noise
- threshold the image to get a binary mask of your band (eg mask = image > threshold)
- potentially use a morphological operation such as binary closing (scipy.ndimage.binary_closing) or hole filling to tidy up the mask

from this mask you have a number of options to get the edges:
A - do a binary dilation and then subtract the original mask - this should give you a mask with all the edge pixels (both top and bottom). Applying ndimage.label to this mask might allow you to extract masks of both edges separately.

B - as you want to find the top and bottom edges (rather than edges which can fold back on themselves), you could take the difference between the mask and a copy of the mask which has been shifted vertically by one pixel (eg mask[:-1,:] - mask[1:,:]). This should give you an image in which the top edge pixels have a value of 1, the bottom edge has a value of -1, and all other pixels are zero. 

once you have an image in which the pixels of each edge have different values, you can find the coordinates using numpy.where.

cheers,
David



----- Original Message ----
From: Dharhas Pothina <Dharhas.Pothina at twdb.state.tx.us>
To: SciPy at yahoo.com 
Sent: Fri, 16 April, 2010 6:54:58 AM
Subject: [SciPy-User] line/edge detection and output of coordinates.

Hi,

I'm trying to do some line/edge detection in python. I've done some googling and found some mailing list archives that talked about ways to do edge detection but they seem to mainly return an image with the edges highlighted. What I need is x,y coordinates of the pixels that make up the lines. I've attached an image that shows a dark band on a light background. The final output I need would be line definitions in terms of a series of x,y coordinates for the upper light to dark interface and the lower dark to light interface of the band.

Any pointers on packages to use or ways to do this are highly appreciated.

Thanks,

- dharhas


      
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user 


      


From gokhansever at gmail.com  Wed Apr 21 01:21:34 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Wed, 21 Apr 2010 00:21:34 -0500
Subject: [SciPy-User] Problem using curve_fit function
Message-ID: <y2z49d6b3501004202221ya110ceces4997f56523628a02@mail.gmail.com>

Hello,

I want to fit a curve to my simple data using the scipy.optimize.curve_fit
function. Here how I define my arrays and fit function:

    tfit = np.array([463.8, 0.5])
    weights = np.ones(16, dtype='float64')
    popt, pcov = curve_fit(my_func, array([16 elements x]), array([16
elements y]), p0=tfit, sigma=weights)

   def my_func(x, a, b):
       return a*x**b

I am following the documentation and example from
http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

When I run the script, this part of the code produces a RunTimeError as
indicated below:

  File "/usr/lib/python2.6/site-packages/scipy/optimize/minpack.py", line
430, in curve_fit
    raise RuntimeError(msg)
RuntimeError: Optimal parameters not found: The cosine of the angle between
func(x) and any column of the
  Jacobian is at most 0.000000 in absolute value

Do you have any ideas how to fix this issue and estimate optimum parameters
out of my data?

Thank you.

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100421/d84eafaf/attachment.html>

From gokhansever at gmail.com  Wed Apr 21 02:21:31 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Wed, 21 Apr 2010 01:21:31 -0500
Subject: [SciPy-User] Problem using curve_fit function
In-Reply-To: <y2z49d6b3501004202221ya110ceces4997f56523628a02@mail.gmail.com>
References: <y2z49d6b3501004202221ya110ceces4997f56523628a02@mail.gmail.com>
Message-ID: <k2p49d6b3501004202321g771c3390p298fe929bb53a575@mail.gmail.com>

On Wed, Apr 21, 2010 at 12:21 AM, G?khan Sever <gokhansever at gmail.com>wrote:

> Hello,
>
> I want to fit a curve to my simple data using the scipy.optimize.curve_fit
> function. Here how I define my arrays and fit function:
>
>     tfit = np.array([463.8, 0.5])
>     weights = np.ones(16, dtype='float64')
>     popt, pcov = curve_fit(my_func, array([16 elements x]), array([16
> elements y]), p0=tfit, sigma=weights)
>
>    def my_func(x, a, b):
>        return a*x**b
>
> I am following the documentation and example from
> http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html
>
> When I run the script, this part of the code produces a RunTimeError as
> indicated below:
>
>   File "/usr/lib/python2.6/site-packages/scipy/optimize/minpack.py", line
> 430, in curve_fit
>     raise RuntimeError(msg)
> RuntimeError: Optimal parameters not found: The cosine of the angle between
> func(x) and any column of the
>   Jacobian is at most 0.000000 in absolute value
>
> Do you have any ideas how to fix this issue and estimate optimum parameters
> out of my data?
>
> Thank you.
>
> --
> G?khan
>

Here comes a quick self-reply:

Converting x and y arrays using np.asarray to float64 resolved the problem.

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100421/e6c3cb1d/attachment.html>

From tom.foutz at gmail.com  Wed Apr 21 10:04:56 2010
From: tom.foutz at gmail.com (Tom Foutz)
Date: Wed, 21 Apr 2010 10:04:56 -0400
Subject: [SciPy-User] Interpolate based on three closest points
Message-ID: <y2i362a43271004210704jf5eea48czd12a4fb491019c4a@mail.gmail.com>

Hi everybody,
I have an irregular mesh of ~1e5 data points, with unreliable connection
data.  I am trying to interpolate based on these points.  My current method,
in pseudocode, is roughly the following:


> Matrix of data is a numpy array with X,Y,Z as columns



point of interest is x,y,z



Find distance from point of interest to all points by ~ numpy.sqrt((X-x)**2
> + (Y-y)**2 + (Z-z)**2)

tri=[]; area=[]

N=3

while True:

     for each triangle that can be made by N closest points:

        if triangle contains point of interest:

            area.append(area of triangle)

            tri.append(three triangle vertices)

     if tri: break

     N+=1



Do a linear interpolation based on triangle with smallest area


This method works great, but is super slow.  I do this for ~1e6 points.  I
was thinking there might be a faster way to find the closest points, perhaps
by doing some sort of binary tree structure?  Or perhaps I can build a
better mesh, and trace mesh connections to the closest points?

Any ideas would be appreciated.
-- Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100421/3b807536/attachment.html>

From peridot.faceted at gmail.com  Wed Apr 21 10:40:58 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 21 Apr 2010 10:40:58 -0400
Subject: [SciPy-User] Interpolate based on three closest points
In-Reply-To: <y2i362a43271004210704jf5eea48czd12a4fb491019c4a@mail.gmail.com>
References: <y2i362a43271004210704jf5eea48czd12a4fb491019c4a@mail.gmail.com>
Message-ID: <z2xce557a361004210740t27e8758kd600ee01bcf8dcf7@mail.gmail.com>

On 21 April 2010 10:04, Tom Foutz <tom.foutz at gmail.com> wrote:
> Hi everybody,
> I have an irregular mesh of ~1e5 data points, with unreliable connection
> data. ?I am trying to interpolate based on these points. ?My current method,
> in pseudocode, is roughly the following:
>
>>
>> Matrix of data is a numpy array with X,Y,Z as columns
>>
>>
>>
>> point of interest is x,y,z
>>
>>
>>
>> Find distance from point of interest to all points by ~
>> numpy.sqrt((X-x)**2 + (Y-y)**2 + (Z-z)**2)
>>
>> tri=[]; area=[]
>>
>> N=3
>>
>> while True:
>>
>> ?? ? for each triangle that can be made by N closest points:
>>
>> ?? ? ? ?if triangle contains point of interest:
>>
>> ?? ? ? ? ? ?area.append(area of triangle)
>>
>> ?? ? ? ? ? ?tri.append(three triangle vertices)
>>
>> ?? ? if tri: break
>>
>> ?? ? N+=1
>>
>>
>>
>> Do a linear interpolation based on triangle with smallest area
>
> This method works great, but is super slow. ?I do this for ~1e6 points. ?I
> was thinking there might be a faster way to find the closest points, perhaps
> by doing some sort of binary tree structure? ?Or perhaps I can build a
> better mesh, and trace mesh connections to the closest points?
> Any ideas would be appreciated.

For the first problem, scipy.spatial includes efficient
nearest-neighbour searching that is well-suited to your task (fixed
set of potential neighbours, many query points, short list of
neighbours for any query, low dimensionality). You may find, if it's
fast enough, you want to do smarter interpolation; a simple
distance-weighted average of the n nearest neighbours for some n might
be fine, or you might look at scipy's radial basis function
interpolation (though I don't know how well this handles large numbers
of points).

If you really want to construct a mesh, I'm not sure what to suggest,
particularly if you care about preserving the topology (connectedness
and holes) of the original object; you might look into algorithms used
in 3D scanning, where the problem of constructing a mesh from a cloud
of points is dealt with.

Anne

> -- Tom
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From edepagne at lcogt.net  Wed Apr 21 11:49:07 2010
From: edepagne at lcogt.net (=?utf-8?q?=C3=89ric_Depagne?=)
Date: Wed, 21 Apr 2010 08:49:07 -0700
Subject: [SciPy-User] numpy.array of mixed type.
Message-ID: <201004210849.08008.edepagne@lcogt.net>

Hi.

I'd like to create an array that would contain data of two different types: 
str and float64.

I've created a dtype accordingly :
dt = dtype({'names': ['Type', 'Chisquare'], 'formats': ['S8', float64]})
then, I initialise my array as follow:
temp = zeros ((1,1), dtype = dt)

that gives me the following:
array([[('', 0.0)]],
      dtype=[('Type', '|S8'), ('Chisquare', '<f8')])

which is almost good.

I'd like to know if it is possible instead of having an array with one column 
that will contain a tuple, to create an array with two columns,  the first 
column being a str and the second a float.


Thanks,

?ric.
-- 
Un clavier azerty en vaut deux
----------------------------------------------------------


From lists at hilboll.de  Wed Apr 21 14:05:40 2010
From: lists at hilboll.de (Andreas Hilboll)
Date: Wed, 21 Apr 2010 20:05:40 +0200 (CEST)
Subject: [SciPy-User] scikits.timeseries: select with date condition
Message-ID: <3acfe1061388626d345bb1ea75fe7ad7.squirrel@srv2.hilboll.net>

Hi there,

I'd like to do something like this:

   condition = (mySeries.dates.quarter == 1)
   mySeries[condition]

However, this is the error I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/home/hilboll/lib/python2.6/site-packages/scikits.timeseries-0.91.3-py2.6-linux-x86_64.egg/scikits/timeseries/tseries.py",
line 863, in __repr__
    timestr = str(_dates)
  File
"/home/hilboll/lib/python2.6/site-packages/numpy-1.4.0-py2.6-linux-x86_64.egg/numpy/core/numeric.py",
line 1404, in array_str
    return array2string(a, max_line_width, precision, suppress_small, ' ',
"", str)
  File
"/home/hilboll/lib/python2.6/site-packages/numpy-1.4.0-py2.6-linux-x86_64.egg/numpy/core/arrayprint.py",
line 300, in array2string
    separator, prefix)
  File
"/home/hilboll/lib/python2.6/site-packages/numpy-1.4.0-py2.6-linux-x86_64.egg/numpy/core/arrayprint.py",
line 222, in _array2string
    _summaryEdgeItems, summary_insert)[:-1]
  File
"/home/hilboll/lib/python2.6/site-packages/numpy-1.4.0-py2.6-linux-x86_64.egg/numpy/core/arrayprint.py",
line 343, in _formatArray
    word = format_function(a[-i]) + separator
  File
"/home/hilboll/lib/python2.6/site-packages/numpy-1.4.0-py2.6-linux-x86_64.egg/numpy/core/arrayprint.py",
line 438, in __call__
    if isnan(x):
TypeError: function not supported for these types, and can't coerce safely
to supported types

Any idea what's going wrong here?

Thanks for your insight,

Andreas.



From wing1127aishi at gmail.com  Wed Apr 21 14:32:45 2010
From: wing1127aishi at gmail.com (Leon Sit)
Date: Wed, 21 Apr 2010 13:32:45 -0500
Subject: [SciPy-User] How to specify the dtype in IO
Message-ID: <m2u3af1998d1004211132p20f08989waddb0f71fafe8036@mail.gmail.com>

Hi All

I have a csv file with columns of the following dtype

int8, int8, int8, float, float, ............

and the column size is arbitrary. Is there a way to specific the dtype
in genfromtxt() such that it knows the first three column is int and
float otherwise?

Thanks


From robert.kern at gmail.com  Wed Apr 21 15:55:27 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 21 Apr 2010 14:55:27 -0500
Subject: [SciPy-User] numpy.array of mixed type.
In-Reply-To: <201004210849.08008.edepagne@lcogt.net>
References: <201004210849.08008.edepagne@lcogt.net>
Message-ID: <x2k3d375d731004211255y943ef4cfk585aecab0ceb0845@mail.gmail.com>

On Wed, Apr 21, 2010 at 10:49, ?ric Depagne <edepagne at lcogt.net> wrote:
> Hi.
>
> I'd like to create an array that would contain data of two different types:
> str and float64.
>
> I've created a dtype accordingly :
> dt = dtype({'names': ['Type', 'Chisquare'], 'formats': ['S8', float64]})
> then, I initialise my array as follow:
> temp = zeros ((1,1), dtype = dt)
>
> that gives me the following:
> array([[('', 0.0)]],
> ? ? ?dtype=[('Type', '|S8'), ('Chisquare', '<f8')])
>
> which is almost good.
>
> I'd like to know if it is possible instead of having an array with one column
> that will contain a tuple, to create an array with two columns, ?the first
> column being a str and the second a float.

No, not really.

You can make a 1D array of records (they're not really tuples, though
we print them like tuples). That's almost as good.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Wed Apr 21 16:07:02 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 21 Apr 2010 16:07:02 -0400
Subject: [SciPy-User] scikits.timeseries: select with date condition
In-Reply-To: <3acfe1061388626d345bb1ea75fe7ad7.squirrel@srv2.hilboll.net>
References: <3acfe1061388626d345bb1ea75fe7ad7.squirrel@srv2.hilboll.net>
Message-ID: <6E4B27EC-166A-42A7-B3AB-F85D7C142246@gmail.com>

On Apr 21, 2010, at 2:05 PM, Andreas Hilboll wrote:
> Hi there,
> 
> I'd like to do something like this:
> 
>   condition = (mySeries.dates.quarter == 1)
>   mySeries[condition]
> 
> However, this is the error I get:


Looks like a problem w/ printing, not necessarily related to scikits.timeseries. 
Please tell us the version numbers of Numpy and scikits.timeseries, as well as a short self-contained example...

From warren.weckesser at enthought.com  Wed Apr 21 16:20:03 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Wed, 21 Apr 2010 15:20:03 -0500
Subject: [SciPy-User] How to specify the dtype in IO
In-Reply-To: <m2u3af1998d1004211132p20f08989waddb0f71fafe8036@mail.gmail.com>
References: <m2u3af1998d1004211132p20f08989waddb0f71fafe8036@mail.gmail.com>
Message-ID: <4BCF5DF3.9090400@enthought.com>

Leon Sit wrote:
> Hi All
>
> I have a csv file with columns of the following dtype
>
> int8, int8, int8, float, float, ............
>
> and the column size is arbitrary. Is there a way to specific the dtype
> in genfromtxt() such that it knows the first three column is int and
> float otherwise?
>
>   

There may be a simpler method, but it looks like a combination of the 
'dtype=None' and appropriate values in 'converters' would work.

For example, suppose this is "data.txt":
-----
index,category,ratio
1, 10, 1.234
2, 21, 4.553
3, 17, 5.113
4, 22, 2.220
-----

To create a structure array with dtypes int8, int32 and float:

In [24]: a = np.genfromtxt('data.txt', delimiter=',', names=True, 
dtype=None, converters={0: lambda s: np.int8(s)})

In [25]: a.dtype
Out[25]: dtype([('index', '|i1'), ('category', '<i4'), ('ratio', '<f8')])

In [26]: for row in a:
   ....:     print row
   ....:    
   ....:    
(1, 10, 1.234)
(2, 21, 4.5529999999999999)
(3, 17, 5.1130000000000004)
(4, 22, 2.2200000000000002)


For your data, you could set dtype=None and provide converters for 
columns 0, 1, and 2.

Warren


> Thanks_______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From schlesin at cshl.edu  Wed Apr 21 17:01:14 2010
From: schlesin at cshl.edu (Felix Schlesinger)
Date: Wed, 21 Apr 2010 21:01:14 +0000 (UTC)
Subject: [SciPy-User] Sparse vector
References: <j2y7317d50c1004151429j5141589dr4bab16e09bd55ae9@mail.gmail.com>
	<s2sce557a361004151518zab4b8f24oba2ee12d991a1c10@mail.gmail.com>
	<i2n1cd32cbb1004151529x9df9d1cdv1ef2c4019eea9551@mail.gmail.com>
Message-ID: <loom.20100421T225604-242@post.gmane.org>

> > I occasionally wonder whether a generalized sparse ndarray object
> > would be useful. You'd want it to specify the value for empty elements
> > so that things like boolean arrays could be done this way, and I think
> > a dictionary of keys approach would be the way to go. In any case,
> > nothing exists now.
> 
> for some applications, I keep just the nonzero values in an array and
> the index separate, which allows easy back and forth conversion to
> dense.

Storing indicies seperate is a good option sometimes but does not allow random
access. Are you aware of any fast hash-table implementation that can store
atomic values (e.g. numpy floats) instead of general python objects? That would
go a long way towards sparse vectors.

felix



From jsseabold at gmail.com  Wed Apr 21 17:15:06 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 21 Apr 2010 17:15:06 -0400
Subject: [SciPy-User] How to specify the dtype in IO
In-Reply-To: <m2u3af1998d1004211132p20f08989waddb0f71fafe8036@mail.gmail.com>
References: <m2u3af1998d1004211132p20f08989waddb0f71fafe8036@mail.gmail.com>
Message-ID: <g2qc048da1c1004211415ufc5dc44dhf4b71203bef8cae1@mail.gmail.com>

On Wed, Apr 21, 2010 at 2:32 PM, Leon Sit <wing1127aishi at gmail.com> wrote:
> Hi All
>
> I have a csv file with columns of the following dtype
>
> int8, int8, int8, float, float, ............
>
> and the column size is arbitrary. Is there a way to specific the dtype
> in genfromtxt() such that it knows the first three column is int and
> float otherwise?
>

I don't think that genfromtxt can handle arbitrary column lengths.  I
am assuming that you don't have missing values but rather differing
(delimited) column lengths.  If I had to do something like this and
had to use genfromtxt, I guess I would iterate through the file twice
and get the max column length and then insert delimiters for missing
data.

I usually know my column length n (which might be variable across
datasets/files but not rows) Then I do something like

dt = ['int']*3 + ['float']*(n-3)

You might be better off not using genfromtxt in this case(?) unless
someone has a better option than iterating through the file twice...

Skipper


From charlesr.harris at gmail.com  Wed Apr 21 20:27:14 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 21 Apr 2010 18:27:14 -0600
Subject: [SciPy-User] numpy.array of mixed type.
In-Reply-To: <201004210849.08008.edepagne@lcogt.net>
References: <201004210849.08008.edepagne@lcogt.net>
Message-ID: <o2ve06186141004211727g78ad390ald34bc2341359cb7c@mail.gmail.com>

On Wed, Apr 21, 2010 at 9:49 AM, ?ric Depagne <edepagne at lcogt.net> wrote:

> Hi.
>
> I'd like to create an array that would contain data of two different types:
> str and float64.
>
> I've created a dtype accordingly :
> dt = dtype({'names': ['Type', 'Chisquare'], 'formats': ['S8', float64]})
> then, I initialise my array as follow:
> temp = zeros ((1,1), dtype = dt)
>
> that gives me the following:
> array([[('', 0.0)]],
>      dtype=[('Type', '|S8'), ('Chisquare', '<f8')])
>
> which is almost good.
>
> I'd like to know if it is possible instead of having an array with one
> column
> that will contain a tuple, to create an array with two columns,  the first
> column being a str and the second a float.
>
>
It's not a tuple, it is just displayed that way, it is more like a packed c
structure. But no, you can't have *actual* ndarray columns of different
types. What problem do you have with using the dtype?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100421/5155342b/attachment.html>

From vanforeest at gmail.com  Thu Apr 22 04:14:00 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Thu, 22 Apr 2010 10:14:00 +0200
Subject: [SciPy-User] How to specify the dtype in IO
In-Reply-To: <m2u3af1998d1004211132p20f08989waddb0f71fafe8036@mail.gmail.com>
References: <m2u3af1998d1004211132p20f08989waddb0f71fafe8036@mail.gmail.com>
Message-ID: <r2pfa510ff81004220114g6a5e77jd8860911e001ad74@mail.gmail.com>

Hi,

Have you tried the csv module? This is part of some code that I use:

    import csv
    reader =  csv.reader(open("dummy.csv"))
    for row in reader:
        Id = str(row[0])

Thus, you can determine the format of each column specifically.

bye

Nicky



On 21 April 2010 20:32, Leon Sit <wing1127aishi at gmail.com> wrote:
> Hi All
>
> I have a csv file with columns of the following dtype
>
> int8, int8, int8, float, float, ............
>
> and the column size is arbitrary. Is there a way to specific the dtype
> in genfromtxt() such that it knows the first three column is int and
> float otherwise?
>
> Thanks
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From jgomezdans at gmail.com  Thu Apr 22 05:51:44 2010
From: jgomezdans at gmail.com (Jose Gomez-Dans)
Date: Thu, 22 Apr 2010 10:51:44 +0100
Subject: [SciPy-User] How to specify the dtype in IO
In-Reply-To: <m2u3af1998d1004211132p20f08989waddb0f71fafe8036@mail.gmail.com>
References: <m2u3af1998d1004211132p20f08989waddb0f71fafe8036@mail.gmail.com>
Message-ID: <v2z91d218431004220251r25091263mafa58436c74a0e02@mail.gmail.com>

Hi,

On 21 April 2010 19:32, Leon Sit <wing1127aishi at gmail.com> wrote:

> I have a csv file with columns of the following dtype
> int8, int8, int8, float, float, ............
> and the column size is arbitrary. Is there a way to specific the dtype
> in genfromtxt() such that it knows the first three column is int and
> float otherwise?
>

Can't you use numpy.loadtxt for this? From the help

 >>> d = StringIO("M 21 72\nF 35 58")

 >>> np.loadtxt(d, dtype={'names': ('gender', 'age', 'weight'),
 ...                      'formats': ('S1', 'i4', 'f4')})
 array([('M', 21, 72.0), ('F', 35, 58.0)],
          dtype=[('gender', '|S1'), ('age', '<i4'), ('weight', '<f4')])


Jose
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100422/43923e00/attachment.html>

From edepagne at lcogt.net  Thu Apr 22 11:16:57 2010
From: edepagne at lcogt.net (=?iso-8859-15?q?=C9ric_Depagne?=)
Date: Thu, 22 Apr 2010 08:16:57 -0700
Subject: [SciPy-User] numpy.array of mixed type.
In-Reply-To: <o2ve06186141004211727g78ad390ald34bc2341359cb7c@mail.gmail.com>
References: <201004210849.08008.edepagne@lcogt.net>
	<o2ve06186141004211727g78ad390ald34bc2341359cb7c@mail.gmail.com>
Message-ID: <201004220816.57162.edepagne@lcogt.net>

Le mercredi 21 avril 2010 17:27:14, Charles R Harris a ?crit :
> On Wed, Apr 21, 2010 at 9:49 AM, ?ric Depagne <edepagne at lcogt.net> wrote:
> > Hi.
> >
> > I'd like to create an array that would contain data of two different
> > types: str and float64.
> >
> > I've created a dtype accordingly :
> > dt = dtype({'names': ['Type', 'Chisquare'], 'formats': ['S8', float64]})
> > then, I initialise my array as follow:
> > temp = zeros ((1,1), dtype = dt)
> >
> > that gives me the following:
> > array([[('', 0.0)]],
> >      dtype=[('Type', '|S8'), ('Chisquare', '<f8')])
> >
> > which is almost good.
> >
> > I'd like to know if it is possible instead of having an array with one
> > column
> > that will contain a tuple, to create an array with two columns,  the
> > first column being a str and the second a float.
> 
> It's not a tuple, it is just displayed that way, it is more like a packed c
> structure. But no, you can't have *actual* ndarray columns of different
> types. What problem do you have with using the dtype?

I tried to sort my array along the Chisquare column and could not.
temp.sort (axis = 0) sorts along the column named Type, but I can't find a way 
to sort along Chisquare. Is it possible?

?ric
> 
> Chuck
> 

-- 
Un clavier azerty en vaut deux
----------------------------------------------------------
?ric Depagne                            edepagne at lcogt.net
Las Cumbres Observatory 
6740 Cortona Dr
Goleta CA, 93117
----------------------------------------------------------


From ralf.gommers at googlemail.com  Thu Apr 22 19:58:51 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Fri, 23 Apr 2010 07:58:51 +0800
Subject: [SciPy-User] [ANN] SciPy 0.7.2
Message-ID: <l2tdde7764a1004221658x2144d253tc870cc121cd2d124@mail.gmail.com>

Hi,

I am pleased to announce the release of SciPy 0.7.2. This maintenance
release is made to keep SciPy compatible with NumPy 1.4.1.

Binaries, sources and release notes can be found at
https://sourceforge.net/projects/scipy/files/<https://sourceforge.net/projects/numpy/files/>

Thank you to everyone who contributed to this release; developers, users who
reported bugs, documentation writers, etc...

Enjoy!
The SciPy developers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100423/23b76c60/attachment.html>

From silva at lma.cnrs-mrs.fr  Fri Apr 23 04:01:02 2010
From: silva at lma.cnrs-mrs.fr (Fabrice Silva)
Date: Fri, 23 Apr 2010 10:01:02 +0200
Subject: [SciPy-User] Operations on Bsplines
Message-ID: <1272009663.1705.6.camel@Portable-s2m.cnrs-mrs.fr>

Hello,
I am currently dealing with the parametrization of curves (1D functions)
by Bsplines, using Dierckx's Fitpack library (more precisely the
scipy.interpolate wrapper).

One thing I would like to achieve is simple operations on such
representation, for example computing the resulting Bsplines from the
addition/multiplication/inversion of Bplines.

Has anyone ever done/heard of something like this ?

Regards
-- 
Fabrice Silva <silva at lma.cnrs-mrs.fr>
LMA UPR CNRS 7051



From baker.alexander at gmail.com  Fri Apr 23 05:15:35 2010
From: baker.alexander at gmail.com (alexander baker)
Date: Fri, 23 Apr 2010 10:15:35 +0100
Subject: [SciPy-User] cholesky
Message-ID: <w2y270620221004230215g79e411b3m42d0526175efb113@mail.gmail.com>

We have computed a decomposition matrix using the cholesky method for a
correlation matrix, some of the eigenvalues are negative and we set those to
zero, the question is how can we estimate the significance of removing these
eigenvalues from the original correlation matrix?

http://docs.scipy.org/doc/scipy-0.7.x/reference/generated/scipy.linalg.cholesky.html#scipy.linalg.cholesky

Regards

Alex Baker


Mobile: 07788 872118
Blog: www.alexfb.com

--
All science is either physics or stamp collecting.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100423/e9d05cc2/attachment.html>

From mlist at re-factory.de  Fri Apr 23 09:41:48 2010
From: mlist at re-factory.de (Robert Elsner)
Date: Fri, 23 Apr 2010 15:41:48 +0200
Subject: [SciPy-User] sparse matrix speed
Message-ID: <1272030108.9342.24.camel@robert-desktop>

Hello,

I am solving PDEs using Scipy. Because the linear systems tend to be
pretty big I need sparse matrices. My Problem is that I found them to be
somewhat slow and I am wondering if I missed the point.
Consider for example a skew-symmetric matrix P with the main diagonal
zero and the upper diagonal all ones. Doing a matrix-vector
multiplication a thousand times with a 1e6 element vector takes ages
(around 25s with csr and 40s with csc format) and consumes 400Mb of
memory. Using the linear operator class this time is cut down to approx.
7 seconds with 30Mb memory, which is pretty much the same time as doing
the calculation directly on the vector. I am aware of the fact that the
creation of the matrix takes a couple of seconds (5) but this is not
sufficient to explain the time difference. Is there any way to speed up
the matrix operations or is the LinearOperator class the way to go?

Thanks
Robert

# some example code to illustrate the problem below

import numpy as np
import scipy.sparse as sp
import scipy.sparse.linalg as linalg

x = np.ones( 1e6 )

diag = np.repeat( 1, 1e6 )
P = sp.lil_diags( ( -diag, diag ), ( -1, 1 ), ( 1e6, 1e6 ) ).tocsr()


def lin_op( x ):
    tmp = x.ravel()
    x[2:] - x[:-2]
    return x

#P = linalg.LinearOperator( ( 1e6, 1e6 ), lin_op )

for i in xrange( 1000 ):

    P * x

    #x[2:] - x[:-2]



From faltet at pytables.org  Fri Apr 23 13:57:24 2010
From: faltet at pytables.org (Francesc Alted)
Date: Fri, 23 Apr 2010 19:57:24 +0200
Subject: [SciPy-User] Sparse vector
In-Reply-To: <loom.20100421T225604-242@post.gmane.org>
References: <j2y7317d50c1004151429j5141589dr4bab16e09bd55ae9@mail.gmail.com>
	<i2n1cd32cbb1004151529x9df9d1cdv1ef2c4019eea9551@mail.gmail.com>
	<loom.20100421T225604-242@post.gmane.org>
Message-ID: <201004231957.24933.faltet@pytables.org>

A Wednesday 21 April 2010 23:01:14 Felix Schlesinger escrigu?:
> Storing indicies seperate is a good option sometimes but does not allow
>  random access. Are you aware of any fast hash-table implementation that
>  can store atomic values (e.g. numpy floats) instead of general python
>  objects? That would go a long way towards sparse vectors.

What I'd suggest is not exactly a hash-table implementation, but rather using 
compression for keeping you sparse vectors (or arrays).  Then use a package 
that can deal with typical numpy math operations and slicing but with 
compressed containers instead of ndarrays, and you will have something very 
close to what you are looking for.

Here it is a simple example with PyTables, showing how you can deal with that 
efficiently:

In [1]: import numpy as np
In [2]: import tables as tb

# Create a sparse vector of length N (you can use matrices too)
In [4]: N = 100*1000*1000
In [5]: a0 = np.zeros(N, dtype='i4')
# Fill a0 with some sparse values (1% fill ratio) on it
In [6]: l = np.random.randint(0, N, N/100).astype('i4')
In [7]: a0[l] = 2*l+3

# Save it in a file using compression
In [9]: f = tb.openFile("/tmp/vectors.h5", "w")
# Use the highly efficent Blosc compressor (http://blosc.pytables.org)
In [10]: filters = tb.Filters(complib='blosc', complevel=9, shuffle=0)
In [11]: v0 = f.createCArray(f.root, 'v0', tb.Int32Atom(), shape=(N,), 
filters=filters)
In [12]: v0[:] = a0
In [13]: a0    # a0 is a typical ndarray container (data is in-memory)
Out[13]: array([0, 0, 0, ..., 0, 0, 0], dtype=int32)
In [14]: v0    # v0 is a CArray container (data is on-disk)
Out[14]:
/v0 (CArray(100000000,), blosc(9)) ''
  atom := Int32Atom(shape=(), dflt=0)
  maindim := 0
  flavor := 'numpy'
  byteorder := 'little'
  chunkshape := (32768,)

# Show the sizes of a0 (in-memory) and v0 (on-disk)
In [15]: a0.size*a0.itemsize
Out[15]: 400000000    # ~ 400 MB
In [16]: !ls -lh /tmp/vectors.h5
-rw-r--r-- 1 faltet users 8,2M 23 abr 19:15 /tmp/vectors.h5
# so, the compressed vector v0 takes ~ 8 MB on disk,
# while ndarray a0 takes ~ 400 MB

# Now, run some slicing operations on v0
# Single element
In [19]: timeit -r1 -n1 v0[1000]
1 loops, best of 1: 129 ?s per loop
# Slice
In [20]: timeit -r1 -n1 v0[1000:10000]
1 loops, best of 1: 162 ?s per loop
# Fancy indexing
In [21]: timeit -r1 -n1 v0[np.random.randint(0, N, 10000)]
1 loops, best of 1: 165 ms per loop

# Of course, ndarray indexing is way faster
In [22]: timeit -r1 -n1 a0[1000]
1 loops, best of 1: 4.05 ?s per loop
In [23]: timeit -r1 -n1 a0[1000:10000]
1 loops, best of 1: 5.01 ?s per loop
In [24]: timeit -r1 -n1 a0[np.random.randint(0, N, 10000)]
1 loops, best of 1: 1.39 ms per loop

# But pytables let's operate with arrays on-disk no matter how large are they
# Firstly, create another vector both in-memory and on-disk
In [25]: a1 = a0/2  # a1 is a new ndarray in-memory
In [26]: v1 = f.createCArray(f.root, 'v1', tb.Int32Atom(), shape=(N,), 
filters=filters)
In [27]: v1[:] = a1  # dump a1 ndarray into v1 CArray on disk

# Now, perform an arithmetic operation completely on-disk
In [29]: expr = tb.Expr('3*v1-2*v0')  # the expression to compute
# Create an array for output
In [31]: r = f.createCArray(f.root, 'result', tb.Int32Atom(), (N,), 
filters=filters)
In [32]: expr.setOutput(r)  # the output for expression will be 'r'
# Perform the computation!
In [33]: timeit -r1 -n1 expr.eval()
1 loops, best of 1: 1.65 s per loop
# numpy is still faster, but not a great deal
In [34]: timeit -r1 -n1 3*a1-2*a0
1 loops, best of 1: 1.09 s per loop

[You will need the version 2.2b3 of PyTables (or trunk) for doing this]

The advantage of the CArray objects in PyTables is that you can have as many 
of them as you want, the only limitation being your available disk space. 
Also, as the CArray containers are compressed, the filesystem OS can hold many 
hundreds of GB of sparsed data simultaneously, which results in much better 
performance during array mathematical computations (almost reaching pure NumPy 
speed).

Not exactly a sparse matrix implementation, but could make your day.

-- 
Francesc Alted


From permafacture at gmail.com  Fri Apr 23 19:53:13 2010
From: permafacture at gmail.com (Elliot Hallmark)
Date: Fri, 23 Apr 2010 18:53:13 -0500
Subject: [SciPy-User] help interpreting univariate spline
Message-ID: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com>

Hi there,

I am wanting to use a scipy interpolation routine to generate
polynomials approximating function I have sampled.  these polynomials
will be solved in c (cython), so I need to extract the coefficencts
from the interpolation method and pass them to a c function for root
finding.

using UnivariateSpline I ran this code:

import numpy as np
from scipy.interpolate import splrep, UnivariateSpline

x = np.linspace(1, 10, 100)
y = 1/x     #approximate a hyperbola
g = UnivariateSpline(x, y, w=None, k=3, s=.0005)
print g.get_knots()
print g.get_coeffs()

import matplotlib.pyplot as plt
plt.clf()
ax = plt.subplot(121)
l1, = ax.plot(x, y, "rs", ms=10)
l2, = ax.plot(x, g(x), "bo")
plt.show()




and the output was:
[  1.           2.18181818   3.27272727   5.54545455  10.        ]

[ 0.98462432  0.66575572  0.423       0.25461626  0.13929847  0.11763985
  0.09929961]

That is 5 knots and 7 coefficients for a degree=3 spline.  naively, I
was expecting 4 coefficents for each interval between knots.

how can I construct a piecewise polynomial from this output? or can I?

thx,

Elliot


From josef.pktd at gmail.com  Sat Apr 24 04:07:09 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 24 Apr 2010 04:07:09 -0400
Subject: [SciPy-User] scipy.stats.beta does not handle small parameters
In-Reply-To: <y2z1cd32cbb1004200652w48a8a69el5b352242d7d542dd@mail.gmail.com>
References: <hqhn0t$gfi$1@dough.gmane.org>
	<y2z1cd32cbb1004200652w48a8a69el5b352242d7d542dd@mail.gmail.com>
Message-ID: <y2j1cd32cbb1004240107m63a42212h27954dfb6e0f917c@mail.gmail.com>

On Tue, Apr 20, 2010 at 9:52 AM,  <josef.pktd at gmail.com> wrote:
> On Mon, Apr 19, 2010 at 9:49 AM, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
>> I've been using scipy.stats.beta and I've noticed it returns NaN when
>> the parameters are small. I've tried using the same parameters in R code
>> and it handles them just fine. R also handles parameters up to 17 orders
>> of magnitude smaller. Is there any documentation on which parameter
>> ranges are acceptable? Can I expect similar results with other
>> distributions? Should I file a bug report?
>
> General answer:
> Many distributions have numerical problems in the boundary range of
> the valid distribution parameters, and there are no systematic tests
> for these. In the last two years, we improved the numerical precision
> in many of these cases. However, in other cases, working around
> numerical precision problems is too much effort, too difficult or
> would require too many special cases, so it's not really worth it for
> the "common" usage.
> Filing a bug report is always good, (and even better with a patch), in
> the worst case it adds to documenting the limitation of the current
> implementation.
>
> In the beta.rvs case the relevant code is in numpy.random
> ? ?def _rvs(self, a, b):
> ? ? ? ?return mtrand.beta(a,b,self._size)
>
> which means digging through the pyrex or c code to find the source of
> the nans. Most of the other code in distributions.beta is from
> scipy.special (either c or fortran code).
>
> I just tried your first example (the following is also intended as
> documentation for myself)
>
>>>> alpha, beta
> (7.1e-008, 4.2199999999999999e-007)
>
> This is very close to a two point distribution, cdf in scipy and R aggree
>>>> stats.beta.cdf(1e-8, alpha, beta)
> 0.85598265330617773
>>>> r.pbeta(1e-8, alpha, beta)[0]
> 0.85598265330617762
>>>> stats.beta.sf(1-1e-8, alpha, beta)
> 0.14401510767081682
>>>> 1-r.pbeta(1-1e-8, alpha, beta)[0]
> 0.14401510767081682
>>>> stats.beta.cdf(1e-8, alpha, beta) + stats.beta.sf(1-1e-8, alpha, beta)
> 0.99999776097699455
>
> quantile/ppf: R and scipy disagree on what almost zero
>
>>>> r.qbeta(0.5, alpha, beta)[0]
> 1.0547236079445126e-230
>>>> stats.beta.ppf(0.5, alpha, beta)
> 4.9406564584124654e-324
>>>> r.qbeta(0.85, alpha, beta)[0]
> 1.7763568394002505e-015
>>>> stats.beta.ppf(0.85, alpha, beta)
> 0.0
>
> pdf looks ok
>>>> r.dbeta(1e-8, alpha, beta)[0]
> 6.0774768992486097
>>>> stats.beta.pdf(1e-8, alpha, beta)
> 6.0774768992486035
>>>> r.dbeta(1-1e-12, alpha, beta)[0]
> 60775.483679644421
>>>> stats.beta.pdf(1-1e-12, alpha, beta)
> 60775.483679644392
>
> ppf cdf roundtripping looks ok in one direction
>
>>>> stats.beta.ppf(stats.beta.cdf(0.5, alpha, beta), alpha, beta)
> 0.49999999974425491
>>>> r.qbeta(r.pbeta(0.5, alpha, beta)[0], alpha, beta)[0]
> 0.50000000006892631
>
> roundtripping doesn't work in R nor in scipy in the other direction
>>>> r.qbeta(r.qbeta(0.5, alpha, beta)[0], alpha, beta)[0]
> 4.0301331108853522e-308
>>>> stats.beta.cdf(stats.beta.ppf(0.5, alpha, beta), alpha, beta)
> 0.85593853078304538
>
> #check roundtripping formula is correct
>>>> stats.norm.cdf(stats.norm.ppf(0.5, alpha, beta), alpha, beta)
> 0.5
>>>> stats.norm.cdf(stats.norm.ppf(0.25, alpha, beta), alpha, beta)
> 0.25
>
> rvs in R looks ok
>>>> rrvs = r.rbeta(1000, alpha, beta)
>>>> rrvsp = [rrvs[i] for i in range(1000)]
>>>> (np.array(rrvsp)<0.5).mean()
> 0.85499999999999998
>>>> (np.array(rrvsp)>0.5).mean()
> 0.14499999999999999
>
> but not in numpy
>>>> np.random.beta(alpha, beta, size=4)
> array([ NaN, ?NaN, ?NaN, ?NaN])
>
> So it might be worth a look into numpy.random.beta to see what causes the nan.
> beta.ppf and qbeta both have problems in the neighborhood of zero (and one ?)
>
> Josef

dirichlet has the same problem

>>> drvs = np.random.dirichlet([7.10e-05, 4.22e-04],size=100)
>>> np.isnan(drvs[:,0]).sum()
68
>>> (drvs[:,0]<1e-4).sum()
27
>>> (drvs[:,0]>1-1e-4).sum()
5
>>> 27/(27.+5)
0.84375
>>> drvs = np.random.dirichlet([7.10e-14, 4.22e-13],size=100)
>>> np.isnan(drvs[:,0]).sum()
100

Josef

>
>
>> Here's the code I used:
>> import scipy.stats as S, numpy as N
>> from rpy2.robjects import r
>>
>> alpha, beta = 0.0710, 0.4222
>> for i in xrange(20):
>> ? ? x_from_scipy = S.beta.rvs(alpha, beta)
>> ? ? x_from_R = r.rbeta(1, alpha, beta)
>> ? ? print 'Alpha=%.2e; Beta=%.2e; scipy.stats.beta.rvs=%.2e;
>> R.rbeta=%.2e' % (alpha, beta, x_from_scipy, x_from_R[0])
>> ? ? alpha /= 10.
>> ? ? beta /= 10.
>>
>> and the output from it:
>> Alpha=7.10e-02; Beta=4.22e-01; scipy.stats.beta.rvs=2.75e-11;
>> R.rbeta=6.60e-02
>> Alpha=7.10e-03; Beta=4.22e-02; scipy.stats.beta.rvs=3.73e-84;
>> R.rbeta=4.50e-124
>> Alpha=7.10e-04; Beta=4.22e-03; scipy.stats.beta.rvs=1.00e+00;
>> R.rbeta=1.00e+00
>> Alpha=7.10e-05; Beta=4.22e-04; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-313
>> Alpha=7.10e-06; Beta=4.22e-05; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
>> Alpha=7.10e-07; Beta=4.22e-06; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-315
>> Alpha=7.10e-08; Beta=4.22e-07; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-316
>> Alpha=7.10e-09; Beta=4.22e-08; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-317
>> Alpha=7.10e-10; Beta=4.22e-09; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-318
>> Alpha=7.10e-11; Beta=4.22e-10; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-319
>> Alpha=7.10e-12; Beta=4.22e-11; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-320
>> Alpha=7.10e-13; Beta=4.22e-12; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-321
>> Alpha=7.10e-14; Beta=4.22e-13; scipy.stats.beta.rvs=nan; R.rbeta=3.95e-322
>> Alpha=7.10e-15; Beta=4.22e-14; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
>> Alpha=7.10e-16; Beta=4.22e-15; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
>> Alpha=7.10e-17; Beta=4.22e-16; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
>> Alpha=7.10e-18; Beta=4.22e-17; scipy.stats.beta.rvs=nan; R.rbeta=1.00e+00
>> Alpha=7.10e-19; Beta=4.22e-18; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
>> Alpha=7.10e-20; Beta=4.22e-19; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
>> Alpha=7.10e-21; Beta=4.22e-20; scipy.stats.beta.rvs=nan; R.rbeta=0.00e+00
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From gael.varoquaux at normalesup.org  Sun Apr 25 05:47:11 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 25 Apr 2010 11:47:11 +0200
Subject: [SciPy-User] cholesky
In-Reply-To: <w2y270620221004230215g79e411b3m42d0526175efb113@mail.gmail.com>
References: <w2y270620221004230215g79e411b3m42d0526175efb113@mail.gmail.com>
Message-ID: <20100425094711.GC27572@phare.normalesup.org>

On Fri, Apr 23, 2010 at 10:15:35AM +0100, alexander baker wrote:
>    We have computed a decomposition matrix using the cholesky method for a
>    correlation matrix, some of the eigenvalues are negative and we set those
>    to zero, the question is how can we estimate the significance of removing
>    these eigenvalues from the original correlation matrix?

I am not sure what you mean by 'the signification'. A correlation matrix
with negative eigenvalues is an undefined correlation matrix: it
corresponds to an impossible signal (you are most probably ending up in
such a situation because you did not have enough independent data samples
to estimate the correlation matrix).

The best thing you can do, IMHO, is to regularise the correlation matrix.
In my experience, the Ledoit-Wolf regularisation works really well:
www.ledoit.net/honey.pdf

I have some code to apply this regularisation, but it is not terribly
performant (could probably be improved).

Out of curiosity: in what context are you using these correlation
matrices? What is your final use case?

HTH,

Ga?l


From pav+sp at iki.fi  Sun Apr 25 06:15:33 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Sun, 25 Apr 2010 10:15:33 +0000 (UTC)
Subject: [SciPy-User] sparse matrix speed
References: <1272030108.9342.24.camel@robert-desktop>
Message-ID: <hr14o5$cfl$2@dough.gmane.org>

Fri, 23 Apr 2010 15:41:48 +0200, Robert Elsner wrote:
> I am solving PDEs using Scipy. Because the linear systems tend to be
> pretty big I need sparse matrices. My Problem is that I found them to be
> somewhat slow and I am wondering if I missed the point. Consider for
> example a skew-symmetric matrix P with the main diagonal zero and the
> upper diagonal all ones. Doing a matrix-vector multiplication a thousand
> times with a 1e6 element vector takes ages (around 25s with csr and 40s
> with csc format) and consumes 400Mb of memory. 

How do you measure the memory usage? I do not see such growth.

> Using the linear operator
> class this time is cut down to approx. 7 seconds with 30Mb memory, which
> is pretty much the same time as doing the calculation directly on the
> vector. I am aware of the fact that the creation of the matrix takes a
> couple of seconds (5) but this is not sufficient to explain the time
> difference. Is there any way to speed up the matrix operations or is the
> LinearOperator class the way to go?

The LinearOperator class is performing the calculation directly on the 
vector, matrix-free.

[clip]
> def lin_op( x ):
>     tmp = x.ravel()
>     x[2:] - x[:-2]
>     return x
[clip]

Here your lin_op is returning the old vector unchanged, so it's not the 
same thing as multiplying by P, which can cause a speed difference. 
Moreover, when implementing the operation this way, you are omitting at 
least NNZ floating-point multiplications which also have a cost. These 
probably can make up much of the speed difference.

-- 
Pauli Virtanen



From emmanuelle.gouillart at normalesup.org  Sun Apr 25 08:34:45 2010
From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart)
Date: Sun, 25 Apr 2010 14:34:45 +0200
Subject: [SciPy-User] watershed question
Message-ID: <20100425123445.GA25175@phare.normalesup.org>

	Dear Scipy users,

	I'm wondering if any of you is familiar with
ndimage.watershed_ift and could provide a usage example. I'm asking this
because my attempts to use this function yield quite counter-intuitive
results compared to what I understand of the watershed algorithm, or what
Matlab's watershed function returns for the same problem.  

	Below is an example of what I'm trying to do; I would like to
separate connected objects along constrictions. In the example, I create
two overlapping circles that I would like to separate with the watershed
transform. To do so, I perform the watershed transform of the opposite
of the distance transform (distance to the background of the objects)
with markers located at the minima of the negative distance transform. 

	I would expect the watershed transform to separate the connected
circles along the constriction, as Matlab does (see
http://sd-2116.dedibox.fr/scipy_examples/watershed_matlab.png), but
instead, it seems that the marker with the smallest (positive) distance
completely "invades" the basin of the marker with the greatest distance
to the background. A picture of what I obtain can be found on 
http://sd-2116.dedibox.fr/scipy_examples/watershed_example.png,
and below is the code that computes the watershed

**********
import numpy as np
from scipy import ndimage

# initial image: two overlapping circles
x, y = np.indices((40, 40))
x1, y1, x2, y2 = 14, 14, 22, 26
r1, r2 = 8, 10
mask_circle1 = (x - x1)**2 + (y - y1)**2 < r1**2
mask_circle2 = (x - x2)**2 + (y - y2)**2 < r2**2
image = np.logical_or(mask_circle1, mask_circle2)
# Use distance transform to perform watershed segmentation
distance = ndimage.distance_transform_bf(image)

# Markers array: two markers located at each circle's center;
# negative markers are background markers 
markers = np.zeros_like(image).astype(np.int8)
markers[distance==0] = -1
markers[x1, y1] = 1
markers[x2, y2] = 2

# ndimage.watershed_ift needs np.uint8 or np.uint16 as input dtype
distance = -distance
distance -= distance.min()
distance = (255*distance/distance.max()).astype(np.uint8)

# Compute watershed transform
ws = ndimage.watershed_ift(distance, markers.astype(np.int8))
***********

>>> (ws == 1).sum()
474
>>> (ws == 2).sum()
1


Is it a bug that the watershed transform doesn't separate regions along
the ridges of the array 'distance'? Am I doing something wrong? 

Any comments will be very welcome.

Thanks in advance,

Emmanuelle


From sebastian.walter at gmail.com  Sun Apr 25 09:26:32 2010
From: sebastian.walter at gmail.com (Sebastian Walter)
Date: Sun, 25 Apr 2010 15:26:32 +0200
Subject: [SciPy-User] cholesky
In-Reply-To: <20100425094711.GC27572@phare.normalesup.org>
References: <w2y270620221004230215g79e411b3m42d0526175efb113@mail.gmail.com>
	<20100425094711.GC27572@phare.normalesup.org>
Message-ID: <g2mec9f80fa1004250626j2389d987o8995287328fa6dcd@mail.gmail.com>

To be honest I also can't quite follow.
For what exactly do you need the Cholesky decomposition, resp. where
does your correlation matrix come from? And how do you compute the
eigenvalues of the Cholesky decomposition?

best regards,
Sebastian




On Sun, Apr 25, 2010 at 11:47 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Fri, Apr 23, 2010 at 10:15:35AM +0100, alexander baker wrote:
>> ? ?We have computed a decomposition matrix using the cholesky method for a
>> ? ?correlation matrix, some of the eigenvalues are negative and we set those
>> ? ?to zero, the question is how can we estimate the significance of removing
>> ? ?these eigenvalues from the original correlation matrix?
>
> I am not sure what you mean by 'the signification'. A correlation matrix
> with negative eigenvalues is an undefined correlation matrix: it
> corresponds to an impossible signal (you are most probably ending up in
> such a situation because you did not have enough independent data samples
> to estimate the correlation matrix).
>
> The best thing you can do, IMHO, is to regularise the correlation matrix.
> In my experience, the Ledoit-Wolf regularisation works really well:
> www.ledoit.net/honey.pdf
>
> I have some code to apply this regularisation, but it is not terribly
> performant (could probably be improved).
>
> Out of curiosity: in what context are you using these correlation
> matrices? What is your final use case?
>
> HTH,
>
> Ga?l
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From robert.kern at gmail.com  Sun Apr 25 10:52:30 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 25 Apr 2010 09:52:30 -0500
Subject: [SciPy-User] numpy.array of mixed type.
In-Reply-To: <201004220816.57162.edepagne@lcogt.net>
References: <201004210849.08008.edepagne@lcogt.net>
	<o2ve06186141004211727g78ad390ald34bc2341359cb7c@mail.gmail.com>
	<201004220816.57162.edepagne@lcogt.net>
Message-ID: <q2v3d375d731004250752i81429bb1s15c979cf61b76850@mail.gmail.com>

On Thu, Apr 22, 2010 at 10:16, ?ric Depagne <edepagne at lcogt.net> wrote:
> Le mercredi 21 avril 2010 17:27:14, Charles R Harris a ?crit :
>> On Wed, Apr 21, 2010 at 9:49 AM, ?ric Depagne <edepagne at lcogt.net> wrote:
>> > Hi.
>> >
>> > I'd like to create an array that would contain data of two different
>> > types: str and float64.
>> >
>> > I've created a dtype accordingly :
>> > dt = dtype({'names': ['Type', 'Chisquare'], 'formats': ['S8', float64]})
>> > then, I initialise my array as follow:
>> > temp = zeros ((1,1), dtype = dt)
>> >
>> > that gives me the following:
>> > array([[('', 0.0)]],
>> > ? ? ?dtype=[('Type', '|S8'), ('Chisquare', '<f8')])
>> >
>> > which is almost good.
>> >
>> > I'd like to know if it is possible instead of having an array with one
>> > column
>> > that will contain a tuple, to create an array with two columns, ?the
>> > first column being a str and the second a float.
>>
>> It's not a tuple, it is just displayed that way, it is more like a packed c
>> structure. But no, you can't have *actual* ndarray columns of different
>> types. What problem do you have with using the dtype?
>
> I tried to sort my array along the Chisquare column and could not.
> temp.sort (axis = 0) sorts along the column named Type, but I can't find a way
> to sort along Chisquare. Is it possible?

Use numpy.lexsort()

i = numpy.lexsort([myarray['Chisquare']])
sorted_array = myarray[i]

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Sun Apr 25 11:00:14 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 25 Apr 2010 11:00:14 -0400
Subject: [SciPy-User] numpy.array of mixed type.
In-Reply-To: <q2v3d375d731004250752i81429bb1s15c979cf61b76850@mail.gmail.com>
References: <201004210849.08008.edepagne@lcogt.net>
	<o2ve06186141004211727g78ad390ald34bc2341359cb7c@mail.gmail.com>
	<201004220816.57162.edepagne@lcogt.net>
	<q2v3d375d731004250752i81429bb1s15c979cf61b76850@mail.gmail.com>
Message-ID: <x2k1cd32cbb1004250800kbc102928p2e0b37f1bf12b38f@mail.gmail.com>

On Sun, Apr 25, 2010 at 10:52 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Apr 22, 2010 at 10:16, ?ric Depagne <edepagne at lcogt.net> wrote:
>> Le mercredi 21 avril 2010 17:27:14, Charles R Harris a ?crit :
>>> On Wed, Apr 21, 2010 at 9:49 AM, ?ric Depagne <edepagne at lcogt.net> wrote:
>>> > Hi.
>>> >
>>> > I'd like to create an array that would contain data of two different
>>> > types: str and float64.
>>> >
>>> > I've created a dtype accordingly :
>>> > dt = dtype({'names': ['Type', 'Chisquare'], 'formats': ['S8', float64]})
>>> > then, I initialise my array as follow:
>>> > temp = zeros ((1,1), dtype = dt)
>>> >
>>> > that gives me the following:
>>> > array([[('', 0.0)]],
>>> > ? ? ?dtype=[('Type', '|S8'), ('Chisquare', '<f8')])
>>> >
>>> > which is almost good.
>>> >
>>> > I'd like to know if it is possible instead of having an array with one
>>> > column
>>> > that will contain a tuple, to create an array with two columns, ?the
>>> > first column being a str and the second a float.
>>>
>>> It's not a tuple, it is just displayed that way, it is more like a packed c
>>> structure. But no, you can't have *actual* ndarray columns of different
>>> types. What problem do you have with using the dtype?
>>
>> I tried to sort my array along the Chisquare column and could not.
>> temp.sort (axis = 0) sorts along the column named Type, but I can't find a way
>> to sort along Chisquare. Is it possible?
>
> Use numpy.lexsort()
>
> i = numpy.lexsort([myarray['Chisquare']])
> sorted_array = myarray[i]

I don't know if I'm missing something but the docstring or numpy.sort
looks pretty good:

order : list, optional
When a is a structured array, this argument specifies which fields to
compare first, second, and so on. This list does not need to include
all of the fields.
<...>
Use the order keyword to specify a field to use when sorting a structured array:

>>> dtype = [('name', 'S10'), ('height', float), ('age', int)]
>>> values = [('Arthur', 1.8, 41), ('Lancelot', 1.9, 38),
...           ('Galahad', 1.7, 38)]
>>> a = np.array(values, dtype=dtype)       # create a structured array
>>> np.sort(a, order='height')
array([('Galahad', 1.7, 38), ('Arthur', 1.8, 41),
       ('Lancelot', 1.8999999999999999, 38)],
      dtype=[('name', '|S10'), ('height', '<f8'), ('age', '<i4')])


Josef


>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From charlesr.harris at gmail.com  Sun Apr 25 12:41:55 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 25 Apr 2010 10:41:55 -0600
Subject: [SciPy-User] numpy.array of mixed type.
In-Reply-To: <x2k1cd32cbb1004250800kbc102928p2e0b37f1bf12b38f@mail.gmail.com>
References: <201004210849.08008.edepagne@lcogt.net>
	<o2ve06186141004211727g78ad390ald34bc2341359cb7c@mail.gmail.com>
	<201004220816.57162.edepagne@lcogt.net>
	<q2v3d375d731004250752i81429bb1s15c979cf61b76850@mail.gmail.com>
	<x2k1cd32cbb1004250800kbc102928p2e0b37f1bf12b38f@mail.gmail.com>
Message-ID: <h2ne06186141004250941gcab8d9e4m13d5110d9f688006@mail.gmail.com>

On Sun, Apr 25, 2010 at 9:00 AM, <josef.pktd at gmail.com> wrote:

> On Sun, Apr 25, 2010 at 10:52 AM, Robert Kern <robert.kern at gmail.com>
> wrote:
> > On Thu, Apr 22, 2010 at 10:16, ?ric Depagne <edepagne at lcogt.net> wrote:
> >> Le mercredi 21 avril 2010 17:27:14, Charles R Harris a ?crit :
> >>> On Wed, Apr 21, 2010 at 9:49 AM, ?ric Depagne <edepagne at lcogt.net>
> wrote:
> >>> > Hi.
> >>> >
> >>> > I'd like to create an array that would contain data of two different
> >>> > types: str and float64.
> >>> >
> >>> > I've created a dtype accordingly :
> >>> > dt = dtype({'names': ['Type', 'Chisquare'], 'formats': ['S8',
> float64]})
> >>> > then, I initialise my array as follow:
> >>> > temp = zeros ((1,1), dtype = dt)
> >>> >
> >>> > that gives me the following:
> >>> > array([[('', 0.0)]],
> >>> >      dtype=[('Type', '|S8'), ('Chisquare', '<f8')])
> >>> >
> >>> > which is almost good.
> >>> >
> >>> > I'd like to know if it is possible instead of having an array with
> one
> >>> > column
> >>> > that will contain a tuple, to create an array with two columns,  the
> >>> > first column being a str and the second a float.
> >>>
> >>> It's not a tuple, it is just displayed that way, it is more like a
> packed c
> >>> structure. But no, you can't have *actual* ndarray columns of different
> >>> types. What problem do you have with using the dtype?
> >>
> >> I tried to sort my array along the Chisquare column and could not.
> >> temp.sort (axis = 0) sorts along the column named Type, but I can't find
> a way
> >> to sort along Chisquare. Is it possible?
> >
> > Use numpy.lexsort()
> >
> > i = numpy.lexsort([myarray['Chisquare']])
> > sorted_array = myarray[i]
>
> I don't know if I'm missing something but the docstring or numpy.sort
> looks pretty good:
>
> order : list, optional
> When a is a structured array, this argument specifies which fields to
> compare first, second, and so on. This list does not need to include
> all of the fields.
> <...>
> Use the order keyword to specify a field to use when sorting a structured
> array:
>
> >>> dtype = [('name', 'S10'), ('height', float), ('age', int)]
> >>> values = [('Arthur', 1.8, 41), ('Lancelot', 1.9, 38),
> ...           ('Galahad', 1.7, 38)]
> >>> a = np.array(values, dtype=dtype)       # create a structured array
> >>> np.sort(a, order='height')
> array([('Galahad', 1.7, 38), ('Arthur', 1.8, 41),
>       ('Lancelot', 1.8999999999999999, 38)],
>      dtype=[('name', '|S10'), ('height', '<f8'), ('age', '<i4')])
>
>
>
I'll bet lexsort is faster, though.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100425/6fbac22d/attachment.html>

From edepagne at lcogt.net  Sun Apr 25 13:25:42 2010
From: edepagne at lcogt.net (=?iso-8859-15?q?=C9ric_Depagne?=)
Date: Sun, 25 Apr 2010 10:25:42 -0700
Subject: [SciPy-User] numpy.array of mixed type.
In-Reply-To: <h2ne06186141004250941gcab8d9e4m13d5110d9f688006@mail.gmail.com>
References: <201004210849.08008.edepagne@lcogt.net>
	<x2k1cd32cbb1004250800kbc102928p2e0b37f1bf12b38f@mail.gmail.com>
	<h2ne06186141004250941gcab8d9e4m13d5110d9f688006@mail.gmail.com>
Message-ID: <201004251025.42843.edepagne@lcogt.net>

Le dimanche 25 avril 2010 09:41:55, Charles R Harris a ?crit :
> On Sun, Apr 25, 2010 at 9:00 AM, <josef.pktd at gmail.com> wrote:
> > On Sun, Apr 25, 2010 at 10:52 AM, Robert Kern <robert.kern at gmail.com>
> >
> > wrote:
> > > On Thu, Apr 22, 2010 at 10:16, ?ric Depagne <edepagne at lcogt.net> wrote:
> > >> Le mercredi 21 avril 2010 17:27:14, Charles R Harris a ?crit :
> > >>> On Wed, Apr 21, 2010 at 9:49 AM, ?ric Depagne <edepagne at lcogt.net>
> >
> > wrote:
> > >>> > Hi.
> > >>> >
> > >>> > I'd like to create an array that would contain data of two
> > >>> > different types: str and float64.
> > >>> >
> > >>> > I've created a dtype accordingly :
> > >>> > dt = dtype({'names': ['Type', 'Chisquare'], 'formats': ['S8',
> >
> > float64]})
> >
> > >>> > then, I initialise my array as follow:
> > >>> > temp = zeros ((1,1), dtype = dt)
> > >>> >
> > >>> > that gives me the following:
> > >>> > array([[('', 0.0)]],
> > >>> >      dtype=[('Type', '|S8'), ('Chisquare', '<f8')])
> > >>> >
> > >>> > which is almost good.
> > >>> >
> > >>> > I'd like to know if it is possible instead of having an array with
> >
> > one
> >
> > >>> > column
> > >>> > that will contain a tuple, to create an array with two columns, 
> > >>> > the first column being a str and the second a float.
> > >>>
> > >>> It's not a tuple, it is just displayed that way, it is more like a
> >
> > packed c
> >
> > >>> structure. But no, you can't have *actual* ndarray columns of
> > >>> different types. What problem do you have with using the dtype?
> > >>
> > >> I tried to sort my array along the Chisquare column and could not.
> > >> temp.sort (axis = 0) sorts along the column named Type, but I can't
> > >> find
> >
> > a way
> >
> > >> to sort along Chisquare. Is it possible?
> > >
> > > Use numpy.lexsort()
> > >
> > > i = numpy.lexsort([myarray['Chisquare']])
> > > sorted_array = myarray[i]
> >
> > I don't know if I'm missing something but the docstring or numpy.sort
> > looks pretty good:
> >
> > order : list, optional
> > When a is a structured array, this argument specifies which fields to
> > compare first, second, and so on. This list does not need to include
> > all of the fields.
> > <...>
> > Use the order keyword to specify a field to use when sorting a structured
> >
> > array:
> > >>> dtype = [('name', 'S10'), ('height', float), ('age', int)]
> > >>> values = [('Arthur', 1.8, 41), ('Lancelot', 1.9, 38),
> >
> > ...           ('Galahad', 1.7, 38)]
> >
> > >>> a = np.array(values, dtype=dtype)       # create a structured array
> > >>> np.sort(a, order='height')
> >
> > array([('Galahad', 1.7, 38), ('Arthur', 1.8, 41),
> >       ('Lancelot', 1.8999999999999999, 38)],
> >      dtype=[('name', '|S10'), ('height', '<f8'), ('age', '<i4')])
> 
> I'll bet lexsort is faster, though.
> 
> Chuck
> 
Thanks Robert for the lexsort, Josef for the order option of numpy.sort (how 
did I miss it ?) and Chuck for a way to choose between them!

?ric.
-- 
Un clavier azerty en vaut deux
----------------------------------------------------------
?ric Depagne                            edepagne at lcogt.net
Las Cumbres Observatory 
6740 Cortona Dr
Goleta CA, 93117
----------------------------------------------------------


From abiel.x.reinhart at jpmchase.com  Mon Apr 26 09:14:17 2010
From: abiel.x.reinhart at jpmchase.com (Abiel X Reinhart)
Date: Mon, 26 Apr 2010 09:14:17 -0400
Subject: [SciPy-User] scikits.timeseries: frequency conversion with
	non-numeric series?
Message-ID: <DF11712721890A4880B3FA1B046893720DECB68BC1@EMARC105VS01.exchad.jpmchase.net>

I would like to use scikits.timeseries to do a high-to-low frequency conversion on a time series composed of strings. A conversion function like mean or sum obviously does not make sense here, but it should be possible to use functions like end-of-period or start-of-period. Unfortunately, I end up with a ValueError. In general, is it possible to do frequency conversion with non-numeric series?

Example:

import scikits.timeseries as ts

data = ['Q1', 'Q2', 'Q3', 'Q4', 'Q5', 'Q6', 'Q7', 'Q8']
t = ts.time_series(data, freq='Q', start_date=ts.Date('Q', year=2010, quarter=1))
t.convert('A', ts.last_unmasked_val)

Traceback (most recent call last):
  File "cratings.py", line 39, in <module>
    t.convert('A', ts.last_unmasked_val)
  File "C:\Python26\lib\site-packages\scikits.timeseries-0.91.3-py2.6-win32.egg\
scikits\timeseries\tseries.py", line 2002, in convert
    obj = _convert1d(series, freq, func, position, *args, **kwargs)
  File "C:\Python26\lib\site-packages\scikits.timeseries-0.91.3-py2.6-win32.egg\
scikits\timeseries\tseries.py", line 1912, in _convert1d
    int(start_date), mask_)
ValueError: data type must provide an itemsize

This communication is for informational purposes only. It is not
intended as an offer or solicitation for the purchase or sale of
any financial instrument or as an official confirmation of any
transaction. All market prices, data and other information are not
warranted as to completeness or accuracy and are subject to change
without notice. Any comments or statements made herein do not
necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
and affiliates.

This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by JPMorgan Chase &
Co., its subsidiaries and affiliates, as applicable, for any loss
or damage arising in any way from its use. If you received this
transmission in error, please immediately contact the sender and
destroy the material in its entirety, whether in electronic or hard
copy format. Thank you.

Please refer to http://www.jpmorgan.com/pages/disclosures for
disclosures relating to European legal entities.


From etrade.griffiths at dsl.pipex.com  Mon Apr 26 12:47:44 2010
From: etrade.griffiths at dsl.pipex.com (Etrade Griffiths)
Date: Mon, 26 Apr 2010 17:47:44 +0100
Subject: [SciPy-User] Warnings with QUAD
Message-ID: <0226e0$9be904@smtp.pipex.tiscali.co.uk>

Hi

I am trying to integrate a function with an infinite upper limit but 
keep getting these warnings from scipy.integrate.quad:

Warning: The maximum number of subdivisions (50) has been achieved.
   If increasing the limit yields no improvement it is advised to analyze
   the integrand in order to determine the difficulties.  If the position of a
   local difficulty can be determined (singularity, discontinuity) one will
   probably gain from splitting up the interval and calling the integrator
   on the subranges.  Perhaps a special-purpose integrator should be used.

Not quite sure what this means.  Increasing the number of 
sub-divisions had no effect and the components of the function seem 
reasonably well behaved.  As the following example shows, I get these 
messages over a wide range of values.  Would be grateful for any 
suggestions to stop these warnings - thanks in advance.

import math
import scipy.special
import scipy.integrate

# function to integrate

def func(u, tDa):
     num = 1.0 - math.exp(-u * u * tDa)
     j0u = scipy.special.j0(u)
     y0u = scipy.special.y0(u)
     den = j0u * j0u + y0u * y0u
     return num / den / u / u / u

# main program

print("    tD          wD")

for tD in [1.0E-3,1.0E-2,1.0E-1,1.0E0,1.0E1,1.0E2,1.0E3,1.0E4,1.0E5]:
     x = scipy.integrate.quad(func, 0.0, scipy.integrate.inf, args=(tD))
     wD = 4.0 * x[0] / math.pi / math.pi
     print("%10.3e  %10.3e" % (tD, wD) )



From devicerandom at gmail.com  Mon Apr 26 13:09:01 2010
From: devicerandom at gmail.com (ms)
Date: Mon, 26 Apr 2010 18:09:01 +0100
Subject: [SciPy-User] ODR , Levenberg-Marquardt,
 non linear fitting and convergence: some assistance needed
Message-ID: <4BD5C8AD.6070805@gmail.com>

Hi,

I am currently smashing my head on the following problem.

I am trying to fit data to two equations, which are two levels of 
approximation for the same model. I am currently using ODR to fit.

Of course the 2nd order approximation is mathematically bit more 
complicated than the first, involving a long summatory etc. but the 
resulting curve and behaviour are overall very similar.

Now, in my tests, the 1st order approx. usually converges, while the 2nd 
order does not converge at all: not that it gives some wrong result, it 
remains stuck to the initial parameters with zero values in the 
covariance, etc. This even when I feed to ODR starting values very close 
to the "true" ones.

I used fmin so far to bypass this problem, but it is really slow. 
Recently a collegue of mine told me that he can get Levenberg-Marquardt 
to minimize the untreatable (in my system) 2nd order approx, using 
Mathematica, out of the box.

I have no idea unfortunately what are the differences between the 
Mathematica and the ODRPACK implementations of Levenberg-Marquardt, but 
if one can do it I think the other one can too. So, what should I try to 
improve my system? I tried increasing iterations, fixing X values etc. 
but nothing seems to work properly. Do you have any hint?

Thanks a lot!
M.


From Jim.Vickroy at noaa.gov  Mon Apr 26 14:15:52 2010
From: Jim.Vickroy at noaa.gov (Jim Vickroy)
Date: Mon, 26 Apr 2010 12:15:52 -0600
Subject: [SciPy-User] f2py : NotImplementedError: Only MS compiler supported
 with gfortran on win64
Message-ID: <4BD5D858.9030108@noaa.gov>

Hello All,

I'm attempting to create a python wrapper, for a Fortran subroutine, 
using f2py.  The attachment (/traceback.log/) is a screen capture of the 
resulting traceback.

My system details are:

 >>> sys.version
'2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)]'
 >>> sys.getwindowsversion()
(5, 1, 2600, 2, 'Service Pack 3')
 >>> scipy.__version__
'0.7.1'
 >>> numpy.__version__
'1.4.0'


I would appreciate assistance in understanding the cause of the 
/*NotImplementedError: Only MS compiler supported with gfortran on 
win64*/ exception.

Thanks,
-- jv



P.S.
The second attachment (/gfortran.log/) is a screen capture of executing 
gfortran with the file as input.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100426/3fdc004e/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: traceback.log
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100426/3fdc004e/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: gfortran.log
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100426/3fdc004e/attachment-0001.ksh>

From permafacture at gmail.com  Mon Apr 26 15:26:31 2010
From: permafacture at gmail.com (Elliot Hallmark)
Date: Mon, 26 Apr 2010 14:26:31 -0500
Subject: [SciPy-User] help interpreting univariate spline
In-Reply-To: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com>
References: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com>
Message-ID: <n2na07a0f1c1004261226m5d5416c7ib79812469d9978b6@mail.gmail.com>

On Fri, Apr 23, 2010 at 6:53 PM, Elliot Hallmark <permafacture at gmail.com> wrote:

> I am wanting to use a scipy interpolation routine to generate
> polynomials approximating function I have sampled.
> ...
> import numpy as np
>from scipy.interpolate import splrep, UnivariateSpline

>x = np.linspace(1, 10, 100)
>y = 1/x     #approximate a hyperbola
>g = UnivariateSpline(x, y, w=None, k=3, s=.0005)
>print g.get_knots()
>print g.get_coeffs()

>and the output was:
>[  1.           2.18181818   3.27272727   5.54545455  10.        ]
>
>[ 0.98462432  0.66575572  0.423       0.25461626  0.13929847  0.11763985
 0.09929961]

>That is 5 knots and 7 coefficients for a degree=3 spline.  naively, I
>was expecting 4 coefficents for each interval between knots.

I found this post:

http://mail.scipy.org/pipermail/scipy-user/2009-June/021572.html

This seems very round-a-bout and ignores the piecewise nature of the
spline.  I mean, really? is there no direct or documented way to get
the polynomial coefficents from a spline in python?  If not, then this
code is the best I have found.  For my purposes I have modified it
(see below)


"""
try_spline_coeff.py

explorations into getting polynomial coefficents from a spline.

Modified from http://mail.scipy.org/pipermail/scipy-user/2009-June/021572.html
by josef.pktd at gmail

please email offonoffoffonoff at gmail with improvements to this technique.
"""

import numpy as np
import scipy as sp
from scipy import interpolate

def cubice(x,a):
    "evaluate a cubic"
    return (a[:,0]+a[:,1]*x+a[:,2]*x**2+a[:,3]*x**3)

def deriv2coeff(x2, tck):
    '''construct a piecewise cubic polynomial from derivatives of
spline at given knots (x2)

	output is an array of knot abscissa and coefficent of poly for that section.

	out = (x_knot,a,b,c,d)'''

    if tck[2] != 3:
	print "interpolation must be cubic!"
	return np.inf
    coeffr = np.vstack(interpolate.spalde(x2,tck))
    coeffr[:,3] /= 6.
    coeffr[:,2] = (coeffr[:,2] - 6*coeffr[:,3]*x2)/2.
    coeffr[:,1] = coeffr[:,1] - 2*coeffr[:,2]*x2 - 3*coeffr[:,3]*x2**2
    coeffr[:,0] = coeffr[:,0] - coeffr[:,1]*x2 - coeffr[:,2]*x2**2 -
coeffr[:,3]*x2**3
    return np.concatenate((np.resize(x2,(x2.size,1)),coeffr), axis=1)

def calc_poly(xs, coeff):
	'''determine which piece of the piecewise polynomial the given x
value corisponds to
	and return the value of the approximation based on evaluating the
correct polynomial'''
	result = np.zeros((xs.size,4))	
	for i,x in enumerate(xs):
		a = b = c = d = 0
		for x_knot,a1,b1,c1,d1 in coeff:
			if x>x_knot:		#x_knot is in ascending order (required by splrep also)
				a=a1
				b=b1
				c=c1
				d=d1
		result[i] = a,b,c,d
		#print "single: ",x, result[i]
	return cubice(xs,result)

npoints = 51
x = np.linspace(1, 20, 2*npoints)
y = 1/x
tck = interpolate.splrep(x, y, k=3)

#calculate coefficents at original x points,
# would using tck knot values be more faithful to original spline?
coeffr = deriv2coeff(x, tck)

#print coeffr

#now, compare the polynomial and original spline over a different dataset
x2 = np.linspace(3, 18, 40)
y_spline = interpolate.splev(x2, tck)
y_polynomial = calc_poly(x2,coeffr)
print "max error: ", np.max(np.abs(y_spline-y_polynomial))

#using the actual knots of the spline
coeffk = deriv2coeff(tck[0][3:-3], tck)
y_polynomial = calc_poly(x2,coeffk)
print "max error using spline's knots: ", np.max(np.abs(y_spline-y_polynomial))


From kwgoodman at gmail.com  Tue Apr 27 17:05:04 2010
From: kwgoodman at gmail.com (Keith Goodman)
Date: Tue, 27 Apr 2010 14:05:04 -0700
Subject: [SciPy-User] [ANN] la 0.2, the labeled array
Message-ID: <j2pf4f93d421004271405la4834eceu7e8b56a7e6a70f5a@mail.gmail.com>

I am pleased to announce the second release of the la package, version 0.2.

The main class of the la package is a labeled array, larry. A larry
consists of a data array and a label list. The data array is stored as
a NumPy array and the label list as a list of lists.

larry has built-in methods such as movingsum, ranking, merge, shuffle,
zscore, demean, lag as well as typical Numpy methods like sum, max,
std, sign, clip. NaNs are treated as missing data.

Alignment by label is automatic when you add (or subtract, multiply,
divide) two larrys.

larry adds the convenience of labels, provides many built-in methods,
and let's you use many of your existing array functions.

Download: https://launchpad.net/larry/+download
docs ?http://larry.sourceforge.net
code ?https://launchpad.net/larry
list ?http://groups.google.ca/group/pystatsmodels

=============
Release Notes
=============

la 0.2 (avocado)
================

*Release date: 2010-04-27*

New larry methods
-----------------
- lix : Index into a larry using labels or index numbers or both
- swapaxes : Swap the two specified axes
- sortaxis : Sort data (and label) according to label along specified axis
- flipaxis : Reverse the order of the elements along the specified axis
- tocsv : Save larry to a csv file
- fromcsv : Load a larry from a csv file
- insertaxis : Insert a new axis at the specified position
- invert : Element by element inverting of True to False and False to True

Enhancements
------------
- All larry methods can now take nd input arrays (some previously 2d only)
- Added ability to save larrys with datetime.date labels to HDF5
- New function (panel) to convert larry of shape (n, m, k) to shape (m*k, n)
- Expanded documentation
- Over 280 new unit tests; testing easier with new assert_larry_equal function

Bug fixes
---------
- #517912: larry([]) == larry([]) raised IndexError
- #518096: larry.fromdict failed due to missing import
- #518106: la.larry.fromdict({}) failed
- #518114: fromlist([]) and fromtuples([]) failed
- #518135: keep_label crashed when there was nothing to keep
- #518210: sum, std, var returned NaN for empty larrys; now return 0.0
- #518215: unflatten crashed on an empty larry
- #518442: sum, std, var returned NaN for shapes that contain zero: (2, 0, 3)
- #568175: larry.std(axis=-1) and var crashed on negative axis input
- #569622: Negative axis input gave wrong output for several larry methods


From schlesin at cshl.edu  Wed Apr 28 00:36:48 2010
From: schlesin at cshl.edu (Felix Schlesinger)
Date: Wed, 28 Apr 2010 04:36:48 +0000 (UTC)
Subject: [SciPy-User] Sparse vector
References: <j2y7317d50c1004151429j5141589dr4bab16e09bd55ae9@mail.gmail.com>
	<i2n1cd32cbb1004151529x9df9d1cdv1ef2c4019eea9551@mail.gmail.com>
	<loom.20100421T225604-242@post.gmane.org>
	<201004231957.24933.faltet@pytables.org>
Message-ID: <loom.20100428T025205-385@post.gmane.org>

Francesc Alted <faltet <at> pytables.org> writes: 
> A Wednesday 21 April 2010 23:01:14 Felix Schlesinger escrigu?:
> > Storing indicies seperate is a good option sometimes but does not allow
> >  random access. Are you aware of any fast hash-table implementation that
> >  can store atomic values (e.g. numpy floats) instead of general python
> >  objects? That would go a long way towards sparse vectors.
> 
> What I'd suggest is not exactly a hash-table implementation, but rather using 
> compression for keeping you sparse vectors (or arrays).  Then use a package 
> that can deal with typical numpy math operations and slicing but with 
> compressed containers instead of ndarrays, and you will have something very 
> close to what you are looking for.

I use pytables a lot (thanks for the great work) and compression can help with
some sparse data tasks, but in my case (and probably many others) random access
is important. And maybe performing some operation only on those entries that are
not 0.

> In [21]: timeit -r1 -n1 v0[np.random.randint(0, N, 10000)]
> 1 loops, best of 1: 165 ms per loop
> In [24]: timeit -r1 -n1 a0[np.random.randint(0, N, 10000)]
> 1 loops, best of 1: 1.39 ms per loop>

This difference is pretty dramatic and the scipy.sparse matrices (compressed row
or dictionary of keys) are much closer to numpy performance. In some use cases
this matters, for others the very good performance on expression evaluation with
large on-disk arrays of pytables works fine.

I think there is room for a general sparse vector class in numpy (maybe based on
something in scipy.sparse). But there are many trade-offs in the design.




From stefan at sun.ac.za  Wed Apr 28 11:36:09 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 28 Apr 2010 17:36:09 +0200
Subject: [SciPy-User] watershed question
In-Reply-To: <20100425123445.GA25175@phare.normalesup.org>
References: <20100425123445.GA25175@phare.normalesup.org>
Message-ID: <r2y9457e7c81004280836x81ae43abv471e6c3e639c596e@mail.gmail.com>

Hi Emmanuelle

On 25 April 2010 14:34, Emmanuelle Gouillart
<emmanuelle.gouillart at normalesup.org> wrote:
> ? ? ? ?I would expect the watershed transform to separate the connected
> circles along the constriction, as Matlab does (see
> http://sd-2116.dedibox.fr/scipy_examples/watershed_matlab.png), but
> instead, it seems that the marker with the smallest (positive) distance
> completely "invades" the basin of the marker with the greatest distance
> to the background. A picture of what I obtain can be found on
> http://sd-2116.dedibox.fr/scipy_examples/watershed_example.png,
> and below is the code that computes the watershed
>
> **********
> ***********
>
> Is it a bug that the watershed transform doesn't separate regions along
> the ridges of the array 'distance'? Am I doing something wrong?

Unless I'm also missing something obvious, the code returns an invalid
result.  I even adjusted the depths of the two "pits", but always one
region overruns the other---not what I would expect to happen.  I
haven't delved into the ndimage code at all, but I wonder weather we
shouldn't implement one of the simpler algorithms as part of
scikits.image.segment for comparison?

Regards
St?fan


From josef.pktd at gmail.com  Wed Apr 28 11:55:27 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 28 Apr 2010 11:55:27 -0400
Subject: [SciPy-User] Warnings with QUAD
In-Reply-To: <0226e0$9be904@smtp.pipex.tiscali.co.uk>
References: <0226e0$9be904@smtp.pipex.tiscali.co.uk>
Message-ID: <j2g1cd32cbb1004280855rd01d265brdeeb783a2feb8a05@mail.gmail.com>

On Mon, Apr 26, 2010 at 12:47 PM, Etrade Griffiths
<etrade.griffiths at dsl.pipex.com> wrote:
> Hi
>
> I am trying to integrate a function with an infinite upper limit but
> keep getting these warnings from scipy.integrate.quad:
>
> Warning: The maximum number of subdivisions (50) has been achieved.
> ? If increasing the limit yields no improvement it is advised to analyze
> ? the integrand in order to determine the difficulties. ?If the position of a
> ? local difficulty can be determined (singularity, discontinuity) one will
> ? probably gain from splitting up the interval and calling the integrator
> ? on the subranges. ?Perhaps a special-purpose integrator should be used.
>
> Not quite sure what this means. ?Increasing the number of
> sub-divisions had no effect and the components of the function seem
> reasonably well behaved. ?As the following example shows, I get these
> messages over a wide range of values. ?Would be grateful for any
> suggestions to stop these warnings - thanks in advance.
>
> import math
> import scipy.special
> import scipy.integrate
>
> # function to integrate
>
> def func(u, tDa):
> ? ? num = 1.0 - math.exp(-u * u * tDa)
> ? ? j0u = scipy.special.j0(u)
> ? ? y0u = scipy.special.y0(u)
> ? ? den = j0u * j0u + y0u * y0u
> ? ? return num / den / u / u / u
>
> # main program
>
> print(" ? ?tD ? ? ? ? ?wD")
>
> for tD in [1.0E-3,1.0E-2,1.0E-1,1.0E0,1.0E1,1.0E2,1.0E3,1.0E4,1.0E5]:
> ? ? x = scipy.integrate.quad(func, 0.0, scipy.integrate.inf, args=(tD))
> ? ? wD = 4.0 * x[0] / math.pi / math.pi
> ? ? print("%10.3e ?%10.3e" % (tD, wD) )

your function  a spike/singularity close to zero, e.g.
ud = np.linspace(1e-8,0.1)
fv = np.array([func(ui, 1.0E-3) for ui in ud])

import matplotlib.pyplot as plt
plt.plot(fv)

Someone offered a version of integrate.quad that can handle
singularities on the mailing list last year.

In some cases the integrate.quad might still give correct answers in
spite of the warning and singularity.

Josef



> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Wed Apr 28 12:10:02 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 28 Apr 2010 12:10:02 -0400
Subject: [SciPy-User] ODR , Levenberg-Marquardt,
	non linear fitting and 	convergence: some assistance needed
In-Reply-To: <4BD5C8AD.6070805@gmail.com>
References: <4BD5C8AD.6070805@gmail.com>
Message-ID: <q2h1cd32cbb1004280910tcdf95e39i549d643a1ca0beb6@mail.gmail.com>

On Mon, Apr 26, 2010 at 1:09 PM, ms <devicerandom at gmail.com> wrote:
> Hi,
>
> I am currently smashing my head on the following problem.
>
> I am trying to fit data to two equations, which are two levels of
> approximation for the same model. I am currently using ODR to fit.
>
> Of course the 2nd order approximation is mathematically bit more
> complicated than the first, involving a long summatory etc. but the
> resulting curve and behaviour are overall very similar.
>
> Now, in my tests, the 1st order approx. usually converges, while the 2nd
> order does not converge at all: not that it gives some wrong result, it
> remains stuck to the initial parameters with zero values in the
> covariance, etc. This even when I feed to ODR starting values very close
> to the "true" ones.
>
> I used fmin so far to bypass this problem, but it is really slow.
> Recently a collegue of mine told me that he can get Levenberg-Marquardt
> to minimize the untreatable (in my system) 2nd order approx, using
> Mathematica, out of the box.
>
> I have no idea unfortunately what are the differences between the
> Mathematica and the ODRPACK implementations of Levenberg-Marquardt, but
> if one can do it I think the other one can too. So, what should I try to
> improve my system? I tried increasing iterations, fixing X values etc.
> but nothing seems to work properly. Do you have any hint?

scipy.optimize.leastsq , and
scipy.optimize.curve_fit (scipy trunk)
use Levenberg-Marquardt

worth a try

Josef


>
> Thanks a lot!
> M.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From amenity at enthought.com  Wed Apr 28 12:28:51 2010
From: amenity at enthought.com (Amenity Applewhite)
Date: Wed, 28 Apr 2010 11:28:51 -0500
Subject: [SciPy-User] SciPy 2010: Talks Announced
References: <e91b4574d5d1709a9dc4f7ab71ddd26b9f5.20100428160847@mcsv71.net>
Message-ID: <920397F1-6129-4054-B4FD-CF3EEA39B01E@enthought.com>

Email not displaying correctly? View it in your browser.

Hello,

As summer approaches, plans for SciPy 2010 are coming into focus! Here  
are a few quick updates:

General conference talks announced
Thanks so much to everyone who submitted an abstract. We received a  
number of fantastic proposals and are indebted to St?fan, Ond?ej, and  
our awesome Program Committee for reviewing them so quickly. We're  
really excited to see these presentations, and definitely encourage  
you to take a look at what is lined up.

Keep in mind that this schedule is tentative, and that we will be  
announcing papers for our Biomedical and Parallel Processing & Cloud  
Computing tracks next week. Stay tuned...

Less than two weeks left for early registration
Yikes, time really does fly! Remember, early registration prices only  
apply if you register before May 10th.

Best,

The SciPy 2010 Team
@SciPy2010 on Twitter

You are receiving this email because you have registered for the SciPy  
2010 conference in Austin, TX.

Unsubscribe amenity at enthought.com from this list | Forward to a friend  
| Update your profile
Our mailing address is:
Enthought, Inc.
515 Congress Ave.
Austin, TX 78701

Add us to your address book

Copyright (C) 2010 Enthought, Inc. All rights reserved.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100428/9cba50b6/attachment.html>

From zachary.pincus at yale.edu  Wed Apr 28 13:18:11 2010
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 28 Apr 2010 13:18:11 -0400
Subject: [SciPy-User] watershed question
In-Reply-To: <r2y9457e7c81004280836x81ae43abv471e6c3e639c596e@mail.gmail.com>
References: <20100425123445.GA25175@phare.normalesup.org>
	<r2y9457e7c81004280836x81ae43abv471e6c3e639c596e@mail.gmail.com>
Message-ID: <A9F18944-1712-41F3-B493-7A2B566FC0BF@yale.edu>

> Unless I'm also missing something obvious, the code returns an invalid
> result.  I even adjusted the depths of the two "pits", but always one
> region overruns the other---not what I would expect to happen.  I
> haven't delved into the ndimage code at all, but I wonder weather we
> shouldn't implement one of the simpler algorithms as part of
> scikits.image.segment for comparison?

Cellprofiler has a watershed algorithm, I believe. And like most of  
the cellprofiler stuff, the implementation seems pretty high-quality  
and well-thought-out.

I wound up extracting the cpmath sub-package, and (after a few  
setup.py changes) it works great standalone with just scipy and numpy  
as dependencies:
https://svn.broadinstitute.org/CellProfiler/trunk/CellProfiler/cellprofiler/cpmath/

Zach


From charlesr.harris at gmail.com  Wed Apr 28 13:34:38 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 28 Apr 2010 11:34:38 -0600
Subject: [SciPy-User] ODR , Levenberg-Marquardt,
	non linear fitting and 	convergence: some assistance needed
In-Reply-To: <q2h1cd32cbb1004280910tcdf95e39i549d643a1ca0beb6@mail.gmail.com>
References: <4BD5C8AD.6070805@gmail.com>
	<q2h1cd32cbb1004280910tcdf95e39i549d643a1ca0beb6@mail.gmail.com>
Message-ID: <t2we06186141004281034m694cae75hfc22737d662e209@mail.gmail.com>

On Wed, Apr 28, 2010 at 10:10 AM, <josef.pktd at gmail.com> wrote:

> On Mon, Apr 26, 2010 at 1:09 PM, ms <devicerandom at gmail.com> wrote:
> > Hi,
> >
> > I am currently smashing my head on the following problem.
> >
> > I am trying to fit data to two equations, which are two levels of
> > approximation for the same model. I am currently using ODR to fit.
> >
> > Of course the 2nd order approximation is mathematically bit more
> > complicated than the first, involving a long summatory etc. but the
> > resulting curve and behaviour are overall very similar.
> >
> > Now, in my tests, the 1st order approx. usually converges, while the 2nd
> > order does not converge at all: not that it gives some wrong result, it
> > remains stuck to the initial parameters with zero values in the
> > covariance, etc. This even when I feed to ODR starting values very close
> > to the "true" ones.
> >
> > I used fmin so far to bypass this problem, but it is really slow.
> > Recently a collegue of mine told me that he can get Levenberg-Marquardt
> > to minimize the untreatable (in my system) 2nd order approx, using
> > Mathematica, out of the box.
> >
> > I have no idea unfortunately what are the differences between the
> > Mathematica and the ODRPACK implementations of Levenberg-Marquardt, but
> > if one can do it I think the other one can too. So, what should I try to
> > improve my system? I tried increasing iterations, fixing X values etc.
> > but nothing seems to work properly. Do you have any hint?
>
> scipy.optimize.leastsq , and
>

I use leastsq a lot and like it. It comes from MINPACK, I didn't know there
was a version of Levenberg-Marquardt available in ODRPACK. Is there?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100428/cba9b603/attachment.html>

From robert.kern at gmail.com  Wed Apr 28 13:44:14 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 28 Apr 2010 12:44:14 -0500
Subject: [SciPy-User] ODR , Levenberg-Marquardt,
	non linear fitting and 	convergence: some assistance needed
In-Reply-To: <t2we06186141004281034m694cae75hfc22737d662e209@mail.gmail.com>
References: <4BD5C8AD.6070805@gmail.com>
	<q2h1cd32cbb1004280910tcdf95e39i549d643a1ca0beb6@mail.gmail.com>
	<t2we06186141004281034m694cae75hfc22737d662e209@mail.gmail.com>
Message-ID: <n2g3d375d731004281044l3ebd5697pf4b88a1f8858049b@mail.gmail.com>

On Wed, Apr 28, 2010 at 12:34, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
> On Wed, Apr 28, 2010 at 10:10 AM, <josef.pktd at gmail.com> wrote:
>>
>> On Mon, Apr 26, 2010 at 1:09 PM, ms <devicerandom at gmail.com> wrote:
>> > Hi,
>> >
>> > I am currently smashing my head on the following problem.
>> >
>> > I am trying to fit data to two equations, which are two levels of
>> > approximation for the same model. I am currently using ODR to fit.
>> >
>> > Of course the 2nd order approximation is mathematically bit more
>> > complicated than the first, involving a long summatory etc. but the
>> > resulting curve and behaviour are overall very similar.
>> >
>> > Now, in my tests, the 1st order approx. usually converges, while the 2nd
>> > order does not converge at all: not that it gives some wrong result, it
>> > remains stuck to the initial parameters with zero values in the
>> > covariance, etc. This even when I feed to ODR starting values very close
>> > to the "true" ones.
>> >
>> > I used fmin so far to bypass this problem, but it is really slow.
>> > Recently a collegue of mine told me that he can get Levenberg-Marquardt
>> > to minimize the untreatable (in my system) 2nd order approx, using
>> > Mathematica, out of the box.
>> >
>> > I have no idea unfortunately what are the differences between the
>> > Mathematica and the ODRPACK implementations of Levenberg-Marquardt, but
>> > if one can do it I think the other one can too. So, what should I try to
>> > improve my system? I tried increasing iterations, fixing X values etc.
>> > but nothing seems to work properly. Do you have any hint?
>>
>> scipy.optimize.leastsq , and
>
> I use leastsq a lot and like it. It comes from MINPACK, I didn't know there
> was a version of Levenberg-Marquardt available in ODRPACK. Is there?

No. ODRPACK uses a trust-region algorithm instead of
Levenberg-Marquardt. They are conceptually similar, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From faltet at pytables.org  Wed Apr 28 14:04:33 2010
From: faltet at pytables.org (Francesc Alted)
Date: Wed, 28 Apr 2010 20:04:33 +0200
Subject: [SciPy-User] Sparse vector
In-Reply-To: <loom.20100428T025205-385@post.gmane.org>
References: <j2y7317d50c1004151429j5141589dr4bab16e09bd55ae9@mail.gmail.com>
	<201004231957.24933.faltet@pytables.org>
	<loom.20100428T025205-385@post.gmane.org>
Message-ID: <201004282004.33341.faltet@pytables.org>

A Wednesday 28 April 2010 06:36:48 Felix Schlesinger escrigu?:
> I use pytables a lot (thanks for the great work) and compression can help
>  with some sparse data tasks, but in my case (and probably many others)
>  random access is important. And maybe performing some operation only on
>  those entries that are not 0.
> 
> > In [21]: timeit -r1 -n1 v0[np.random.randint(0, N, 10000)]
> > 1 loops, best of 1: 165 ms per loop
> > In [24]: timeit -r1 -n1 a0[np.random.randint(0, N, 10000)]
> > 1 loops, best of 1: 1.39 ms per loop>
> 
> This difference is pretty dramatic and the scipy.sparse matrices
>  (compressed row or dictionary of keys) are much closer to numpy
>  performance. In some use cases this matters, for others the very good
>  performance on expression evaluation with large on-disk arrays of pytables
>  works fine.
> 
> I think there is room for a general sparse vector class in numpy (maybe
>  based on something in scipy.sparse). But there are many trade-offs in the
>  design.

True.  I, for one, have been lately pondering about a container that would use 
compression specific for sparse matrices with emphasis in fast random access.  
I'd say that a combination of Blosc and Cython would be really good at this, 
and the implementation would not be too difficult.

The hairy think is how to provide mathematical functionality for this sparse 
container (making numexpr to understand these containers is appealing, but 
would require a non-negligible amount of time).  Mmh, food for thought.

-- 
Francesc Alted


From schlesin at cshl.edu  Wed Apr 28 17:21:06 2010
From: schlesin at cshl.edu (Felix Schlesinger)
Date: Wed, 28 Apr 2010 21:21:06 +0000 (UTC)
Subject: [SciPy-User] Sparse vector
References: <j2y7317d50c1004151429j5141589dr4bab16e09bd55ae9@mail.gmail.com>
	<201004231957.24933.faltet@pytables.org>
	<loom.20100428T025205-385@post.gmane.org>
	<201004282004.33341.faltet@pytables.org>
Message-ID: <loom.20100428T231521-788@post.gmane.org>

Francesc Alted <faltet <at> pytables.org> writes:
> A Wednesday 28 April 2010 06:36:48 Felix Schlesinger escrigu?:
> True.  I, for one, have been lately pondering about a container that would use 
> compression specific for sparse matrices with emphasis in fast random access.  
> I'd say that a combination of Blosc and Cython would be really good at this, 
> and the implementation would not be too difficult.

A cython interface would be great to allow fast operations on the sparse 
array that cannot be expressed as simple numexpr expressions. One 
could simply loop over all non-zero entries with C speed.

> The hairy think is how to provide mathematical functionality for this sparse 
> container (making numexpr to understand these containers is appealing, but 
> would require a non-negligible amount of time).  Mmh, food for thought.

It might not be necessary. A combination of methods to extract 
non-zero values or convert slices to a (compressed) dense 
representation when linear access is needed and a fast  (cython)
 random-access interface would probably cover many use-cases.

Felix






From peridot.faceted at gmail.com  Wed Apr 28 18:33:05 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 28 Apr 2010 18:33:05 -0400
Subject: [SciPy-User] scipy.integrate.ode method docs not findable in
	reference manual
Message-ID: <k2qce557a361004281533yd45aa68cte20c74790db31af6@mail.gmail.com>

Hi,

I was recently trying to look up some information on the
scipy.integrate.ode objects, and I found that the reference manual
lists the methods but does not link to their docstrings (which are
unhelpful but that's beside the point). In the doc editor I was able
to follow the links, so I suspect that the problem is in the doc
generator rather than the docstrings themselves. Has anyone else
encountered similar problems?

Thanks,
Anne


From seb.haase at gmail.com  Thu Apr 29 03:37:45 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Thu, 29 Apr 2010 09:37:45 +0200
Subject: [SciPy-User] [ANN] la 0.2, the labeled array
In-Reply-To: <j2pf4f93d421004271405la4834eceu7e8b56a7e6a70f5a@mail.gmail.com>
References: <j2pf4f93d421004271405la4834eceu7e8b56a7e6a70f5a@mail.gmail.com>
Message-ID: <p2gbc657ead1004290037ze53a0942xf038360f79f9e2f4@mail.gmail.com>

Hi Keith,

how would you say do larrys compare to numpy recarrays ? Can you
convert one to the other ? - IOW why do put the labels in a separate
list instead of the dtype ?

Regards,
Sebastian Haase


On Tue, Apr 27, 2010 at 11:05 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
> I am pleased to announce the second release of the la package, version 0.2.
>
> The main class of the la package is a labeled array, larry. A larry
> consists of a data array and a label list. The data array is stored as
> a NumPy array and the label list as a list of lists.
>
> larry has built-in methods such as movingsum, ranking, merge, shuffle,
> zscore, demean, lag as well as typical Numpy methods like sum, max,
> std, sign, clip. NaNs are treated as missing data.
>
> Alignment by label is automatic when you add (or subtract, multiply,
> divide) two larrys.
>
> larry adds the convenience of labels, provides many built-in methods,
> and let's you use many of your existing array functions.
>
> Download: https://launchpad.net/larry/+download
> docs ?http://larry.sourceforge.net
> code ?https://launchpad.net/larry
> list ?http://groups.google.ca/group/pystatsmodels
>
> =============
> Release Notes
> =============
>
> la 0.2 (avocado)
> ================
>
> *Release date: 2010-04-27*
>
> New larry methods
> -----------------
> - lix : Index into a larry using labels or index numbers or both
> - swapaxes : Swap the two specified axes
> - sortaxis : Sort data (and label) according to label along specified axis
> - flipaxis : Reverse the order of the elements along the specified axis
> - tocsv : Save larry to a csv file
> - fromcsv : Load a larry from a csv file
> - insertaxis : Insert a new axis at the specified position
> - invert : Element by element inverting of True to False and False to True
>
> Enhancements
> ------------
> - All larry methods can now take nd input arrays (some previously 2d only)
> - Added ability to save larrys with datetime.date labels to HDF5
> - New function (panel) to convert larry of shape (n, m, k) to shape (m*k, n)
> - Expanded documentation
> - Over 280 new unit tests; testing easier with new assert_larry_equal function
>
> Bug fixes
> ---------
> - #517912: larry([]) == larry([]) raised IndexError
> - #518096: larry.fromdict failed due to missing import
> - #518106: la.larry.fromdict({}) failed
> - #518114: fromlist([]) and fromtuples([]) failed
> - #518135: keep_label crashed when there was nothing to keep
> - #518210: sum, std, var returned NaN for empty larrys; now return 0.0
> - #518215: unflatten crashed on an empty larry
> - #518442: sum, std, var returned NaN for shapes that contain zero: (2, 0, 3)
> - #568175: larry.std(axis=-1) and var crashed on negative axis input
> - #569622: Negative axis input gave wrong output for several larry methods
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From devicerandom at gmail.com  Thu Apr 29 08:23:58 2010
From: devicerandom at gmail.com (ms)
Date: Thu, 29 Apr 2010 13:23:58 +0100
Subject: [SciPy-User] [solved] ODR , Levenberg-Marquardt,
 non linear fitting and 	convergence: some assistance needed
In-Reply-To: <n2g3d375d731004281044l3ebd5697pf4b88a1f8858049b@mail.gmail.com>
References: <4BD5C8AD.6070805@gmail.com>	<q2h1cd32cbb1004280910tcdf95e39i549d643a1ca0beb6@mail.gmail.com>	<t2we06186141004281034m694cae75hfc22737d662e209@mail.gmail.com>
	<n2g3d375d731004281044l3ebd5697pf4b88a1f8858049b@mail.gmail.com>
Message-ID: <4BD97A5E.30707@gmail.com>

On 28/04/10 18:44, Robert Kern wrote:
> On Wed, Apr 28, 2010 at 12:34, Charles R Harris
> <charlesr.harris at gmail.com>  wrote:
>>
>> On Wed, Apr 28, 2010 at 10:10 AM,<josef.pktd at gmail.com>  wrote:
>>>
>>> On Mon, Apr 26, 2010 at 1:09 PM, ms<devicerandom at gmail.com>  wrote:
>>>> Hi,
>>>>
>>>> I am currently smashing my head on the following problem.
>>>>
>>>> I am trying to fit data to two equations, which are two levels of
>>>> approximation for the same model. I am currently using ODR to fit.
>>>>
>>>> Of course the 2nd order approximation is mathematically bit more
>>>> complicated than the first, involving a long summatory etc. but the
>>>> resulting curve and behaviour are overall very similar.
>>>>
>>>> Now, in my tests, the 1st order approx. usually converges, while the 2nd
>>>> order does not converge at all: not that it gives some wrong result, it
>>>> remains stuck to the initial parameters with zero values in the
>>>> covariance, etc. This even when I feed to ODR starting values very close
>>>> to the "true" ones.
>>>>
>>>> I used fmin so far to bypass this problem, but it is really slow.
>>>> Recently a collegue of mine told me that he can get Levenberg-Marquardt
>>>> to minimize the untreatable (in my system) 2nd order approx, using
>>>> Mathematica, out of the box.
>>>>
>>>> I have no idea unfortunately what are the differences between the
>>>> Mathematica and the ODRPACK implementations of Levenberg-Marquardt, but
>>>> if one can do it I think the other one can too. So, what should I try to
>>>> improve my system? I tried increasing iterations, fixing X values etc.
>>>> but nothing seems to work properly. Do you have any hint?
>>>
>>> scipy.optimize.leastsq , and
>>
>> I use leastsq a lot and like it. It comes from MINPACK, I didn't know there
>> was a version of Levenberg-Marquardt available in ODRPACK. Is there?
>
> No. ODRPACK uses a trust-region algorithm instead of
> Levenberg-Marquardt. They are conceptually similar, though.

Thank you for your feedback. I used leastsq and it works well, even if 
it's definitely slower than the ODR -but at least now I'm reassured I 
was not doing anything wrong.

I thought ODR was using LM too -thanks for the information!

cheers,
m.


From archeryguru2000 at gmail.com  Thu Apr 29 10:04:25 2010
From: archeryguru2000 at gmail.com (archeryguru2000)
Date: Thu, 29 Apr 2010 07:04:25 -0700 (PDT)
Subject: [SciPy-User] [SciPy-user] lambda functions within matrix
Message-ID: <28393844.post@talk.nabble.com>


Hello all,
I'm trying to create a matrix that will allow me to specify which cells will
contain certain values.  I have a line of code that I just cannot straighten
out.  In SymPy I can use the following:

>>>sympy.Matrix(N,N,lambda i,j: i==j & j<int(M/2+1))

where N and M are integers.  This creates a zero-matrix with ones in the
upper left diagonal but the ones do not continue to the lower left.  However
in SciPy, I assumed I would've used the following:

>>>scipy.fromfunction(lambda i,j: i==j & j<int(M/2+1), (N,N))

but this gives an error that seems to pertain to my lambda function?
unsupported operand type(s) for &: 'float' and 'float'

Any suggestions?
Thanks,
~~archery~~
-- 
View this message in context: http://old.nabble.com/lambda-functions-within-matrix-tp28393844p28393844.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From algebraicamente at gmail.com  Thu Apr 29 12:17:09 2010
From: algebraicamente at gmail.com (Oscar Gerardo Lazo Arjona)
Date: Thu, 29 Apr 2010 11:17:09 -0500
Subject: [SciPy-User] Help with fast fourier transform
Message-ID: <4BD9B105.8040501@gmail.com>

Hello! I'm new to this mailing list and to numpy in general.
I need to calculate fft for my optics class. But I'm having trouble.

This is the code I'm using (commented):

import pylab
import numpy as np

def g(x):
   if x>450 and x<550:
       return 1
   else:
       return 0
      f=[g(x) for x in range(0,1000)]
f=np.array(f)

#this funtion can be ploted as
#http://i41.tinypic.com/k0shnk.png
#which is a step function "centered" at 500


F=np.fft.fft(f)
#when calculate the fft of f i get an array of complex numbers
#whose absolute value can be ploted as
#http://i40.tinypic.com/1zcecxx.png

#But that is not the desired output.
#Instead of that i expect something that can be ploted like this
#http://i39.tinypic.com/1zmch0g.png


#what i think must be happening because my function f
#has an offset of 500 (it's supposed to be centereed at 500)

#So i think it all reduces to two options:
#somehow telling fft to consider the origin at 500
#(indicate thetell the offset)

#Or make fft accept a list of points like
#[[-2,0],[-1,1],[0,1],[1,1],[2,0]]
#so that it can know the position of the step relative to the origin

Please help!

Oscar


From kwgoodman at gmail.com  Thu Apr 29 12:51:27 2010
From: kwgoodman at gmail.com (Keith Goodman)
Date: Thu, 29 Apr 2010 09:51:27 -0700
Subject: [SciPy-User] [ANN] la 0.2, the labeled array
In-Reply-To: <p2gbc657ead1004290037ze53a0942xf038360f79f9e2f4@mail.gmail.com>
References: <j2pf4f93d421004271405la4834eceu7e8b56a7e6a70f5a@mail.gmail.com>
	<p2gbc657ead1004290037ze53a0942xf038360f79f9e2f4@mail.gmail.com>
Message-ID: <m2uf4f93d421004290951r7373be14g54092059d163c3b6@mail.gmail.com>

On Thu, Apr 29, 2010 at 12:37 AM, Sebastian Haase <seb.haase at gmail.com> wrote:
> how would you say do larrys compare to numpy recarrays ? Can you
> convert one to the other ? - IOW why do put the labels in a separate
> list instead of the dtype ?

(Lately I've been getting mail from the numpy and scipy lists all at
once, once a day or so. Anyone else seeing the same?)

I'm not familiar with recarrays. But I think recarrays rely on
position to determine the dtype. So the first element of a record will
always have the same dtype (well, unless you change it).

With larrys the position of the data doesn't matter. Only the label
matters. In binary operations, for example, two larrys will be aligned
by labels:

>> from la import larry
>> x = [[1, 2], [3, 4]]
>> label = [['north', 'south'], ['west', 'east']]
>> lar1 = larry(x,  label)
>>
>> x = [[1, 2], [3, 4]]
>> label = [['south', 'north'], ['west', 'east']]  # flip north <--> south
>> lar2 = larry(x,  label)
>>
>> lar1 + lar2

label_0
    north
    south
label_1
    west
    east
x
array([[4, 6],
       [4, 6]])

Also all data in any one larry has the same dtype since the data is
stored in a numpy array.

Now for the real answer (I think):

recarrays:

>> x = np.array([(1.0, 2), (3.0, 4)], dtype=[('x', float), ('y', int)])
>> x.sum()
<...>
TypeError: cannot perform reduce with flexible type

larry:

>> x = lar1 + lar2
>> x.sum()
   20


From josef.pktd at gmail.com  Thu Apr 29 12:57:39 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 29 Apr 2010 12:57:39 -0400
Subject: [SciPy-User] scipy.integrate.ode method docs not findable in
	reference manual
In-Reply-To: <k2qce557a361004281533yd45aa68cte20c74790db31af6@mail.gmail.com>
References: <k2qce557a361004281533yd45aa68cte20c74790db31af6@mail.gmail.com>
Message-ID: <q2s1cd32cbb1004290957u9deb5e7ex2c5ea7ee037f3e53@mail.gmail.com>

On Wed, Apr 28, 2010 at 6:33 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> Hi,
>
> I was recently trying to look up some information on the
> scipy.integrate.ode objects, and I found that the reference manual
> lists the methods but does not link to their docstrings (which are
> unhelpful but that's beside the point). In the doc editor I was able
> to follow the links, so I suspect that the problem is in the doc
> generator rather than the docstrings themselves. Has anyone else
> encountered similar problems?

I haven't looked at it in a year, but last year I was complaining that
the class shadows the module, so access to the underlying module is
not possible.

integrate __init__.py:
from ode import *

>From a quick look it seems that this has not changed.

Josef

>
> Thanks,
> Anne
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From wesmckinn at gmail.com  Thu Apr 29 12:58:27 2010
From: wesmckinn at gmail.com (Wes McKinney)
Date: Thu, 29 Apr 2010 12:58:27 -0400
Subject: [SciPy-User] [ANN] la 0.2, the labeled array
In-Reply-To: <p2gbc657ead1004290037ze53a0942xf038360f79f9e2f4@mail.gmail.com>
References: <j2pf4f93d421004271405la4834eceu7e8b56a7e6a70f5a@mail.gmail.com> 
	<p2gbc657ead1004290037ze53a0942xf038360f79f9e2f4@mail.gmail.com>
Message-ID: <z2q6c476c8a1004290958wb53bd367t3847c414dbea38b1@mail.gmail.com>

On Thu, Apr 29, 2010 at 3:37 AM, Sebastian Haase <seb.haase at gmail.com> wrote:
> Hi Keith,
>
> how would you say do larrys compare to numpy recarrays ? Can you
> convert one to the other ? - IOW why do put the labels in a separate
> list instead of the dtype ?
>
> Regards,
> Sebastian Haase
>
>
> On Tue, Apr 27, 2010 at 11:05 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>> I am pleased to announce the second release of the la package, version 0.2.
>>
>> The main class of the la package is a labeled array, larry. A larry
>> consists of a data array and a label list. The data array is stored as
>> a NumPy array and the label list as a list of lists.
>>
>> larry has built-in methods such as movingsum, ranking, merge, shuffle,
>> zscore, demean, lag as well as typical Numpy methods like sum, max,
>> std, sign, clip. NaNs are treated as missing data.
>>
>> Alignment by label is automatic when you add (or subtract, multiply,
>> divide) two larrys.
>>
>> larry adds the convenience of labels, provides many built-in methods,
>> and let's you use many of your existing array functions.
>>
>> Download: https://launchpad.net/larry/+download
>> docs ?http://larry.sourceforge.net
>> code ?https://launchpad.net/larry
>> list ?http://groups.google.ca/group/pystatsmodels
>>
>> =============
>> Release Notes
>> =============
>>
>> la 0.2 (avocado)
>> ================
>>
>> *Release date: 2010-04-27*
>>
>> New larry methods
>> -----------------
>> - lix : Index into a larry using labels or index numbers or both
>> - swapaxes : Swap the two specified axes
>> - sortaxis : Sort data (and label) according to label along specified axis
>> - flipaxis : Reverse the order of the elements along the specified axis
>> - tocsv : Save larry to a csv file
>> - fromcsv : Load a larry from a csv file
>> - insertaxis : Insert a new axis at the specified position
>> - invert : Element by element inverting of True to False and False to True
>>
>> Enhancements
>> ------------
>> - All larry methods can now take nd input arrays (some previously 2d only)
>> - Added ability to save larrys with datetime.date labels to HDF5
>> - New function (panel) to convert larry of shape (n, m, k) to shape (m*k, n)
>> - Expanded documentation
>> - Over 280 new unit tests; testing easier with new assert_larry_equal function
>>
>> Bug fixes
>> ---------
>> - #517912: larry([]) == larry([]) raised IndexError
>> - #518096: larry.fromdict failed due to missing import
>> - #518106: la.larry.fromdict({}) failed
>> - #518114: fromlist([]) and fromtuples([]) failed
>> - #518135: keep_label crashed when there was nothing to keep
>> - #518210: sum, std, var returned NaN for empty larrys; now return 0.0
>> - #518215: unflatten crashed on an empty larry
>> - #518442: sum, std, var returned NaN for shapes that contain zero: (2, 0, 3)
>> - #568175: larry.std(axis=-1) and var crashed on negative axis input
>> - #569622: Negative axis input gave wrong output for several larry methods
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>

Record arrays only provide labels for the columns-- here the rows can
also have labels (example: dates, for a named collection of time
series).


From josef.pktd at gmail.com  Thu Apr 29 13:09:51 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 29 Apr 2010 13:09:51 -0400
Subject: [SciPy-User] [ANN] la 0.2, the labeled array
In-Reply-To: <p2gbc657ead1004290037ze53a0942xf038360f79f9e2f4@mail.gmail.com>
References: <j2pf4f93d421004271405la4834eceu7e8b56a7e6a70f5a@mail.gmail.com>
	<p2gbc657ead1004290037ze53a0942xf038360f79f9e2f4@mail.gmail.com>
Message-ID: <m2j1cd32cbb1004291009sba8b2c25kc50d9785f7fca82f@mail.gmail.com>

On Thu, Apr 29, 2010 at 3:37 AM, Sebastian Haase <seb.haase at gmail.com> wrote:
> Hi Keith,
>
> how would you say do larrys compare to numpy recarrays ? Can you
> convert one to the other ? - IOW why do put the labels in a separate
> list instead of the dtype ?

my impression:

the main difference is that
larrys have labels along all dimensions, e.g also for rows.

larrys data array has a single, homogenous dtype, not structured
dtypes, so standard array slicing and indexing applies along all axis,
mean, var,... is as usual with ndarrays

with recarrays you have to create a view first, assuming the column
dtypes are identical, otherwise it's a bit of a pain to get the mean
of all columns.

I'm not sure about larry.from_recarrays, larry.to_recarrays but if the
recarray has homogenous dtype it should be possible to use names and
views directly in the constructors.

Josef

Josef


>
> Regards,
> Sebastian Haase
>
>
> On Tue, Apr 27, 2010 at 11:05 PM, Keith Goodman <kwgoodman at gmail.com> wrote:
>> I am pleased to announce the second release of the la package, version 0.2.
>>
>> The main class of the la package is a labeled array, larry. A larry
>> consists of a data array and a label list. The data array is stored as
>> a NumPy array and the label list as a list of lists.
>>
>> larry has built-in methods such as movingsum, ranking, merge, shuffle,
>> zscore, demean, lag as well as typical Numpy methods like sum, max,
>> std, sign, clip. NaNs are treated as missing data.
>>
>> Alignment by label is automatic when you add (or subtract, multiply,
>> divide) two larrys.
>>
>> larry adds the convenience of labels, provides many built-in methods,
>> and let's you use many of your existing array functions.
>>
>> Download: https://launchpad.net/larry/+download
>> docs ?http://larry.sourceforge.net
>> code ?https://launchpad.net/larry
>> list ?http://groups.google.ca/group/pystatsmodels
>>
>> =============
>> Release Notes
>> =============
>>
>> la 0.2 (avocado)
>> ================
>>
>> *Release date: 2010-04-27*
>>
>> New larry methods
>> -----------------
>> - lix : Index into a larry using labels or index numbers or both
>> - swapaxes : Swap the two specified axes
>> - sortaxis : Sort data (and label) according to label along specified axis
>> - flipaxis : Reverse the order of the elements along the specified axis
>> - tocsv : Save larry to a csv file
>> - fromcsv : Load a larry from a csv file
>> - insertaxis : Insert a new axis at the specified position
>> - invert : Element by element inverting of True to False and False to True
>>
>> Enhancements
>> ------------
>> - All larry methods can now take nd input arrays (some previously 2d only)
>> - Added ability to save larrys with datetime.date labels to HDF5
>> - New function (panel) to convert larry of shape (n, m, k) to shape (m*k, n)
>> - Expanded documentation
>> - Over 280 new unit tests; testing easier with new assert_larry_equal function
>>
>> Bug fixes
>> ---------
>> - #517912: larry([]) == larry([]) raised IndexError
>> - #518096: larry.fromdict failed due to missing import
>> - #518106: la.larry.fromdict({}) failed
>> - #518114: fromlist([]) and fromtuples([]) failed
>> - #518135: keep_label crashed when there was nothing to keep
>> - #518210: sum, std, var returned NaN for empty larrys; now return 0.0
>> - #518215: unflatten crashed on an empty larry
>> - #518442: sum, std, var returned NaN for shapes that contain zero: (2, 0, 3)
>> - #568175: larry.std(axis=-1) and var crashed on negative axis input
>> - #569622: Negative axis input gave wrong output for several larry methods
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From emmanuelle.gouillart at normalesup.org  Thu Apr 29 15:38:26 2010
From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart)
Date: Thu, 29 Apr 2010 21:38:26 +0200
Subject: [SciPy-User] watershed question
In-Reply-To: <A9F18944-1712-41F3-B493-7A2B566FC0BF@yale.edu>
References: <20100425123445.GA25175@phare.normalesup.org>
	<r2y9457e7c81004280836x81ae43abv471e6c3e639c596e@mail.gmail.com>
	<A9F18944-1712-41F3-B493-7A2B566FC0BF@yale.edu>
Message-ID: <20100429193826.GA9432@phare.normalesup.org>

	Hi St?fan and Zach,

	thank you for your answers. So it seems that
ndimage.watershed_ift is quite buggy, maybe some warnings should be
added to its docstring? We can't afford people spending too much time
trying to use the function if it doesn't work.

	Using the cellprofile/cpmath package is a neat trick, I tried it
and it works perfectly. It even works for 3-D arrays (using the
fast_watershed function)! Too bad that it's GPL-licensed and it's not
possible to integrate the code in the image processing scikit :(. 

	Thanks again,

	Emmanuelle



On Wed, Apr 28, 2010 at 01:18:11PM -0400, Zachary Pincus wrote:
> > Unless I'm also missing something obvious, the code returns an invalid
> > result.  I even adjusted the depths of the two "pits", but always one
> > region overruns the other---not what I would expect to happen.  I
> > haven't delved into the ndimage code at all, but I wonder weather we
> > shouldn't implement one of the simpler algorithms as part of
> > scikits.image.segment for comparison?

> Cellprofiler has a watershed algorithm, I believe. And like most of  
> the cellprofiler stuff, the implementation seems pretty high-quality  
> and well-thought-out.

> I wound up extracting the cpmath sub-package, and (after a few  
> setup.py changes) it works great standalone with just scipy and numpy  
> as dependencies:
> https://svn.broadinstitute.org/CellProfiler/trunk/CellProfiler/cellprofiler/cpmath/

> Zach
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From robert.kern at gmail.com  Thu Apr 29 19:54:01 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 29 Apr 2010 18:54:01 -0500
Subject: [SciPy-User] [ANN] la 0.2, the labeled array
In-Reply-To: <m2uf4f93d421004290951r7373be14g54092059d163c3b6@mail.gmail.com>
References: <j2pf4f93d421004271405la4834eceu7e8b56a7e6a70f5a@mail.gmail.com> 
	<p2gbc657ead1004290037ze53a0942xf038360f79f9e2f4@mail.gmail.com> 
	<m2uf4f93d421004290951r7373be14g54092059d163c3b6@mail.gmail.com>
Message-ID: <i2p3d375d731004291654s2bb89c31vab8d0618799e8840@mail.gmail.com>

On Thu, Apr 29, 2010 at 11:51, Keith Goodman <kwgoodman at gmail.com> wrote:

> (Lately I've been getting mail from the numpy and scipy lists all at
> once, once a day or so. Anyone else seeing the same?)

Yes. The mail server has been unstable recently. It is currently being
migrated to a more stable machine. In the meantime, it is being
manually kicked into submission.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From warren.weckesser at enthought.com  Thu Apr 29 20:05:37 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Thu, 29 Apr 2010 19:05:37 -0500
Subject: [SciPy-User] [SciPy-user] lambda functions within matrix
In-Reply-To: <28393844.post@talk.nabble.com>
References: <28393844.post@talk.nabble.com>
Message-ID: <4BDA1ED1.10604@enthought.com>

archeryguru2000 wrote:
> Hello all,
> I'm trying to create a matrix that will allow me to specify which cells will
> contain certain values.  I have a line of code that I just cannot straighten
> out.  In SymPy I can use the following:
>
>   
>>>> sympy.Matrix(N,N,lambda i,j: i==j & j<int(M/2+1))
>>>>         
>
> where N and M are integers.  This creates a zero-matrix with ones in the
> upper left diagonal but the ones do not continue to the lower left.  However
> in SciPy, I assumed I would've used the following:
>
>   
>>>> scipy.fromfunction(lambda i,j: i==j & j<int(M/2+1), (N,N))
>>>>         
>
> but this gives an error that seems to pertain to my lambda function?
> unsupported operand type(s) for &: 'float' and 'float'
>
>   

In numpy, & has higher precedence than the relational operators == and 
<.  Use additional parentheses:

scipy.fromfunction(lambda i,j: (i==j) & (j<int(M/2+1)), (N,N))



For example:

In [16]: N = 4

In [17]: M = 2

In [18]: scipy.fromfunction(lambda i,j: (i==j) & (j<int(M/2+1)), (N,N))
Out[18]:
array([[ True, False, False, False],
       [False,  True, False, False],
       [False, False, False, False],
       [False, False, False, False]], dtype=bool)

In [19]: scipy.fromfunction(lambda i,j: (i==j) & (j<int(M/2+1)), 
(N,N)).astype(float)
Out[19]:
array([[ 1.,  0.,  0.,  0.],
       [ 0.,  1.,  0.,  0.],
       [ 0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.]])



Warren

> Any suggestions?
> Thanks,
> ~~archery~~
>   



From cycomanic at gmail.com  Thu Apr 29 21:00:46 2010
From: cycomanic at gmail.com (Jochen Schroeder)
Date: Fri, 30 Apr 2010 11:00:46 +1000
Subject: [SciPy-User] Help with fast fourier transform
In-Reply-To: <4BD9B105.8040501@gmail.com>
References: <4BD9B105.8040501@gmail.com>
Message-ID: <20100430010044.GA2175@cudos0803>

OK there's a number of things going on. First, fft results in a shifted array
(google for fftshift to find out more). Therefore your result should be:
F=np.fft.fftshift(np.fft.fft(f))

That will centre the result properly. Secondly the width of your resulting
array is a lot smaller than in the desired output (you only cut out the
centre). This is because your stepsize in the stepfunction is too large you
could fix that e.g. by doing f = g(np.arange(0,1000,0.1)). Finally your values
are all positive because you take the absolute value while in the expected result
you do not.

HTH

Jochen

On 04/29/10 11:17, Oscar Gerardo Lazo Arjona wrote:
> Hello! I'm new to this mailing list and to numpy in general.
> I need to calculate fft for my optics class. But I'm having trouble.
> 
> This is the code I'm using (commented):
> 
> import pylab
> import numpy as np
> 
> def g(x):
>    if x>450 and x<550:
>        return 1
>    else:
>        return 0
>       f=[g(x) for x in range(0,1000)]
> f=np.array(f)
> 
> #this funtion can be ploted as
> #http://i41.tinypic.com/k0shnk.png
> #which is a step function "centered" at 500
> 
> 
> F=np.fft.fft(f)
> #when calculate the fft of f i get an array of complex numbers
> #whose absolute value can be ploted as
> #http://i40.tinypic.com/1zcecxx.png
> 
> #But that is not the desired output.
> #Instead of that i expect something that can be ploted like this
> #http://i39.tinypic.com/1zmch0g.png
> 
> 
> #what i think must be happening because my function f
> #has an offset of 500 (it's supposed to be centereed at 500)
> 
> #So i think it all reduces to two options:
> #somehow telling fft to consider the origin at 500
> #(indicate thetell the offset)
> 
> #Or make fft accept a list of points like
> #[[-2,0],[-1,1],[0,1],[1,1],[2,0]]
> #so that it can know the position of the step relative to the origin
> 
> Please help!
> 
> Oscar
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From josef.pktd at gmail.com  Thu Apr 29 21:10:26 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 29 Apr 2010 21:10:26 -0400
Subject: [SciPy-User] Help with fast fourier transform
In-Reply-To: <4BD9B105.8040501@gmail.com>
References: <4BD9B105.8040501@gmail.com>
Message-ID: <t2n1cd32cbb1004291810r7701c674lef4bbb83f9204dce@mail.gmail.com>

On Thu, Apr 29, 2010 at 12:17 PM, Oscar Gerardo Lazo Arjona
<algebraicamente at gmail.com> wrote:
> Hello! I'm new to this mailing list and to numpy in general.
> I need to calculate fft for my optics class. But I'm having trouble.
>
> This is the code I'm using (commented):
>
> import pylab
> import numpy as np
>
> def g(x):
> ? if x>450 and x<550:
> ? ? ? return 1
> ? else:
> ? ? ? return 0
> ? ? ?f=[g(x) for x in range(0,1000)]
> f=np.array(f)
>
> #this funtion can be ploted as
> #http://i41.tinypic.com/k0shnk.png
> #which is a step function "centered" at 500
>
>
> F=np.fft.fft(f)
> #when calculate the fft of f i get an array of complex numbers
> #whose absolute value can be ploted as
> #http://i40.tinypic.com/1zcecxx.png
>
> #But that is not the desired output.
> #Instead of that i expect something that can be ploted like this
> #http://i39.tinypic.com/1zmch0g.png
>
>
> #what i think must be happening because my function f
> #has an offset of 500 (it's supposed to be centereed at 500)
>
> #So i think it all reduces to two options:
> #somehow telling fft to consider the origin at 500
> #(indicate thetell the offset)
>
> #Or make fft accept a list of points like
> #[[-2,0],[-1,1],[0,1],[1,1],[2,0]]
> #so that it can know the position of the step relative to the origin

>>> x = np.zeros(1000)
>>> x[450:550]=1
>>> y = np.fft.fftshift(np.fft.fft(np.fft.fftshift(x)))

this looks similar to your second graph, with recentering

Josef

>
> Please help!
>
> Oscar
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From charlesr.harris at gmail.com  Thu Apr 29 21:19:01 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 29 Apr 2010 19:19:01 -0600
Subject: [SciPy-User] Help with fast fourier transform
In-Reply-To: <4BD9B105.8040501@gmail.com>
References: <4BD9B105.8040501@gmail.com>
Message-ID: <u2we06186141004291819h4123f34ez1872a182861979a7@mail.gmail.com>

On Thu, Apr 29, 2010 at 10:17 AM, Oscar Gerardo Lazo Arjona <
algebraicamente at gmail.com> wrote:

> Hello! I'm new to this mailing list and to numpy in general.
> I need to calculate fft for my optics class. But I'm having trouble.
>
> This is the code I'm using (commented):
>
> import pylab
> import numpy as np
>
> def g(x):
>   if x>450 and x<550:
>       return 1
>   else:
>       return 0
>      f=[g(x) for x in range(0,1000)]
> f=np.array(f)
>
> #this funtion can be ploted as
> #http://i41.tinypic.com/k0shnk.png
> #which is a step function "centered" at 500
>
>
> F=np.fft.fft(f)
> #when calculate the fft of f i get an array of complex numbers
> #whose absolute value can be ploted as
> #http://i40.tinypic.com/1zcecxx.png
>
> #But that is not the desired output.
> #Instead of that i expect something that can be ploted like this
> #http://i39.tinypic.com/1zmch0g.png
>
>
> #what i think must be happening because my function f
> #has an offset of 500 (it's supposed to be centereed at 500)
>
>
Yep. Think of the data points as lying on a circle. You want the function
centered on index 0, so what this will look like is a chunk at the
beginning, a bunch of zeros, and a chunk at the end. Alternatively, you can
phase shift the transform, which in the case where the function is centered
amounts to multiplying all the odd indices by -1. Note that transformed data
will also be split between the two ends of the transform and you will have
to paste the right end before the left end to get the sinc-like plot. In any
case the result will not be centered.

#So i think it all reduces to two options:
> #somehow telling fft to consider the origin at 500
> #(indicate thetell the offset)
>
> #Or make fft accept a list of points like
> #[[-2,0],[-1,1],[0,1],[1,1],[2,0]]
> #so that it can know the position of the step relative to the origin
>
>
Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100429/253b2f8b/attachment.html>

From warren.weckesser at enthought.com  Thu Apr 29 21:46:08 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Thu, 29 Apr 2010 20:46:08 -0500
Subject: [SciPy-User] Help with fast fourier transform
In-Reply-To: <4BD9B105.8040501@gmail.com>
References: <4BD9B105.8040501@gmail.com>
Message-ID: <4BDA3660.7040103@enthought.com>

Oscar Gerardo Lazo Arjona wrote:
> Hello! I'm new to this mailing list and to numpy in general.
> I need to calculate fft for my optics class. But I'm having trouble.
>
> This is the code I'm using (commented):
>
> import pylab
> import numpy as np
>
> def g(x):
>    if x>450 and x<550:
>        return 1
>    else:
>        return 0
>       f=[g(x) for x in range(0,1000)]
> f=np.array(f)
>
> #this funtion can be ploted as
> #http://i41.tinypic.com/k0shnk.png
> #which is a step function "centered" at 500
>
>
> F=np.fft.fft(f)
> #when calculate the fft of f i get an array of complex numbers
> #whose absolute value can be ploted as
> #http://i40.tinypic.com/1zcecxx.png
>
> #But that is not the desired output.
> #Instead of that i expect something that can be ploted like this
> #http://i39.tinypic.com/1zmch0g.png
>
>
> #what i think must be happening because my function f
> #has an offset of 500 (it's supposed to be centereed at 500)
>
> #So i think it all reduces to two options:
> #somehow telling fft to consider the origin at 500
> #(indicate thetell the offset)
>
> #Or make fft accept a list of points like
> #[[-2,0],[-1,1],[0,1],[1,1],[2,0]]
> #so that it can know the position of the step relative to the origin
>
> Please help!
>   

Take a look at the functions numpy.fft.fftshift and numpy.fft.fftfreq.

The attached script shows an example.

Warren

> Oscar
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fft_example.py
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100429/4e592f46/attachment.ksh>

From david_baddeley at yahoo.com.au  Thu Apr 29 23:42:40 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Thu, 29 Apr 2010 20:42:40 -0700 (PDT)
Subject: [SciPy-User] Help with fast fourier transform
In-Reply-To: <4BD9B105.8040501@gmail.com>
References: <4BD9B105.8040501@gmail.com>
Message-ID: <105491.24353.qm@web33001.mail.mud.yahoo.com>

Your output looks correct. Most fft functions will give you the 0 frequency in the zeroeth output element, and the -ve frequencies in the second half of the array. It's more effcient to do it this way, and is clearly stated in the docs for fft. What you're probably after is fftshift (which can be found in the same palce as fft etc...) and swaps the halves of the array to give you the more traditional representation with the 0 frequency in the centre.

cheers,
David


----- Original Message ----
From: Oscar Gerardo Lazo Arjona <algebraicamente at gmail.com>
To: scipy-user at scipy.org
Sent: Fri, 30 April, 2010 4:17:09 AM
Subject: [SciPy-User] Help with fast fourier transform

Hello! I'm new to this mailing list and to numpy in general.
I need to calculate fft for my optics class. But I'm having trouble.

This is the code I'm using (commented):

import pylab
import numpy as np

def g(x):
   if x>450 and x<550:
       return 1
   else:
       return 0
      f=[g(x) for x in range(0,1000)]
f=np.array(f)

#this funtion can be ploted as
#http://i41.tinypic.com/k0shnk.png
#which is a step function "centered" at 500


F=np.fft.fft(f)
#when calculate the fft of f i get an array of complex numbers
#whose absolute value can be ploted as
#http://i40.tinypic.com/1zcecxx.png

#But that is not the desired output.
#Instead of that i expect something that can be ploted like this
#http://i39.tinypic.com/1zmch0g.png


#what i think must be happening because my function f
#has an offset of 500 (it's supposed to be centereed at 500)

#So i think it all reduces to two options:
#somehow telling fft to consider the origin at 500
#(indicate thetell the offset)

#Or make fft accept a list of points like
#[[-2,0],[-1,1],[0,1],[1,1],[2,0]]
#so that it can know the position of the step relative to the origin

Please help!

Oscar
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From chris.michalski at gmail.com  Fri Apr 30 00:06:56 2010
From: chris.michalski at gmail.com (Chris Michalski)
Date: Thu, 29 Apr 2010 21:06:56 -0700
Subject: [SciPy-User] Help with fast fourier transform
In-Reply-To: <4BD9B105.8040501@gmail.com>
References: <4BD9B105.8040501@gmail.com>
Message-ID: <50D26A58-F9DC-4556-B172-F0E029EBB8BC@gmail.com>

Looks like your problem is one of expectations.  

You did an FFT on real data - specifically a "pulse" that is 100 point long surrounded by 450 "zeros" on either side.  Since the FFT function computes a complex FFT - you got negative and positive frequencies.  As the FFT samples come out, they represent "DC" (zero freq) through just shy of fs/2 (fs = sample rate), then the output wraps to -fs/2 which then advances to just shy of  "0".

Your original output is correct - it just needs to be reordered.  The "desired" output you referenced (the picture) takes the last "N/2" complex points from the FFT function and moves then to the first N/s points.   Once the data is reordered, before you plot you have to set the frequency resolution of the FFT.  In a complex FFT output, the first sample in the reordered output would correspond to -fs/2 and the uppermost frequency would be (about fs/2)  The complication is that there is a zero frequency (DC offset) term.  Since the FFT returns an even number of samples (specifically a power of two), the convention I remember is that there are more "negative frequency" samples than positive frequency samples.

By the way.  There is a much simpler way to get a pulse, specifically:

import numpy as np

f = np.zeros(1000, dtype='d')
f[450:550] = 1.0

If "F" is the FFT of "f" (the time function), then the reordering can be accomplished with

# Create new array to hold the reordered output (holding complex data)
size = len(F)
FF = np.array(size, dtype='D')

FF[0:size/2] = F[size/2:]
FF[size/2+1:]  = F[0:size/2]

Getting the hang of thinking of operations as vectored code rather than loops can take some practice - but well worth the results.  

chris


On Apr 29, 2010, at 9:17 AM, Oscar Gerardo Lazo Arjona wrote:

> Hello! I'm new to this mailing list and to numpy in general.
> I need to calculate fft for my optics class. But I'm having trouble.
> 
> This is the code I'm using (commented):
> 
> import pylab
> import numpy as np
> 
> def g(x):
>   if x>450 and x<550:
>       return 1
>   else:
>       return 0
>      f=[g(x) for x in range(0,1000)]
> f=np.array(f)
> 
> #this funtion can be ploted as
> #http://i41.tinypic.com/k0shnk.png
> #which is a step function "centered" at 500
> 
> 
> F=np.fft.fft(f)
> #when calculate the fft of f i get an array of complex numbers
> #whose absolute value can be ploted as
> #http://i40.tinypic.com/1zcecxx.png
> 
> #But that is not the desired output.
> #Instead of that i expect something that can be ploted like this
> #http://i39.tinypic.com/1zmch0g.png
> 
> 
> #what i think must be happening because my function f
> #has an offset of 500 (it's supposed to be centereed at 500)
> 
> #So i think it all reduces to two options:
> #somehow telling fft to consider the origin at 500
> #(indicate thetell the offset)
> 
> #Or make fft accept a list of points like
> #[[-2,0],[-1,1],[0,1],[1,1],[2,0]]
> #so that it can know the position of the step relative to the origin
> 
> Please help!
> 
> Oscar
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From dwf at cs.toronto.edu  Fri Apr 30 03:31:51 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 30 Apr 2010 03:31:51 -0400
Subject: [SciPy-User] help interpreting univariate spline
In-Reply-To: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com>
References: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com>
Message-ID: <4BDA8767.4060202@cs.toronto.edu>

Elliot Hallmark wrote:
> Hi there,
>
> I am wanting to use a scipy interpolation routine to generate
> polynomials approximating function I have sampled.  these polynomials
> will be solved in c (cython), so I need to extract the coefficencts
> from the interpolation method and pass them to a c function for root
> finding.
>
> using UnivariateSpline I ran this code:
>
> import numpy as np
> from scipy.interpolate import splrep, UnivariateSpline
>
> x = np.linspace(1, 10, 100)
> y = 1/x     #approximate a hyperbola
> g = UnivariateSpline(x, y, w=None, k=3, s=.0005)
> print g.get_knots()
> print g.get_coeffs()
>
> import matplotlib.pyplot as plt
> plt.clf()
> ax = plt.subplot(121)
> l1, = ax.plot(x, y, "rs", ms=10)
> l2, = ax.plot(x, g(x), "bo")
> plt.show()
>
>
>
>
> and the output was:
> [  1.           2.18181818   3.27272727   5.54545455  10.        ]
>
> [ 0.98462432  0.66575572  0.423       0.25461626  0.13929847  0.11763985
>   0.09929961]
>
> That is 5 knots and 7 coefficients for a degree=3 spline.  naively, I
> was expecting 4 coefficents for each interval between knots.
>
> how can I construct a piecewise polynomial from this output? or can I?
>   
You might do well to look through the original documentation for Paul 
Dierckx's FITPACK, which scipy.interpolate wraps.

A textbook he wrote that describes some of the routines is available on 
Google Books: http://tinyurl.com/2djtlnl

David


From jason-sage at creativetrax.com  Fri Apr 30 08:05:26 2010
From: jason-sage at creativetrax.com (Jason Grout)
Date: Fri, 30 Apr 2010 07:05:26 -0500
Subject: [SciPy-User] help interpreting univariate spline
In-Reply-To: <4BDA8767.4060202@cs.toronto.edu>
References: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com>
	<4BDA8767.4060202@cs.toronto.edu>
Message-ID: <4BDAC786.6010501@creativetrax.com>

>
> Elliot Hallmark wrote:
>    
>> Hi there,
>>
>> I am wanting to use a scipy interpolation routine to generate
>> polynomials approximating function I have sampled.  these polynomials
>> will be solved in c (cython), so I need to extract the coefficencts
>> from the interpolation method and pass them to a c function for root
>> finding.
>>
>> using UnivariateSpline I ran this code:
>>
>> import numpy as np
>> from scipy.interpolate import splrep, UnivariateSpline
>>
>> x = np.linspace(1, 10, 100)
>> y = 1/x     #approximate a hyperbola
>> g = UnivariateSpline(x, y, w=None, k=3, s=.0005)
>> print g.get_knots()
>> print g.get_coeffs()
>>
>> import matplotlib.pyplot as plt
>> plt.clf()
>> ax = plt.subplot(121)
>> l1, = ax.plot(x, y, "rs", ms=10)
>> l2, = ax.plot(x, g(x), "bo")
>> plt.show()
>>
>>
>>
>>
>> and the output was:
>> [  1.           2.18181818   3.27272727   5.54545455  10.        ]
>>
>> [ 0.98462432  0.66575572  0.423       0.25461626  0.13929847  0.11763985
>>    0.09929961]
>>
>> That is 5 knots and 7 coefficients for a degree=3 spline.  naively, I
>> was expecting 4 coefficents for each interval between knots.
>>
>> how can I construct a piecewise polynomial from this output? or can I?
>>
>>      
> You might do well to look through the original documentation for Paul
> Dierckx's FITPACK, which scipy.interpolate wraps.
>
>    

Also, if you looked at using PPPACK via f2py to get the coefficients of 
a cubic spline, you might find the following useful:

http://www.sagenb.org/home/pub/1708/ [1]

Note that in this function, the 4 coefficients returned for each piece 
are not the coefficients of x^3, x^2, etc., but rather, the coefficients 
are numbers for a Horner-like form of the polynomial (f(x) = 
c(1,i)+h*(c(2,i)+h*(c(3,i)+h*c(4,i)/3.)/2.)) where h = x - tau(i) -- see 
the documentation for cubspl or the example above to construct the 
actual polynomial.)

Thanks,

Jason

[1] (In Sage, using %fortran at the top of a cell wraps the code in an 
f2py wrapper.)



From peridot.faceted at gmail.com  Fri Apr 30 10:04:55 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 30 Apr 2010 10:04:55 -0400
Subject: [SciPy-User] help interpreting univariate spline
In-Reply-To: <n2na07a0f1c1004261226m5d5416c7ib79812469d9978b6@mail.gmail.com>
References: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com> 
	<n2na07a0f1c1004261226m5d5416c7ib79812469d9978b6@mail.gmail.com>
Message-ID: <o2kce557a361004300704ucf8b7b52tc975fd46f5785508@mail.gmail.com>

On 26 April 2010 15:26, Elliot Hallmark <permafacture at gmail.com> wrote:
> On Fri, Apr 23, 2010 at 6:53 PM, Elliot Hallmark <permafacture at gmail.com> wrote:
>
>> I am wanting to use a scipy interpolation routine to generate
>> polynomials approximating function I have sampled.
>> ...
>> import numpy as np
>>from scipy.interpolate import splrep, UnivariateSpline
>
>>x = np.linspace(1, 10, 100)
>>y = 1/x ? ? #approximate a hyperbola
>>g = UnivariateSpline(x, y, w=None, k=3, s=.0005)
>>print g.get_knots()
>>print g.get_coeffs()
>
>>and the output was:
>>[ ?1. ? ? ? ? ? 2.18181818 ? 3.27272727 ? 5.54545455 ?10. ? ? ? ?]
>>
>>[ 0.98462432 ?0.66575572 ?0.423 ? ? ? 0.25461626 ?0.13929847 ?0.11763985
> ?0.09929961]
>
>>That is 5 knots and 7 coefficients for a degree=3 spline. ?naively, I
>>was expecting 4 coefficents for each interval between knots.
>
> I found this post:
>
> http://mail.scipy.org/pipermail/scipy-user/2009-June/021572.html
>
> This seems very round-a-bout and ignores the piecewise nature of the
> spline. ?I mean, really? is there no direct or documented way to get
> the polynomial coefficents from a spline in python? ?If not, then this
> code is the best I have found. ?For my purposes I have modified it
> (see below)

There is indeed no direct way to get the polynomials in a familiar
basis. (I'll point out that you lose accuracy - possibly a lot of it -
by converting polynomials from one representation to another.) If all
you want to do is solve the polynomials, though, scipy already
provides root-finding functionality in its splines, based on routines
built into FITPACK. Also integration and differentiation.

If you really want the polynomials, though, the tck representation
scipy uses is semi-standard for representing splines; you may find the
non-object-oriented interface (splrep, splev, etc.) somewhat less
opaque in this respect. If you do decide to decipher the results, keep
in mind that with the knots held fixed, it's a linear representation
of the space of piecewise-cubic functions, so if you can find
representations for your basis functions (e.g. 1, x, x**2, x**3) you
can easily work out the conversion. And since the interpolating spline
for each of those functions is itself, all you need to do is four
interpolations on a fixed set of knots.

Anne


From wing1127aishi at gmail.com  Fri Apr 30 10:28:41 2010
From: wing1127aishi at gmail.com (Leon Sit)
Date: Fri, 30 Apr 2010 09:28:41 -0500
Subject: [SciPy-User] SWIG/numpy help
Message-ID: <u2z3af1998d1004300728t7d0fb9d6t612e9f4053c355f7@mail.gmail.com>

Hi all:

I am trying to pass data around the numpy and boost::ublas layers. I
have written an ultra thin wrapper because swig cannot parse ublas'
header correctly. The code is shown below

-----------------------------------------------------------------------------------
#include <boost/numeric/ublas/vector.hpp>
#include <boost/numeric/ublas/matrix.hpp>
#include <boost/lexical_cast.hpp>

#include <algorithm>
#include <sstream>
#include <string>

using std::copy;
using namespace boost;

typedef boost::numeric::ublas::matrix<double> dm;
typedef boost::numeric::ublas::vector<double> dv;

class dvector : public dv{
 public:
 dvector(const int rhs):dv(rhs){;};
  dvector();
 dvector(const int size, double* ptr):dv(size){
    copy(ptr, ptr+sizeof(double)*size, &(dv::data()[0]));
  }
 ~dvector(){}
};
-------------------------------------------------------------------------------------------------
with the SWIG interface that looks something like
-----------------------------------------------------------------------------------------------

%apply(int DIM1, double* INPLACE_ARRAY1) {(const int size, double* ptr)}
 class dvector{
 public:
 dvector(const int rhs);
  dvector();
  dvector(const int size, double* ptr);
	%newobject toString;
  char* toString();
	~dvector();
};
---------------------------------------------------------------------------------------------------
I have compiled them successfully via gcc 4.3 and vc++9.0. However
when I simply run

a = dvector(array([1.,2.,3.]))

it gives me a segfault. This is the first time I use swigh with numpy
and not have fully understanding between the data conversion and
memory buffer passing. Does anyone see something obvious I have
missed? Anything is appreciated.

Best Regards,

Leon Sit
Graduate Student in Statistics at Rutgers University
wing1127aishi at gmail.com
http://eden.rutgers.edu/~wingsit/


From archeryguru2000 at gmail.com  Fri Apr 30 11:25:57 2010
From: archeryguru2000 at gmail.com (archeryguru2000)
Date: Fri, 30 Apr 2010 08:25:57 -0700 (PDT)
Subject: [SciPy-User] [SciPy-user] lambda functions within matrix
In-Reply-To: <4BDA1ED1.10604@enthought.com>
References: <28393844.post@talk.nabble.com> <4BDA1ED1.10604@enthought.com>
Message-ID: <28413564.post@talk.nabble.com>



Warren Weckesser-3 wrote:
> 
> 
> In [18]: scipy.fromfunction(lambda i,j: (i==j) & (j<int(M/2+1)), (N,N))
> 
> Warren
> 
> 
_______________________________________________

Thank you very much.  This helped out greatly.

~~archery~~
-- 
View this message in context: http://old.nabble.com/lambda-functions-within-matrix-tp28393844p28413564.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From permafacture at gmail.com  Fri Apr 30 16:14:36 2010
From: permafacture at gmail.com (Elliot Hallmark)
Date: Fri, 30 Apr 2010 15:14:36 -0500
Subject: [SciPy-User] help interpreting univariate spline
In-Reply-To: <o2kce557a361004300704ucf8b7b52tc975fd46f5785508@mail.gmail.com>
References: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com>
	<n2na07a0f1c1004261226m5d5416c7ib79812469d9978b6@mail.gmail.com>
	<o2kce557a361004300704ucf8b7b52tc975fd46f5785508@mail.gmail.com>
Message-ID: <v2sa07a0f1c1004301314z6dbf63d0p9177299c6cfda4fb@mail.gmail.com>

>  (I'll point out that you lose accuracy - possibly a lot of it -
> by converting polynomials from one representation to another.)

error between evaluation of the interpolation and eval of polynomial
is consistently 10^-16.  I don't know if this is huge in computer
world, but it is
quite reasonable to me.  10^-16 is about zero in my view.


 >If all
> you want to do is solve the polynomials, though, scipy already
> provides root-finding functionality in its splines

But I will be using c code through cython to solve for roots, which is
why I want a transparent way to solve the interpolation.

>
> If you really want the polynomials, though, the tck representation
> scipy uses is semi-standard for representing splines; you may find the
> non-object-oriented interface (splrep, splev, etc.) somewhat less
> opaque in this respect. If you do decide to decipher the results, keep
> in mind that with the knots held fixed, it's a linear representation
> of the space of piecewise-cubic functions, so if you can find
> representations for your basis functions (e.g. 1, x, x**2, x**3) you
> can easily work out the conversion. And since the interpolating spline
> for each of those functions is itself, all you need to do is four
> interpolations on a fixed set of knots.


um, I think this is what I already have done?  But the "semi standard"
spline representation in tck is completely undocumented as far as I
can tell.  The only way to get the polynomial coefficents I can tell
is through evaluating the derivatives.

Do you know what is the meaning of the coefficents splrep generates?
How could such a small set of coefficents represent all the
information of a cubic function?


From t.charrett at cranfield.ac.uk  Fri Apr 30 16:38:04 2010
From: t.charrett at cranfield.ac.uk (Charrett, Thomas)
Date: Fri, 30 Apr 2010 21:38:04 +0100
Subject: [SciPy-User] [ANN] PythonToolkit (PTK) - open source,
 wxPython interactive environment
Message-ID: <CB2B2AE46366814B817A6E0B01EE7B3704D6311D65@CCEXCHSTORE-1.central.cranfield.ac.uk>

Hello all,
I would like to announce a interactive environment for python, PythonToolkit, or PTK, is loosely based on the MATLAB gui interface and is written in wxpython.:
http://pythontoolkit.sourceforge.net

It started life a personal project for use in day-to-day lab work and teaching (including myself) python so it is far from complete, but usable.
Key features:

- A console window with  support for muliple isolated python interpreters (engines) running as external processes,
- External (process) engines allow interactive use of GUI toolkits (currently wxPython and Tk)
- Full object inspection, auto-completions and call tips in internal and external engines.
- Matlab style namespace/workspace browser than can be extended/customised for different python types.
- GUI views for strings, unicode, lists and numpy arrays (more can be easily added)
- Python path management .
- A simple python code editor.
- Searchable command history that is stored between sessions.
- Extendible via a tool plugin system.

I would be interested in any comments/suggestion/feedback.

Thanks,

Tom

From josef.pktd at gmail.com  Fri Apr 30 16:40:33 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 30 Apr 2010 16:40:33 -0400
Subject: [SciPy-User] help interpreting univariate spline
In-Reply-To: <v2sa07a0f1c1004301314z6dbf63d0p9177299c6cfda4fb@mail.gmail.com>
References: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com>
	<n2na07a0f1c1004261226m5d5416c7ib79812469d9978b6@mail.gmail.com>
	<o2kce557a361004300704ucf8b7b52tc975fd46f5785508@mail.gmail.com>
	<v2sa07a0f1c1004301314z6dbf63d0p9177299c6cfda4fb@mail.gmail.com>
Message-ID: <v2l1cd32cbb1004301340i8383dddbk1bc8719c34bed277@mail.gmail.com>

On Fri, Apr 30, 2010 at 4:14 PM, Elliot Hallmark <permafacture at gmail.com> wrote:
>> ?(I'll point out that you lose accuracy - possibly a lot of it -
>> by converting polynomials from one representation to another.)
>
> error between evaluation of the interpolation and eval of polynomial
> is consistently 10^-16. ?I don't know if this is huge in computer
> world, but it is
> quite reasonable to me. ?10^-16 is about zero in my view.
>
>
> ?>If all
>> you want to do is solve the polynomials, though, scipy already
>> provides root-finding functionality in its splines
>
> But I will be using c code through cython to solve for roots, which is
> why I want a transparent way to solve the interpolation.
>
>>
>> If you really want the polynomials, though, the tck representation
>> scipy uses is semi-standard for representing splines; you may find the
>> non-object-oriented interface (splrep, splev, etc.) somewhat less
>> opaque in this respect. If you do decide to decipher the results, keep
>> in mind that with the knots held fixed, it's a linear representation
>> of the space of piecewise-cubic functions, so if you can find
>> representations for your basis functions (e.g. 1, x, x**2, x**3) you
>> can easily work out the conversion. And since the interpolating spline
>> for each of those functions is itself, all you need to do is four
>> interpolations on a fixed set of knots.
>
>
> um, I think this is what I already have done? ?But the "semi standard"
> spline representation in tck is completely undocumented as far as I
> can tell. ?The only way to get the polynomial coefficents I can tell
> is through evaluating the derivatives.
>
> Do you know what is the meaning of the coefficents splrep generates?
> How could such a small set of coefficents represent all the
> information of a cubic function?

just a guess:
http://en.wikipedia.org/wiki/B-spline#Cubic_B-Spline

has a representation of the curve as linear combination of spline
basis functions which requires only m-2 coefficients.

The fortran and f2py code is not informative about this, maybe the
original documentation by Dierckx shows the exact specification.

But I still wouldn't see from the wikipedia representation, how to get
the coefficients of the piecewise polynomial back.

Josef


> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From dwf at cs.toronto.edu  Fri Apr 30 21:32:55 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 30 Apr 2010 21:32:55 -0400
Subject: [SciPy-User] help interpreting univariate spline
In-Reply-To: <v2sa07a0f1c1004301314z6dbf63d0p9177299c6cfda4fb@mail.gmail.com>
References: <t2va07a0f1c1004231653g630f0a71ma1c2e1779c594941@mail.gmail.com>
	<n2na07a0f1c1004261226m5d5416c7ib79812469d9978b6@mail.gmail.com>
	<o2kce557a361004300704ucf8b7b52tc975fd46f5785508@mail.gmail.com>
	<v2sa07a0f1c1004301314z6dbf63d0p9177299c6cfda4fb@mail.gmail.com>
Message-ID: <E6155323-AD72-4E53-AAC8-11A809B85AB8@cs.toronto.edu>


On 2010-04-30, at 4:14 PM, Elliot Hallmark wrote:

> um, I think this is what I already have done?  But the "semi standard"
> spline representation in tck is completely undocumented as far as I
> can tell.  The only way to get the polynomial coefficents I can tell
> is through evaluating the derivatives.

It's documented, in the FITPACK user's manual, and possibly in that book that I pointed you to in another reply.

> Do you know what is the meaning of the coefficents splrep generates?
> How could such a small set of coefficents represent all the
> information of a cubic function?

Probably because they are not general cubic functions. They are constrained to have equal 0th through 2nd derivatives at each of the knot points. This limits the degrees of freedom considerably.

David

From solomon.negusse at twdb.state.tx.us  Thu Apr  8 17:49:33 2010
From: solomon.negusse at twdb.state.tx.us (Solomon M Negusse)
Date: Thu, 08 Apr 2010 21:49:33 -0000
Subject: [SciPy-User] re[SciPy-user] ading time series file using tsfromtxt
Message-ID: <28185028.post@talk.nabble.com>


Hello, 
I'm trying to read a time series of stream discharge using ts.genfromtxt.
The file looks like:

#agency_cd	site_no	datetime	02_00060_00003	02_00060_00003_cd
#5s	15s	16d	14n	10s
agency	  station	  1940-07-01	    1700	A 
agency	  station       1940-07-02	    350   	A

I'm trying to read in the third 'date' column and the 4th  'data' column
using: 

dateconverter = lambda y:
datetime.date(year=int(y.split('-')[0]),month=int(y.split('-')[1]),day=int(y.split('-')[2]))

flowts = ts.tsfromtxt('file', comments='#',datecols=2,
usecols=(2,3),dateconverter=dateconverter,freq='D')

This is returning an error 
Line #25309 (got 3 columns instead of 2). 

I'm fairly new with python and I'm hoping this will be a quick fix for the
experienced users here. I appreciate
any help I can get on this.

Thanks,
Solomon






-- 
View this message in context: http://old.nabble.com/reading-time-series-file-using-tsfromtxt-tp28185028p28185028.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From Nikolaus at rath.org  Tue Apr 20 13:27:28 2010
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Tue, 20 Apr 2010 17:27:28 -0000
Subject: [SciPy-User] matplotlib: ticks at 0 and 2pi
References: <87bpdf5ete.fsf@inspiron.ap.columbia.edu>
Message-ID: <87hbn684by.fsf@inspiron.ap.columbia.edu>

Nikolaus Rath <Nikolaus at rath.org> writes:
> Hello,
>
> I'm trying to plot something from 0 to 2pi:
[...]

Nevermind, the problem disappeared after restarting the interpreter.
Maybe I accidentally messed up some internal data structure.



Best,

   -Nikolaus

-- 
 ?Time flies like an arrow, fruit flies like a Banana.?

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C