From fabian.pedregosa at inria.fr  Mon Feb  1 04:28:20 2010
From: fabian.pedregosa at inria.fr (Fabian Pedregosa)
Date: Mon, 01 Feb 2010 10:28:20 +0100
Subject: [SciPy-dev] Ball Tree code updated (ticket 1048)
In-Reply-To: <58df6dc21001061448l4dc9ef05x478437a7837d3c32@mail.gmail.com>
References: <58df6dc21001061448l4dc9ef05x478437a7837d3c32@mail.gmail.com>
Message-ID: <4B669EB4.9070903@inria.fr>

Jake VanderPlas wrote:
> Hello,
> I have had comments from a few people over the last two months on the
> Ball Tree code that I submitted (ticket 1048).  I cleaned up the code
> a bit and posted the changes on the tracker.  Any other comments would
> be appreciated!
>    -Jake
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
> 

Hi! Your code works great, with impressive speed improvements in high 
dimensions. I could just find a minor thing: leaf_size in the docstring 
should read leafsize.

If this turns out to be too specific for scipy, would you mind that it 
get's included in scikit.learn [1] (BSD-license)?

Thanks,

fabian


[1] http://scikit-learn.sourceforge.net



From david at silveregg.co.jp  Thu Feb  4 01:27:57 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Thu, 04 Feb 2010 15:27:57 +0900
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent (computing
	a few eigenvalues only)
Message-ID: <4B6A68ED.6070007@silveregg.co.jp>

Hi,

I have played a bit with adding support for computing a few eigenvalues 
of full symmetric matrices. I would like some comments on the current API:

import numpy as np
from scipy.linalg import eigs
x = np.random.randn(10, 10)
x = np.dot(x.T, x)
# Retrieve the 3 biggest eigenvalues
eigs(x, 3)[0]
# Retrieve the 3 smallest eigenvalues
eigs(x, -3)[0]
# Retrieve the 3 biggest eigenvalues
eigs(x, [0, 3], mode="index")[0]
# Retrieve the 2nd and 3rd biggest
eigs(x, [1, 3], mode="index")[0]
# Retrieve all the eigenvalues in the range [1.5, 3.5[
eigs(x, [1.5, 3.5], mode="range")[0]

One thing which does not feel right is that that the range in the 
"index" mode is exactly inverted compared to the output (i.e. if you ask 
for the range [0, 3], you get the last three items from what you would 
get if you asked for the full range [0, 10]), but this is because I kept 
compatibility with Octave (always showing from biggest to smallest). It 
would be easy to always do the contrary (from smallest to biggest) - 
besides consistency, it has the advantages that it is the actual order 
you get back from the underlying LAPACK function.

Also, I needed to modify the f2py file for the related LAPACK functions, 
effectively changing their API in a backward incompatible way. Are the 
low-level f2py wrappers considered public API ?

cheers,

David


From charlesr.harris at gmail.com  Thu Feb  4 01:37:59 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 3 Feb 2010 23:37:59 -0700
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <4B6A68ED.6070007@silveregg.co.jp>
References: <4B6A68ED.6070007@silveregg.co.jp>
Message-ID: <e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>

On Wed, Feb 3, 2010 at 11:27 PM, David Cournapeau <david at silveregg.co.jp>wrote:

> Hi,
>
> I have played a bit with adding support for computing a few eigenvalues
> of full symmetric matrices. I would like some comments on the current API:
>
> import numpy as np
> from scipy.linalg import eigs
> x = np.random.randn(10, 10)
> x = np.dot(x.T, x)
> # Retrieve the 3 biggest eigenvalues
> eigs(x, 3)[0]
> # Retrieve the 3 smallest eigenvalues
> eigs(x, -3)[0]
> # Retrieve the 3 biggest eigenvalues
> eigs(x, [0, 3], mode="index")[0]
> # Retrieve the 2nd and 3rd biggest
> eigs(x, [1, 3], mode="index")[0]
> # Retrieve all the eigenvalues in the range [1.5, 3.5[
> eigs(x, [1.5, 3.5], mode="range")[0]
>
> One thing which does not feel right is that that the range in the
> "index" mode is exactly inverted compared to the output (i.e. if you ask
>

Why not use separate keywords for index and range. Changing the meaning of
an argument using another keyword is just weird. I've seen it used
elsewhere, but still... At some point you might even want the largest three
in a range. So something like

eigs(x, index=3)
eigs(x, range=[1,4])

Why the index on the end? Is the return a list?


> for the range [0, 3], you get the last three items from what you would
> get if you asked for the full range [0, 10]), but this is because I kept
> compatibility with Octave (always showing from biggest to smallest). It
>

Why follow Octave, isn't Octave like Matlab? Matlab functions are a mess.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100203/26491538/attachment.html>

From david at silveregg.co.jp  Thu Feb  4 01:49:42 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Thu, 04 Feb 2010 15:49:42 +0900
Subject: [SciPy-dev] Comments on API for Matlab's eigs
	equivalent	(computing a few eigenvalues only)
In-Reply-To: <e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
Message-ID: <4B6A6E06.7060102@silveregg.co.jp>

Charles R Harris wrote:
> 
> 
> On Wed, Feb 3, 2010 at 11:27 PM, David Cournapeau <david at silveregg.co.jp 
> <mailto:david at silveregg.co.jp>> wrote:
> 
>     Hi,
> 
>     I have played a bit with adding support for computing a few eigenvalues
>     of full symmetric matrices. I would like some comments on the
>     current API:
> 
>     import numpy as np
>     from scipy.linalg import eigs
>     x = np.random.randn(10, 10)
>     x = np.dot(x.T, x)
>     # Retrieve the 3 biggest eigenvalues
>     eigs(x, 3)[0]
>     # Retrieve the 3 smallest eigenvalues
>     eigs(x, -3)[0]
>     # Retrieve the 3 biggest eigenvalues
>     eigs(x, [0, 3], mode="index")[0]
>     # Retrieve the 2nd and 3rd biggest
>     eigs(x, [1, 3], mode="index")[0]
>     # Retrieve all the eigenvalues in the range [1.5, 3.5[
>     eigs(x, [1.5, 3.5], mode="range")[0]
> 
>     One thing which does not feel right is that that the range in the
>     "index" mode is exactly inverted compared to the output (i.e. if you ask
> 
> 
> Why not use separate keywords for index and range.

Then I am not sure how to handle the "give me the k biggest/smallest" 
case, which is the most common I think (that's the only one I care 
personally :) ).

Maybe this just warrants several functions ? But then there is the issue 
of naming (supporting non-symmetric matrices would be nice, but requires 
a totally different implementation, as LAPACK does not support it AFAIK 
- the easiest way would be to use ARPACK ATM).

> Changing the meaning 
> of an argument using another keyword is just weird. 

Agreed - this is just the best I could came up with one function to 
support all cases while keeping the common case simple. But I feel a bit 
like I outsmarted myself here.

> Why the index on the end? Is the return a list?

Yes, it also returns eigenvectors

> 
>     for the range [0, 3], you get the last three items from what you would
>     get if you asked for the full range [0, 10]), but this is because I kept
>     compatibility with Octave (always showing from biggest to smallest). It
> 
> 
> Why follow Octave, isn't Octave like Matlab? Matlab functions are a mess.

Actually, there is another reason I forgot to mention: that's how eigen 
in scipy.sparse does as well (in decreasing order). I think that the 
ARPACK wrappers are not that well thought out, though.

cheers,

David


From charlesr.harris at gmail.com  Thu Feb  4 02:06:20 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 4 Feb 2010 00:06:20 -0700
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <4B6A6E06.7060102@silveregg.co.jp>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
Message-ID: <e06186141002032306s711685f0n5454502496f72706@mail.gmail.com>

On Wed, Feb 3, 2010 at 11:49 PM, David Cournapeau <david at silveregg.co.jp>wrote:

> Charles R Harris wrote:
> >
> >
> > On Wed, Feb 3, 2010 at 11:27 PM, David Cournapeau <david at silveregg.co.jp
> > <mailto:david at silveregg.co.jp>> wrote:
> >
> >     Hi,
> >
> >     I have played a bit with adding support for computing a few
> eigenvalues
> >     of full symmetric matrices. I would like some comments on the
> >     current API:
> >
> >     import numpy as np
> >     from scipy.linalg import eigs
> >     x = np.random.randn(10, 10)
> >     x = np.dot(x.T, x)
> >     # Retrieve the 3 biggest eigenvalues
> >     eigs(x, 3)[0]
> >     # Retrieve the 3 smallest eigenvalues
> >     eigs(x, -3)[0]
> >     # Retrieve the 3 biggest eigenvalues
> >     eigs(x, [0, 3], mode="index")[0]
> >     # Retrieve the 2nd and 3rd biggest
> >     eigs(x, [1, 3], mode="index")[0]
> >     # Retrieve all the eigenvalues in the range [1.5, 3.5[
> >     eigs(x, [1.5, 3.5], mode="range")[0]
> >
> >     One thing which does not feel right is that that the range in the
> >     "index" mode is exactly inverted compared to the output (i.e. if you
> ask
> >
> >
> > Why not use separate keywords for index and range.
>
> Then I am not sure how to handle the "give me the k biggest/smallest"
> case, which is the most common I think (that's the only one I care
> personally :) ).
>
>
That would be eigs(x, index=3) or eigs(x, index=-3) respectively, the
default value of both range and index would be None, which could possibly
return all eigenvalues. I'm not sure that index is the best word but I can't
think of a better at the moment.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100204/a1b3fd4f/attachment.html>

From dagss at student.matnat.uio.no  Thu Feb  4 03:12:04 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Feb 2010 09:12:04 +0100
Subject: [SciPy-dev] Comments on API for Matlab's
 eigs	equivalent	(computing a few eigenvalues only)
In-Reply-To: <4B6A6E06.7060102@silveregg.co.jp>
References: <4B6A68ED.6070007@silveregg.co.jp>	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
Message-ID: <4B6A8154.2040705@student.matnat.uio.no>

David Cournapeau wrote:
> Charles R Harris wrote:
>>
>> On Wed, Feb 3, 2010 at 11:27 PM, David Cournapeau <david at silveregg.co.jp 
>> <mailto:david at silveregg.co.jp>> wrote:
>>
>>     Hi,
>>
>>     I have played a bit with adding support for computing a few eigenvalues
>>     of full symmetric matrices. I would like some comments on the
>>     current API:
>>
>>     import numpy as np
>>     from scipy.linalg import eigs
>>     x = np.random.randn(10, 10)
>>     x = np.dot(x.T, x)
>>     # Retrieve the 3 biggest eigenvalues
>>     eigs(x, 3)[0]
>>     # Retrieve the 3 smallest eigenvalues
>>     eigs(x, -3)[0]
>>     # Retrieve the 3 biggest eigenvalues
>>     eigs(x, [0, 3], mode="index")[0]
>>     # Retrieve the 2nd and 3rd biggest
>>     eigs(x, [1, 3], mode="index")[0]
>>     # Retrieve all the eigenvalues in the range [1.5, 3.5[
>>     eigs(x, [1.5, 3.5], mode="range")[0]
>>
>>     One thing which does not feel right is that that the range in the
>>     "index" mode is exactly inverted compared to the output (i.e. if you ask
>>
>>
>> Why not use separate keywords for index and range.
> 
> Then I am not sure how to handle the "give me the k biggest/smallest" 
> case, which is the most common I think (that's the only one I care 
> personally :) ).
> 
> Maybe this just warrants several functions ? But then there is the issue 
> of naming (supporting non-symmetric matrices would be nice, but requires 
> a totally different implementation, as LAPACK does not support it AFAIK 
> - the easiest way would be to use ARPACK ATM).
> 
>> Changing the meaning 
>> of an argument using another keyword is just weird. 
> 
> Agreed - this is just the best I could came up with one function to 
> support all cases while keeping the common case simple. But I feel a bit 
> like I outsmarted myself here.
> 
>> Why the index on the end? Is the return a list?
> 
> Yes, it also returns eigenvectors
> 
>>     for the range [0, 3], you get the last three items from what you would
>>     get if you asked for the full range [0, 10]), but this is because I kept
>>     compatibility with Octave (always showing from biggest to smallest). It
>>
>>
>> Why follow Octave, isn't Octave like Matlab? Matlab functions are a mess.
> 
> Actually, there is another reason I forgot to mention: that's how eigen 
> in scipy.sparse does as well (in decreasing order). I think that the 
> ARPACK wrappers are not that well thought out, though.

I feel there's also a certain convention of listing the largest 
eigenvalues first; "let \lambda_i be the eigenvalues...assume \lambda_2 
< \lambda_1 ...".


I know I'd assume that the largest eigenvalue came out first, and I've 
barely used either MATLAB or Octave.

(Also with SVD one would typically put the largest singular values 
first, and so on.)

Just a data-point, I don't really care either way...

-- 
Dag Sverre


From sturla at molden.no  Thu Feb  4 05:40:52 2010
From: sturla at molden.no (Sturla Molden)
Date: Thu, 4 Feb 2010 11:40:52 +0100
Subject: [SciPy-dev] compilation with fort77
In-Reply-To: <4B5CEA49.3040604@silveregg.co.jp>
References: <4B5A054C.7060308@gmx.de> <4B5CEA49.3040604@silveregg.co.jp>
Message-ID: <A37BDEC0-ABFA-4601-A473-B0322055AB66@molden.no>


>>
>>
>>
>> the problem is that fort77 cannot deal with those variable sized
>> input arguments:
>>
>>      SUBROUTINE mvnun(d, n, lower, upper, means, covar, maxpts,
>>     &                   abseps, releps, value, inform)
>> ...
>>      integer n, d, infin(d), maxpts, inform, tmpinf
>>      double precision lower(d), upper(d), releps, abseps,
>>     &                 error, value, stdev(d), rho(d*(d-1)/2),
>>     &                 covar(d,d),
>>     &                 nlower(d), nupper(d), means(d,n), tmpval
>>      integer i, j
>>
>> Could some Fortran expert please help me to make it fort77  
>> compatible?

The code posted here (I haven't looked in SVN) looks like valid  
Fortran 77 to me. Dummy arguments can have variable size. f2c might  
not accept it because variable-size arrays are not supported in C89,  
and the Fortran to C conversion is rather primitive.

Sturla





>
> This file is using some F90/F95 - I am not sure what the policy is on
> this point, but don't think we want to guarantee that we will never  
> use
> fortran > F77.
>
> Can't you get gfortran running on your platform ? Cross-compiling
> gfortran  for your platform should not be difficult
>
> David
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev


From pav+sp at iki.fi  Thu Feb  4 06:30:25 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Thu, 4 Feb 2010 11:30:25 +0000 (UTC)
Subject: [SciPy-dev] compilation with fort77
References: <4B5A054C.7060308@gmx.de> <4B5CEA49.3040604@silveregg.co.jp>
	<A37BDEC0-ABFA-4601-A473-B0322055AB66@molden.no>
Message-ID: <hkeb4g$mh5$1@ger.gmane.org>

Thu, 04 Feb 2010 11:40:52 +0100, Sturla Molden wrote:
>>> the problem is that fort77 cannot deal with those variable sized input
>>> arguments:
>>>
>>>      SUBROUTINE mvnun(d, n, lower, upper, means, covar, maxpts,
>>>     &                   abseps, releps, value, inform)
>>> ...
>>>      integer n, d, infin(d), maxpts, inform, tmpinf double precision
>>>      lower(d), upper(d), releps, abseps,
>>>     &                 error, value, stdev(d), rho(d*(d-1)/2), &       
>>>              covar(d,d),
>>>     &                 nlower(d), nupper(d), means(d,n), tmpval
>>>      integer i, j
>>>
>>> Could some Fortran expert please help me to make it fort77 compatible?
> 
> The code posted here (I haven't looked in SVN) looks like valid Fortran
> 77 to me. Dummy arguments can have variable size. f2c might not accept
> it because variable-size arrays are not supported in C89, and the
> Fortran to C conversion is rather primitive.

Note that infin, stdev, rho, nlower, and nupper are not dummy variables, 
so it's not valid F77.

-- 
Pauli Virtanen



From dwf at cs.toronto.edu  Thu Feb  4 14:31:14 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Thu, 4 Feb 2010 14:31:14 -0500
Subject: [SciPy-dev] Comments on API for Matlab's
	eigs	equivalent	(computing a few eigenvalues only)
In-Reply-To: <4B6A8154.2040705@student.matnat.uio.no>
References: <4B6A68ED.6070007@silveregg.co.jp>	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
Message-ID: <750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>

On 4-Feb-10, at 3:12 AM, Dag Sverre Seljebotn wrote:

> I feel there's also a certain convention of listing the largest
> eigenvalues first; "let \lambda_i be the eigenvalues...assume  
> \lambda_2
> < \lambda_1 ...".

+1, though I've seen them listed from smallest to largest as well.

David


From rob.clewley at gmail.com  Thu Feb  4 15:01:56 2010
From: rob.clewley at gmail.com (Rob Clewley)
Date: Thu, 4 Feb 2010 15:01:56 -0500
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
Message-ID: <a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>

On Thu, Feb 4, 2010 at 2:31 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> On 4-Feb-10, at 3:12 AM, Dag Sverre Seljebotn wrote:
>
>> I feel there's also a certain convention of listing the largest
>> eigenvalues first; "let \lambda_i be the eigenvalues...assume
>> \lambda_2
>> < \lambda_1 ...".
>
> +1, though I've seen them listed from smallest to largest as well.
>
> David

I'm +1 too, but are these going to be ordered by norm of the
eigenvalue if they are complex, or by the real part only? I suggest by
norm, and the docstring needs to make the choice clear. This
functionality has to be consistent for all uses of eig()!

-Rob


From peridot.faceted at gmail.com  Thu Feb  4 15:18:02 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 4 Feb 2010 15:18:02 -0500
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
Message-ID: <ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>

On 4 February 2010 15:01, Rob Clewley <rob.clewley at gmail.com> wrote:
> On Thu, Feb 4, 2010 at 2:31 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>> On 4-Feb-10, at 3:12 AM, Dag Sverre Seljebotn wrote:
>>
>>> I feel there's also a certain convention of listing the largest
>>> eigenvalues first; "let \lambda_i be the eigenvalues...assume
>>> \lambda_2
>>> < \lambda_1 ...".
>>
>> +1, though I've seen them listed from smallest to largest as well.
>>
>> David
>
> I'm +1 too, but are these going to be ordered by norm of the
> eigenvalue if they are complex, or by the real part only? I suggest by
> norm, and the docstring needs to make the choice clear. This
> functionality has to be consistent for all uses of eig()!

I have to say using the norm for complex eigenvalues is asking for
trouble, since it puts -2 between 1 and 3. Not that this is relevant
for a symmetric (Hermitian) eigensolver, which will always have real
eigenvalues.

Anne


From rob.clewley at gmail.com  Thu Feb  4 19:39:37 2010
From: rob.clewley at gmail.com (Rob Clewley)
Date: Thu, 4 Feb 2010 19:39:37 -0500
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
Message-ID: <a749952d1002041639v34ea66fgbf1e090d0b5a8703@mail.gmail.com>

> Not that this is relevant
> for a symmetric (Hermitian) eigensolver, which will always have real
> eigenvalues.
>
> Anne

Yep, sorry I failed to notice this wasn't an extension to eig, but
about a new function eigs only for symmetric matrices. My bad.


From warren.weckesser at enthought.com  Thu Feb  4 19:43:46 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Thu, 04 Feb 2010 18:43:46 -0600
Subject: [SciPy-dev] Comments on API for Matlab's eigs
 equivalent	(computing a few eigenvalues only)
In-Reply-To: <ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>	<4B6A6E06.7060102@silveregg.co.jp>	<4B6A8154.2040705@student.matnat.uio.no>	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
Message-ID: <4B6B69C2.7000907@enthought.com>

Anne Archibald wrote:
> On 4 February 2010 15:01, Rob Clewley <rob.clewley at gmail.com> wrote:
>   
>> On Thu, Feb 4, 2010 at 2:31 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>>     
>>> On 4-Feb-10, at 3:12 AM, Dag Sverre Seljebotn wrote:
>>>
>>>       
>>>> I feel there's also a certain convention of listing the largest
>>>> eigenvalues first; "let \lambda_i be the eigenvalues...assume
>>>> \lambda_2
>>>> < \lambda_1 ...".
>>>>         
>>> +1, though I've seen them listed from smallest to largest as well.
>>>
>>> David
>>>       
>> I'm +1 too, but are these going to be ordered by norm of the
>> eigenvalue if they are complex, or by the real part only? I suggest by
>> norm, and the docstring needs to make the choice clear. This
>> functionality has to be consistent for all uses of eig()!
>>     
>
> I have to say using the norm for complex eigenvalues is asking for
> trouble, since it puts -2 between 1 and 3. Not that this is relevant
> for a symmetric (Hermitian) eigensolver, which will always have real
> eigenvalues.
>
> Anne
>   

David C,

You said this was for symmetric matrices, but do you envision later 
allowing nonsymmetric matrices?

If not, then perhaps the name of the function should be 'eigsh', 
following the precedent set by numpy.linalg and scipy.linalg.

On the other hand, if the intent is to eventually handle nonsymmetric 
matrices, then it would be nice to provide an API that is as flexible as 
(but definitetly not the same as) Matlab's eigs function:

    http://www.mathworks.com/access/helpdesk/help/techdoc/ref/eigs.html

In particular, the choice of ordering by magnitude or by real part is 
convenient.

Warren


> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   



From cournape at gmail.com  Thu Feb  4 20:27:07 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 5 Feb 2010 10:27:07 +0900
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <4B6B69C2.7000907@enthought.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
	<4B6B69C2.7000907@enthought.com>
Message-ID: <5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>

On Fri, Feb 5, 2010 at 9:43 AM, Warren Weckesser
<warren.weckesser at enthought.com> wrote:

> David C,
>
> You said this was for symmetric matrices, but do you envision later
> allowing nonsymmetric matrices?

Yes. I have only implemented symmetric/Hermitian because that's the
only solver that handles getting only a few eigenvalues in LAPACK.
Matlab own eigs function uses ARPACK for both symmetric/unsymmetric
cases, which is not good according to one of the Lapack developer
(http://mail.scipy.org/pipermail/scipy-dev/2007-March/006778.html).
But we could use ARPACK for the general case in eigs.

>
> If not, then perhaps the name of the function should be 'eigsh',
> following the precedent set by numpy.linalg and scipy.linalg.

I wonder whether those eigh functions are a good idea: I fear that
most people will always use eig - maybe one could use the underlying
eig*h solver in eig* if the matrix is detected as being symmetric ? I
am not really knowledgeable about those issues, though. For example, I
don't know whether the symmetric aspect should be checked exactly, or
if it is better to use a symmetric solver  even if there are very
small errors in say A'-A.

>
> In particular, the choice of ordering by magnitude or by real part is
> convenient.

It seems that one would need to implement the non-symmetric
capabilities to sort this out. I fear that those options are solver
specific, though - maybe the solution is to have two levels of API,
one low-level and one high level.

cheers,

David


From josef.pktd at gmail.com  Thu Feb  4 21:24:41 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 4 Feb 2010 21:24:41 -0500
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
	<4B6B69C2.7000907@enthought.com>
	<5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>
Message-ID: <1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>

On Thu, Feb 4, 2010 at 8:27 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Fri, Feb 5, 2010 at 9:43 AM, Warren Weckesser
> <warren.weckesser at enthought.com> wrote:
>
>> David C,
>>
>> You said this was for symmetric matrices, but do you envision later
>> allowing nonsymmetric matrices?
>
> Yes. I have only implemented symmetric/Hermitian because that's the
> only solver that handles getting only a few eigenvalues in LAPACK.
> Matlab own eigs function uses ARPACK for both symmetric/unsymmetric
> cases, which is not good according to one of the Lapack developer
> (http://mail.scipy.org/pipermail/scipy-dev/2007-March/006778.html).
> But we could use ARPACK for the general case in eigs.
>
>>
>> If not, then perhaps the name of the function should be 'eigsh',
>> following the precedent set by numpy.linalg and scipy.linalg.
>
> I wonder whether those eigh functions are a good idea: I fear that
> most people will always use eig - maybe one could use the underlying
> eig*h solver in eig* if the matrix is detected as being symmetric ? I
> am not really knowledgeable about those issues, though. For example, I
> don't know whether the symmetric aspect should be checked exactly, or
> if it is better to use a symmetric solver ?even if there are very
> small errors in say A'-A.
>
>>
>> In particular, the choice of ordering by magnitude or by real part is
>> convenient.
>
> It seems that one would need to implement the non-symmetric
> capabilities to sort this out. I fear that those options are solver
> specific, though - maybe the solution is to have two levels of API,
> one low-level and one high level.
>
> cheers,
>
> David
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>

the current version of scipy.linalg.eigh sorts from smallest to largest

>>> scipy.linalg.eigh(x)[0]
array([  4.04457427e-03,   1.84073286e-01,   6.74875960e-01,
         3.23328824e+00,   4.00741304e+00,   6.98333680e+00,
         1.45314842e+01,   1.49260377e+01,   2.47166702e+01,
         2.98955755e+01])
>>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
array([ 0.00404457,  0.18407329,  0.67487596,  3.23328824])
>>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
array([ 14.92603769,  24.71667021,  29.89557547])


>>> x = np.random.randn(10, 10) + 1j*np.random.randn(10, 10)
>>> x = np.dot(x.T, x)
>>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
array([-36.82039662, -18.20362967, -12.21716065,  -9.79752274])
>>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
array([ 14.44815917,  28.8394026 ,  42.45471058])

Josef


From charlesr.harris at gmail.com  Thu Feb  4 21:54:21 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 4 Feb 2010 19:54:21 -0700
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
	<4B6B69C2.7000907@enthought.com>
	<5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>
	<1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>
Message-ID: <e06186141002041854g1fc9c052t7a11a22c2c90dbed@mail.gmail.com>

On Thu, Feb 4, 2010 at 7:24 PM, <josef.pktd at gmail.com> wrote:

> On Thu, Feb 4, 2010 at 8:27 PM, David Cournapeau <cournape at gmail.com>
> wrote:
> > On Fri, Feb 5, 2010 at 9:43 AM, Warren Weckesser
> > <warren.weckesser at enthought.com> wrote:
> >
> >> David C,
> >>
> >> You said this was for symmetric matrices, but do you envision later
> >> allowing nonsymmetric matrices?
> >
> > Yes. I have only implemented symmetric/Hermitian because that's the
> > only solver that handles getting only a few eigenvalues in LAPACK.
> > Matlab own eigs function uses ARPACK for both symmetric/unsymmetric
> > cases, which is not good according to one of the Lapack developer
> > (http://mail.scipy.org/pipermail/scipy-dev/2007-March/006778.html).
> > But we could use ARPACK for the general case in eigs.
> >
> >>
> >> If not, then perhaps the name of the function should be 'eigsh',
> >> following the precedent set by numpy.linalg and scipy.linalg.
> >
> > I wonder whether those eigh functions are a good idea: I fear that
> > most people will always use eig - maybe one could use the underlying
> > eig*h solver in eig* if the matrix is detected as being symmetric ? I
> > am not really knowledgeable about those issues, though. For example, I
> > don't know whether the symmetric aspect should be checked exactly, or
> > if it is better to use a symmetric solver  even if there are very
> > small errors in say A'-A.
> >
> >>
> >> In particular, the choice of ordering by magnitude or by real part is
> >> convenient.
> >
> > It seems that one would need to implement the non-symmetric
> > capabilities to sort this out. I fear that those options are solver
> > specific, though - maybe the solution is to have two levels of API,
> > one low-level and one high level.
> >
> > cheers,
> >
> > David
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-dev
> >
>
> the current version of scipy.linalg.eigh sorts from smallest to largest
>
> >>> scipy.linalg.eigh(x)[0]
> array([  4.04457427e-03,   1.84073286e-01,   6.74875960e-01,
>         3.23328824e+00,   4.00741304e+00,   6.98333680e+00,
>         1.45314842e+01,   1.49260377e+01,   2.47166702e+01,
>         2.98955755e+01])
> >>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
> array([ 0.00404457,  0.18407329,  0.67487596,  3.23328824])
> >>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
> array([ 14.92603769,  24.71667021,  29.89557547])
>
>
> >>> x = np.random.randn(10, 10) + 1j*np.random.randn(10, 10)
> >>> x = np.dot(x.T, x)
> >>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
> array([-36.82039662, -18.20362967, -12.21716065,  -9.79752274])
> >>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
> array([ 14.44815917,  28.8394026 ,  42.45471058])
>
>
Yeah. It's my fault ;) It didn't use to sort at all.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100204/eddc9873/attachment.html>

From josef.pktd at gmail.com  Thu Feb  4 22:08:42 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 4 Feb 2010 22:08:42 -0500
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <e06186141002041854g1fc9c052t7a11a22c2c90dbed@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
	<4B6B69C2.7000907@enthought.com>
	<5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>
	<1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>
	<e06186141002041854g1fc9c052t7a11a22c2c90dbed@mail.gmail.com>
Message-ID: <1cd32cbb1002041908i42e37c28sd4e54ee1d2bedadb@mail.gmail.com>

On Thu, Feb 4, 2010 at 9:54 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Thu, Feb 4, 2010 at 7:24 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Thu, Feb 4, 2010 at 8:27 PM, David Cournapeau <cournape at gmail.com>
>> wrote:
>> > On Fri, Feb 5, 2010 at 9:43 AM, Warren Weckesser
>> > <warren.weckesser at enthought.com> wrote:
>> >
>> >> David C,
>> >>
>> >> You said this was for symmetric matrices, but do you envision later
>> >> allowing nonsymmetric matrices?
>> >
>> > Yes. I have only implemented symmetric/Hermitian because that's the
>> > only solver that handles getting only a few eigenvalues in LAPACK.
>> > Matlab own eigs function uses ARPACK for both symmetric/unsymmetric
>> > cases, which is not good according to one of the Lapack developer
>> > (http://mail.scipy.org/pipermail/scipy-dev/2007-March/006778.html).
>> > But we could use ARPACK for the general case in eigs.
>> >
>> >>
>> >> If not, then perhaps the name of the function should be 'eigsh',
>> >> following the precedent set by numpy.linalg and scipy.linalg.
>> >
>> > I wonder whether those eigh functions are a good idea: I fear that
>> > most people will always use eig - maybe one could use the underlying
>> > eig*h solver in eig* if the matrix is detected as being symmetric ? I
>> > am not really knowledgeable about those issues, though. For example, I
>> > don't know whether the symmetric aspect should be checked exactly, or
>> > if it is better to use a symmetric solver ?even if there are very
>> > small errors in say A'-A.
>> >
>> >>
>> >> In particular, the choice of ordering by magnitude or by real part is
>> >> convenient.
>> >
>> > It seems that one would need to implement the non-symmetric
>> > capabilities to sort this out. I fear that those options are solver
>> > specific, though - maybe the solution is to have two levels of API,
>> > one low-level and one high level.
>> >
>> > cheers,
>> >
>> > David
>> > _______________________________________________
>> > SciPy-Dev mailing list
>> > SciPy-Dev at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-dev
>> >
>>
>> the current version of scipy.linalg.eigh sorts from smallest to largest
>>
>> >>> scipy.linalg.eigh(x)[0]
>> array([ ?4.04457427e-03, ? 1.84073286e-01, ? 6.74875960e-01,
>> ? ? ? ? 3.23328824e+00, ? 4.00741304e+00, ? 6.98333680e+00,
>> ? ? ? ? 1.45314842e+01, ? 1.49260377e+01, ? 2.47166702e+01,
>> ? ? ? ? 2.98955755e+01])
>> >>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
>> array([ 0.00404457, ?0.18407329, ?0.67487596, ?3.23328824])
>> >>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
>> array([ 14.92603769, ?24.71667021, ?29.89557547])
>>
>>
>> >>> x = np.random.randn(10, 10) + 1j*np.random.randn(10, 10)
>> >>> x = np.dot(x.T, x)
>> >>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
>> array([-36.82039662, -18.20362967, -12.21716065, ?-9.79752274])
>> >>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
>> array([ 14.44815917, ?28.8394026 , ?42.45471058])
>>
>
> Yeah. It's my fault ;) It didn't use to sort at all.

I thought that's how Tiziano added them, I didn't see a sort when I
briefly browsed the source.
But I'm looking at lapack only from the outside.

Josef

>
> Chuck
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>


From david at silveregg.co.jp  Thu Feb  4 22:45:42 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Fri, 05 Feb 2010 12:45:42 +0900
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
Message-ID: <4B6B9466.4010507@silveregg.co.jp>

Hi,

I wanted to know if there was a rationale for using svd to 
orthonormalize the columns of a matrix (in scipy.linalg). QR-based 
methods are likely to be much faster, and I thought this was the 
standard, numerically-stable method to orthonormalize a basis ? If the 
reason is to deal with rank-deficient matrices, maybe we could add an 
option to choose between them ?

cheers,

David


From charlesr.harris at gmail.com  Fri Feb  5 00:37:55 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 4 Feb 2010 22:37:55 -0700
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <1cd32cbb1002041908i42e37c28sd4e54ee1d2bedadb@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
	<4B6B69C2.7000907@enthought.com>
	<5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>
	<1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>
	<e06186141002041854g1fc9c052t7a11a22c2c90dbed@mail.gmail.com>
	<1cd32cbb1002041908i42e37c28sd4e54ee1d2bedadb@mail.gmail.com>
Message-ID: <e06186141002042137y5c272f39g1e073116819d14a8@mail.gmail.com>

On Thu, Feb 4, 2010 at 8:08 PM, <josef.pktd at gmail.com> wrote:

> On Thu, Feb 4, 2010 at 9:54 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
> >
> >
> > On Thu, Feb 4, 2010 at 7:24 PM, <josef.pktd at gmail.com> wrote:
> >>
> >> On Thu, Feb 4, 2010 at 8:27 PM, David Cournapeau <cournape at gmail.com>
> >> wrote:
> >> > On Fri, Feb 5, 2010 at 9:43 AM, Warren Weckesser
> >> > <warren.weckesser at enthought.com> wrote:
> >> >
> >> >> David C,
> >> >>
> >> >> You said this was for symmetric matrices, but do you envision later
> >> >> allowing nonsymmetric matrices?
> >> >
> >> > Yes. I have only implemented symmetric/Hermitian because that's the
> >> > only solver that handles getting only a few eigenvalues in LAPACK.
> >> > Matlab own eigs function uses ARPACK for both symmetric/unsymmetric
> >> > cases, which is not good according to one of the Lapack developer
> >> > (http://mail.scipy.org/pipermail/scipy-dev/2007-March/006778.html).
> >> > But we could use ARPACK for the general case in eigs.
> >> >
> >> >>
> >> >> If not, then perhaps the name of the function should be 'eigsh',
> >> >> following the precedent set by numpy.linalg and scipy.linalg.
> >> >
> >> > I wonder whether those eigh functions are a good idea: I fear that
> >> > most people will always use eig - maybe one could use the underlying
> >> > eig*h solver in eig* if the matrix is detected as being symmetric ? I
> >> > am not really knowledgeable about those issues, though. For example, I
> >> > don't know whether the symmetric aspect should be checked exactly, or
> >> > if it is better to use a symmetric solver  even if there are very
> >> > small errors in say A'-A.
> >> >
> >> >>
> >> >> In particular, the choice of ordering by magnitude or by real part is
> >> >> convenient.
> >> >
> >> > It seems that one would need to implement the non-symmetric
> >> > capabilities to sort this out. I fear that those options are solver
> >> > specific, though - maybe the solution is to have two levels of API,
> >> > one low-level and one high level.
> >> >
> >> > cheers,
> >> >
> >> > David
> >> > _______________________________________________
> >> > SciPy-Dev mailing list
> >> > SciPy-Dev at scipy.org
> >> > http://mail.scipy.org/mailman/listinfo/scipy-dev
> >> >
> >>
> >> the current version of scipy.linalg.eigh sorts from smallest to largest
> >>
> >> >>> scipy.linalg.eigh(x)[0]
> >> array([  4.04457427e-03,   1.84073286e-01,   6.74875960e-01,
> >>         3.23328824e+00,   4.00741304e+00,   6.98333680e+00,
> >>         1.45314842e+01,   1.49260377e+01,   2.47166702e+01,
> >>         2.98955755e+01])
> >> >>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
> >> array([ 0.00404457,  0.18407329,  0.67487596,  3.23328824])
> >> >>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
> >> array([ 14.92603769,  24.71667021,  29.89557547])
> >>
> >>
> >> >>> x = np.random.randn(10, 10) + 1j*np.random.randn(10, 10)
> >> >>> x = np.dot(x.T, x)
> >> >>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
> >> array([-36.82039662, -18.20362967, -12.21716065,  -9.79752274])
> >> >>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
> >> array([ 14.44815917,  28.8394026 ,  42.45471058])
> >>
> >
> > Yeah. It's my fault ;) It didn't use to sort at all.
>
> I thought that's how Tiziano added them, I didn't see a sort when I
> briefly browsed the source.
> But I'm looking at lapack only from the outside.
>
>
I was thinking of the numpy version. But now that I think about it, it was a
test that needed the eigenvalues sorted because the routine didn't sort
them. I seem to be developing one of those trick memories...

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100204/79b81c78/attachment.html>

From charlesr.harris at gmail.com  Fri Feb  5 00:58:23 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 4 Feb 2010 22:58:23 -0700
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <4B6B9466.4010507@silveregg.co.jp>
References: <4B6B9466.4010507@silveregg.co.jp>
Message-ID: <e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>

On Thu, Feb 4, 2010 at 8:45 PM, David Cournapeau <david at silveregg.co.jp>wrote:

> Hi,
>
> I wanted to know if there was a rationale for using svd to
> orthonormalize the columns of a matrix (in scipy.linalg). QR-based
> methods are likely to be much faster, and I thought this was the
> standard, numerically-stable method to orthonormalize a basis ? If the
> reason is to deal with rank-deficient matrices, maybe we could add an
> option to choose between them ?
>
>
QR with column rotation would deal with rank-deficient matrices and routines
for that are available in LAPACK <http://netlib.org/lapack/lug/node42.html>.
The SVD was probably used because it was available. The diagonal elements of
the R matrix can somewhat take the place of the singular values when column
rotation is used.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100204/6aa7c58a/attachment.html>

From d.l.goldsmith at gmail.com  Fri Feb  5 02:04:26 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Thu, 4 Feb 2010 23:04:26 -0800
Subject: [SciPy-dev] WOT: Experiences running 64 bit Vista on a 32 bit
	machine
Message-ID: <45d1ab481002042304o375b7c69rb79d1af5b2de37d9@mail.gmail.com>

Please forgive the widely OT request for input: I'm thinking about trying to
run 64 bit Vista on my 32 bit machine (Pentium Dual-Core T4300 @ 2x2.10Ghz);
following
http://windows.microsoft.com/en-US/windows-vista/32-bit-and-64-bit-Windows-frequently-asked-questions#How-do-I-tell...
I've confirmed that I'm "64 bit capable."  Has anyone reading this far ;-)
had a particularly positive/negative experience doing this?  Also, how do I
determine the 64 bit linux capability of my hardware?  Thanks!

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100204/ec661ba8/attachment.html>

From david at silveregg.co.jp  Fri Feb  5 02:15:50 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Fri, 05 Feb 2010 16:15:50 +0900
Subject: [SciPy-dev] WOT: Experiences running 64 bit Vista on a 32 bit
	machine
In-Reply-To: <45d1ab481002042304o375b7c69rb79d1af5b2de37d9@mail.gmail.com>
References: <45d1ab481002042304o375b7c69rb79d1af5b2de37d9@mail.gmail.com>
Message-ID: <4B6BC5A6.1000006@silveregg.co.jp>

David Goldsmith wrote:
> Please forgive the widely OT request for input: I'm thinking about 
> trying to run 64 bit Vista on my 32 bit machine (Pentium Dual-Core T4300 
> @ 2x2.10Ghz); following 
> http://windows.microsoft.com/en-US/windows-vista/32-bit-and-64-bit-Windows-frequently-asked-questions#How-do-I-tell... 
> I've confirmed that I'm "64 bit capable."  Has anyone reading this far 
> ;-) had a particularly positive/negative experience doing this?  Also, 
> how do I determine the 64 bit linux capability of my hardware?

If windows 64 runs, linux 64 will as well; as far as OS are concerned, 
64 bits mode is like a new architecture (like ppc vs intel). When the PC 
starts, the kernel of whatever OS you are running will refuse to run if 
it is 64 bits and you run on a 32 bits machine. The details are OS/Boot 
loader dependent, but most installers will refuse to install a 64 bits 
installers on a 32bits-only machine anyway. You can also run a 64 bits 
VM inside a 32 bits, assuming your CPU is 64 bits capable (so you can 
run 64 bits Ubuntu in vmware on top of 32 bits Ubuntu for example).

I strongly suggest using windows 7 instead of Vista if you can.

cheers,

David


From david at silveregg.co.jp  Fri Feb  5 02:18:02 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Fri, 05 Feb 2010 16:18:02 +0900
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
Message-ID: <4B6BC62A.8090109@silveregg.co.jp>

Charles R Harris wrote:
> 
> 
> On Thu, Feb 4, 2010 at 8:45 PM, David Cournapeau <david at silveregg.co.jp 
> <mailto:david at silveregg.co.jp>> wrote:
> 
>     Hi,
> 
>     I wanted to know if there was a rationale for using svd to
>     orthonormalize the columns of a matrix (in scipy.linalg). QR-based
>     methods are likely to be much faster, and I thought this was the
>     standard, numerically-stable method to orthonormalize a basis ? If the
>     reason is to deal with rank-deficient matrices, maybe we could add an
>     option to choose between them ?
> 
> 
> QR with column rotation would deal with rank-deficient matrices and 
> routines for that are available in LAPACK 
> <http://netlib.org/lapack/lug/node42.html>. The SVD was probably used 
> because it was available. The diagonal elements of the R matrix can 
> somewhat take the place of the singular values when column rotation is used.

So would be it ok to use this column-rotated QR in place of svd for 
every case in orth ? I would have to check that QR with column rotation 
is still significantly faster than svd, but I would surprised if if were 
not the case. QR has also the advantage of being implemented in PLASMA 
already contrary to eigen/svd solvers,

cheers,

David


From charlesr.harris at gmail.com  Fri Feb  5 02:47:09 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Feb 2010 00:47:09 -0700
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <4B6BC62A.8090109@silveregg.co.jp>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
Message-ID: <e06186141002042347x74d585dcu3e45837242cd8a90@mail.gmail.com>

On Fri, Feb 5, 2010 at 12:18 AM, David Cournapeau <david at silveregg.co.jp>wrote:

> Charles R Harris wrote:
> >
> >
> > On Thu, Feb 4, 2010 at 8:45 PM, David Cournapeau <david at silveregg.co.jp
> > <mailto:david at silveregg.co.jp>> wrote:
> >
> >     Hi,
> >
> >     I wanted to know if there was a rationale for using svd to
> >     orthonormalize the columns of a matrix (in scipy.linalg). QR-based
> >     methods are likely to be much faster, and I thought this was the
> >     standard, numerically-stable method to orthonormalize a basis ? If
> the
> >     reason is to deal with rank-deficient matrices, maybe we could add an
> >     option to choose between them ?
> >
> >
> > QR with column rotation would deal with rank-deficient matrices and
> > routines for that are available in LAPACK
> > <http://netlib.org/lapack/lug/node42.html>. The SVD was probably used
> > because it was available. The diagonal elements of the R matrix can
> > somewhat take the place of the singular values when column rotation is
> used.
>
> So would be it ok to use this column-rotated QR in place of svd for
> every case in orth ? I would have to check that QR with column rotation
> is still significantly faster than svd, but I would surprised if if were
> not the case. QR has also the advantage of being implemented in PLASMA
> already contrary to eigen/svd solvers,
>
>
I don't know how the two methods compare in practice. SVD algorithms
generally use iterated QR reductions in their implementation, so QR
reductions can't be worse numerically. But the SVD probably provides a
better metric for rank determination. A google search turns up some
literature on the subject that I can't access from home.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/14205d07/attachment.html>

From gael.varoquaux at normalesup.org  Fri Feb  5 03:10:12 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 5 Feb 2010 09:10:12 +0100
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <4B6BC62A.8090109@silveregg.co.jp>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
Message-ID: <20100205081012.GA20893@phare.normalesup.org>

On Fri, Feb 05, 2010 at 04:18:02PM +0900, David Cournapeau wrote:
> So would be it ok to use this column-rotated QR in place of svd for 
> every case in orth ? I would have to check that QR with column rotation 
> is still significantly faster than svd, but I would surprised if if were 
> not the case. QR has also the advantage of being implemented in PLASMA 
> already contrary to eigen/svd solvers,

Out of curiosity, what's PLASMA, in this context?

Ga?l


From charlesr.harris at gmail.com  Fri Feb  5 03:12:06 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Feb 2010 01:12:06 -0700
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <e06186141002042347x74d585dcu3e45837242cd8a90@mail.gmail.com>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<e06186141002042347x74d585dcu3e45837242cd8a90@mail.gmail.com>
Message-ID: <e06186141002050012m61754875rfc0bd9dfa1eaa8ea@mail.gmail.com>

On Fri, Feb 5, 2010 at 12:47 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Fri, Feb 5, 2010 at 12:18 AM, David Cournapeau <david at silveregg.co.jp>wrote:
>
>> Charles R Harris wrote:
>> >
>> >
>> > On Thu, Feb 4, 2010 at 8:45 PM, David Cournapeau <david at silveregg.co.jp
>> > <mailto:david at silveregg.co.jp>> wrote:
>> >
>> >     Hi,
>> >
>> >     I wanted to know if there was a rationale for using svd to
>> >     orthonormalize the columns of a matrix (in scipy.linalg). QR-based
>> >     methods are likely to be much faster, and I thought this was the
>> >     standard, numerically-stable method to orthonormalize a basis ? If
>> the
>> >     reason is to deal with rank-deficient matrices, maybe we could add
>> an
>> >     option to choose between them ?
>> >
>> >
>> > QR with column rotation would deal with rank-deficient matrices and
>> > routines for that are available in LAPACK
>> > <http://netlib.org/lapack/lug/node42.html>. The SVD was probably used
>> > because it was available. The diagonal elements of the R matrix can
>> > somewhat take the place of the singular values when column rotation is
>> used.
>>
>> So would be it ok to use this column-rotated QR in place of svd for
>> every case in orth ? I would have to check that QR with column rotation
>> is still significantly faster than svd, but I would surprised if if were
>> not the case. QR has also the advantage of being implemented in PLASMA
>> already contrary to eigen/svd solvers,
>>
>>
> I don't know how the two methods compare in practice. SVD algorithms
> generally use iterated QR reductions in their implementation, so QR
> reductions can't be worse numerically. But the SVD probably provides a
> better metric for rank determination. A google search turns up some
> literature on the subject that I can't access from home.
>
>
OK, here's a good
reference<http://www.math.sjsu.edu/%7Efoster/rank/rank_revealing_s.pdf>.
A quick look seems to indicate that the  SVD is the way to go.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/76b38e9b/attachment.html>

From david at silveregg.co.jp  Fri Feb  5 03:29:51 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Fri, 05 Feb 2010 17:29:51 +0900
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <e06186141002050012m61754875rfc0bd9dfa1eaa8ea@mail.gmail.com>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<e06186141002042347x74d585dcu3e45837242cd8a90@mail.gmail.com>
	<e06186141002050012m61754875rfc0bd9dfa1eaa8ea@mail.gmail.com>
Message-ID: <4B6BD6FF.6070201@silveregg.co.jp>

Charles R Harris wrote:
> 
> 
> On Fri, Feb 5, 2010 at 12:47 AM, Charles R Harris 
> <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
> 
> 
> 
>     On Fri, Feb 5, 2010 at 12:18 AM, David Cournapeau
>     <david at silveregg.co.jp <mailto:david at silveregg.co.jp>> wrote:
> 
>         Charles R Harris wrote:
>          >
>          >
>          > On Thu, Feb 4, 2010 at 8:45 PM, David Cournapeau
>         <david at silveregg.co.jp <mailto:david at silveregg.co.jp>
>          > <mailto:david at silveregg.co.jp
>         <mailto:david at silveregg.co.jp>>> wrote:
>          >
>          >     Hi,
>          >
>          >     I wanted to know if there was a rationale for using svd to
>          >     orthonormalize the columns of a matrix (in scipy.linalg).
>         QR-based
>          >     methods are likely to be much faster, and I thought this
>         was the
>          >     standard, numerically-stable method to orthonormalize a
>         basis ? If the
>          >     reason is to deal with rank-deficient matrices, maybe we
>         could add an
>          >     option to choose between them ?
>          >
>          >
>          > QR with column rotation would deal with rank-deficient
>         matrices and
>          > routines for that are available in LAPACK
>          > <http://netlib.org/lapack/lug/node42.html>. The SVD was
>         probably used
>          > because it was available. The diagonal elements of the R
>         matrix can
>          > somewhat take the place of the singular values when column
>         rotation is used.
> 
>         So would be it ok to use this column-rotated QR in place of svd for
>         every case in orth ? I would have to check that QR with column
>         rotation
>         is still significantly faster than svd, but I would surprised if
>         if were
>         not the case. QR has also the advantage of being implemented in
>         PLASMA
>         already contrary to eigen/svd solvers,
> 
> 
>     I don't know how the two methods compare in practice. SVD algorithms
>     generally use iterated QR reductions in their implementation, so QR
>     reductions can't be worse numerically. But the SVD probably provides
>     a better metric for rank determination. A google search turns up
>     some literature on the subject that I can't access from home.
> 
> 
> OK, here's a good reference 
> <http://www.math.sjsu.edu/%7Efoster/rank/rank_revealing_s.pdf>. A quick 
> look seems to indicate that the  SVD is the way to go.

AFAIK, SVD is indeed the way to go, but do we really need this for the 
orth function ? I am wrong to think that orthonormalizing a matrix of 
linearly independent vectors is the most common usage for orth ? The 
difference in terms of speed is really significant (for example, svd of 
a 2000x100 matrix takes ~1.9 second vs 0.1 s for QR).

cheers,

David


From charlesr.harris at gmail.com  Fri Feb  5 03:31:17 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Feb 2010 01:31:17 -0700
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <e06186141002050012m61754875rfc0bd9dfa1eaa8ea@mail.gmail.com>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<e06186141002042347x74d585dcu3e45837242cd8a90@mail.gmail.com>
	<e06186141002050012m61754875rfc0bd9dfa1eaa8ea@mail.gmail.com>
Message-ID: <e06186141002050031t4d4e381x9c94568e0fecc9a4@mail.gmail.com>

On Fri, Feb 5, 2010 at 1:12 AM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Fri, Feb 5, 2010 at 12:47 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>>
>>
>> On Fri, Feb 5, 2010 at 12:18 AM, David Cournapeau <david at silveregg.co.jp>wrote:
>>
>>> Charles R Harris wrote:
>>> >
>>> >
>>> > On Thu, Feb 4, 2010 at 8:45 PM, David Cournapeau <
>>> david at silveregg.co.jp
>>> > <mailto:david at silveregg.co.jp>> wrote:
>>> >
>>> >     Hi,
>>> >
>>> >     I wanted to know if there was a rationale for using svd to
>>> >     orthonormalize the columns of a matrix (in scipy.linalg). QR-based
>>> >     methods are likely to be much faster, and I thought this was the
>>> >     standard, numerically-stable method to orthonormalize a basis ? If
>>> the
>>> >     reason is to deal with rank-deficient matrices, maybe we could add
>>> an
>>> >     option to choose between them ?
>>> >
>>> >
>>> > QR with column rotation would deal with rank-deficient matrices and
>>> > routines for that are available in LAPACK
>>> > <http://netlib.org/lapack/lug/node42.html>. The SVD was probably used
>>> > because it was available. The diagonal elements of the R matrix can
>>> > somewhat take the place of the singular values when column rotation is
>>> used.
>>>
>>> So would be it ok to use this column-rotated QR in place of svd for
>>> every case in orth ? I would have to check that QR with column rotation
>>> is still significantly faster than svd, but I would surprised if if were
>>> not the case. QR has also the advantage of being implemented in PLASMA
>>> already contrary to eigen/svd solvers,
>>>
>>>
>> I don't know how the two methods compare in practice. SVD algorithms
>> generally use iterated QR reductions in their implementation, so QR
>> reductions can't be worse numerically. But the SVD probably provides a
>> better metric for rank determination. A google search turns up some
>> literature on the subject that I can't access from home.
>>
>>
> OK, here's a good reference<http://www.math.sjsu.edu/%7Efoster/rank/rank_revealing_s.pdf>.
> A quick look seems to indicate that the  SVD is the way to go.
>
>
I take that back. The QR algorithms are faster, but the SVD is more robust.
In practice the LAPACK QR algorithm with column pivoting works well for most
things, but there are even faster versions.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/955fb547/attachment.html>

From david at silveregg.co.jp  Fri Feb  5 03:33:49 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Fri, 05 Feb 2010 17:33:49 +0900
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <20100205081012.GA20893@phare.normalesup.org>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<20100205081012.GA20893@phare.normalesup.org>
Message-ID: <4B6BD7ED.9000205@silveregg.co.jp>

Gael Varoquaux wrote:
> On Fri, Feb 05, 2010 at 04:18:02PM +0900, David Cournapeau wrote:
>> So would be it ok to use this column-rotated QR in place of svd for 
>> every case in orth ? I would have to check that QR with column rotation 
>> is still significantly faster than svd, but I would surprised if if were 
>> not the case. QR has also the advantage of being implemented in PLASMA 
>> already contrary to eigen/svd solvers,
> 
> Out of curiosity, what's PLASMA, in this context?

http://icl.cs.utk.edu/projectsfiles/plasma/html/README.html

"""
The main purpose of PLASMA is to address the performance shortcomings of 
the LAPACK and ScaLAPACK libraries on multicore processors and 
multi-socket systems of multicore processors. PLASMA provides routines 
to solve dense general systems of linear equations, symmetric positive 
definite systems of linear equations and linear least squares problems, 
using LU, Cholesky, QR and LQ factorizations. Real arithmetic and 
complex arithmetic are supported in both single precision and double 
precision.
"""

It is under BSD license, and as a bonus point, may be compiled easily on 
windows (MS is one of the sponsor of the project). The main drawback I 
can see is that it requires a serial BLAS, and ATLAS cannot be switched 
dynamically between serial and parallel (you have to relink).

I am hoping to provide a basic set of wrappers for scipy,

cheers,

David


From charlesr.harris at gmail.com  Fri Feb  5 03:37:43 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Feb 2010 01:37:43 -0700
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <4B6BD6FF.6070201@silveregg.co.jp>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<e06186141002042347x74d585dcu3e45837242cd8a90@mail.gmail.com>
	<e06186141002050012m61754875rfc0bd9dfa1eaa8ea@mail.gmail.com>
	<4B6BD6FF.6070201@silveregg.co.jp>
Message-ID: <e06186141002050037v7988aed8g55c854b9d15acf10@mail.gmail.com>

On Fri, Feb 5, 2010 at 1:29 AM, David Cournapeau <david at silveregg.co.jp>wrote:

> Charles R Harris wrote:
> >
> >
> > On Fri, Feb 5, 2010 at 12:47 AM, Charles R Harris
> > <charlesr.harris at gmail.com <mailto:charlesr.harris at gmail.com>> wrote:
> >
> >
> >
> >     On Fri, Feb 5, 2010 at 12:18 AM, David Cournapeau
> >     <david at silveregg.co.jp <mailto:david at silveregg.co.jp>> wrote:
> >
> >         Charles R Harris wrote:
> >          >
> >          >
> >          > On Thu, Feb 4, 2010 at 8:45 PM, David Cournapeau
> >         <david at silveregg.co.jp <mailto:david at silveregg.co.jp>
> >          > <mailto:david at silveregg.co.jp
> >         <mailto:david at silveregg.co.jp>>> wrote:
> >          >
> >          >     Hi,
> >          >
> >          >     I wanted to know if there was a rationale for using svd to
> >          >     orthonormalize the columns of a matrix (in scipy.linalg).
> >         QR-based
> >          >     methods are likely to be much faster, and I thought this
> >         was the
> >          >     standard, numerically-stable method to orthonormalize a
> >         basis ? If the
> >          >     reason is to deal with rank-deficient matrices, maybe we
> >         could add an
> >          >     option to choose between them ?
> >          >
> >          >
> >          > QR with column rotation would deal with rank-deficient
> >         matrices and
> >          > routines for that are available in LAPACK
> >          > <http://netlib.org/lapack/lug/node42.html>. The SVD was
> >         probably used
> >          > because it was available. The diagonal elements of the R
> >         matrix can
> >          > somewhat take the place of the singular values when column
> >         rotation is used.
> >
> >         So would be it ok to use this column-rotated QR in place of svd
> for
> >         every case in orth ? I would have to check that QR with column
> >         rotation
> >         is still significantly faster than svd, but I would surprised if
> >         if were
> >         not the case. QR has also the advantage of being implemented in
> >         PLASMA
> >         already contrary to eigen/svd solvers,
> >
> >
> >     I don't know how the two methods compare in practice. SVD algorithms
> >     generally use iterated QR reductions in their implementation, so QR
> >     reductions can't be worse numerically. But the SVD probably provides
> >     a better metric for rank determination. A google search turns up
> >     some literature on the subject that I can't access from home.
> >
> >
> > OK, here's a good reference
> > <http://www.math.sjsu.edu/%7Efoster/rank/rank_revealing_s.pdf>. A quick
> > look seems to indicate that the  SVD is the way to go.
>
> AFAIK, SVD is indeed the way to go, but do we really need this for the
> orth function ? I am wrong to think that orthonormalizing a matrix of
> linearly independent vectors is the most common usage for orth ? The
> difference in terms of speed is really significant (for example, svd of
> a 2000x100 matrix takes ~1.9 second vs 0.1 s for QR).
>
>
This looks a bit like sorting: quicksort is almost always fastest, but no
quarantee. The other methods are safer but slower. Maybe the way to go is
use a keyword to choose between methods.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/f056c0a7/attachment.html>

From gael.varoquaux at normalesup.org  Fri Feb  5 03:45:42 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 5 Feb 2010 09:45:42 +0100
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <4B6BD7ED.9000205@silveregg.co.jp>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<20100205081012.GA20893@phare.normalesup.org>
	<4B6BD7ED.9000205@silveregg.co.jp>
Message-ID: <20100205084542.GC20893@phare.normalesup.org>

On Fri, Feb 05, 2010 at 05:33:49PM +0900, David Cournapeau wrote:
> I am hoping to provide a basic set of wrappers for scipy,

Whoho!

Ga?l


From charlesr.harris at gmail.com  Fri Feb  5 03:54:23 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Feb 2010 01:54:23 -0700
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <4B6BD7ED.9000205@silveregg.co.jp>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<20100205081012.GA20893@phare.normalesup.org>
	<4B6BD7ED.9000205@silveregg.co.jp>
Message-ID: <e06186141002050054q3c47a87fx1154fa137c16c488@mail.gmail.com>

On Fri, Feb 5, 2010 at 1:33 AM, David Cournapeau <david at silveregg.co.jp>wrote:

> Gael Varoquaux wrote:
> > On Fri, Feb 05, 2010 at 04:18:02PM +0900, David Cournapeau wrote:
> >> So would be it ok to use this column-rotated QR in place of svd for
> >> every case in orth ? I would have to check that QR with column rotation
> >> is still significantly faster than svd, but I would surprised if if were
> >> not the case. QR has also the advantage of being implemented in PLASMA
> >> already contrary to eigen/svd solvers,
> >
> > Out of curiosity, what's PLASMA, in this context?
>
> http://icl.cs.utk.edu/projectsfiles/plasma/html/README.html
>
> """
> The main purpose of PLASMA is to address the performance shortcomings of
> the LAPACK and ScaLAPACK libraries on multicore processors and
> multi-socket systems of multicore processors. PLASMA provides routines
> to solve dense general systems of linear equations, symmetric positive
> definite systems of linear equations and linear least squares problems,
> using LU, Cholesky, QR and LQ factorizations. Real arithmetic and
> complex arithmetic are supported in both single precision and double
> precision.
> """
>
> It is under BSD license, and as a bonus point, may be compiled easily on
> windows (MS is one of the sponsor of the project). The main drawback I
> can see is that it requires a serial BLAS, and ATLAS cannot be switched
> dynamically between serial and parallel (you have to relink).
>
> I am hoping to provide a basic set of wrappers for scipy,
>
>
Looks like the column pivoting QR algorithm isn't there. I'm not sure what
orth is supposed to be used for, but if there is no danger of rank
deficiency, then the usual QR algorithm should work just fine.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/9c8d57cc/attachment.html>

From d.l.goldsmith at gmail.com  Fri Feb  5 04:23:27 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Fri, 5 Feb 2010 01:23:27 -0800
Subject: [SciPy-dev] WOT: Experiences running 64 bit Vista on a 32 bit
	machine
In-Reply-To: <4B6BC5A6.1000006@silveregg.co.jp>
References: <45d1ab481002042304o375b7c69rb79d1af5b2de37d9@mail.gmail.com>
	<4B6BC5A6.1000006@silveregg.co.jp>
Message-ID: <45d1ab481002050123p3a3abf5do2198c22d0911c708@mail.gmail.com>

On Thu, Feb 4, 2010 at 11:15 PM, David Cournapeau <david at silveregg.co.jp>wrote:

> David Goldsmith wrote:
> > Please forgive the widely OT request for input: I'm thinking about
> > trying to run 64 bit Vista on my 32 bit machine (Pentium Dual-Core T4300
> > @ 2x2.10Ghz); following
> >
> http://windows.microsoft.com/en-US/windows-vista/32-bit-and-64-bit-Windows-frequently-asked-questions#How-do-I-tell.
> ..
> > I've confirmed that I'm "64 bit capable."  Has anyone reading this far
> > ;-) had a particularly positive/negative experience doing this?  Also,
> > how do I determine the 64 bit linux capability of my hardware?
>
> If windows 64 runs, linux 64 will as well; as far as OS are concerned,
> 64 bits mode is like a new architecture (like ppc vs intel). When the PC
> starts, the kernel of whatever OS you are running will refuse to run if
> it is 64 bits and you run on a 32 bits machine. The details are OS/Boot
> loader dependent, but most installers will refuse to install a 64 bits
> installers on a 32bits-only machine anyway. You can also run a 64 bits
> VM inside a 32 bits, assuming your CPU is 64 bits capable (so you can
> run 64 bits Ubuntu in vmware on top of 32 bits Ubuntu for example).
>
> I strongly suggest using windows 7 instead of Vista if you can.
>

OK, thank you very much for the input.  Do you have first-hand experience w/
that vmware configuration?  My main issue of concern, IIRC the responses to
a post I made about a year-and-a-half ago, is a single object being able to
address more than 4GB of memory...  Thanks again,

DG


>
> cheers,
>
> David
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/ad317943/attachment.html>

From david at silveregg.co.jp  Fri Feb  5 04:45:51 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Fri, 05 Feb 2010 18:45:51 +0900
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <e06186141002050037v7988aed8g55c854b9d15acf10@mail.gmail.com>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<e06186141002042347x74d585dcu3e45837242cd8a90@mail.gmail.com>
	<e06186141002050012m61754875rfc0bd9dfa1eaa8ea@mail.gmail.com>
	<4B6BD6FF.6070201@silveregg.co.jp>
	<e06186141002050037v7988aed8g55c854b9d15acf10@mail.gmail.com>
Message-ID: <4B6BE8CF.1020207@silveregg.co.jp>

Charles R Harris wrote:
> 
> Maybe the way to 
> go is use a keyword to choose between methods.

'safe' vs 'fast' ? :)

I wonder whether it would be ok to change the default to QR, though. I 
will look at the pivoted QR thing,

cheers,

David


From josef.pktd at gmail.com  Fri Feb  5 06:59:05 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 5 Feb 2010 06:59:05 -0500
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<e06186141002032237oe333d5bj85c2a17d3e2de256@mail.gmail.com>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
	<4B6B69C2.7000907@enthought.com>
	<5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>
	<1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>
Message-ID: <1cd32cbb1002050359g14c4b4d5m5e38c6f92349669b@mail.gmail.com>

On Thu, Feb 4, 2010 at 9:24 PM,  <josef.pktd at gmail.com> wrote:
> On Thu, Feb 4, 2010 at 8:27 PM, David Cournapeau <cournape at gmail.com> wrote:
>> On Fri, Feb 5, 2010 at 9:43 AM, Warren Weckesser
>> <warren.weckesser at enthought.com> wrote:
>>
>>> David C,
>>>
>>> You said this was for symmetric matrices, but do you envision later
>>> allowing nonsymmetric matrices?
>>
>> Yes. I have only implemented symmetric/Hermitian because that's the
>> only solver that handles getting only a few eigenvalues in LAPACK.
>> Matlab own eigs function uses ARPACK for both symmetric/unsymmetric
>> cases, which is not good according to one of the Lapack developer
>> (http://mail.scipy.org/pipermail/scipy-dev/2007-March/006778.html).
>> But we could use ARPACK for the general case in eigs.
>>
>>>
>>> If not, then perhaps the name of the function should be 'eigsh',
>>> following the precedent set by numpy.linalg and scipy.linalg.
>>
>> I wonder whether those eigh functions are a good idea: I fear that
>> most people will always use eig - maybe one could use the underlying
>> eig*h solver in eig* if the matrix is detected as being symmetric ? I
>> am not really knowledgeable about those issues, though. For example, I
>> don't know whether the symmetric aspect should be checked exactly, or
>> if it is better to use a symmetric solver ?even if there are very
>> small errors in say A'-A.
>>
>>>
>>> In particular, the choice of ordering by magnitude or by real part is
>>> convenient.
>>
>> It seems that one would need to implement the non-symmetric
>> capabilities to sort this out. I fear that those options are solver
>> specific, though - maybe the solution is to have two levels of API,
>> one low-level and one high level.
>>
>> cheers,
>>
>> David
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>
> the current version of scipy.linalg.eigh sorts from smallest to largest
>
>>>> scipy.linalg.eigh(x)[0]
> array([ ?4.04457427e-03, ? 1.84073286e-01, ? 6.74875960e-01,
> ? ? ? ? 3.23328824e+00, ? 4.00741304e+00, ? 6.98333680e+00,
> ? ? ? ? 1.45314842e+01, ? 1.49260377e+01, ? 2.47166702e+01,
> ? ? ? ? 2.98955755e+01])
>>>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
> array([ 0.00404457, ?0.18407329, ?0.67487596, ?3.23328824])
>>>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
> array([ 14.92603769, ?24.71667021, ?29.89557547])
>
>
>>>> x = np.random.randn(10, 10) + 1j*np.random.randn(10, 10)
>>>> x = np.dot(x.T, x)
>>>> scipy.linalg.eigh(x, eigvals=(0,3))[0]
> array([-36.82039662, -18.20362967, -12.21716065, ?-9.79752274])
>>>> scipy.linalg.eigh(x, eigvals=(len(x)-3, len(x)-1))[0]
> array([ 14.44815917, ?28.8394026 , ?42.45471058])

How does the new implementation relate to the existing implementation
of selecting just a few eigenvalues in a range that is possible with
the current scipy.linalg.eigh ?

Tiziano added this with the integration of symeig into scipy.linalg.

Josef


> Josef
>


From cournape at gmail.com  Fri Feb  5 07:59:23 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 5 Feb 2010 21:59:23 +0900
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <1cd32cbb1002050359g14c4b4d5m5e38c6f92349669b@mail.gmail.com>
References: <4B6A68ED.6070007@silveregg.co.jp>
	<4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
	<4B6B69C2.7000907@enthought.com>
	<5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>
	<1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>
	<1cd32cbb1002050359g14c4b4d5m5e38c6f92349669b@mail.gmail.com>
Message-ID: <5b8d13221002050459l7b582b7eo503ded93fee80adf@mail.gmail.com>

On Fri, Feb 5, 2010 at 8:59 PM,  <josef.pktd at gmail.com> wrote:

>
> How does the new implementation relate to the existing implementation
> of selecting just a few eigenvalues in a range that is possible with
> the current scipy.linalg.eigh ?

Mostly a different interface (the underlying lapack function is the
same). What bothers me with the current API for eigen/svd
decompositions is the lack of consistency. The current eigh also does
not enable to look for eigenvalues in a value range (e.g. all eigen
values between 2 and 3), and I intend to add support for non-symmetric
eigenvalues as well.

cheers,

David


From charlesr.harris at gmail.com  Fri Feb  5 11:53:25 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Feb 2010 09:53:25 -0700
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <4B6BE8CF.1020207@silveregg.co.jp>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<e06186141002042347x74d585dcu3e45837242cd8a90@mail.gmail.com>
	<e06186141002050012m61754875rfc0bd9dfa1eaa8ea@mail.gmail.com>
	<4B6BD6FF.6070201@silveregg.co.jp>
	<e06186141002050037v7988aed8g55c854b9d15acf10@mail.gmail.com>
	<4B6BE8CF.1020207@silveregg.co.jp>
Message-ID: <e06186141002050853j351330d0vf77f25b0c5da7f50@mail.gmail.com>

On Fri, Feb 5, 2010 at 2:45 AM, David Cournapeau <david at silveregg.co.jp>wrote:

> Charles R Harris wrote:
> >
> > Maybe the way to
> > go is use a keyword to choose between methods.
>
> 'safe' vs 'fast' ? :)
>
> I wonder whether it would be ok to change the default to QR, though. I
> will look at the pivoted QR thing,
>
>
That sounds OK. The documentation should probably mention that the
orthogonal columns produced by the two methods will be different, otherwise
we will probably get some bug reports.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/7cfd4468/attachment.html>

From charlesr.harris at gmail.com  Fri Feb  5 13:15:51 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Feb 2010 11:15:51 -0700
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <e06186141002050853j351330d0vf77f25b0c5da7f50@mail.gmail.com>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<e06186141002042347x74d585dcu3e45837242cd8a90@mail.gmail.com>
	<e06186141002050012m61754875rfc0bd9dfa1eaa8ea@mail.gmail.com>
	<4B6BD6FF.6070201@silveregg.co.jp>
	<e06186141002050037v7988aed8g55c854b9d15acf10@mail.gmail.com>
	<4B6BE8CF.1020207@silveregg.co.jp>
	<e06186141002050853j351330d0vf77f25b0c5da7f50@mail.gmail.com>
Message-ID: <e06186141002051015o14887016u2bb4e10321cc6349@mail.gmail.com>

On Fri, Feb 5, 2010 at 9:53 AM, Charles R Harris
<charlesr.harris at gmail.com>wrote:

>
>
> On Fri, Feb 5, 2010 at 2:45 AM, David Cournapeau <david at silveregg.co.jp>wrote:
>
>> Charles R Harris wrote:
>> >
>> > Maybe the way to
>> > go is use a keyword to choose between methods.
>>
>> 'safe' vs 'fast' ? :)
>>
>> I wonder whether it would be ok to change the default to QR, though. I
>> will look at the pivoted QR thing,
>>
>>
> That sounds OK. The documentation should probably mention that the
> orthogonal columns produced by the two methods will be different, otherwise
> we will probably get some bug reports.
>
>
I don't know if you read the paper, but the fastest and safest QR
factorizations seem to be the DGEQPX and DGEQPY algorithms in ACM Algorithm
782. It looks like one might need to get permission from the ACM to use the
algorithms in a BSD licensed library.

 Submittal of an algorithm for publication in one of the ACM Transactions
impliesthat unrestricted use of the algorithm within a computer is
permissible.General permission to copy and distribute the algorithm without
fee is granted provided that the copies are not made or distributed for
direct commercial advantage. The ACM copyright notice and the title of the
publication and its date appear, and notice is given that copying is by
permission of the Association for Computing Machinery. To copy otherwise, or
to republish, requires a fee and/or specific permission.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/faf1a43b/attachment.html>

From smattacus at gmail.com  Fri Feb  5 14:51:24 2010
From: smattacus at gmail.com (Sean Mattingly)
Date: Fri, 5 Feb 2010 11:51:24 -0800
Subject: [SciPy-dev] WOT: Experiences running 64 bit Vista on a 32 bit
	machine
In-Reply-To: <45d1ab481002050123p3a3abf5do2198c22d0911c708@mail.gmail.com>
References: <45d1ab481002042304o375b7c69rb79d1af5b2de37d9@mail.gmail.com>
	<4B6BC5A6.1000006@silveregg.co.jp>
	<45d1ab481002050123p3a3abf5do2198c22d0911c708@mail.gmail.com>
Message-ID: <856175f81002051151s1d610d8cia9116f41a1431a2d@mail.gmail.com>

Hi David,

I've installed Linux x64, Vista x64, and Windows 7 x64 on my desktop
computer which has this kind of processor:

http://www.newegg.com/Product/Product.aspx?Item=N82E16819115029

Looking around a bit on the web, I can find instances of computers with the
same processor you have with 64 bit OS's already installed. If I had a
computer with that processor, I wouldn't think twice about putting a 64 bit
OS on it.

I doubt that, with the 64 bit VM on a 32 bit host OS on a 64 bit CPU, you'd
be able to address more than the ~ 2.8 gigs of memory or so which is the
limit for 32 bit machines running windows (after hardware and whatever takes
its chunk of the 4g visible). Indeed, finding some threads on this:

http://superuser.com/questions/15434/how-does-vmware-guest-os-memory-usage-work

There's some links to forum discussion on that page where people have tried
that, but the gist of it is that the VM can be told to use more than the
amount of ram physically provided, but it's paging memory, and not
physically accessing the unused memory. What you're asking about, if it can
use more than the 4g limit, would require the VM making its own OS - like
interface with the hardware...which means it's not a VM anymore!

Also, I want to agree with an earlier message that you go with Windows 7. I
had a dual - boot configuration of Vista and Linux (Openbox on Ubuntu), and
Vista was just giving me all sorts of problems. I replaced the Vista
installation with Windows 7 about 2 weeks ago, so these weren't "early
rollout" issues. 7 is a much smoother OS; under the hood, it's running a lot
of the good things from Vista anyways, so there's no reason to go with
Vista. Really, IMO, going with Vista would be like taking windows ME over XP
SP2, at this point. And the driver support is much better, as some
manufacturers and software developers have just stopped caring about Vista.

Finally, if you decide to install Vista or Windows 7 and you have more than
one hard drive partition, feel free to ping me with questions, because
Microsoft made some...interesting...design decisions in the install process
there.

Hope some of this helps...
 - Sean


On Fri, Feb 5, 2010 at 1:23 AM, David Goldsmith <d.l.goldsmith at gmail.com>wrote:

> On Thu, Feb 4, 2010 at 11:15 PM, David Cournapeau <david at silveregg.co.jp>wrote:
>
>> David Goldsmith wrote:
>> > Please forgive the widely OT request for input: I'm thinking about
>> > trying to run 64 bit Vista on my 32 bit machine (Pentium Dual-Core T4300
>> > @ 2x2.10Ghz); following
>> >
>> http://windows.microsoft.com/en-US/windows-vista/32-bit-and-64-bit-Windows-frequently-asked-questions#How-do-I-tell.
>> ..
>> > I've confirmed that I'm "64 bit capable."  Has anyone reading this far
>> > ;-) had a particularly positive/negative experience doing this?  Also,
>> > how do I determine the 64 bit linux capability of my hardware?
>>
>> If windows 64 runs, linux 64 will as well; as far as OS are concerned,
>> 64 bits mode is like a new architecture (like ppc vs intel). When the PC
>> starts, the kernel of whatever OS you are running will refuse to run if
>> it is 64 bits and you run on a 32 bits machine. The details are OS/Boot
>> loader dependent, but most installers will refuse to install a 64 bits
>> installers on a 32bits-only machine anyway. You can also run a 64 bits
>> VM inside a 32 bits, assuming your CPU is 64 bits capable (so you can
>> run 64 bits Ubuntu in vmware on top of 32 bits Ubuntu for example).
>>
>> I strongly suggest using windows 7 instead of Vista if you can.
>>
>
> OK, thank you very much for the input.  Do you have first-hand experience
> w/ that vmware configuration?  My main issue of concern, IIRC the responses
> to a post I made about a year-and-a-half ago, is a single object being able
> to address more than 4GB of memory...  Thanks again,
>
> DG
>
>
>>
>> cheers,
>>
>> David
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/d0f66be1/attachment.html>

From d.l.goldsmith at gmail.com  Fri Feb  5 15:07:25 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Fri, 5 Feb 2010 12:07:25 -0800
Subject: [SciPy-dev] WOT: Experiences running 64 bit Vista on a 32 bit
	machine
In-Reply-To: <856175f81002051151s1d610d8cia9116f41a1431a2d@mail.gmail.com>
References: <45d1ab481002042304o375b7c69rb79d1af5b2de37d9@mail.gmail.com>
	<4B6BC5A6.1000006@silveregg.co.jp>
	<45d1ab481002050123p3a3abf5do2198c22d0911c708@mail.gmail.com>
	<856175f81002051151s1d610d8cia9116f41a1431a2d@mail.gmail.com>
Message-ID: <45d1ab481002051207s7afce8f8paf6cf0681a613457@mail.gmail.com>

On Fri, Feb 5, 2010 at 11:51 AM, Sean Mattingly <smattacus at gmail.com> wrote:

> Hi David,
>
> I've installed Linux x64, Vista x64, and Windows 7 x64 on my desktop
> computer which has this kind of processor:
>
> http://www.newegg.com/Product/Product.aspx?Item=N82E16819115029
>
> Looking around a bit on the web, I can find instances of computers with the
> same processor you have with 64 bit OS's already installed. If I had a
> computer with that processor, I wouldn't think twice about putting a 64 bit
> OS on it.
>
> I doubt that, with the 64 bit VM on a 32 bit host OS on a 64 bit CPU, you'd
> be able to address more than the ~ 2.8 gigs of memory or so which is the
> limit for 32 bit machines running windows (after hardware and whatever takes
> its chunk of the 4g visible). Indeed, finding some threads on this:
>
>
> http://superuser.com/questions/15434/how-does-vmware-guest-os-memory-usage-work
>
> There's some links to forum discussion on that page where people have tried
> that, but the gist of it is that the VM can be told to use more than the
> amount of ram physically provided, but it's paging memory, and not
> physically accessing the unused memory. What you're asking about, if it can
> use more than the 4g limit, would require the VM making its own OS - like
> interface with the hardware...which means it's not a VM anymore!
>
> Also, I want to agree with an earlier message that you go with Windows 7. I
> had a dual - boot configuration of Vista and Linux (Openbox on Ubuntu), and
> Vista was just giving me all sorts of problems. I replaced the Vista
> installation with Windows 7 about 2 weeks ago, so these weren't "early
> rollout" issues. 7 is a much smoother OS; under the hood, it's running a lot
> of the good things from Vista anyways, so there's no reason to go with
> Vista. Really, IMO, going with Vista would be like taking windows ME over XP
> SP2, at this point. And the driver support is much better, as some
> manufacturers and software developers have just stopped caring about Vista.
>
> Finally, if you decide to install Vista or Windows 7 and you have more than
> one hard drive partition, feel free to ping me with questions, because
> Microsoft made some...interesting...design decisions in the install process
> there.
>
> Hope some of this helps...
>  - Sean


It helps a lot, thanks, but just to be clear, mine is dual core 32 bit
hardware, yes?  So the info about the 64 bit VM on a 32 bit host OS on a 64
bit CPU isn't exactly applicable, is it?

Right now, I'm leaning toward keeping my Windows install as is and trying a
64-bit linux install in a dual boot config., as the only thing presently I
need the extra memory addressing for is Python-related software. :-)

Thanks again.

DG

PS: now that I've netted a couple "volunteer instructors," if the linux
majority would prefer it if I took this exchange off list...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100205/ad4e7eb0/attachment.html>

From luszczek at eecs.utk.edu  Fri Feb  5 23:05:12 2010
From: luszczek at eecs.utk.edu (Piotr Luszczek)
Date: Sat, 6 Feb 2010 04:05:12 +0000 (UTC)
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<20100205081012.GA20893@phare.normalesup.org>
	<4B6BD7ED.9000205@silveregg.co.jp>
Message-ID: <loom.20100206T045617-309@post.gmane.org>

David Cournapeau <david <at> silveregg.co.jp> writes:

> 
> Gael Varoquaux wrote:
> > On Fri, Feb 05, 2010 at 04:18:02PM +0900, David Cournapeau wrote:
> >> So would be it ok to use this column-rotated QR in place of svd for 
> >> every case in orth ? I would have to check that QR with column rotation 
> >> is still significantly faster than svd, but I would surprised if if were 
> >> not the case. QR has also the advantage of being implemented in PLASMA 
> >> already contrary to eigen/svd solvers,
> > 
> > Out of curiosity, what's PLASMA, in this context?
> http://icl.cs.utk.edu/projectsfiles/plasma/html/README.html 
> It is under BSD license, and as a bonus point, may be compiled easily on 
> windows (MS is one of the sponsor of the project). The main drawback I 
> can see is that it requires a serial BLAS, and ATLAS cannot be switched 
> dynamically between serial and parallel (you have to relink).
> 
> I am hoping to provide a basic set of wrappers for scipy,

We (PLASMA team) are looking into this problem. In principle
it should be possible to just use parallel ATLAS and call
internal functions of ATLAS to get at the serial interface.
We might add this feature to PLASMA some time soon.

Piotr





From luszczek at eecs.utk.edu  Fri Feb  5 23:21:35 2010
From: luszczek at eecs.utk.edu (Piotr Luszczek)
Date: Sat, 6 Feb 2010 04:21:35 +0000 (UTC)
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<20100205081012.GA20893@phare.normalesup.org>
	<4B6BD7ED.9000205@silveregg.co.jp>
	<e06186141002050054q3c47a87fx1154fa137c16c488@mail.gmail.com>
Message-ID: <loom.20100206T051831-630@post.gmane.org>

Charles R Harris <charlesr.harris <at> gmail.com> writes:

> 
> 
> On Fri, Feb 5, 2010 at 1:33 AM, David Cournapeau <david <at> silveregg.co.jp>
wrote:
> Gael Varoquaux wrote:
> > On Fri, Feb 05, 2010 at 04:18:02PM +0900, David Cournapeau wrote:
> >> So would be it ok to use this column-rotated QR in place of svd for
> >> every case in orth ? I would have to check that QR with column rotation
> >> is still significantly faster than svd, but I would surprised if if were
> >> not the case. QR has also the advantage of being implemented in PLASMA
> >> already contrary to eigen/svd solvers,
> >
> > Out of curiosity, what's PLASMA, in this context?
> http://icl.cs.utk.edu/projectsfiles/plasma/html/README.html
> It is under BSD license, and as a bonus point, may be compiled easily on
> windows (MS is one of the sponsor of the project). The main drawback I
> can see is that it requires a serial BLAS, and ATLAS cannot be switched
> dynamically between serial and parallel (you have to relink).
> I am hoping to provide a basic set of wrappers for scipy,
> 
> 
> Looks like the column pivoting QR algorithm isn't there. I'm not sure what
orth is supposed to be used for, but if there is no danger of rank deficiency,
then the usual QR algorithm should work just fine.Chuck 

Indeed, there is no rank-revealing QR in PLASMA (I'm assuming this is what
you're after). And there are no immediate plans for it.

Piotr

PS. I'm part of PLASMA team.



From opossumnano at gmail.com  Sun Feb  7 03:14:29 2010
From: opossumnano at gmail.com (Tiziano Zito)
Date: Sun, 7 Feb 2010 09:14:29 +0100
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
 (computing a few eigenvalues only)
In-Reply-To: <5b8d13221002050459l7b582b7eo503ded93fee80adf@mail.gmail.com>
References: <4B6A6E06.7060102@silveregg.co.jp>
	<4B6A8154.2040705@student.matnat.uio.no>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
	<4B6B69C2.7000907@enthought.com>
	<5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>
	<1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>
	<1cd32cbb1002050359g14c4b4d5m5e38c6f92349669b@mail.gmail.com>
	<5b8d13221002050459l7b582b7eo503ded93fee80adf@mail.gmail.com>
Message-ID: <20100207081429.GB3065@localhost>

> > How does the new implementation relate to the existing implementation
> > of selecting just a few eigenvalues in a range that is possible with
> > the current scipy.linalg.eigh ?
> 
> Mostly a different interface (the underlying lapack function is the
> same). What bothers me with the current API for eigen/svd
> decompositions is the lack of consistency. The current eigh also does
> not enable to look for eigenvalues in a value range (e.g. all eigen
> values between 2 and 3), and I intend to add support for non-symmetric
> eigenvalues as well.

Is the new eigs function going to supersede eigh? I did not know
that LAPACK allowed selecting a range of eigenvalues by value. Which
are the LAPACK routines you are referring to?
I agree that a consistent interface for eigen/svd decomposition is
good, I am on the other hand a little unsure that linalg needs an
eigs function together with an eigh function. Why not simply change the
eigh interface in a backward compatible way?

ciao,
tiziano



From tmp50 at ukr.net  Sun Feb  7 04:37:12 2010
From: tmp50 at ukr.net (Dmitrey)
Date: Sun, 07 Feb 2010 11:37:12 +0200
Subject: [SciPy-dev] Don't you mind me to put a minor fix into some
	scipy.optimize docstrings?
Message-ID: <E1Ne3Zo-000LjI-NV@ffe6.ukr.net>

Hello,  
I've been informed scipy.optimize docstrings still point to scikits.openopt (thus google search yields deprecated webpages), so if you don't mind I'll point it to mere openopt.  
  
Regards, D.  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100207/597071a5/attachment.html>

From cournape at gmail.com  Sun Feb  7 07:05:31 2010
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 7 Feb 2010 21:05:31 +0900
Subject: [SciPy-dev] Comments on API for Matlab's eigs equivalent
	(computing a few eigenvalues only)
In-Reply-To: <20100207081429.GB3065@localhost>
References: <4B6A6E06.7060102@silveregg.co.jp>
	<750A37B6-052E-4F91-A63A-97EC5D422AD5@cs.toronto.edu>
	<a749952d1002041201g37c90abctc2b20045d906ed14@mail.gmail.com>
	<ce557a361002041218u6efa7d3bod9b9f2492e65ab17@mail.gmail.com>
	<4B6B69C2.7000907@enthought.com>
	<5b8d13221002041727w131d1854sde448b76fcd9155c@mail.gmail.com>
	<1cd32cbb1002041824s646b6a37g7a0e4c761975401e@mail.gmail.com>
	<1cd32cbb1002050359g14c4b4d5m5e38c6f92349669b@mail.gmail.com>
	<5b8d13221002050459l7b582b7eo503ded93fee80adf@mail.gmail.com>
	<20100207081429.GB3065@localhost>
Message-ID: <5b8d13221002070405j116fd747n8ff52f84e677fd30@mail.gmail.com>

On Sun, Feb 7, 2010 at 5:14 PM, Tiziano Zito <opossumnano at gmail.com> wrote:
>> > How does the new implementation relate to the existing implementation
>> > of selecting just a few eigenvalues in a range that is possible with
>> > the current scipy.linalg.eigh ?
>>
>> Mostly a different interface (the underlying lapack function is the
>> same). What bothers me with the current API for eigen/svd
>> decompositions is the lack of consistency. The current eigh also does
>> not enable to look for eigenvalues in a value range (e.g. all eigen
>> values between 2 and 3), and I intend to add support for non-symmetric
>> eigenvalues as well.
>
> Is the new eigs function going to supersede eigh? I did not know
> that LAPACK allowed selecting a range of eigenvalues by value. Which
> are the LAPACK routines you are referring to?

The ones used in eigh, e.g. dsyevr. The range mode may be an addition
in lapack 3.0, I am not sure:

"""
*  DSYEVR computes selected eigenvalues and, optionally, eigenvectors
*  of a real symmetric matrix A.  Eigenvalues and eigenvectors can be
*  selected by specifying either a range of values or a range of
*  indices for the desired eigenvalues.
"""

> I agree that a consistent interface for eigen/svd decomposition is
> good, I am on the other hand a little unsure that linalg needs an
> eigs function together with an eigh function. Why not simply change the
> eigh interface in a backward compatible way?

Because I intend to support non-symmetric matrices (using arpack for
the unsymmetric case).

David


From warren.weckesser at enthought.com  Sun Feb  7 20:57:14 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sun, 07 Feb 2010 19:57:14 -0600
Subject: [SciPy-dev] Ticket #1105 -- patch submitted for
	scipy.signal.waveforms.chirp()
Message-ID: <4B6F6F7A.70506@enthought.com>

I just added a patch to ticket #1105; a summary of the changes is given 
in the ticket.  If any scipy.signal users (especially chirp()) users) 
have chance, please take a look a let me know what you think.  The 
docstrings could use polishing, but I'd like to get some feedback before 
doing more work on it.

Warren



From cournape at gmail.com  Mon Feb  8 20:45:00 2010
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 9 Feb 2010 10:45:00 +0900
Subject: [SciPy-dev] Why does orth use svd instead of QR ?
In-Reply-To: <loom.20100206T045617-309@post.gmane.org>
References: <4B6B9466.4010507@silveregg.co.jp>
	<e06186141002042158n5dfc6a0bn3043f89c974b8f4@mail.gmail.com>
	<4B6BC62A.8090109@silveregg.co.jp>
	<20100205081012.GA20893@phare.normalesup.org>
	<4B6BD7ED.9000205@silveregg.co.jp>
	<loom.20100206T045617-309@post.gmane.org>
Message-ID: <5b8d13221002081745t7eee8405g72b3d1bc2da10740@mail.gmail.com>

On Sat, Feb 6, 2010 at 1:05 PM, Piotr Luszczek <luszczek at eecs.utk.edu> wrote:

>
> We (PLASMA team) are looking into this problem. In principle
> it should be possible to just use parallel ATLAS and call
> internal functions of ATLAS to get at the serial interface.
> We might add this feature to PLASMA some time soon.

Great, thank you for the information.

cheers,

David


From stefan at sun.ac.za  Tue Feb  9 04:31:12 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 9 Feb 2010 11:31:12 +0200
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
Message-ID: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>

Hi all,

I noticed the following changeset coming through. Do you think docstrings
are the right place for advertising external packages?  If not, should they
be moved to the module docstring, or removed entirely?

Regards
St?fan

---------- Forwarded message ----------
Author: dmitrey.kroshko
Date: 2010-02-09 02:41:34 -0600 (Tue, 09 Feb 2010)
New Revision: 6221

Modified:
  trunk/scipy/optimize/anneal.py
  trunk/scipy/optimize/cobyla.py
  trunk/scipy/optimize/lbfgsb.py
  trunk/scipy/optimize/minpack.py
  trunk/scipy/optimize/nnls.py
  trunk/scipy/optimize/optimize.py
  trunk/scipy/optimize/slsqp.py
  trunk/scipy/optimize/tnc.py
Log:
scikits.openopt replaced by mere openopt


Modified: trunk/scipy/optimize/anneal.py
===================================================================
--- trunk/scipy/optimize/anneal.py      2010-02-08 15:18:27 UTC (rev 6220)
+++ trunk/scipy/optimize/anneal.py      2010-02-09 08:41:34 UTC (rev 6221)
@@ -217,6 +217,8 @@

      fixed_point -- scalar fixed-point finder

+      OpenOpt -- Python package with more optimization solvers
+
    """
    x0 = asarray(x0)
    lower = asarray(lower)

Modified: trunk/scipy/optimize/cobyla.py
===================================================================
--- trunk/scipy/optimize/cobyla.py      2010-02-08 15:18:27 UTC (rev 6220)
+++ trunk/scipy/optimize/cobyla.py      2010-02-09 08:41:34 UTC (rev 6221)
@@ -46,8 +46,6 @@

    See also:

-        scikits.openopt, which offers a unified syntax to call this and
other solvers
-
        fmin, fmin_powell, fmin_cg,
              fmin_bfgs, fmin_ncg -- multivariate local optimizers
        leastsq -- nonlinear least squares minimizer
@@ -65,6 +63,9 @@

        fixed_point -- scalar fixed-point finder

+        OpenOpt -- a tool which offers a unified syntax to call this and
+         other solvers with possibility of automatic differentiation
+
    """
    err = "cons must be a sequence of callable functions or a single"\
              " callable function."

Modified: trunk/scipy/optimize/lbfgsb.py
===================================================================
--- trunk/scipy/optimize/lbfgsb.py      2010-02-08 15:18:27 UTC (rev 6220)
+++ trunk/scipy/optimize/lbfgsb.py      2010-02-09 08:41:34 UTC (rev 6221)
@@ -119,8 +119,6 @@
       ACM Transactions on Mathematical Software, Vol 23, Num. 4, pp. 550 -
560.

    See also:
-        scikits.openopt, which offers a unified syntax to call this and
other solvers
-
        fmin, fmin_powell, fmin_cg,
               fmin_bfgs, fmin_ncg -- multivariate local optimizers
        leastsq -- nonlinear least squares minimizer
@@ -138,6 +136,9 @@

        fixed_point -- scalar fixed-point finder

+        OpenOpt -- a tool which offers a unified syntax to call this and
+         other solvers with possibility of automatic differentiation
+
    """
    n = len(x0)


Modified: trunk/scipy/optimize/minpack.py
===================================================================
--- trunk/scipy/optimize/minpack.py     2010-02-08 15:18:27 UTC (rev 6220)
+++ trunk/scipy/optimize/minpack.py     2010-02-09 08:41:34 UTC (rev 6221)
@@ -102,8 +102,6 @@

    See Also
    --------
-    scikits.openopt : offers a unified syntax to call this and other
solvers
-
    fmin, fmin_powell, fmin_cg, fmin_bfgs, fmin_ncg : multivariate local
optimizers

    leastsq : nonlinear least squares minimizer
@@ -118,6 +116,9 @@

    fixed_point : scalar and vector fixed-point finder

+    OpenOpt : a tool which offers a unified syntax to call this and
+     other solvers with possibility of automatic differentiation
+
    """
    if not warning :
        msg = "The warning keyword is deprecated. Use the warnings module."
@@ -263,7 +264,6 @@

    See Also
    --------
-    scikits.openopt: offers a unified syntax to call this and other solvers
    fmin, fmin_powell, fmin_cg, fmin_bfgs, fmin_ncg: multivariate local
optimizers
    fmin_l_bfgs_b, fmin_tnc, fmin_cobyla: constrained multivariate
optimizers
    anneal, brute: global optimizers
@@ -272,6 +272,9 @@
    brentq, brenth, ridder, bisect, newton: one-dimensional root-finding
    fixed_point: scalar and vector fixed-point finder
    curve_fit: find parameters for a curve-fitting problem.
+    OpenOpt : a tool which offers a unified syntax to call this and
+     other solvers with possibility of automatic differentiation
+
    """
    if not warning :
        msg = "The warning keyword is deprecated. Use the warnings module."

Modified: trunk/scipy/optimize/nnls.py
===================================================================
--- trunk/scipy/optimize/nnls.py        2010-02-08 15:18:27 UTC (rev 6220)
+++ trunk/scipy/optimize/nnls.py        2010-02-09 08:41:34 UTC (rev 6221)
@@ -16,6 +16,8 @@

    wrapper around NNLS.F code below nnls/ directory

+    Check OpenOpt for more LLSP solvers
+
    """

    A,b = map(asarray_chkfinite, (A,b))

Modified: trunk/scipy/optimize/optimize.py
===================================================================
--- trunk/scipy/optimize/optimize.py    2010-02-08 15:18:27 UTC (rev 6220)
+++ trunk/scipy/optimize/optimize.py    2010-02-09 08:41:34 UTC (rev 6221)
@@ -156,7 +156,8 @@

        Uses a Nelder-Mead simplex algorithm to find the minimum of
        function of one or more variables.
-
+        Check OpenOpt - a tool which offers a unified syntax to call
+        this and other solvers with possibility of automatic
differentiation.
    """
    fcalls, func = wrap_function(func, args)
    x0 = asfarray(x0).flatten()
@@ -694,8 +695,8 @@

    *See Also*:

-      scikits.openopt : SciKit which offers a unified syntax to call
-                        this and other solvers.
+      OpenOpt : a tool which offers a unified syntax to call
+                this and other solvers with possibility of automatic
differentiation.

    """
    x0 = asarray(x0).squeeze()
@@ -862,7 +863,8 @@
        using the nonlinear conjugate gradient algorithm of Polak and
        Ribiere See Wright, and Nocedal 'Numerical Optimization',
        1999, pg. 120-122.
-
+        Check OpenOpt - a tool which offers a unified syntax to call
+        this and other solvers with possibility of automatic
differentiation.
    """
    x0 = asarray(x0).flatten()
    if maxiter is None:
@@ -1018,8 +1020,7 @@
            If True, return a list of results at each iteration.

    :Notes:
-      1. scikits.openopt offers a unified syntax to call this and other
solvers.
-      2. Only one of `fhess_p` or `fhess` need to be given.  If `fhess`
+      1. Only one of `fhess_p` or `fhess` need to be given.  If `fhess`
      is provided, then `fhess_p` will be ignored.  If neither `fhess`
      nor `fhess_p` is provided, then the hessian product will be
      approximated using finite differences on `fprime`. `fhess_p`
@@ -1027,6 +1028,8 @@
      given, finite-differences on `fprime` are used to compute
      it. See Wright, and Nocedal 'Numerical Optimization', 1999,
      pg. 140.
+      2. Check OpenOpt - a tool which offers a unified syntax to call
+      this and other solvers with possibility of automatic differentiation.

    """
    x0 = asarray(x0).flatten()
@@ -1179,8 +1182,9 @@
        Finds a local minimizer of the scalar function `func` in the
        interval x1 < xopt < x2 using Brent's method.  (See `brent`
        for auto-bracketing).
+        Check OpenOpt - a tool which offers a unified syntax to call
+        this and other solvers with possibility of automatic
differentiation.

-
    """
    # Test bounds are of correct form

@@ -1722,7 +1726,8 @@

        Uses a modification of Powell's method to find the minimum of
        a function of N variables.
-
+        Check OpenOpt - a tool which offers a unified syntax to call
+        this and other solvers with possibility of automatic
differentiation.
    """
    # we need to use a mutable object here that we can update in the
    # wrapper function

Modified: trunk/scipy/optimize/slsqp.py
===================================================================
--- trunk/scipy/optimize/slsqp.py       2010-02-08 15:18:27 UTC (rev 6220)
+++ trunk/scipy/optimize/slsqp.py       2010-02-09 08:41:34 UTC (rev 6221)
@@ -146,6 +146,11 @@

    for examples see :ref:`in the tutorial <tutorial-sqlsp>`

+    See also
+    --------
+    OpenOpt - a tool which offers a unified syntax to call this
+    and other solvers with possibility of automatic differentiation.
+
    """

    exit_modes = { -1 : "Gradient evaluation required (g & a)",

Modified: trunk/scipy/optimize/tnc.py
===================================================================
--- trunk/scipy/optimize/tnc.py 2010-02-08 15:18:27 UTC (rev 6220)
+++ trunk/scipy/optimize/tnc.py 2010-02-09 08:41:34 UTC (rev 6221)
@@ -164,8 +164,6 @@
            Return code as defined in the RCSTRINGS dict.

    :SeeAlso:
-      - scikits.openopt, which offers a unified syntax to call this and
other solvers
-
      - fmin, fmin_powell, fmin_cg, fmin_bfgs, fmin_ncg :
         multivariate local optimizers

@@ -184,6 +182,9 @@

      - fixed_point : scalar fixed-point finder

+      - OpenOpt : a tool which offers a unified syntax to call this and
+         other solvers with possibility of automatic differentiation.
+
 """
    x0 = asarray(x0, dtype=float).tolist()
    n = len(x0)

_______________________________________________
Scipy-svn mailing list
Scipy-svn at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-svn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100209/05d47761/attachment.html>

From millman at berkeley.edu  Tue Feb  9 05:14:45 2010
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 9 Feb 2010 02:14:45 -0800
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
Message-ID: <c7009a551002090214ofe9a60dp2e894c35a5e316fd@mail.gmail.com>

2010/2/9 St?fan van der Walt <stefan at sun.ac.za>:
> I noticed the following changeset coming through. Do you think docstrings
> are the right place for advertising external packages?? If not, should they
> be moved to the module docstring, or removed entirely?

My vote would be to remove them.  I think the 'See Also' section
should only point to numpy/scipy/scikits code.  We can point to
external packages on the website.

-- 
Jarrod Millman
Helen Wills Neuroscience Institute
10 Giannini Hall, UC Berkeley
http://cirl.berkeley.edu/


From ralf.gommers at googlemail.com  Tue Feb  9 05:15:08 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 9 Feb 2010 18:15:08 +0800
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
Message-ID: <dde7764a1002090215r3274f79ak4a1ec360b81201a7@mail.gmail.com>

2010/2/9 St?fan van der Walt <stefan at sun.ac.za>

> Hi all,
>
> I noticed the following changeset coming through. Do you think docstrings
> are the right place for advertising external packages?  If not, should they
> be moved to the module docstring, or removed entirely?
>
>
> First, thanks to Dimitry for asking on this list before making the change.
That said, docstrings do not seem like the right place for this, module
docstring and/or tutorial seem like more appropriate places.

I do not see any reason to remove mention of OpenOpt completely, many
external packages are mentioned in our docs and OpenOpt is clearly relevant.
That it moved from scikits to some other location does not matter.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100209/f2edec99/attachment.html>

From tmp50 at ukr.net  Tue Feb  9 05:19:50 2010
From: tmp50 at ukr.net (Dmitrey)
Date: Tue, 09 Feb 2010 12:19:50 +0200
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
Message-ID: <E1NenCA-000Ai8-JR@ffe14.ukr.net>

I had asked it 2 days ago before today's commit  
http://permalink.gmane.org/gmane.comp.python.scientific.devel/12740  
why couldn't you answer earlier?  
as for mention scikits.openopt, it was allowed that time I had asked for it (about 2 years ago).  
  
In any way, of course, you can undo the changes / remove it completely / do anything else what you want.  
D.  
  
--- ???????? ????????? ---  
?? ????: St?fan van der Walt <stefan at sun.ac.za>  
????: SciPy Developers List <scipy-dev at scipy.org>  
????: 9 ???????, 11:31:12  
????: [SciPy-dev] Changes to trunk/scipy/optimize  
  
  Hi all,  
  
I noticed the following changeset coming through. Do you think docstrings  
are the right place for advertising external packages? If not, should they  
be moved to the module docstring, or removed entirely?  
  
Regards  
St?fan  
  
---------- Forwarded message ----------  
Author: dmitrey.kroshko  
Date: 2010-02-09 02:41:34 -0600 (Tue, 09 Feb 2010)  
New Revision: 6221  
  
Modified:  
trunk/scipy/optimize/anneal.py  
trunk/scipy/optimize/cobyla.py  
trunk/scipy/optimize/lbfgsb.py  
trunk/scipy/optimize/minpack.py  
trunk/scipy/optimize/nnls.py  
trunk/scipy/optimize/optimize.py  
trunk/scipy/optimize/slsqp.py  
trunk/scipy/optimize/tnc.py  
Log:  
scikits.openopt replaced by mere openopt  
  
Modified: trunk/scipy/optimize/anneal.py  
===================================================================  
--- trunk/scipy/optimize/anneal.py 2010-02-08 15:18:27 UTC (rev 6220)  
+++ trunk/scipy/optimize/anneal.py 2010-02-09 08:41:34 UTC (rev 6221)  
@@ -217,6 +217,8 @@  
  
fixed_point -- scalar fixed-point finder  
  
+ OpenOpt -- Python package with more optimization solvers  
+  
"""  
x0 = asarray(x0)  
lower = asarray(lower)  
  
Modified: trunk/scipy/optimize/cobyla.py  
===================================================================  
--- trunk/scipy/optimize/cobyla.py 2010-02-08 15:18:27 UTC (rev 6220)  
+++ trunk/scipy/optimize/cobyla.py 2010-02-09 08:41:34 UTC (rev 6221)  
@@ -46,8 +46,6 @@  
  
See also:  
  
- scikits.openopt, which offers a unified syntax to call this and  
other solvers  
-  
fmin, fmin_powell, fmin_cg,  
fmin_bfgs, fmin_ncg -- multivariate local optimizers  
leastsq -- nonlinear least squares minimizer  
@@ -65,6 +63,9 @@  
  
fixed_point -- scalar fixed-point finder  
  
+ OpenOpt -- a tool which offers a unified syntax to call this and  
+ other solvers with possibility of automatic differentiation  
+  
"""  
err = "cons must be a sequence of callable functions or a single"\  
" callable function."  
  
Modified: trunk/scipy/optimize/lbfgsb.py  
===================================================================  
--- trunk/scipy/optimize/lbfgsb.py 2010-02-08 15:18:27 UTC (rev 6220)  
+++ trunk/scipy/optimize/lbfgsb.py 2010-02-09 08:41:34 UTC (rev 6221)  
@@ -119,8 +119,6 @@  
ACM Transactions on Mathematical Software, Vol 23, Num. 4, pp. 550 -  
560.  
  
See also:  
- scikits.openopt, which offers a unified syntax to call this and  
other solvers  
-  
fmin, fmin_powell, fmin_cg,  
fmin_bfgs, fmin_ncg -- multivariate local optimizers  
leastsq -- nonlinear least squares minimizer  
@@ -138,6 +136,9 @@  
  
fixed_point -- scalar fixed-point finder  
  
+ OpenOpt -- a tool which offers a unified syntax to call this and  
+ other solvers with possibility of automatic differentiation  
+  
"""  
n = len(x0)  
  
Modified: trunk/scipy/optimize/minpack.py  
===================================================================  
--- trunk/scipy/optimize/minpack.py 2010-02-08 15:18:27 UTC (rev 6220)  
+++ trunk/scipy/optimize/minpack.py 2010-02-09 08:41:34 UTC (rev 6221)  
@@ -102,8 +102,6 @@  
  
See Also  
--------  
- scikits.openopt : offers a unified syntax to call this and other  
solvers  
-  
fmin, fmin_powell, fmin_cg, fmin_bfgs, fmin_ncg : multivariate local  
optimizers  
  
leastsq : nonlinear least squares minimizer  
@@ -118,6 +116,9 @@  
  
fixed_point : scalar and vector fixed-point finder  
  
+ OpenOpt : a tool which offers a unified syntax to call this and  
+ other solvers with possibility of automatic differentiation  
+  
"""  
if not warning :  
msg = "The warning keyword is deprecated. Use the warnings module."  
@@ -263,7 +264,6 @@  
  
See Also  
--------  
- scikits.openopt: offers a unified syntax to call this and other solvers  
fmin, fmin_powell, fmin_cg, fmin_bfgs, fmin_ncg: multivariate local  
optimizers  
fmin_l_bfgs_b, fmin_tnc, fmin_cobyla: constrained multivariate  
optimizers  
anneal, brute: global optimizers  
@@ -272,6 +272,9 @@  
brentq, brenth, ridder, bisect, newton: one-dimensional root-finding  
fixed_point: scalar and vector fixed-point finder  
curve_fit: find parameters for a curve-fitting problem.  
+ OpenOpt : a tool which offers a unified syntax to call this and  
+ other solvers with possibility of automatic differentiation  
+  
"""  
if not warning :  
msg = "The warning keyword is deprecated. Use the warnings module."  
  
Modified: trunk/scipy/optimize/nnls.py  
===================================================================  
--- trunk/scipy/optimize/nnls.py 2010-02-08 15:18:27 UTC (rev 6220)  
+++ trunk/scipy/optimize/nnls.py 2010-02-09 08:41:34 UTC (rev 6221)  
@@ -16,6 +16,8 @@  
  
wrapper around NNLS.F code below nnls/ directory  
  
+ Check OpenOpt for more LLSP solvers  
+  
"""  
  
A,b = map(asarray_chkfinite, (A,b))  
  
Modified: trunk/scipy/optimize/optimize.py  
===================================================================  
--- trunk/scipy/optimize/optimize.py 2010-02-08 15:18:27 UTC (rev 6220)  
+++ trunk/scipy/optimize/optimize.py 2010-02-09 08:41:34 UTC (rev 6221)  
@@ -156,7 +156,8 @@  
  
Uses a Nelder-Mead simplex algorithm to find the minimum of  
function of one or more variables.  
-  
+ Check OpenOpt - a tool which offers a unified syntax to call  
+ this and other solvers with possibility of automatic  
differentiation.  
"""  
fcalls, func = wrap_function(func, args)  
x0 = asfarray(x0).flatten()  
@@ -694,8 +695,8 @@  
  
*See Also*:  
  
- scikits.openopt : SciKit which offers a unified syntax to call  
- this and other solvers.  
+ OpenOpt : a tool which offers a unified syntax to call  
+ this and other solvers with possibility of automatic  
differentiation.  
  
"""  
x0 = asarray(x0).squeeze()  
@@ -862,7 +863,8 @@  
using the nonlinear conjugate gradient algorithm of Polak and  
Ribiere See Wright, and Nocedal 'Numerical Optimization',  
1999, pg. 120-122.  
-  
+ Check OpenOpt - a tool which offers a unified syntax to call  
+ this and other solvers with possibility of automatic  
differentiation.  
"""  
x0 = asarray(x0).flatten()  
if maxiter is None:  
@@ -1018,8 +1020,7 @@  
If True, return a list of results at each iteration.  
  
:Notes:  
- 1. scikits.openopt offers a unified syntax to call this and other  
solvers.  
- 2. Only one of `fhess_p` or `fhess` need to be given. If `fhess`  
+ 1. Only one of `fhess_p` or `fhess` need to be given. If `fhess`  
is provided, then `fhess_p` will be ignored. If neither `fhess`  
nor `fhess_p` is provided, then the hessian product will be  
approximated using finite differences on `fprime`. `fhess_p`  
@@ -1027,6 +1028,8 @@  
given, finite-differences on `fprime` are used to compute  
it. See Wright, and Nocedal 'Numerical Optimization', 1999,  
pg. 140.  
+ 2. Check OpenOpt - a tool which offers a unified syntax to call  
+ this and other solvers with possibility of automatic differentiation.  
  
"""  
x0 = asarray(x0).flatten()  
@@ -1179,8 +1182,9 @@  
Finds a local minimizer of the scalar function `func` in the  
interval x1 < xopt < x2 using Brent's method. (See `brent`  
for auto-bracketing).  
+ Check OpenOpt - a tool which offers a unified syntax to call  
+ this and other solvers with possibility of automatic  
differentiation.  
  
-  
"""  
# Test bounds are of correct form  
  
@@ -1722,7 +1726,8 @@  
  
Uses a modification of Powell's method to find the minimum of  
a function of N variables.  
-  
+ Check OpenOpt - a tool which offers a unified syntax to call  
+ this and other solvers with possibility of automatic  
differentiation.  
"""  
# we need to use a mutable object here that we can update in the  
# wrapper function  
  
Modified: trunk/scipy/optimize/slsqp.py  
===================================================================  
--- trunk/scipy/optimize/slsqp.py 2010-02-08 15:18:27 UTC (rev 6220)  
+++ trunk/scipy/optimize/slsqp.py 2010-02-09 08:41:34 UTC (rev 6221)  
@@ -146,6 +146,11 @@  
  
for examples see :ref:`in the tutorial <tutorial-sqlsp>`  
  
+ See also  
+ --------  
+ OpenOpt - a tool which offers a unified syntax to call this  
+ and other solvers with possibility of automatic differentiation.  
+  
"""  
  
exit_modes = { -1 : "Gradient evaluation required (g & a)",  
  
Modified: trunk/scipy/optimize/tnc.py  
===================================================================  
--- trunk/scipy/optimize/tnc.py 2010-02-08 15:18:27 UTC (rev 6220)  
+++ trunk/scipy/optimize/tnc.py 2010-02-09 08:41:34 UTC (rev 6221)  
@@ -164,8 +164,6 @@  
Return code as defined in the RCSTRINGS dict.  
  
:SeeAlso:  
- - scikits.openopt, which offers a unified syntax to call this and  
other solvers  
-  
- fmin, fmin_powell, fmin_cg, fmin_bfgs, fmin_ncg :  
multivariate local optimizers  
  
@@ -184,6 +182,9 @@  
  
- fixed_point : scalar fixed-point finder  
  
+ - OpenOpt : a tool which offers a unified syntax to call this and  
+ other solvers with possibility of automatic differentiation.  
+  
"""  
x0 = asarray(x0, dtype=float).tolist()  
n = len(x0)  
  
_______________________________________________  
Scipy-svn mailing list  
Scipy-svn at scipy.org  
http://mail.scipy.org/mailman/listinfo/scipy-svn  
  
_______________________________________________  
SciPy-Dev mailing list  
SciPy-Dev at scipy.org  
http://mail.scipy.org/mailman/listinfo/scipy-dev  
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100209/343879ac/attachment.html>

From millman at berkeley.edu  Tue Feb  9 06:05:41 2010
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 9 Feb 2010 03:05:41 -0800
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <E1NenCA-000Ai8-JR@ffe14.ukr.net>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
	<E1NenCA-000Ai8-JR@ffe14.ukr.net>
Message-ID: <c7009a551002090305x5c33edd6h6b8b6361281ba894@mail.gmail.com>

2010/2/9 Dmitrey <tmp50 at ukr.net>:
> I had asked it 2 days ago before today's commit
> http://permalink.gmane.org/gmane.comp.python.scientific.devel/12740
> why couldn't you answer earlier?

Thanks for raising the issue to the list.  Sorry, I missed your
original email and I am sure Stefan must have missed it as well.
Please don't take this discussion as in any way an attempt to
discourage people from using OpenOpt or suggesting you did anything
wrong.  I know that OpenOpt is extremely useful and you've done a
great job developing it.

> as for mention scikits.openopt, it was allowed that time I had asked for it
> (about 2 years ago).

Just to be clear, I think Stefan was asking a more general question
that arose due to this specific instance.  The OpenOpt situation is a
bit unique since it was originally a Google SoC project for SciPy, but
after funding ran out became a successful stand-alone project of its
own.

Now the question is where is the best place for us to reference
external, but relevant and useful Python projects.  My personal
feeling is that it shouldn't be in the 'see also' section of our
docstrings, but we don't have any official policy on that yet.  So we
need to have a general discussion about what the general policy should
be.

Personally I would say that the primary place to point to external
packages is in the topical software section of website.  For instance,
OpenOpt is pointed to here:
http://www.scipy.org/Topical_Software#head-d21a11d2d173826993e03eb937fac7e6347e6d5f
I also think it would be fine to occasionally use external packages in
the tutorials if deemed useful.  But, in general, I would expect
external packages to have their own tutorials.  I would prefer to
limit the docstrings to just our core projects (numpy and scipy for
certain and perhaps the scikits as well).  If we don't limit the
docstrings in this way, I can see us either 1) getting in the
situation where it isn't clear how much more we should add to the
docstrings for the sake of completeness or 2) inadvertently getting
into political battles by appearing to favor certain external projects
while not mentioning others.

I am very interested in hearing what everyone else thinks about this
issue.  However, I think it would be most useful to discuss this in
general, rather than with a focus on openopt.  So if we decide not to
reference external packages in scipy and it turns out that we
reference several others in addition to openopt, then we should apply
the same standard to all the cases.

Best,
Jarrod


From luethi at vaw.baug.ethz.ch  Tue Feb  9 07:22:28 2010
From: luethi at vaw.baug.ethz.ch (Martin =?ISO-8859-1?Q?L=FCthi?=)
Date: Tue, 09 Feb 2010 13:22:28 +0100
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <c7009a551002090305x5c33edd6h6b8b6361281ba894@mail.gmail.com>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
	<E1NenCA-000Ai8-JR@ffe14.ukr.net>
	<c7009a551002090305x5c33edd6h6b8b6361281ba894@mail.gmail.com>
Message-ID: <87mxziegzf.wl%luethi@vaw.baug.ethz.ch>

Hi

At Tue, 9 Feb 2010 03:05:41 -0800,
Jarrod Millman wrote:
> Now the question is where is the best place for us to reference
> external, but relevant and useful Python projects.  My personal
> feeling is that it shouldn't be in the 'see also' section of our
> docstrings, but we don't have any official policy on that yet.  So we
> need to have a general discussion about what the general policy should
> be.

I completely understand your reasoning. However, I would still find it useful
if there were a docstring section named "similar packages", mentioning either of

a) relevant packages that are closely painfully integrated with scipy and are
   generally regarded as useful

b) a link to the relevant URL of the topical software guide.

Option (a) might lead to some "political" problems, which I cannot imagine
being difficult. Option (b) would be useful in any case. While checking out
code the Topical Software Pages or the Scipy Wiki are not the immediately
obvious places to look for more relevant information.

Best, Martin


From josef.pktd at gmail.com  Tue Feb  9 09:31:35 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 9 Feb 2010 09:31:35 -0500
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <87mxziegzf.wl%luethi@vaw.baug.ethz.ch>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
	<E1NenCA-000Ai8-JR@ffe14.ukr.net>
	<c7009a551002090305x5c33edd6h6b8b6361281ba894@mail.gmail.com>
	<87mxziegzf.wl%luethi@vaw.baug.ethz.ch>
Message-ID: <1cd32cbb1002090631q1d2623cbwda6216f3e1b34a97@mail.gmail.com>

2010/2/9 Martin L?thi <luethi at vaw.baug.ethz.ch>:
> Hi
>
> At Tue, 9 Feb 2010 03:05:41 -0800,
> Jarrod Millman wrote:
>> Now the question is where is the best place for us to reference
>> external, but relevant and useful Python projects. ?My personal
>> feeling is that it shouldn't be in the 'see also' section of our
>> docstrings, but we don't have any official policy on that yet. ?So we
>> need to have a general discussion about what the general policy should
>> be.
>
> I completely understand your reasoning. However, I would still find it useful
> if there were a docstring section named "similar packages", mentioning either of
>
> a) relevant packages that are closely painfully integrated with scipy and are
> ? generally regarded as useful
>
> b) a link to the relevant URL of the topical software guide.
>
> Option (a) might lead to some "political" problems, which I cannot imagine
> being difficult. Option (b) would be useful in any case. While checking out
> code the Topical Software Pages or the Scipy Wiki are not the immediately
> obvious places to look for more relevant information.

I think this would be useful on the module or scipy subpackage level
but not for the docstrings for individual functions or classes. In my
opinion, the docstring of function should provide only closely related
functions and classes for quick lookup. Extra information should go
into a different section, for example in the statsmodels docs, I
included a page with related packages.

For the specific case of scipy.optimize, I think also some of the
internal See Also can be removed, since many of the docstrings are
almost a table of content of scipy.optimize, which also can be
directly seen on the subpackage level. If I remember correctly, in
some parts of scipy some weeding of umbrella SeeAlso has already
occured.

On Wiki versus documentation pages, I think it would be helpful to
introduce additional descriptive sections on the scipy.subpackage
level, which could include specific Topical Software.

I'm using the numpy/scipy docs mainly through htmlhelp (.chm) where
the tree view makes browsing very fast (similar to matlab help), maybe
the html pages need more links than this.

Josef


>
> Best, Martin
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From robert.kern at gmail.com  Tue Feb  9 10:30:44 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 9 Feb 2010 09:30:44 -0600
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <1cd32cbb1002090631q1d2623cbwda6216f3e1b34a97@mail.gmail.com>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com> 
	<E1NenCA-000Ai8-JR@ffe14.ukr.net>
	<c7009a551002090305x5c33edd6h6b8b6361281ba894@mail.gmail.com> 
	<87mxziegzf.wl%luethi@vaw.baug.ethz.ch>
	<1cd32cbb1002090631q1d2623cbwda6216f3e1b34a97@mail.gmail.com>
Message-ID: <3d375d731002090730i7bc7da2eybda44eee094b1dde@mail.gmail.com>

On Tue, Feb 9, 2010 at 08:31,  <josef.pktd at gmail.com> wrote:
> 2010/2/9 Martin L?thi <luethi at vaw.baug.ethz.ch>:
>> Hi
>>
>> At Tue, 9 Feb 2010 03:05:41 -0800,
>> Jarrod Millman wrote:
>>> Now the question is where is the best place for us to reference
>>> external, but relevant and useful Python projects. ?My personal
>>> feeling is that it shouldn't be in the 'see also' section of our
>>> docstrings, but we don't have any official policy on that yet. ?So we
>>> need to have a general discussion about what the general policy should
>>> be.
>>
>> I completely understand your reasoning. However, I would still find it useful
>> if there were a docstring section named "similar packages", mentioning either of
>>
>> a) relevant packages that are closely painfully integrated with scipy and are
>> ? generally regarded as useful
>>
>> b) a link to the relevant URL of the topical software guide.
>>
>> Option (a) might lead to some "political" problems, which I cannot imagine
>> being difficult. Option (b) would be useful in any case. While checking out
>> code the Topical Software Pages or the Scipy Wiki are not the immediately
>> obvious places to look for more relevant information.
>
> I think this would be useful on the module or scipy subpackage level
> but not for the docstrings for individual functions or classes. In my
> opinion, the docstring of function should provide only closely related
> functions and classes for quick lookup. Extra information should go
> into a different section, for example in the statsmodels docs, I
> included a page with related packages.

+1

> For the specific case of scipy.optimize, I think also some of the
> internal See Also can be removed, since many of the docstrings are
> almost a table of content of scipy.optimize, which also can be
> directly seen on the subpackage level. If I remember correctly, in
> some parts of scipy some weeding of umbrella SeeAlso has already
> occured.

+1

> On Wiki versus documentation pages, I think it would be helpful to
> introduce additional descriptive sections on the scipy.subpackage
> level, which could include specific Topical Software.

+0.5

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jsseabold at gmail.com  Tue Feb  9 11:03:07 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 9 Feb 2010 11:03:07 -0500
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <1cd32cbb1002090631q1d2623cbwda6216f3e1b34a97@mail.gmail.com>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com> 
	<E1NenCA-000Ai8-JR@ffe14.ukr.net>
	<c7009a551002090305x5c33edd6h6b8b6361281ba894@mail.gmail.com> 
	<87mxziegzf.wl%luethi@vaw.baug.ethz.ch>
	<1cd32cbb1002090631q1d2623cbwda6216f3e1b34a97@mail.gmail.com>
Message-ID: <c048da1c1002090803p155ef172t2053dfec76a9b752@mail.gmail.com>

On Tue, Feb 9, 2010 at 9:31 AM,  <josef.pktd at gmail.com> wrote:
> 2010/2/9 Martin L?thi <luethi at vaw.baug.ethz.ch>:
>> Hi
>>
>> At Tue, 9 Feb 2010 03:05:41 -0800,
>> Jarrod Millman wrote:
>>> Now the question is where is the best place for us to reference
>>> external, but relevant and useful Python projects. ?My personal
>>> feeling is that it shouldn't be in the 'see also' section of our
>>> docstrings, but we don't have any official policy on that yet. ?So we
>>> need to have a general discussion about what the general policy should
>>> be.
>>
>> I completely understand your reasoning. However, I would still find it useful
>> if there were a docstring section named "similar packages", mentioning either of
>>
>> a) relevant packages that are closely painfully integrated with scipy and are
>> ? generally regarded as useful
>>
>> b) a link to the relevant URL of the topical software guide.
>>
>> Option (a) might lead to some "political" problems, which I cannot imagine
>> being difficult. Option (b) would be useful in any case. While checking out
>> code the Topical Software Pages or the Scipy Wiki are not the immediately
>> obvious places to look for more relevant information.
>

>

+1 to (b).  While I think (a), would be of more use, it could be
difficult to keep such sections clean, short, and relevant, but having
them some other place that's readily accessible is a good compromise.

I've been hearing some very mild push back against using Python from
others recently, because it's a "dark horse."  Whether or not this is
a fair moniker is irrelevant, but I think part of this is because it
doesn't feel like a totally integrated environment for problem solving
across packages (the fact that Python is much more general than this
seems to get lost...).  The question then is, is scipy the place to
act as a gateway for scientific problem solving writ large?  I'd say
yes...

> I think this would be useful on the module or scipy subpackage level
> but not for the docstrings for individual functions or classes. In my
> opinion, the docstring of function should provide only closely related
> functions and classes for quick lookup. Extra information should go
> into a different section, for example in the statsmodels docs, I
> included a page with related packages.
>

+1 Fair compromise.

> For the specific case of scipy.optimize, I think also some of the
> internal See Also can be removed, since many of the docstrings are
> almost a table of content of scipy.optimize, which also can be
> directly seen on the subpackage level. If I remember correctly, in
> some parts of scipy some weeding of umbrella SeeAlso has already
> occured.
>
> On Wiki versus documentation pages, I think it would be helpful to
> introduce additional descriptive sections on the scipy.subpackage
> level, which could include specific Topical Software.
>
> I'm using the numpy/scipy docs mainly through htmlhelp (.chm) where
> the tree view makes browsing very fast (similar to matlab help), maybe
> the html pages need more links than this.
>
> Josef
>
>
>>
>> Best, Martin

Skipper


From matthew.brett at gmail.com  Tue Feb  9 13:24:30 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 9 Feb 2010 10:24:30 -0800
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <3d375d731002090730i7bc7da2eybda44eee094b1dde@mail.gmail.com>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
	<E1NenCA-000Ai8-JR@ffe14.ukr.net>
	<c7009a551002090305x5c33edd6h6b8b6361281ba894@mail.gmail.com>
	<87mxziegzf.wl%luethi@vaw.baug.ethz.ch>
	<1cd32cbb1002090631q1d2623cbwda6216f3e1b34a97@mail.gmail.com>
	<3d375d731002090730i7bc7da2eybda44eee094b1dde@mail.gmail.com>
Message-ID: <1e2af89e1002091024hbfc2d01q6ef3e7a7eadfaae2@mail.gmail.com>

Hi,

>> I think this would be useful on the module or scipy subpackage level
>> but not for the docstrings for individual functions or classes. In my
>> opinion, the docstring of function should provide only closely related
>> functions and classes for quick lookup. Extra information should go
>> into a different section, for example in the statsmodels docs, I
>> included a page with related packages.
>
> +1

That seems a sensible suggestion.

Matthew


From rob.clewley at gmail.com  Tue Feb  9 15:49:07 2010
From: rob.clewley at gmail.com (Rob Clewley)
Date: Tue, 9 Feb 2010 15:49:07 -0500
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <87mxziegzf.wl%luethi@vaw.baug.ethz.ch>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com>
	<E1NenCA-000Ai8-JR@ffe14.ukr.net>
	<c7009a551002090305x5c33edd6h6b8b6361281ba894@mail.gmail.com>
	<87mxziegzf.wl%luethi@vaw.baug.ethz.ch>
Message-ID: <a749952d1002091249w24a6f48ao485e8d9b9cd2a240@mail.gmail.com>

> I completely understand your reasoning. However, I would still find it useful
> if there were a docstring section named "similar packages", mentioning either of
>
> a) relevant packages that are closely painfully integrated with scipy and are
> ? generally regarded as useful
>
> b) a link to the relevant URL of the topical software guide.
>
> Option (a) might lead to some "political" problems, which I cannot imagine
> being difficult. Option (b) would be useful in any case. While checking out
> code the Topical Software Pages or the Scipy Wiki are not the immediately
> obvious places to look for more relevant information.

+1 for (b). It achieves the same thing as (a) with only minimal
additional inconvenience to the user but much greater convenience to
developers of the external codes and to the maintainers of the scipy
code. There is already a one stop shop for this info at the Topical
Software wiki page, and this just has to be better advertised IMO.

-Rob


From david at silveregg.co.jp  Tue Feb  9 21:12:14 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Wed, 10 Feb 2010 11:12:14 +0900
Subject: [SciPy-dev] Latex and docstrings
Message-ID: <4B7215FE.5030803@silveregg.co.jp>

Hi,

I noticed that some of the docstrings I have written for DCT have been 
changed to latex format. While I have no issue with having latex in the 
documentation, I thought the consensus was to use them sparingly in 
docstrings ?

For example, the dct I formula used to be (fixed width font assumed):

for 0 <= k < N,

                                    N-1
y[k] = x[0] + (-1)**k x[N-1] + 2 * sum x[n]*cos(pi*k*n/(N-1))
                                    n=0

But now, it is:

y_k = x_0 + (-1)^k x_{N-1} + 2\\sum_{n=1}^{N-2} x_n
         \\cos\\left({\\pi nk\\over N-1}\\right),
         \\qquad 0 \\le k < N.

I much prefer the former (the latter is unreadable in a terminal IMO). I 
have of course no issue in putting the latex formula in the scipy docs,

thanks,

David


From robertlayton at gmail.com  Tue Feb  9 23:38:56 2010
From: robertlayton at gmail.com (Robert Layton)
Date: Wed, 10 Feb 2010 15:38:56 +1100
Subject: [SciPy-dev] Ticket #467 pending decision for 3 months
Message-ID: <585dc5e21002092038j3678d516ma01c79c10c5831d3@mail.gmail.com>

I submitted a fix for ticket #467 a while ago, which is quite a simple fix.

As scipy's mean and std functions are now passing through to numpy, there is
little reason to test them as part of scipy (the appropriate tests should be
in numpy).
Even if its decided that the tests should be retained, theres a patch for
that (r) as well.

Thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100210/d2ce1984/attachment.html>

From stefan at sun.ac.za  Wed Feb 10 02:44:36 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 10 Feb 2010 09:44:36 +0200
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <E1NenCA-000Ai8-JR@ffe14.ukr.net>
References: <9457e7c81002090131v148a86d1w1eef996c9f3c19a2@mail.gmail.com> 
	<E1NenCA-000Ai8-JR@ffe14.ukr.net>
Message-ID: <9457e7c81002092344i72886a26gcee3460782e4647e@mail.gmail.com>

2010/2/9 Dmitrey <tmp50 at ukr.net>:
> I had asked it 2 days ago before today's commit
> http://permalink.gmane.org/gmane.comp.python.scientific.devel/12740
> why couldn't you answer earlier?

Sorry, Dmitrey, I didn't see your earlier message.  I included a link
to OpenOpt in the optimize module docstring, and removed
the See Also sections entirely (they were difficult to maintain, so I
would have removed them either way).  I hope you find this solution to
your satisfaction.

Kind regards
St?fan


From stefan at sun.ac.za  Wed Feb 10 03:18:27 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 10 Feb 2010 10:18:27 +0200
Subject: [SciPy-dev] Ticket #467 pending decision for 3 months
In-Reply-To: <585dc5e21002092038j3678d516ma01c79c10c5831d3@mail.gmail.com>
References: <585dc5e21002092038j3678d516ma01c79c10c5831d3@mail.gmail.com>
Message-ID: <9457e7c81002100018k69022e86n967f052dc6dc13c8@mail.gmail.com>

Hi Robert

On 10 February 2010 06:38, Robert Layton <robertlayton at gmail.com> wrote:
> I submitted a fix for ticket #467 a while ago, which is quite a simple fix.
>
> As scipy's mean and std functions are now passing through to numpy, there is
> little reason to test them as part of scipy (the appropriate tests should be
> in numpy).
> Even if its decided that the tests should be retained, theres a patch for
> that (r) as well.

Sorry for not paying attention to this earlier.  I think we should
remove tests that only validate numpy's behaviour, so your
`without_numpy.patch' looks good.  Unfortunately, it doesn't apply
cleanly; would you have a chance to look at it again?

Thanks,
St?fan


From pav+sp at iki.fi  Wed Feb 10 04:48:51 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Wed, 10 Feb 2010 09:48:51 +0000 (UTC)
Subject: [SciPy-dev] Latex and docstrings
References: <4B7215FE.5030803@silveregg.co.jp>
Message-ID: <hktve3$8c7$1@ger.gmane.org>

Wed, 10 Feb 2010 11:12:14 +0900, David Cournapeau wrote:
[clip]
> for 0 <= k < N,
> 
>                                     N-1
> y[k] = x[0] + (-1)**k x[N-1] + 2 * sum x[n]*cos(pi*k*n/(N-1))
>                                     n=0
> 
> But now, it is:
> 
> y_k = x_0 + (-1)^k x_{N-1} + 2\\sum_{n=1}^{N-2} x_n
>          \\cos\\left({\\pi nk\\over N-1}\\right), \\qquad 0 \\le k < N.
> 
> I much prefer the former (the latter is unreadable in a terminal IMO). I
> have of course no issue in putting the latex formula in the scipy docs,

Someone could try to write a "simple math" plugin for Sphinx that 
converted poor man's math notation to Latex.

Cheers,
Pauli



From stefan at sun.ac.za  Wed Feb 10 05:24:16 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 10 Feb 2010 12:24:16 +0200
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <hktve3$8c7$1@ger.gmane.org>
References: <4B7215FE.5030803@silveregg.co.jp> <hktve3$8c7$1@ger.gmane.org>
Message-ID: <9457e7c81002100224s261b648u6c5a81412aa55448@mail.gmail.com>

On 10 February 2010 11:48, Pauli Virtanen <pav+sp at iki.fi> wrote:
>> I much prefer the former (the latter is unreadable in a terminal IMO). I
>> have of course no issue in putting the latex formula in the scipy docs,
>
> Someone could try to write a "simple math" plugin for Sphinx that
> converted poor man's math notation to Latex.

We could look at AsciiMathML for inspiration:

http://asciimathml.sourceforge.net/

Regards
St?fan


From dagss at student.matnat.uio.no  Wed Feb 10 06:30:16 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Feb 2010 12:30:16 +0100
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <9457e7c81002100224s261b648u6c5a81412aa55448@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp> <hktve3$8c7$1@ger.gmane.org>
	<9457e7c81002100224s261b648u6c5a81412aa55448@mail.gmail.com>
Message-ID: <4B7298C8.30000@student.matnat.uio.no>

St?fan van der Walt wrote:
> On 10 February 2010 11:48, Pauli Virtanen <pav+sp at iki.fi> wrote:
>   
>>> I much prefer the former (the latter is unreadable in a terminal IMO). I
>>> have of course no issue in putting the latex formula in the scipy docs,
>>>       
>> Someone could try to write a "simple math" plugin for Sphinx that
>> converted poor man's math notation to Latex.
>>     
>
> We could look at AsciiMathML for inspiration:
>
> http://asciimathml.sourceforge.net/
>   
+1, that looks great.

One should ban expressions like "a/b/c/d" though -- apparently 
AsciiMathML typesets that as (a/b)/(c/d), while in Python that would be 
a/(b*c*d).

Dag Sverre


From josef.pktd at gmail.com  Wed Feb 10 10:44:55 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 10 Feb 2010 10:44:55 -0500
Subject: [SciPy-dev] Ticket #467 pending decision for 3 months
In-Reply-To: <9457e7c81002100018k69022e86n967f052dc6dc13c8@mail.gmail.com>
References: <585dc5e21002092038j3678d516ma01c79c10c5831d3@mail.gmail.com>
	<9457e7c81002100018k69022e86n967f052dc6dc13c8@mail.gmail.com>
Message-ID: <1cd32cbb1002100744i47c5b87bv91f46f6e65b597f5@mail.gmail.com>

2010/2/10 St?fan van der Walt <stefan at sun.ac.za>:
> Hi Robert
>
> On 10 February 2010 06:38, Robert Layton <robertlayton at gmail.com> wrote:
>> I submitted a fix for ticket #467 a while ago, which is quite a simple fix.
>>
>> As scipy's mean and std functions are now passing through to numpy, there is
>> little reason to test them as part of scipy (the appropriate tests should be
>> in numpy).
>> Even if its decided that the tests should be retained, theres a patch for
>> that (r) as well.
>
> Sorry for not paying attention to this earlier. ?I think we should
> remove tests that only validate numpy's behaviour, so your
> `without_numpy.patch' looks good. ?Unfortunately, it doesn't apply
> cleanly; would you have a chance to look at it again?

this was sitting in my drafts folder since November

'''
Sorry, for not replying earlier, I have seen your patches before but I
don't know
what I would prefer.

I agree that numpy functions should be tested in numpy. On the other hand,
the stats tests already include additional test matrices, that can be used
to check the precision of the numpy functions. And I would like to

(As an example, numpy random is mostly tested in scipy.stats since there
pdf, pmf and cdf of the distributions are available.)
''''

The point for stats is that I didn't find any precision test in the
numpy test suite for mean, var and so on.

When I wrote the anova tests using the NIST reference cases,
numpy.mean did pretty badly for the badly scaled test cases. I never
checked the NIST test cases specifically for mean, var, ...

I still don't know where precision tests should be, but the outcome
would be very useful. In ANOVA I ended up calculating the mean twice
(if the dataset is badly scaled) to pass the NIST test.

Josef
(google is not very informative about donkeys and piles of hay)



>
> Thanks,
> St?fan
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From josef.pktd at gmail.com  Wed Feb 10 11:03:42 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 10 Feb 2010 11:03:42 -0500
Subject: [SciPy-dev] Ticket #467 pending decision for 3 months
In-Reply-To: <1cd32cbb1002100744i47c5b87bv91f46f6e65b597f5@mail.gmail.com>
References: <585dc5e21002092038j3678d516ma01c79c10c5831d3@mail.gmail.com>
	<9457e7c81002100018k69022e86n967f052dc6dc13c8@mail.gmail.com>
	<1cd32cbb1002100744i47c5b87bv91f46f6e65b597f5@mail.gmail.com>
Message-ID: <1cd32cbb1002100803i4d3b73fcle72db01f6c3c7b37@mail.gmail.com>

On Wed, Feb 10, 2010 at 10:44 AM,  <josef.pktd at gmail.com> wrote:
> 2010/2/10 St?fan van der Walt <stefan at sun.ac.za>:
>> Hi Robert
>>
>> On 10 February 2010 06:38, Robert Layton <robertlayton at gmail.com> wrote:
>>> I submitted a fix for ticket #467 a while ago, which is quite a simple fix.
>>>
>>> As scipy's mean and std functions are now passing through to numpy, there is
>>> little reason to test them as part of scipy (the appropriate tests should be
>>> in numpy).
>>> Even if its decided that the tests should be retained, theres a patch for
>>> that (r) as well.
>>
>> Sorry for not paying attention to this earlier. ?I think we should
>> remove tests that only validate numpy's behaviour, so your
>> `without_numpy.patch' looks good. ?Unfortunately, it doesn't apply
>> cleanly; would you have a chance to look at it again?
>
> this was sitting in my drafts folder since November
>
> '''
> Sorry, for not replying earlier, I have seen your patches before but I
> don't know
> what I would prefer.
>
> I agree that numpy functions should be tested in numpy. On the other hand,
> the stats tests already include additional test matrices, that can be used
> to check the precision of the numpy functions. And I would like to
>
> (As an example, numpy random is mostly tested in scipy.stats since there
> pdf, pmf and cdf of the distributions are available.)
> ''''
>
> The point for stats is that I didn't find any precision test in the
> numpy test suite for mean, var and so on.
>
> When I wrote the anova tests using the NIST reference cases,
> numpy.mean did pretty badly for the badly scaled test cases. I never
> checked the NIST test cases specifically for mean, var, ...
>
> I still don't know where precision tests should be, but the outcome
> would be very useful. In ANOVA I ended up calculating the mean twice
> (if the dataset is badly scaled) to pass the NIST test.
>
> Josef
> (google is not very informative about donkeys and piles of hay)
>

Speaking of piles of hay:

http://projects.scipy.org/scipy/ticket/999    (O(n log(n)) )  in python
http://projects.scipy.org/scipy/ticket/893    )(n**2)  (I think) in cython

I don't have much experience guessing which would perform better and
no time for comparing them.

One of the two should go into scipy, since both are much better than
the current implementation.

Which?

Josef


>
>
>>
>> Thanks,
>> St?fan
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>


From robertlayton at gmail.com  Wed Feb 10 16:54:03 2010
From: robertlayton at gmail.com (Robert Layton)
Date: Thu, 11 Feb 2010 08:54:03 +1100
Subject: [SciPy-dev] SciPy-Dev Digest, Vol 76, Issue 14
In-Reply-To: <mailman.4912.1265817834.2454.scipy-dev@scipy.org>
References: <mailman.4912.1265817834.2454.scipy-dev@scipy.org>
Message-ID: <585dc5e21002101354v187b7e04m7f089ba1d2e3f4e4@mail.gmail.com>

>2010/2/10 St?fan van der Walt <stefan at sun.ac.za>:
>> Hi Robert
>>
>> On 10 February 2010 06:38, Robert Layton <robertlayton at gmail.com> wrote:
>>> I submitted a fix for ticket #467 a while ago, which is quite a simple
fix.
>>>
>>> As scipy's mean and std functions are now passing through to numpy,
there is
>>> little reason to test them as part of scipy (the appropriate tests
should be
>>> in numpy).
>>> Even if its decided that the tests should be retained, theres a patch
for
>>> that (r) as well.
>>
>> Sorry for not paying attention to this earlier. ?I think we should
>> remove tests that only validate numpy's behaviour, so your
>> `without_numpy.patch' looks good. ?Unfortunately, it doesn't apply
>> cleanly; would you have a chance to look at it again?
>
>this was sitting in my drafts folder since November
>
>'''
>Sorry, for not replying earlier, I have seen your patches before but I
>don't know
>what I would prefer.
>
>I agree that numpy functions should be tested in numpy. On the other hand,
>the stats tests already include additional test matrices, that can be used
>to check the precision of the numpy functions. And I would like to
>
>(As an example, numpy random is mostly tested in scipy.stats since there
>pdf, pmf and cdf of the distributions are available.)
>''''
>
>The point for stats is that I didn't find any precision test in the
>numpy test suite for mean, var and so on.
>
>When I wrote the anova tests using the NIST reference cases,
>numpy.mean did pretty badly for the badly scaled test cases. I never
>checked the NIST test cases specifically for mean, var, ...
>
>I still don't know where precision tests should be, but the outcome
>would be very useful. In ANOVA I ended up calculating the mean twice
>(if the dataset is badly scaled) to pass the NIST test.
>
>Josef
>(google is not very informative about donkeys and piles of hay)
>


I have updated the patch and it now applys to the current SVN version.
As I stated in the ticket, there is a problem with my build environment with
a different test (test_ltisys.TestSS2TF.test_basic(0, 3, 3) ...  ** On entry
to DGEEV  parameter number  5 had an illegal value ) so I can't verify that
the test works, but it does build and install.

Thanks

Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100211/ab531985/attachment.html>

From robert.kern at gmail.com  Wed Feb 10 17:48:28 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 10 Feb 2010 16:48:28 -0600
Subject: [SciPy-dev] Web space for docstring-linked PDFs?
Message-ID: <3d375d731002101448x6c817043sd9eb6351455c643b@mail.gmail.com>

I would like to make the PDFs of the ODRPACK User's Guide available
somewhere on (docs.)scipy.org in order to provide a stable URL for
them. The URL currently in the scipy.odr docstring is now broken. I
currently have them on my own web space, but it would be nice to have
them available somewhere stable on scipy.org (although not as a wiki
attachment). I am now getting an email every month or so telling me
that the link is broken. Is there a reasonable place in the
docs.scipy.org infrastructure where I can place these static files
such that they will be backed up and eventually migrated along with
the rest of the infrastructure as it evolves?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Thu Feb 11 01:41:45 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 11 Feb 2010 08:41:45 +0200
Subject: [SciPy-dev] Web space for docstring-linked PDFs?
In-Reply-To: <3d375d731002101448x6c817043sd9eb6351455c643b@mail.gmail.com>
References: <3d375d731002101448x6c817043sd9eb6351455c643b@mail.gmail.com>
Message-ID: <9457e7c81002102241n57ffff69w26c1592dbd411437@mail.gmail.com>

On 11 February 2010 00:48, Robert Kern <robert.kern at gmail.com> wrote:
> I would like to make the PDFs of the ODRPACK User's Guide available
> somewhere on (docs.)scipy.org in order to provide a stable URL for
> them. The URL currently in the scipy.odr docstring is now broken. I
> currently have them on my own web space, but it would be nice to have
> them available somewhere stable on scipy.org (although not as a wiki
> attachment). I am now getting an email every month or so telling me
> that the link is broken. Is there a reasonable place in the
> docs.scipy.org infrastructure where I can place these static files
> such that they will be backed up and eventually migrated along with
> the rest of the infrastructure as it evolves?

If the files are not to big, let's add them to the SVN repo under either

/trunk/doc/frontpage or the new
/scipy.org

The new scipy site can be previewed at new.scipy.org.

Regards
St?fan


From dwf at cs.toronto.edu  Thu Feb 11 02:20:24 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Thu, 11 Feb 2010 02:20:24 -0500
Subject: [SciPy-dev] Web space for docstring-linked PDFs?
In-Reply-To: <3d375d731002101448x6c817043sd9eb6351455c643b@mail.gmail.com>
References: <3d375d731002101448x6c817043sd9eb6351455c643b@mail.gmail.com>
Message-ID: <93CB8BF7-4DEC-4FAA-A2B3-7F54385ED863@cs.toronto.edu>

On 10-Feb-10, at 5:48 PM, Robert Kern wrote:

> I would like to make the PDFs of the ODRPACK User's Guide available
> somewhere on (docs.)scipy.org in order to provide a stable URL for
> them. The URL currently in the scipy.odr docstring is now broken. I
> currently have them on my own web space, but it would be nice to have
> them available somewhere stable on scipy.org (although not as a wiki
> attachment). I am now getting an email every month or so telling me
> that the link is broken. Is there a reasonable place in the
> docs.scipy.org infrastructure where I can place these static files
> such that they will be backed up and eventually migrated along with
> the rest of the infrastructure as it evolves?

I don't have an answer but I think it's an important question. A  
number of times in the scipy.org-Sphinx effort I've been worried about  
where to stick old wiki attachments and how to link to them so that  
those links will be fairly stable for years to come. Wiki attachments  
are nice in this way but the wiki has other problems...

David


From robert.kern at gmail.com  Thu Feb 11 10:37:16 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 11 Feb 2010 09:37:16 -0600
Subject: [SciPy-dev] Web space for docstring-linked PDFs?
In-Reply-To: <9457e7c81002102241n57ffff69w26c1592dbd411437@mail.gmail.com>
References: <3d375d731002101448x6c817043sd9eb6351455c643b@mail.gmail.com> 
	<9457e7c81002102241n57ffff69w26c1592dbd411437@mail.gmail.com>
Message-ID: <3d375d731002110737i5b0d6cebo7fdfbeb52f6cdfd7@mail.gmail.com>

2010/2/11 St?fan van der Walt <stefan at sun.ac.za>:
> On 11 February 2010 00:48, Robert Kern <robert.kern at gmail.com> wrote:
>> I would like to make the PDFs of the ODRPACK User's Guide available
>> somewhere on (docs.)scipy.org in order to provide a stable URL for
>> them. The URL currently in the scipy.odr docstring is now broken. I
>> currently have them on my own web space, but it would be nice to have
>> them available somewhere stable on scipy.org (although not as a wiki
>> attachment). I am now getting an email every month or so telling me
>> that the link is broken. Is there a reasonable place in the
>> docs.scipy.org infrastructure where I can place these static files
>> such that they will be backed up and eventually migrated along with
>> the rest of the infrastructure as it evolves?
>
> If the files are not to big, let's add them to the SVN repo under either
>
> /trunk/doc/frontpage or the new
> /scipy.org
>
> The new scipy site can be previewed at new.scipy.org.

How big is too big?

% ls -lh
total 900K
-rw-r--r-- 1 rkern rkern 156K May 27  2009 odr_ams.pdf
-rw-r--r-- 1 rkern rkern 551K May 27  2009 odrpack_guide.pdf
-rw-r--r-- 1 rkern rkern 167K May 27  2009 odr_vcv.pdf

The really important one is the 551K file, though all of them are useful.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Thu Feb 11 11:27:42 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 11 Feb 2010 18:27:42 +0200
Subject: [SciPy-dev] Web space for docstring-linked PDFs?
In-Reply-To: <3d375d731002110737i5b0d6cebo7fdfbeb52f6cdfd7@mail.gmail.com>
References: <3d375d731002101448x6c817043sd9eb6351455c643b@mail.gmail.com> 
	<9457e7c81002102241n57ffff69w26c1592dbd411437@mail.gmail.com> 
	<3d375d731002110737i5b0d6cebo7fdfbeb52f6cdfd7@mail.gmail.com>
Message-ID: <9457e7c81002110827r398b128dx21f6c58738e3539d@mail.gmail.com>

On 11 February 2010 17:37, Robert Kern <robert.kern at gmail.com> wrote:
> How big is too big?
>
> % ls -lh
> total 900K
> -rw-r--r-- 1 rkern rkern 156K May 27 ?2009 odr_ams.pdf
> -rw-r--r-- 1 rkern rkern 551K May 27 ?2009 odrpack_guide.pdf
> -rw-r--r-- 1 rkern rkern 167K May 27 ?2009 odr_vcv.pdf
>
> The really important one is the 551K file, though all of them are useful.

Even here, where our internet connectivity is something not discussed
in civilised conversation, 500K is quite manageable, so I'd say go
ahead (if it were up to me).

Regards
St?fan


From d.l.goldsmith at gmail.com  Thu Feb 11 19:20:57 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Thu, 11 Feb 2010 16:20:57 -0800
Subject: [SciPy-dev] Not Re: Removing datetime support. :-)
Message-ID: <45d1ab481002111620s103360e6m35606a0336e4986a@mail.gmail.com>

In the doc Wiki, how does one make a three line, a.k.a. "identically" equal
to sign?  Thanks!

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100211/12851965/attachment.html>

From robert.kern at gmail.com  Thu Feb 11 19:21:59 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 11 Feb 2010 18:21:59 -0600
Subject: [SciPy-dev] Not Re: Removing datetime support. :-)
In-Reply-To: <45d1ab481002111620s103360e6m35606a0336e4986a@mail.gmail.com>
References: <45d1ab481002111620s103360e6m35606a0336e4986a@mail.gmail.com>
Message-ID: <3d375d731002111621jc31d987v9211aec3e021b9c4@mail.gmail.com>

On Thu, Feb 11, 2010 at 18:20, David Goldsmith <d.l.goldsmith at gmail.com> wrote:
> In the doc Wiki, how does one make a three line, a.k.a. "identically" equal
> to sign?? Thanks!

Preferably, you don't.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david.kirkby at onetel.net  Thu Feb 11 19:35:06 2010
From: david.kirkby at onetel.net (Dr. David Kirkby)
Date: Fri, 12 Feb 2010 00:35:06 +0000
Subject: [SciPy-dev] Not Re: Removing datetime support. :-)
In-Reply-To: <45d1ab481002111620s103360e6m35606a0336e4986a@mail.gmail.com>
References: <45d1ab481002111620s103360e6m35606a0336e4986a@mail.gmail.com>
Message-ID: <4B74A23A.7050801@onetel.net>

David Goldsmith wrote:
> In the doc Wiki, how does one make a three line, a.k.a. "identically" 
> equal to sign?  Thanks!
> 
> DG

On some Wikis, an exclamation mark will remove formatting. Try

!===

Dave


From jh at physics.ucf.edu  Sat Feb 13 13:30:18 2010
From: jh at physics.ucf.edu (Joe Harrington)
Date: Sat, 13 Feb 2010 13:30:18 -0500
Subject: [SciPy-dev] 2-review system on doc wiki
Message-ID: <wl8ocjtq98l.fsf@glup.physics.ucf.edu>

Chuck Harris recently posted on the numpy-discussion list a request
for numpy 2.0 requirements.  I suggested that we include reviewed docs
in 2.0.

The problem is that the docstring review is stuck.  We need to
implement both a technical and a presentation review, but we currently
lack labor to do the job.  Pauli Virtanen has been busy with other
commitments, so there has been little progress on doc wiki changes.

So, this is also a call for a Django programmer who can add a second
review capability to the doc wiki.

If nobody steps forward, then we'll have to abandon the idea.  I think
this would be a shame, because there are many docstrings that are
technically complete but impenetrable, and others that are well
presented but technically incomplete.  It won't be hard to fix these,
but we need a system to find them.

Any takers?  Ideally, it should be someone who has written docs in our
system and who has Django experience, but likely Django or similar
experience is more important for this.

Thanks,

--jh--
Prof. Joseph Harrington
Planetary Sciences Group
Department of Physics
MAP 414
4000 Central Florida Blvd.
University of Central Florida
Orlando, FL 32816-2385
jh at physics.ucf.edu
planets.ucf.edu


From millman at berkeley.edu  Sat Feb 13 15:39:30 2010
From: millman at berkeley.edu (Jarrod Millman)
Date: Sat, 13 Feb 2010 14:39:30 -0600
Subject: [SciPy-dev] 2-review system on doc wiki
In-Reply-To: <wl8ocjtq98l.fsf@glup.physics.ucf.edu>
References: <wl8ocjtq98l.fsf@glup.physics.ucf.edu>
Message-ID: <c7009a551002131239i43b8ffc0w51a1654f0f640ebb@mail.gmail.com>

On Sat, Feb 13, 2010 at 12:30 PM, Joe Harrington <jh at physics.ucf.edu> wrote:
> Chuck Harris recently posted on the numpy-discussion list a request
> for numpy 2.0 requirements. ?I suggested that we include reviewed docs
> in 2.0.

NumPy 2.0 is being released to address a recent ABI change (and is not
going to include any major new functionality).  It will be released in
3-4 weeks, so this is unfortunately way out of scope for this release.
 However, I am all for continuing the doc project and happy to see you
are still moving forward with it.

Thanks,

-- 
Jarrod Millman
Helen Wills Neuroscience Institute
10 Giannini Hall, UC Berkeley
http://cirl.berkeley.edu/


From bsouthey at gmail.com  Sat Feb 13 20:22:19 2010
From: bsouthey at gmail.com (Bruce Southey)
Date: Sat, 13 Feb 2010 19:22:19 -0600
Subject: [SciPy-dev] 2-review system on doc wiki
In-Reply-To: <wl8ocjtq98l.fsf@glup.physics.ucf.edu>
References: <wl8ocjtq98l.fsf@glup.physics.ucf.edu>
Message-ID: <bbcd77d01002131722n79eaf1f9n78803fcfe0092634@mail.gmail.com>

On Sat, Feb 13, 2010 at 12:30 PM, Joe Harrington <jh at physics.ucf.edu> wrote:
> Chuck Harris recently posted on the numpy-discussion list a request
> for numpy 2.0 requirements. ?I suggested that we include reviewed docs
> in 2.0.

I think you need to a provide a plan on how this can be done although
clearly it will not be the 2.0 release but perhaps the somewhere in
the 2.x series.

>
> The problem is that the docstring review is stuck. ?We need to
> implement both a technical and a presentation review, but we currently
> lack labor to do the job. ?Pauli Virtanen has been busy with other
> commitments, so there has been little progress on doc wiki changes.

What do you actually need by technical review and presentation review?
I think that we need some sort of checklist that people can go through
for each part. Really just numpy alone it a big task so it would be
nice to get people to proof other people's work.

> So, this is also a call for a Django programmer who can add a second
> review capability to the doc wiki.
>
> If nobody steps forward, then we'll have to abandon the idea. ?I think
> this would be a shame, because there are many docstrings that are
> technically complete but impenetrable, and others that are well
> presented but technically incomplete. ?It won't be hard to fix these,
> but we need a system to find them.

Can you provide examples that illustrate these two problems?

I am prepared to try to do something if I can understand what to do
and I can get the time.  But the Django side is beyond me as I am
trying to learn it for my own project.


>
> Any takers? ?Ideally, it should be someone who has written docs in our
> system and who has Django experience, but likely Django or similar
> experience is more important for this.
>
> Thanks,
>
> --jh--

Bruce


From d.l.goldsmith at gmail.com  Sat Feb 13 23:39:04 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Sat, 13 Feb 2010 20:39:04 -0800
Subject: [SciPy-dev] 2-review system on doc wiki
In-Reply-To: <bbcd77d01002131722n79eaf1f9n78803fcfe0092634@mail.gmail.com>
References: <wl8ocjtq98l.fsf@glup.physics.ucf.edu>
	<bbcd77d01002131722n79eaf1f9n78803fcfe0092634@mail.gmail.com>
Message-ID: <45d1ab481002132039r4823c067je07d3d19089088d7@mail.gmail.com>

On Sat, Feb 13, 2010 at 5:22 PM, Bruce Southey <bsouthey at gmail.com> wrote:

> On Sat, Feb 13, 2010 at 12:30 PM, Joe Harrington <jh at physics.ucf.edu>
> wrote:
> > Chuck Harris recently posted on the numpy-discussion list a request
> > for numpy 2.0 requirements.  I suggested that we include reviewed docs
> > in 2.0.
>
> I think you need to a provide a plan on how this can be done although
> clearly it will not be the 2.0 release but perhaps the somewhere in
> the 2.x series.
>
> >
> > The problem is that the docstring review is stuck.  We need to
> > implement both a technical and a presentation review, but we currently
> > lack labor to do the job.  Pauli Virtanen has been busy with other
> > commitments, so there has been little progress on doc wiki changes.
>
> What do you actually need by technical review and presentation review?
> I think that we need some sort of checklist that people can go through
>

http://docs.scipy.org/numpy/Questions+Answers/#reviewer-guidelines

First promulgated *2009-07-22, updated 2009-09-28, 2009-10-01

Unfortunately, as I say there, asking for a precise "checklist" for the
"presentation" review is _perhaps_ intractable (too much inherent
subjectivity IMO), but I don't feel that should preclude it being done.

DG
*

> for each part. Really just numpy alone it a big task so it would be
> nice to get people to proof other people's work.
>
> > So, this is also a call for a Django programmer who can add a second
> > review capability to the doc wiki.
> >
> > If nobody steps forward, then we'll have to abandon the idea.  I think
> > this would be a shame, because there are many docstrings that are
> > technically complete but impenetrable, and others that are well
> > presented but technically incomplete.  It won't be hard to fix these,
> > but we need a system to find them.
>
> Can you provide examples that illustrate these two problems?
>
> I am prepared to try to do something if I can understand what to do
> and I can get the time.  But the Django side is beyond me as I am
> trying to learn it for my own project.
>
>
> >
> > Any takers?  Ideally, it should be someone who has written docs in our
> > system and who has Django experience, but likely Django or similar
> > experience is more important for this.
> >
> > Thanks,
> >
> > --jh--
>
> Bruce
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100213/8cd8b896/attachment.html>

From charlesr.harris at gmail.com  Sun Feb 14 01:24:09 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 13 Feb 2010 23:24:09 -0700
Subject: [SciPy-dev] Buildbots showing red
Message-ID: <e06186141002132224q4b54d286i91d7ab2badcc32d5@mail.gmail.com>

*All* the buildbots are showing errors. Here are some:

======================================================================
ERROR: test_view_to_flexible_dtype (test_core.TestMaskedView)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/numpybb/Buildbot/numpy/b13/numpy-install/lib/python2.4/site-packages/numpy/ma/tests/test_core.py",
line 3333, in test_view_to_flexible_dtype
    test = a[0].view([('A', float), ('B', float)])
  File "../numpy-install/lib/python2.4/site-packages/numpy/ma/core.py",
line 2866, in view
  File "../numpy-install/lib/python2.4/site-packages/numpy/ma/core.py",
line 2786, in __array_finalize__
TypeError: attribute 'shape' of 'numpy.generic' objects is not writable

======================================================================
ERROR: test_view_to_subdtype (test_core.TestMaskedView)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/numpybb/Buildbot/numpy/b13/numpy-install/lib/python2.4/site-packages/numpy/ma/tests/test_core.py",
line 3354, in test_view_to_subdtype
    test = a[0].view((float, 2))
  File "../numpy-install/lib/python2.4/site-packages/numpy/ma/core.py",
line 2866, in view
  File "../numpy-install/lib/python2.4/site-packages/numpy/ma/core.py",
line 2786, in __array_finalize__
TypeError: attribute 'shape' of 'numpy.generic' objects is not writable

======================================================================
FAIL: test_buffer_hashlib (test_regression.TestRegression)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/numpybb/Buildbot/numpy/b13/numpy-install/lib/python2.4/site-packages/numpy/core/tests/test_regression.py",
line 1255, in test_buffer_hashlib
    assert_equal(md5(x).hexdigest(), '2a1dd1e1e59d0a384c26951e316cd7e6')
  File "../numpy-install/lib/python2.4/site-packages/numpy/testing/utils.py",
line 305, in assert_equal
AssertionError:
Items are not equal:
 ACTUAL: '1264d4a9f74dc462700fd163e3ff09a6'
 DESIRED: '2a1dd1e1e59d0a384c26951e316cd7e6'

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100213/4aa717e5/attachment.html>

From bsouthey at gmail.com  Sun Feb 14 10:12:03 2010
From: bsouthey at gmail.com (Bruce Southey)
Date: Sun, 14 Feb 2010 09:12:03 -0600
Subject: [SciPy-dev] 2-review system on doc wiki
In-Reply-To: <45d1ab481002132039r4823c067je07d3d19089088d7@mail.gmail.com>
References: <wl8ocjtq98l.fsf@glup.physics.ucf.edu>
	<bbcd77d01002131722n79eaf1f9n78803fcfe0092634@mail.gmail.com>
	<45d1ab481002132039r4823c067je07d3d19089088d7@mail.gmail.com>
Message-ID: <bbcd77d01002140712u1cfe6a3dr3f49c4dc94fe9d0@mail.gmail.com>

On Sat, Feb 13, 2010 at 10:39 PM, David Goldsmith
<d.l.goldsmith at gmail.com> wrote:
> On Sat, Feb 13, 2010 at 5:22 PM, Bruce Southey <bsouthey at gmail.com> wrote:
>>
>> On Sat, Feb 13, 2010 at 12:30 PM, Joe Harrington <jh at physics.ucf.edu>
>> wrote:
>> > Chuck Harris recently posted on the numpy-discussion list a request
>> > for numpy 2.0 requirements. ?I suggested that we include reviewed docs
>> > in 2.0.
>>
>> I think you need to a provide a plan on how this can be done although
>> clearly it will not be the 2.0 release but perhaps the somewhere in
>> the 2.x series.
>>
>> >
>> > The problem is that the docstring review is stuck. ?We need to
>> > implement both a technical and a presentation review, but we currently
>> > lack labor to do the job. ?Pauli Virtanen has been busy with other
>> > commitments, so there has been little progress on doc wiki changes.
>>
>> What do you actually need by technical review and presentation review?
>> I think that we need some sort of checklist that people can go through
>
> http://docs.scipy.org/numpy/Questions+Answers/#reviewer-guidelines
>
> First promulgated 2009-07-22, updated 2009-09-28, 2009-10-01
>
> Unfortunately, as I say there, asking for a precise "checklist" for the
> "presentation" review is _perhaps_ intractable (too much inherent
> subjectivity IMO), but I don't feel that should preclude it being done.
>
> DG

Never knew that existed because it is rather different from:
http://docs.scipy.org/numpy/Front%20Page/#roles-reviewing

But my question is what is 'technical review' and what is 'presentation review'?
Without clarifying those any steps are rather pointless. From the page
you linked, these two things are combined. Questions like 'is it
clear?' and 'is it helpful?' is not what I would call presentation but
technical.

To me, presentation review should only address if it meets the
docstring standard
http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#docstring-standard)
and is displayed correctly.


I just think that the 'bar' here is set too high for a volunteer
project. Also I think that this 'new version' is asking too much
especially when people have been working under a rather different
approach. Also there is no conflict resolution between all the steps
involved.



Bruce


From stefan at sun.ac.za  Sun Feb 14 11:07:43 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 14 Feb 2010 18:07:43 +0200
Subject: [SciPy-dev] 2-review system on doc wiki
In-Reply-To: <bbcd77d01002140712u1cfe6a3dr3f49c4dc94fe9d0@mail.gmail.com>
References: <wl8ocjtq98l.fsf@glup.physics.ucf.edu>
	<bbcd77d01002131722n79eaf1f9n78803fcfe0092634@mail.gmail.com> 
	<45d1ab481002132039r4823c067je07d3d19089088d7@mail.gmail.com> 
	<bbcd77d01002140712u1cfe6a3dr3f49c4dc94fe9d0@mail.gmail.com>
Message-ID: <9457e7c81002140807g348a5a5naa6f13deb82b92c5@mail.gmail.com>

On 14 February 2010 17:12, Bruce Southey <bsouthey at gmail.com> wrote:
> I just think that the 'bar' here is set too high for a volunteer
> project. Also I think that this 'new version' is asking too much
> especially when people have been working under a rather different
> approach. Also there is no conflict resolution between all the steps
> involved.

This whole issue looks much more complicated than it is.  We simply
need two (possibly overlapping) groups of people to answer the
following questions:

1) Is the docstring technically accurate? (Examples correct, docstring
format followed, etc.)
2) Is the docstring well written and easily understandable? (Language
use, simplicity)

However, I think the reviewing process is much too daunting as it is,
so we'll need to simply everything by providing:

- An easy way to reach numpy / scipy docstrings for editing (currently
requires >1 click)
- An easy way to review (maybe an automated weekly post to the list
with prioritised request for review)
- Simpler explanations of the differences between docstring and
reference guide editing
- Easy integration of patches into the source code (everything is in
place, but we need to make the process clear)

Regards
St?fan


From tmp50 at ukr.net  Sun Feb 14 11:45:23 2010
From: tmp50 at ukr.net (Dmitrey)
Date: Sun, 14 Feb 2010 18:45:23 +0200
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <9457e7c81002092344i72886a26gcee3460782e4647e@mail.gmail.com>
Message-ID: <E1Nghb1-0008mm-1R@ffe5.ukr.net>

Well, if anyone doesn't mind, I'll add one more string "and automatic differentiation" to the docstring, I guess it's quite essential.  
  
Regards, D.  
  
?? ????: St?fan van der Walt <stefan at sun.ac.za>  
  
  Sorry, Dmitrey, I didn't see your earlier message. I included a link  
to OpenOpt in the optimize module docstring, and removed  
the See Also sections entirely (they were difficult to maintain, so I  
would have removed them either way). I hope you find this solution to  
your satisfaction.  
  
Kind regards  
St?fan  
___________________________  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100214/ba43e77e/attachment.html>

From jh at physics.ucf.edu  Sun Feb 14 14:07:59 2010
From: jh at physics.ucf.edu (Joe Harrington)
Date: Sun, 14 Feb 2010 14:07:59 -0500
Subject: [SciPy-dev] 2-review system on doc wiki
In-Reply-To: <mailman.5075.1266163689.2454.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.5075.1266163689.2454.scipy-dev@scipy.org>
Message-ID: <wl8ljevr5yo.fsf@glup.physics.ucf.edu>

On 14 February 2010 17:12, Bruce Southey <bsouthey at gmail.com> wrote:
> I just think that the 'bar' here is set too high for a volunteer
> project. Also I think that this 'new version' is asking too much
> especially when people have been working under a rather different
> approach. Also there is no conflict resolution between all the steps
> involved.

Sorry for the length here.  Hopefully this clarifies a lot of
questions.  See, in particular, example 3 if you're not convinced we
need this.

I agree with Stefan that this really isn't that complicated.  David
and I have discussed the two-review system here, in doc telecons, at
the SciPy09 conference, and in its proceedings; this is nothing new.
The motivation is simple: I read a number of the reviewed pages and
found problems that should not have passed review.  The plan is a
slight modification of our one-review plan.  David pointed to that
already (thanks, David).

There are no differences of approach other than the change in the
review system.  Since only a tiny fraction (8%) of the pages has
undergone any level of review, and only 4% have passed review, the
change will not cause a major upset to what we are doing.

As always, we resolve conflicts by discussion and use of the comment
field on each page.

We are aiming at a product of equal or greater quality to similar
manuals for software such as IDL or Matlab.  Whether this can all be
done by volunteers is an irrelevant question.  I expect that the
number of reviewers will be much smaller than the number of writers.
We will identify and vet technical and presentation reviewers, and if
necessary we can seek funds to pay them.  Of course, we'll try the
volunteer way first.  I hope that we can find volunteer technical
reviewers from among the developers.  Presentation reviewers will
likely have substantial technical writing experience; we have a list
of a few potentials already.  A professional copy editor will proof
the doc the first time we have fully-reviewed pages and hopefully for
each major release thereafter, but that's a future problem.

I give some examples and clarification on the review roles below.

EXAMPLES

1. numpy.core.umath.sqrt does not define the "out" argument (technical
omission) and uses language "branch cut", "continuous from above on
it" that will confuse the majority of readers who have not taken a
course in complex variables, such as high-school students and perhaps
many of their teachers (presentation review).  This could be solved
with an external reference, which is missing, or even just a rewording
of the sentence, like:

In the terminology of complex-variable calculus (ref), sqrt has a
branch cut [-inf, 0) and is continuous from above on it.

This is what I call "introducing an expert section".  It signifies to
our target audience (one level below the likely users of a function)
that we're about to go over their heads, where to go to come up to
speed, and otherwise not to sweat it if they don't get it.  (Actually,
in this particular case, it's not clear to me why we need to document
the analytic properties of taking roots.  There's *lots* more one
could say about roots, and trig functions, and....  We should leave
that to the textbooks.)

2. Most routines are missing pointers to relevant pages of the
numpy.doc package that discuss things like "along and axis" or "out".
In many cases, that's because these pages didn't exist when the
function docstrings were written.

3. From scipy, some of the ready-for-review pages in scipy.stats are
likely technically good, but are totally impenetrable to anyone
without several semesters' equivalent college education in statistics.
While you may need that level of description to use all the tests to
their fullest, a beginner should be able to do things like plot,
evaluate, and integrate standard PDFs within a few minutes of starting
to read the docs there.  If two stats experts wrote all the pages and
reviewed each others' writing, such improvements would never be
suggested.  Yet, a single presentation-oriented reviewer might not
catch technical errors.  That's why we need two types of reviewers.

TECHNICAL REVIEW

A technical review ensures that all the features, API points,
underlying methods that affect the results, and limitations of the
item are noted properly in the docstring.  It implies familiarity with
(or at least a good, hard look at) the source code and the general
topic (e.g., fitting, stats, etc.).  In the ideal case, an expert
should be able to take the doc and write a more-or-less equivalent
routine.  This review also should check that internal cross-references
are complete and that external references are sufficient (and
long-lived).

PRESENTATION REVIEW

A presentation review ensures that our target audience - which we long
ago defined at one level *below* that of a likely user of a given
routine - can read and understand all but the expert parts of the
document, that the doc follows the docstring format, that it is as
clear as reasonably possible, that, if expert sections are needed,
they are properly introduced as such, that the examples are the right
ones to have and that they work, etc.

--jh--


From josef.pktd at gmail.com  Sun Feb 14 14:56:25 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 14 Feb 2010 14:56:25 -0500
Subject: [SciPy-dev] 2-review system on doc wiki
In-Reply-To: <wl8ljevr5yo.fsf@glup.physics.ucf.edu>
References: <mailman.5075.1266163689.2454.scipy-dev@scipy.org>
	<wl8ljevr5yo.fsf@glup.physics.ucf.edu>
Message-ID: <1cd32cbb1002141156s37147298md272f15d356fd934@mail.gmail.com>

On Sun, Feb 14, 2010 at 2:07 PM, Joe Harrington <jh at physics.ucf.edu> wrote:
> On 14 February 2010 17:12, Bruce Southey <bsouthey at gmail.com> wrote:
>> I just think that the 'bar' here is set too high for a volunteer
>> project. Also I think that this 'new version' is asking too much
>> especially when people have been working under a rather different
>> approach. Also there is no conflict resolution between all the steps
>> involved.
>
> Sorry for the length here. ?Hopefully this clarifies a lot of
> questions. ?See, in particular, example 3 if you're not convinced we
> need this.
>
> I agree with Stefan that this really isn't that complicated. ?David
> and I have discussed the two-review system here, in doc telecons, at
> the SciPy09 conference, and in its proceedings; this is nothing new.
> The motivation is simple: I read a number of the reviewed pages and
> found problems that should not have passed review. ?The plan is a
> slight modification of our one-review plan. ?David pointed to that
> already (thanks, David).
>
> There are no differences of approach other than the change in the
> review system. ?Since only a tiny fraction (8%) of the pages has
> undergone any level of review, and only 4% have passed review, the
> change will not cause a major upset to what we are doing.
>
> As always, we resolve conflicts by discussion and use of the comment
> field on each page.
>
> We are aiming at a product of equal or greater quality to similar
> manuals for software such as IDL or Matlab. ?Whether this can all be
> done by volunteers is an irrelevant question. ?I expect that the
> number of reviewers will be much smaller than the number of writers.
> We will identify and vet technical and presentation reviewers, and if
> necessary we can seek funds to pay them. ?Of course, we'll try the
> volunteer way first. ?I hope that we can find volunteer technical
> reviewers from among the developers. ?Presentation reviewers will
> likely have substantial technical writing experience; we have a list
> of a few potentials already. ?A professional copy editor will proof
> the doc the first time we have fully-reviewed pages and hopefully for
> each major release thereafter, but that's a future problem.
>
> I give some examples and clarification on the review roles below.
>
> EXAMPLES
>
> 1. numpy.core.umath.sqrt does not define the "out" argument (technical
> omission) and uses language "branch cut", "continuous from above on
> it" that will confuse the majority of readers who have not taken a
> course in complex variables, such as high-school students and perhaps
> many of their teachers (presentation review). ?This could be solved
> with an external reference, which is missing, or even just a rewording
> of the sentence, like:
>
> In the terminology of complex-variable calculus (ref), sqrt has a
> branch cut [-inf, 0) and is continuous from above on it.
>
> This is what I call "introducing an expert section". ?It signifies to
> our target audience (one level below the likely users of a function)
> that we're about to go over their heads, where to go to come up to
> speed, and otherwise not to sweat it if they don't get it. ?(Actually,
> in this particular case, it's not clear to me why we need to document
> the analytic properties of taking roots. ?There's *lots* more one
> could say about roots, and trig functions, and.... ?We should leave
> that to the textbooks.)
>
> 2. Most routines are missing pointers to relevant pages of the
> numpy.doc package that discuss things like "along and axis" or "out".
> In many cases, that's because these pages didn't exist when the
> function docstrings were written.
>
> 3. From scipy, some of the ready-for-review pages in scipy.stats are
> likely technically good, but are totally impenetrable to anyone
> without several semesters' equivalent college education in statistics.
> While you may need that level of description to use all the tests to
> their fullest, a beginner should be able to do things like plot,
> evaluate, and integrate standard PDFs within a few minutes of starting
> to read the docs there. ?If two stats experts wrote all the pages and
> reviewed each others' writing, such improvements would never be
> suggested. ?Yet, a single presentation-oriented reviewer might not
> catch technical errors. ?That's why we need two types of reviewers.

I agree with all the reviewing proposals, but I have two
qualifications for specialized parts in scipy

Most scipy subpackages have tutorials, and if a user needs an
introduction then it is necessary to read the tutorial. I went through
them when I looked at the part of scipy that I didn't know. A basic
introduction cannot be included in every docstring. I think the
presentation review for accessibility with less prior information
should focus also more on the tutorials. For the
scipy.stats.distributions, I tried to do this in the stats.tutorial
(although my plots seemed to have disappeared again.)

For some functions it will be difficult to determine what one level
below the likely user actually means. I recently fixed a problem with
http://docs.scipy.org/scipy/docs/scipy.signal.signaltools.hilbert/
but I still don't know what the use for the analytical signal is or
what it really means. I don't think a random user will bump into
signal.hilbert, for example. And for a basic introduction Wikipedia is
more informative than a docstring can be.

Josef


>
> TECHNICAL REVIEW
>
> A technical review ensures that all the features, API points,
> underlying methods that affect the results, and limitations of the
> item are noted properly in the docstring. ?It implies familiarity with
> (or at least a good, hard look at) the source code and the general
> topic (e.g., fitting, stats, etc.). ?In the ideal case, an expert
> should be able to take the doc and write a more-or-less equivalent
> routine. ?This review also should check that internal cross-references
> are complete and that external references are sufficient (and
> long-lived).
>
> PRESENTATION REVIEW
>
> A presentation review ensures that our target audience - which we long
> ago defined at one level *below* that of a likely user of a given
> routine - can read and understand all but the expert parts of the
> document, that the doc follows the docstring format, that it is as
> clear as reasonably possible, that, if expert sections are needed,
> they are properly introduced as such, that the examples are the right
> ones to have and that they work, etc.
>
> --jh--
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From stefan at sun.ac.za  Sun Feb 14 15:48:14 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 14 Feb 2010 22:48:14 +0200
Subject: [SciPy-dev] Changes to trunk/scipy/optimize
In-Reply-To: <E1Nghb1-0008mm-1R@ffe5.ukr.net>
References: <9457e7c81002092344i72886a26gcee3460782e4647e@mail.gmail.com> 
	<E1Nghb1-0008mm-1R@ffe5.ukr.net>
Message-ID: <9457e7c81002141248q3b213e15off65a76c617b7242@mail.gmail.com>

2010/2/14 Dmitrey <tmp50 at ukr.net>:
> Well, if anyone doesn't mind, I'll add one more string "and automatic
> differentiation" to the docstring, I guess it's quite essential.

Sure, go ahead.

Regards
St?fan


From warren.weckesser at enthought.com  Sun Feb 14 19:06:18 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sun, 14 Feb 2010 18:06:18 -0600
Subject: [SciPy-dev] Ticket #1105 -- patch submitted
	for	scipy.signal.waveforms.chirp()
In-Reply-To: <4B6F6F7A.70506@enthought.com>
References: <4B6F6F7A.70506@enthought.com>
Message-ID: <4B788FFA.4060709@enthought.com>

Warren Weckesser wrote:
> I just added a patch to ticket #1105; a summary of the changes is given 
> in the ticket.  If any scipy.signal users (especially chirp()) users) 
> have chance, please take a look a let me know what you think.  The 
> docstrings could use polishing, but I'd like to get some feedback before 
> doing more work on it.
>
> Warren
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   

I created a new patch today; any feedback would be appreciated.

Examples of signals created by the patched functions are available here:

    http://www.scipy.org/NewChirp


Cheers,

Warren



From d.l.goldsmith at gmail.com  Mon Feb 15 00:30:58 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Sun, 14 Feb 2010 21:30:58 -0800
Subject: [SciPy-dev] 2-review system on doc wiki
In-Reply-To: <1cd32cbb1002141156s37147298md272f15d356fd934@mail.gmail.com>
References: <mailman.5075.1266163689.2454.scipy-dev@scipy.org>
	<wl8ljevr5yo.fsf@glup.physics.ucf.edu>
	<1cd32cbb1002141156s37147298md272f15d356fd934@mail.gmail.com>
Message-ID: <45d1ab481002142130i3306d68pbb3771265a42c8b6@mail.gmail.com>

On Sun, Feb 14, 2010 at 11:56 AM, <josef.pktd at gmail.com> wrote:

>
> For some functions it will be difficult to determine what one level
> below the likely user actually means. I recently fixed a problem with
>

Maybe, maybe not: if in doubt, one could query the list "who uses this
function?"  Reply: "I do."  (Hopefully someone pipes up.)  Next query: "what
do you do/how do you use it?"  Reply: "I'm a physicist; I use it for my
research on quantum gravity."  Conclusion: "one level below" equals a
physics graduate student (or maybe post-doc).  Another (hopefully less
plausible) example: "Who uses 'dot'?"  A tsunami of replies: "I do."
Follow-up: "What do you do?"  Lowest level of replies: a high school student
taking AP Physics.  Conclusion: "one level below" equals a high school
student taking non-AP physics.  Not so hard - "loud" on the list, perhaps,
but not so hard.

DG


> http://docs.scipy.org/scipy/docs/scipy.signal.signaltools.hilbert/
> but I still don't know what the use for the analytical signal is or
> what it really means. I don't think a random user will bump into
> signal.hilbert, for example. And for a basic introduction Wikipedia is
> more informative than a docstring can be.
>
> Josef
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100214/d2f9663d/attachment.html>

From jh at physics.ucf.edu  Mon Feb 15 13:34:44 2010
From: jh at physics.ucf.edu (Joe Harrington)
Date: Mon, 15 Feb 2010 13:34:44 -0500
Subject: [SciPy-dev] 2-review system on doc wiki
In-Reply-To: <mailman.17.1266256805.28994.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.17.1266256805.28994.scipy-dev@scipy.org>
Message-ID: <wl8sk92pcu3.fsf@glup.physics.ucf.edu>

On Sun, Feb 14, 2010 at 11:56 AM, <josef.pktd at gmail.com> wrote:

> Most scipy subpackages have tutorials, and if a user needs an
> introduction then it is necessary to read the tutorial.

Agreed that a lot can go into the tutorials; we need clearer
guidelines on what should go in tutorial vs. function docstrings.
Perhaps more importantly, we need to educate people about the
existence of these tutorials, perhaps in the see-also to each routine,
or at least in the module docstrings.

> And for a basic introduction Wikipedia is
> more informative than a docstring can be.

Agreed that it is our job to inform about what the routines are and
how to use them to do likely tasks, not to teach people the math
behind them.  But, it is pretty standard to include enough of that
math (such as a formula) to demonstrate what the routine does.  This
is particuarly important where there is more than one way it's
defined, such as the normalization for Fourier transforms (does it go
on the forward transform, the inverse, or split between them).

--jh--



From njs at pobox.com  Tue Feb 16 15:18:49 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 16 Feb 2010 12:18:49 -0800
Subject: [SciPy-dev] code for incremental least squares
Message-ID: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>

Hello all,
I have a need for solving some very large least squares regression
problems -- but fortunately my model matrix can be generated
incrementally, so there are some tricks to do the important
calculations without actually loading the whole thing into memory. The
resulting code is rather general, and might be of wider interest, so I
thought I'd send a note here and see what people thought about
upstreaming it somehow...

Specifically, what it implements is:
  -- Least squares regression for very large numbers of observations
  -- Using either Cholesky or QR methods
  -- Multivariate or univariate
  -- Full statistical tests (t tests, linear hypothesis F tests,
univariate and multivariate)
  -- "Grouped" weighted least squares (i.e., if your observations fall
into a small number of different classes, and are homoskedastic within
each class, but not across classes, it can handle that), with simple
EM estimation of the weights ("feasible generalized least squares")
  -- The Cholesky solver's initial "accumulation" step has a
parallelization-friendly API, and can take advantage of sparseness in
the model matrix

Most of the code is unencumbered, but I cribbed some of the
calculations for multivariate statistics (Wilk's lambda and all that)
from R, so those functions are GPL'ed (but could presumably be
rewritten if anyone cares). Also the sparse Cholesky method should
probably take advantage of the GPL'ed scikits.sparse.cholmod, but that
isn't actually implemented yet (it just uses sparse matrices for the
accumulation step, and then converts to a dense matrix at the end).

So, any thoughts on what would be best to do with this? I'd be happy
to contribute it to SciPy or whatever. In some ways it overlaps with
scikits.statsmodels, but of course the API is quite different.

-- Nathaniel


From stefan at sun.ac.za  Tue Feb 16 15:24:08 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 16 Feb 2010 22:24:08 +0200
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
Message-ID: <9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>

Hi Nathan

On 16 February 2010 22:18, Nathaniel Smith <njs at pobox.com> wrote:
> I have a need for solving some very large least squares regression
> problems -- but fortunately my model matrix can be generated
> incrementally, so there are some tricks to do the important
> calculations without actually loading the whole thing into memory. The
> resulting code is rather general, and might be of wider interest, so I
> thought I'd send a note here and see what people thought about
> upstreaming it somehow...

This sounds interesting!  Could you expand on the incremental
generation of the model matrix, and how it is made general?

Thanks
St?fan


From josef.pktd at gmail.com  Tue Feb 16 15:37:55 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 15:37:55 -0500
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
Message-ID: <1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>

2010/2/16 St?fan van der Walt <stefan at sun.ac.za>:
> Hi Nathan
>
> On 16 February 2010 22:18, Nathaniel Smith <njs at pobox.com> wrote:
>> I have a need for solving some very large least squares regression
>> problems -- but fortunately my model matrix can be generated
>> incrementally, so there are some tricks to do the important
>> calculations without actually loading the whole thing into memory. The
>> resulting code is rather general, and might be of wider interest, so I
>> thought I'd send a note here and see what people thought about
>> upstreaming it somehow...
>
> This sounds interesting! ?Could you expand on the incremental
> generation of the model matrix, and how it is made general?

I have the same thought, Are you increasing by observation or by
explanatory variable?

For either case, I would be very interested to see this kind of
incremental least squares in statsmodels. If you are able to license
your parts as BSD, then I will look at it for sure.

We have some plans for this but not yet implemented.

Josef

>
> Thanks
> St?fan
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From njs at pobox.com  Tue Feb 16 15:49:12 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 16 Feb 2010 12:49:12 -0800
Subject: [SciPy-dev] Best interface for computing the logarithm of the
	determinant?
Message-ID: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>

So when you have a matrix whose determinant you want, it's often wise
to compute the logarithm of the determinant instead of the determinant
itself, because determinants involve lots and lots of multiplications
and the result might otherwise underflow/overflow. Therefore, in
scikits.sparse, I'd like to provide an API for doing this (and this is
well-supported by the underlying libraries).

But the problem is that for a general matrix, the determinant may be
zero or negative. Obviously we can deal with this, but what's the best
API? I'd like to use one consistently across the different
factorizations in scikits.sparse, and perhaps eventually in numpy as
well.

Some options:

1) Split off the sign into a separate return value ('sign' may be 1, -1, 0):
sign, value = logdet(A)
actual_determinant = sign * exp(value)

2) Allow complex/infinite return values, even when A is a real matrix:
logdet(eye(3)) == pi*1j
logdet(zeros((3, 3))) == -Inf

3) "Scientific notation" (This is what UMFPACK's API does): return a
mantissa and base-10 exponent:
mantissa, exponent = logdet(A)
actual_determinant = mantissa * 10 ** exponent

4) Have separate functions for computing the sign, and the log of the
absolute value (This is what GSL does, though it seems pointlessly
inefficient):
sign = sgndet(A)
value = logdet(A)
actual_determinant = sign * exp(value)

These are all kind of ugly looking, unfortunately, but that seems
unavoidable, unless someone has a clever idea.

Any preferences?

-- Nathaniel


From warren.weckesser at enthought.com  Tue Feb 16 16:01:42 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Tue, 16 Feb 2010 15:01:42 -0600
Subject: [SciPy-dev] Best interface for computing the logarithm of the
 determinant?
In-Reply-To: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
References: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
Message-ID: <4B7B07B6.7040802@enthought.com>

Nathaniel Smith wrote:
> So when you have a matrix whose determinant you want, it's often wise
> to compute the logarithm of the determinant instead of the determinant
> itself, because determinants involve lots and lots of multiplications
> and the result might otherwise underflow/overflow. Therefore, in
> scikits.sparse, I'd like to provide an API for doing this (and this is
> well-supported by the underlying libraries).
>
> But the problem is that for a general matrix, the determinant may be
> zero or negative.

For a general matrix, the determinant may be complex.  Or are you only 
considering real matrices?



Warren



From jsseabold at gmail.com  Tue Feb 16 16:10:12 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 16 Feb 2010 16:10:12 -0500
Subject: [SciPy-dev] maxentropy docs / parent class methods
Message-ID: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com>

The docs on the methods for the maxentropy model class that are
inherited from basemodel are not picked up by sphinx.  It seems that
most of the .rst files explicitly list the methods, but since
basemodel is not intended to be public (and some subclasses overwrite
the parent class methods), is there a better way than to start listing
the basemodel methods?

I started to make the changes, but I don't think this is the right way
forward.  Other thoughts?

http://docs.scipy.org/scipy/docs/scipy-docs/maxentropy.rst/

Cheers,

Skipper


From njs at pobox.com  Tue Feb 16 16:16:05 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 16 Feb 2010 13:16:05 -0800
Subject: [SciPy-dev] Best interface for computing the logarithm of the
	determinant?
In-Reply-To: <4B7B07B6.7040802@enthought.com>
References: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
	<4B7B07B6.7040802@enthought.com>
Message-ID: <961fa2b41002161316o63db3d74o9403a9df0284d329@mail.gmail.com>

On Tue, Feb 16, 2010 at 1:01 PM, Warren Weckesser
<warren.weckesser at enthought.com> wrote:
> Nathaniel Smith wrote:
>> So when you have a matrix whose determinant you want, it's often wise
>> to compute the logarithm of the determinant instead of the determinant
>> itself, because determinants involve lots and lots of multiplications
>> and the result might otherwise underflow/overflow. Therefore, in
>> scikits.sparse, I'd like to provide an API for doing this (and this is
>> well-supported by the underlying libraries).
>>
>> But the problem is that for a general matrix, the determinant may be
>> zero or negative.
>
> For a general matrix, the determinant may be complex. ?Or are you only
> considering real matrices?

Sorry, yes. For a complex matrix, obviously you expect a complex
determinant and there is no problem. But it may be confusing for
people to get a complex number out when they are computing the (real)
determinant of a real matrix; for instance, np.log on real-value
inputs never returns a complex number, even if this means it must
return NaN. Or perhaps this would be just fine, that's why it's one of
the options I mentioned :-).

It is a good point though that whatever API we pick should not be too
annoying for people with complex matrices, since we want to use the
same API for both real and complex.

-- Nathaniel


From dagss at student.matnat.uio.no  Tue Feb 16 16:19:58 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 16 Feb 2010 22:19:58 +0100
Subject: [SciPy-dev] Best interface for computing the logarithm of the
 determinant?
In-Reply-To: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
References: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
Message-ID: <4B7B0BFE.8060109@student.matnat.uio.no>

Nathaniel Smith wrote:
> So when you have a matrix whose determinant you want, it's often wise
> to compute the logarithm of the determinant instead of the determinant
> itself, because determinants involve lots and lots of multiplications
> and the result might otherwise underflow/overflow. Therefore, in
> scikits.sparse, I'd like to provide an API for doing this (and this is
> well-supported by the underlying libraries).
> 
> But the problem is that for a general matrix, the determinant may be
> zero or negative. Obviously we can deal with this, but what's the best
> API? I'd like to use one consistently across the different
> factorizations in scikits.sparse, and perhaps eventually in numpy as
> well.
> 
> Some options:
> 
> 1) Split off the sign into a separate return value ('sign' may be 1, -1, 0):
> sign, value = logdet(A)
> actual_determinant = sign * exp(value)
> 
> 2) Allow complex/infinite return values, even when A is a real matrix:
> logdet(eye(3)) == pi*1j
> logdet(zeros((3, 3))) == -Inf

I'm +1 for this one. It is easy to interpret, and if one knows that the 
result is positive (like after a successful CHOLMOD LL^T) is is easy 
enough to take the real part by appending .real.

Also it makes more sense for code that may be dealing with both complex 
and real matrices compared to 1).

> 
> 3) "Scientific notation" (This is what UMFPACK's API does): return a
> mantissa and base-10 exponent:
> mantissa, exponent = logdet(A)
> actual_determinant = mantissa * 10 ** exponent

This is my least preferred option, because "10" has nothing to do with 
floats.

> 
> 4) Have separate functions for computing the sign, and the log of the
> absolute value (This is what GSL does, though it seems pointlessly
> inefficient):
> sign = sgndet(A)
> value = logdet(A)
> actual_determinant = sign * exp(value)

Well, it doesn't have to be inefficient, you use cache the return value 
internally on id(A) and possibly weak reference callbacks to remove the 
cached values...but pretty ugly, yeah.

-- 
Dag Sverre


From josef.pktd at gmail.com  Tue Feb 16 16:29:12 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 16:29:12 -0500
Subject: [SciPy-dev] Best interface for computing the logarithm of the
	determinant?
In-Reply-To: <4B7B0BFE.8060109@student.matnat.uio.no>
References: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
	<4B7B0BFE.8060109@student.matnat.uio.no>
Message-ID: <1cd32cbb1002161329s29f8da19l298ce62fd726125e@mail.gmail.com>

On Tue, Feb 16, 2010 at 4:19 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Nathaniel Smith wrote:
>> So when you have a matrix whose determinant you want, it's often wise
>> to compute the logarithm of the determinant instead of the determinant
>> itself, because determinants involve lots and lots of multiplications
>> and the result might otherwise underflow/overflow. Therefore, in
>> scikits.sparse, I'd like to provide an API for doing this (and this is
>> well-supported by the underlying libraries).
>>
>> But the problem is that for a general matrix, the determinant may be
>> zero or negative. Obviously we can deal with this, but what's the best
>> API? I'd like to use one consistently across the different
>> factorizations in scikits.sparse, and perhaps eventually in numpy as
>> well.
>>
>> Some options:
>>
>> 1) Split off the sign into a separate return value ('sign' may be 1, -1, 0):
>> sign, value = logdet(A)
>> actual_determinant = sign * exp(value)
>>
>> 2) Allow complex/infinite return values, even when A is a real matrix:
>> logdet(eye(3)) == pi*1j
>> logdet(zeros((3, 3))) == -Inf
>
> I'm +1 for this one. It is easy to interpret, and if one knows that the
> result is positive (like after a successful CHOLMOD LL^T) is is easy
> enough to take the real part by appending .real.
>
> Also it makes more sense for code that may be dealing with both complex
> and real matrices compared to 1).

I'm also in favor of this, but would prefer if promotion to complex
only occurs if the result requires it.
np.real_if_close  might be safer than .real

Josef

>
>>
>> 3) "Scientific notation" (This is what UMFPACK's API does): return a
>> mantissa and base-10 exponent:
>> mantissa, exponent = logdet(A)
>> actual_determinant = mantissa * 10 ** exponent
>
> This is my least preferred option, because "10" has nothing to do with
> floats.
>
>>
>> 4) Have separate functions for computing the sign, and the log of the
>> absolute value (This is what GSL does, though it seems pointlessly
>> inefficient):
>> sign = sgndet(A)
>> value = logdet(A)
>> actual_determinant = sign * exp(value)
>
> Well, it doesn't have to be inefficient, you use cache the return value
> internally on id(A) and possibly weak reference callbacks to remove the
> cached values...but pretty ugly, yeah.
>
> --
> Dag Sverre
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From josef.pktd at gmail.com  Tue Feb 16 16:41:23 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 16:41:23 -0500
Subject: [SciPy-dev] maxentropy docs / parent class methods
In-Reply-To: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com>
References: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com>
Message-ID: <1cd32cbb1002161341r412704c8y65a3eed7000fce8f@mail.gmail.com>

On Tue, Feb 16, 2010 at 4:10 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> The docs on the methods for the maxentropy model class that are
> inherited from basemodel are not picked up by sphinx. ?It seems that
> most of the .rst files explicitly list the methods, but since
> basemodel is not intended to be public (and some subclasses overwrite
> the parent class methods), is there a better way than to start listing
> the basemodel methods?
>
> I started to make the changes, but I don't think this is the right way
> forward. ?Other thoughts?
>
> http://docs.scipy.org/scipy/docs/scipy-docs/maxentropy.rst/

Try to reference directly the basemodel.xxx method in the autoclass for model

.. autoclass:: model

.. autosummary::
   :toctree: generated/

.....
   model.dual
   basemodel.fit
   model.grad
   model.log

Otherwise, it's better to have too much than too little information in the docs

Josef

>
> Cheers,
>
> Skipper
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From njs at pobox.com  Tue Feb 16 16:56:39 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 16 Feb 2010 13:56:39 -0800
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
Message-ID: <961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>

On Tue, Feb 16, 2010 at 12:37 PM,  <josef.pktd at gmail.com> wrote:
> 2010/2/16 St?fan van der Walt <stefan at sun.ac.za>:
>> Hi Nathan
>>
>> On 16 February 2010 22:18, Nathaniel Smith <njs at pobox.com> wrote:
>>> I have a need for solving some very large least squares regression
>>> problems -- but fortunately my model matrix can be generated
>>> incrementally, so there are some tricks to do the important
>>> calculations without actually loading the whole thing into memory. The
>>> resulting code is rather general, and might be of wider interest, so I
>>> thought I'd send a note here and see what people thought about
>>> upstreaming it somehow...
>>
>> This sounds interesting! ?Could you expand on the incremental
>> generation of the model matrix, and how it is made general?
>
> I have the same thought, Are you increasing by observation or by
> explanatory variable?

By observation. (This is good since hopefully you have more
observations than explanatory variables!) Actually generating the
model matrix in an incremental way is your problem; but as you
generate each 'strip' of the model matrix, you just hand it to the
code's 'update' method and then forget about it.

The basic idea inside 'update' is pretty elementary... if you have a
model matrix
  X = np.row_stack([X1, X2, X3, ...])
then what we need for least squares calculations is
  X'X = X1'X1 + X2'X2 + X3'X3 + ...
and we can compute this sum incrementally as the X_i matrices arrive.
Also, it is obviously easy to parallelize the matrix products (and
potentially the generation of the strips themselves, depending on your
situation), and those seem to be the bottleneck.

There's some linear algebra in working out how to calculate the
residual sum of squares and products without actually calculating the
residuals, you need to also accumulate X'y, weight handling, etc., but
no real magic.

For the QR code I use a recurrence relation I stole from some slides
by Simon Wood to compute R and Q'y incrementally; probably a real
incremental QR (e.g., "AS274", which is what R's biglm package uses)
would be better, but this one is easy to implement in terms of
non-incremental QR.

> For either case, I would be very interested to see this kind of
> incremental least squares in statsmodels. If you are able to license
> your parts as BSD, then I will look at it for sure.

I am.

> We have some plans for this but not yet implemented.

Oh? Do tell...

Anyway, attaching the code so you can see the details for yourselves.
It won't quite run as is, since it uses some utility routines I
haven't included, but you should be able to get the idea.

-- Nathaniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: incremental_qr.py
Type: text/x-python
Size: 4845 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100216/77969d62/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: incremental_ls.py
Type: text/x-python
Size: 28500 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100216/77969d62/attachment-0001.py>

From jsseabold at gmail.com  Tue Feb 16 17:05:44 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 16 Feb 2010 17:05:44 -0500
Subject: [SciPy-dev] maxentropy docs / parent class methods
In-Reply-To: <1cd32cbb1002161341r412704c8y65a3eed7000fce8f@mail.gmail.com>
References: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com> 
	<1cd32cbb1002161341r412704c8y65a3eed7000fce8f@mail.gmail.com>
Message-ID: <c048da1c1002161405t397449a2vc16ea42e39c02193@mail.gmail.com>

On Tue, Feb 16, 2010 at 4:41 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Feb 16, 2010 at 4:10 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> The docs on the methods for the maxentropy model class that are
>> inherited from basemodel are not picked up by sphinx. ?It seems that
>> most of the .rst files explicitly list the methods, but since
>> basemodel is not intended to be public (and some subclasses overwrite
>> the parent class methods), is there a better way than to start listing
>> the basemodel methods?
>>
>> I started to make the changes, but I don't think this is the right way
>> forward. ?Other thoughts?
>>
>> http://docs.scipy.org/scipy/docs/scipy-docs/maxentropy.rst/
>
> Try to reference directly the basemodel.xxx method in the autoclass for model
>
> .. autoclass:: model
>
> .. autosummary::
> ? :toctree: generated/
>
> .....
> ? model.dual
> ? basemodel.fit
> ? model.grad
> ? model.log
>
> Otherwise, it's better to have too much than too little information in the docs
>

That works, but it's still the basemodel namespace under model.  I was
looking to statsmodels to see how we autogenerate the inherited
methods.  I might play around with a local build so I don't clutter up
the logs on the wiki.  It's changed so that they show up for now while
I clean up a little more but is probably not a final solution.

Skipper


From arokem at berkeley.edu  Tue Feb 16 17:11:15 2010
From: arokem at berkeley.edu (Ariel Rokem)
Date: Tue, 16 Feb 2010 14:11:15 -0800
Subject: [SciPy-dev] bugfix in optimize.leastsq
Message-ID: <43958ee61002161411h27d3ee36y47ac263ccbe6f97b@mail.gmail.com>

Hi everyone,

I just submitted this ticket:

http://projects.scipy.org/scipy/ticket/1115

with a bug-fix in optimize.leastsq (a typo: "warning" s.b. "warnings").

Could someone take a look and tell me if it is correct/apply it?

Best -

Ariel

-- 
Ariel Rokem
Helen Wills Neuroscience Institute
University of California, Berkeley
http://argentum.ucbso.berkeley.edu/ariel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100216/720bb3d3/attachment.html>

From josef.pktd at gmail.com  Tue Feb 16 17:56:32 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 17:56:32 -0500
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
Message-ID: <1cd32cbb1002161456p2a5e2414t80e901165b3b2422@mail.gmail.com>

On Tue, Feb 16, 2010 at 4:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Feb 16, 2010 at 12:37 PM, ?<josef.pktd at gmail.com> wrote:
>> 2010/2/16 St?fan van der Walt <stefan at sun.ac.za>:
>>> Hi Nathan
>>>
>>> On 16 February 2010 22:18, Nathaniel Smith <njs at pobox.com> wrote:
>>>> I have a need for solving some very large least squares regression
>>>> problems -- but fortunately my model matrix can be generated
>>>> incrementally, so there are some tricks to do the important
>>>> calculations without actually loading the whole thing into memory. The
>>>> resulting code is rather general, and might be of wider interest, so I
>>>> thought I'd send a note here and see what people thought about
>>>> upstreaming it somehow...
>>>
>>> This sounds interesting! ?Could you expand on the incremental
>>> generation of the model matrix, and how it is made general?
>>
>> I have the same thought, Are you increasing by observation or by
>> explanatory variable?
>
> By observation. (This is good since hopefully you have more
> observations than explanatory variables!) Actually generating the
> model matrix in an incremental way is your problem; but as you
> generate each 'strip' of the model matrix, you just hand it to the
> code's 'update' method and then forget about it.
>
> The basic idea inside 'update' is pretty elementary... if you have a
> model matrix
> ?X = np.row_stack([X1, X2, X3, ...])
> then what we need for least squares calculations is
> ?X'X = X1'X1 + X2'X2 + X3'X3 + ...
> and we can compute this sum incrementally as the X_i matrices arrive.
> Also, it is obviously easy to parallelize the matrix products (and
> potentially the generation of the strips themselves, depending on your
> situation), and those seem to be the bottleneck.
>
> There's some linear algebra in working out how to calculate the
> residual sum of squares and products without actually calculating the
> residuals, you need to also accumulate X'y, weight handling, etc., but
> no real magic.
>
> For the QR code I use a recurrence relation I stole from some slides
> by Simon Wood to compute R and Q'y incrementally; probably a real
> incremental QR (e.g., "AS274", which is what R's biglm package uses)
> would be better, but this one is easy to implement in terms of
> non-incremental QR.
>
>> For either case, I would be very interested to see this kind of
>> incremental least squares in statsmodels. If you are able to license
>> your parts as BSD, then I will look at it for sure.
>
> I am.
>
>> We have some plans for this but not yet implemented.
>
> Oh? Do tell...

pandas has expanding ols, (besides moving ols) which is similar in the
idea that, for each new observation (or groups of observations seems
to be in your case), the ols estimate is calculated in a recursive
way.

However your code looks more efficient because of the incremental QR
updating, and that you update more summary/sufficient statistics, but
I just had a brief look and I don't remember the details of pandas.

My application is also the more in time series when validation of the
estimator requires continuous updating of the estimate. In pandas case
and in my case it is more computational efficiency that makes
incremental estimation attractive than memory requirements.
For this part I was looking more into using Kalman Filter, but,
although I have seen papers that use matrix decomposition to do more
efficient updating, my knowledge of recursive QR, or cholesky or ?? is
not great.

Actually, having more observations than explanatory variables is not
necessarily the only standard case anymore. In machine learning and in
econometrics in a "Data-Rich Environment", the number of observations
and variables might be of the same order. But none of the applications
that I know of in econometrics would struggle with memory problems.
There are many interesting cases for example for forecasting where
checking the forecast performance requires variable selection and
re-estimation in each period.

I will have a much closer look, but my impression is that the basic
structure is not so different from the model structure in statsmodels
that it wouldn't fit in. Also, I think that some of your functions
might also be useful for other models, and maybe also for pandas. And
I will be learning more about how to work with QR directly.

Thanks,

Josef

>
> Anyway, attaching the code so you can see the details for yourselves.
> It won't quite run as is, since it uses some utility routines I
> haven't included, but you should be able to get the idea.
>
> -- Nathaniel
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>


From josef.pktd at gmail.com  Tue Feb 16 18:04:44 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 18:04:44 -0500
Subject: [SciPy-dev] Best interface for computing the logarithm of the
	determinant?
In-Reply-To: <1cd32cbb1002161329s29f8da19l298ce62fd726125e@mail.gmail.com>
References: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
	<4B7B0BFE.8060109@student.matnat.uio.no>
	<1cd32cbb1002161329s29f8da19l298ce62fd726125e@mail.gmail.com>
Message-ID: <1cd32cbb1002161504h3c24f22q55cb7ed4ac749b35@mail.gmail.com>

On Tue, Feb 16, 2010 at 4:29 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Feb 16, 2010 at 4:19 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Nathaniel Smith wrote:
>>> So when you have a matrix whose determinant you want, it's often wise
>>> to compute the logarithm of the determinant instead of the determinant
>>> itself, because determinants involve lots and lots of multiplications
>>> and the result might otherwise underflow/overflow. Therefore, in
>>> scikits.sparse, I'd like to provide an API for doing this (and this is
>>> well-supported by the underlying libraries).
>>>
>>> But the problem is that for a general matrix, the determinant may be
>>> zero or negative. Obviously we can deal with this, but what's the best
>>> API? I'd like to use one consistently across the different
>>> factorizations in scikits.sparse, and perhaps eventually in numpy as
>>> well.
>>>
>>> Some options:
>>>
>>> 1) Split off the sign into a separate return value ('sign' may be 1, -1, 0):
>>> sign, value = logdet(A)
>>> actual_determinant = sign * exp(value)
>>>
>>> 2) Allow complex/infinite return values, even when A is a real matrix:
>>> logdet(eye(3)) == pi*1j
>>> logdet(zeros((3, 3))) == -Inf
>>
>> I'm +1 for this one. It is easy to interpret, and if one knows that the
>> result is positive (like after a successful CHOLMOD LL^T) is is easy
>> enough to take the real part by appending .real.
>>
>> Also it makes more sense for code that may be dealing with both complex
>> and real matrices compared to 1).
>
> I'm also in favor of this, but would prefer if promotion to complex
> only occurs if the result requires it.
> np.real_if_close ?might be safer than .real
>
> Josef
>
>>
>>>
>>> 3) "Scientific notation" (This is what UMFPACK's API does): return a
>>> mantissa and base-10 exponent:
>>> mantissa, exponent = logdet(A)
>>> actual_determinant = mantissa * 10 ** exponent
>>
>> This is my least preferred option, because "10" has nothing to do with
>> floats.
>>
>>>
>>> 4) Have separate functions for computing the sign, and the log of the
>>> absolute value (This is what GSL does, though it seems pointlessly
>>> inefficient):
>>> sign = sgndet(A)
>>> value = logdet(A)
>>> actual_determinant = sign * exp(value)
>>
>> Well, it doesn't have to be inefficient, you use cache the return value
>> internally on id(A) and possibly weak reference callbacks to remove the
>> cached values...but pretty ugly, yeah.

An alternative would be to provide 2 public functions where logdet()
would have the same behavior as np.log(np.det())  and any convenient
representation for an internal functions.

for multivariate normal distribution (loglikelihood) logdet would be
very useful. Is it possible to get an efficient implementation in
scipy or are the functions only available in CHOLMOD ?

Josef


>>
>> --
>> Dag Sverre
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>


From peridot.faceted at gmail.com  Tue Feb 16 18:14:02 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 16 Feb 2010 18:14:02 -0500
Subject: [SciPy-dev] Best interface for computing the logarithm of the
	determinant?
In-Reply-To: <4B7B0BFE.8060109@student.matnat.uio.no>
References: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
	<4B7B0BFE.8060109@student.matnat.uio.no>
Message-ID: <ce557a361002161514q4bd31908ud7ce9e8070a40a5b@mail.gmail.com>

On 16 February 2010 16:19, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Nathaniel Smith wrote:
>> So when you have a matrix whose determinant you want, it's often wise
>> to compute the logarithm of the determinant instead of the determinant
>> itself, because determinants involve lots and lots of multiplications
>> and the result might otherwise underflow/overflow. Therefore, in
>> scikits.sparse, I'd like to provide an API for doing this (and this is
>> well-supported by the underlying libraries).
>>
>> But the problem is that for a general matrix, the determinant may be
>> zero or negative. Obviously we can deal with this, but what's the best
>> API? I'd like to use one consistently across the different
>> factorizations in scikits.sparse, and perhaps eventually in numpy as
>> well.
>>
>> Some options:
>>
>> 1) Split off the sign into a separate return value ('sign' may be 1, -1, 0):
>> sign, value = logdet(A)
>> actual_determinant = sign * exp(value)

I kind of like this one, but it could definitely become a pain. For
what it's worth, for complex numbers it means you get a polar
representation of the complex number (complex number of magnitude 1
and real log of the magnitude).

>> 2) Allow complex/infinite return values, even when A is a real matrix:
>> logdet(eye(3)) == pi*1j
>> logdet(zeros((3, 3))) == -Inf
>
> I'm +1 for this one. It is easy to interpret, and if one knows that the
> result is positive (like after a successful CHOLMOD LL^T) is is easy
> enough to take the real part by appending .real.
>
> Also it makes more sense for code that may be dealing with both complex
> and real matrices compared to 1).

I don't really like this very much, since it can be really startling
to suddenly have complex numbers appear out of nowhere. But at least
it's consistent.

>>
>> 3) "Scientific notation" (This is what UMFPACK's API does): return a
>> mantissa and base-10 exponent:
>> mantissa, exponent = logdet(A)
>> actual_determinant = mantissa * 10 ** exponent
>
> This is my least preferred option, because "10" has nothing to do with
> floats.

This lends itself well to a multiply-and-rescale-when-necessary
approach, and avoids transcendental functions entirely; if the 10
offends, you could use 2 or 16, I suppose. But this kind of
count-every-flop approach is not really appropriate for scipy/numpy.

>> 4) Have separate functions for computing the sign, and the log of the
>> absolute value (This is what GSL does, though it seems pointlessly
>> inefficient):
>> sign = sgndet(A)
>> value = logdet(A)
>> actual_determinant = sign * exp(value)
>
> Well, it doesn't have to be inefficient, you use cache the return value
> internally on id(A) and possibly weak reference callbacks to remove the
> cached values...but pretty ugly, yeah.

Since dets are (potentially) expensive operations, no, I don't like
this either. Of course, one might want a sgndet function in addition
to the usual API.

Anne


From josef.pktd at gmail.com  Tue Feb 16 23:51:40 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Feb 2010 23:51:40 -0500
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <1cd32cbb1002161456p2a5e2414t80e901165b3b2422@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
	<1cd32cbb1002161456p2a5e2414t80e901165b3b2422@mail.gmail.com>
Message-ID: <1cd32cbb1002162051t2290ac6y2a8bcf243de5127@mail.gmail.com>

On Tue, Feb 16, 2010 at 5:56 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Feb 16, 2010 at 4:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Tue, Feb 16, 2010 at 12:37 PM, ?<josef.pktd at gmail.com> wrote:
>>> 2010/2/16 St?fan van der Walt <stefan at sun.ac.za>:
>>>> Hi Nathan
>>>>
>>>> On 16 February 2010 22:18, Nathaniel Smith <njs at pobox.com> wrote:
>>>>> I have a need for solving some very large least squares regression
>>>>> problems -- but fortunately my model matrix can be generated
>>>>> incrementally, so there are some tricks to do the important
>>>>> calculations without actually loading the whole thing into memory. The
>>>>> resulting code is rather general, and might be of wider interest, so I
>>>>> thought I'd send a note here and see what people thought about
>>>>> upstreaming it somehow...
>>>>
>>>> This sounds interesting! ?Could you expand on the incremental
>>>> generation of the model matrix, and how it is made general?
>>>
>>> I have the same thought, Are you increasing by observation or by
>>> explanatory variable?
>>
>> By observation. (This is good since hopefully you have more
>> observations than explanatory variables!) Actually generating the
>> model matrix in an incremental way is your problem; but as you
>> generate each 'strip' of the model matrix, you just hand it to the
>> code's 'update' method and then forget about it.
>>
>> The basic idea inside 'update' is pretty elementary... if you have a
>> model matrix
>> ?X = np.row_stack([X1, X2, X3, ...])
>> then what we need for least squares calculations is
>> ?X'X = X1'X1 + X2'X2 + X3'X3 + ...
>> and we can compute this sum incrementally as the X_i matrices arrive.
>> Also, it is obviously easy to parallelize the matrix products (and
>> potentially the generation of the strips themselves, depending on your
>> situation), and those seem to be the bottleneck.
>>
>> There's some linear algebra in working out how to calculate the
>> residual sum of squares and products without actually calculating the
>> residuals, you need to also accumulate X'y, weight handling, etc., but
>> no real magic.
>>
>> For the QR code I use a recurrence relation I stole from some slides
>> by Simon Wood to compute R and Q'y incrementally; probably a real
>> incremental QR (e.g., "AS274", which is what R's biglm package uses)
>> would be better, but this one is easy to implement in terms of
>> non-incremental QR.
>>
>>> For either case, I would be very interested to see this kind of
>>> incremental least squares in statsmodels. If you are able to license
>>> your parts as BSD, then I will look at it for sure.
>>
>> I am.
>>
>>> We have some plans for this but not yet implemented.
>>
>> Oh? Do tell...
>
> pandas has expanding ols, (besides moving ols) which is similar in the
> idea that, for each new observation (or groups of observations seems
> to be in your case), the ols estimate is calculated in a recursive
> way.
>
> However your code looks more efficient because of the incremental QR
> updating, and that you update more summary/sufficient statistics, but
> I just had a brief look and I don't remember the details of pandas.
>
> My application is also the more in time series when validation of the
> estimator requires continuous updating of the estimate. In pandas case
> and in my case it is more computational efficiency that makes
> incremental estimation attractive than memory requirements.
> For this part I was looking more into using Kalman Filter, but,
> although I have seen papers that use matrix decomposition to do more
> efficient updating, my knowledge of recursive QR, or cholesky or ?? is
> not great.
>
> Actually, having more observations than explanatory variables is not
> necessarily the only standard case anymore. In machine learning and in
> econometrics in a "Data-Rich Environment", the number of observations
> and variables might be of the same order. But none of the applications
> that I know of in econometrics would struggle with memory problems.
> There are many interesting cases for example for forecasting where
> checking the forecast performance requires variable selection and
> re-estimation in each period.
>
> I will have a much closer look, but my impression is that the basic
> structure is not so different from the model structure in statsmodels
> that it wouldn't fit in. Also, I think that some of your functions
> might also be useful for other models, and maybe also for pandas. And
> I will be learning more about how to work with QR directly.
>
> Thanks,
>
> Josef
>
>>
>> Anyway, attaching the code so you can see the details for yourselves.
>> It won't quite run as is, since it uses some utility routines I
>> haven't included, but you should be able to get the idea.

just a BTW:

I found this comment in incremental_ls

        # R and Scipy disagree by about 3 orders of magnitude here:
        #  pf(4704.7767809675142416, 4, 12, lower.tail=FALSE) == 4.68-19
        #  1 - f.cdf(4704.7767809675142416, 4, 12) == 1.11-16
        # which I guess is most likely Scipy having limited resolution in the
        # tails.


>>> from scipy import stats
>>> stats.f.sf(4704.7767809675142416, 4, 12)
4.6848221938640787e-019

some of our tails are also pretty good

Josef

>>
>> -- Nathaniel
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>>
>


From njs at pobox.com  Wed Feb 17 00:31:16 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 16 Feb 2010 21:31:16 -0800
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <1cd32cbb1002162051t2290ac6y2a8bcf243de5127@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
	<1cd32cbb1002161456p2a5e2414t80e901165b3b2422@mail.gmail.com>
	<1cd32cbb1002162051t2290ac6y2a8bcf243de5127@mail.gmail.com>
Message-ID: <961fa2b41002162131s7e1815e8gb338bfbef0268346@mail.gmail.com>

On Tue, Feb 16, 2010 at 8:51 PM,  <josef.pktd at gmail.com> wrote:
> I found this comment in incremental_ls
>
> ? ? ? ?# R and Scipy disagree by about 3 orders of magnitude here:
> ? ? ? ?# ?pf(4704.7767809675142416, 4, 12, lower.tail=FALSE) == 4.68-19
> ? ? ? ?# ?1 - f.cdf(4704.7767809675142416, 4, 12) == 1.11-16
> ? ? ? ?# which I guess is most likely Scipy having limited resolution in the
> ? ? ? ?# tails.
>
>>>> from scipy import stats
>>>> stats.f.sf(4704.7767809675142416, 4, 12)
> 4.6848221938640787e-019
>
> some of our tails are also pretty good

Oh, awesome, I'd missed .sf() -- thanks for the pointer! It's
aesthetically annoying to plot p-values on a graph and always have
weird discontinuities where they underflow...

-- Nathaniel


From josef.pktd at gmail.com  Wed Feb 17 00:55:58 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 17 Feb 2010 00:55:58 -0500
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <961fa2b41002162131s7e1815e8gb338bfbef0268346@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
	<1cd32cbb1002161456p2a5e2414t80e901165b3b2422@mail.gmail.com>
	<1cd32cbb1002162051t2290ac6y2a8bcf243de5127@mail.gmail.com>
	<961fa2b41002162131s7e1815e8gb338bfbef0268346@mail.gmail.com>
Message-ID: <1cd32cbb1002162155q303cac6ekc0c8570266bcc3ce@mail.gmail.com>

On Wed, Feb 17, 2010 at 12:31 AM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Feb 16, 2010 at 8:51 PM, ?<josef.pktd at gmail.com> wrote:
>> I found this comment in incremental_ls
>>
>> ? ? ? ?# R and Scipy disagree by about 3 orders of magnitude here:
>> ? ? ? ?# ?pf(4704.7767809675142416, 4, 12, lower.tail=FALSE) == 4.68-19
>> ? ? ? ?# ?1 - f.cdf(4704.7767809675142416, 4, 12) == 1.11-16
>> ? ? ? ?# which I guess is most likely Scipy having limited resolution in the
>> ? ? ? ?# tails.
>>
>>>>> from scipy import stats
>>>>> stats.f.sf(4704.7767809675142416, 4, 12)
>> 4.6848221938640787e-019
>>
>> some of our tails are also pretty good
>
> Oh, awesome, I'd missed .sf() -- thanks for the pointer! It's
> aesthetically annoying to plot p-values on a graph and always have
> weird discontinuities where they underflow...

I'm glad to help, (especially if I can show off things that work)

There are still some strange things left in the range smaller than
1e-8, but every once in a while we catch one and stats.distributions
is slowly improving.
Filing a ticket for these cases is useful, sometimes it's easy to
improve when we find the loss in precision, sometimes it's buried in
the c or fortran code in scipy.special and we have to live with 1e-8
or whichever precision we get. (Although it won't make much difference
to the statistical significance of any test.)

Josef

>
> -- Nathaniel
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From njs at pobox.com  Wed Feb 17 00:58:57 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 16 Feb 2010 21:58:57 -0800
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <1cd32cbb1002161456p2a5e2414t80e901165b3b2422@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
	<1cd32cbb1002161456p2a5e2414t80e901165b3b2422@mail.gmail.com>
Message-ID: <961fa2b41002162158u77f1e776xd732be0548ff9f05@mail.gmail.com>

On Tue, Feb 16, 2010 at 2:56 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Feb 16, 2010 at 4:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
> pandas has expanding ols, (besides moving ols) which is similar in the
> idea that, for each new observation (or groups of observations seems
> to be in your case), the ols estimate is calculated in a recursive
> way.

Yes, and similar code can be used -- certainly you can calculate both
expanding and moving ols using what I've been calling "the cholesky
approach". (CHOLMOD actually includes code for "update" and "downdate"
operations, i.e., adding/removing rows to your model matrix directly
on the decomposition, without saving X'X separately and
re-decomposing. Of course, if your model matrix isn't sparse then
CHOLMOD is irrelevant.)

> However your code looks more efficient because of the incremental QR
> updating, and that you update more summary/sufficient statistics, but
> I just had a brief look and I don't remember the details of pandas.

I actually don't use the QR approach much, because it is much slower
than the Cholesky approach. In principle the QR version might be more
numerically stable (so far I've been lucky and haven't had problems),
and it is useful in some other cases (e.g. the thoroughly magic spline
fitting routines in R package "mgcv" have a mode that lets you fit
very large data sets by doing the QR decomposition in advance with
something like incremental_qr, and then pass in the decomposition to
the magic fitting functions), so it's worth keeping around. But I
don't know if it's possible to do a QR "downdate" for the "moving ols"
case to toss out observations at the trailing edge.

> My application is also the more in time series when validation of the
> estimator requires continuous updating of the estimate. In pandas case
> and in my case it is more computational efficiency that makes
> incremental estimation attractive than memory requirements.
> For this part I was looking more into using Kalman Filter, but,
> although I have seen papers that use matrix decomposition to do more
> efficient updating, my knowledge of recursive QR, or cholesky or ?? is
> not great.

If your goal is robust, fast, incremental OLS for general consumption
on potentially misbehaved data, then I'd definitely recommend looking
into true incremental QR algorithms, like that "AS274" I mentioned. It
looks like the biglm folks figured out how to do incremental sandwich
estimation too: see slide 32-ish:
http://faculty.washington.edu/tlumley/tutorials/user-biglm.pdf

> Actually, having more observations than explanatory variables is not
> necessarily the only standard case anymore. In machine learning and in
> econometrics in a "Data-Rich Environment", the number of observations
> and variables might be of the same order.

Yes, but then you need some regularization to make the problem
well-defined -- simple OLS will blow up. (Of course, penalized least
squares may work fine, and is just as tractable as OLS; I think the
same techniques work, though I haven't checked the details.)

> But none of the applications
> that I know of in econometrics would struggle with memory problems.

Yes, my data sets are from EEG (brain waves), so I get 250 samples per
second (or more), and I'm trying to estimate whole impulse response
curves (so I have predictors like "event with property x1 occurred 100
ms ago", "event with property x1 occurred 104 ms ago", ...), which
leads to thousands of predictors and millions of observations = tens
of gigabyte sized model matrices. It's a somewhat unusual situation to
be in, but I guess data sets grow fast enough that it may become more
common...

-- Nathaniel


From stefan at sun.ac.za  Wed Feb 17 02:07:23 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 17 Feb 2010 09:07:23 +0200
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com> 
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com> 
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com> 
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
Message-ID: <9457e7c81002162307q3dc59492y9905dcc525a49575@mail.gmail.com>

Hi Nathaniel

On 16 February 2010 23:56, Nathaniel Smith <njs at pobox.com> wrote:
> The basic idea inside 'update' is pretty elementary... if you have a
> model matrix
> ?X = np.row_stack([X1, X2, X3, ...])
> then what we need for least squares calculations is
> ?X'X = X1'X1 + X2'X2 + X3'X3 + ...
> and we can compute this sum incrementally as the X_i matrices arrive.

Forming the product A^T A is often a bad idea from a numerical
perspective.  In Chapter 3 of Ake Bjork's "Numerical Methods for Least
Squares Problems", he talks about "update" problems.  He mentions that
"...the solution should be accurate up to the limitation of data and
conditioning of the problem; i.e., a stable numerical method must be
used."

He describes the Kalman-based update method (which bears some
resemblance to yours), but says that "the main disadvantage...is its
serious sensitivity to roundoff errors.  The updating algorithms based
on orthogonal transformations developed in the following sections are
generally to be preferred."  He then goes into more detail on updating
the QR and Gram-Schmidt decompositions.

Not sure if that helps, but it may be worth reading that chapter.

> For the QR code I use a recurrence relation I stole from some slides
> by Simon Wood to compute R and Q'y incrementally; probably a real
> incremental QR (e.g., "AS274", which is what R's biglm package uses)
> would be better, but this one is easy to implement in terms of
> non-incremental QR.

Incremental QR is something we should implement in scipy.linalg, for sure.

Regards
St?fan


From charlesr.harris at gmail.com  Wed Feb 17 02:46:25 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 17 Feb 2010 00:46:25 -0700
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
Message-ID: <e06186141002162346r1299f45bi49a54639cf535b08@mail.gmail.com>

On Tue, Feb 16, 2010 at 2:56 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Tue, Feb 16, 2010 at 12:37 PM,  <josef.pktd at gmail.com> wrote:
> > 2010/2/16 St?fan van der Walt <stefan at sun.ac.za>:
> >> Hi Nathan
> >>
> >> On 16 February 2010 22:18, Nathaniel Smith <njs at pobox.com> wrote:
> >>> I have a need for solving some very large least squares regression
> >>> problems -- but fortunately my model matrix can be generated
> >>> incrementally, so there are some tricks to do the important
> >>> calculations without actually loading the whole thing into memory. The
> >>> resulting code is rather general, and might be of wider interest, so I
> >>> thought I'd send a note here and see what people thought about
> >>> upstreaming it somehow...
> >>
> >> This sounds interesting!  Could you expand on the incremental
> >> generation of the model matrix, and how it is made general?
> >
> > I have the same thought, Are you increasing by observation or by
> > explanatory variable?
>
> By observation. (This is good since hopefully you have more
> observations than explanatory variables!) Actually generating the
> model matrix in an incremental way is your problem; but as you
> generate each 'strip' of the model matrix, you just hand it to the
> code's 'update' method and then forget about it.
>
> The basic idea inside 'update' is pretty elementary... if you have a
> model matrix
>  X = np.row_stack([X1, X2, X3, ...])
> then what we need for least squares calculations is
>  X'X = X1'X1 + X2'X2 + X3'X3 + ...
> and we can compute this sum incrementally as the X_i matrices arrive.
> Also, it is obviously easy to parallelize the matrix products (and
> potentially the generation of the strips themselves, depending on your
> situation), and those seem to be the bottleneck.
>
>
Did you look into Kalman filters? They probably aren't the most efficient
approach but they give you incremental solutions along the way if you want
them. There are also various factored versions available which avoid the
final decompostion.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100217/f56ee5dc/attachment.html>

From dagss at student.matnat.uio.no  Wed Feb 17 02:49:59 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Feb 2010 08:49:59 +0100
Subject: [SciPy-dev] Best interface for computing the logarithm of the
 determinant?
In-Reply-To: <1cd32cbb1002161504h3c24f22q55cb7ed4ac749b35@mail.gmail.com>
References: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>	<4B7B0BFE.8060109@student.matnat.uio.no>	<1cd32cbb1002161329s29f8da19l298ce62fd726125e@mail.gmail.com>
	<1cd32cbb1002161504h3c24f22q55cb7ed4ac749b35@mail.gmail.com>
Message-ID: <4B7B9FA7.4060806@student.matnat.uio.no>

josef.pktd at gmail.com wrote:
> On Tue, Feb 16, 2010 at 4:29 PM,  <josef.pktd at gmail.com> wrote:
>   
>> On Tue, Feb 16, 2010 at 4:19 PM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
>>     
>>> Nathaniel Smith wrote:
>>>       
>>>> So when you have a matrix whose determinant you want, it's often wise
>>>> to compute the logarithm of the determinant instead of the determinant
>>>> itself, because determinants involve lots and lots of multiplications
>>>> and the result might otherwise underflow/overflow. Therefore, in
>>>> scikits.sparse, I'd like to provide an API for doing this (and this is
>>>> well-supported by the underlying libraries).
>>>>
>>>> But the problem is that for a general matrix, the determinant may be
>>>> zero or negative. Obviously we can deal with this, but what's the best
>>>> API? I'd like to use one consistently across the different
>>>> factorizations in scikits.sparse, and perhaps eventually in numpy as
>>>> well.
>>>>
>>>> Some options:
>>>>
>>>> 1) Split off the sign into a separate return value ('sign' may be 1, -1, 0):
>>>> sign, value = logdet(A)
>>>> actual_determinant = sign * exp(value)
>>>>
>>>> 2) Allow complex/infinite return values, even when A is a real matrix:
>>>> logdet(eye(3)) == pi*1j
>>>> logdet(zeros((3, 3))) == -Inf
>>>>         
>>> I'm +1 for this one. It is easy to interpret, and if one knows that the
>>> result is positive (like after a successful CHOLMOD LL^T) is is easy
>>> enough to take the real part by appending .real.
>>>
>>> Also it makes more sense for code that may be dealing with both complex
>>> and real matrices compared to 1).
>>>       
>> I'm also in favor of this, but would prefer if promotion to complex
>> only occurs if the result requires it.
>> np.real_if_close  might be safer than .real
>>
>> Josef
>>
>>     
>>>> 3) "Scientific notation" (This is what UMFPACK's API does): return a
>>>> mantissa and base-10 exponent:
>>>> mantissa, exponent = logdet(A)
>>>> actual_determinant = mantissa * 10 ** exponent
>>>>         
>>> This is my least preferred option, because "10" has nothing to do with
>>> floats.
>>>
>>>       
>>>> 4) Have separate functions for computing the sign, and the log of the
>>>> absolute value (This is what GSL does, though it seems pointlessly
>>>> inefficient):
>>>> sign = sgndet(A)
>>>> value = logdet(A)
>>>> actual_determinant = sign * exp(value)
>>>>         
>>> Well, it doesn't have to be inefficient, you use cache the return value
>>> internally on id(A) and possibly weak reference callbacks to remove the
>>> cached values...but pretty ugly, yeah.
>>>       
>
> An alternative would be to provide 2 public functions where logdet()
> would have the same behavior as np.log(np.det())  and any convenient
> representation for an internal functions.
>
> for multivariate normal distribution (loglikelihood) logdet would be
> very useful. Is it possible to get an efficient implementation in
> scipy or are the functions only available in CHOLMOD ?
>   
You mean like this?

2 * np.sum(np.log(np.diagonal(np.linalg.cholesky(A))))

(Using scipy.linalg.cho_solve would be very slightly faster I guess.)

Of course, in practice you usually get the Cholesky factor once and then 
reuse it both for solving and for getting the determinant. So what you 
want is a symbolic "solution" object which allows solving, getting 
determinant etc. scikits.sparse has this for Cholesky factors but SciPy 
doesn't.

I don't think SciPy should get it though -- SciPy usually has a 
"lower-level" approach. Therefore I've written "oomatrix.py" 
(polymorphic matrices) which does this in a higher-level way for my own 
purposes (not available online yet...will have to see when I get time to 
polish it...):

A, B = ...numpy arrays...
M = oomatrix.matrix(A) # Create immutable matrix object from array
C M.solve_right(B, algorithm='cholesky') # does cholesky and caches it
M.log_determinant() # from cached decomposition
P, L = M.cholesky()

Alternatively, constructing oomatrix.matrix(A, sparse=True) will 
transparently switch to using CHOLMOD as backend.

Dag Sverre


From njs at pobox.com  Wed Feb 17 03:32:45 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 17 Feb 2010 00:32:45 -0800
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <9457e7c81002162307q3dc59492y9905dcc525a49575@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
	<9457e7c81002162307q3dc59492y9905dcc525a49575@mail.gmail.com>
Message-ID: <961fa2b41002170032u72d6a22ajafd95af99da6f57d@mail.gmail.com>

2010/2/16 St?fan van der Walt <stefan at sun.ac.za>:
> Forming the product A^T A is often a bad idea from a numerical
> perspective.

Right.

> In Chapter 3 of Ake Bjork's "Numerical Methods for Least
> Squares Problems", he talks about "update" problems. ?He mentions that
> "...the solution should be accurate up to the limitation of data and
> conditioning of the problem; i.e., a stable numerical method must be
> used."
>
> He describes the Kalman-based update method (which bears some
> resemblance to yours), but says that "the main disadvantage...is its
> serious sensitivity to roundoff errors. ?The updating algorithms based
> on orthogonal transformations developed in the following sections are
> generally to be preferred." ?He then goes into more detail on updating
> the QR and Gram-Schmidt decompositions.
>
> Not sure if that helps, but it may be worth reading that chapter.

Definitely -- thanks for the reference! Looks like the library has a
copy available, too...

(Not that I'll necessarily bother for my own use, since like I said,
my problems seem to be fairly well conditioned and the sparse X'X +
Cholesky approach is very fast, and also generalizes easily to mixed
effect models, which I'm currently working on implementing.)

>> For the QR code I use a recurrence relation I stole from some slides
>> by Simon Wood to compute R and Q'y incrementally; probably a real
>> incremental QR (e.g., "AS274", which is what R's biglm package uses)
>> would be better, but this one is easy to implement in terms of
>> non-incremental QR.
>
> Incremental QR is something we should implement in scipy.linalg, for sure.

+1

-- Nathaniel


From tom.grydeland at gmail.com  Wed Feb 17 05:22:17 2010
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Wed, 17 Feb 2010 11:22:17 +0100
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <4B7215FE.5030803@silveregg.co.jp>
References: <4B7215FE.5030803@silveregg.co.jp>
Message-ID: <c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com>

On Wed, Feb 10, 2010 at 3:12 AM, David Cournapeau <david at silveregg.co.jp> wrote:
> Hi,
>
> I noticed that some of the docstrings I have written for DCT have been
> changed to latex format. While I have no issue with having latex in the
> documentation, I thought the consensus was to use them sparingly in
> docstrings ?

I am probably the one to "blame" here.  I know I have edited the DCT docstrings.

What is considered "sparingly" is obviously different from one person
to the next, and I have taken it to mean roughly "where pure text is
insufficient", and (along the discussion re numpy.fft and friends
<http://docs.scipy.org/numpy/docs/numpy.fft/>) okay for module-level
docstrings but avoid it for functions.  Since I have written a lot of
latex, I might have a lower threshold than others here.

> For example, the dct I formula used to be (fixed width font assumed):
>
> for 0 <= k < N,
>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?N-1
> y[k] = x[0] + (-1)**k x[N-1] + 2 * sum x[n]*cos(pi*k*n/(N-1))
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?n=0
>
> But now, it is:
>
> y_k = x_0 + (-1)^k x_{N-1} + 2\\sum_{n=1}^{N-2} x_n
> ? ? ? ? \\cos\\left({\\pi nk\\over N-1}\\right),
> ? ? ? ? \\qquad 0 \\le k < N.
>
> I much prefer the former (the latter is unreadable in a terminal IMO). I
> have of course no issue in putting the latex formula in the scipy docs,

Similarly, the former is unreadable or incorrect in the web interface
where the latter is useful:
http://docs.scipy.org/scipy/docs/scipy.fftpack.realtransforms.dct/

If you change it back, please observe that the limits on the summation
are different for the two versions.

> David

Regards,

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From ralf.gommers at googlemail.com  Wed Feb 17 06:16:21 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Wed, 17 Feb 2010 19:16:21 +0800
Subject: [SciPy-dev] maxentropy docs / parent class methods
In-Reply-To: <c048da1c1002161405t397449a2vc16ea42e39c02193@mail.gmail.com>
References: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com>
	<1cd32cbb1002161341r412704c8y65a3eed7000fce8f@mail.gmail.com>
	<c048da1c1002161405t397449a2vc16ea42e39c02193@mail.gmail.com>
Message-ID: <dde7764a1002170316s760e2335kd402db5b5c82ec78@mail.gmail.com>

On Wed, Feb 17, 2010 at 6:05 AM, Skipper Seabold <jsseabold at gmail.com>wrote:

> On Tue, Feb 16, 2010 at 4:41 PM,  <josef.pktd at gmail.com> wrote:
> > On Tue, Feb 16, 2010 at 4:10 PM, Skipper Seabold <jsseabold at gmail.com>
> wrote:
> >> The docs on the methods for the maxentropy model class that are
> >> inherited from basemodel are not picked up by sphinx.  It seems that
> >> most of the .rst files explicitly list the methods, but since
> >> basemodel is not intended to be public (and some subclasses overwrite
> >> the parent class methods), is there a better way than to start listing
> >> the basemodel methods?
> >>
> >> I started to make the changes, but I don't think this is the right way
> >> forward.  Other thoughts?
> >>
> >> http://docs.scipy.org/scipy/docs/scipy-docs/maxentropy.rst/
> >
> > Try to reference directly the basemodel.xxx method in the autoclass for
> model
> >
> > .. autoclass:: model
> >
> > .. autosummary::
> >   :toctree: generated/
> >
> > .....
> >   model.dual
> >   basemodel.fit
> >   model.grad
> >   model.log
> >
> > Otherwise, it's better to have too much than too little information in
> the docs
> >
>
> That works, but it's still the basemodel namespace under model.  I was
> looking to statsmodels to see how we autogenerate the inherited
> methods.  I might play around with a local build so I don't clutter up
> the logs on the wiki.  It's changed so that they show up for now while
> I clean up a little more but is probably not a final solution.
>
> Those methods do not belong just to 'model' but also to 'bigmodel' and
'conditionalmodel'. So listing them under 'model' is a bit arbitrary. I
think the correct way to do this is to list them under an .. autoclass::
basemodel, with a note that this class contains shared functionality and
should not be instantiated directly.

Also, I do not see in the code that basemodel is not public. Why did you
conclude this? If a user wants to implement a new model, shouldn't he
inherit from basemodel? In that case it should be public. If that's not the
case, an __all__ dict should be added to the module to indicate what is
public and what is not.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100217/92a34d32/attachment.html>

From ralf.gommers at googlemail.com  Wed Feb 17 07:21:05 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Wed, 17 Feb 2010 20:21:05 +0800
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com>
Message-ID: <dde7764a1002170421y4da939aaxbc14caaff373b0d@mail.gmail.com>

On Wed, Feb 17, 2010 at 6:22 PM, Tom Grydeland <tom.grydeland at gmail.com>wrote:

> On Wed, Feb 10, 2010 at 3:12 AM, David Cournapeau <david at silveregg.co.jp>
> wrote:
> > Hi,
> >
> > I noticed that some of the docstrings I have written for DCT have been
> > changed to latex format. While I have no issue with having latex in the
> > documentation, I thought the consensus was to use them sparingly in
> > docstrings ?
>
> I am probably the one to "blame" here.  I know I have edited the DCT
> docstrings.
>
> What is considered "sparingly" is obviously different from one person
> to the next, and I have taken it to mean roughly "where pure text is
> insufficient", and (along the discussion re numpy.fft and friends
> <http://docs.scipy.org/numpy/docs/numpy.fft/>) okay for module-level
> docstrings but avoid it for functions.
>

That's about right. In function docstrings latex should only sparingly be
used in the Notes section (see
http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines). This policy has
worked pretty well so far, the problem for this particular docstring is that
the Notes section is huge with lots of maths.

>
> > For example, the dct I formula used to be (fixed width font assumed):
> >
> > for 0 <= k < N,
> >
> >                                    N-1
> > y[k] = x[0] + (-1)**k x[N-1] + 2 * sum x[n]*cos(pi*k*n/(N-1))
> >                                    n=0
> >
> > But now, it is:
> >
> > y_k = x_0 + (-1)^k x_{N-1} + 2\\sum_{n=1}^{N-2} x_n
> >         \\cos\\left({\\pi nk\\over N-1}\\right),
> >         \\qquad 0 \\le k < N.
> >
> > I much prefer the former (the latter is unreadable in a terminal IMO). I
> > have of course no issue in putting the latex formula in the scipy docs,
>
> Similarly, the former is unreadable or incorrect in the web interface
> where the latter is useful:
> http://docs.scipy.org/scipy/docs/scipy.fftpack.realtransforms.dct/
>
> If you change it back, please observe that the limits on the summation
> are different for the two versions.


Fixing this docstring for both terminal and html/pdf does not seem possible,
short of removing content or writing a new Sphinx plugin. For now I would be
in favor of keeping the latex, because going to the html doc for terminal
users is doable. Users of the html docs on the other hand are unlikely to
all know their way around a terminal.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100217/1711d2c3/attachment.html>

From josef.pktd at gmail.com  Wed Feb 17 09:16:31 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 17 Feb 2010 09:16:31 -0500
Subject: [SciPy-dev] Best interface for computing the logarithm of the
	determinant?
In-Reply-To: <4B7B9FA7.4060806@student.matnat.uio.no>
References: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
	<4B7B0BFE.8060109@student.matnat.uio.no>
	<1cd32cbb1002161329s29f8da19l298ce62fd726125e@mail.gmail.com>
	<1cd32cbb1002161504h3c24f22q55cb7ed4ac749b35@mail.gmail.com>
	<4B7B9FA7.4060806@student.matnat.uio.no>
Message-ID: <1cd32cbb1002170616r60779c28p3ac6258f6c74b6d9@mail.gmail.com>

On Wed, Feb 17, 2010 at 2:49 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> josef.pktd at gmail.com wrote:
>> On Tue, Feb 16, 2010 at 4:29 PM, ?<josef.pktd at gmail.com> wrote:
>>
>>> On Tue, Feb 16, 2010 at 4:19 PM, Dag Sverre Seljebotn
>>> <dagss at student.matnat.uio.no> wrote:
>>>
>>>> Nathaniel Smith wrote:
>>>>
>>>>> So when you have a matrix whose determinant you want, it's often wise
>>>>> to compute the logarithm of the determinant instead of the determinant
>>>>> itself, because determinants involve lots and lots of multiplications
>>>>> and the result might otherwise underflow/overflow. Therefore, in
>>>>> scikits.sparse, I'd like to provide an API for doing this (and this is
>>>>> well-supported by the underlying libraries).
>>>>>
>>>>> But the problem is that for a general matrix, the determinant may be
>>>>> zero or negative. Obviously we can deal with this, but what's the best
>>>>> API? I'd like to use one consistently across the different
>>>>> factorizations in scikits.sparse, and perhaps eventually in numpy as
>>>>> well.
>>>>>
>>>>> Some options:
>>>>>
>>>>> 1) Split off the sign into a separate return value ('sign' may be 1, -1, 0):
>>>>> sign, value = logdet(A)
>>>>> actual_determinant = sign * exp(value)
>>>>>
>>>>> 2) Allow complex/infinite return values, even when A is a real matrix:
>>>>> logdet(eye(3)) == pi*1j
>>>>> logdet(zeros((3, 3))) == -Inf
>>>>>
>>>> I'm +1 for this one. It is easy to interpret, and if one knows that the
>>>> result is positive (like after a successful CHOLMOD LL^T) is is easy
>>>> enough to take the real part by appending .real.
>>>>
>>>> Also it makes more sense for code that may be dealing with both complex
>>>> and real matrices compared to 1).
>>>>
>>> I'm also in favor of this, but would prefer if promotion to complex
>>> only occurs if the result requires it.
>>> np.real_if_close ?might be safer than .real
>>>
>>> Josef
>>>
>>>
>>>>> 3) "Scientific notation" (This is what UMFPACK's API does): return a
>>>>> mantissa and base-10 exponent:
>>>>> mantissa, exponent = logdet(A)
>>>>> actual_determinant = mantissa * 10 ** exponent
>>>>>
>>>> This is my least preferred option, because "10" has nothing to do with
>>>> floats.
>>>>
>>>>
>>>>> 4) Have separate functions for computing the sign, and the log of the
>>>>> absolute value (This is what GSL does, though it seems pointlessly
>>>>> inefficient):
>>>>> sign = sgndet(A)
>>>>> value = logdet(A)
>>>>> actual_determinant = sign * exp(value)
>>>>>
>>>> Well, it doesn't have to be inefficient, you use cache the return value
>>>> internally on id(A) and possibly weak reference callbacks to remove the
>>>> cached values...but pretty ugly, yeah.
>>>>
>>
>> An alternative would be to provide 2 public functions where logdet()
>> would have the same behavior as np.log(np.det()) ?and any convenient
>> representation for an internal functions.
>>
>> for multivariate normal distribution (loglikelihood) logdet would be
>> very useful. Is it possible to get an efficient implementation in
>> scipy or are the functions only available in CHOLMOD ?
>>
> You mean like this?
>
> 2 * np.sum(np.log(np.diagonal(np.linalg.cholesky(A))))

Thanks, I didn't know it is as simple as this.

>
> (Using scipy.linalg.cho_solve would be very slightly faster I guess.)
>
> Of course, in practice you usually get the Cholesky factor once and then
> reuse it both for solving and for getting the determinant. So what you
> want is a symbolic "solution" object which allows solving, getting
> determinant etc. scikits.sparse has this for Cholesky factors but SciPy
> doesn't.
>
> I don't think SciPy should get it though -- SciPy usually has a
> "lower-level" approach. Therefore I've written "oomatrix.py"
> (polymorphic matrices) which does this in a higher-level way for my own
> purposes (not available online yet...will have to see when I get time to
> polish it...):

I like to keep some one-liner functions around just to tell or remind
me how to do it.
Scipy.stats currently doesn't need it, (maybe when a multivariate
normal class is included, and I just saw that it might be possible to
improve stats.kde by storing the determinant instead of calculating it
repeatedly.)

In statsmodels, we store currently the cholesky of the pinv of the
covariance matrix, but statsmodels is still mostly least squares based
where we don't need the determinant, and we are still expanding MLE.

So, these linear algebra tricks will come in handy.

Josef

>
> A, B = ...numpy arrays...
> M = oomatrix.matrix(A) # Create immutable matrix object from array
> C M.solve_right(B, algorithm='cholesky') # does cholesky and caches it
> M.log_determinant() # from cached decomposition
> P, L = M.cholesky()
>
> Alternatively, constructing oomatrix.matrix(A, sparse=True) will
> transparently switch to using CHOLMOD as backend.
>
> Dag Sverre
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From jsseabold at gmail.com  Wed Feb 17 09:46:52 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 17 Feb 2010 09:46:52 -0500
Subject: [SciPy-dev] maxentropy docs / parent class methods
In-Reply-To: <dde7764a1002170316s760e2335kd402db5b5c82ec78@mail.gmail.com>
References: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com> 
	<1cd32cbb1002161341r412704c8y65a3eed7000fce8f@mail.gmail.com> 
	<c048da1c1002161405t397449a2vc16ea42e39c02193@mail.gmail.com> 
	<dde7764a1002170316s760e2335kd402db5b5c82ec78@mail.gmail.com>
Message-ID: <c048da1c1002170646x30473884r4d948d07e4bf8c1b@mail.gmail.com>

On Wed, Feb 17, 2010 at 6:16 AM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
>
>
> On Wed, Feb 17, 2010 at 6:05 AM, Skipper Seabold <jsseabold at gmail.com>
> wrote:
>>
>> On Tue, Feb 16, 2010 at 4:41 PM, ?<josef.pktd at gmail.com> wrote:
>> > On Tue, Feb 16, 2010 at 4:10 PM, Skipper Seabold <jsseabold at gmail.com>
>> > wrote:
>> >> The docs on the methods for the maxentropy model class that are
>> >> inherited from basemodel are not picked up by sphinx. ?It seems that
>> >> most of the .rst files explicitly list the methods, but since
>> >> basemodel is not intended to be public (and some subclasses overwrite
>> >> the parent class methods), is there a better way than to start listing
>> >> the basemodel methods?
>> >>
>> >> I started to make the changes, but I don't think this is the right way
>> >> forward. ?Other thoughts?
>> >>
>> >> http://docs.scipy.org/scipy/docs/scipy-docs/maxentropy.rst/
>> >
>> > Try to reference directly the basemodel.xxx method in the autoclass for
>> > model
>> >
>> > .. autoclass:: model
>> >
>> > .. autosummary::
>> > ? :toctree: generated/
>> >
>> > .....
>> > ? model.dual
>> > ? basemodel.fit
>> > ? model.grad
>> > ? model.log
>> >
>> > Otherwise, it's better to have too much than too little information in
>> > the docs
>> >
>>
>> That works, but it's still the basemodel namespace under model. ?I was
>> looking to statsmodels to see how we autogenerate the inherited
>> methods. ?I might play around with a local build so I don't clutter up
>> the logs on the wiki. ?It's changed so that they show up for now while
>> I clean up a little more but is probably not a final solution.
>>
> Those methods do not belong just to 'model' but also to 'bigmodel' and
> 'conditionalmodel'. So listing them under 'model' is a bit arbitrary. I
> think the correct way to do this is to list them under an .. autoclass::
> basemodel, with a note that this class contains shared functionality and
> should not be instantiated directly.
>

Agreed.  I will try this and make sure the note that is already in the
docstring shows up.

> Also, I do not see in the code that basemodel is not public. Why did you
> conclude this? If a user wants to implement a new model, shouldn't he
> inherit from basemodel? In that case it should be public. If that's not the
> case, an __all__ dict should be added to the module to indicate what is
> public and what is not.
>

I guess I am not using public/private right.  I was thinking private
== can't be instantiated, but what you say is correct (I will be using
it for generalized maxent, etc.), so I guess it's public.

Thanks,

Skipper


From josef.pktd at gmail.com  Wed Feb 17 10:10:05 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 17 Feb 2010 10:10:05 -0500
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <961fa2b41002170032u72d6a22ajafd95af99da6f57d@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
	<9457e7c81002162307q3dc59492y9905dcc525a49575@mail.gmail.com>
	<961fa2b41002170032u72d6a22ajafd95af99da6f57d@mail.gmail.com>
Message-ID: <1cd32cbb1002170710l8365a7bic20c33a0cc5f1379@mail.gmail.com>

On Wed, Feb 17, 2010 at 3:32 AM, Nathaniel Smith <njs at pobox.com> wrote:
> 2010/2/16 St?fan van der Walt <stefan at sun.ac.za>:
>> Forming the product A^T A is often a bad idea from a numerical
>> perspective.
>
> Right.
>
>> In Chapter 3 of Ake Bjork's "Numerical Methods for Least
>> Squares Problems", he talks about "update" problems. ?He mentions that
>> "...the solution should be accurate up to the limitation of data and
>> conditioning of the problem; i.e., a stable numerical method must be
>> used."
>>
>> He describes the Kalman-based update method (which bears some
>> resemblance to yours), but says that "the main disadvantage...is its
>> serious sensitivity to roundoff errors. ?The updating algorithms based
>> on orthogonal transformations developed in the following sections are
>> generally to be preferred." ?He then goes into more detail on updating
>> the QR and Gram-Schmidt decompositions.
>>
>> Not sure if that helps, but it may be worth reading that chapter.
>
> Definitely -- thanks for the reference! Looks like the library has a
> copy available, too...
>
> (Not that I'll necessarily bother for my own use, since like I said,
> my problems seem to be fairly well conditioned and the sparse X'X +
> Cholesky approach is very fast, and also generalizes easily to mixed
> effect models, which I'm currently working on implementing.)
>
>>> For the QR code I use a recurrence relation I stole from some slides
>>> by Simon Wood to compute R and Q'y incrementally; probably a real
>>> incremental QR (e.g., "AS274", which is what R's biglm package uses)
>>> would be better, but this one is easy to implement in terms of
>>> non-incremental QR.
>>
>> Incremental QR is something we should implement in scipy.linalg, for sure.
>
> +1

Just a follow up question for the experts.

In the description of some estimators and the optimization routines
used, I have read that the updating is done on the cholesky or QR
decomposition of the inverse matrix ( (x'x)^{-1} or inverse Hessian).
Except for simple least squares problem the inverse is often more
important than the the original X'X.

Are there algorithms available for this, and is there an advantage for
either way? From what I have seen in incremental_ls, the updating
works on QR of x'x and then the inverse of the QR decomposition is
taken. Assuming a well behaved non-singular X'X.

Nathaniel,
I will need more time to work through this but will come back to it soon.
from njs.util import block_diagonal_stack   seems to be the only
function that is missing

Thanks,

Josef




>
> -- Nathaniel
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From charlesr.harris at gmail.com  Wed Feb 17 11:03:23 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 17 Feb 2010 09:03:23 -0700
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <1cd32cbb1002170710l8365a7bic20c33a0cc5f1379@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
	<9457e7c81002162307q3dc59492y9905dcc525a49575@mail.gmail.com>
	<961fa2b41002170032u72d6a22ajafd95af99da6f57d@mail.gmail.com>
	<1cd32cbb1002170710l8365a7bic20c33a0cc5f1379@mail.gmail.com>
Message-ID: <e06186141002170803g737177a9s6e242343655f29bf@mail.gmail.com>

On Wed, Feb 17, 2010 at 8:10 AM, <josef.pktd at gmail.com> wrote:

> On Wed, Feb 17, 2010 at 3:32 AM, Nathaniel Smith <njs at pobox.com> wrote:
> > 2010/2/16 St?fan van der Walt <stefan at sun.ac.za>:
> >> Forming the product A^T A is often a bad idea from a numerical
> >> perspective.
> >
> > Right.
> >
> >> In Chapter 3 of Ake Bjork's "Numerical Methods for Least
> >> Squares Problems", he talks about "update" problems.  He mentions that
> >> "...the solution should be accurate up to the limitation of data and
> >> conditioning of the problem; i.e., a stable numerical method must be
> >> used."
> >>
> >> He describes the Kalman-based update method (which bears some
> >> resemblance to yours), but says that "the main disadvantage...is its
> >> serious sensitivity to roundoff errors.  The updating algorithms based
> >> on orthogonal transformations developed in the following sections are
> >> generally to be preferred."  He then goes into more detail on updating
> >> the QR and Gram-Schmidt decompositions.
> >>
> >> Not sure if that helps, but it may be worth reading that chapter.
> >
> > Definitely -- thanks for the reference! Looks like the library has a
> > copy available, too...
> >
> > (Not that I'll necessarily bother for my own use, since like I said,
> > my problems seem to be fairly well conditioned and the sparse X'X +
> > Cholesky approach is very fast, and also generalizes easily to mixed
> > effect models, which I'm currently working on implementing.)
> >
> >>> For the QR code I use a recurrence relation I stole from some slides
> >>> by Simon Wood to compute R and Q'y incrementally; probably a real
> >>> incremental QR (e.g., "AS274", which is what R's biglm package uses)
> >>> would be better, but this one is easy to implement in terms of
> >>> non-incremental QR.
> >>
> >> Incremental QR is something we should implement in scipy.linalg, for
> sure.
> >
> > +1
>
> Just a follow up question for the experts.
>
> In the description of some estimators and the optimization routines
> used, I have read that the updating is done on the cholesky or QR
> decomposition of the inverse matrix ( (x'x)^{-1} or inverse Hessian).
> Except for simple least squares problem the inverse is often more
> important than the the original X'X.
>
> Are there algorithms available for this, and is there an advantage for
> either way? From what I have seen in incremental_ls, the updating
> works on QR of x'x and then the inverse of the QR decomposition is
> taken. Assuming a well behaved non-singular X'X.
>
>
Both versions are available for Kalman filters. Using the covariance
(inverse) is the usual approach, but the other version is used and goes
under the name Kalman information filter. There are various ways to minimise
roundoff error depending on how the updates are done. For high precision
there are implementations that work with factorizations, either the Cholesky
factorization or what amounts to QR. There is a large literature on the
subject going back to the 60's.

Kalman filters are Bayesian and require a prior to get started, but such
isn't hard to come by, zero with a large covariance suffices. However, that
might make them less suitable for an application where all the data is
available and dynamic estimates of the parameters aren't needed. But the
numerical parts may offer some useful bits.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100217/3ea77dbe/attachment.html>

From robert.kern at gmail.com  Wed Feb 17 11:05:03 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Feb 2010 10:05:03 -0600
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com>
Message-ID: <3d375d731002170805h13928409lc95af1dcff0ca9c5@mail.gmail.com>

On Wed, Feb 17, 2010 at 04:22, Tom Grydeland <tom.grydeland at gmail.com> wrote:
> On Wed, Feb 10, 2010 at 3:12 AM, David Cournapeau <david at silveregg.co.jp> wrote:
>> Hi,
>>
>> I noticed that some of the docstrings I have written for DCT have been
>> changed to latex format. While I have no issue with having latex in the
>> documentation, I thought the consensus was to use them sparingly in
>> docstrings ?
>
> I am probably the one to "blame" here. ?I know I have edited the DCT docstrings.
>
> What is considered "sparingly" is obviously different from one person
> to the next, and I have taken it to mean roughly "where pure text is
> insufficient", and (along the discussion re numpy.fft and friends
> <http://docs.scipy.org/numpy/docs/numpy.fft/>) okay for module-level
> docstrings but avoid it for functions. ?Since I have written a lot of
> latex, I might have a lower threshold than others here.
>
>> For example, the dct I formula used to be (fixed width font assumed):
>>
>> for 0 <= k < N,
>>
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?N-1
>> y[k] = x[0] + (-1)**k x[N-1] + 2 * sum x[n]*cos(pi*k*n/(N-1))
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?n=0
>>
>> But now, it is:
>>
>> y_k = x_0 + (-1)^k x_{N-1} + 2\\sum_{n=1}^{N-2} x_n
>> ? ? ? ? \\cos\\left({\\pi nk\\over N-1}\\right),
>> ? ? ? ? \\qquad 0 \\le k < N.
>>
>> I much prefer the former (the latter is unreadable in a terminal IMO). I
>> have of course no issue in putting the latex formula in the scipy docs,
>
> Similarly, the former is unreadable or incorrect in the web interface
> where the latter is useful:
> http://docs.scipy.org/scipy/docs/scipy.fftpack.realtransforms.dct/

The former can be returned to readable in the web interface simply by
using the "::" literal marker at the end of the previous line. E.g.

"""
There are several definitions of the DCT-I; we use the following
(for ``norm=None``)::

                                     N-2
  y[k] = x[0] + (-1)**k x[N-1] + 2 * sum x[n]*cos(pi*k*n/(N-1))
                                     n=0
"""

Additionally, the font on the edit box should be monospaced to assist
editors in editing the docstrings to look good in plain text. I'll
look into changing that.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jdh2358 at gmail.com  Wed Feb 17 11:15:45 2010
From: jdh2358 at gmail.com (John Hunter)
Date: Wed, 17 Feb 2010 10:15:45 -0600
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <dde7764a1002170421y4da939aaxbc14caaff373b0d@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com>
	<dde7764a1002170421y4da939aaxbc14caaff373b0d@mail.gmail.com>
Message-ID: <88e473831002170815wb788b34qa5389d5af33355a3@mail.gmail.com>

On Wed, Feb 17, 2010 at 6:21 AM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:

> Fixing this docstring for both terminal and html/pdf does not seem possible,
> short of removing content or writing a new Sphinx plugin. For now I would be
> in favor of keeping the latex, because going to the html doc for terminal
> users is doable. Users of the html docs on the other hand are unlikely to
> all know their way around a terminal.

In mpl we have a related issue -- we render rest tables for the html
and pdf, but ascii tables for the interactive docs (eg "help" in
python)

We achieve this by sacrificing the readability of the source
docstrings, eg for Text

    def __init__(self,
                 ....
                 **kwargs
                 ):
        """
        Create a :class:`~matplotlib.text.Text` instance at *x*, *y*
        with string *text*.

        Valid kwargs are
        %(Text)s
        """

and have a module for inserting formated strings into the docstrings

   docstring.interpd.update(Text = artist.kwdoc(Text))

This module respects an rc setting which determines whether to render
in "hardcopy" mode or not.  numpy could adopt a similar system in
which equations are interpolated in as either ascii or latex depending
on a runtime setting.

As an example, compare the text rendered help for Text vs the html
rendered at http://matplotlib.sourceforge.net/api/artist_api.html#matplotlib.text.Text

Help on Text in module matplotlib.text object:

class Text(matplotlib.artist.Artist)
 |  Handle storing and drawing of text in window or data coordinates.
 |
 |  Method resolution order:
 |      Text
 |      matplotlib.artist.Artist
 |      __builtin__.object
 |
 |  Methods defined here:
 |
 |  __init__(self, x=0, y=0, text='', color=None,
verticalalignment='bottom', horizontalalignment='left',
multialignment=None, fontproperties=None, rotation=None,
linespacing=None, rotation_mode=None, path_effects=None,
**kwargs)text*.
 |
 |      Valid kwargs are
 |        agg_filter: unknown
 |        alpha: float (0.0 transparent through 1.0 opaque)
 |        animated: [True | False]
 |        axes: an :class:`~matplotlib.axes.Axes` instance
 |        backgroundcolor: any matplotlib color
 |        bbox: rectangle prop dict
 |        clip_box: a :class:`matplotlib.transforms.Bbox` instance
 |        clip_on: [True | False]


From charlesr.harris at gmail.com  Wed Feb 17 11:16:50 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 17 Feb 2010 09:16:50 -0700
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <dde7764a1002170421y4da939aaxbc14caaff373b0d@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com>
	<dde7764a1002170421y4da939aaxbc14caaff373b0d@mail.gmail.com>
Message-ID: <e06186141002170816w5408bb63p84cbd7e0ce9bc195@mail.gmail.com>

On Wed, Feb 17, 2010 at 5:21 AM, Ralf Gommers
<ralf.gommers at googlemail.com>wrote:

>
>
> On Wed, Feb 17, 2010 at 6:22 PM, Tom Grydeland <tom.grydeland at gmail.com>wrote:
>
>> On Wed, Feb 10, 2010 at 3:12 AM, David Cournapeau <david at silveregg.co.jp>
>> wrote:
>> > Hi,
>> >
>> > I noticed that some of the docstrings I have written for DCT have been
>> > changed to latex format. While I have no issue with having latex in the
>> > documentation, I thought the consensus was to use them sparingly in
>> > docstrings ?
>>
>> I am probably the one to "blame" here.  I know I have edited the DCT
>> docstrings.
>>
>> What is considered "sparingly" is obviously different from one person
>> to the next, and I have taken it to mean roughly "where pure text is
>> insufficient", and (along the discussion re numpy.fft and friends
>> <http://docs.scipy.org/numpy/docs/numpy.fft/>) okay for module-level
>> docstrings but avoid it for functions.
>>
>
> That's about right. In function docstrings latex should only sparingly be
> used in the Notes section (see
> http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines). This policy
> has worked pretty well so far, the problem for this particular docstring is
> that the Notes section is huge with lots of maths.
>
>>
>> > For example, the dct I formula used to be (fixed width font assumed):
>> >
>> > for 0 <= k < N,
>> >
>> >                                    N-1
>> > y[k] = x[0] + (-1)**k x[N-1] + 2 * sum x[n]*cos(pi*k*n/(N-1))
>> >                                    n=0
>> >
>> > But now, it is:
>> >
>> > y_k = x_0 + (-1)^k x_{N-1} + 2\\sum_{n=1}^{N-2} x_n
>> >         \\cos\\left({\\pi nk\\over N-1}\\right),
>> >         \\qquad 0 \\le k < N.
>> >
>> > I much prefer the former (the latter is unreadable in a terminal IMO). I
>> > have of course no issue in putting the latex formula in the scipy docs,
>>
>> Similarly, the former is unreadable or incorrect in the web interface
>> where the latter is useful:
>> http://docs.scipy.org/scipy/docs/scipy.fftpack.realtransforms.dct/
>>
>> If you change it back, please observe that the limits on the summation
>> are different for the two versions.
>
>
> Fixing this docstring for both terminal and html/pdf does not seem
> possible, short of removing content or writing a new Sphinx plugin. For now
> I would be in favor of keeping the latex, because going to the html doc for
> terminal users is doable. Users of the html docs on the other hand are
> unlikely to all know their way around a terminal.
>
>
I don't think the docstrings should be tutorials, rather, they should state
what the function does in simple terms. A place for more extended
explanation might be desirable at some point for teaching purposes but I
don't think it belongs in the docstrings.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100217/57d54687/attachment.html>

From njs at pobox.com  Wed Feb 17 12:35:48 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 17 Feb 2010 09:35:48 -0800
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <1cd32cbb1002170710l8365a7bic20c33a0cc5f1379@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
	<9457e7c81002162307q3dc59492y9905dcc525a49575@mail.gmail.com>
	<961fa2b41002170032u72d6a22ajafd95af99da6f57d@mail.gmail.com>
	<1cd32cbb1002170710l8365a7bic20c33a0cc5f1379@mail.gmail.com>
Message-ID: <961fa2b41002170935i57937a7coe4c00b5b69b6ce4c@mail.gmail.com>

On Wed, Feb 17, 2010 at 7:10 AM,  <josef.pktd at gmail.com> wrote:
> In the description of some estimators and the optimization routines
> used, I have read that the updating is done on the cholesky or QR
> decomposition of the inverse matrix ( (x'x)^{-1} or inverse Hessian).
> Except for simple least squares problem the inverse is often more
> important than the the original X'X.
>
> Are there algorithms available for this, and is there an advantage for
> either way? From what I have seen in incremental_ls, the updating
> works on QR of x'x and then the inverse of the QR decomposition is
> taken. Assuming a well behaved non-singular X'X.

I'm not sure what you mean here -- when using QR for OLS, you take the
QR decomposition of X, not X'X (this is what gives the improved
numerical stability -- if X is ill-conditioned, then X'X is
ill-conditioned squared, so you want to avoid it). See:
  http://en.wikipedia.org/wiki/Linear_least_squares#Orthogonal_decomposition_methods
You do need to explicitly calculate inv(X'X) at the end in order to
get standard error estimates, and it's nice if you can re-use any
decompositions you have lying around to make this faster (for the QR
case, inv(X'X) = inv(R'Q'QR) = inv(R'R), which can take advantage of
R's triangular structure).

But I'm not sure what you're referring to when you say "some
estimators", or why those routines want to incrementally compute the
inverse Hessian, so I can't help you there.

-- Nathaniel


From gokhansever at gmail.com  Wed Feb 17 13:44:23 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Wed, 17 Feb 2010 12:44:23 -0600
Subject: [SciPy-dev] Two points
Message-ID: <49d6b3501002171044g4c4f455n79b7c570a4b39c0a@mail.gmail.com>

Hi,

Currently http://svn.scipy.org/ pointing to code.enthought.com. Wouldn't be
nice it to show a page like svn.python.org does and show NumPy and SciPy dev
instructions + browser and the actual svn base?

http://conference.scipy.org/SciPy2009/slides link down. Could someone make
it up again?

Thanks.

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100217/a97ffaa6/attachment.html>

From josef.pktd at gmail.com  Wed Feb 17 14:01:37 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 17 Feb 2010 14:01:37 -0500
Subject: [SciPy-dev] code for incremental least squares
In-Reply-To: <961fa2b41002170935i57937a7coe4c00b5b69b6ce4c@mail.gmail.com>
References: <961fa2b41002161218q1a2c5242o6ae7f2f39d05a179@mail.gmail.com>
	<9457e7c81002161224r383a5e33sd04f8d1e195cc7bd@mail.gmail.com>
	<1cd32cbb1002161237q13a3d27ct1cccc1f2927bc25e@mail.gmail.com>
	<961fa2b41002161356vd281f75i46092d4a6e79c559@mail.gmail.com>
	<9457e7c81002162307q3dc59492y9905dcc525a49575@mail.gmail.com>
	<961fa2b41002170032u72d6a22ajafd95af99da6f57d@mail.gmail.com>
	<1cd32cbb1002170710l8365a7bic20c33a0cc5f1379@mail.gmail.com>
	<961fa2b41002170935i57937a7coe4c00b5b69b6ce4c@mail.gmail.com>
Message-ID: <1cd32cbb1002171101v3313d740yabbba6c0eddb7add@mail.gmail.com>

On Wed, Feb 17, 2010 at 12:35 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Wed, Feb 17, 2010 at 7:10 AM, ?<josef.pktd at gmail.com> wrote:
>> In the description of some estimators and the optimization routines
>> used, I have read that the updating is done on the cholesky or QR
>> decomposition of the inverse matrix ( (x'x)^{-1} or inverse Hessian).
>> Except for simple least squares problem the inverse is often more
>> important than the the original X'X.
>>
>> Are there algorithms available for this, and is there an advantage for
>> either way? From what I have seen in incremental_ls, the updating
>> works on QR of x'x and then the inverse of the QR decomposition is
>> taken. Assuming a well behaved non-singular X'X.

Maybe, I'm getting ahead of myself and mixing oranges and apples.

One of the estimators that I have been looking at recently is a F-GLS
with a general covariance matrix
http://en.wikipedia.org/wiki/Generalized_least_squares
as in mixed/random effects models or panel data. I was trying to
figure out how to work more efficiently with the matrix decompositions
to calculate the covariance matrix of the parameter estimates.

>
> I'm not sure what you mean here -- when using QR for OLS, you take the
> QR decomposition of X, not X'X (this is what gives the improved
> numerical stability -- if X is ill-conditioned, then X'X is
> ill-conditioned squared, so you want to avoid it).

I didn't realize QR is on X and not on X'X. I think, I was looking
more at XtXIncrementalLS than at QRIncrementalLS. Because of the block
structure of the heteroscedasticity, you update separately by groups,
if my reading is correct.
I assume now XtXIncrementalLS is your second implementation that
doesn't use QR. I think, I understand now the basic structure of the
QR method for OLS.

See:
> ?http://en.wikipedia.org/wiki/Linear_least_squares#Orthogonal_decomposition_methods
> You do need to explicitly calculate inv(X'X) at the end in order to
> get standard error estimates, and it's nice if you can re-use any
> decompositions you have lying around to make this faster (for the QR
> case, inv(X'X) = inv(R'Q'QR) = inv(R'R), which can take advantage of
> R's triangular structure).

I saw this, and it reminded me of the possibility to update the
decomposition of the inverse directly. This part, I really have to
look at later when I work on GLS again.

>
> But I'm not sure what you're referring to when you say "some
> estimators", or why those routines want to incrementally compute the
> inverse Hessian, so I can't help you there.

most likely apples and oranges again. I was also working on time
series models where the optimization problem is non-linear, but the
underlying structure is linear, eg. arma with maximum likelihood
estimation.

Since I have only a vague idea what the low level linear algebra does,
I better stop until I find time to figure this out. I just learn it as
the need or the opportunity arises. (Similar to my fight with fft a
while ago.)

Thanks,

Josef

>
> -- Nathaniel
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From jh at physics.ucf.edu  Wed Feb 17 14:22:14 2010
From: jh at physics.ucf.edu (Joe Harrington)
Date: Wed, 17 Feb 2010 14:22:14 -0500
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <mailman.5296.1266428157.2454.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.5296.1266428157.2454.scipy-dev@scipy.org>
Message-ID: <wl8sk8zoeft.fsf@glup.physics.ucf.edu>

On Wed, 17 Feb 2010 09:16:50 -0700, Charles R Harris wrote:
> I don't think the docstrings should be tutorials, rather, they should state
> what the function does in simple terms. A place for more extended
> explanation might be desirable at some point for teaching purposes but I
> don't think it belongs in the docstrings.

This is rather irrelevant to the discussion at hand.  ASCII or LaTeX
formulae belong in any docstring implementing something best described
by one, particularly if there are differences in how the concept is
implemented in different numerical packages.  For example, in an FFT,
is the normalization part of the forward transform, the inverse, or
both?  A formula defines exactly what is implemented in the code,
and is usually the tersest way of presenting that (which I think is
what you're after).

If you have concerns about the general content of the docstrings, it
would be helpful if you could post them on another thread and give
some specific examples of what you are concerned about.

--jh--


From dwf at cs.toronto.edu  Wed Feb 17 15:03:53 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 17 Feb 2010 15:03:53 -0500
Subject: [SciPy-dev] cephes on Mac OS X ppc64?
Message-ID: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu>

I noticed that for some reason, cephes gets built as a universal  
binary with 3 architectures: i386, ppc, and x86_64. Is there any  
reason why it's not building ppc64? (I ask because I'm trying to use  
NumPy/SciPy/etc. in 64-bit mode on a G5).

David


From robert.kern at gmail.com  Wed Feb 17 15:10:03 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Feb 2010 14:10:03 -0600
Subject: [SciPy-dev] cephes on Mac OS X ppc64?
In-Reply-To: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu>
References: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu>
Message-ID: <3d375d731002171210o6154bddetbb125eb9bdde389@mail.gmail.com>

On Wed, Feb 17, 2010 at 14:03, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> I noticed that for some reason, cephes gets built as a universal
> binary with 3 architectures: i386, ppc, and x86_64. Is there any
> reason why it's not building ppc64? (I ask because I'm trying to use
> NumPy/SciPy/etc. in 64-bit mode on a G5).

I think we test each architecture with the compiler(s) before adding
them to the list. It is possible that either your gcc or your gfortran
does not support ppc64 or we are failing to recognize that it does. Or
we might've dropped support for it. I think David Cournapeau was the
last person to touch the architecture selection code in
numpy.distutils.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From mohitiiit at gmail.com  Wed Feb 17 16:14:15 2010
From: mohitiiit at gmail.com (Mohit Goyal)
Date: Thu, 18 Feb 2010 02:44:15 +0530
Subject: [SciPy-dev] Reg: gsoc 2010
Message-ID: <9e78df6d1002171314r3e0a1f35s96d4b51c3e9d23dc@mail.gmail.com>

I am a masters student doing research in data engineering. I know python and
have done projects on data mining and web mining.
I want to contribute to scipy via gsoc 2010. I am  interested in working on
data related projects in scipy like improving  data sources and integrate it
into all the numpy/scipy , adding data sets and examples to scipy, data
analysis etc.  Please guide me.

Thank You.





-- 
The Game is Not Over Because i haven't Won Yet.
-Mohit Goyal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100218/82cf72f7/attachment.html>

From gael.varoquaux at normalesup.org  Wed Feb 17 16:21:04 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 17 Feb 2010 22:21:04 +0100
Subject: [SciPy-dev] Reg: gsoc 2010
In-Reply-To: <9e78df6d1002171314r3e0a1f35s96d4b51c3e9d23dc@mail.gmail.com>
References: <9e78df6d1002171314r3e0a1f35s96d4b51c3e9d23dc@mail.gmail.com>
Message-ID: <20100217212104.GA20266@phare.normalesup.org>

On Thu, Feb 18, 2010 at 02:44:15AM +0530, Mohit Goyal wrote:
>    I am a masters student doing research in data engineering. I know python
>    and have done projects on data mining and web mining.
>    I want to contribute to scipy via gsoc 2010. I am? interested in working
>    on data related projects in scipy like improving? data sources and
>    integrate it into all the numpy/scipy , adding data sets and examples to
>    scipy, data analysis etc.? Please guide me.

Hey,

Would you be interested in contributing to the scikit learn, which aims
to have reference machine learning and data mining implemented in Python
with high quality implementation, documentation and example?

The project has been asleep a bit, but is now picking up speed, as my
research group has been able to hire a very talented engineer
to contribute full-time. We are right now starting to get the ball
rolling :

    http://scikit-learn.sourceforge.net/
    http://fseoane.net/blog/2010/scikit-learn-01/

We should be able to define goals according to your interests, and
provide tutoring both on the scientific aspects (which algorithms) and on
the practical aspects to end up with successful code integration.

Cheers,

Ga?l


From dwf at cs.toronto.edu  Wed Feb 17 16:26:05 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 17 Feb 2010 16:26:05 -0500
Subject: [SciPy-dev] Reg: gsoc 2010
In-Reply-To: <9e78df6d1002171314r3e0a1f35s96d4b51c3e9d23dc@mail.gmail.com>
References: <9e78df6d1002171314r3e0a1f35s96d4b51c3e9d23dc@mail.gmail.com>
Message-ID: <4015AEDE-C7D0-49F0-B557-FA18331C878C@cs.toronto.edu>

On 17-Feb-10, at 4:14 PM, Mohit Goyal wrote:

> I want to contribute to scipy via gsoc 2010. I am  interested in  
> working on
> data related projects in scipy like improving  data sources

Can you explain what you mean? Everything in SciPy is 'data-related'  
in one way or another. Do you mean I/O code?

> and integrate it into all the numpy/scipy , adding data sets and  
> examples to scipy,

More examples and documentation are always a welcome addition, but I  
don't know about 'datasets'.

> data analysis etc.

You'll have to be more specific than 'data analysis'. Again, that  
encompasses nearly everything one could conceivably do with SciPy.

David


From dwf at cs.toronto.edu  Wed Feb 17 16:48:02 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 17 Feb 2010 16:48:02 -0500
Subject: [SciPy-dev] cephes on Mac OS X ppc64?
In-Reply-To: <3d375d731002171210o6154bddetbb125eb9bdde389@mail.gmail.com>
References: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu>
	<3d375d731002171210o6154bddetbb125eb9bdde389@mail.gmail.com>
Message-ID: <87BA7EE3-14EC-43C3-8BB4-A14E146AB49D@cs.toronto.edu>


On 17-Feb-10, at 3:10 PM, Robert Kern wrote:

> I think we test each architecture with the compiler(s) before adding
> them to the list. It is possible that either your gcc or your gfortran
> does not support ppc64 or we are failing to recognize that it does. Or
> we might've dropped support for it. I think David Cournapeau was the
> last person to touch the architecture selection code in
> numpy.distutils.

Hmm. Interesting. Doing some digging in the build log I encountered  
stuff like this:

/usr/local/bin/gfortran -Wall -arch ppc -arch i686 -arch x86_64 -Wall - 
undefined dynamic_lookup -bundle build/temp.macosx-10.5-universal-2.6/ 
build/src.macosx-10.5-universal-2.6/specfunmodule.o build/ 
temp.macosx-10.5-universal-2.6/build/src.macosx-10.5-universal-2.6/ 
fortranobject.o -Lbuild/temp.macosx-10.5-universal-2.6 -lsc_specfun - 
lgfortran -o build/lib.macosx-10.5-universal-2.6/special/specfun.so

Looking at the .o's and .so's generated, gcc is definitely generating  
ppc64-capable binaries. gfortran (it's the AT&T one to which you  
usually point people) *seems* perfectly capable of generating ppc64  
binaries (it doesn't complain when run with -arch ppc64).

I've upgraded to the latest gfortran from the AT&T R Tools page and  
tried mucking with FFLAGS, F90FLAGS, F77FLAGS, etc. No dice.

Anyone have ideas for anything else I should try?

David


From pgmdevlist at gmail.com  Wed Feb 17 16:51:19 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 17 Feb 2010 16:51:19 -0500
Subject: [SciPy-dev] Reg: gsoc 2010
In-Reply-To: <9e78df6d1002171314r3e0a1f35s96d4b51c3e9d23dc@mail.gmail.com>
References: <9e78df6d1002171314r3e0a1f35s96d4b51c3e9d23dc@mail.gmail.com>
Message-ID: <10ED3C13-C633-4B64-B2FC-0AD62EDFD447@gmail.com>

On Feb 17, 2010, at 4:14 PM, Mohit Goyal wrote:
> I am a masters student doing research in data engineering. I know python and have done projects on data mining and web mining. 
> I want to contribute to scipy via gsoc 2010. I am  interested in working on data related projects in scipy like improving  data sources and integrate it into all the numpy/scipy , adding data sets and examples to scipy, data analysis etc.  Please guide me.
> 
> Thank You.


In addition to the previous answers, I would strongly suggest you to lurk around the mailing list for a while, in order to get familiarized with the hot topics (stuff we need to get done but didn't have time to address yet). That way,  you can refine your proposal and not make it sound too generic (spam-like).

From robert.kern at gmail.com  Wed Feb 17 17:12:09 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Feb 2010 16:12:09 -0600
Subject: [SciPy-dev] cephes on Mac OS X ppc64?
In-Reply-To: <87BA7EE3-14EC-43C3-8BB4-A14E146AB49D@cs.toronto.edu>
References: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu> 
	<3d375d731002171210o6154bddetbb125eb9bdde389@mail.gmail.com> 
	<87BA7EE3-14EC-43C3-8BB4-A14E146AB49D@cs.toronto.edu>
Message-ID: <3d375d731002171412xdf8e9d3ieb810de5bd1e602d@mail.gmail.com>

On Wed, Feb 17, 2010 at 15:48, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>
> On 17-Feb-10, at 3:10 PM, Robert Kern wrote:
>
>> I think we test each architecture with the compiler(s) before adding
>> them to the list. It is possible that either your gcc or your gfortran
>> does not support ppc64 or we are failing to recognize that it does. Or
>> we might've dropped support for it. I think David Cournapeau was the
>> last person to touch the architecture selection code in
>> numpy.distutils.
>
> Hmm. Interesting. Doing some digging in the build log I encountered
> stuff like this:
>
> /usr/local/bin/gfortran -Wall -arch ppc -arch i686 -arch x86_64 -Wall -
> undefined dynamic_lookup -bundle build/temp.macosx-10.5-universal-2.6/
> build/src.macosx-10.5-universal-2.6/specfunmodule.o build/
> temp.macosx-10.5-universal-2.6/build/src.macosx-10.5-universal-2.6/
> fortranobject.o -Lbuild/temp.macosx-10.5-universal-2.6 -lsc_specfun -
> lgfortran -o build/lib.macosx-10.5-universal-2.6/special/specfun.so
>
> Looking at the .o's and .so's generated, gcc is definitely generating
> ppc64-capable binaries. gfortran (it's the AT&T one to which you
> usually point people) *seems* perfectly capable of generating ppc64
> binaries (it doesn't complain when run with -arch ppc64).
>
> I've upgraded to the latest gfortran from the AT&T R Tools page and
> tried mucking with FFLAGS, F90FLAGS, F77FLAGS, etc. No dice.
>
> Anyone have ideas for anything else I should try?

Check earlier in the log for where numpy.distutils is checking the
Fortran configuration. It would also be worth locating the code in
numpy/distutils/fcompiler/gnu.py that picks the architecture flags and
putting in some print statements to see what it's doing. Using the
environment variable DISTUTILS_DEBUG=1 may help.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Wed Feb 17 17:27:35 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Feb 2010 16:27:35 -0600
Subject: [SciPy-dev] Two points
In-Reply-To: <49d6b3501002171044g4c4f455n79b7c570a4b39c0a@mail.gmail.com>
References: <49d6b3501002171044g4c4f455n79b7c570a4b39c0a@mail.gmail.com>
Message-ID: <3d375d731002171427x332a1b01h98cde15abd3ae83e@mail.gmail.com>

On Wed, Feb 17, 2010 at 12:44, G?khan Sever <gokhansever at gmail.com> wrote:
> Hi,
>
> Currently http://svn.scipy.org/ pointing to code.enthought.com. Wouldn't be
> nice it to show a page like svn.python.org does and show NumPy and SciPy dev
> instructions + browser and the actual svn base?

It now points to http://www.scipy.org/Developer_Zone/. Thanks for
reporting this.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From millman at berkeley.edu  Wed Feb 17 17:44:53 2010
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 17 Feb 2010 14:44:53 -0800
Subject: [SciPy-dev] Reg: gsoc 2010
In-Reply-To: <4015AEDE-C7D0-49F0-B557-FA18331C878C@cs.toronto.edu>
References: <9e78df6d1002171314r3e0a1f35s96d4b51c3e9d23dc@mail.gmail.com>
	<4015AEDE-C7D0-49F0-B557-FA18331C878C@cs.toronto.edu>
Message-ID: <c7009a551002171444l420bc655p5bce33e71951306a@mail.gmail.com>

On Wed, Feb 17, 2010 at 1:26 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> On 17-Feb-10, at 4:14 PM, Mohit Goyal wrote:
>> I want to contribute to scipy via gsoc 2010. I am ?interested in
>> working on
>> data related projects in scipy like improving ?data sources
>
> Can you explain what you mean? Everything in SciPy is 'data-related'
> in one way or another. Do you mean I/O code?

I think he was thinking about improving and integrating this:
http://projects.scipy.org/numpy/browser/trunk/numpy/lib/_datasource.py

Of course, he would need to flesh out the project a bit.

Jarrod


From david at silveregg.co.jp  Wed Feb 17 19:34:32 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Thu, 18 Feb 2010 09:34:32 +0900
Subject: [SciPy-dev] cephes on Mac OS X ppc64?
In-Reply-To: <87BA7EE3-14EC-43C3-8BB4-A14E146AB49D@cs.toronto.edu>
References: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu>
	<3d375d731002171210o6154bddetbb125eb9bdde389@mail.gmail.com>
	<87BA7EE3-14EC-43C3-8BB4-A14E146AB49D@cs.toronto.edu>
Message-ID: <4B7C8B18.1070408@silveregg.co.jp>

David Warde-Farley wrote:
> On 17-Feb-10, at 3:10 PM, Robert Kern wrote:
> 
>> I think we test each architecture with the compiler(s) before adding
>> them to the list. It is possible that either your gcc or your gfortran
>> does not support ppc64 or we are failing to recognize that it does. Or
>> we might've dropped support for it. I think David Cournapeau was the
>> last person to touch the architecture selection code in
>> numpy.distutils.
> 
> Hmm. Interesting. Doing some digging in the build log I encountered  
> stuff like this:
> 
> /usr/local/bin/gfortran -Wall -arch ppc -arch i686 -arch x86_64 -Wall - 
> undefined dynamic_lookup -bundle build/temp.macosx-10.5-universal-2.6/ 
> build/src.macosx-10.5-universal-2.6/specfunmodule.o build/ 
> temp.macosx-10.5-universal-2.6/build/src.macosx-10.5-universal-2.6/ 
> fortranobject.o -Lbuild/temp.macosx-10.5-universal-2.6 -lsc_specfun - 
> lgfortran -o build/lib.macosx-10.5-universal-2.6/special/specfun.so

If you look at numpy/distutils/fcompiler/gnu.py, the list of tested arch 
does not contain ppc64. I don't know why it is missing here; can you try 
rebuilding scipy with ppc64 added as a target in the function 
_universal_flags ?

David


From ralf.gommers at googlemail.com  Wed Feb 17 20:01:15 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Thu, 18 Feb 2010 09:01:15 +0800
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <3d375d731002170805h13928409lc95af1dcff0ca9c5@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com>
	<3d375d731002170805h13928409lc95af1dcff0ca9c5@mail.gmail.com>
Message-ID: <dde7764a1002171701sf8eaee5g7886816a5a800eaa@mail.gmail.com>

On Thu, Feb 18, 2010 at 12:05 AM, Robert Kern <robert.kern at gmail.com> wrote:

> > Similarly, the former is unreadable or incorrect in the web interface
> > where the latter is useful:
> > http://docs.scipy.org/scipy/docs/scipy.fftpack.realtransforms.dct/
>
> The former can be returned to readable in the web interface simply by
> using the "::" literal marker at the end of the previous line.
>
> Missed that one. Straightforward and looks perfectly fine in html. Thanks
Robert.

I changed the docstring back to ascii.


> Additionally, the font on the edit box should be monospaced to assist
> editors in editing the docstrings to look good in plain text. I'll
> look into changing that.
>
> The font is monospaced already.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100218/14cc983f/attachment.html>

From robert.kern at gmail.com  Wed Feb 17 20:08:25 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Feb 2010 19:08:25 -0600
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <dde7764a1002171701sf8eaee5g7886816a5a800eaa@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com> 
	<3d375d731002170805h13928409lc95af1dcff0ca9c5@mail.gmail.com> 
	<dde7764a1002171701sf8eaee5g7886816a5a800eaa@mail.gmail.com>
Message-ID: <3d375d731002171708s2e545ac5g40194829a443c262@mail.gmail.com>

On Wed, Feb 17, 2010 at 19:01, Ralf Gommers <ralf.gommers at googlemail.com> wrote:
>
>
> On Thu, Feb 18, 2010 at 12:05 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> > Similarly, the former is unreadable or incorrect in the web interface
>> > where the latter is useful:
>> > http://docs.scipy.org/scipy/docs/scipy.fftpack.realtransforms.dct/
>>
>> The former can be returned to readable in the web interface simply by
>> using the "::" literal marker at the end of the previous line.
>>
> Missed that one. Straightforward and looks perfectly fine in html. Thanks
> Robert.
>
> I changed the docstring back to ascii.

Thanks!

>> Additionally, the font on the edit box should be monospaced to assist
>> editors in editing the docstrings to look good in plain text. I'll
>> look into changing that.
>>
> The font is monospaced already.

Hmm, not to me. Might be my browser.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ralf.gommers at googlemail.com  Wed Feb 17 20:18:05 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Thu, 18 Feb 2010 09:18:05 +0800
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <88e473831002170815wb788b34qa5389d5af33355a3@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com>
	<dde7764a1002170421y4da939aaxbc14caaff373b0d@mail.gmail.com>
	<88e473831002170815wb788b34qa5389d5af33355a3@mail.gmail.com>
Message-ID: <dde7764a1002171718m2547a30ayd0ce7ae7b9221a6b@mail.gmail.com>

On Thu, Feb 18, 2010 at 12:15 AM, John Hunter <jdh2358 at gmail.com> wrote:

> On Wed, Feb 17, 2010 at 6:21 AM, Ralf Gommers
> <ralf.gommers at googlemail.com> wrote:
>
> > Fixing this docstring for both terminal and html/pdf does not seem
> possible,
> > short of removing content or writing a new Sphinx plugin. For now I would
> be
> > in favor of keeping the latex, because going to the html doc for terminal
> > users is doable. Users of the html docs on the other hand are unlikely to
> > all know their way around a terminal.
>
> In mpl we have a related issue -- we render rest tables for the html
> and pdf, but ascii tables for the interactive docs (eg "help" in
> python)
>
> We achieve this by sacrificing the readability of the source
> docstrings, eg for Text
>
>    def __init__(self,
>                 ....
>                 **kwargs
>                 ):
>        """
>        Create a :class:`~matplotlib.text.Text` instance at *x*, *y*
>        with string *text*.
>
>        Valid kwargs are
>        %(Text)s
>        """
>
> and have a module for inserting formated strings into the docstrings
>
>   docstring.interpd.update(Text = artist.kwdoc(Text))
>
> This module respects an rc setting which determines whether to render
> in "hardcopy" mode or not.  numpy could adopt a similar system in
> which equations are interpolated in as either ascii or latex depending
> on a runtime setting.
>
> String substitutions are handy when you need to share content between
several objects, but imho is overkill for each equation you want to typeset.
Something like AsciiMathML would still be the ideal solution here.

It also won't work with the doc editor unless you make it aware of how the
substitution system works. Which is not trivial.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100218/c7b0ccf8/attachment.html>

From dwf at cs.toronto.edu  Wed Feb 17 20:19:57 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 17 Feb 2010 20:19:57 -0500
Subject: [SciPy-dev] cephes on Mac OS X ppc64?
In-Reply-To: <4B7C8B18.1070408@silveregg.co.jp>
References: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu>
	<3d375d731002171210o6154bddetbb125eb9bdde389@mail.gmail.com>
	<87BA7EE3-14EC-43C3-8BB4-A14E146AB49D@cs.toronto.edu>
	<4B7C8B18.1070408@silveregg.co.jp>
Message-ID: <79F24F8A-279B-400C-B7B0-65FF881E9E73@cs.toronto.edu>


On 17-Feb-10, at 7:34 PM, David Cournapeau wrote:

> If you look at numpy/distutils/fcompiler/gnu.py, the list of tested  
> arch
> does not contain ppc64. I don't know why it is missing here; can you  
> try
> rebuilding scipy with ppc64 added as a target in the function
> _universal_flags ?

Hey David,

I just saw that, and am attempting a build with ppc64 added.

I have this command failing now, claiming undefined symbols for a  
whole bunch of things that look like Python API symbols:

/usr/local/bin/gfortran -Wall -arch i386 -arch ppc -arch x86_64 -arch  
ppc64 build/temp.macosx-10.5-universal-2.6/build/src.macosx-10.5- 
universal-2.6/scipy/fftpack/_fftpackmodule.o build/temp.macosx-10.5- 
universal-2.6/scipy/fftpack/src/zfft.o build/temp.macosx-10.5- 
universal-2.6/scipy/fftpack/src/drfft.o build/temp.macosx-10.5- 
universal-2.6/scipy/fftpack/src/zrfft.o build/temp.macosx-10.5- 
universal-2.6/scipy/fftpack/src/zfftnd.o build/temp.macosx-10.5- 
universal-2.6/build/src.macosx-10.5-universal-2.6/scipy/fftpack/src/ 
dct.o build/temp.macosx-10.5-universal-2.6/build/src.macosx-10.5- 
universal-2.6/fortranobject.o -Lbuild/temp.macosx-10.5-universal-2.6 - 
ldfftpack -lfftpack -lgfortran -o build/lib.macosx-10.5-universal-2.6/ 
scipy/fftpack/_fftpack.so

Partial error log (these and others are repeated for each architecture  
it seems):

Undefined symbols for architecture i386:
   "_PyModule_GetDict", referenced from:
       _init_fftpack in _fftpackmodule.o
   "_Py_FindMethod", referenced from:
       _fortran_getattr in fortranobject.o
   "_PyExc_AttributeError", referenced from:
       _PyExc_AttributeError$non_lazy_ptr in fortranobject.o
   "_PyString_FromString", referenced from:
       _init_fftpack in _fftpackmodule.o
       _init_fftpack in _fftpackmodule.o
       _fortran_getattr in fortranobject.o
       _fortran_getattr in fortranobject.o
   "_PyCObject_FromVoidPtr", referenced from:
       _fortran_getattr in fortranobject.o
   "_PyMem_Free", referenced from:
       _fortran_dealloc in fortranobject.o
       _fortran_dealloc in fortranobject.o
   "_PyObject_Type", referenced from:
       _array_from_pyobj in fortranobject.o
   "_PyType_IsSubtype", referenced from:
       _int_from_pyobj in _fftpackmodule.o
       _array_from_pyobj in fortranobject.o
   "_PyCObject_Type", referenced from:
       _PyCObject_Type$non_lazy_ptr in _fftpackmodule.o
   "_PyDict_DelItemString", referenced from:
       _fortran_setattr in fortranobject.o
   "_PyType_Type", referenced from:
       _PyType_Type$non_lazy_ptr in _fftpackmodule.o
   "_PyComplex_Type", referenced from:
       _PyComplex_Type$non_lazy_ptr in _fftpackmodule.o
   "_PyCObject_AsVoidPtr", referenced from:
       _init_fftpack in _fftpackmodule.o
....

Incidentally, if I check with "file" on the mentioned .o's and .so's,  
they are all quad-architecture as desired.

It seems that this is not a problem with the extra architecutre but  
somehow some Python dependencies are not being met (for ANY  
architecture). Note that my Python was built 4-way universal using the  
distributed build script:

dwf at morrislab:~/src/scipy-svn$ file `which python`
/Library/Frameworks/Python.framework/Versions/2.6/bin/python: Mach-O  
universal binary with 4 architectures
/Library/Frameworks/Python.framework/Versions/2.6/bin/python (for  
architecture i386):	Mach-O executable i386
/Library/Frameworks/Python.framework/Versions/2.6/bin/python (for  
architecture ppc7400):	Mach-O executable ppc
/Library/Frameworks/Python.framework/Versions/2.6/bin/python (for  
architecture ppc64):	Mach-O 64-bit executable ppc64
/Library/Frameworks/Python.framework/Versions/2.6/bin/python (for  
architecture x86_64):	Mach-O 64-bit executable x86_64

Any idea why this is failing?

Thanks,

David




From david at silveregg.co.jp  Wed Feb 17 20:38:30 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Thu, 18 Feb 2010 10:38:30 +0900
Subject: [SciPy-dev] cephes on Mac OS X ppc64?
In-Reply-To: <79F24F8A-279B-400C-B7B0-65FF881E9E73@cs.toronto.edu>
References: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu>
	<3d375d731002171210o6154bddetbb125eb9bdde389@mail.gmail.com>
	<87BA7EE3-14EC-43C3-8BB4-A14E146AB49D@cs.toronto.edu>
	<4B7C8B18.1070408@silveregg.co.jp>
	<79F24F8A-279B-400C-B7B0-65FF881E9E73@cs.toronto.edu>
Message-ID: <4B7C9A16.5040403@silveregg.co.jp>

David Warde-Farley wrote:
> On 17-Feb-10, at 7:34 PM, David Cournapeau wrote:
> 
>> If you look at numpy/distutils/fcompiler/gnu.py, the list of tested  
>> arch
>> does not contain ppc64. I don't know why it is missing here; can you  
>> try
>> rebuilding scipy with ppc64 added as a target in the function
>> _universal_flags ?
> 
> Hey David,
> 
> I just saw that, and am attempting a build with ppc64 added.
> 
> I have this command failing now, claiming undefined symbols for a  
> whole bunch of things that look like Python API symbols:

Unset any flags you set before (CFLAGS, LDFLAGS, etc...). The error you 
see is caused by -shared missing from LDFLAGS, and gfortran trying 
building an executable instead of a library,

cheers,

David


From dwf at cs.toronto.edu  Wed Feb 17 22:11:17 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 17 Feb 2010 22:11:17 -0500
Subject: [SciPy-dev] cephes on Mac OS X ppc64?
In-Reply-To: <4B7C9A16.5040403@silveregg.co.jp>
References: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu>
	<3d375d731002171210o6154bddetbb125eb9bdde389@mail.gmail.com>
	<87BA7EE3-14EC-43C3-8BB4-A14E146AB49D@cs.toronto.edu>
	<4B7C8B18.1070408@silveregg.co.jp>
	<79F24F8A-279B-400C-B7B0-65FF881E9E73@cs.toronto.edu>
	<4B7C9A16.5040403@silveregg.co.jp>
Message-ID: <6D6E5840-6E37-45B8-9D64-B014586324DE@cs.toronto.edu>

On 17-Feb-10, at 8:38 PM, David Cournapeau wrote:

> Unset any flags you set before (CFLAGS, LDFLAGS, etc...). The error  
> you
> see is caused by -shared missing from LDFLAGS, and gfortran trying
> building an executable instead of a library,

Oops. I unset FFLAGS/F90FLAGS/F77FLAGS but forgot about LDFLAGS. Sorry  
about that.

The build works fine once "ppc64" appears in that list. There are some  
stupid minor test failures where the results are different but the  
difference bounds for assert_array_almost_equal are probably too  
small, but cephes is building correctly now.

I put it in a ticket and attached a patch at http://projects.scipy.org/numpy/ticket/1399 
  just so it's kept track of.

Also, I noticed a duplicate of my last ppc64-related bug showed up in http://projects.scipy.org/numpy/ticket/1272 
  but I don't have privileges necessary to mark it as a duplicate.

Thanks a lot,

David

P.S. Is there a difference between i386 and i686 as far as the Apple  
gcc is concerned? I usually see it named 'i386' but in the source for  
numpy/distutils/fcompiler/gnu.py it's listed as i686.


From robert.kern at gmail.com  Wed Feb 17 23:49:37 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Feb 2010 22:49:37 -0600
Subject: [SciPy-dev] cephes on Mac OS X ppc64?
In-Reply-To: <6D6E5840-6E37-45B8-9D64-B014586324DE@cs.toronto.edu>
References: <74914F3D-16B5-4F8D-9A88-574687036C83@cs.toronto.edu> 
	<3d375d731002171210o6154bddetbb125eb9bdde389@mail.gmail.com> 
	<87BA7EE3-14EC-43C3-8BB4-A14E146AB49D@cs.toronto.edu>
	<4B7C8B18.1070408@silveregg.co.jp> 
	<79F24F8A-279B-400C-B7B0-65FF881E9E73@cs.toronto.edu>
	<4B7C9A16.5040403@silveregg.co.jp> 
	<6D6E5840-6E37-45B8-9D64-B014586324DE@cs.toronto.edu>
Message-ID: <3d375d731002172049l43cd2474r8fe2019254d93f57@mail.gmail.com>

On Wed, Feb 17, 2010 at 21:11, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> P.S. Is there a difference between i386 and i686 as far as the Apple
> gcc is concerned? I usually see it named 'i386' but in the source for
> numpy/distutils/fcompiler/gnu.py it's listed as i686.

As far as the -arch flag is concerned, they are aliases.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Thu Feb 18 01:57:33 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 18 Feb 2010 08:57:33 +0200
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <3d375d731002171708s2e545ac5g40194829a443c262@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002170222g20e1de4fi64343b2d6b1ea8f6@mail.gmail.com> 
	<3d375d731002170805h13928409lc95af1dcff0ca9c5@mail.gmail.com> 
	<dde7764a1002171701sf8eaee5g7886816a5a800eaa@mail.gmail.com> 
	<3d375d731002171708s2e545ac5g40194829a443c262@mail.gmail.com>
Message-ID: <9457e7c81002172257y26111097yff090445b38ffc0b@mail.gmail.com>

On 18 February 2010 03:08, Robert Kern <robert.kern at gmail.com> wrote:
>> The font is monospaced already.
>
> Hmm, not to me. Might be my browser.

Firefox uses monospace, but Safari and Chrome don't.  With Chrome you
can right click on the edit box and change its CSS properties to have
a monospace font, but I'm sure there is an easier way.

Cheers
St?fan


From tom.grydeland at gmail.com  Thu Feb 18 03:58:12 2010
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Thu, 18 Feb 2010 09:58:12 +0100
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <4B7215FE.5030803@silveregg.co.jp>
References: <4B7215FE.5030803@silveregg.co.jp>
Message-ID: <c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>

On Wed, Feb 10, 2010 at 3:12 AM, David Cournapeau <david at silveregg.co.jp> wrote:
> Hi,
>
> I noticed that some of the docstrings I have written for DCT have been
> changed to latex format. While I have no issue with having latex in the
> documentation, I thought the consensus was to use them sparingly in
> docstrings ?

I knew I had seen this discussion before:

http://mail.scipy.org/pipermail/scipy-dev/2009-February/010975.html

Now, I'm not suggesting that my efforts in this are of any
significance whatsoever, but it is annoying to have contributed
something _upon_request_, and then having the exact same thing
reverted, even to applause.

If you don't want it, don't ask for it.

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From josef.pktd at gmail.com  Thu Feb 18 10:22:38 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 10:22:38 -0500
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
Message-ID: <1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>

On Thu, Feb 18, 2010 at 3:58 AM, Tom Grydeland <tom.grydeland at gmail.com> wrote:
> On Wed, Feb 10, 2010 at 3:12 AM, David Cournapeau <david at silveregg.co.jp> wrote:
>> Hi,
>>
>> I noticed that some of the docstrings I have written for DCT have been
>> changed to latex format. While I have no issue with having latex in the
>> documentation, I thought the consensus was to use them sparingly in
>> docstrings ?
>
> I knew I had seen this discussion before:
>
> http://mail.scipy.org/pipermail/scipy-dev/2009-February/010975.html
>
> Now, I'm not suggesting that my efforts in this are of any
> significance whatsoever, but it is annoying to have contributed
> something _upon_request_, and then having the exact same thing
> reverted, even to applause.
>
> If you don't want it, don't ask for it.

Before all the nice Latex disappears, can we rescue it and start a fft
tutorial page in scipy with the collection of all the formulas that
are currently in the docstrings?

I'd rather read rendered Latex than ASCII math expressions.

Josef

>
> --
> Tom Grydeland
> ?<Tom.Grydeland@(gmail.com)>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From jsseabold at gmail.com  Thu Feb 18 13:18:17 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 18 Feb 2010 13:18:17 -0500
Subject: [SciPy-dev] maxentropy docs / parent class methods
In-Reply-To: <dde7764a1002170316s760e2335kd402db5b5c82ec78@mail.gmail.com>
References: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com> 
	<1cd32cbb1002161341r412704c8y65a3eed7000fce8f@mail.gmail.com> 
	<c048da1c1002161405t397449a2vc16ea42e39c02193@mail.gmail.com> 
	<dde7764a1002170316s760e2335kd402db5b5c82ec78@mail.gmail.com>
Message-ID: <c048da1c1002181018u5ee3db68w13b3f26279b823e2@mail.gmail.com>

On Wed, Feb 17, 2010 at 6:16 AM, Ralf Gommers
<ralf.gommers at googlemail.com> wrote:
<snip>
> Those methods do not belong just to 'model' but also to 'bigmodel' and
> 'conditionalmodel'. So listing them under 'model' is a bit arbitrary. I
> think the correct way to do this is to list them under an .. autoclass::
> basemodel, with a note that this class contains shared functionality and
> should not be instantiated directly.

I used autosummary to generate the methods of basemodel.

The one thing I don't understand is why the autoclass directive
doesn't work unless I give the whole path to the class.  I tried
adding

.. currentmodule:: scipy.maxentropy (or just .. module::)

.. autoclass:: basemodel

etc.

But it still didn't seem to respect the current module and show the
class docstrings.  Not a big deal I guess.

Skipper


From josef.pktd at gmail.com  Thu Feb 18 13:29:31 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 13:29:31 -0500
Subject: [SciPy-dev] maxentropy docs / parent class methods
In-Reply-To: <c048da1c1002181018u5ee3db68w13b3f26279b823e2@mail.gmail.com>
References: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com>
	<1cd32cbb1002161341r412704c8y65a3eed7000fce8f@mail.gmail.com>
	<c048da1c1002161405t397449a2vc16ea42e39c02193@mail.gmail.com>
	<dde7764a1002170316s760e2335kd402db5b5c82ec78@mail.gmail.com>
	<c048da1c1002181018u5ee3db68w13b3f26279b823e2@mail.gmail.com>
Message-ID: <1cd32cbb1002181029o47c9a50bk678a81e001abec85@mail.gmail.com>

On Thu, Feb 18, 2010 at 1:18 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Wed, Feb 17, 2010 at 6:16 AM, Ralf Gommers
> <ralf.gommers at googlemail.com> wrote:
> <snip>
>> Those methods do not belong just to 'model' but also to 'bigmodel' and
>> 'conditionalmodel'. So listing them under 'model' is a bit arbitrary. I
>> think the correct way to do this is to list them under an .. autoclass::
>> basemodel, with a note that this class contains shared functionality and
>> should not be instantiated directly.
>
> I used autosummary to generate the methods of basemodel.
>
> The one thing I don't understand is why the autoclass directive
> doesn't work unless I give the whole path to the class. ?I tried
> adding
>
> .. currentmodule:: scipy.maxentropy (or just .. module::)
>
> .. autoclass:: basemodel

does it require the full module path ?

>>> scipy.maxentropy.basemodel
<class 'scipy.maxentropy.maxentropy.basemodel'>

Josef

>
> etc.
>
> But it still didn't seem to respect the current module and show the
> class docstrings. ?Not a big deal I guess.
>
> Skipper
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From jsseabold at gmail.com  Thu Feb 18 13:35:02 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 18 Feb 2010 13:35:02 -0500
Subject: [SciPy-dev] maxentropy docs / parent class methods
In-Reply-To: <1cd32cbb1002181029o47c9a50bk678a81e001abec85@mail.gmail.com>
References: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com> 
	<1cd32cbb1002161341r412704c8y65a3eed7000fce8f@mail.gmail.com> 
	<c048da1c1002161405t397449a2vc16ea42e39c02193@mail.gmail.com> 
	<dde7764a1002170316s760e2335kd402db5b5c82ec78@mail.gmail.com> 
	<c048da1c1002181018u5ee3db68w13b3f26279b823e2@mail.gmail.com> 
	<1cd32cbb1002181029o47c9a50bk678a81e001abec85@mail.gmail.com>
Message-ID: <c048da1c1002181035v6d7ee136h5df0c69f21f96c5c@mail.gmail.com>

On Thu, Feb 18, 2010 at 1:29 PM,  <josef.pktd at gmail.com> wrote:
> On Thu, Feb 18, 2010 at 1:18 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> On Wed, Feb 17, 2010 at 6:16 AM, Ralf Gommers
>> <ralf.gommers at googlemail.com> wrote:
>> <snip>
>>> Those methods do not belong just to 'model' but also to 'bigmodel' and
>>> 'conditionalmodel'. So listing them under 'model' is a bit arbitrary. I
>>> think the correct way to do this is to list them under an .. autoclass::
>>> basemodel, with a note that this class contains shared functionality and
>>> should not be instantiated directly.
>>
>> I used autosummary to generate the methods of basemodel.
>>
>> The one thing I don't understand is why the autoclass directive
>> doesn't work unless I give the whole path to the class. ?I tried
>> adding
>>
>> .. currentmodule:: scipy.maxentropy (or just .. module::)
>>
>> .. autoclass:: basemodel
>
> does it require the full module path ?
>
>>>> scipy.maxentropy.basemodel
> <class 'scipy.maxentropy.maxentropy.basemodel'>
>

Well, that would make more sense, but still doesn't work.

.. currentmodule:: scipy.maxentropy.maxentropy

.. autoclass:: basemodel

Skipper


From josef.pktd at gmail.com  Thu Feb 18 14:05:48 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Feb 2010 14:05:48 -0500
Subject: [SciPy-dev] maxentropy docs / parent class methods
In-Reply-To: <c048da1c1002181035v6d7ee136h5df0c69f21f96c5c@mail.gmail.com>
References: <c048da1c1002161310n23408e80k4d2c184b921666a9@mail.gmail.com>
	<1cd32cbb1002161341r412704c8y65a3eed7000fce8f@mail.gmail.com>
	<c048da1c1002161405t397449a2vc16ea42e39c02193@mail.gmail.com>
	<dde7764a1002170316s760e2335kd402db5b5c82ec78@mail.gmail.com>
	<c048da1c1002181018u5ee3db68w13b3f26279b823e2@mail.gmail.com>
	<1cd32cbb1002181029o47c9a50bk678a81e001abec85@mail.gmail.com>
	<c048da1c1002181035v6d7ee136h5df0c69f21f96c5c@mail.gmail.com>
Message-ID: <1cd32cbb1002181105s25239dafg1001409a1f3e28da@mail.gmail.com>

On Thu, Feb 18, 2010 at 1:35 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Thu, Feb 18, 2010 at 1:29 PM, ?<josef.pktd at gmail.com> wrote:
>> On Thu, Feb 18, 2010 at 1:18 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>>> On Wed, Feb 17, 2010 at 6:16 AM, Ralf Gommers
>>> <ralf.gommers at googlemail.com> wrote:
>>> <snip>
>>>> Those methods do not belong just to 'model' but also to 'bigmodel' and
>>>> 'conditionalmodel'. So listing them under 'model' is a bit arbitrary. I
>>>> think the correct way to do this is to list them under an .. autoclass::
>>>> basemodel, with a note that this class contains shared functionality and
>>>> should not be instantiated directly.
>>>
>>> I used autosummary to generate the methods of basemodel.
>>>
>>> The one thing I don't understand is why the autoclass directive
>>> doesn't work unless I give the whole path to the class. ?I tried
>>> adding
>>>
>>> .. currentmodule:: scipy.maxentropy (or just .. module::)
>>>
>>> .. autoclass:: basemodel
>>
>> does it require the full module path ?
>>
>>>>> scipy.maxentropy.basemodel
>> <class 'scipy.maxentropy.maxentropy.basemodel'>
>>
>
> Well, that would make more sense, but still doesn't work.
>
> .. currentmodule:: scipy.maxentropy.maxentropy
>
> .. autoclass:: basemodel

no idea what's special about autoclass, autosummary works to get the
list of classes (as used in stats.rst), e.g.

.. module:: scipy.maxentropy

.. autosummary::
   :toctree: generated/

   model
   basemodel
   bigmodel
   conditionalmodel


.. autoclass:: scipy.maxentropy.basemodel

Josef

> Skipper
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From farmckon at gmail.com  Thu Feb 18 14:32:54 2010
From: farmckon at gmail.com (Far McKon)
Date: Thu, 18 Feb 2010 14:32:54 -0500
Subject: [SciPy-dev] Build error for scipy
Message-ID: <aae9232e1002181132v6668c6beh56796d35aee1e038@mail.gmail.com>

Hey,
I'm working on a MacBookPro, and trying to get scipy an numpy happy. I
am working from instructions at :
http://www.scipy.org/Installing_SciPy/Mac_OS_X

I just 'setup.py' built and installed numpy, no problems.

When try to build scipy via 'sudo setup.py build" I get the error
stream below.   I checked, and indeed, the file 'npymath.ini' does not
exist on my machine.

Help? What went wrong? Do I have to do some werid comfig change and
rebuild numpy?

thanks in advance for any help,
- Far McKon

== Error Stream =====
Warning: No configuration returned, assuming unavailable.
blas_opt_info:
  FOUND:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
    extra_compile_args = ['-msse3',
'-I/System/Library/Frameworks/vecLib.framework/Headers']

lapack_opt_info:
  FOUND:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
    extra_compile_args = ['-msse3']

umfpack_info:
  libraries umfpack not found in
/System/Library/Frameworks/Python.framework/Versions/2.5/lib
  libraries umfpack not found in /usr/local/lib
  libraries umfpack not found in /usr/lib
  libraries umfpack not found in /opt/local/lib
  libraries umfpack not found in /sw/lib
/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/system_info.py:452:
UserWarning:
    UMFPACK sparse solver (http://www.cise.ufl.edu/research/sparse/umfpack/)
    not found. Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [umfpack]) or by setting
    the UMFPACK environment variable.
  warnings.warn(self.notfounderror.__doc__)
  NOT AVAILABLE

Traceback (most recent call last):
  File "setup.py", line 160, in <module>
    setup_package()
  File "setup.py", line 152, in setup_package
    configuration=configuration )
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/core.py",
line 152, in setup
    config = configuration()
  File "setup.py", line 118, in configuration
    config.add_subpackage('scipy')
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/misc_util.py",
line 957, in add_subpackage
    caller_level = 2)
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/misc_util.py",
line 926, in get_subpackage
    caller_level = caller_level + 1)
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/misc_util.py",
line 863, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "scipy/setup.py", line 20, in configuration
    config.add_subpackage('special')
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/misc_util.py",
line 957, in add_subpackage
    caller_level = 2)
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/misc_util.py",
line 926, in get_subpackage
    caller_level = caller_level + 1)
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/misc_util.py",
line 863, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "scipy/special/setup.py", line 45, in configuration
    extra_info=get_info("npymath")
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/misc_util.py",
line 2068, in get_info
    pkg_info = get_pkg_info(pkgname, dirs)
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/misc_util.py",
line 2020, in get_pkg_info
    return read_config(pkgname, dirs)
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/npy_pkg_config.py",
line 343, in read_config
    v = _read_config_imp(pkg_to_filename(pkgname), dirs)
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/npy_pkg_config.py",
line 317, in _read_config_imp
    meta, vars, sections, reqs = _read_config(filenames)
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/npy_pkg_config.py",
line 301, in _read_config
    meta, vars, sections, reqs = parse_config(f, dirs)
  File "/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/distutils/npy_pkg_config.py",
line 273, in parse_config
    raise PkgNotFound("Could not find file(s) %s" % str(filenames))
numpy.distutils.npy_pkg_config.PkgNotFound: Could not find file(s)
['/Library/Python/2.5/site-packages/numpy-1.4.0-py2.5-macosx-10.5-i386.egg/numpy/core/lib/npy-pkg-config/npymath.ini']


From david at silveregg.co.jp  Thu Feb 18 19:34:18 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Fri, 19 Feb 2010 09:34:18 +0900
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
Message-ID: <4B7DDC8A.5050400@silveregg.co.jp>

josef.pktd at gmail.com wrote:

> Before all the nice Latex disappears, can we rescue it and start a fft
> tutorial page in scipy with the collection of all the formulas that
> are currently in the docstrings?

I never suggested to remove the latex from the doc, only from the 
docstring. I agree rendered latex is better than ASCII when put in the 
scipy.fftpack doc (or tutorial) - I did not know how to do it, that's 
why I did not remove the latex for now.

cheers,

David


From josef.pktd at gmail.com  Fri Feb 19 01:10:22 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 01:10:22 -0500
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <4B7DDC8A.5050400@silveregg.co.jp>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
	<4B7DDC8A.5050400@silveregg.co.jp>
Message-ID: <1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>

On Thu, Feb 18, 2010 at 7:34 PM, David Cournapeau <david at silveregg.co.jp> wrote:
> josef.pktd at gmail.com wrote:
>
>> Before all the nice Latex disappears, can we rescue it and start a fft
>> tutorial page in scipy with the collection of all the formulas that
>> are currently in the docstrings?
>
> I never suggested to remove the latex from the doc, only from the
> docstring. I agree rendered latex is better than ASCII when put in the
> scipy.fftpack doc (or tutorial) - I did not know how to do it, that's
> why I did not remove the latex for now.

Almost none of the scipy sub packages has module level docs, for most
it's just at most a few paragraphs and then the list of routines.

The tutorials are a mixture of definitions, background material and
examples, e.g. linalg and signal contain mostly definitions and
explanations.

I created a stub tutorial page for fftpack,
http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/fftpack.rst/
and copied the last correct latex version for dct.
I latex math doesn't render, so I changed the first two equations to
the same pattern as the equations are in the signal tutorial which
works. I also copied the first paragraph from the numpy fft module
description.,

Also I'm not sure what the section structure should be, but it's setup
for anyone to correct and add information.

Josef


>
> cheers,
>
> David
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From josef.pktd at gmail.com  Fri Feb 19 01:16:42 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 01:16:42 -0500
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
	<4B7DDC8A.5050400@silveregg.co.jp>
	<1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
Message-ID: <1cd32cbb1002182216g99f001dwa1e66e42d2ffa27a@mail.gmail.com>

On Fri, Feb 19, 2010 at 1:10 AM,  <josef.pktd at gmail.com> wrote:
> On Thu, Feb 18, 2010 at 7:34 PM, David Cournapeau <david at silveregg.co.jp> wrote:
>> josef.pktd at gmail.com wrote:
>>
>>> Before all the nice Latex disappears, can we rescue it and start a fft
>>> tutorial page in scipy with the collection of all the formulas that
>>> are currently in the docstrings?
>>
>> I never suggested to remove the latex from the doc, only from the
>> docstring. I agree rendered latex is better than ASCII when put in the
>> scipy.fftpack doc (or tutorial) - I did not know how to do it, that's
>> why I did not remove the latex for now.
>
> Almost none of the scipy sub packages has module level docs, for most
> it's just at most a few paragraphs and then the list of routines.
>
> The tutorials are a mixture of definitions, background material and
> examples, e.g. linalg and signal contain mostly definitions and
> explanations.
>
> I created a stub tutorial page for fftpack,
> http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/fftpack.rst/
> and copied the last correct latex version for dct.
> I latex math doesn't render, so I changed the first two equations to
> the same pattern as the equations are in the signal tutorial which
> works. I also copied the first paragraph from the numpy fft module
> description.,
>
> Also I'm not sure what the section structure should be, but it's setup
> for anyone to correct and add information.

And I haven't linked it yet from the tutorial front page, so will not
yet be included in the docs (as long as it's mostly a stub page)
>
> Josef
>
>
>>
>> cheers,
>>
>> David
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>


From d.l.goldsmith at gmail.com  Fri Feb 19 01:23:22 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Thu, 18 Feb 2010 22:23:22 -0800
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
	<4B7DDC8A.5050400@silveregg.co.jp>
	<1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
Message-ID: <45d1ab481002182223m669b4b07ga267487895706d46@mail.gmail.com>

Thanks, Josef!

DG

On Thu, Feb 18, 2010 at 10:10 PM, <josef.pktd at gmail.com> wrote:

> On Thu, Feb 18, 2010 at 7:34 PM, David Cournapeau <david at silveregg.co.jp>
> wrote:
> > josef.pktd at gmail.com wrote:
> >
> >> Before all the nice Latex disappears, can we rescue it and start a fft
> >> tutorial page in scipy with the collection of all the formulas that
> >> are currently in the docstrings?
> >
> > I never suggested to remove the latex from the doc, only from the
> > docstring. I agree rendered latex is better than ASCII when put in the
> > scipy.fftpack doc (or tutorial) - I did not know how to do it, that's
> > why I did not remove the latex for now.
>
> Almost none of the scipy sub packages has module level docs, for most
> it's just at most a few paragraphs and then the list of routines.
>
> The tutorials are a mixture of definitions, background material and
> examples, e.g. linalg and signal contain mostly definitions and
> explanations.
>
> I created a stub tutorial page for fftpack,
> http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/fftpack.rst/
> and copied the last correct latex version for dct.
> I latex math doesn't render, so I changed the first two equations to
> the same pattern as the equations are in the signal tutorial which
> works. I also copied the first paragraph from the numpy fft module
> description.,
>
> Also I'm not sure what the section structure should be, but it's setup
> for anyone to correct and add information.
>
> Josef
>
>
> >
> > cheers,
> >
> > David
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-dev
> >
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100218/d559b210/attachment.html>

From tom.grydeland at gmail.com  Fri Feb 19 02:45:32 2010
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Fri, 19 Feb 2010 08:45:32 +0100
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
	<4B7DDC8A.5050400@silveregg.co.jp>
	<1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
Message-ID: <c21c0b2f1002182345l44850424o375938c5c8e0e900@mail.gmail.com>

On Fri, Feb 19, 2010 at 7:10 AM,  <josef.pktd at gmail.com> wrote:

> I created a stub tutorial page for fftpack,
> http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/fftpack.rst/
> and copied the last correct latex version for dct.
> I latex math doesn't render, so I changed the first two equations to
> the same pattern as the equations are in the signal tutorial which
> works. I also copied the first paragraph from the numpy fft module
> description.,

Thanks Josef!

If the DCT matter is right for a tutorial, I believe a lot more of the
numpy.fft module description would fit also.

Do you want me to fix the latex errors?

> Josef

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From oliphant at ee.byu.edu  Fri Feb 19 02:49:33 2010
From: oliphant at ee.byu.edu (oliphant at ee.byu.edu)
Date: Fri, 19 Feb 2010 16:49:33 +0900
Subject: [SciPy-dev] Mail Delivery (failure scipy-dev@scipy.org)
Message-ID: <20100219074927.3F13C39CAF6@scipy.org>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100219/a1c988ae/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: message.scr
Type: audio/x-wav
Size: 29568 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100219/a1c988ae/attachment.wav>

From njs at pobox.com  Fri Feb 19 03:08:50 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 19 Feb 2010 00:08:50 -0800
Subject: [SciPy-dev] Best interface for computing the logarithm of the
	determinant?
In-Reply-To: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
References: <961fa2b41002161249g66dde100s251cf8874d2a996e@mail.gmail.com>
Message-ID: <961fa2b41002190008v81f5cd1o57f20b6d27211623@mail.gmail.com>

Thanks for your comments, all. Since it occurs to me that this is a
general need, not just for sparse matrices, and it would be very
annoying to settle on one API for scikits.sparse and then have another
show up in one of the main packages later, I've just submitted a patch
for option (1) to numpy:
  http://projects.scipy.org/numpy/ticket/1402
And we'll see what happens with that :-)

On Tue, Feb 16, 2010 at 12:49 PM, Nathaniel Smith <njs at pobox.com> wrote:
> So when you have a matrix whose determinant you want, it's often wise
> to compute the logarithm of the determinant instead of the determinant
> itself, because determinants involve lots and lots of multiplications
> and the result might otherwise underflow/overflow. Therefore, in
> scikits.sparse, I'd like to provide an API for doing this (and this is
> well-supported by the underlying libraries).
>
> But the problem is that for a general matrix, the determinant may be
> zero or negative. Obviously we can deal with this, but what's the best
> API? I'd like to use one consistently across the different
> factorizations in scikits.sparse, and perhaps eventually in numpy as
> well.
>
> Some options:
>
> 1) Split off the sign into a separate return value ('sign' may be 1, -1, 0):
> sign, value = logdet(A)
> actual_determinant = sign * exp(value)
>
> 2) Allow complex/infinite return values, even when A is a real matrix:
> logdet(eye(3)) == pi*1j
> logdet(zeros((3, 3))) == -Inf
>
> 3) "Scientific notation" (This is what UMFPACK's API does): return a
> mantissa and base-10 exponent:
> mantissa, exponent = logdet(A)
> actual_determinant = mantissa * 10 ** exponent
>
> 4) Have separate functions for computing the sign, and the log of the
> absolute value (This is what GSL does, though it seems pointlessly
> inefficient):
> sign = sgndet(A)
> value = logdet(A)
> actual_determinant = sign * exp(value)
>
> These are all kind of ugly looking, unfortunately, but that seems
> unavoidable, unless someone has a clever idea.
>
> Any preferences?
>
> -- Nathaniel
>


From josef.pktd at gmail.com  Fri Feb 19 07:24:28 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 07:24:28 -0500
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <c21c0b2f1002182345l44850424o375938c5c8e0e900@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
	<4B7DDC8A.5050400@silveregg.co.jp>
	<1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
	<c21c0b2f1002182345l44850424o375938c5c8e0e900@mail.gmail.com>
Message-ID: <1cd32cbb1002190424p681f3111ic3718d8b5d260053@mail.gmail.com>

On Fri, Feb 19, 2010 at 2:45 AM, Tom Grydeland <tom.grydeland at gmail.com> wrote:
> On Fri, Feb 19, 2010 at 7:10 AM, ?<josef.pktd at gmail.com> wrote:
>
>> I created a stub tutorial page for fftpack,
>> http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/fftpack.rst/
>> and copied the last correct latex version for dct.
>> I latex math doesn't render, so I changed the first two equations to
>> the same pattern as the equations are in the signal tutorial which
>> works. I also copied the first paragraph from the numpy fft module
>> description.,
>
> Thanks Josef!
>
> If the DCT matter is right for a tutorial, I believe a lot more of the
> numpy.fft module description would fit also.
>
> Do you want me to fix the latex errors?

Yes, please, and make any adjustment that you find usefule.

One formatting: I think the header ordering is === ----- ^^^^^ ~~~  ,
if my inference from other rst pages is correct.
Because I don't know what the section hierarchy might be, I didn't
change to the correct underlines for headers of  subsections.

I didn't want to copy too much from the numpy docs since we can also
just refer to it. For my reading an abridged explanation refering to
the numpy docs, and a more complete collection of the extra parts that
in scipy would be useful. But we haven't had much discussion yet on
doc duplication between scipy and numpy (which is also relevant for
the descriptions of the distributions in numpy.random and
scipy.stats.distributions)

Thanks,

Josef

>
>> Josef
>
> --
> Tom Grydeland
> ?<Tom.Grydeland@(gmail.com)>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From tom.grydeland at gmail.com  Fri Feb 19 08:05:05 2010
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Fri, 19 Feb 2010 14:05:05 +0100
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <1cd32cbb1002190424p681f3111ic3718d8b5d260053@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
	<4B7DDC8A.5050400@silveregg.co.jp>
	<1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
	<c21c0b2f1002182345l44850424o375938c5c8e0e900@mail.gmail.com>
	<1cd32cbb1002190424p681f3111ic3718d8b5d260053@mail.gmail.com>
Message-ID: <c21c0b2f1002190505g6c3114e2v9e111a7779e8cbb6@mail.gmail.com>

> On Fri, Feb 19, 2010 at 2:45 AM, Tom Grydeland <tom.grydeland at gmail.com> wrote:
>> Do you want me to fix the latex errors?

On Fri, Feb 19, 2010 at 1:24 PM,  <josef.pktd at gmail.com> wrote:
> Yes, please, and make any adjustment that you find usefule.

Okay, they are fixed, and I unified all the references cited also.

> I didn't want to copy too much from the numpy docs since we can also
> just refer to it. For my reading an abridged explanation refering to
> the numpy docs, and a more complete collection of the extra parts that
> in scipy would be useful. But we haven't had much discussion yet on
> doc duplication between scipy and numpy (which is also relevant for
> the descriptions of the distributions in numpy.random and
> scipy.stats.distributions)

I'm fine either way, but I think for a tutorial page for fftpack it
seems odd to define the DCT in that level of detail and then refer
elsewhere for the DFT definition.

> Josef

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From josef.pktd at gmail.com  Fri Feb 19 09:24:52 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 09:24:52 -0500
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <c21c0b2f1002190505g6c3114e2v9e111a7779e8cbb6@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
	<4B7DDC8A.5050400@silveregg.co.jp>
	<1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
	<c21c0b2f1002182345l44850424o375938c5c8e0e900@mail.gmail.com>
	<1cd32cbb1002190424p681f3111ic3718d8b5d260053@mail.gmail.com>
	<c21c0b2f1002190505g6c3114e2v9e111a7779e8cbb6@mail.gmail.com>
Message-ID: <1cd32cbb1002190624q75bca956oc1cc1d44a8821ebc@mail.gmail.com>

On Fri, Feb 19, 2010 at 8:05 AM, Tom Grydeland <tom.grydeland at gmail.com> wrote:
>> On Fri, Feb 19, 2010 at 2:45 AM, Tom Grydeland <tom.grydeland at gmail.com> wrote:
>>> Do you want me to fix the latex errors?
>
> On Fri, Feb 19, 2010 at 1:24 PM, ?<josef.pktd at gmail.com> wrote:
>> Yes, please, and make any adjustment that you find usefule.
>
> Okay, they are fixed, and I unified all the references cited also.
>
>> I didn't want to copy too much from the numpy docs since we can also
>> just refer to it. For my reading an abridged explanation refering to
>> the numpy docs, and a more complete collection of the extra parts that
>> in scipy would be useful. But we haven't had much discussion yet on
>> doc duplication between scipy and numpy (which is also relevant for
>> the descriptions of the distributions in numpy.random and
>> scipy.stats.distributions)
>
> I'm fine either way, but I think for a tutorial page for fftpack it
> seems odd to define the DCT in that level of detail and then refer
> elsewhere for the DFT definition.

I don't have a strong opinion either.
The numpy.fft module docstring is short enough (when rendered on the
webpage) that most of it can be repeated. My initial thought was that
fft in scipy could be explained more in terms of fft2 and fftn, with
examples for those. (I just recently struggled with fftn and there is
not much example code available.)
I think, the definitions and formulas should be included for all
transforms in any case.

Josef

>
>> Josef
>
> --
> Tom Grydeland
> ?<Tom.Grydeland@(gmail.com)>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From nwagner at iam.uni-stuttgart.de  Fri Feb 19 11:41:48 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 19 Feb 2010 17:41:48 +0100
Subject: [SciPy-dev] New scipy.test() errors in trunk
Message-ID: <web-130528976@uni-stuttgart.de>

======================================================================
ERROR: Regression test for #880: empty array for zi 
crashes.
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py", 
line 458, in test_empty_zi
     y, zf = lfilter(b, a, x, zi=zi)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py", 
line 603, in lfilter
     return sigtools._linear_filter(b, a, x, axis, zi)
NotImplementedError: input type 'object' not supported


======================================================================
ERROR: test_rank1 
(test_signaltools.TestLinearFilterDecimal)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py", 
line 360, in test_rank1
     assert_array_almost_equal(lfilter(b, a, x), y_r)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py", 
line 601, in lfilter
     return sigtools._linear_filter(b, a, x, axis)
NotImplementedError: input type 'object' not supported


======================================================================
ERROR: test_rank2 
(test_signaltools.TestLinearFilterDecimal)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py", 
line 401, in test_rank2
     y = lfilter(b, a, x, axis = 0)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py", 
line 601, in lfilter
     return sigtools._linear_filter(b, a, x, axis)
NotImplementedError: input type 'object' not supported


======================================================================
ERROR: test_rank2_init_cond_a0 
(test_signaltools.TestLinearFilterDecimal)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py", 
line 435, in test_rank2_init_cond_a0
     y, zf = lfilter(b, a, x, axis = 0, zi = np.ones((1, 
3)))
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py", 
line 603, in lfilter
     return sigtools._linear_filter(b, a, x, axis, zi)
NotImplementedError: input type 'object' not supported


======================================================================
ERROR: test_rank2_init_cond_a1 
(test_signaltools.TestLinearFilterDecimal)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py", 
line 419, in test_rank2_init_cond_a1
     y, zf = lfilter(b, a, x, axis = 1, zi = np.ones((4, 
1)))
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py", 
line 603, in lfilter
     return sigtools._linear_filter(b, a, x, axis, zi)
NotImplementedError: input type 'object' not supported


======================================================================
ERROR: test_rank3 
(test_signaltools.TestLinearFilterDecimal)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py", 
line 447, in test_rank3
     y = lfilter(b, a, x)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py", 
line 601, in lfilter
     return sigtools._linear_filter(b, a, x, axis)
NotImplementedError: input type 'object' not supported


======================================================================
ERROR: test_mpmath.test_hyp2f1_real_some
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", 
line 146, in skipper_func
     return f(*args, **kwargs)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_mpmath.py", 
line 117, in test_hyp2f1_real_some
     except (TypeError, mpmath.libhyper.NoConvergence):
AttributeError: 'module' object has no attribute 
'libhyper'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.bernoulli_gen 
object at 0x2636ad0>, (0.29999999999999999,), array([0, 0, 
0, ..., 1, 0, 0]), 0.01, 'bernoulli chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.binom_gen 
object at 0x26368d0>, (5, 0.40000000000000002), array([2, 
2, 2, ..., 4, 1, 3]), 0.01, 'binom chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.boltzmann_gen 
object at 0x26ec590>, (1.3999999999999999, 19), array([0, 
0, 0, ..., 2, 0, 0]), 0.01, 'boltzmann chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.dlaplace_gen 
object at 0x26eca50>, (0.80000000000000004,), array([ 0, 
 0,  0, ...,  4, -1,  0]), 0.01, 'dlaplace chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.geom_gen 
object at 0x2636dd0>, (0.5,), array([1, 1, 2, ..., 6, 1, 
2]), 0.01, 'geom chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.hypergeom_gen 
object at 0x2636f50>, (30, 12, 6), array([1, 1, 4, ..., 3, 
2, 2]), 0.01, 'hypergeom chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.hypergeom_gen 
object at 0x2636f50>, (21, 3, 12), array([2, 3, 2, ..., 2, 
2, 1]), 0.01, 'hypergeom chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.hypergeom_gen 
object at 0x2636f50>, (21, 18, 11), array([ 9,  8,  9, 
..., 10, 10, 10]), 0.01, 'hypergeom chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.logser_gen 
object at 0x26ec110>, (0.59999999999999998,), array([1, 1, 
1, ..., 1, 1, 4]), 0.01, 'logser chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.nbinom_gen 
object at 0x2636c50>, (5, 0.5), array([0, 2, 6, ..., 3, 3, 
3]), 0.01, 'nbinom chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.nbinom_gen 
object at 0x2636c50>, (0.40000000000000002, 
0.40000000000000002), array([0, 0, 0, ..., 0, 0, 0]), 
0.01, 'nbinom chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.planck_gen 
object at 0x26ec410>, (0.51000000000000001,), array([1, 1, 
1, ..., 7, 0, 2]), 0.01, 'planck chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.poisson_gen 
object at 0x26ec290>, (0.59999999999999998,), array([0, 0, 
0, ..., 1, 0, 0]), 0.01, 'poisson chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.randint_gen 
object at 0x26ec710>, (7, 31), array([27, 10, 15, ..., 16, 
 9, 17]), 0.01, 'randint chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.skellam_gen 
object at 0x26ecbd0>, (15, 8), array([ 4,  6, 10, ...,  5, 
14, 15]), 0.01, 'skellam chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
ERROR: 
test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.zipf_gen 
object at 0x26ec8d0>, (4,), array([1, 1, 1, ..., 1, 1, 
1]), 0.01, 'zipf chisquare')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py", 
line 252, in check_discrete_chisquare
     freq,hsupp = np.histogram(rvs,histsupp,new=True)
TypeError: histogram() got an unexpected keyword argument 
'new'

======================================================================
FAIL: test_lambertw.test_values
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_lambertw.py", 
line 80, in test_values
     FuncData(w, data, (0,1), 2, rtol=1e-10, 
atol=1e-13).check()
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/testutils.py", 
line 187, in check
     assert False, "\n".join(msg)
AssertionError:
Max |adiff|: 1.77636e-15
Max |rdiff|: 2.06237e-16
Bad results for the following points (in output 0):
(-0.40000000000000002+0.40000000000000002j) 
                             0j => 
                   (nan+nan*j) != 
(-0.10396515323290657+0.61899273315171632j)  (rdiff 
                           0.0)
(-0.44800000000000001+0.40000000000000002j) 
                             0j => 
                   (nan+nan*j) != 
(-0.11855133765652383+0.66570534313583418j)  (rdiff 
                           0.0)
(-0.44800000000000001-0.40000000000000002j) 
                             0j => 
                   (nan+nan*j) != 
(-0.11855133765652383-0.66570534313583418j)  (rdiff 
                           0.0)

======================================================================
FAIL: test_mpmath.test_hyp2f1_some_points_2
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", 
line 183, in runTest
     self.test(*self.arg)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", 
line 146, in skipper_func
     return f(*args, **kwargs)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_mpmath.py", 
line 106, in test_hyp2f1_some_points_2
     FuncData(sc.hyp2f1, dataset, (0,1,2,3), 4, 
rtol=1e-10).check()
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/testutils.py", 
line 187, in check
     assert False, "\n".join(msg)
AssertionError:
Max |adiff|: 1.48367e-22
Max |rdiff|: 0.989172
Bad results for the following points (in output 0):
                          112.0 
             5.0999999999999996 
           -0.90000000000000002 
           -0.99999000000000005 => 
       -1.6241361047970947e-24 != 
        -1.499910195656332e-22  (rdiff 
           0.98917177768708742)

----------------------------------------------------------------------
Ran 4332 tests in 76.236s

FAILED (KNOWNFAIL=11, SKIP=17, errors=24, failures=2)
<nose.result.TextTestResult run=4332 errors=24 failures=2>


  


From josef.pktd at gmail.com  Fri Feb 19 11:49:10 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 11:49:10 -0500
Subject: [SciPy-dev] New scipy.test() errors in trunk
In-Reply-To: <web-130528976@uni-stuttgart.de>
References: <web-130528976@uni-stuttgart.de>
Message-ID: <1cd32cbb1002190849i3082e2c7h709ae53f07c274d9@mail.gmail.com>

Are you running scipy against numpy trunk?

np.histogram(rvs,histsupp,new=True)

I think this is a very recent change that will need adjustments.
However, it might be difficult to write code that works with numpy 1.3
and numpy 1.5, with or without "new" keyword. I need to check when the
default for np.histogram changed.

Josef


On Fri, Feb 19, 2010 at 11:41 AM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> ======================================================================
> ERROR: Regression test for #880: empty array for zi
> crashes.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
> line 458, in test_empty_zi
> ? ? y, zf = lfilter(b, a, x, zi=zi)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
> line 603, in lfilter
> ? ? return sigtools._linear_filter(b, a, x, axis, zi)
> NotImplementedError: input type 'object' not supported
>
>
> ======================================================================
> ERROR: test_rank1
> (test_signaltools.TestLinearFilterDecimal)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
> line 360, in test_rank1
> ? ? assert_array_almost_equal(lfilter(b, a, x), y_r)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
> line 601, in lfilter
> ? ? return sigtools._linear_filter(b, a, x, axis)
> NotImplementedError: input type 'object' not supported
>
>
> ======================================================================
> ERROR: test_rank2
> (test_signaltools.TestLinearFilterDecimal)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
> line 401, in test_rank2
> ? ? y = lfilter(b, a, x, axis = 0)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
> line 601, in lfilter
> ? ? return sigtools._linear_filter(b, a, x, axis)
> NotImplementedError: input type 'object' not supported
>
>
> ======================================================================
> ERROR: test_rank2_init_cond_a0
> (test_signaltools.TestLinearFilterDecimal)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
> line 435, in test_rank2_init_cond_a0
> ? ? y, zf = lfilter(b, a, x, axis = 0, zi = np.ones((1,
> 3)))
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
> line 603, in lfilter
> ? ? return sigtools._linear_filter(b, a, x, axis, zi)
> NotImplementedError: input type 'object' not supported
>
>
> ======================================================================
> ERROR: test_rank2_init_cond_a1
> (test_signaltools.TestLinearFilterDecimal)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
> line 419, in test_rank2_init_cond_a1
> ? ? y, zf = lfilter(b, a, x, axis = 1, zi = np.ones((4,
> 1)))
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
> line 603, in lfilter
> ? ? return sigtools._linear_filter(b, a, x, axis, zi)
> NotImplementedError: input type 'object' not supported
>
>
> ======================================================================
> ERROR: test_rank3
> (test_signaltools.TestLinearFilterDecimal)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
> line 447, in test_rank3
> ? ? y = lfilter(b, a, x)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
> line 601, in lfilter
> ? ? return sigtools._linear_filter(b, a, x, axis)
> NotImplementedError: input type 'object' not supported
>
>
> ======================================================================
> ERROR: test_mpmath.test_hyp2f1_real_some
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py",
> line 146, in skipper_func
> ? ? return f(*args, **kwargs)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_mpmath.py",
> line 117, in test_hyp2f1_real_some
> ? ? except (TypeError, mpmath.libhyper.NoConvergence):
> AttributeError: 'module' object has no attribute
> 'libhyper'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.bernoulli_gen
> object at 0x2636ad0>, (0.29999999999999999,), array([0, 0,
> 0, ..., 1, 0, 0]), 0.01, 'bernoulli chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.binom_gen
> object at 0x26368d0>, (5, 0.40000000000000002), array([2,
> 2, 2, ..., 4, 1, 3]), 0.01, 'binom chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.boltzmann_gen
> object at 0x26ec590>, (1.3999999999999999, 19), array([0,
> 0, 0, ..., 2, 0, 0]), 0.01, 'boltzmann chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.dlaplace_gen
> object at 0x26eca50>, (0.80000000000000004,), array([ 0,
> ?0, ?0, ..., ?4, -1, ?0]), 0.01, 'dlaplace chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.geom_gen
> object at 0x2636dd0>, (0.5,), array([1, 1, 2, ..., 6, 1,
> 2]), 0.01, 'geom chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.hypergeom_gen
> object at 0x2636f50>, (30, 12, 6), array([1, 1, 4, ..., 3,
> 2, 2]), 0.01, 'hypergeom chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.hypergeom_gen
> object at 0x2636f50>, (21, 3, 12), array([2, 3, 2, ..., 2,
> 2, 1]), 0.01, 'hypergeom chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.hypergeom_gen
> object at 0x2636f50>, (21, 18, 11), array([ 9, ?8, ?9,
> ..., 10, 10, 10]), 0.01, 'hypergeom chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.logser_gen
> object at 0x26ec110>, (0.59999999999999998,), array([1, 1,
> 1, ..., 1, 1, 4]), 0.01, 'logser chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.nbinom_gen
> object at 0x2636c50>, (5, 0.5), array([0, 2, 6, ..., 3, 3,
> 3]), 0.01, 'nbinom chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.nbinom_gen
> object at 0x2636c50>, (0.40000000000000002,
> 0.40000000000000002), array([0, 0, 0, ..., 0, 0, 0]),
> 0.01, 'nbinom chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.planck_gen
> object at 0x26ec410>, (0.51000000000000001,), array([1, 1,
> 1, ..., 7, 0, 2]), 0.01, 'planck chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.poisson_gen
> object at 0x26ec290>, (0.59999999999999998,), array([0, 0,
> 0, ..., 1, 0, 0]), 0.01, 'poisson chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.randint_gen
> object at 0x26ec710>, (7, 31), array([27, 10, 15, ..., 16,
> ?9, 17]), 0.01, 'randint chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.skellam_gen
> object at 0x26ecbd0>, (15, 8), array([ 4, ?6, 10, ..., ?5,
> 14, 15]), 0.01, 'skellam chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> ERROR:
> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.zipf_gen
> object at 0x26ec8d0>, (4,), array([1, 1, 1, ..., 1, 1,
> 1]), 0.01, 'zipf chisquare')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
> line 252, in check_discrete_chisquare
> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
> TypeError: histogram() got an unexpected keyword argument
> 'new'
>
> ======================================================================
> FAIL: test_lambertw.test_values
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_lambertw.py",
> line 80, in test_values
> ? ? FuncData(w, data, (0,1), 2, rtol=1e-10,
> atol=1e-13).check()
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/testutils.py",
> line 187, in check
> ? ? assert False, "\n".join(msg)
> AssertionError:
> Max |adiff|: 1.77636e-15
> Max |rdiff|: 2.06237e-16
> Bad results for the following points (in output 0):
> (-0.40000000000000002+0.40000000000000002j)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0j =>
> ? ? ? ? ? ? ? ? ? (nan+nan*j) !=
> (-0.10396515323290657+0.61899273315171632j) ?(rdiff
> ? ? ? ? ? ? ? ? ? ? ? ? ? 0.0)
> (-0.44800000000000001+0.40000000000000002j)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0j =>
> ? ? ? ? ? ? ? ? ? (nan+nan*j) !=
> (-0.11855133765652383+0.66570534313583418j) ?(rdiff
> ? ? ? ? ? ? ? ? ? ? ? ? ? 0.0)
> (-0.44800000000000001-0.40000000000000002j)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0j =>
> ? ? ? ? ? ? ? ? ? (nan+nan*j) !=
> (-0.11855133765652383-0.66570534313583418j) ?(rdiff
> ? ? ? ? ? ? ? ? ? ? ? ? ? 0.0)
>
> ======================================================================
> FAIL: test_mpmath.test_hyp2f1_some_points_2
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
> line 183, in runTest
> ? ? self.test(*self.arg)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py",
> line 146, in skipper_func
> ? ? return f(*args, **kwargs)
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_mpmath.py",
> line 106, in test_hyp2f1_some_points_2
> ? ? FuncData(sc.hyp2f1, dataset, (0,1,2,3), 4,
> rtol=1e-10).check()
> ? File
> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/testutils.py",
> line 187, in check
> ? ? assert False, "\n".join(msg)
> AssertionError:
> Max |adiff|: 1.48367e-22
> Max |rdiff|: 0.989172
> Bad results for the following points (in output 0):
> ? ? ? ? ? ? ? ? ? ? ? ? ?112.0
> ? ? ? ? ? ? 5.0999999999999996
> ? ? ? ? ? -0.90000000000000002
> ? ? ? ? ? -0.99999000000000005 =>
> ? ? ? -1.6241361047970947e-24 !=
> ? ? ? ?-1.499910195656332e-22 ?(rdiff
> ? ? ? ? ? 0.98917177768708742)
>
> ----------------------------------------------------------------------
> Ran 4332 tests in 76.236s
>
> FAILED (KNOWNFAIL=11, SKIP=17, errors=24, failures=2)
> <nose.result.TextTestResult run=4332 errors=24 failures=2>
>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From nwagner at iam.uni-stuttgart.de  Fri Feb 19 11:51:44 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 19 Feb 2010 17:51:44 +0100
Subject: [SciPy-dev] New scipy.test() errors in trunk
In-Reply-To: <1cd32cbb1002190849i3082e2c7h709ae53f07c274d9@mail.gmail.com>
References: <web-130528976@uni-stuttgart.de>
	<1cd32cbb1002190849i3082e2c7h709ae53f07c274d9@mail.gmail.com>
Message-ID: <web-130529242@uni-stuttgart.de>

On Fri, 19 Feb 2010 11:49:10 -0500
  josef.pktd at gmail.com wrote:
> Are you running scipy against numpy trunk?
> 
Yes.

> np.histogram(rvs,histsupp,new=True)
> 
> I think this is a very recent change that will need 
>adjustments.

See

r8116 | dhuard | 2010-02-16 19:52:08 +0100 (Di, 16. Feb 
2010) | 1 Zeile

removed old behavior for the histogram function.


> However, it might be difficult to write code that works 
>with numpy 1.3
> and numpy 1.5, with or without "new" keyword. I need to 
>check when the
> default for np.histogram changed.
> 
> Josef
> 

Nils


From josef.pktd at gmail.com  Fri Feb 19 11:54:29 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 19 Feb 2010 11:54:29 -0500
Subject: [SciPy-dev] New scipy.test() errors in trunk
In-Reply-To: <1cd32cbb1002190849i3082e2c7h709ae53f07c274d9@mail.gmail.com>
References: <web-130528976@uni-stuttgart.de>
	<1cd32cbb1002190849i3082e2c7h709ae53f07c274d9@mail.gmail.com>
Message-ID: <1cd32cbb1002190854j4469e8f1s4e498c4d8f8550a7@mail.gmail.com>

On Fri, Feb 19, 2010 at 11:49 AM,  <josef.pktd at gmail.com> wrote:
> Are you running scipy against numpy trunk?
>
> np.histogram(rvs,histsupp,new=True)
>
> I think this is a very recent change that will need adjustments.
> However, it might be difficult to write code that works with numpy 1.3
> and numpy 1.5, with or without "new" keyword. I need to check when the
> default for np.histogram changed.

No, this won't matter, since scipy trunk requires at least numpy 1.4

Thanks for reporting,

Josef

>
> Josef
>
>
> On Fri, Feb 19, 2010 at 11:41 AM, Nils Wagner
> <nwagner at iam.uni-stuttgart.de> wrote:
>> ======================================================================
>> ERROR: Regression test for #880: empty array for zi
>> crashes.
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
>> line 458, in test_empty_zi
>> ? ? y, zf = lfilter(b, a, x, zi=zi)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
>> line 603, in lfilter
>> ? ? return sigtools._linear_filter(b, a, x, axis, zi)
>> NotImplementedError: input type 'object' not supported
>>
>>
>> ======================================================================
>> ERROR: test_rank1
>> (test_signaltools.TestLinearFilterDecimal)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
>> line 360, in test_rank1
>> ? ? assert_array_almost_equal(lfilter(b, a, x), y_r)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
>> line 601, in lfilter
>> ? ? return sigtools._linear_filter(b, a, x, axis)
>> NotImplementedError: input type 'object' not supported
>>
>>
>> ======================================================================
>> ERROR: test_rank2
>> (test_signaltools.TestLinearFilterDecimal)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
>> line 401, in test_rank2
>> ? ? y = lfilter(b, a, x, axis = 0)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
>> line 601, in lfilter
>> ? ? return sigtools._linear_filter(b, a, x, axis)
>> NotImplementedError: input type 'object' not supported
>>
>>
>> ======================================================================
>> ERROR: test_rank2_init_cond_a0
>> (test_signaltools.TestLinearFilterDecimal)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
>> line 435, in test_rank2_init_cond_a0
>> ? ? y, zf = lfilter(b, a, x, axis = 0, zi = np.ones((1,
>> 3)))
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
>> line 603, in lfilter
>> ? ? return sigtools._linear_filter(b, a, x, axis, zi)
>> NotImplementedError: input type 'object' not supported
>>
>>
>> ======================================================================
>> ERROR: test_rank2_init_cond_a1
>> (test_signaltools.TestLinearFilterDecimal)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
>> line 419, in test_rank2_init_cond_a1
>> ? ? y, zf = lfilter(b, a, x, axis = 1, zi = np.ones((4,
>> 1)))
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
>> line 603, in lfilter
>> ? ? return sigtools._linear_filter(b, a, x, axis, zi)
>> NotImplementedError: input type 'object' not supported
>>
>>
>> ======================================================================
>> ERROR: test_rank3
>> (test_signaltools.TestLinearFilterDecimal)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/tests/test_signaltools.py",
>> line 447, in test_rank3
>> ? ? y = lfilter(b, a, x)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/signal/signaltools.py",
>> line 601, in lfilter
>> ? ? return sigtools._linear_filter(b, a, x, axis)
>> NotImplementedError: input type 'object' not supported
>>
>>
>> ======================================================================
>> ERROR: test_mpmath.test_hyp2f1_real_some
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py",
>> line 146, in skipper_func
>> ? ? return f(*args, **kwargs)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_mpmath.py",
>> line 117, in test_hyp2f1_real_some
>> ? ? except (TypeError, mpmath.libhyper.NoConvergence):
>> AttributeError: 'module' object has no attribute
>> 'libhyper'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.bernoulli_gen
>> object at 0x2636ad0>, (0.29999999999999999,), array([0, 0,
>> 0, ..., 1, 0, 0]), 0.01, 'bernoulli chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.binom_gen
>> object at 0x26368d0>, (5, 0.40000000000000002), array([2,
>> 2, 2, ..., 4, 1, 3]), 0.01, 'binom chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.boltzmann_gen
>> object at 0x26ec590>, (1.3999999999999999, 19), array([0,
>> 0, 0, ..., 2, 0, 0]), 0.01, 'boltzmann chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.dlaplace_gen
>> object at 0x26eca50>, (0.80000000000000004,), array([ 0,
>> ?0, ?0, ..., ?4, -1, ?0]), 0.01, 'dlaplace chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.geom_gen
>> object at 0x2636dd0>, (0.5,), array([1, 1, 2, ..., 6, 1,
>> 2]), 0.01, 'geom chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.hypergeom_gen
>> object at 0x2636f50>, (30, 12, 6), array([1, 1, 4, ..., 3,
>> 2, 2]), 0.01, 'hypergeom chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.hypergeom_gen
>> object at 0x2636f50>, (21, 3, 12), array([2, 3, 2, ..., 2,
>> 2, 1]), 0.01, 'hypergeom chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.hypergeom_gen
>> object at 0x2636f50>, (21, 18, 11), array([ 9, ?8, ?9,
>> ..., 10, 10, 10]), 0.01, 'hypergeom chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.logser_gen
>> object at 0x26ec110>, (0.59999999999999998,), array([1, 1,
>> 1, ..., 1, 1, 4]), 0.01, 'logser chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.nbinom_gen
>> object at 0x2636c50>, (5, 0.5), array([0, 2, 6, ..., 3, 3,
>> 3]), 0.01, 'nbinom chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.nbinom_gen
>> object at 0x2636c50>, (0.40000000000000002,
>> 0.40000000000000002), array([0, 0, 0, ..., 0, 0, 0]),
>> 0.01, 'nbinom chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.planck_gen
>> object at 0x26ec410>, (0.51000000000000001,), array([1, 1,
>> 1, ..., 7, 0, 2]), 0.01, 'planck chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.poisson_gen
>> object at 0x26ec290>, (0.59999999999999998,), array([0, 0,
>> 0, ..., 1, 0, 0]), 0.01, 'poisson chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.randint_gen
>> object at 0x26ec710>, (7, 31), array([27, 10, 15, ..., 16,
>> ?9, 17]), 0.01, 'randint chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.skellam_gen
>> object at 0x26ecbd0>, (15, 8), array([ 4, ?6, 10, ..., ?5,
>> 14, 15]), 0.01, 'skellam chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> ERROR:
>> test_discrete_basic.test_discrete_basic(<scipy.stats.distributions.zipf_gen
>> object at 0x26ec8d0>, (4,), array([1, 1, 1, ..., 1, 1,
>> 1]), 0.01, 'zipf chisquare')
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_discrete_basic.py",
>> line 252, in check_discrete_chisquare
>> ? ? freq,hsupp = np.histogram(rvs,histsupp,new=True)
>> TypeError: histogram() got an unexpected keyword argument
>> 'new'
>>
>> ======================================================================
>> FAIL: test_lambertw.test_values
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_lambertw.py",
>> line 80, in test_values
>> ? ? FuncData(w, data, (0,1), 2, rtol=1e-10,
>> atol=1e-13).check()
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/testutils.py",
>> line 187, in check
>> ? ? assert False, "\n".join(msg)
>> AssertionError:
>> Max |adiff|: 1.77636e-15
>> Max |rdiff|: 2.06237e-16
>> Bad results for the following points (in output 0):
>> (-0.40000000000000002+0.40000000000000002j)
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0j =>
>> ? ? ? ? ? ? ? ? ? (nan+nan*j) !=
>> (-0.10396515323290657+0.61899273315171632j) ?(rdiff
>> ? ? ? ? ? ? ? ? ? ? ? ? ? 0.0)
>> (-0.44800000000000001+0.40000000000000002j)
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0j =>
>> ? ? ? ? ? ? ? ? ? (nan+nan*j) !=
>> (-0.11855133765652383+0.66570534313583418j) ?(rdiff
>> ? ? ? ? ? ? ? ? ? ? ? ? ? 0.0)
>> (-0.44800000000000001-0.40000000000000002j)
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0j =>
>> ? ? ? ? ? ? ? ? ? (nan+nan*j) !=
>> (-0.11855133765652383-0.66570534313583418j) ?(rdiff
>> ? ? ? ? ? ? ? ? ? ? ? ? ? 0.0)
>>
>> ======================================================================
>> FAIL: test_mpmath.test_hyp2f1_some_points_2
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py",
>> line 183, in runTest
>> ? ? self.test(*self.arg)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py",
>> line 146, in skipper_func
>> ? ? return f(*args, **kwargs)
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_mpmath.py",
>> line 106, in test_hyp2f1_some_points_2
>> ? ? FuncData(sc.hyp2f1, dataset, (0,1,2,3), 4,
>> rtol=1e-10).check()
>> ? File
>> "/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/testutils.py",
>> line 187, in check
>> ? ? assert False, "\n".join(msg)
>> AssertionError:
>> Max |adiff|: 1.48367e-22
>> Max |rdiff|: 0.989172
>> Bad results for the following points (in output 0):
>> ? ? ? ? ? ? ? ? ? ? ? ? ?112.0
>> ? ? ? ? ? ? 5.0999999999999996
>> ? ? ? ? ? -0.90000000000000002
>> ? ? ? ? ? -0.99999000000000005 =>
>> ? ? ? -1.6241361047970947e-24 !=
>> ? ? ? ?-1.499910195656332e-22 ?(rdiff
>> ? ? ? ? ? 0.98917177768708742)
>>
>> ----------------------------------------------------------------------
>> Ran 4332 tests in 76.236s
>>
>> FAILED (KNOWNFAIL=11, SKIP=17, errors=24, failures=2)
>> <nose.result.TextTestResult run=4332 errors=24 failures=2>
>>
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>


From d.l.goldsmith at gmail.com  Fri Feb 19 16:30:34 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Fri, 19 Feb 2010 13:30:34 -0800
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <1cd32cbb1002190624q75bca956oc1cc1d44a8821ebc@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
	<4B7DDC8A.5050400@silveregg.co.jp>
	<1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
	<c21c0b2f1002182345l44850424o375938c5c8e0e900@mail.gmail.com>
	<1cd32cbb1002190424p681f3111ic3718d8b5d260053@mail.gmail.com>
	<c21c0b2f1002190505g6c3114e2v9e111a7779e8cbb6@mail.gmail.com>
	<1cd32cbb1002190624q75bca956oc1cc1d44a8821ebc@mail.gmail.com>
Message-ID: <45d1ab481002191330g195bc72fuf6f5a7d3f6c4bed9@mail.gmail.com>

Hi, folks!  OK, at this point, can someone(s) please summarize precisely
where changes have been made as a result of this thread?  (I don't need to
know what the changes were, of course, 'cause I can get those from the
Wiki.)  Thanks!

DG

On Fri, Feb 19, 2010 at 6:24 AM, <josef.pktd at gmail.com> wrote:

> On Fri, Feb 19, 2010 at 8:05 AM, Tom Grydeland <tom.grydeland at gmail.com>
> wrote:
> >> On Fri, Feb 19, 2010 at 2:45 AM, Tom Grydeland <tom.grydeland at gmail.com>
> wrote:
> >>> Do you want me to fix the latex errors?
> >
> > On Fri, Feb 19, 2010 at 1:24 PM,  <josef.pktd at gmail.com> wrote:
> >> Yes, please, and make any adjustment that you find usefule.
> >
> > Okay, they are fixed, and I unified all the references cited also.
> >
> >> I didn't want to copy too much from the numpy docs since we can also
> >> just refer to it. For my reading an abridged explanation refering to
> >> the numpy docs, and a more complete collection of the extra parts that
> >> in scipy would be useful. But we haven't had much discussion yet on
> >> doc duplication between scipy and numpy (which is also relevant for
> >> the descriptions of the distributions in numpy.random and
> >> scipy.stats.distributions)
> >
> > I'm fine either way, but I think for a tutorial page for fftpack it
> > seems odd to define the DCT in that level of detail and then refer
> > elsewhere for the DFT definition.
>
> I don't have a strong opinion either.
> The numpy.fft module docstring is short enough (when rendered on the
> webpage) that most of it can be repeated. My initial thought was that
> fft in scipy could be explained more in terms of fft2 and fftn, with
> examples for those. (I just recently struggled with fftn and there is
> not much example code available.)
> I think, the definitions and formulas should be included for all
> transforms in any case.
>
> Josef
>
> >
> >> Josef
> >
> > --
> > Tom Grydeland
> >  <Tom.Grydeland@(gmail.com)>
> > _______________________________________________
> > SciPy-Dev mailing list
> > SciPy-Dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-dev
> >
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100219/6ca07385/attachment.html>

From d.l.goldsmith at gmail.com  Fri Feb 19 16:32:23 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Fri, 19 Feb 2010 13:32:23 -0800
Subject: [SciPy-dev] Latex and docstrings
In-Reply-To: <45d1ab481002191330g195bc72fuf6f5a7d3f6c4bed9@mail.gmail.com>
References: <4B7215FE.5030803@silveregg.co.jp>
	<c21c0b2f1002180058ved1c97fufc552a4380511435@mail.gmail.com>
	<1cd32cbb1002180722j7dfae867hc15975b32b050da7@mail.gmail.com>
	<4B7DDC8A.5050400@silveregg.co.jp>
	<1cd32cbb1002182210w5e6bb614kca47dc247beee080@mail.gmail.com>
	<c21c0b2f1002182345l44850424o375938c5c8e0e900@mail.gmail.com>
	<1cd32cbb1002190424p681f3111ic3718d8b5d260053@mail.gmail.com>
	<c21c0b2f1002190505g6c3114e2v9e111a7779e8cbb6@mail.gmail.com>
	<1cd32cbb1002190624q75bca956oc1cc1d44a8821ebc@mail.gmail.com>
	<45d1ab481002191330g195bc72fuf6f5a7d3f6c4bed9@mail.gmail.com>
Message-ID: <45d1ab481002191332n56400b34jfb83745e74875f1a@mail.gmail.com>

Never mind, I remembered I can get this from the Wiki also.  SFTN.

DG

On Fri, Feb 19, 2010 at 1:30 PM, David Goldsmith <d.l.goldsmith at gmail.com>wrote:

> Hi, folks!  OK, at this point, can someone(s) please summarize precisely
> where changes have been made as a result of this thread?  (I don't need to
> know what the changes were, of course, 'cause I can get those from the
> Wiki.)  Thanks!
>
> DG
>
>
> On Fri, Feb 19, 2010 at 6:24 AM, <josef.pktd at gmail.com> wrote:
>
>> On Fri, Feb 19, 2010 at 8:05 AM, Tom Grydeland <tom.grydeland at gmail.com>
>> wrote:
>> >> On Fri, Feb 19, 2010 at 2:45 AM, Tom Grydeland <
>> tom.grydeland at gmail.com> wrote:
>> >>> Do you want me to fix the latex errors?
>> >
>> > On Fri, Feb 19, 2010 at 1:24 PM,  <josef.pktd at gmail.com> wrote:
>> >> Yes, please, and make any adjustment that you find usefule.
>> >
>> > Okay, they are fixed, and I unified all the references cited also.
>> >
>> >> I didn't want to copy too much from the numpy docs since we can also
>> >> just refer to it. For my reading an abridged explanation refering to
>> >> the numpy docs, and a more complete collection of the extra parts that
>> >> in scipy would be useful. But we haven't had much discussion yet on
>> >> doc duplication between scipy and numpy (which is also relevant for
>> >> the descriptions of the distributions in numpy.random and
>> >> scipy.stats.distributions)
>> >
>> > I'm fine either way, but I think for a tutorial page for fftpack it
>> > seems odd to define the DCT in that level of detail and then refer
>> > elsewhere for the DFT definition.
>>
>> I don't have a strong opinion either.
>> The numpy.fft module docstring is short enough (when rendered on the
>> webpage) that most of it can be repeated. My initial thought was that
>> fft in scipy could be explained more in terms of fft2 and fftn, with
>> examples for those. (I just recently struggled with fftn and there is
>> not much example code available.)
>> I think, the definitions and formulas should be included for all
>> transforms in any case.
>>
>> Josef
>>
>> >
>> >> Josef
>> >
>> > --
>> > Tom Grydeland
>> >  <Tom.Grydeland@(gmail.com)>
>> > _______________________________________________
>> > SciPy-Dev mailing list
>> > SciPy-Dev at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-dev
>> >
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100219/cfe75e16/attachment.html>

From warren.weckesser at enthought.com  Sun Feb 21 12:28:09 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Sun, 21 Feb 2010 11:28:09 -0600
Subject: [SciPy-dev] 0.8 release date?
Message-ID: <4B816D29.3070704@enthought.com>

Is there currently a target date (even a tentative one) for the release 
of 0.8?

Warren


From josef.pktd at gmail.com  Sun Feb 21 12:37:33 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 21 Feb 2010 12:37:33 -0500
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <4B816D29.3070704@enthought.com>
References: <4B816D29.3070704@enthought.com>
Message-ID: <1cd32cbb1002210937n1c44760y8c80988ca263924d@mail.gmail.com>

On Sun, Feb 21, 2010 at 12:28 PM, Warren Weckesser
<warren.weckesser at enthought.com> wrote:
> Is there currently a target date (even a tentative one) for the release
> of 0.8?

And related, does scipy trunk stay numpy 1.4 compatible or will there
be a numpy 2.0 requirement?
scipy 0.8 will require >= numpy 1.4?

Josef

>
> Warren
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From xavier.gnata at gmail.com  Sun Feb 21 13:57:31 2010
From: xavier.gnata at gmail.com (Xavier Gnata)
Date: Sun, 21 Feb 2010 19:57:31 +0100
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <1cd32cbb1002210937n1c44760y8c80988ca263924d@mail.gmail.com>
References: <4B816D29.3070704@enthought.com>
	<1cd32cbb1002210937n1c44760y8c80988ca263924d@mail.gmail.com>
Message-ID: <4B81821B.2020401@gmail.com>

On 02/21/2010 06:37 PM, josef.pktd at gmail.com wrote:
> On Sun, Feb 21, 2010 at 12:28 PM, Warren Weckesser
> <warren.weckesser at enthought.com> wrote:
>   
>> Is there currently a target date (even a tentative one) for the release
>> of 0.8?
>>     
> And related, does scipy trunk stay numpy 1.4 compatible or will there
> be a numpy 2.0 requirement?
> scipy 0.8 will require >= numpy 1.4?
>
> Josef
>
>   
Great news from the numpy list:
"The test suite passes now on Pythons 2.4 - 3.1." with the current numpy svn trunk.
If there's a numpy 2.0 requirement for 0.8 ,are there plans to have a draft support of python3.X? 

Xavier


From david at silveregg.co.jp  Sun Feb 21 20:31:11 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Mon, 22 Feb 2010 10:31:11 +0900
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <4B816D29.3070704@enthought.com>
References: <4B816D29.3070704@enthought.com>
Message-ID: <4B81DE5F.305@silveregg.co.jp>

Warren Weckesser wrote:
> Is there currently a target date (even a tentative one) for the release 
> of 0.8?

Not that I know of, but I think it would be good to have one. I would 
also like to have a binary release for scipy which is compatible with 
whatever version will be used for numpy based on 1.4.x branch, something 
like:
	- Scipy 0.8.0 and Numpy 2.0.
	- Scipy 0.7.2 and Numpy 1.4.x

This only concerns binaries - at the source level, I think both numpy 
1.4.x and 2.0 can be used against scipy trunk (at least the scipy trunk 
as recent as last week works well with 1.4.x, that's what I am using for 
my own work).

The big question is whether we should release scipy 0.8.0 without caring 
about py3k compatibility at all, in which case we could aim at a release 
in April, and go into bug fixing mode only from now.

cheers,

David


From david at silveregg.co.jp  Sun Feb 21 20:34:35 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Mon, 22 Feb 2010 10:34:35 +0900
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <1cd32cbb1002210937n1c44760y8c80988ca263924d@mail.gmail.com>
References: <4B816D29.3070704@enthought.com>
	<1cd32cbb1002210937n1c44760y8c80988ca263924d@mail.gmail.com>
Message-ID: <4B81DF2B.7040008@silveregg.co.jp>

josef.pktd at gmail.com wrote:
> On Sun, Feb 21, 2010 at 12:28 PM, Warren Weckesser
> <warren.weckesser at enthought.com> wrote:
>> Is there currently a target date (even a tentative one) for the release
>> of 0.8?
> 
> And related, does scipy trunk stay numpy 1.4 compatible or will there
> be a numpy 2.0 requirement?

Scipy trunk requires at least the changes Pauli, Chuck and me have been 
working on for improving scipy.special.

At the source level, numpy 2.0 is compatible with numpy 1.4.0 AFAIK, and 
scipy does not use datetime, so the scipy trunk should not require numpy 
2.0. IOW, the story would be:
	- Scipy 0.7.2 (created from the 0.7.x svn branch): compatible with 
numpy >= 1.3.0 (including 2.0)
	- Scipy 0.8.0 (from trunk): compatible with numpy >= 1.4.0.

cheers,

David


From charlesr.harris at gmail.com  Sun Feb 21 20:59:55 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 21 Feb 2010 18:59:55 -0700
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <4B81DF2B.7040008@silveregg.co.jp>
References: <4B816D29.3070704@enthought.com>
	<1cd32cbb1002210937n1c44760y8c80988ca263924d@mail.gmail.com>
	<4B81DF2B.7040008@silveregg.co.jp>
Message-ID: <e06186141002211759j1aca497aucc0c765728319f8c@mail.gmail.com>

On Sun, Feb 21, 2010 at 6:34 PM, David Cournapeau <david at silveregg.co.jp>wrote:

> josef.pktd at gmail.com wrote:
> > On Sun, Feb 21, 2010 at 12:28 PM, Warren Weckesser
> > <warren.weckesser at enthought.com> wrote:
> >> Is there currently a target date (even a tentative one) for the release
> >> of 0.8?
> >
> > And related, does scipy trunk stay numpy 1.4 compatible or will there
> > be a numpy 2.0 requirement?
>
> Scipy trunk requires at least the changes Pauli, Chuck and me have been
> working on for improving scipy.special.
>
>
I haven't done much, if anything, on scipy.special. There are two things I
could contribute:

1) Fix the orthogonal polynomials to use the new Chebyshev feature in numpy
>=1.4

2) Add the generalized Remez (hermitean complex) algorithm. I'm not sure
where it should go, maybe in the optimize category? The other possibility
might be interpolate. I wish we had a 'fitting' category for such things.

I'm going to be darn busy from March through April, however.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100221/31cf9f82/attachment.html>

From charlesr.harris at gmail.com  Sun Feb 21 21:05:01 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 21 Feb 2010 19:05:01 -0700
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <e06186141002211759j1aca497aucc0c765728319f8c@mail.gmail.com>
References: <4B816D29.3070704@enthought.com>
	<1cd32cbb1002210937n1c44760y8c80988ca263924d@mail.gmail.com>
	<4B81DF2B.7040008@silveregg.co.jp>
	<e06186141002211759j1aca497aucc0c765728319f8c@mail.gmail.com>
Message-ID: <e06186141002211805o6352ba34y4a7b4dd28d52f02f@mail.gmail.com>

On Sun, Feb 21, 2010 at 6:59 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Sun, Feb 21, 2010 at 6:34 PM, David Cournapeau <david at silveregg.co.jp>wrote:
>
>> josef.pktd at gmail.com wrote:
>> > On Sun, Feb 21, 2010 at 12:28 PM, Warren Weckesser
>> > <warren.weckesser at enthought.com> wrote:
>> >> Is there currently a target date (even a tentative one) for the release
>> >> of 0.8?
>> >
>> > And related, does scipy trunk stay numpy 1.4 compatible or will there
>> > be a numpy 2.0 requirement?
>>
>> Scipy trunk requires at least the changes Pauli, Chuck and me have been
>> working on for improving scipy.special.
>>
>>
> I haven't done much, if anything, on scipy.special. There are two things I
> could contribute:
>
> 1) Fix the orthogonal polynomials to use the new Chebyshev feature in numpy
> >=1.4
>
> 2) Add the generalized Remez (hermitean complex) algorithm. I'm not sure
> where it should go, maybe in the optimize category? The other possibility
> might be interpolate. I wish we had a 'fitting' category for such things.
>
> I'm going to be darn busy from March through April, however.
>
>
PS, the ball tree code could use a looksie.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100221/27d265ed/attachment.html>

From loredo at astro.cornell.edu  Sun Feb 21 21:35:01 2010
From: loredo at astro.cornell.edu (Tom Loredo)
Date: Sun, 21 Feb 2010 21:35:01 -0500
Subject: [SciPy-dev] Segfault on OS X 10.6.2/Py-2.6.4 with SciPy FFT tests
Message-ID: <1266806101.4b81ed55d759b@astrosun2.astro.cornell.edu>


Hi folks-

I'm trying to install NumPy & SciPy on 10.6.2 (Snow Leopard) with
64-bit Python 2.6.4 (a new framework build, not Apple's 2.6.1).  I've
built FFTW and AMD/UMFPACK and they pass their respective tests.
NumPy-1.4.0 builds fine and passes all tests (except for 3 Knowns
and 1 Skip).  I've verified it's finding FFTW & AMD/UMF.  
SciPy-0.7.1 builds fine and imports without issue, but quickly 
into the tests has several failures & then a segfault crash.  I tried
installing a recent SVN SciPy (0.8.0.dev6220) but it crashes with
a similar segfault.  In both cases it's during the FFT tests.

I removed both packages from site-packages and re-installed, this
time having removed FFTW (verifying no fftw lib was found), and
again there is a crash in the FFT tests, presumably pointing to
a problem with numpy/scipy rather than FFTW.

I've copied below the tail end of the output from scipy.test(verbose=10)
from both SciPy versions (without FFTW).  There's an indication that
an object is being modified after its memory is freed.  I haven't
done malloc-level debugging so I'm not sure how to help at this point.
If anyone can have a look or provide more explicit pointers, I'd be
grateful!

Thanks,
Tom

~~~~~~~~~~~~
scipy-0.7.1:

test_definition (test_basic.TestFft) ... ok
test_djbfft (test_basic.TestFft) ... ok
test_n_argument_real (test_basic.TestFft) ... ok
test_axes_argument (test_basic.TestFftn) ... ok
test_definition (test_basic.TestFftn) ... ok
test_shape_argument (test_basic.TestFftn) ... ok
test_shape_argument_more (test_basic.TestFftn) ... ok
test_shape_axes_argument (test_basic.TestFftn) ... ok
test_shape_axes_argument2 (test_basic.TestFftn) ... ok
test_definition (test_basic.TestIfft) ... FAIL
test_djbfft (test_basic.TestIfft) ... FAIL
test_random_complex (test_basic.TestIfft) ... FAIL
test_random_real (test_basic.TestIfft) ... Python(15804,0x7fff7065dbe0) malloc: *** error for object 0x105376448: 
incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Abort trap

~~~~~~~~~~~~
scipy-0.8.0.dev6220

test_definition (test_basic.TestDoubleFFT) ... ok
test_djbfft (test_basic.TestDoubleFFT) ... ok
test_n_argument_real (test_basic.TestDoubleFFT) ... ok
test_definition (test_basic.TestDoubleIFFT) ... Python(29515,0x7fff7065dbe0) malloc: *** error for object 0x1012fbf28: 
incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Abort trap


-------------------------------------------------
This mail sent through IMP: http://horde.org/imp/


From josef.pktd at gmail.com  Sun Feb 21 21:54:33 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 21 Feb 2010 21:54:33 -0500
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <4B81DE5F.305@silveregg.co.jp>
References: <4B816D29.3070704@enthought.com> <4B81DE5F.305@silveregg.co.jp>
Message-ID: <1cd32cbb1002211854u393cbd88mcad4418b34fafe6b@mail.gmail.com>

On Sun, Feb 21, 2010 at 8:31 PM, David Cournapeau <david at silveregg.co.jp> wrote:
> Warren Weckesser wrote:
>> Is there currently a target date (even a tentative one) for the release
>> of 0.8?
>
> Not that I know of, but I think it would be good to have one. I would
> also like to have a binary release for scipy which is compatible with
> whatever version will be used for numpy based on 1.4.x branch, something
> like:
> ? ? ? ?- Scipy 0.8.0 and Numpy 2.0.
> ? ? ? ?- Scipy 0.7.2 and Numpy 1.4.x
>
> This only concerns binaries - at the source level, I think both numpy
> 1.4.x and 2.0 can be used against scipy trunk (at least the scipy trunk
> as recent as last week works well with 1.4.x, that's what I am using for
> my own work).
>
> The big question is whether we should release scipy 0.8.0 without caring
> about py3k compatibility at all, in which case we could aim at a release
> in April, and go into bug fixing mode only from now.

Improving py3k compatibility would then require numpy 2.0 instead of
numpy 1.4, does it?

In terms of compatibilities, for me the main question is how soon
other packages, e.g. matplotlib, h5py, tables, ... , will be able to
provide binary distributions for numpy 2.0. Is it difficult for those
packages to adjust, or does it basically only require recompiling?
If there is a time lag, I would prefer if we can get a numpy 1.4
compatible scipy 0.8 release.

In scipy.stats, I have two things that I really want in scipy 0.8.
(spearmanr rewrite and removing pdf_approx), the rest can wait if I
don't find the time (and no one else does either).

Josef

>
> cheers,
>
> David
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From arkapravobhaumik at gmail.com  Mon Feb 22 09:09:46 2010
From: arkapravobhaumik at gmail.com (Arkapravo Bhaumik)
Date: Mon, 22 Feb 2010 19:39:46 +0530
Subject: [SciPy-dev] Scipy meets pychecker
Message-ID: <ba4564351002220609t4d84e209q5deb3424187591dd@mail.gmail.com>

Dear Friends

*Has anyone tried to 'test' scipy using pychecker ?* What results have you
got ?

Please reply, preferably with examples etc.

Much appreciated

Regards

Arkapravo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100222/31580bf6/attachment.html>

From rmay31 at gmail.com  Mon Feb 22 09:41:15 2010
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 22 Feb 2010 08:41:15 -0600
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <e06186141002211759j1aca497aucc0c765728319f8c@mail.gmail.com>
References: <4B816D29.3070704@enthought.com>
	<1cd32cbb1002210937n1c44760y8c80988ca263924d@mail.gmail.com>
	<4B81DF2B.7040008@silveregg.co.jp>
	<e06186141002211759j1aca497aucc0c765728319f8c@mail.gmail.com>
Message-ID: <c7b95a021002220641v7c201f5ay688f777ef7508c37@mail.gmail.com>

On Sun, Feb 21, 2010 at 7:59 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Sun, Feb 21, 2010 at 6:34 PM, David Cournapeau <david at silveregg.co.jp>
> wrote:
>>
>> josef.pktd at gmail.com wrote:
>> > On Sun, Feb 21, 2010 at 12:28 PM, Warren Weckesser
>> > <warren.weckesser at enthought.com> wrote:
>> >> Is there currently a target date (even a tentative one) for the release
>> >> of 0.8?
>> >
>> > And related, does scipy trunk stay numpy 1.4 compatible or will there
>> > be a numpy 2.0 requirement?
>>
>> Scipy trunk requires at least the changes Pauli, Chuck and me have been
>> working on for improving scipy.special.
>>
>
> I haven't done much, if anything, on scipy.special. There are two things I
> could contribute:
>
> 1) Fix the orthogonal polynomials to use the new Chebyshev feature in numpy
>>=1.4
>
> 2) Add the generalized Remez (hermitean complex) algorithm. I'm not sure
> where it should go, maybe in the optimize category? The other possibility
> might be interpolate. I wish we had a 'fitting' category for such things.

There's also Continuous Wavelet Transform code that's been sitting
here for almost a year:

http://projects.scipy.org/scipy/ticket/922

which points to:

http://github.com/lesserwhirls/scipy-cwt

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From mohitiiit at gmail.com  Mon Feb 22 13:46:59 2010
From: mohitiiit at gmail.com (Mohit Goyal)
Date: Tue, 23 Feb 2010 00:16:59 +0530
Subject: [SciPy-dev] Reg: google summer of code 2010
Message-ID: <9e78df6d1002221046i18b2ac2fu3415444e6bb0a851@mail.gmail.com>

I am a masters student doing research in data engineering. I know python and
have done projects on data mining and web mining past 3 years.
I want to contribute to scipy via gsoc 2010. I am  interested in working on
data related projects in scipy like improving  data sources and integrate it
into all the numpy/scipy , adding data sets and examples to scipy, data
analysis etc.

Somebody suggest how to proceed or info about the concerned persons.

Thank You.


-- 
The Game is Not Over Because i haven't Won Yet.
-Mohit Goyal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100223/c46f2016/attachment.html>

From warren.weckesser at enthought.com  Mon Feb 22 15:39:37 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Mon, 22 Feb 2010 14:39:37 -0600
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <4B81DE5F.305@silveregg.co.jp>
References: <4B816D29.3070704@enthought.com> <4B81DE5F.305@silveregg.co.jp>
Message-ID: <4B82EB89.4060605@enthought.com>

David Cournapeau wrote:
> Warren Weckesser wrote:
>   
>> Is there currently a target date (even a tentative one) for the release 
>> of 0.8?
>>     
>
> Not that I know of, but I think it would be good to have one.

Definitely.

>  I would 
> also like to have a binary release for scipy which is compatible with 
> whatever version will be used for numpy based on 1.4.x branch, something 
> like:
> 	- Scipy 0.8.0 and Numpy 2.0.
> 	- Scipy 0.7.2 and Numpy 1.4.x
>
> This only concerns binaries - at the source level, I think both numpy 
> 1.4.x and 2.0 can be used against scipy trunk (at least the scipy trunk 
> as recent as last week works well with 1.4.x, that's what I am using for 
> my own work).
>
> The big question is whether we should release scipy 0.8.0 without caring 
> about py3k compatibility at all, in which case we could aim at a release 
> in April, and go into bug fixing mode only from now.
>
>   

"Bug fixing only from now" might be a bit short notice for people.  So 
far, I've contributed just a few patches, but I plan on doing a lot more 
work over the next month or so--mostly in scipy.signal--and some of this 
will involve adding new functions and enhancing old ones.

Here's a suggestion that provides some more time for enhancements: 
release 0.8 in time for the SciPy 2010 conference.  The conference 
begins June 28.   Suppose we work backwards from there.  How much in 
advance of the conference should the final release be?  Let's say four 
weeks, so we aim for a June 1 release.  If the schedule on the 
"Milestone 0.8.0" web page 
(http://projects.scipy.org/scipy/milestone/0.8.0) is still reasonable, 
then the 0.8 branch freeze should be mid-May.  That leaves about 12 
weeks of development time.

Considering that the "MakingReleases" wiki page says there will be a 
stable release every three months, I don't think anyone will complain 
that this is not enough time. :)  However, some people may want the 
release sooner.

What do you think?


Warren



From pav at iki.fi  Mon Feb 22 16:18:10 2010
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 22 Feb 2010 23:18:10 +0200
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <4B81DE5F.305@silveregg.co.jp>
References: <4B816D29.3070704@enthought.com>  <4B81DE5F.305@silveregg.co.jp>
Message-ID: <1266873490.5645.46.camel@idol>

ma, 2010-02-22 kello 10:31 +0900, David Cournapeau kirjoitti:
> Warren Weckesser wrote:
> > Is there currently a target date (even a tentative one) for the release 
> > of 0.8?
> 
> Not that I know of, but I think it would be good to have one. I would 
> also like to have a binary release for scipy which is compatible with 
> whatever version will be used for numpy based on 1.4.x branch, something 
> like:
> 	- Scipy 0.8.0 and Numpy 2.0.
> 	- Scipy 0.7.2 and Numpy 1.4.x
> 
> This only concerns binaries - at the source level, I think both numpy 
> 1.4.x and 2.0 can be used against scipy trunk (at least the scipy trunk 
> as recent as last week works well with 1.4.x, that's what I am using for 
> my own work).

Do we want even to release a separate 0.7.x? I'd say this is not very
well spent work, unless (i) 0.8 will still take a long time, and (ii)
there have been substantial bug fixes.

Of course, it might be confusing to users to have separate 0.8.x
binaries for Numpy 1.4 and 2.0. Still, I'd rather add a suffix to the
version than have the minor number be different since that's also
confusing...

> The big question is whether we should release scipy 0.8.0 without caring 
> about py3k compatibility at all, in which case we could aim at a release 
> in April, and go into bug fixing mode only from now.

I don't have an clear idea how much work it will take to port Scipy to
Py3. I'm kind of hoping that it won't be too much: Scipy is mostly just
using the Numpy APIs which haven't really changed. Also, apart from
scipy.io, it mostly does not mess with strings too much.

One issue is also whether f2py works at all on Py3. All of its 0 unit
tests pass in Numpy SVN on Py3, so YMMV :)

	Pauli





From dwf at cs.toronto.edu  Mon Feb 22 17:03:52 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Mon, 22 Feb 2010 17:03:52 -0500
Subject: [SciPy-dev] Reg: google summer of code 2010
In-Reply-To: <9e78df6d1002221046i18b2ac2fu3415444e6bb0a851@mail.gmail.com>
References: <9e78df6d1002221046i18b2ac2fu3415444e6bb0a851@mail.gmail.com>
Message-ID: <044343FA-45ED-4996-BCEA-07FEE7D56BD0@cs.toronto.edu>

On 22-Feb-10, at 1:46 PM, Mohit Goyal wrote:

> Somebody suggest how to proceed or info about the concerned persons.

There were several replies to your last post, if you'd like to  
continue the conversation there.

David


From cournape at gmail.com  Mon Feb 22 17:48:28 2010
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 23 Feb 2010 07:48:28 +0900
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <1266873490.5645.46.camel@idol>
References: <4B816D29.3070704@enthought.com> <4B81DE5F.305@silveregg.co.jp>
	<1266873490.5645.46.camel@idol>
Message-ID: <5b8d13221002221448p22571c1chb21a0470edac2ab9@mail.gmail.com>

On Tue, Feb 23, 2010 at 6:18 AM, Pauli Virtanen <pav at iki.fi> wrote:
> ma, 2010-02-22 kello 10:31 +0900, David Cournapeau kirjoitti:
>> Warren Weckesser wrote:
>> > Is there currently a target date (even a tentative one) for the release
>> > of 0.8?
>>
>> Not that I know of, but I think it would be good to have one. I would
>> also like to have a binary release for scipy which is compatible with
>> whatever version will be used for numpy based on 1.4.x branch, something
>> like:
>> ? ? ? - Scipy 0.8.0 and Numpy 2.0.
>> ? ? ? - Scipy 0.7.2 and Numpy 1.4.x
>>
>> This only concerns binaries - at the source level, I think both numpy
>> 1.4.x and 2.0 can be used against scipy trunk (at least the scipy trunk
>> as recent as last week works well with 1.4.x, that's what I am using for
>> my own work).
>
> Do we want even to release a separate 0.7.x? I'd say this is not very
> well spent work, unless (i) 0.8 will still take a long time, and (ii)
> there have been substantial bug fixes.

I think a 0.7.2 is worthwhile for two reasons:
  - it can work with numpy 1.3.0 as well
  - I think having two scipy binaries at the same version for two
different numpy is too confusing. My experience is that many people
will not look into any information, and just say it does not work.
Versions are easier to explain.

>
> One issue is also whether f2py works at all on Py3. All of its 0 unit
> tests pass in Numpy SVN on Py3, so YMMV :)

Well, there is one way to know for sure: scipy could be considered as
a test suite for f2py :)

David


From guyer at nist.gov  Tue Feb 23 09:36:36 2010
From: guyer at nist.gov (Jonathan Guyer)
Date: Tue, 23 Feb 2010 09:36:36 -0500
Subject: [SciPy-dev] 0.8 release date?
In-Reply-To: <5b8d13221002221448p22571c1chb21a0470edac2ab9@mail.gmail.com>
References: <4B816D29.3070704@enthought.com> <4B81DE5F.305@silveregg.co.jp>
	<1266873490.5645.46.camel@idol>
	<5b8d13221002221448p22571c1chb21a0470edac2ab9@mail.gmail.com>
Message-ID: <961E3E70-C8B0-4E28-A0C4-C36FE92EF96B@nist.gov>


On Feb 22, 2010, at 5:48 PM, David Cournapeau wrote:

>  - I think having two scipy binaries at the same version for two
> different numpy is too confusing. My experience is that many people
> will not look into any information, and just say it does not work.
> Versions are easier to explain.

+1



From stefan at sun.ac.za  Tue Feb 23 12:24:08 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 23 Feb 2010 19:24:08 +0200
Subject: [SciPy-dev] SciPy2010 Call for Papers
Message-ID: <9457e7c81002230924g1d5be5bet36fe0aee6aa03886@mail.gmail.com>

==========================
SciPy 2010 Call for Papers
==========================

SciPy 2010, the 9th Python in Science Conference, will be held from
June 28th - July 3rd, 2010 in Austin, Texas.

At this conference, novel applications and breakthroughs made in the
pursuit of science using Python are presented.  Attended by leading
figures from both academia and industry, it is an excellent
opportunity to experience the cutting edge of scientific software
development.

The conference is preceded by two days of paid tutorials, during which
community experts provide training on several scientific Python
packages.

We invite you to take part by submitting a talk abstract on the
conference website at:

http://conference.scipy.org

Talk/Paper Submission
=====================

We solicit talks and accompanying papers (either formal academic or
magazine-style articles) that discuss topics regarding scientific
computing using Python, including applications, teaching, development
and research.  Papers are included in the peer-reviewed conference
proceedings, published online.

Please note that submissions primarily aimed at the promotion of a
commercial product or service will not be considered.

Important dates for authors include:

 * 11 April: Talk abstracts due
 * 20 April: Notification of acceptance
 * 13 June: Papers due
 * 15 August: Publication of proceedings

Further detail will be made available on http://conference.scipy.org

Conference Dates
================

 * Friday, 10 May: Early registration ends
 * Monday-Tuesday, 28-29 June: Tutorials
 * Wednesday-Thursday, June 30-July 1: Conference
 * Friday-Saturday, July 2-3: Coding Sprints

Executive Committee
===================

 * Conference: Jarrod Millman & Eric Jones
 * Program: Stefan van der Walt & Ondrej Certik
 * Student Sponsorship: Travis Oliphant

For more information on Python, visit http://www.python.org.


From oliphant at enthought.com  Tue Feb 23 18:46:02 2010
From: oliphant at enthought.com (Travis Oliphant)
Date: Tue, 23 Feb 2010 17:46:02 -0600
Subject: [SciPy-dev] Fwd: [Info] Fwd: Reg: gsoc 2010
References: <83ae36ae1002231543va73547ej68503e90226c4f0c@mail.gmail.com>
Message-ID: <C42D9DC5-B0E5-42CF-B71A-B8321D46FF7C@enthought.com>

Here is a message we received.  Who is handling GSOC mentoring this  
year?

-Travis


Begin forwarded message:

> From: Aaron River <ariver at enthought.com>
> Date: February 23, 2010 5:43:45 PM CST
> To: info at enthought.com
> Subject: [Info] Fwd: Reg: gsoc 2010
> Reply-To: info at enthought.com
>
> -------- Original Message --------
> Subject: 	Reg: gsoc 2010
> Date: 	Mon, 22 Feb 2010 23:56:17 +0530
> From: 	Mohit Goyal <mohitiiit at gmail.com>
> To: 	scipy-dev-owner at scipy.org
>
> I am a masters student doing research in data engineering. I know  
> python
> and have done projects on data mining and web mining.
> I want to contribute to scipy via gsoc 2010. I am  interested in  
> working
> on data related projects in scipy like improving  data sources and
> integrate it into all the numpy/scipy , adding data sets and  
> examples to
> scipy, data analysis etc.  Please tell me how to proceed.
>
> Thank You.
>
> -- 
> The Game is Not Over Because i haven't Won Yet.
> -Mohit Goyal
> _______________________________________________
> Info mailing list
> Info at enthought.com
> https://mail.enthought.com/mailman/listinfo/info

--
Travis Oliphant
Enthought Inc.
1-512-536-1057
http://www.enthought.com
oliphant at enthought.com





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100223/79b31d6b/attachment.html>

From bouloumag at gmail.com  Tue Feb 23 20:26:02 2010
From: bouloumag at gmail.com (Darcoux Christine)
Date: Tue, 23 Feb 2010 20:26:02 -0500
Subject: [SciPy-dev] implement reorthogonalization method in gmres
Message-ID: <477ac73a1002231726h18b1rcd5f884e3c0cc036@mail.gmail.com>

It seems that the vectors produced by the Gram-Schmidt process may not be
orthogonal (probably due to the presence of roundoff errors or when the
matrix is ill conditionned). I would suggest to add an option to test for
loss of orthogonality and reorthogonalize if needed. See
http://www4.ncsu.edu/~ctk/newton/SOLVERS/nsoli.m for an example.

regards,

Christine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100223/fb26a0a2/attachment.html>

From pgmdevlist at gmail.com  Tue Feb 23 22:33:26 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 23 Feb 2010 22:33:26 -0500
Subject: [SciPy-dev] Fwd: [Info] Fwd: Reg: gsoc 2010
In-Reply-To: <C42D9DC5-B0E5-42CF-B71A-B8321D46FF7C@enthought.com>
References: <83ae36ae1002231543va73547ej68503e90226c4f0c@mail.gmail.com>
	<C42D9DC5-B0E5-42CF-B71A-B8321D46FF7C@enthought.com>
Message-ID: <13001F62-CB0D-4173-9C1B-26AB2FAC7F2D@gmail.com>

On Feb 23, 2010, at 6:46 PM, Travis Oliphant wrote:
> 
> Here is a message we received.  Who is handling GSOC mentoring this year?   

Wow, that's some heavy carpet-emailing. The list already received a couple of emails from him. Interestingly, the second was just a copy of the first one we'd already answered... 

From dwf at cs.toronto.edu  Tue Feb 23 23:41:42 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 23 Feb 2010 23:41:42 -0500
Subject: [SciPy-dev] Fwd: [Info] Fwd: Reg: gsoc 2010
In-Reply-To: <13001F62-CB0D-4173-9C1B-26AB2FAC7F2D@gmail.com>
References: <83ae36ae1002231543va73547ej68503e90226c4f0c@mail.gmail.com>
	<C42D9DC5-B0E5-42CF-B71A-B8321D46FF7C@enthought.com>
	<13001F62-CB0D-4173-9C1B-26AB2FAC7F2D@gmail.com>
Message-ID: <A2F370A7-B425-4858-9DCF-31196D5C1EE1@cs.toronto.edu>

On 23-Feb-10, at 10:33 PM, Pierre GM wrote:

> Wow, that's some heavy carpet-emailing. The list already received a  
> couple of emails from him. Interestingly, the second was just a copy  
> of the first one we'd already answered...

I wonder if he realizes that replies will go to the list and he has to  
stay subscribed to get them?

David


From jsseabold at gmail.com  Tue Feb 23 23:43:14 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 23 Feb 2010 23:43:14 -0500
Subject: [SciPy-dev] Fwd: [Info] Fwd: Reg: gsoc 2010
In-Reply-To: <A2F370A7-B425-4858-9DCF-31196D5C1EE1@cs.toronto.edu>
References: <83ae36ae1002231543va73547ej68503e90226c4f0c@mail.gmail.com> 
	<C42D9DC5-B0E5-42CF-B71A-B8321D46FF7C@enthought.com>
	<13001F62-CB0D-4173-9C1B-26AB2FAC7F2D@gmail.com> 
	<A2F370A7-B425-4858-9DCF-31196D5C1EE1@cs.toronto.edu>
Message-ID: <c048da1c1002232043u7ea358fbn9f486e07131a2600@mail.gmail.com>

On Tue, Feb 23, 2010 at 11:41 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> On 23-Feb-10, at 10:33 PM, Pierre GM wrote:
>
>> Wow, that's some heavy carpet-emailing. The list already received a
>> couple of emails from him. Interestingly, the second was just a copy
>> of the first one we'd already answered...
>
> I wonder if he realizes that replies will go to the list and he has to
> stay subscribed to get them?
>

This was my thought as well.  Do you have to be subscribed to post?

Skipper


From josef.pktd at gmail.com  Tue Feb 23 23:53:20 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 23 Feb 2010 23:53:20 -0500
Subject: [SciPy-dev] Fwd: [Info] Fwd: Reg: gsoc 2010
In-Reply-To: <c048da1c1002232043u7ea358fbn9f486e07131a2600@mail.gmail.com>
References: <83ae36ae1002231543va73547ej68503e90226c4f0c@mail.gmail.com>
	<C42D9DC5-B0E5-42CF-B71A-B8321D46FF7C@enthought.com>
	<13001F62-CB0D-4173-9C1B-26AB2FAC7F2D@gmail.com>
	<A2F370A7-B425-4858-9DCF-31196D5C1EE1@cs.toronto.edu>
	<c048da1c1002232043u7ea358fbn9f486e07131a2600@mail.gmail.com>
Message-ID: <1cd32cbb1002232053g2cb6ce93k960dacf99beeee5@mail.gmail.com>

On Tue, Feb 23, 2010 at 11:43 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Tue, Feb 23, 2010 at 11:41 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>> On 23-Feb-10, at 10:33 PM, Pierre GM wrote:
>>
>>> Wow, that's some heavy carpet-emailing. The list already received a
>>> couple of emails from him. Interestingly, the second was just a copy
>>> of the first one we'd already answered...
>>
>> I wonder if he realizes that replies will go to the list and he has to
>> stay subscribed to get them?
>>
>
> This was my thought as well. ?Do you have to be subscribed to post?

No, only signing up is required, subscription is not necessary to mail
to the list from other places, e.g. google groups interface or ...
But in these mailing list interfaces it's also possible/obvious to see
the messages.

Josef

>
> Skipper
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From pav+sp at iki.fi  Wed Feb 24 07:07:55 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Wed, 24 Feb 2010 12:07:55 +0000 (UTC)
Subject: [SciPy-dev] implement reorthogonalization method in gmres
References: <477ac73a1002231726h18b1rcd5f884e3c0cc036@mail.gmail.com>
Message-ID: <hm34qr$ndi$1@dough.gmane.org>

Tue, 23 Feb 2010 20:26:02 -0500, Darcoux Christine wrote:
> It seems that the vectors produced by the Gram-Schmidt process may not
> be orthogonal (probably due to the presence of roundoff errors or when
> the matrix is ill conditionned). I would suggest to add an option to
> test for loss of orthogonality and reorthogonalize if needed. See
> http://www4.ncsu.edu/~ctk/newton/SOLVERS/nsoli.m for an example.

>From the literature, it appears that orthogonality of the Krylov vectors 
is not very important for GMRES convergence, see e.g.

	A. Greenbaum, M. Rozlo?n?k, Z. Strako?
	BIT Numerical Mathematics, 37, 706 (1997).
	http://www.springerlink.com/content/277614q38kj6q376/

The GMRES implementations in Scipy use the modified Gram-Schmid 
procedure, so they should be OK in this respect.

Do you have some evidence suggesting that reorthogonalization is a 
significant improvement for some problems?  I'll accept patches providing 
such a flag in any case, though.

Best regards,
Pauli



From vanforeest at gmail.com  Wed Feb 24 17:43:44 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Wed, 24 Feb 2010 23:43:44 +0100
Subject: [SciPy-dev] implement reorthogonalization method in gmres
In-Reply-To: <hm34qr$ndi$1@dough.gmane.org>
References: <477ac73a1002231726h18b1rcd5f884e3c0cc036@mail.gmail.com>
	<hm34qr$ndi$1@dough.gmane.org>
Message-ID: <fa510ff81002241443w5a1a027fi9160b1db022db104@mail.gmail.com>

Hi,

I am not at all a specialist on this, but I recall that Numerical
Recipes states that the Gram-Schmidt process gives terrible results,
and should be avoided nearly always. Instead, singular value
decomposition should be used. To avoid the problem below, would is be
an idea to replace the GM algo by SVD?

NIcky

On 24 February 2010 13:07, Pauli Virtanen <pav+sp at iki.fi> wrote:
> Tue, 23 Feb 2010 20:26:02 -0500, Darcoux Christine wrote:
>> It seems that the vectors produced by the Gram-Schmidt process may not
>> be orthogonal (probably due to the presence of roundoff errors or when
>> the matrix is ill conditionned). I would suggest to add an option to
>> test for loss of orthogonality and reorthogonalize if needed. See
>> http://www4.ncsu.edu/~ctk/newton/SOLVERS/nsoli.m for an example.
>
> From the literature, it appears that orthogonality of the Krylov vectors
> is not very important for GMRES convergence, see e.g.
>
> ? ? ? ?A. Greenbaum, M. Rozlo?n?k, Z. Strako?
> ? ? ? ?BIT Numerical Mathematics, 37, 706 (1997).
> ? ? ? ?http://www.springerlink.com/content/277614q38kj6q376/
>
> The GMRES implementations in Scipy use the modified Gram-Schmid
> procedure, so they should be OK in this respect.
>
> Do you have some evidence suggesting that reorthogonalization is a
> significant improvement for some problems? ?I'll accept patches providing
> such a flag in any case, though.
>
> Best regards,
> Pauli
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From pav at iki.fi  Wed Feb 24 19:26:40 2010
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 25 Feb 2010 02:26:40 +0200
Subject: [SciPy-dev] implement reorthogonalization method in gmres
In-Reply-To: <fa510ff81002241443w5a1a027fi9160b1db022db104@mail.gmail.com>
References: <477ac73a1002231726h18b1rcd5f884e3c0cc036@mail.gmail.com>
	<hm34qr$ndi$1@dough.gmane.org>
	<fa510ff81002241443w5a1a027fi9160b1db022db104@mail.gmail.com>
Message-ID: <1267057599.24576.0.camel@idol>

ke, 2010-02-24 kello 23:43 +0100, nicky van foreest kirjoitti:
> I am not at all a specialist on this, but I recall that Numerical
> Recipes states that the Gram-Schmidt process gives terrible results,
> and should be avoided nearly always.

There's a balance here: more accurate methods tend to require more work,
and the orthogonalization is probably the most expensive part of GMRES.
So one should use a method that is good enough.

> Instead, singular value
> decomposition should be used. To avoid the problem below, would is be
> an idea to replace the GM algo by SVD?

One could also QR factorize using Householder transforms, which should
be more accurate than GS, and is probably less expensive than SVD; or
reorthogonalize.

But as far as I understand, this would not really improve the
performance of GMRES: as shown in the article I linked to (and also in
others), the quality of the orthogonalization is not expected to matter
for GMRES -- modified Gram-Schmid is good enough. The point is that
breakdown of orthogonality happens simultaneously with the solution
reaching its maximal accuracy, so it will not essentially affect
convergence or accuracy. I'd like to have a reference or a test case at
hand stating otherwise before changing things here.

As far as I understand, however, the situation with orthogonality is
different if you want to solve eigenvalue problems with Arnoldi stuff.
But that's in the domain of ARPACK, and I would trust without looking
that the ARPACK people know what they are doing.

Cheers,
Pauli






From mohitiiit4u at gmail.com  Thu Feb 25 03:01:43 2010
From: mohitiiit4u at gmail.com (Mohit Goyal)
Date: Thu, 25 Feb 2010 13:31:43 +0530
Subject: [SciPy-dev] GSOC project discussion on improving datasource and its
	integration.
Message-ID: <f34db9ee1002250001p2e32bdfak243dff44841380b2@mail.gmail.com>

I am masters students in data engineering and interested in contributing to
scipy. I need some directions to how to start with it.  I know python and
have worked on projects including data mining , web mining and  web crawlers
etc.  I want to contribute to scipy by improving datasource and its
integrations like handling malformed URLs, proxy settings , other file
extensions support.

 http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas
http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/lib/_datasource.py



Thank You.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100225/84dc2b04/attachment.html>

From robert.kern at gmail.com  Thu Feb 25 10:45:14 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 25 Feb 2010 09:45:14 -0600
Subject: [SciPy-dev] GSOC project discussion on improving datasource and
	its integration.
In-Reply-To: <f34db9ee1002250001p2e32bdfak243dff44841380b2@mail.gmail.com>
References: <f34db9ee1002250001p2e32bdfak243dff44841380b2@mail.gmail.com>
Message-ID: <3d375d731002250745o593cd0eai6cf38f926185d530@mail.gmail.com>

On Thu, Feb 25, 2010 at 02:01, Mohit Goyal <mohitiiit4u at gmail.com> wrote:
>
> I am masters students in data engineering and interested in contributing to
> scipy. I need some directions to how to start with it.

Yes. We know. We have seen your previous messages and some people have
responded. Please go back to those threads and respond to those
responses. Please do not post another introduction like this again. We
appreciate your interest and enthusiasm, but we would also appreciate
some improvement with how you communicate with the list.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From mohitiiit at gmail.com  Thu Feb 25 12:22:44 2010
From: mohitiiit at gmail.com (Mohit Goyal)
Date: Thu, 25 Feb 2010 22:52:44 +0530
Subject: [SciPy-dev] GSOC project discussion on improving datasource and
	its integration.
In-Reply-To: <3d375d731002250745o593cd0eai6cf38f926185d530@mail.gmail.com>
References: <f34db9ee1002250001p2e32bdfak243dff44841380b2@mail.gmail.com> 
	<3d375d731002250745o593cd0eai6cf38f926185d530@mail.gmail.com>
Message-ID: <9e78df6d1002250922n339f180bu21a85d7e67e9efbe@mail.gmail.com>

I am extremely sorry for inconvenience caused. I am subscribed to the list
but i am not receiving any replies, mails etc from this list.  That's why i
sent it twice.

Thank You.

Best Regards,
Mohit Goyal.



On Thu, Feb 25, 2010 at 9:15 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Thu, Feb 25, 2010 at 02:01, Mohit Goyal <mohitiiit4u at gmail.com> wrote:
> >
> > I am masters students in data engineering and interested in contributing
> to
> > scipy. I need some directions to how to start with it.
>
> Yes. We know. We have seen your previous messages and some people have
> responded. Please go back to those threads and respond to those
> responses. Please do not post another introduction like this again. We
> appreciate your interest and enthusiasm, but we would also appreciate
> some improvement with how you communicate with the list.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
>



-- 
The Game is Not Over Because i haven't Won Yet.
-Mohit Goyal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100225/42cf2b5f/attachment.html>

From smattacus at gmail.com  Thu Feb 25 13:40:46 2010
From: smattacus at gmail.com (Sean Mattingly)
Date: Thu, 25 Feb 2010 10:40:46 -0800
Subject: [SciPy-dev] GSOC project discussion on improving datasource and
	its integration.
In-Reply-To: <9e78df6d1002250922n339f180bu21a85d7e67e9efbe@mail.gmail.com>
References: <f34db9ee1002250001p2e32bdfak243dff44841380b2@mail.gmail.com>
	<3d375d731002250745o593cd0eai6cf38f926185d530@mail.gmail.com>
	<9e78df6d1002250922n339f180bu21a85d7e67e9efbe@mail.gmail.com>
Message-ID: <856175f81002251040v6f932a1cn72bf97294307a6dd@mail.gmail.com>

This mailing list, like most others of its form, has an archive. This page
has your original thread and the replies:

http://mail.scipy.org/pipermail/scipy-dev/2010-February/thread.html

Or, to go straight to your original post:

http://mail.scipy.org/pipermail/scipy-dev/2010-February/013874.html

You can click through the replies there.

Finally, there's a few scipy mailing lists, so it's possible you're
subscribed to a different one than you sent this to:
http://www.scipy.org/Mailing_Lists

Hope that helps.

 - Sean

On Thu, Feb 25, 2010 at 9:22 AM, Mohit Goyal <mohitiiit at gmail.com> wrote:

>
> I am extremely sorry for inconvenience caused. I am subscribed to the list
> but i am not receiving any replies, mails etc from this list.  That's why i
> sent it twice.
>
> Thank You.
>
> Best Regards,
> Mohit Goyal.
>
>
>
>
> On Thu, Feb 25, 2010 at 9:15 PM, Robert Kern <robert.kern at gmail.com>wrote:
>
>> On Thu, Feb 25, 2010 at 02:01, Mohit Goyal <mohitiiit4u at gmail.com> wrote:
>> >
>> > I am masters students in data engineering and interested in contributing
>> to
>> > scipy. I need some directions to how to start with it.
>>
>> Yes. We know. We have seen your previous messages and some people have
>> responded. Please go back to those threads and respond to those
>> responses. Please do not post another introduction like this again. We
>> appreciate your interest and enthusiasm, but we would also appreciate
>> some improvement with how you communicate with the list.
>>
>> --
>> Robert Kern
>>
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>>  -- Umberto Eco
>>
>
>
>
> --
> The Game is Not Over Because i haven't Won Yet.
> -Mohit Goyal
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100225/f5d96b0f/attachment.html>

From bouloumag at gmail.com  Thu Feb 25 14:16:39 2010
From: bouloumag at gmail.com (Darcoux Christine)
Date: Thu, 25 Feb 2010 14:16:39 -0500
Subject: [SciPy-dev] implement reorthogonalization method in gmres
In-Reply-To: <1267057599.24576.0.camel@idol>
References: <477ac73a1002231726h18b1rcd5f884e3c0cc036@mail.gmail.com>
	<hm34qr$ndi$1@dough.gmane.org>
	<fa510ff81002241443w5a1a027fi9160b1db022db104@mail.gmail.com>
	<1267057599.24576.0.camel@idol>
Message-ID: <477ac73a1002251116m16f43a32sbe02b3e65fd6a9b2@mail.gmail.com>

Dear Pauli,

Thank you for your quick answer with a pointer to the litterature.

The result in the paper you cite assumes that the matrix is
well-conditioned, and I'm not sure I can count on that in a
general-purpose code which many people will use.

The books "Solving nonlinear equations with Newton's method" and
"Iterative methods for linear and nonlinear equations" (some parts are
availlable from google book) by C. T. Kelley gives a good presentation
on the importance of orthogonality. It also show a simple 3x3 example
where paying attention to orthogonality is very important.

Checking for orthogonality and reorthogonalizing the way there author
of these books do it costs almost nothing (see the link to the code in
my previous post). Matlab's "official" GMRES code uses Householder
reflections to build the basis and gets orthogonality automatically,
but at double the cost of Gram-Schmidt. The Sandia Trilinos code uses
classical Gram-Schmidt for good parallel performance and
reorthogonalizes at every iteration, so also costs double. The way
proposed by Kelley costs nothing because the reorthogonalization
happens rarely, so I do not see any reason not to do it. Of course,
this could be added as an option so users will decide if they want to
use it.

Best Regards,

Christine

2010/2/24, Pauli Virtanen <pav at iki.fi>:
> ke, 2010-02-24 kello 23:43 +0100, nicky van foreest kirjoitti:
>> I am not at all a specialist on this, but I recall that Numerical
>> Recipes states that the Gram-Schmidt process gives terrible results,
>> and should be avoided nearly always.
>
> There's a balance here: more accurate methods tend to require more work,
> and the orthogonalization is probably the most expensive part of GMRES.
> So one should use a method that is good enough.
>
>> Instead, singular value
>> decomposition should be used. To avoid the problem below, would is be
>> an idea to replace the GM algo by SVD?
>
> One could also QR factorize using Householder transforms, which should
> be more accurate than GS, and is probably less expensive than SVD; or
> reorthogonalize.
>
> But as far as I understand, this would not really improve the
> performance of GMRES: as shown in the article I linked to (and also in
> others), the quality of the orthogonalization is not expected to matter
> for GMRES -- modified Gram-Schmid is good enough. The point is that
> breakdown of orthogonality happens simultaneously with the solution
> reaching its maximal accuracy, so it will not essentially affect
> convergence or accuracy. I'd like to have a reference or a test case at
> hand stating otherwise before changing things here.
>
> As far as I understand, however, the situation with orthogonality is
> different if you want to solve eigenvalue problems with Arnoldi stuff.
> But that's in the domain of ARPACK, and I would trust without looking
> that the ARPACK people know what they are doing.
>
> Cheers,
> Pauli
>
>
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From pav at iki.fi  Thu Feb 25 15:18:51 2010
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 25 Feb 2010 22:18:51 +0200
Subject: [SciPy-dev] implement reorthogonalization method in gmres
In-Reply-To: <477ac73a1002251116m16f43a32sbe02b3e65fd6a9b2@mail.gmail.com>
References: <477ac73a1002231726h18b1rcd5f884e3c0cc036@mail.gmail.com>
	<hm34qr$ndi$1@dough.gmane.org>
	<fa510ff81002241443w5a1a027fi9160b1db022db104@mail.gmail.com>
	<1267057599.24576.0.camel@idol>
	<477ac73a1002251116m16f43a32sbe02b3e65fd6a9b2@mail.gmail.com>
Message-ID: <1267129131.5836.37.camel@idol>

to, 2010-02-25 kello 14:16 -0500, Darcoux Christine kirjoitti:
> Thank you for your quick answer with a pointer to the litterature.
> 
> The result in the paper you cite assumes that the matrix is
> well-conditioned, and I'm not sure I can count on that in a
> general-purpose code which many people will use.

Yes, it's a good point that the situation might not be the same for near
singular matrices. Perhaps someone looked into that...

> The books "Solving nonlinear equations with Newton's method" and
> "Iterative methods for linear and nonlinear equations" (some parts are
> availlable from google book) by C. T. Kelley gives a good presentation
> on the importance of orthogonality. It also show a simple 3x3 example
> where paying attention to orthogonality is very important.

I note that the book is from ca 1995, whereas the papers are newer. For
the particular example in the book, 

        [ 0.001  0  0;  0  0.0011  0; 0 0 1e4 ]

MGS (which we currently have) required an additional step at the final
stage of the convergence, as compared to GS + selective
reorthogonalization. Up to the last step, the convergence histories were
identical.

> Checking for orthogonality and reorthogonalizing the way there author
> of these books do it costs almost nothing (see the link to the code in
> my previous post). Matlab's "official" GMRES code uses Householder
> reflections to build the basis and gets orthogonality automatically,
> but at double the cost of Gram-Schmidt. The Sandia Trilinos code uses
> classical Gram-Schmidt for good parallel performance and
> reorthogonalizes at every iteration, so also costs double.
> The way proposed by Kelley costs nothing because the
> reorthogonalization happens rarely, so I do not see any reason not to
> do it. Of course, this could be added as an option so users will
> decide if they want to use it.

Well, as I said, if someone implements it + writes tests, it will get
in. If you are not going to implement it, please submit an enhancement
request at

        http://projects.scipy.org/scipy

to keep track of the idea. I don't have the time to work at this right
now, though, but later is possible.

Out of interest: do you have an actual matrix at hand where you know
that MGS + selective reorthogonalization will give significantly better
results than plain MGS, or a reference that shows an example like that?

-- 
Pauli Virtanen






From pearu.peterson at gmail.com  Fri Feb 26 03:20:56 2010
From: pearu.peterson at gmail.com (pearu.peterson at gmail.com)
Date: Fri, 26 Feb 2010 09:20:56 +0100
Subject: [SciPy-dev] Hello
Message-ID: <20100226082032.A478E39CB98@scipy.org>

The message cannot be represented in 7-bit ASCII encoding and has been sent as a binary attachment.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: message.scr
Type: application/octet-stream
Size: 19273 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100226/b542d503/attachment.obj>

From josef.pktd at gmail.com  Fri Feb 26 09:00:05 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 26 Feb 2010 09:00:05 -0500
Subject: [SciPy-dev] spam filter for scipy-tickets or
	scipy-tickets-bounces@scipy.org
Message-ID: <1cd32cbb1002260600l464665cft1ec8128d8fdc03fd@mail.gmail.com>

Is there a spam filter on the scipy ticket mailing list ?

Or am I the only one to get spam through it?

Josef


From rmay31 at gmail.com  Fri Feb 26 09:39:22 2010
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 26 Feb 2010 08:39:22 -0600
Subject: [SciPy-dev] spam filter for scipy-tickets or
	scipy-tickets-bounces@scipy.org
In-Reply-To: <1cd32cbb1002260600l464665cft1ec8128d8fdc03fd@mail.gmail.com>
References: <1cd32cbb1002260600l464665cft1ec8128d8fdc03fd@mail.gmail.com>
Message-ID: <c7b95a021002260639y189d834cv893b20193e7d9708@mail.gmail.com>

On Fri, Feb 26, 2010 at 8:00 AM,  <josef.pktd at gmail.com> wrote:
> Is there a spam filter on the scipy ticket mailing list ?
>
> Or am I the only one to get spam through it?

No, I've been seeing it too.  On scipy-svn as well.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From ariver at enthought.com  Fri Feb 26 17:38:39 2010
From: ariver at enthought.com (Aaron River)
Date: Fri, 26 Feb 2010 16:38:39 -0600
Subject: [SciPy-Dev] [SciPy-dev] GSOC project discussion on improving
	datasource and its integration.
In-Reply-To: <9e78df6d1002250922n339f180bu21a85d7e67e9efbe@mail.gmail.com>
References: <f34db9ee1002250001p2e32bdfak243dff44841380b2@mail.gmail.com>
	<3d375d731002250745o593cd0eai6cf38f926185d530@mail.gmail.com>
	<9e78df6d1002250922n339f180bu21a85d7e67e9efbe@mail.gmail.com>
Message-ID: <E28E7164-58F0-4502-91E4-D2657E162C9E@enthought.com>

Hello Mohit,

I'm the "IT-Guy" over here at Enthought, where these lists are hosted.

In order to make sure we didn't have something breaking on our end
that might be keeping you from getting emails from the list, I reviewed
our logs and reconciled them with the list archives.

All of your emails to the lists from the 17th up to today were sent from
"mohitiiit", and all replies to those emails were sent on the same day,
and to that same address.

Your first and only email from "mohitiit4u" was sent today to scipy-dev.
You did receive Robert's reply to "mohitiiit4u", to which you replied.
But, your reply came from "mohitiiit".

I see you are now subscribed to scipy-dev with both of your addresses.

Perhaps you are expecting the emails to be arriving at "mohitiiit4u"?

I hope this helps resolve any issues you're having with the lists.

Let me know if I can assist you any further.

Thanks,

-- 
Aaron River
ariver at enthought.com



On Thu, Feb 25, 2010 at 11:22, Mohit Goyal <mohitiiit at gmail.com> wrote:
> 
> I am extremely sorry for inconvenience caused. I am subscribed to the list
> but i am not receiving any replies, mails etc from this list.  That's why i
> sent it twice.
> 
> Thank You.
> 
> Best Regards,
> Mohit Goyal.
> 
> 
> 
> On Thu, Feb 25, 2010 at 9:15 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> 
>> On Thu, Feb 25, 2010 at 02:01, Mohit Goyal <mohitiiit4u at gmail.com> wrote:
>>> 
>>> I am masters students in data engineering and interested in contributing
>>> to
>>> scipy. I need some directions to how to start with it.
>> 
>> Yes. We know. We have seen your previous messages and some people have
>> responded. Please go back to those threads and respond to those
>> responses. Please do not post another introduction like this again. We
>> appreciate your interest and enthusiasm, but we would also appreciate
>> some improvement with how you communicate with the list.
>> 
>> --
>> Robert Kern
>> 
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>>  -- Umberto Eco
> 
> 
> 
> --
> The Game is Not Over Because i haven't Won Yet.
> -Mohit Goyal
> 
> 
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
> 
> 


From ralf.gommers at googlemail.com  Sat Feb 27 09:02:36 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sat, 27 Feb 2010 22:02:36 +0800
Subject: [SciPy-Dev] build issue on OS X 10.6 - fails on fftpack
Message-ID: <dde7764a1002270602h3aea67cdo2e0cc9922349a261@mail.gmail.com>

After switching from Apple Python to the python.org version this is the
first time I tried to build scipy. I see the same problem when building
scipy trunk against numpy trunk or 0.7.x against numpy 1.3.

Compilers:
$ gcc --version
i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5646) (dot 1)
$ gcc-4.0 --version
i686-apple-darwin10-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5493)
$ gfortran --version
GNU Fortran (GCC) 4.2.3
(obtained from http://r.research.att.com/tools/)

Python:
$ python
Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin


Below is the relevant part (I think) of the build output. The "missing
required architecture x86_64 in file" complaints happen for numpy builds as
well, and seem to be harmless there.

Any ideas?

Thanks,
Ralf



$ LDFLAGS="-lgfortran -arch x86_64" FFLAGS="-arch x86_64" python setup.py
build
Warning: No configuration returned, assuming unavailable.blas_opt_info:
  FOUND:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
    extra_compile_args = ['-faltivec',
'-I/System/Library/Frameworks/vecLib.framework/Headers']

lapack_opt_info:
  FOUND:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
    extra_compile_args = ['-faltivec']

umfpack_info:
  libraries umfpack not found in
/Library/Frameworks/Python.framework/Versions/2.6/lib
  libraries umfpack not found in /usr/local/lib
  libraries umfpack not found in /usr/lib
/Users/rgommers/Code/numpy/numpy/distutils/system_info.py:459: UserWarning:
    UMFPACK sparse solver (http://www.cise.ufl.edu/research/sparse/umfpack/)
    not found. Directories to search for the libraries can be specified in
the
    numpy/distutils/site.cfg file (section [umfpack]) or by setting
    the UMFPACK environment variable.
  warnings.warn(self.notfounderror.__doc__)
  NOT AVAILABLE

running build


<SNIP>


building 'scipy.fftpack._fftpack' extension
compiling C sources
C compiler: gcc-4.0 -arch ppc -arch i386 -isysroot
/Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic
-DNDEBUG -g -O3

creating build/temp.macosx-10.3-fat-2.6/build
creating build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6
creating build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy
creating
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack
creating
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/src
compile options: '-Iscipy/fftpack/src -Ibuild/src.macosx-10.3-fat-2.6
-I/Users/rgommers/Code/numpy/numpy/core/include
-I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c'
gcc-4.0: scipy/fftpack/src/zrfft.c
gcc-4.0: build/src.macosx-10.3-fat-2.6/scipy/fftpack/src/dct.c
gcc-4.0: scipy/fftpack/src/zfftnd.c
gcc-4.0: build/src.macosx-10.3-fat-2.6/fortranobject.c
gcc-4.0: build/src.macosx-10.3-fat-2.6/scipy/fftpack/_fftpackmodule.c
gcc-4.0: scipy/fftpack/src/drfft.c
gcc-4.0: scipy/fftpack/src/zfft.c
/usr/local/bin/gfortran -Wall -lgfortran -arch x86_64
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/_fftpackmodule.o
build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfft.o
build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/drfft.o
build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zrfft.o
build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfftnd.o
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/src/dct.o
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/fortranobject.o
-L/usr/local/lib/gcc/i686-apple-darwin8/4.2.3/x86_64
-Lbuild/temp.macosx-10.3-fat-2.6 -ldfftpack -lfftpack -lgfortran -o
build/lib.macosx-10.3-fat-2.6/scipy/fftpack/_fftpack.so
ld: warning: in
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/_fftpackmodule.o,
missing required architecture x86_64 in file
ld: warning: in build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfft.o,
missing required architecture x86_64 in file
ld: warning: in build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/drfft.o,
missing required architecture x86_64 in file
ld: warning: in build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zrfft.o,
missing required architecture x86_64 in file
ld: warning: in build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfftnd.o,
missing required architecture x86_64 in file
ld: warning: in
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/src/dct.o,
missing required architecture x86_64 in file
ld: warning: in
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/fortranobject.o,
missing required architecture x86_64 in file
Undefined symbols:
  "_MAIN__", referenced from:
      _main in libgfortranbegin.a(fmain.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
ld: warning: in
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/_fftpackmodule.o,
missing required architecture x86_64 in file
ld: warning: in build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfft.o,
missing required architecture x86_64 in file
ld: warning: in build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/drfft.o,
missing required architecture x86_64 in file
ld: warning: in build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zrfft.o,
missing required architecture x86_64 in file
ld: warning: in build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfftnd.o,
missing required architecture x86_64 in file
ld: warning: in
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/src/dct.o,
missing required architecture x86_64 in file
ld: warning: in
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/fortranobject.o,
missing required architecture x86_64 in file
Undefined symbols:
  "_MAIN__", referenced from:
      _main in libgfortranbegin.a(fmain.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
error: Command "/usr/local/bin/gfortran -Wall -lgfortran -arch x86_64
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/_fftpackmodule.o
build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfft.o
build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/drfft.o
build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zrfft.o
build/temp.macosx-10.3-fat-2.6/scipy/fftpack/src/zfftnd.o
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/scipy/fftpack/src/dct.o
build/temp.macosx-10.3-fat-2.6/build/src.macosx-10.3-fat-2.6/fortranobject.o
-L/usr/local/lib/gcc/i686-apple-darwin8/4.2.3/x86_64
-Lbuild/temp.macosx-10.3-fat-2.6 -ldfftpack -lfftpack -lgfortran -o
build/lib.macosx-10.3-fat-2.6/scipy/fftpack/_fftpack.so" failed with exit
status 1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100227/fd5c8661/attachment.html>

From david at silveregg.co.jp  Sun Feb 28 20:50:29 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Mon, 01 Mar 2010 10:50:29 +0900
Subject: [SciPy-Dev] build issue on OS X 10.6 - fails on fftpack
In-Reply-To: <dde7764a1002270602h3aea67cdo2e0cc9922349a261@mail.gmail.com>
References: <dde7764a1002270602h3aea67cdo2e0cc9922349a261@mail.gmail.com>
Message-ID: <4B8B1D65.107@silveregg.co.jp>

Ralf Gommers wrote:

> $ LDFLAGS="-lgfortran -arch x86_64" FFLAGS="-arch x86_64" python

Whenever you redefine one of the CFLAGS/FFLAGS/CXXFLAGS/LDFLAGS, it 
*overwrites* the default value (contrary to the usual behavior of having 
its value appended to the default value).

here, by overwriting LDFLAGS, you remove all the link options necessary 
to build a bundle, and the linker believes you want to build an 
executable (hence the missing main symbol message, and all the 
unresolved symbols from libpython.dylib).

You can see which flags to use in the scipy paver script (note in 
particular that the gfortran library should be linked statically, to 
avoid adding a dependency on gfortran for scipy binaries - I used to run 
the test suite in a  Mac OS X chroot check this).

David


From ralf.gommers at googlemail.com  Sun Feb 28 21:56:26 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Mon, 1 Mar 2010 10:56:26 +0800
Subject: [SciPy-Dev] build issue on OS X 10.6 - fails on fftpack
In-Reply-To: <4B8B1D65.107@silveregg.co.jp>
References: <dde7764a1002270602h3aea67cdo2e0cc9922349a261@mail.gmail.com>
	<4B8B1D65.107@silveregg.co.jp>
Message-ID: <dde7764a1002281856h7c3b621fs172100803e0b6b4@mail.gmail.com>

On Mon, Mar 1, 2010 at 9:50 AM, David Cournapeau <david at silveregg.co.jp>wrote:

> Ralf Gommers wrote:
>
> > $ LDFLAGS="-lgfortran -arch x86_64" FFLAGS="-arch x86_64" python
>
> Whenever you redefine one of the CFLAGS/FFLAGS/CXXFLAGS/LDFLAGS, it
> *overwrites* the default value (contrary to the usual behavior of having
> its value appended to the default value).
>

I followed the instructions under "Obtaining and Building NumPy and SciPy on
this page: http://www.scipy.org/Installing_SciPy/Mac_OS_X . There it says
that the above command *extends* the build flags. So that info is wrong? Or
maybe I'm still not understanding something.

>
> here, by overwriting LDFLAGS, you remove all the link options necessary
> to build a bundle, and the linker believes you want to build an
> executable (hence the missing main symbol message, and all the
> unresolved symbols from libpython.dylib).
>

To be clear, I'm just trying to build scipy on my own machine, so I can test
the numpy binary installer against it. I'm not trying to build a scipy
installer. Although figuring out how to do that would be good as well.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20100301/6d201377/attachment.html>

From david at silveregg.co.jp  Sun Feb 28 22:03:01 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Mon, 01 Mar 2010 12:03:01 +0900
Subject: [SciPy-Dev] build issue on OS X 10.6 - fails on fftpack
In-Reply-To: <dde7764a1002281856h7c3b621fs172100803e0b6b4@mail.gmail.com>
References: <dde7764a1002270602h3aea67cdo2e0cc9922349a261@mail.gmail.com>
	<4B8B1D65.107@silveregg.co.jp>
	<dde7764a1002281856h7c3b621fs172100803e0b6b4@mail.gmail.com>
Message-ID: <4B8B2E65.1030708@silveregg.co.jp>

Ralf Gommers wrote:
> 
> 
> On Mon, Mar 1, 2010 at 9:50 AM, David Cournapeau <david at silveregg.co.jp 
> <mailto:david at silveregg.co.jp>> wrote:
> 
>     Ralf Gommers wrote:
> 
>      > $ LDFLAGS="-lgfortran -arch x86_64" FFLAGS="-arch x86_64" python
> 
>     Whenever you redefine one of the CFLAGS/FFLAGS/CXXFLAGS/LDFLAGS, it
>     *overwrites* the default value (contrary to the usual behavior of having
>     its value appended to the default value).
> 
> 
> I followed the instructions under "Obtaining and Building NumPy and 
> SciPy on this page: http://www.scipy.org/Installing_SciPy/Mac_OS_X . 
> There it says that the above command *extends* the build flags. So that 
> info is wrong? Or maybe I'm still not understanding something.

Both are true :) If you build with numscons, then the flags are 
extended. If you build with distutils, the flags are overriden.

The wiki could be made clearer there.

> 
> 
>     here, by overwriting LDFLAGS, you remove all the link options necessary
>     to build a bundle, and the linker believes you want to build an
>     executable (hence the missing main symbol message, and all the
>     unresolved symbols from libpython.dylib).
> 
> 
> To be clear, I'm just trying to build scipy on my own machine

The flag issue is a distutils issue, and happens whenever you call 
setup.py. You should not try to edit flags if you build with distutils, 
really - the overriding support implies that it should be used as last 
resort, and otherwise, everything should be handled within distutils.

cheers,

David