From david at ar.media.kyoto-u.ac.jp  Mon Dec  1 02:02:04 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 01 Dec 2008 16:02:04 +0900
Subject: [SciPy-dev] How to test for easy_install scipy from sources ?
Message-ID: <49338BEC.1090404@ar.media.kyoto-u.ac.jp>

Hi,

    I would like to see if I fixed the following issue:

http://scipy.org/scipy/scipy/ticket/801

Problem is, I don't know how easy_install is used for something not
released on pypi ?

thanks,

David


From cournape at gmail.com  Mon Dec  1 02:24:11 2008
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 1 Dec 2008 16:24:11 +0900
Subject: [SciPy-dev] 1 Failure in 0.7.0b1
In-Reply-To: <49331DE0.1040700@gmail.com>
References: <c7009a550811252204h2a9ad237p5bba6a5028b6da4@mail.gmail.com>
	<ggjil1$7kq$1@ger.gmane.org> <49331DE0.1040700@gmail.com>
Message-ID: <5b8d13220811302324i69d45b8bpa817031a79efce40@mail.gmail.com>

On Mon, Dec 1, 2008 at 8:12 AM, Xavier Gnata <xavier.gnata at gmail.com> wrote:

> Ok I can reproduce this one on intrepid ibex 64bits (gfortran).
> The funny think is that I think (if I read the correct mathword pages)
> that 1.0 is the correct answer (but it is so easy to be wrong with these
> special function).

I can't reproduce on RHEL 5 (64 bits/python2.4). I opened a ticket:

http://scipy.org/scipy/scipy/ticket/803

Neal, Xavier, which version of numpy are you using ? Which version of
gcc is used in FC 9 ?

thanks,

David


From gael.varoquaux at normalesup.org  Mon Dec  1 03:09:01 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 1 Dec 2008 09:09:01 +0100
Subject: [SciPy-dev] How to test for easy_install scipy from sources ?
In-Reply-To: <49338BEC.1090404@ar.media.kyoto-u.ac.jp>
References: <49338BEC.1090404@ar.media.kyoto-u.ac.jp>
Message-ID: <20081201080901.GB18450@phare.normalesup.org>

On Mon, Dec 01, 2008 at 04:02:04PM +0900, David Cournapeau wrote:
>     I would like to see if I fixed the following issue:

> http://scipy.org/scipy/scipy/ticket/801

> Problem is, I don't know how easy_install is used for something not
> released on pypi ?

Use the -f switch of easy_install to point to a local directory on your
box where you build an egg, an simply put a source tar.gz.

Ga?l


From pav at iki.fi  Mon Dec  1 03:19:55 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 1 Dec 2008 08:19:55 +0000 (UTC)
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
References: <1227987645.8264.200.camel@idol>
	<4932250D.7040003@ar.media.kyoto-u.ac.jp>
	<9457e7c80811301410x333f6352r5fa4e81f4ac628@mail.gmail.com>
Message-ID: <gh06nb$1ug$1@ger.gmane.org>

Mon, 01 Dec 2008 00:10:53 +0200, St?fan van der Walt wrote:
> 2008/11/30 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>> Pauli Virtanen wrote:
>>> * Getting this to 0.7.0; this is a complete rewrite, so is it too
>>> late,
>>>   and is it better to wait for 0.8?
>>>
>> I would much prefer delaying this after 0.7 release. Unfortunately, I
>> have nothing else to say, since I know nothing to optimization :)
> 
> Postponing a patch often leads to the author losing interest, and to the
> patch never being applied.  I don't know if we can afford that. I'd be
> quite happy to see this going into 0.7 rc2 and trying it out for beta 1
> (or whatever the naming scheme is), especially if we can fine-tune the
> tests a bit.

Well, I promise not to lose interest if this gets delayed until 0.8 :)

The main reason for proposing this for 0.7.0 is that IMO having tolerance-
based termination conditions for these methods makes them much more 
useful, especially as they give only NaNs out if you set iter (the fixed 
number of iterations they do) too high.

But maybe it's best to play safe and ship 0.7.0 out first, and make sure 
the new nonlin.py is OK and well-tested in 0.8.

-- 
Pauli Virtanen



From pav at iki.fi  Mon Dec  1 03:28:15 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 1 Dec 2008 08:28:15 +0000 (UTC)
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
References: <1227987645.8264.200.camel@idol>
	<920F563C-A1BB-487A-81C6-8571C4DC526C@math.toronto.edu>
	<ce557a360811301859s5a91a55cid9e3049e42bf5795@mail.gmail.com>
Message-ID: <gh076v$1ug$2@ger.gmane.org>

Sun, 30 Nov 2008 21:59:32 -0500, Anne Archibald wrote:

> 2008/11/30 Gideon Simpson <simpson at math.toronto.edu>:
>> Still no args input for inputting arguments to the function F?
>>
>> Sorry to complain, but the absence of this has put me off using these
>> routines as it would require a rewrite of much of my code.
> 
> Why?
> 
> Instead of
> optimize.whatever(F, args=extra)
> just use
> optimize.whatever(lambda x: F(x,extra))

Exactly; I was under the impression that this 'args' business was some 
legacy from times when Python didn't support nested scopes, cf. eg.

	http://www.python.org/doc/2.1/ref/lambda.html
	http://www.python.org/dev/peps/pep-0227/

I don't see a need for the 'args' option except maybe for conformance 
with the other optimization functions in Scipy. But maybe there are also 
other reasons to include it?

-- 
Pauli Virtanen



From benny.malengier at gmail.com  Mon Dec  1 07:07:24 2008
From: benny.malengier at gmail.com (Benny Malengier)
Date: Mon, 1 Dec 2008 13:07:24 +0100
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <gh076v$1ug$2@ger.gmane.org>
References: <1227987645.8264.200.camel@idol>
	<920F563C-A1BB-487A-81C6-8571C4DC526C@math.toronto.edu>
	<ce557a360811301859s5a91a55cid9e3049e42bf5795@mail.gmail.com>
	<gh076v$1ug$2@ger.gmane.org>
Message-ID: <bcafe8290812010407h87266ffm402167a3d9b9ac5d@mail.gmail.com>

2008/12/1 Pauli Virtanen <pav at iki.fi>

> Sun, 30 Nov 2008 21:59:32 -0500, Anne Archibald wrote:
>
> > 2008/11/30 Gideon Simpson <simpson at math.toronto.edu>:
> >> Still no args input for inputting arguments to the function F?
> >>
> >> Sorry to complain, but the absence of this has put me off using these
> >> routines as it would require a rewrite of much of my code.
> >
> > Why?
> >
> > Instead of
> > optimize.whatever(F, args=extra)
> > just use
> > optimize.whatever(lambda x: F(x,extra))
>
> Exactly; I was under the impression that this 'args' business was some
> legacy from times when Python didn't support nested scopes, cf. eg.
>
>        http://www.python.org/doc/2.1/ref/lambda.html
>        http://www.python.org/dev/peps/pep-0227/
>
> I don't see a need for the 'args' option except maybe for conformance
> with the other optimization functions in Scipy. But maybe there are also
> other reasons to include it?
>

I have noticed these args not even working well in some cases in integrate
ode method (due to the fortran wrapper code not handling them). Eg vode.pyf
has
     double precision intent(hide) :: rpar
While the ode class does contain:

    def set_f_params(self,*args):
        """Set extra parameters for user-supplied function f."""
        self.f_params = args
        return self

I tend to use class functions, which have access to other class
methods/attributes, so
optimize.whatever(problem.F)

Benny
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081201/6ac15b73/attachment.html>

From ndbecker2 at gmail.com  Mon Dec  1 07:36:34 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 01 Dec 2008 07:36:34 -0500
Subject: [SciPy-dev] 1 Failure in 0.7.0b1
References: <c7009a550811252204h2a9ad237p5bba6a5028b6da4@mail.gmail.com>
	<ggjil1$7kq$1@ger.gmane.org> <49331DE0.1040700@gmail.com>
	<5b8d13220811302324i69d45b8bpa817031a79efce40@mail.gmail.com>
Message-ID: <gh0loi$ko7$1@ger.gmane.org>

David Cournapeau wrote:

> On Mon, Dec 1, 2008 at 8:12 AM, Xavier Gnata <xavier.gnata at gmail.com>
> wrote:
> 
>> Ok I can reproduce this one on intrepid ibex 64bits (gfortran).
>> The funny think is that I think (if I read the correct mathword pages)
>> that 1.0 is the correct answer (but it is so easy to be wrong with these
>> special function).
> 
> I can't reproduce on RHEL 5 (64 bits/python2.4). I opened a ticket:
> 
> http://scipy.org/scipy/scipy/ticket/803
> 
> Neal, Xavier, which version of numpy are you using ? Which version of
> gcc is used in FC 9 ?
> 
> thanks,
> 
> David
: numpy.__version__
Out[3]: '1.2.0'

gcc-4.3.0-8.x86_64



From stefan at sun.ac.za  Mon Dec  1 07:48:58 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 1 Dec 2008 14:48:58 +0200
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <5b8d13220811301932x7fdf8f0bn72ec63496bad8423@mail.gmail.com>
References: <1227987645.8264.200.camel@idol>
	<4932250D.7040003@ar.media.kyoto-u.ac.jp>
	<9457e7c80811301410x333f6352r5fa4e81f4ac628@mail.gmail.com>
	<5b8d13220811301932x7fdf8f0bn72ec63496bad8423@mail.gmail.com>
Message-ID: <9457e7c80812010448l29c56eceyd9a1adb9e047a1f4@mail.gmail.com>

2008/12/1 David Cournapeau <cournape at gmail.com>:
> first beta. Also, rushing to add new code may lead to oversight some
> limitation, some API bug, etc...

I'm happy to postpone, but then we should accept that the API may
change just after 0.7.  It concerns me that non-reviewed code becomes
the standard quo, and then sets the API for the future.  Also, if
Pauli is willing to write some tests, we can commit those so long.

Cheers
St?fan


From david at ar.media.kyoto-u.ac.jp  Mon Dec  1 07:46:41 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 01 Dec 2008 21:46:41 +0900
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <9457e7c80812010448l29c56eceyd9a1adb9e047a1f4@mail.gmail.com>
References: <1227987645.8264.200.camel@idol>	<4932250D.7040003@ar.media.kyoto-u.ac.jp>	<9457e7c80811301410x333f6352r5fa4e81f4ac628@mail.gmail.com>	<5b8d13220811301932x7fdf8f0bn72ec63496bad8423@mail.gmail.com>
	<9457e7c80812010448l29c56eceyd9a1adb9e047a1f4@mail.gmail.com>
Message-ID: <4933DCB1.1070803@ar.media.kyoto-u.ac.jp>

St?fan van der Walt wrote:
> 2008/12/1 David Cournapeau <cournape at gmail.com>:
>> first beta. Also, rushing to add new code may lead to oversight some
>> limitation, some API bug, etc...
>
> I'm happy to postpone, but then we should accept that the API may
> change just after 0.7.  It concerns me that non-reviewed code becomes
> the standard quo, and then sets the API for the future.

Hey, that's one of the reason why I prefer avoiding changes during the
beta timeline :) By rushing code into the release, we are less likely to
review changes correctly; from the discussion, it looks quite likely
that the rewrite is not ready yet.

The only goal of a beta is to see whether we have unexpected failures,
and solve them until the release. If we add new code, there is no point
in doing beta. Even without taking into account this, having a window
where no new feature is allowed is a really common process; even linux,
which changes extremely rapidly, has a limited time window for merges.
Every project with a proper release management enforce those very
strictly, because that's the whole point, really. Otherwise, releases
are nothing more than a snapshot of the source code at a given time

The problem of people not seeing their code quickly should be solved by
doing more frequent releases IMO; "freezing" the trunk for some time
before releases helps that, rushing new code does not. It is not
theoretical: almost all my time spent on scipy the last few weeks have
been related to recently added code.

cheers,

David


From pav at iki.fi  Mon Dec  1 08:06:16 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 1 Dec 2008 13:06:16 +0000 (UTC)
Subject: [SciPy-dev] Umfpack deprecation warning
Message-ID: <gh0ng8$s2i$1@ger.gmane.org>

Hi,

Can something be done to this warning:

>>> import scipy.linalg
/usr/local/lib/python2.5/site-packages/scipy-0.7.0.dev5203-py2.5-linux-
x86_64.egg/scipy/sparse/linalg/dsolve/linsolve.py:20: DeprecationWarning: 
scipy.sparse.linalg.dsolve.umfpack will be removed, install 
scikits.umfpack instead
  ' install scikits.umfpack instead', DeprecationWarning )

This is a bit annoying as it comes up basically for all scripts that 
import Scipy.

-- 
Pauli Virtanen



From stefan at sun.ac.za  Mon Dec  1 08:30:16 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 1 Dec 2008 15:30:16 +0200
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <4933DCB1.1070803@ar.media.kyoto-u.ac.jp>
References: <1227987645.8264.200.camel@idol>
	<4932250D.7040003@ar.media.kyoto-u.ac.jp>
	<9457e7c80811301410x333f6352r5fa4e81f4ac628@mail.gmail.com>
	<5b8d13220811301932x7fdf8f0bn72ec63496bad8423@mail.gmail.com>
	<9457e7c80812010448l29c56eceyd9a1adb9e047a1f4@mail.gmail.com>
	<4933DCB1.1070803@ar.media.kyoto-u.ac.jp>
Message-ID: <9457e7c80812010530m6eefb97m6ddd96b16e2694e5@mail.gmail.com>

2008/12/1 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> The only goal of a beta is to see whether we have unexpected failures,
> and solve them until the release. If we add new code, there is no point
> in doing beta. Even without taking into account this, having a window
> where no new feature is allowed is a really common process; even linux,
> which changes extremely rapidly, has a limited time window for merges.

You are preaching to the converted, but just to point out a subtlety:
Linux and most other projects allow contributions to be accepted while
they freeze, albeit to a different branch, tree, etc.  We should have
a similar mechanism in place (we've tried SVN branches without much
success in the past).

> Every project with a proper release management enforce those very
> strictly, because that's the whole point, really. Otherwise, releases
> are nothing more than a snapshot of the source code at a given time

Ideally, releases should simply be snapshots.  In practice, that is
seldom feasible.

> before releases helps that, rushing new code does not. It is not
> theoretical: almost all my time spent on scipy the last few weeks have
> been related to recently added code.

Just because the old code builds, does not mean that there isn't deep
underlying pathology.  But yes, "rushing" code through is a bad idea.
Only, in this case I trust the capabilities of the programmer, and if
he were to provide unit tests, I think we would have been better off.
But let me re-iterate, I am quite happy to postpone, as long as there
is no back-lash upon changing the API just after 0.7 has been
released.

Cheers
St?fan


From josef.pktd at gmail.com  Mon Dec  1 08:39:07 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 1 Dec 2008 08:39:07 -0500
Subject: [SciPy-dev] How to test for easy_install scipy from sources ?
In-Reply-To: <20081201080901.GB18450@phare.normalesup.org>
References: <49338BEC.1090404@ar.media.kyoto-u.ac.jp>
	<20081201080901.GB18450@phare.normalesup.org>
Message-ID: <1cd32cbb0812010539j3d690765j9376207a572972fd@mail.gmail.com>

On Mon, Dec 1, 2008 at 3:09 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Mon, Dec 01, 2008 at 04:02:04PM +0900, David Cournapeau wrote:
>>     I would like to see if I fixed the following issue:
>
>> http://scipy.org/scipy/scipy/ticket/801
>
>> Problem is, I don't know how easy_install is used for something not
>> released on pypi ?
>
> Use the -f switch of easy_install to point to a local directory on your
> box where you build an egg, an simply put a source tar.gz.
>
> Ga?l
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>

I usually just do
 easy_install filename,
when I have either source archive or egg distribution downloaded, or
created myself

on windowsXP:
I have a .bat file in SendTo, with

C:\Programs\Python24\Scripts\easy_install-script.py -U %1

Right click source archive or egg file in Windows Explorer, and send
to this bat file. This will easy_install it.

Josef


From david at ar.media.kyoto-u.ac.jp  Mon Dec  1 08:28:58 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 01 Dec 2008 22:28:58 +0900
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <9457e7c80812010530m6eefb97m6ddd96b16e2694e5@mail.gmail.com>
References: <1227987645.8264.200.camel@idol>	<4932250D.7040003@ar.media.kyoto-u.ac.jp>	<9457e7c80811301410x333f6352r5fa4e81f4ac628@mail.gmail.com>	<5b8d13220811301932x7fdf8f0bn72ec63496bad8423@mail.gmail.com>	<9457e7c80812010448l29c56eceyd9a1adb9e047a1f4@mail.gmail.com>	<4933DCB1.1070803@ar.media.kyoto-u.ac.jp>
	<9457e7c80812010530m6eefb97m6ddd96b16e2694e5@mail.gmail.com>
Message-ID: <4933E69A.8080101@ar.media.kyoto-u.ac.jp>

St?fan van der Walt wrote:
>
> You are preaching to the converted, but just to point out a subtlety:
> Linux and most other projects allow contributions to be accepted while
> they freeze, albeit to a different branch, tree, etc.  We should have
> a similar mechanism in place (we've tried SVN branches without much
> success in the past).
>   

I agree 100 % here; I certainly don't care about the code being put
somewhere in svn which will not be used as the basis for the release
(0.7 in this case).

David


From wnbell at gmail.com  Mon Dec  1 10:03:34 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Mon, 1 Dec 2008 10:03:34 -0500
Subject: [SciPy-dev] Umfpack deprecation warning
In-Reply-To: <gh0ng8$s2i$1@ger.gmane.org>
References: <gh0ng8$s2i$1@ger.gmane.org>
Message-ID: <d05265cb0812010703s789b5917lb7fb25c9a20be135@mail.gmail.com>

On Mon, Dec 1, 2008 at 8:06 AM, Pauli Virtanen <pav at iki.fi> wrote:
>
> This is a bit annoying as it comes up basically for all scripts that
> import Scipy.
>

I don't have umfpack installed, so I can't test it myself.  Please try
the following:

Index: scipy/sparse/linalg/dsolve/linsolve.py
===================================================================
--- scipy/sparse/linalg/dsolve/linsolve.py	(revision 5213)
+++ scipy/sparse/linalg/dsolve/linsolve.py	(working copy)
@@ -15,10 +15,6 @@

 isUmfpack = hasattr( umfpack, 'UMFPACK_OK' )

-if isUmfpack and noScikit:
-    warn( 'scipy.sparse.linalg.dsolve.umfpack will be removed,'
-          ' install scikits.umfpack instead', DeprecationWarning )
-
 useUmfpack = True


@@ -77,6 +73,9 @@


     if isUmfpack and useUmfpack:
+        if noScikit:
+            warn( 'scipy.sparse.linalg.dsolve.umfpack will be removed,'\
+                    ' install scikits.umfpack instead', DeprecationWarning )
         if A.dtype.char not in 'dD':
             raise ValueError, "convert matrix data to double, please, using"\
                   " .astype(), or set linsolve.useUmfpack = False"
@@ -139,6 +138,10 @@
       x2 = solve( rhs2 ) # Uses again the LU factors.
     """
     if isUmfpack and useUmfpack:
+        if noScikit:
+            warn( 'scipy.sparse.linalg.dsolve.umfpack will be removed,'\
+                    ' install scikits.umfpack instead', DeprecationWarning )
+
         if not isspmatrix_csc(A):
             A = csc_matrix(A)
             warn('splu requires CSC matrix format', SparseEfficiencyWarning)



-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From pav at iki.fi  Mon Dec  1 10:43:08 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 1 Dec 2008 15:43:08 +0000 (UTC)
Subject: [SciPy-dev] Umfpack deprecation warning
References: <gh0ng8$s2i$1@ger.gmane.org>
	<d05265cb0812010703s789b5917lb7fb25c9a20be135@mail.gmail.com>
Message-ID: <gh10mc$qh7$1@ger.gmane.org>

Mon, 01 Dec 2008 10:03:34 -0500, Nathan Bell wrote:
> On Mon, Dec 1, 2008 at 8:06 AM, Pauli Virtanen <pav at iki.fi> wrote:
>>
>> This is a bit annoying as it comes up basically for all scripts that
>> import Scipy.
>>
>>
> I don't have umfpack installed, so I can't test it myself.  Please try
> the following:

Excellent, this seems to work. (However, finding the Umfpack scikit is 
not so easy, as the code is only in SVN.)

Btw, would it make sense to raise the warning even though the scikit is 
installed (and remove the umfpack support altogether in the future), and 
modify scikits.umfpack so that the user could call scikits.umfpack.solve?

Can we get this to 0.7.0?

Like so:

>>> import scipy
>>> import scipy.sparse
>>> import scipy.sparse.linalg
>>> a = scipy.sparse.coo_matrix([[1,2],[3,4]])
>>> b = scipy.array([5,6])
>>> scipy.sparse.linalg.spsolve(a, b)
/usr/local/lib/python2.5/site-packages/scipy-0.7.0.dev5203-py2.5-linux-
x86_64.egg/scipy/sparse/linalg/dsolve/linsolve.py:62: 
SparseEfficiencyWarning: spsolve requires CSC or CSR matrix format
  warn('spsolve requires CSC or CSR matrix format', 
SparseEfficiencyWarning)
/usr/local/lib/python2.5/site-packages/scipy-0.7.0.dev5203-py2.5-linux-
x86_64.egg/scipy/sparse/linalg/dsolve/linsolve.py:78: DeprecationWarning: 
scipy.sparse.linalg.dsolve.umfpack will be removed, install 
scikits.umfpack instead
  ' install scikits.umfpack instead', DeprecationWarning )
array([-4. ,  4.5])
>>> scipy.sparse.linalg.spsolve(a, b)
array([-4. ,  4.5])
>>> scipy.sparse.linalg.factorized(a)
/usr/local/lib/python2.5/site-packages/scipy-0.7.0.dev5203-py2.5-linux-
x86_64.egg/scipy/sparse/linalg/dsolve/linsolve.py:143: 
DeprecationWarning: scipy.sparse.linalg.dsolve.umfpack will be removed, 
install scikits.umfpack instead
  ' install scikits.umfpack instead', DeprecationWarning )
/usr/local/lib/python2.5/site-packages/scipy-0.7.0.dev5203-py2.5-linux-
x86_64.egg/scipy/sparse/linalg/dsolve/linsolve.py:147: 
SparseEfficiencyWarning: splu requires CSC matrix format
  warn('splu requires CSC matrix format', SparseEfficiencyWarning)
<function solve at 0x2ba7eadfbc08>
>>> scipy.sparse.linalg.factorized(a)
<function solve at 0x2ba7eadff578>

and after installing the scikit:

>>> import scipy
>>> import scipy.sparse
>>> import scipy.sparse.linalg
>>> a = scipy.sparse.coo_matrix([[1,2],[3,4]])
>>> b = scipy.array([5,6])
>>> scipy.sparse.linalg.spsolve(a, b)
/usr/local/lib/python2.5/site-packages/scipy-0.7.0.dev5203-py2.5-linux-
x86_64.egg/scipy/sparse/linalg/dsolve/linsolve.py:62: 
SparseEfficiencyWarning: spsolve requires CSC or CSR matrix format
  warn('spsolve requires CSC or CSR matrix format', 
SparseEfficiencyWarning)
array([-4. ,  4.5])
>>> scipy.sparse.linalg.factorized(a)
/usr/local/lib/python2.5/site-packages/scipy-0.7.0.dev5203-py2.5-linux-
x86_64.egg/scipy/sparse/linalg/dsolve/linsolve.py:147: 
SparseEfficiencyWarning: splu requires CSC matrix format
  warn('splu requires CSC matrix format', SparseEfficiencyWarning)
<function solve at 0x2ac1391a1488>

-- 
Pauli Virtanen



From josef.pktd at gmail.com  Mon Dec  1 11:06:48 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 1 Dec 2008 11:06:48 -0500
Subject: [SciPy-dev] Warning about remaining issues in stats.distributions ?
Message-ID: <1cd32cbb0812010806x6eb5bdcdt684c404e4e5e8027@mail.gmail.com>

There are still some known failures or methods, that don't work very
well for some distributions:

* skew. kurtosis: a few remaining incorrect results, no exact test available
* entropy: returns nan in some cases, correctness of values not tested
* fit: does not converge, or does not estimate correctly for some distributions,
       this is expected since maximum likelihood does not always work
* distributions that have problems for some range of parameters
* incorrect random number generator for one discrete distribution

Is it useful to put a warning about these issues in
stats.distributions in the release?

A friend of mine is doing value-at-risk calculations for operational
risk analysis, but I am
still reluctant to recommend scipy.stats, when it is not clear whether
the results are correct or
not, although scripting in Python would be much easier than to
struggle with some
commercial package with a GUI user interface. The problem is that in
this application they are
using many distributions automatically, and not just a few that are
well tested by frequent use.

Per Brodtkorb has an improved version of distributions.py with an
additional estimation
procedure, which we can discuss as an enhancement after 0.7 is out of
the way, if there is
an interest in this.

Josef


From pav at iki.fi  Mon Dec  1 12:22:24 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 1 Dec 2008 17:22:24 +0000 (UTC)
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
References: <1227987645.8264.200.camel@idol>
	<4932250D.7040003@ar.media.kyoto-u.ac.jp>
	<9457e7c80811301410x333f6352r5fa4e81f4ac628@mail.gmail.com>
	<5b8d13220811301932x7fdf8f0bn72ec63496bad8423@mail.gmail.com>
	<9457e7c80812010448l29c56eceyd9a1adb9e047a1f4@mail.gmail.com>
Message-ID: <gh16gg$lcu$1@ger.gmane.org>

Mon, 01 Dec 2008 14:48:58 +0200, St?fan van der Walt wrote:
> I'm happy to postpone, but then we should accept that the API may change
> just after 0.7.  It concerns me that non-reviewed code becomes the
> standard quo, and then sets the API for the future.  Also, if Pauli is
> willing to write some tests, we can commit those so long.

To clarify: there are unit tests, cf. test_nonlin.py in the branch, old 
and a couple of new ones. The problem is that these currently only show 
that the algorithms (well, the quasi-Newton ones) find solutions to 
several test problems without  problems, but not really that they work 
optimally.

-- 
Pauli Virtanen



From wnbell at gmail.com  Mon Dec  1 12:23:43 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Mon, 1 Dec 2008 12:23:43 -0500
Subject: [SciPy-dev] Umfpack deprecation warning
In-Reply-To: <gh10mc$qh7$1@ger.gmane.org>
References: <gh0ng8$s2i$1@ger.gmane.org>
	<d05265cb0812010703s789b5917lb7fb25c9a20be135@mail.gmail.com>
	<gh10mc$qh7$1@ger.gmane.org>
Message-ID: <d05265cb0812010923jd0e6d93gf602d222adfdb299@mail.gmail.com>

On Mon, Dec 1, 2008 at 10:43 AM, Pauli Virtanen <pav at iki.fi> wrote:
>
> Btw, would it make sense to raise the warning even though the scikit is
> installed (and remove the umfpack support altogether in the future), and
> modify scikits.umfpack so that the user could call scikits.umfpack.solve?
>

Yes, that would be more sensible.

> Can we get this to 0.7.0?

I don't have any objection to that.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From simpson at math.toronto.edu  Mon Dec  1 12:39:18 2008
From: simpson at math.toronto.edu (Gideon Simpson)
Date: Mon, 1 Dec 2008 12:39:18 -0500
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <ce557a360811301859s5a91a55cid9e3049e42bf5795@mail.gmail.com>
References: <1227987645.8264.200.camel@idol>
	<920F563C-A1BB-487A-81C6-8571C4DC526C@math.toronto.edu>
	<ce557a360811301859s5a91a55cid9e3049e42bf5795@mail.gmail.com>
Message-ID: <DECAFF5E-135F-4999-A678-2342974F122A@math.toronto.edu>

That's a neat trick.  Is that innate to python or a part of scipy?
-gideon

On Nov 30, 2008, at 9:59 PM, Anne Archibald wrote:

> 2008/11/30 Gideon Simpson <simpson at math.toronto.edu>:
>> Still no args input for inputting arguments to the function F?
>>
>> Sorry to complain, but the absence of this has put me off using these
>> routines as it would require a rewrite of much of my code.
>
> Why?
>
> Instead of
> optimize.whatever(F, args=extra)
> just use
> optimize.whatever(lambda x: F(x,extra))
>
>
> Anne
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev



From robert.kern at gmail.com  Mon Dec  1 12:42:22 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 1 Dec 2008 11:42:22 -0600
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <DECAFF5E-135F-4999-A678-2342974F122A@math.toronto.edu>
References: <1227987645.8264.200.camel@idol>
	<920F563C-A1BB-487A-81C6-8571C4DC526C@math.toronto.edu>
	<ce557a360811301859s5a91a55cid9e3049e42bf5795@mail.gmail.com>
	<DECAFF5E-135F-4999-A678-2342974F122A@math.toronto.edu>
Message-ID: <3d375d730812010942y1a5d31c1kcde105857ff601b2@mail.gmail.com>

On Mon, Dec 1, 2008 at 11:39, Gideon Simpson <simpson at math.toronto.edu> wrote:
> That's a neat trick.  Is that innate to python or a part of scipy?

It's a Python thing.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From xavier.gnata at gmail.com  Mon Dec  1 15:31:08 2008
From: xavier.gnata at gmail.com (Xavier Gnata)
Date: Mon, 01 Dec 2008 21:31:08 +0100
Subject: [SciPy-dev] 1 Failure in 0.7.0b1
In-Reply-To: <gh0loi$ko7$1@ger.gmane.org>
References: <c7009a550811252204h2a9ad237p5bba6a5028b6da4@mail.gmail.com>	<ggjil1$7kq$1@ger.gmane.org>
	<49331DE0.1040700@gmail.com>	<5b8d13220811302324i69d45b8bpa817031a79efce40@mail.gmail.com>
	<gh0loi$ko7$1@ger.gmane.org>
Message-ID: <4934498C.8000908@gmail.com>

>> On Mon, Dec 1, 2008 at 8:12 AM, Xavier Gnata <xavier.gnata at gmail.com>
>> wrote:
>>
>>> Ok I can reproduce this one on intrepid ibex 64bits (gfortran).
>>> The funny think is that I think (if I read the correct mathword pages)
>>> that 1.0 is the correct answer (but it is so easy to be wrong with these
>>> special function).
>> I can't reproduce on RHEL 5 (64 bits/python2.4). I opened a ticket:
>>
>> http://scipy.org/scipy/scipy/ticket/803
>>
>> Neal, Xavier, which version of numpy are you using ? Which version of
>> gcc is used in FC 9 ?
>>
>> thanks,
>>
>> David
> : numpy.__version__
> Out[3]: '1.2.0'
> 
> gcc-4.3.0-8.x86_64

It is starting to be a bit fun :)

gcc version 4.3.2 (Ubuntu 4.3.2-1ubuntu11)
no g77, only gfortran
numpy.__version__  '1.3.0.dev6129'
scipy.__version__ '0.7.0.dev5213'

If there is any other test I should do please tell (I have no time to
try to really debug this week but I can perform tests).

Xavier


From bsouthey at gmail.com  Mon Dec  1 17:02:19 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Mon, 01 Dec 2008 16:02:19 -0600
Subject: [SciPy-dev] 1 Failure in 0.7.0b1
In-Reply-To: <4934498C.8000908@gmail.com>
References: <c7009a550811252204h2a9ad237p5bba6a5028b6da4@mail.gmail.com>	<ggjil1$7kq$1@ger.gmane.org>	<49331DE0.1040700@gmail.com>	<5b8d13220811302324i69d45b8bpa817031a79efce40@mail.gmail.com>	<gh0loi$ko7$1@ger.gmane.org>
	<4934498C.8000908@gmail.com>
Message-ID: <49345EEB.8020203@gmail.com>

Xavier Gnata wrote:
>>> On Mon, Dec 1, 2008 at 8:12 AM, Xavier Gnata <xavier.gnata at gmail.com>
>>> wrote:
>>>
>>>       
>>>> Ok I can reproduce this one on intrepid ibex 64bits (gfortran).
>>>> The funny think is that I think (if I read the correct mathword pages)
>>>> that 1.0 is the correct answer (but it is so easy to be wrong with these
>>>> special function).
>>>>         
>>> I can't reproduce on RHEL 5 (64 bits/python2.4). I opened a ticket:
>>>
>>> http://scipy.org/scipy/scipy/ticket/803
>>>
>>> Neal, Xavier, which version of numpy are you using ? Which version of
>>> gcc is used in FC 9 ?
>>>
>>> thanks,
>>>
>>> David
>>>       
>> : numpy.__version__
>> Out[3]: '1.2.0'
>>
>> gcc-4.3.0-8.x86_64
>>     
>
> It is starting to be a bit fun :)
>
> gcc version 4.3.2 (Ubuntu 4.3.2-1ubuntu11)
> no g77, only gfortran
> numpy.__version__  '1.3.0.dev6129'
> scipy.__version__ '0.7.0.dev5213'
>
> If there is any other test I should do please tell (I have no time to
> try to really debug this week but I can perform tests).
>
> Xavier
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>   
Hi
I also get this on my system:
Linux x86-64 Fedora rawhide/10 using Fedora Python 2.5.2 (r252:60911), 
GCC (4.3.2 20080917 (Red Hat 4.3.2-4)), and Fedora supplied blas/atlas 
libraries. My Numpy is '1.3.0.dev6006' or an earlier version of 1.3 but 
Numpy was not built with any blas/atlas libraries. (Yes, scipy will 
build if a site.cfg is provided otherwise it will not because it looks 
at the numpy installation).

It also fails for
Python 2.4.5 built from source using gcc 4.3.2
Python 2.6 (r26:66714, Oct  2 2008, 11:15:01) built from source using 
gcc 4.3.2 - others also fail here.

Bruce





From millman at berkeley.edu  Tue Dec  2 01:16:48 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 1 Dec 2008 22:16:48 -0800
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
Message-ID: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>

Hey,

I would like to get 0.7.0 out ASAP.  Is there any objection to tagging
a release candidate this weekend?  Are there any major regressions or
release blockers that I should be aware of?

Once we are ready to make 0.7.0rc1, I will create a 0.7.x branch and
tag the rc1 from the branch.  At that point the trunk will be open for
0.8 development and only bug-fixes will be allowed on the 0.7.x
branch.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From opossumnano at gmail.com  Tue Dec  2 04:52:41 2008
From: opossumnano at gmail.com (Tiziano Zito)
Date: Tue, 2 Dec 2008 10:52:41 +0100
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
Message-ID: <20081202095241.GA4699@localhost>

> I would like to get 0.7.0 out ASAP.  Is there any objection to tagging
> a release candidate this weekend?  Are there any major regressions or
> release blockers that I should be aware of?
> 

well, what about
#800 : numpy's test_poly1d_nan_roots is broken if scipy.linalg is imported
#243: linalg.qr to take mode argument
both have one or more fixes possible, a decision is needed.

another IMO seriuos problem:
#794: linalg.eig segmentation fault on unpickled arrays
there's no fix to that one.

#630: Support for generalized symmetric eigenvalue problem in SciPy
can be closed.

#593: flapack.error: ((lwork==-1) || (lwork>=MAX(1,8*n))) failed for 3rd keyword lwork
can be closed.

#632: Matrix gallery aka test matrices
should be scheduled for 0.8 (I also have some code for that).

#585: scipy.linalg.pinv() returns array when input is matrix
the fix seems easy in this case, but I think a decision is needed.

#492: Problem in linalg.svd ?
no fix. (but does the problem still exists?)

I can work on some of these tickets if needed.

ciao,
tiziano



From malkarouri at yahoo.co.uk  Tue Dec  2 09:16:15 2008
From: malkarouri at yahoo.co.uk (Muhammad Alkarouri)
Date: Tue, 2 Dec 2008 14:16:15 +0000 (GMT)
Subject: [SciPy-dev] Scipy not compiling in OS X 10.4
In-Reply-To: <c7009a550811261936r2d11e201g7b42e129048ad5f@mail.gmail.com>
Message-ID: <361031.75207.qm@web27903.mail.ukl.yahoo.com>

Thanks Jarrod and David, and sorry for the late reply.
I was using Python 2.6 and I tripped over some problems there. I am now using Python 2.5 on OS X 10.4.11, Numpy 1.2.1 and Scipy works fine.

In [3]: scipy.test()

#details omitted

Ran 3307 tests in 95.752s

OK (KNOWNFAIL=2, SKIP=32)

Out[3]: <nose.result.TextTestResult run=3307 errors=0 failures=0>

In [4]: scipy.__version__

Out[4]: '0.7.0.dev5213'

SciPy was installed after grabbing dev5213 and doing 'python setup.py sdist' then installing from the tarball.

Many thanks,

Muhammad


      


From chanley at stsci.edu  Tue Dec  2 11:04:36 2008
From: chanley at stsci.edu (Christopher Hanley)
Date: Tue, 02 Dec 2008 11:04:36 -0500
Subject: [SciPy-dev] astropy.scipy.org
Message-ID: <49355C94.7030006@stsci.edu>

Hi,

The site astropy.scipy.org appears to be offline.  I am getting a Fedora 
Core Test Page.

Chris

-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From chanley at stsci.edu  Tue Dec  2 11:30:35 2008
From: chanley at stsci.edu (Christopher Hanley)
Date: Tue, 02 Dec 2008 11:30:35 -0500
Subject: [SciPy-dev] astropy.scipy.org
In-Reply-To: <49355C94.7030006@stsci.edu>
References: <49355C94.7030006@stsci.edu>
Message-ID: <493562AB.4080004@stsci.edu>

Christopher Hanley wrote:
> Hi,
> 
> The site astropy.scipy.org appears to be offline.  I am getting a Fedora 
> Core Test Page.
> 
> Chris
> 

Thank you.  The SVN seems to be happy again.

Chris


-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From ondrej at certik.cz  Tue Dec  2 20:19:15 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 3 Dec 2008 02:19:15 +0100
Subject: [SciPy-dev] license for code published in articles
In-Reply-To: <3d375d730811241222h23a06153s9d34848962559e3a@mail.gmail.com>
References: <1cd32cbb0811241217o34389c4dqee28e819f3758e28@mail.gmail.com>
	<3d375d730811241222h23a06153s9d34848962559e3a@mail.gmail.com>
Message-ID: <85b5c3130812021719r250a263epaf5a303431ab7ab1@mail.gmail.com>

On Mon, Nov 24, 2008 at 9:22 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Nov 24, 2008 at 14:17,  <josef.pktd at gmail.com> wrote:
>> Is code that is published verbatim inside a paper (not as attachment)
>> license restricted?
>
> Yes, of course, provided that the code is otherwise copyrightable.
>
>> The Journal of Statistical Software publishes code under GPL.
>> Does GPL also apply to source snippets or functions that are included
>> in the text of the article, or is there a fair use assumption on the
>> text of the article?
>
> Including code in scipy does not fall under fair use, so that's
> neither here nor there.
>
> You can ask the authors for permission, if you like, but it's best
> just to read the paper, and write new code.

Puting it the other way round --- when writing a paper I should
explicitely state, that all code is BSD, then people should not be
afraid of looking at it, right?

Ondrej


From robert.kern at gmail.com  Tue Dec  2 20:37:38 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 2 Dec 2008 19:37:38 -0600
Subject: [SciPy-dev] license for code published in articles
In-Reply-To: <85b5c3130812021719r250a263epaf5a303431ab7ab1@mail.gmail.com>
References: <1cd32cbb0811241217o34389c4dqee28e819f3758e28@mail.gmail.com>
	<3d375d730811241222h23a06153s9d34848962559e3a@mail.gmail.com>
	<85b5c3130812021719r250a263epaf5a303431ab7ab1@mail.gmail.com>
Message-ID: <3d375d730812021737s49c5c1faq661fcfe7630e3f2c@mail.gmail.com>

On Tue, Dec 2, 2008 at 19:19, Ondrej Certik <ondrej at certik.cz> wrote:
> On Mon, Nov 24, 2008 at 9:22 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Mon, Nov 24, 2008 at 14:17,  <josef.pktd at gmail.com> wrote:
>>> Is code that is published verbatim inside a paper (not as attachment)
>>> license restricted?
>>
>> Yes, of course, provided that the code is otherwise copyrightable.
>>
>>> The Journal of Statistical Software publishes code under GPL.
>>> Does GPL also apply to source snippets or functions that are included
>>> in the text of the article, or is there a fair use assumption on the
>>> text of the article?
>>
>> Including code in scipy does not fall under fair use, so that's
>> neither here nor there.
>>
>> You can ask the authors for permission, if you like, but it's best
>> just to read the paper, and write new code.
>
> Puting it the other way round --- when writing a paper I should
> explicitely state, that all code is BSD, then people should not be
> afraid of looking at it, right?

You might have to arrange such matters with the journal, which might
get sticky. Probably the best way is to have tarballs of the code in
the paper available for download with a LICENSE.txt. But I have no
direct experience negotiating with journals, so take that for what
it's worth.

But in general, people shouldn't be afraid of looking at code snippets
in papers. If the code is short enough to be in a paper, verbatim, it
is likely short enough to not be copyrightable by itself. And you
probably don't want to copy the code directly, anyways.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From eads at soe.ucsc.edu  Tue Dec  2 20:48:31 2008
From: eads at soe.ucsc.edu (Damian Eads)
Date: Tue, 2 Dec 2008 17:48:31 -0800
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
Message-ID: <91b4b1ab0812021748m484ceaf4tadcf9f8745537285@mail.gmail.com>

Is what's on the trunk now at the time you tag 0.7.0rc1 going into the
release? I ask because I added some regression tests to the trunk
after you tagged the 0.7.0b1. This is the first release I am a part of
so I'm curious if we are leaving open the possibility of more regular
0.7.x minor releases.

Damian

On 12/1/08, Jarrod Millman <millman at berkeley.edu> wrote:
> Hey,
>
> I would like to get 0.7.0 out ASAP.  Is there any objection to tagging
> a release candidate this weekend?  Are there any major regressions or
> release blockers that I should be aware of?
>
> Once we are ready to make 0.7.0rc1, I will create a 0.7.x branch and
> tag the rc1 from the branch.  At that point the trunk will be open for
> 0.8 development and only bug-fixes will be allowed on the 0.7.x
> branch.
>
> Thanks,
>
> --
> Jarrod Millman
> Computational Infrastructure for Research Labs
> 10 Giannini Hall, UC Berkeley
> phone: 510.643.4014
> http://cirl.berkeley.edu/
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>

-- 
Sent from my mobile device

-----------------------------------------------------
Damian Eads                             Ph.D. Student
Jack Baskin School of Engineering, UCSC        E2-489
1156 High Street                 Machine Learning Lab
Santa Cruz, CA 95064    http://www.soe.ucsc.edu/~eads


From hardbyte at gmail.com  Tue Dec  2 21:36:53 2008
From: hardbyte at gmail.com (Brian Thorne)
Date: Wed, 3 Dec 2008 15:36:53 +1300
Subject: [SciPy-dev] Possible bug in distutils compiler when using scipy's
	weave on windows
Message-ID: <1f10aea40812021836m6e1e0eeax8d778712436a78e7@mail.gmail.com>

I am running python(x,y) which installs python and scipy and weave
under the "program files" folder on windows. When using weave by
having some inline C/C++ it uses a modified distutils to compile the
inline source right?
Unfortunately it doesn't seem to handle spaces in the path to either
the output or to the temporary source files.
I checked that this was the problem by copying the weave directory
from site-packages and put it in another directory without spaces in
the path - this worked fine.
I then tried the exact g++ command from the stack trace at the command
line and got the same error - so altered the command by adding quotes
to the  -c "path/to source/a.c" and the -o "objects".
This worked at the command line so I set out changing it in weave...
After spending a while debugging I found where quotes were added to
the included libraries. After a lot more tracing found where the
command line gets stringed together for the linker - in
unixcompiler.py (surprising as I am under windows!)
So I changed it to quote the source and the output objects and now it goes fine.

What I changed was:
In distutils.cygwinccompiler __revision__ = "$Id: cygwinccompiler.py
37828 2004-11-10 22:23:15Z loewis $":

> def _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts):
>         if ext == '.rc' or ext == '.res':
>             # gcc needs '.res' and '.rc' compiled to object files !!!
>             try:
>                 self.spawn(["windres", "-i", src, "-o", obj])
>             except DistutilsExecError, msg:
>                 raise CompileError, msg
>         else: # for other files use the C-compiler
>             try:
>                 lis = ['"' + src + '"', '-o', '"' + obj + '"']
>                 cmd = self.compiler_so + cc_args + lis + extra_postargs
>                 self.spawn(cmd)
>             except DistutilsExecError, msg:
>                 raise CompileError, msg

In distutils.unixccompiler __revision__ = "$Id: unixccompiler.py 54954
2007-04-25 06:42:41Z neal.norwitz $":

> def link(self, target_desc, objects,
>              output_filename, output_dir=None, libraries=None,
>              library_dirs=None, runtime_library_dirs=None,
>              export_symbols=None, debug=0, extra_preargs=None,
>              extra_postargs=None, build_temp=None, target_lang=None):
>         objects, output_dir = self._fix_object_args(objects, output_dir)
>         libraries, library_dirs, runtime_library_dirs = \
>             self._fix_lib_args(libraries, library_dirs, runtime_library_dirs)
>
>         lib_opts = gen_lib_options(self, library_dirs, runtime_library_dirs, libraries)
>         if type(output_dir) not in (StringType, NoneType):
>             raise TypeError, "'output_dir' must be a string or None"
>         if output_dir is not None:
>             output_filename = os.path.join(output_dir, output_filename)
>
>         if self._need_link(objects, output_filename):
>             objects = ['"' + o + '"' for o in objects]
>             self.objects = ['"' + o + '"' for o in self.objects]
>             ld_args = (objects + self.objects +
>                        lib_opts + ['-o', '"' + output_filename + '"'])
>             if debug:
>                 ld_args[:0] = ['-g']
>             if extra_preargs:
>                 ld_args[:0] = extra_preargs

I don't know if that would still work on gnu/linux systems tho...
Brian Thorne


From millman at berkeley.edu  Tue Dec  2 22:16:42 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 2 Dec 2008 19:16:42 -0800
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <91b4b1ab0812021748m484ceaf4tadcf9f8745537285@mail.gmail.com>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
	<91b4b1ab0812021748m484ceaf4tadcf9f8745537285@mail.gmail.com>
Message-ID: <c7009a550812021916h57e6dc01j1bc3d11f57487c60@mail.gmail.com>

On Tue, Dec 2, 2008 at 5:48 PM, Damian Eads <eads at soe.ucsc.edu> wrote:
> Is what's on the trunk now at the time you tag 0.7.0rc1 going into the
> release? I ask because I added some regression tests to the trunk
> after you tagged the 0.7.0b1. This is the first release I am a part of
> so I'm curious if we are leaving open the possibility of more regular
> 0.7.x minor releases.

Once we are ready to make a release candidate for 0.7.0, I will create
a 0.7.x branch from the trunk.  At that point the trunk will be
reopened for new features and will where we work on the next minor
release (i.e., SciPy 0.8.x).

I will then tag 0.7.0rc1 from the 0.7.x branch.  If there are any
major regressions or release blockers found during the testing of the
0.7.0rc1, fixes should be applied to the trunk and back-ported to the
0.7.x branch.  If necessary, I will tag 0.7.0rc2 from the fixed 0.7.x
branch; otherwise, I will tag 0.7.0 from the 0.7.x branch.

At that point, attention will focus on the trunk.  Ideally, most
bug-fixes should be back-ported to the 0.7.x branch.  I would like to
release at least a couple of bug-fix releases (i.e., 0.7.1, 0.7.2,
etc.).  Ideally, I would love to see us getting monthly bug-fix
releases out.  Even if there are a just a few bugs fixed on the
branch, I think it would be reasonable to make a release.  (If you or
anyone else is willing to fix bugs on the trunk and back-port them to
the 0.7.x branch, I will be happy to help get a bug-fix release out.)

I would also like to be more disciplined with getting minor releases
out more regularly.  It seems that a 6 month feature release is a
reasonable aim.  Once we get 0.7.0 released, I hope that we can have a
dicsussion about our intentions for the 0.8 release and come to a
consensus on a release schedule.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Tue Dec  2 22:48:55 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 2 Dec 2008 19:48:55 -0800
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <20081202095241.GA4699@localhost>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
	<20081202095241.GA4699@localhost>
Message-ID: <c7009a550812021948h12482038rf6178557f99c66a2@mail.gmail.com>

On Tue, Dec 2, 2008 at 1:52 AM, Tiziano Zito <opossumnano at gmail.com> wrote:
> #800 : numpy's test_poly1d_nan_roots is broken if scipy.linalg is imported
> #243: linalg.qr to take mode argument
> both have one or more fixes possible, a decision is needed.
>
> another IMO seriuos problem:
> #794: linalg.eig segmentation fault on unpickled arrays
> there's no fix to that one.

I don't think we are going to be able to close these tickets in time
for the 0.7.0 release, but I am not sure whether they should be
considered release blockers.  If you disagree, please let me know.

#800 -- I think we should leave scipy as is and consider this a numpy
bug and fix numpy (i.e., use asarray_chkfinite in numpy.linalg.eig)

#243 -- I think that we should make some changes to qr, but I don't
think there is a general agreement on what we should do.  There is
another thread about this, which I will look at again.

#794 -- I am not sure what to do about this, but since pickling arrays
is discouraged I don't think we should hold the release for this.  Of
course, it would be great if you could resolve this in the next few
days.

> #630: Support for generalized symmetric eigenvalue problem in SciPy
> can be closed.

Please close this ticket.  Your account (tzito) should now have
permission.  If not, please let me know.

> #593: flapack.error: ((lwork==-1) || (lwork>=MAX(1,8*n))) failed for 3rd keyword lwork
> can be closed.

Please close this ticket.

> #632: Matrix gallery aka test matrices
> should be scheduled for 0.8 (I also have some code for that).

Please feel free to take this ticket.  I for one would be interested
in seeing this in 0.8.

> #585: scipy.linalg.pinv() returns array when input is matrix
> the fix seems easy in this case, but I think a decision is needed.

I think everything should be an array (i.e., we should remove
matrices), so I would rather leave this as is.  But I am sure that
many people will disagree with me....

> #492: Problem in linalg.svd ?
> no fix. (but does the problem still exists?)

Not sure.  But I don't think this should hold up the release.

> I can work on some of these tickets if needed.

While I don't think that any of these things constitute grounds for
postponing the release candidate, I would love to see as many tickets
closed as possible.  If you have any other tickets that should be
reviewed, please let us know.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Tue Dec  2 23:12:58 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 2 Dec 2008 20:12:58 -0800
Subject: [SciPy-dev] Warning about remaining issues in
	stats.distributions ?
In-Reply-To: <1cd32cbb0812010806x6eb5bdcdt684c404e4e5e8027@mail.gmail.com>
References: <1cd32cbb0812010806x6eb5bdcdt684c404e4e5e8027@mail.gmail.com>
Message-ID: <c7009a550812022012t184f2d31i420b6ba5b10b5e41@mail.gmail.com>

On Mon, Dec 1, 2008 at 8:06 AM,  <josef.pktd at gmail.com> wrote:
> There are still some known failures or methods, that don't work very
> well for some distributions:
>
> * skew. kurtosis: a few remaining incorrect results, no exact test available
> * entropy: returns nan in some cases, correctness of values not tested
> * fit: does not converge, or does not estimate correctly for some distributions,
>       this is expected since maximum likelihood does not always work
> * distributions that have problems for some range of parameters
> * incorrect random number generator for one discrete distribution
>
> Is it useful to put a warning about these issues in
> stats.distributions in the release?

It would be great to add something like this to the docstrings (as
long as you weren't proposing to raise a warning).

> Per Brodtkorb has an improved version of distributions.py with an
> additional estimation
> procedure, which we can discuss as an enhancement after 0.7 is out of
> the way, if there is
> an interest in this.

That would be great.  Once I create the 0.7.x branch, the trunk will
be open for 0.8 development.  Hopefully we can get this in ASAP in the
0.8 development.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From cournape at gmail.com  Wed Dec  3 02:09:49 2008
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 3 Dec 2008 16:09:49 +0900
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <20081202095241.GA4699@localhost>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
	<20081202095241.GA4699@localhost>
Message-ID: <5b8d13220812022309r40e0d810kce8918ed17ab12f2@mail.gmail.com>

On Tue, Dec 2, 2008 at 6:52 PM, Tiziano Zito <opossumnano at gmail.com> wrote:
>> I would like to get 0.7.0 out ASAP.  Is there any objection to tagging
>> a release candidate this weekend?  Are there any major regressions or
>> release blockers that I should be aware of?
>>
>
> well, what about
> #800 : numpy's test_poly1d_nan_roots is broken if scipy.linalg is imported
> #243: linalg.qr to take mode argument
> both have one or more fixes possible, a decision is needed.

The fix will not be in 0.7, but in 0.8. 0.7 will only warn that the
API will change in 0.8. I will close/update the other tickets
according to your comments, thanks for the help.

I will need to find a fix for scipy to work on machines without SSE,
too, I am not sure about the best way to solve it (the problem is in
numpy.distutils, so I may need to do something dirty to circumvent
it).

cheers,

David


From tom.grydeland at gmail.com  Wed Dec  3 02:57:59 2008
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Wed, 3 Dec 2008 08:57:59 +0100
Subject: [SciPy-dev] Hankel transform
Message-ID: <c21c0b2f0812022357p61a32c87j8bd4dde244caf309@mail.gmail.com>

Hi developers,

When I recently needed a Hankel transform I was unable to find one in
Scipy.  I found one in MATLAB though[1], written by Prof. Brian
Borchers at New Mexico Tech. The code is based on a previous FORTRAN
implementation by W. L. Anderson [2], and the MATLAB code is not
marked with any copyright statements.  Hankel transforms of the first
and second order can be computed through digital filtering.

I have rewritten the code in Python/numpy, complete with tests and
acknowledgements of the origins, and my employer has agreed that I can
donate the code to Scipy.  I believe this should be of interest.
Hankel transforms arise often in acoustics and other systems with
cylinder symmetry.  If you want it, I would like a suggestion where it
belongs (with other integral transforms, probably) and how I should
shape the tests to make them suitable for inclusion.

The tests I currently have use Hankel transforms of five different
functions with known analytical transforms and compares the
transformed values to the numerically evaluated analytical
expressions.  Currently plots are generated, but for automated testing
I suppose something else would be better.  Pointing me at an example
is sufficient.

[1] http://infohost.nmt.edu/~borchers/hankel.html
[2] Anderson, W. L., 1979, Computer Program Numerical Integration of
Related Hankel Transforms of Orders 0 and 1 by Adaptive Digital
Filtering. Geophysics, 44(7):1287-1305.

Best regards,

-- 
Tom Grydeland
  <Tom.Grydeland@(discoverpetroleum.com|gmail.com)>


From millman at berkeley.edu  Wed Dec  3 05:19:15 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 3 Dec 2008 02:19:15 -0800
Subject: [SciPy-dev] stats.cov versus np.cov
In-Reply-To: <1cd32cbb0811240912x8161a68k68cd7dc1d5adb93e@mail.gmail.com>
References: <1cd32cbb0811240912x8161a68k68cd7dc1d5adb93e@mail.gmail.com>
Message-ID: <c7009a550812030219x89fac75v3596cb0b29c7ca69@mail.gmail.com>

On Mon, Nov 24, 2008 at 9:12 AM,  <josef.pktd at gmail.com> wrote:
> in http://scipy.org/scipy/scipy/ticket/425, stats mean, var and median
> got depreciated.
>
> Should this also be done for cov?

+1
If no one objects in the next day, would you add the deprecation warning?

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From stefan at sun.ac.za  Wed Dec  3 11:30:12 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 3 Dec 2008 18:30:12 +0200
Subject: [SciPy-dev] #693 (Evaluating bivariate piecewise splines at
	arbitrary points)
In-Reply-To: <ggn14f$ibr$3@ger.gmane.org>
References: <ggn14f$ibr$3@ger.gmane.org>
Message-ID: <9457e7c80812030830q46a29babh61b4b7f2de827e44@mail.gmail.com>

Hi Pauli

2008/11/27 Pauli Virtanen <pav at iki.fi>:
> Do we want to get this:
>
>        http://www.scipy.org/scipy/scipy/ticket/693
>
> to Scipy 0.7.0, or does it need improvements etc.?
>
> It's pretty straightforward; probably the only question is whether
>
>        >>> zi = spline_object.ev(xi, yi)
>
> is a good interface for getting zi[j] = spline(xi[j], yi[j]). [As opposed
> to BivariateSpline.__call__(x,y), which returns answers evaluated on
> meshgrid(x,y)]

Did this make it into 0.7?  It looks like a sensible change.

Cheers
St?fan


From stefan at sun.ac.za  Wed Dec  3 11:38:32 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 3 Dec 2008 18:38:32 +0200
Subject: [SciPy-dev] two tickets with patches
In-Reply-To: <c7009a550811261931k461c1606id2f4a3b0fed3d31c@mail.gmail.com>
References: <c7009a550811261931k461c1606id2f4a3b0fed3d31c@mail.gmail.com>
Message-ID: <9457e7c80812030838v716376d4w2e8342c079831e25@mail.gmail.com>

2008/11/27 Jarrod Millman <millman at berkeley.edu>:
> There are two tickets that contain donated code, which have been
> sitting around for over a year or more:
>  http://scipy.org/scipy/scipy/ticket/354
>  http://scipy.org/scipy/scipy/ticket/457
>
> I would like to see at least some comments on them about why they
> haven't been accepted.  They are both first attempts but no one has
> made any critical comments or useful suggestions for what the authors
> need to do in order to have their code accepted.  One provides support
> the Harwell-Boeing format and the other provides a FIR filter.

Do you know if the author of the FIR filter code ("tash") is still
around?  If we can verify that the code does what it is supposed to, I
think this would make a good addition to SciPy.

Regards
St?fan


From pav at iki.fi  Wed Dec  3 11:42:24 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 3 Dec 2008 16:42:24 +0000 (UTC)
Subject: [SciPy-dev] #693 (Evaluating bivariate piecewise splines
	at	arbitrary points)
References: <ggn14f$ibr$3@ger.gmane.org>
	<9457e7c80812030830q46a29babh61b4b7f2de827e44@mail.gmail.com>
Message-ID: <gh6ctg$3bd$1@ger.gmane.org>

Wed, 03 Dec 2008 18:30:12 +0200, St?fan van der Walt wrote:
> 2008/11/27 Pauli Virtanen <pav at iki.fi>:
>> Do we want to get this:
>>
>>        http://www.scipy.org/scipy/scipy/ticket/693
>>
>> to Scipy 0.7.0, or does it need improvements etc.?
>>
>> It's pretty straightforward; probably the only question is whether
>>
>>        >>> zi = spline_object.ev(xi, yi)
>>
>> is a good interface for getting zi[j] = spline(xi[j], yi[j]). [As
>> opposed to BivariateSpline.__call__(x,y), which returns answers
>> evaluated on meshgrid(x,y)]
> 
> Did this make it into 0.7?  It looks like a sensible change.

It's not committed.

	Pauli



From jh at physics.ucf.edu  Wed Dec  3 11:42:31 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Wed, 03 Dec 2008 11:42:31 -0500
Subject: [SciPy-dev] Docs: it's not over yet! (and T-shirts)
Message-ID: <wl88wqxtdl4.fsf@glup.physics.ucf.edu>

Thanks to all the volunteers who have contributed docs in the Doc
Marathon.  The past month has seen an increase in activity,
particularly for numpy.ma and the ufuncs.  The PDF manual is now 445
pages.  This is fantastic!  But there's still much work to do.

Here's the status of numpy documentation:

State			 %  	Count
Needs editing		38 	 374
Being written / Changed	30 	 296
Needs review	 	27 	 264
Needs review (revised) 	 2 	  20
Needs work (reviewed) 	 1 	   8
Reviewed (needs proof) 	 1 	   5
Proofed		 	 3 	  26
Unimportant	 	 ? 	1488

So, while about 1/3 of the items have docs that someone thinks might
be done ("Needs Review" or higher), 38% of the pages have not even
been touched since the start of the marathon, including stuff in fft,
lots more in ma, even some ndarray pages.

Like these functions?  Use them?  Want others to use them?  I hope
that the semester/holiday break is a good chance for everyone to write
some more.  You don't need to be a developer to pitch in, just go to
docs.scipy.org, click into the editing wiki, read the front page and
guidelines, make a wiki name, and email this list to get edit
privileges.  There's a link to the source code on each function's doc
wiki page, so you can easily refer to it when writing.

We promised a cool T-shirt to everyone who has written 1000 words on
the doc wiki.  As of a few weeks ago, I think we are caught up on
sending shirts out.  If you think you should get a shirt but haven't,
please email me (off list).

Thanks!

--jh--
Prof. Joseph Harrington
Department of Physics
MAP 414
4000 Central Florida Blvd.
University of Central Florida
Orlando, FL 32816-2385
(407) 823-3416 voice
(407) 823-5112 fax
(407) 823-2325 physics office
jh at physics.ucf.edu


From charlesr.harris at gmail.com  Wed Dec  3 12:02:00 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 3 Dec 2008 10:02:00 -0700
Subject: [SciPy-dev] Docs: it's not over yet! (and T-shirts)
In-Reply-To: <wl88wqxtdl4.fsf@glup.physics.ucf.edu>
References: <wl88wqxtdl4.fsf@glup.physics.ucf.edu>
Message-ID: <e06186140812030902y39fae983s64cc7c8147e256c6@mail.gmail.com>

On Wed, Dec 3, 2008 at 9:42 AM, <jh at physics.ucf.edu> wrote:

> Thanks to all the volunteers who have contributed docs in the Doc
> Marathon.  The past month has seen an increase in activity,
> particularly for numpy.ma and the ufuncs.  The PDF manual is now 445
> pages.  This is fantastic!  But there's still much work to do.
>

The document site needs to be merged with the trunk as the two are out of
sync. Because new functions don't show up on the web site I document them in
svn. Merges should happen daily, IMHO.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081203/a7e36512/attachment.html>

From nwagner at iam.uni-stuttgart.de  Wed Dec  3 14:34:46 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Wed, 03 Dec 2008 20:34:46 +0100
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <20081202095241.GA4699@localhost>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
	<20081202095241.GA4699@localhost>
Message-ID: <web-114726230@uni-stuttgart.de>

On Tue, 2 Dec 2008 10:52:41 +0100
  Tiziano Zito <opossumnano at gmail.com> wrote:
>> I would like to get 0.7.0 out ASAP.  Is there any 
>>objection to tagging
>> a release candidate this weekend?  Are there any major 
>>regressions or
>> release blockers that I should be aware of?
>> 
> 
> well, what about
> #800 : numpy's test_poly1d_nan_roots is broken if 
>scipy.linalg is imported
> #243: linalg.qr to take mode argument
> both have one or more fixes possible, a decision is 
>needed.
> 
> another IMO seriuos problem:
> #794: linalg.eig segmentation fault on unpickled arrays
> there's no fix to that one.
> 
> #630: Support for generalized symmetric eigenvalue 
>problem in SciPy
> can be closed.
> 
> #593: flapack.error: ((lwork==-1) || 
>(lwork>=MAX(1,8*n))) failed for 3rd keyword lwork
> can be closed.
> 
> #632: Matrix gallery aka test matrices
> should be scheduled for 0.8 (I also have some code for 
>that).
> 
> #585: scipy.linalg.pinv() returns array when input is 
>matrix
> the fix seems easy in this case, but I think a decision 
>is needed.
> 
> #492: Problem in linalg.svd ?
> no fix. (but does the problem still exists?)
> 
> I can work on some of these tickets if needed.
> 
> ciao,
> tiziano
> 
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev

Ticket

http://projects.scipy.org/scipy/scipy/ticket/795

can be closed.

  Nils

  


From pav at iki.fi  Wed Dec  3 15:00:53 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 3 Dec 2008 20:00:53 +0000 (UTC)
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
Message-ID: <gh6ohl$dg6$1@ger.gmane.org>

Mon, 01 Dec 2008 22:16:48 -0800, Jarrod Millman wrote:
> I would like to get 0.7.0 out ASAP.  Is there any objection to tagging a
> release candidate this weekend?  Are there any major regressions or
> release blockers that I should be aware of?

Maybe this should be resolved before the release (from the users list):

Matthew Brett wrote:
> > def ks_2samp(data1, data2):
> >     """ Computes the Kolmogorov-Smirnof statistic on 2 samples.  Modified
> >     from Numerical Recipies in C, page 493.  Returns KS D-value, prob.  Not
> >     ufunc- like.
>
> Wait - really?  We can't use Numerical Recipes code, it has strict and
> incompatible licensing...  If it's in there it really has to come out as
> fast as possible.

According to svn log, the code for these functions originates from way
back, r71 travo 2001-06-25 18:20:52. Apparently much of the module is
written using NR as a reference, though I haven't checked if the code
is similar.

-- 
Pauli Virtanen



From josef.pktd at gmail.com  Wed Dec  3 15:23:55 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 3 Dec 2008 15:23:55 -0500
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <gh6ohl$dg6$1@ger.gmane.org>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
	<gh6ohl$dg6$1@ger.gmane.org>
Message-ID: <1cd32cbb0812031223l2e46e519l5cafe45e420624ed@mail.gmail.com>

On Wed, Dec 3, 2008 at 3:00 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Mon, 01 Dec 2008 22:16:48 -0800, Jarrod Millman wrote:
>> I would like to get 0.7.0 out ASAP.  Is there any objection to tagging a
>> release candidate this weekend?  Are there any major regressions or
>> release blockers that I should be aware of?
>
> Maybe this should be resolved before the release (from the users list):
>
> Matthew Brett wrote:
>> > def ks_2samp(data1, data2):
>> >     """ Computes the Kolmogorov-Smirnof statistic on 2 samples.  Modified
>> >     from Numerical Recipies in C, page 493.  Returns KS D-value, prob.  Not
>> >     ufunc- like.
>>
>> Wait - really?  We can't use Numerical Recipes code, it has strict and
>> incompatible licensing...  If it's in there it really has to come out as
>> fast as possible.
>
> According to svn log, the code for these functions originates from way
> back, r71 travo 2001-06-25 18:20:52. Apparently much of the module is
> written using NR as a reference, though I haven't checked if the code
> is similar.
>
> --
> Pauli Virtanen
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>
There is one more case in stats.stats:

def ttest_rel(a,b,axis=None):
    """Calculates the t-obtained T-test on TWO RELATED samples of scores, a
    and b.  From Numerical Recipies, p.483. Axis can equal None (ravel array
    first), or an integer (the axis over which to operate on a and b).

    Returns: t-value, two-tailed p-value
    """

In both cases it is just a few lines, and I assume the overall
algorithm changed through the adjustment to numpy/scipy
I would just drop the reference to Numerical Recipies, the algorithm
should be pretty generic.

I haven't looked in these 2 cases, but for the one-sample ks-test, in
the ticket and in the newsgroup the discussion refered to the
numerical recipes. I never looked at NR, but the proposed correction
was exactly the same as in any reference that I found on the internet,
or in articles.

Josef


From millman at berkeley.edu  Wed Dec  3 15:35:44 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 3 Dec 2008 12:35:44 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
Message-ID: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>

Given the discussion about Numerical Recipes derived-code, I did a
little grepping and found 7 instances where code is referencing
Numerical Recipes.  Before branching for 0.7, I would like to see this
resolved.  I am not sure what we should do, but the license for using
Numerical Recipes derived code clearly does not permit us to include
it in scipy:
  http://www.nr.com/licenses/redistribute.html

Below I include all the instances I could easily find.  Thoughts?

--------------------------------------------------

In scipy/optimize/zeros.py:

def ridder(f, a, b, args=(),
           xtol=_xtol, rtol=_rtol, maxiter=_iter,
           full_output=False, disp=False):
    """Find root of f in [a,b]

    Ridder routine to find a zero of the function
    f between the arguments a and b. f(a) and f(b) can not
    have the same signs. Faster than bisection, but not
    generaly as fast as the brent rountines. A description
    may be found in a recent edition of Numerical Recipes.
    The routine here is modified a bit to be more careful
    of tolerance.

def brentq(f, a, b, args=(),
           xtol=_xtol, rtol=_rtol, maxiter=_iter,
           full_output=False, disp=False):
    """Find root of f in [a,b]

    The classic Brent routine to find a zero of the function
    f between the arguments a and b. f(a) and f(b) can not
    have the same signs. Generally the best of the routines here.
    It is a safe version of the secant method that uses inverse
    quadratic extrapolation. The version here is a slight
    modification that uses a different formula in the extrapolation
    step. A description may be found in Numerical Recipes, but the
    code here is probably easier to understand.

In scipy/signal/medianfilter.c:

/*
 *  This Quickselect routine is based on the algorithm described in
 *  "Numerical recipies in C", Second Edition,
 *  Cambridge University Press, 1992, Section 8.5, ISBN 0-521-43108-5
 */

In  scipy/special/orthogonal.py:

"""
A collection of functions to find the weights and abscissas for
Gaussian Quadrature.

These calculations are done by finding the eigenvalues of a
tridiagonal matrix whose entries are dependent on the coefficients
in the recursion formula for the orthogonal polynomials with the
corresponding weighting function over the interval.

Many recursion relations for orthogonal polynomials are given:

a1n f_n+1 (x) = (a2n + a3n x ) f_n (x) - a4n f_n-1 (x)

The recursion relation of interest is

P_n+1 (x) = (x - A_n) P_n (x) - B_n P_n-1 (x)

where P has a different normalization than f.

The coefficients can be found as:

A_n = -a2n / a3n

B_n = ( a4n / a3n sqrt(h_n-1 / h_n))**2

     where
             h_n = int_a^b w(x) f_n(x)^2
assume:
P_0(x) = 1
P_-1(x) == 0

See Numerical Recipies in C, page 156 and
Abramowitz and Stegun p. 774, 782

In scipy/stats/stats.py:

def ttest_ind(a, b, axis=0):
    """Calculates the t-obtained T-test on TWO INDEPENDENT samples of scores
    a, and b.  From Numerical Recipies, p.483. Axis can equal None (ravel
    array first), or an integer (the axis over which to operate on a and b).

def ttest_rel(a,b,axis=None):
    """Calculates the t-obtained T-test on TWO RELATED samples of scores, a
    and b.  From Numerical Recipies, p.483. Axis can equal None (ravel array
    first), or an integer (the axis over which to operate on a and b).

def ks_2samp(data1, data2):
    """ Computes the Kolmogorov-Smirnof statistic on 2 samples.  Modified
    from Numerical Recipies in C, page 493.  Returns KS D-value, prob.  Not
    ufunc- like.


From pav at iki.fi  Wed Dec  3 15:45:51 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 3 Dec 2008 20:45:51 +0000 (UTC)
Subject: [SciPy-dev] Docs: it's not over yet! (and T-shirts)
References: <wl88wqxtdl4.fsf@glup.physics.ucf.edu>
	<e06186140812030902y39fae983s64cc7c8147e256c6@mail.gmail.com>
Message-ID: <gh6r5v$n6q$1@ger.gmane.org>

Wed, 03 Dec 2008 10:02:00 -0700, Charles R Harris wrote:
> On Wed, Dec 3, 2008 at 9:42 AM, <jh at physics.ucf.edu> wrote:
>> Thanks to all the volunteers who have contributed docs in the Doc
>> Marathon.  The past month has seen an increase in activity,
>> particularly for numpy.ma and the ufuncs.  The PDF manual is now 445
>> pages.  This is fantastic!  But there's still much work to do.
>
> The document site needs to be merged with the trunk as the two are out
> of sync. Because new functions don't show up on the web site I document
> them in svn. Merges should happen daily, IMHO.

It's daily now.

-- 
Pauli Virtanen



From robert.kern at gmail.com  Wed Dec  3 15:48:40 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 3 Dec 2008 14:48:40 -0600
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
Message-ID: <3d375d730812031248m380ef171uae47a22e677f2666@mail.gmail.com>

On Wed, Dec 3, 2008 at 14:35, Jarrod Millman <millman at berkeley.edu> wrote:
> Given the discussion about Numerical Recipes derived-code, I did a
> little grepping and found 7 instances where code is referencing
> Numerical Recipes.  Before branching for 0.7, I would like to see this
> resolved.  I am not sure what we should do, but the license for using
> Numerical Recipes derived code clearly does not permit us to include
> it in scipy:
>  http://www.nr.com/licenses/redistribute.html
>
> Below I include all the instances I could easily find.  Thoughts?

There's a difference between referencing NR's descriptions of
algorithms and translating its code. I suspect that the functions in
stats.py, originally written by Gary Perlman, probably fall into the
latter category, but the others probably don't. Before pulling out
functions, it would be worthwhile to compare our implementations with
NR's to see if they in fact do look translated.

But do keep in mind that the algorithms are not copyrightable, and the
fact that there are only so many ways to implement said algorithms
correctly is fairly substantial protection for us.

IANAL. TINLA.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From millman at berkeley.edu  Wed Dec  3 15:53:29 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 3 Dec 2008 12:53:29 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <3d375d730812031248m380ef171uae47a22e677f2666@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<3d375d730812031248m380ef171uae47a22e677f2666@mail.gmail.com>
Message-ID: <c7009a550812031253n9fb7deet14d37e70b58f38c2@mail.gmail.com>

On Wed, Dec 3, 2008 at 12:48 PM, Robert Kern <robert.kern at gmail.com> wrote:
> There's a difference between referencing NR's descriptions of
> algorithms and translating its code. I suspect that the functions in
> stats.py, originally written by Gary Perlman, probably fall into the
> latter category, but the others probably don't. Before pulling out
> functions, it would be worthwhile to compare our implementations with
> NR's to see if they in fact do look translated.
>
> But do keep in mind that the algorithms are not copyrightable, and the
> fact that there are only so many ways to implement said algorithms
> correctly is fairly substantial protection for us.

Agreed.  I don't necessarily think that we need to rip out the
functions nor simply remove the reference to Numerical Recipes.  If it
turns out that we all ready have (or could easily have with a quick
rewrite) code that is based on the algorithms described in Numerical
Recipes (but not *derived* from their code), we should leave them in
and make sure to clarify in the comments that this is the case.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From charlesr.harris at gmail.com  Wed Dec  3 16:36:06 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 3 Dec 2008 14:36:06 -0700
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812031253n9fb7deet14d37e70b58f38c2@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<3d375d730812031248m380ef171uae47a22e677f2666@mail.gmail.com>
	<c7009a550812031253n9fb7deet14d37e70b58f38c2@mail.gmail.com>
Message-ID: <e06186140812031336m2fed26edwae1b339835f54f3@mail.gmail.com>

On Wed, Dec 3, 2008 at 1:53 PM, Jarrod Millman <millman at berkeley.edu> wrote:

> On Wed, Dec 3, 2008 at 12:48 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> > There's a difference between referencing NR's descriptions of
> > algorithms and translating its code. I suspect that the functions in
> > stats.py, originally written by Gary Perlman, probably fall into the
> > latter category, but the others probably don't. Before pulling out
> > functions, it would be worthwhile to compare our implementations with
> > NR's to see if they in fact do look translated.
> >
> > But do keep in mind that the algorithms are not copyrightable, and the
> > fact that there are only so many ways to implement said algorithms
> > correctly is fairly substantial protection for us.
>
> Agreed.  I don't necessarily think that we need to rip out the
> functions nor simply remove the reference to Numerical Recipes.  If it
> turns out that we all ready have (or could easily have with a quick
> rewrite) code that is based on the algorithms described in Numerical
> Recipes (but not *derived* from their code), we should leave them in
> and make sure to clarify in the comments that this is the case.
>

Especially since much of the code in NR is often a slight rewrite of other
code. For instance, Brent's original was written in Algol-68 and published
in a book among other places. Moler did a nice Fortran version with lots of
gotos, still the easiest to understand IMHO. I did structured version myself
in the 80's, a poor fit, actually, as the algorithm is best regarded as a
finite state machine. NR uses a structured version but I have no idea where
it comes from; their formulas for inverse quadratic extrapolation are
Brent's and not the versions I used, etc., etc....  These things really have
to be checked on a case by case basis and judgement brought to bear. There
is very little that is original in NR except the collection into a book with
explanatory text. That's a major accomplishment but it doesn't render the
code theirs. IIRC, their code for such things as computing elliptic
integrals looked very much "copied" in early editions.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081203/a9b77ae1/attachment.html>

From aisaac at american.edu  Wed Dec  3 16:59:46 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 03 Dec 2008 16:59:46 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
Message-ID: <49370152.3010905@american.edu>

As for `ridder` and `brentq`, I hardly think NR
would have a hope of copyrighting anything but
their actual code implementing these.
A classic reference should be given in any case:

Ridders, C. F. J. "A New Algorithm for Computing a
Single Root of a Real Continuous Function."
IEEE Trans. Circuits Systems 26, 979-980, 1979.

Alan Isaac



From doutriaux1 at llnl.gov  Wed Dec  3 17:25:21 2008
From: doutriaux1 at llnl.gov (=?UTF-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=)
Date: Wed, 3 Dec 2008 14:25:21 -0800
Subject: [SciPy-dev] Fwd: scipy/lapack
References: <982FA462-EB78-41A4-B755-FE0F3ED9AAE7@llnl.gov>
Message-ID: <057C5BFA-38EA-487C-B08A-ED449E27D6E6@llnl.gov>

Just forwarding, maybe a more general way could be implemented? or at  
least an option for lapack3.2

Here's what's happening

We have no lapack/blas on the system
we built numpy (1.2.1 i believe)

now we're trying to build scipy using LAPACK_SRC / BLAS_SRC

But we downloaded LAPACK 3.2

it seems as the name of the source files is hardcoded somewhere at the  
numpy level and in 3.2 there's a lot of new source files.

Any clues on this? (in the mean time we're building with 3.1)

C.


Begin forwarded message:

> From: Bob Drach <drach at llnl.gov>
> Date: December 3, 2008 2:04:35 PM PST
> To: Bob Drach <drach at llnl.gov>
> Cc: Charles Doutriaux <doutriaux1 at llnl.gov>
> Subject: Re: scipy/lapack
>
> numpy.distutils.system_info is hardwired for lapack 3.1. It has a  
> list of the routines to build which differ from 3.2. This module is  
> used by scipy to figure out which sources to build. There's no way  
> it would build with lapack 3.2 that I can see. I'm rebuilding from  
> scratch with blas and lapack enabled to see the current repository  
> version works.
>
> lapack-lite has the same functions as lapack, without the html and  
> man pages. So I'm happy with that if it works.
>
> Bob
>
>
>
> On Dec 2, 2008, at 7:10 PM, Bob Drach wrote:
>
>> Hi Charles,
>>
>> Do you know a straightforward way to build a usable scipy.linalg  
>> module? I don't care about having optimized Atlas libraries, etc.
>>
>> Bob
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081203/f6d53324/attachment.html>

From doutriaux1 at llnl.gov  Wed Dec  3 17:29:00 2008
From: doutriaux1 at llnl.gov (=?UTF-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=)
Date: Wed, 3 Dec 2008 14:29:00 -0800
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
Message-ID: <77EED93B-72FD-401E-809C-1B77AEFA4313@llnl.gov>

Hi there (i'll post the attachments of the logs in separate emails  
because it goes other the very restricitve 40k limit)

I'm trying to build scipy on a Mac OS X
uname -a
Darwin omar.llnl.gov 9.5.0 Darwin Kernel Version 9.5.0: Wed Sep  3  
11:29:43 PDT 2008; root:xnu-1228.7.58~1/RELEASE_I386 i386

using gcc and gfortran
omar /svn/cdat/trunk/logs >gcc --version
i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5488)
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There  
is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR  
PURPOSE.

omar /svn/cdat/trunk/logs >gfortran --version
GNU Fortran (GCC) 4.3.0 20080125 (experimental)
Copyright (C) 2007 Free Software Foundation, Inc.

GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of GNU Fortran
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING

I'm having no luck with either 0.7.b1 or trunk

I'm attaching the logs for both
On Dec 1, 2008, at 10:16 PM, Jarrod Millman wrote:

> Hey,
>
> I would like to get 0.7.0 out ASAP.  Is there any objection to tagging
> a release candidate this weekend?  Are there any major regressions or
> release blockers that I should be aware of?
>
> Once we are ready to make 0.7.0rc1, I will create a 0.7.x branch and
> tag the rc1 from the branch.  At that point the trunk will be open for
> 0.8 development and only bug-fixes will be allowed on the 0.7.x
> branch.
>
> Thanks,
>
> -- 
> Jarrod Millman
> Computational Infrastructure for Research Labs
> 10 Giannini Hall, UC Berkeley
> phone: 510.643.4014
> http:// cirl.berkeley.edu/
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http:// projects.scipy.org/mailman/listinfo/scipy-dev
>



From doutriaux1 at llnl.gov  Wed Dec  3 17:29:24 2008
From: doutriaux1 at llnl.gov (=?UTF-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=)
Date: Wed, 3 Dec 2008 14:29:24 -0800
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
Message-ID: <7669ABDB-CC9C-4C3C-A459-406229A98DEF@llnl.gov>

attachment for beta1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scipy.LOG.b1.bz2
Type: application/octet-stream
Size: 19166 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081203/1cb2f8d6/attachment.obj>
-------------- next part --------------

On Dec 1, 2008, at 10:16 PM, Jarrod Millman wrote:

> Hey,
>
> I would like to get 0.7.0 out ASAP.  Is there any objection to tagging
> a release candidate this weekend?  Are there any major regressions or
> release blockers that I should be aware of?
>
> Once we are ready to make 0.7.0rc1, I will create a 0.7.x branch and
> tag the rc1 from the branch.  At that point the trunk will be open for
> 0.8 development and only bug-fixes will be allowed on the 0.7.x
> branch.
>
> Thanks,
>
> -- 
> Jarrod Millman
> Computational Infrastructure for Research Labs
> 10 Giannini Hall, UC Berkeley
> phone: 510.643.4014
> http:// cirl.berkeley.edu/
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http:// projects.scipy.org/mailman/listinfo/scipy-dev
>


From doutriaux1 at llnl.gov  Wed Dec  3 17:29:48 2008
From: doutriaux1 at llnl.gov (=?UTF-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=)
Date: Wed, 3 Dec 2008 14:29:48 -0800
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
Message-ID: <C38245D0-5D32-4893-884F-E7F366A70816@llnl.gov>

attachment for trunk

-------------- next part --------------
A non-text attachment was scrubbed...
Name: scipy.LOG.bz2
Type: application/octet-stream
Size: 26108 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081203/d0f5ceab/attachment.obj>
-------------- next part --------------

On Dec 1, 2008, at 10:16 PM, Jarrod Millman wrote:

> Hey,
>
> I would like to get 0.7.0 out ASAP.  Is there any objection to tagging
> a release candidate this weekend?  Are there any major regressions or
> release blockers that I should be aware of?
>
> Once we are ready to make 0.7.0rc1, I will create a 0.7.x branch and
> tag the rc1 from the branch.  At that point the trunk will be open for
> 0.8 development and only bug-fixes will be allowed on the 0.7.x
> branch.
>
> Thanks,
>
> -- 
> Jarrod Millman
> Computational Infrastructure for Research Labs
> 10 Giannini Hall, UC Berkeley
> phone: 510.643.4014
> http:// cirl.berkeley.edu/
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http:// projects.scipy.org/mailman/listinfo/scipy-dev
>


From cournape at gmail.com  Wed Dec  3 18:06:34 2008
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 4 Dec 2008 08:06:34 +0900
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <77EED93B-72FD-401E-809C-1B77AEFA4313@llnl.gov>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
	<77EED93B-72FD-401E-809C-1B77AEFA4313@llnl.gov>
Message-ID: <5b8d13220812031506q42f56045pa1312f0f2f12b4a@mail.gmail.com>

On Thu, Dec 4, 2008 at 7:29 AM, Charles ???? Doutriaux
<doutriaux1 at llnl.gov> wrote:
> Hi there (i'll post the attachments of the logs in separate emails
> because it goes other the very restricitve 40k limit)
>

Hi Charles,

You should use the system compiler (gcc 4.0.*), not a more recent
version, unless you compile your own python.

David

From doutriaux1 at llnl.gov  Wed Dec  3 18:31:20 2008
From: doutriaux1 at llnl.gov (=?UTF-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=)
Date: Wed, 3 Dec 2008 15:31:20 -0800
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <5b8d13220812031506q42f56045pa1312f0f2f12b4a@mail.gmail.com>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
	<77EED93B-72FD-401E-809C-1B77AEFA4313@llnl.gov>
	<5b8d13220812031506q42f56045pa1312f0f2f12b4a@mail.gmail.com>
Message-ID: <5E4145D9-8AAF-4816-9CE2-2D7A13BADD6D@llnl.gov>

Hi David,

I am compiling my own python.

C.

On Dec 3, 2008, at 3:06 PM, David Cournapeau wrote:

> On Thu, Dec 4, 2008 at 7:29 AM, Charles ???? Doutriaux
> <doutriaux1 at llnl.gov> wrote:
>> Hi there (i'll post the attachments of the logs in separate emails
>> because it goes other the very restricitve 40k limit)
>>
>
> Hi Charles,
>
> You should use the system compiler (gcc 4.0.*), not a more recent
> version, unless you compile your own python.
>
> David
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http:// projects.scipy.org/mailman/listinfo/scipy-dev



From pav at iki.fi  Wed Dec  3 18:35:53 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 3 Dec 2008 23:35:53 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
Message-ID: <gh754p$n6q$2@ger.gmane.org>

Dear all,

I've gone through a first pass of updating the old Scipy Tutorial to 
present day. The result is here (and sources in Scipy SVN trunk/doc/
source/tutorial):

	http://docs.scipy.org/doc/scipy/reference/tutorial/index.html

It would be good if some brave souls went through it and helped to check 
if there are still some bugs hiding in the text.

You can also fix any issues you find immediately here:

	http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/index.rst/

provided you have an account on the site. Note that plots won't be 
visible on the wiki page -- you'll need to make sure that the stuff you 
have in the plot:: directives produces the appropriate plots manually.

-- 
Pauli Virtanen



From charlesr.harris at gmail.com  Wed Dec  3 18:44:26 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 3 Dec 2008 16:44:26 -0700
Subject: [SciPy-dev] Docs: it's not over yet! (and T-shirts)
In-Reply-To: <gh6r5v$n6q$1@ger.gmane.org>
References: <wl88wqxtdl4.fsf@glup.physics.ucf.edu>
	<e06186140812030902y39fae983s64cc7c8147e256c6@mail.gmail.com>
	<gh6r5v$n6q$1@ger.gmane.org>
Message-ID: <e06186140812031544l24566febq1d302a7287e783f@mail.gmail.com>

On Wed, Dec 3, 2008 at 1:45 PM, Pauli Virtanen <pav at iki.fi> wrote:

> Wed, 03 Dec 2008 10:02:00 -0700, Charles R Harris wrote:
> > On Wed, Dec 3, 2008 at 9:42 AM, <jh at physics.ucf.edu> wrote:
> >> Thanks to all the volunteers who have contributed docs in the Doc
> >> Marathon.  The past month has seen an increase in activity,
> >> particularly for numpy.ma and the ufuncs.  The PDF manual is now 445
> >> pages.  This is fantastic!  But there's still much work to do.
> >
> > The document site needs to be merged with the trunk as the two are out
> > of sync. Because new functions don't show up on the web site I document
> > them in svn. Merges should happen daily, IMHO.
>
> It's daily now.
>

Great. I think the website for editing docs is really slick, but it was
frustrating when new ufuncs never showed up for editing.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081203/f387c6f6/attachment.html>

From cournape at gmail.com  Wed Dec  3 18:45:25 2008
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 4 Dec 2008 08:45:25 +0900
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <5E4145D9-8AAF-4816-9CE2-2D7A13BADD6D@llnl.gov>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>
	<77EED93B-72FD-401E-809C-1B77AEFA4313@llnl.gov>
	<5b8d13220812031506q42f56045pa1312f0f2f12b4a@mail.gmail.com>
	<5E4145D9-8AAF-4816-9CE2-2D7A13BADD6D@llnl.gov>
Message-ID: <5b8d13220812031545j52ae07c9mb621d07adb1d813d@mail.gmail.com>

On Thu, Dec 4, 2008 at 8:31 AM, Charles ???? Doutriaux
<doutriaux1 at llnl.gov> wrote:
> Hi David,
>
> I am compiling my own python.

I thought the problem was that since numpy compiler flags come from
the ones used to build python, when using distutils, there was a
discrepency between python and numpy:

python -c "from distutils import sysconfig; print
sysconfig.get_config_vars('CFLAGS')"

Will tell you which compiler flags have been used for python.

But it seems that the problem is a bit more subtle: -no-cpp-precomp is
an Apple-specific flag, and the gcc you are using is more than likely
not the Apple one (since its version is 4.3, and AFAIK, no
apple-distributed gcc has such a high version, at least for released
OS). This may be a bug in python configure, which unconditionally uses
this option. If that's the case, I am afraid there is no easy solution
to your problem.

David

From michael.abshoff at googlemail.com  Wed Dec  3 18:55:26 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 03 Dec 2008 15:55:26 -0800
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <5b8d13220812031545j52ae07c9mb621d07adb1d813d@mail.gmail.com>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>	<77EED93B-72FD-401E-809C-1B77AEFA4313@llnl.gov>	<5b8d13220812031506q42f56045pa1312f0f2f12b4a@mail.gmail.com>	<5E4145D9-8AAF-4816-9CE2-2D7A13BADD6D@llnl.gov>
	<5b8d13220812031545j52ae07c9mb621d07adb1d813d@mail.gmail.com>
Message-ID: <49371C6E.5040709@gmail.com>

David Cournapeau wrote:
> On Thu, Dec 4, 2008 at 8:31 AM, Charles ???? Doutriaux
> <doutriaux1 at llnl.gov> wrote:
>> Hi David,

Hi,

>> I am compiling my own python.
> 
> I thought the problem was that since numpy compiler flags come from
> the ones used to build python, when using distutils, there was a
> discrepency between python and numpy:
> 
> python -c "from distutils import sysconfig; print
> sysconfig.get_config_vars('CFLAGS')"
> 
> Will tell you which compiler flags have been used for python.
> 
> But it seems that the problem is a bit more subtle: -no-cpp-precomp is
> an Apple-specific flag, and the gcc you are using is more than likely
> not the Apple one (since its version is 4.3, and AFAIK, no
> apple-distributed gcc has such a high version, at least for released
> OS). This may be a bug in python configure, which unconditionally uses
> this option. If that's the case, I am afraid there is no easy solution
> to your problem.

I ran into the same problem with Python 2.5.2 on OSX when using a 
non-XCode gcc. The hackish solution is to delete those flags from the 
configure script since they are hardcoded as David mentions. Then Python 
2.5.2 build and worked fine in 32 and 64 bit mode on OSX 10.5.

David: do you know if the problem with hard coded flags on OSX was ever 
reported upstream? I am not a Python 2.6 user, so I have not tried that 
version to see if it has been fixed.


> David

Cheers,

Michael

> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev



From ellisonbg.net at gmail.com  Wed Dec  3 20:35:37 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 3 Dec 2008 17:35:37 -0800
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <gh754p$n6q$2@ger.gmane.org>
References: <gh754p$n6q$2@ger.gmane.org>
Message-ID: <6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>

Great!

I think the tutorial would be much more usable if it were broken up
into different pages (each as a separate .rst file).  That keeps it
much easier to navigate.  We use this approach in IPython's docs and
it works very well.

Cheers,

Brian

On Wed, Dec 3, 2008 at 3:35 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Dear all,
>
> I've gone through a first pass of updating the old Scipy Tutorial to
> present day. The result is here (and sources in Scipy SVN trunk/doc/
> source/tutorial):
>
>        http://docs.scipy.org/doc/scipy/reference/tutorial/index.html
>
> It would be good if some brave souls went through it and helped to check
> if there are still some bugs hiding in the text.
>
> You can also fix any issues you find immediately here:
>
>        http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/index.rst/
>
> provided you have an account on the site. Note that plots won't be
> visible on the wiki page -- you'll need to make sure that the stuff you
> have in the plot:: directives produces the appropriate plots manually.
>
> --
> Pauli Virtanen
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>


From david at ar.media.kyoto-u.ac.jp  Wed Dec  3 20:53:17 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 04 Dec 2008 10:53:17 +0900
Subject: [SciPy-dev] tagging 0.7rc1 this weekend?
In-Reply-To: <49371C6E.5040709@gmail.com>
References: <c7009a550812012216h1fcef5f5sad7817e68441548f@mail.gmail.com>	<77EED93B-72FD-401E-809C-1B77AEFA4313@llnl.gov>	<5b8d13220812031506q42f56045pa1312f0f2f12b4a@mail.gmail.com>	<5E4145D9-8AAF-4816-9CE2-2D7A13BADD6D@llnl.gov>	<5b8d13220812031545j52ae07c9mb621d07adb1d813d@mail.gmail.com>
	<49371C6E.5040709@gmail.com>
Message-ID: <4937380D.9000306@ar.media.kyoto-u.ac.jp>

Michael Abshoff wrote:
>
> I ran into the same problem with Python 2.5.2 on OSX when using a 
> non-XCode gcc. The hackish solution is to delete those flags from the 
> configure script since they are hardcoded as David mentions.

It is not more hackish than putting them in the first place :) They have
been removed in python trunk, I don't know if they are there in python 2.6.

cheers,

David


From josef.pktd at gmail.com  Wed Dec  3 23:45:51 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 3 Dec 2008 23:45:51 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <49370152.3010905@american.edu>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<49370152.3010905@american.edu>
Message-ID: <1cd32cbb0812032045q3395bbc0ge10aa1efd73a4aa8@mail.gmail.com>

I looked at ttest_rel and ttest_ind, both are simple t-tests for
equality of two samples, either in mean or of a twice ("related")
observed random variable.

step 1: calculate difference between samples (means)
step 2: divide by appropriate standard deviation measure
step 3: look up distribution to report p-value

The doc strings don't explain well what the test is useful for, but it
looks a straight forward implementation of the statistical formulas.
The only messy part is to make it more general so that it also applies
for higher dimensional arrays, and that looks all numpy to me.

ttest_ind is identical to
http://en.wikipedia.org/wiki/T-test#Independent_two-sample_t-test
ttest_rel is described in http://en.wikipedia.org/wiki/T-test#Dependent_t-test

So, if there is any resemblance left to NR (which I don't know) then
it's purely because it calculates the same thing. I think the two
Wikipedia references are a lot better than NR, since Wikipedia also
explains a bit the background and usage.

The only part that I wouldn't have immediately implemented, is
handling of zerodivision problems when the variance (of both samples
or of difference of samples) is zero, which is an unusual boundary
case.

Josef


From rbastian at free.fr  Thu Dec  4 04:25:36 2008
From: rbastian at free.fr (R. Bastian)
Date: Thu, 4 Dec 2008 10:25:36 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
Message-ID: <20081204102536.5faad9d5.rbastian@free.fr>

On Wed, 3 Dec 2008 17:35:37 -0800
"Brian Granger" <ellisonbg.net at gmail.com> wrote:

> Great!
> 
> I think the tutorial would be much more usable if it were broken up
> into different pages (each as a separate .rst file).  

No !

It's only easier for those not needing a tutorial ;-)

> That keeps it
> much easier to navigate.  We use this approach in IPython's docs and
> it works very well.
> 
> Cheers,
> 
> Brian
> 
> On Wed, Dec 3, 2008 at 3:35 PM, Pauli Virtanen <pav at iki.fi> wrote:
> > Dear all,
> >
> > I've gone through a first pass of updating the old Scipy Tutorial to
> > present day. The result is here (and sources in Scipy SVN trunk/doc/
> > source/tutorial):
> >
> >        http://docs.scipy.org/doc/scipy/reference/tutorial/index.html
> >
> > It would be good if some brave souls went through it and helped to check
> > if there are still some bugs hiding in the text.
> >
> > You can also fix any issues you find immediately here:
> >
> >        http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/index.rst/
> >
> > provided you have an account on the site. Note that plots won't be
> > visible on the wiki page -- you'll need to make sure that the stuff you
> > have in the plot:: directives produces the appropriate plots manually.
> >
> > --
> > Pauli Virtanen
> >
> > _______________________________________________
> > Scipy-dev mailing list
> > Scipy-dev at scipy.org
> > http://projects.scipy.org/mailman/listinfo/scipy-dev
> >
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
> 


-- 
R. Bastian 
www.musiques-rb.org



From gael.varoquaux at normalesup.org  Thu Dec  4 04:39:44 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 4 Dec 2008 10:39:44 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081204102536.5faad9d5.rbastian@free.fr>
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
Message-ID: <20081204093944.GB18992@phare.normalesup.org>

On Thu, Dec 04, 2008 at 10:25:36AM +0100, R. Bastian wrote:
> > I think the tutorial would be much more usable if it were broken up
> > into different pages (each as a separate .rst file).  

> No !

> It's only easier for those not needing a tutorial ;-)

I don't agree. Long HTML pages are bad practice, every web designer will
tell you that (http://webstyleguide.com/page/length.html,
http://www.astrodigital.org/digital/shortvslongpages.html,
http://www.webpagesthatsuck.com/biggest-mistakes-in-web-design-1995-2015.html
- 10 ...).

The good rule should be: one subject by pages, pages a few screen long
maximum.

If you want a long document, you need the pdf. That's why we use Sphinx.

Ga?l


From pav at iki.fi  Thu Dec  4 04:50:00 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 4 Dec 2008 09:50:00 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
Message-ID: <gh8948$p6v$1@ger.gmane.org>

Wed, 03 Dec 2008 17:35:37 -0800, Brian Granger wrote:
> I think the tutorial would be much more usable if it were broken up into
> different pages (each as a separate .rst file).  That keeps it much
> easier to navigate.  We use this approach in IPython's docs and it works
> very well.

I intended to split it, but didn't get to that yet.

If someone wants to do this, it can be done in the wiki, too; just create 
new .rst files (click the 'tutorial' directory link on the top), and copy 
and paste the relevant portions to the relevant files. Finally, replace 
the copied material in the tutorial's index.rst with a toctree:: 
reference.

-- 
Pauli Virtanen



From rbastian at free.fr  Thu Dec  4 05:15:31 2008
From: rbastian at free.fr (R. Bastian)
Date: Thu, 4 Dec 2008 11:15:31 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081204093944.GB18992@phare.normalesup.org>
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
Message-ID: <20081204111531.82da715a.rbastian@free.fr>

On Thu, 4 Dec 2008 10:39:44 +0100
Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:

> On Thu, Dec 04, 2008 at 10:25:36AM +0100, R. Bastian wrote:
> > > I think the tutorial would be much more usable if it were broken up
> > > into different pages (each as a separate .rst file).  
> 
> > No !
> 
> > It's only easier for those not needing a tutorial ;-)
> 
> I don't agree. Long HTML pages are bad practice, every web designer will
> tell you that (http://webstyleguide.com/page/length.html,
> http://www.astrodigital.org/digital/shortvslongpages.html,
> http://www.webpagesthatsuck.com/biggest-mistakes-in-web-design-1995-2015.html
> - 10 ...).

Not every webdesigner ...

> 
> The good rule should be: one subject by pages, pages a few screen long
> maximum.

I think (or hope) that's easy to get two forms: a long and single page,
or a lot of small subject centered pages.
In a single page it is easy to read with PageDown, PageUp
and it is easy to search with Ctrl-F.

Different pages are the best choice for treating the subjets, but
a tutorial _is_ a subject. Anyway, everyone can choice
the prefered form. 

> 
> If you want a long document, you need the pdf. That's why we use Sphinx.
> 
> Ga?l
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
> 


-- 
R. Bastian 
www.musiques-rb.org



From aisaac at american.edu  Thu Dec  4 09:01:41 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 04 Dec 2008 09:01:41 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081204093944.GB18992@phare.normalesup.org>
References: <gh754p$n6q$2@ger.gmane.org>	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
Message-ID: <4937E2C5.5090001@american.edu>

> On Thu, Dec 04, 2008 at 10:25:36AM +0100, R. Bastian  wrote:
>>> I think the tutorial would be much more usable if it were broken up 
>>> into different pages (each as a separate .rst file). 


On 12/4/2008 4:39 AM Gael Varoquaux apparently wrote:
> I don't agree. Long HTML pages are bad practice, every web designer will 
> tell you that (http://webstyleguide.com/page/length.html, 
> http://www.astrodigital.org/digital/shortvslongpages.html, 
> http://www.webpagesthatsuck.com/biggest-mistakes-in-web-design-1995-2015.html 
> - 10 ...). 
> The good rule should be: one subject by pages, pages a few screen long 
> maximum. 


I think this gets into areas of "true belief".
My view is that an adequately linked single
HTML document can be just as useful or more useful
than a broken up document.  Example: the NumPy
Example List is much more useful if we can search
nicely across all the examples, which requires a single
page document.

That said, there is no reason by broken up HTML document
requires a broken up reST source.  Just write each
section to a separate file, with appropriate headers.
I think the rst2html writer used to support that out
of the box, although I'm not positive.

Cheers,
Alan Isaac




From gael.varoquaux at normalesup.org  Thu Dec  4 09:11:14 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 4 Dec 2008 15:11:14 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <4937E2C5.5090001@american.edu>
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
Message-ID: <20081204141114.GH18992@phare.normalesup.org>

On Thu, Dec 04, 2008 at 09:01:41AM -0500, Alan G Isaac wrote:
> I think this gets into areas of "true belief".
> My view is that an adequately linked single
> HTML document can be just as useful or more useful
> than a broken up document.  Example: the NumPy
> Example List is much more useful if we can search
> nicely across all the examples, which requires a single
> page document.

I really don't understand this argument. It does not scale, and thats the
reason why there is a search box with sphinx. What is the problem with
the search box? It is not as good as google, but I find it works quite
well, and certain as well as "Ctrl-F".

> That said, there is no reason by broken up HTML document
> requires a broken up reST source.  Just write each
> section to a separate file, with appropriate headers.
> I think the rst2html writer used to support that out
> of the box, although I'm not positive.

I don't think so, and in addition we are using sphinx, not vanilla rst,
so we need to use the sphinx tools. That said, once again, sphinx creates
pdf that is single page, and can thus be searched. I think this is the
answer to your request.

My 2 cents,

Ga?l


From bsouthey at gmail.com  Thu Dec  4 09:19:32 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Thu, 04 Dec 2008 08:19:32 -0600
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <1cd32cbb0812032045q3395bbc0ge10aa1efd73a4aa8@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>	<49370152.3010905@american.edu>
	<1cd32cbb0812032045q3395bbc0ge10aa1efd73a4aa8@mail.gmail.com>
Message-ID: <4937E6F4.9000609@gmail.com>

josef.pktd at gmail.com wrote:
> I looked at ttest_rel and ttest_ind, both are simple t-tests for
> equality of two samples, either in mean or of a twice ("related")
> observed random variable.
>
> step 1: calculate difference between samples (means)
> step 2: divide by appropriate standard deviation measure
> step 3: look up distribution to report p-value
>
> The doc strings don't explain well what the test is useful for, but it
> looks a straight forward implementation of the statistical formulas.
> The only messy part is to make it more general so that it also applies
> for higher dimensional arrays, and that looks all numpy to me.
>
> ttest_ind is identical to
> http://en.wikipedia.org/wiki/T-test#Independent_two-sample_t-test
> ttest_rel is described in http://en.wikipedia.org/wiki/T-test#Dependent_t-test
>
> So, if there is any resemblance left to NR (which I don't know) then
> it's purely because it calculates the same thing. I think the two
> Wikipedia references are a lot better than NR, since Wikipedia also
> explains a bit the background and usage.
>
> The only part that I wouldn't have immediately implemented, is
> handling of zerodivision problems when the variance (of both samples
> or of difference of samples) is zero, which is an unusual boundary
> case.
>
> Josef
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>   
Hi,
Could you also please replace the use of 'betai' with the appropriate 
call to the t-distribution to the the probabilities?
Numerical Recipes uses betai and also it is more understandable to use 
the actual t-distribution.

Bruce



From cimrman3 at ntc.zcu.cz  Thu Dec  4 09:51:28 2008
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Thu, 04 Dec 2008 15:51:28 +0100
Subject: [SciPy-dev] ANN: SfePy 2008.4
Message-ID: <4937EE70.4030603@ntc.zcu.cz>

I am pleased to announce the release of SfePy 2008.4.

SfePy (simple finite elements in Python) is a finite element analysis 
software based primarily on Numpy and SciPy.

Mailing lists, issue tracking, mercurial repository: http://sfepy.org
Home page: http://sfepy.kme.zcu.cz

Major improvements:
- framework for running parametric studies
- greatly improved support for time-dependent problems
   - time derivatives of variables as term arguments
   - initial conditions via ics, ic_<number> keywords
- live plotting using multiprocessing module
- type of term arguments determined fully at run-time
- new terms, namely piezo-coupling

Applications:
- enhanced acoustic band gaps code
   - dispersion analysis (polarization angle calculation)
   - applied load tensor computation
   - phase velocity computation for periodic perforated media with empty 
holes
- improved schroedinger.py
   - plotting DFT iterations

For more information on this release, see
http://sfepy.googlecode.com/svn/web/releases/2008.4_RELEASE_NOTES.txt

Best regards,
Robert Cimrman


From aisaac at american.edu  Thu Dec  4 10:01:26 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 04 Dec 2008 10:01:26 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081204141114.GH18992@phare.normalesup.org>
References: <gh754p$n6q$2@ger.gmane.org>	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>	<20081204102536.5faad9d5.rbastian@free.fr>	<20081204093944.GB18992@phare.normalesup.org>	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
Message-ID: <4937F0C6.8000202@american.edu>

> On Thu, Dec 04, 2008 at 09:01:41AM -0500, Alan G Isaac wrote:
>> My view is that an adequately linked single
>> HTML document can be just as useful or more useful
>> than a broken up document.  Example: the NumPy
>> Example List is much more useful if we can search
>> nicely across all the examples, which requires a single
>> page document.


On 12/4/2008 9:11 AM Gael Varoquaux apparently wrote:
> I really don't understand this argument. It does not scale, and thats the
> reason why there is a search box with sphinx. What is the problem with
> the search box? It is not as good as google, but I find it works quite
> well, and certain as well as "Ctrl-F".

It has been so long since I have used `ctrl-F` in my
browser that I had to check if the functionality was
still the same as `/`.  It is, in FireFox at least.
As a user, I hit `/` (one keystroke) and start typing.
Search is incremental and immediate.  If I have typed
all I wish as a search term, I see the result immediately
in front of me **in context**.  If I want the next result,
I just hit ctrl-g, repeatedly if necessary, and each time
I see the result *truly* in context.


>> That said, there is no reason by broken up HTML document
>> requires a broken up reST source.  Just write each
>> section to a separate file, with appropriate headers.
>> I think the rst2html writer used to support that out
>> of the box, although I'm not positive.
> 
> I don't think so, and in addition we are using sphinx, not vanilla rst,
> so we need to use the sphinx tools. That said, once again, sphinx creates
> pdf that is single page, and can thus be searched. I think this is the
> answer to your request.


1. It should be trivial to modify the writer to write sections
to separate files.  Many projects offer both a fragmented HTML
document and a complete one, allowing users to follow their tastes.
2. I doubt anyone who actually uses Adobe's PDF search
is very happy with it: it is slow and provides too little
context.  (Better than it used to be, but still.) Plus,
not everyone viewing PDFs is using Reader or Acrobat.

Alan Isaac



From pav at iki.fi  Thu Dec  4 10:14:50 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 4 Dec 2008 15:14:50 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu>
Message-ID: <gh8s5a$b0n$1@ger.gmane.org>

Thu, 04 Dec 2008 10:01:26 -0500, Alan G Isaac wrote:
[clip]
> 1. It should be trivial to modify the writer to write sections to
> separate files.  Many projects offer both a fragmented HTML document and
> a complete one, allowing users to follow their tastes.

I don't think it's that trivial; you'd have to take care that cross-
references, the search, index, table of contents, etc. still work 
properly. I believe HTML splitting has come up one the Sphinx mailing 
list, but don't remember what was the outcome. If you want to take a shot 
at this, I'd believe the Sphinx author wouldn't object to including the 
feature.

The RST files are easier to handle if the document is split into parts, 
so I'm going to split them. It should be fairly easy to generate a single-
page tutorial by combining them using the include:: directive, so we can 
do also that if some people really object to reading the tutorial in 
smaller pieces.

-- 
Pauli Virtanen



From josef.pktd at gmail.com  Thu Dec  4 10:29:52 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 4 Dec 2008 10:29:52 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <4937E6F4.9000609@gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<49370152.3010905@american.edu>
	<1cd32cbb0812032045q3395bbc0ge10aa1efd73a4aa8@mail.gmail.com>
	<4937E6F4.9000609@gmail.com>
Message-ID: <1cd32cbb0812040729m9966855jf7b83e8b6cb21002@mail.gmail.com>

>>
> Hi,
> Could you also please replace the use of 'betai' with the appropriate
> call to the t-distribution to the the probabilities?
> Numerical Recipes uses betai and also it is more understandable to use
> the actual t-distribution.
>
> Bruce
>

betai is used in several places in scipy.stats, until now I never
touched scipy.special calls unless there was a real bug.

There is some duplication in scipy.special, but I have no idea about
the relative advantages and disadvantages of the different
implementations. For some functions, I know that they don't work
correctly over some parameter range. So, until now I'm reluctant to
change any of the calls to special.

Usually I don't change anything without having some tests first, and
neither ttest_ind nor ttest_rand have any tests in the test suite and
I'm running out of time to write tests before 0.70.

but this replacement looks simple enough:

>>> df=5;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
0.6382988716409288
0.63829887164092902
>>> df=500;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
0.61729505009465102
0.61729505009466568
>>> df=5000;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
0.61709708085302761
0.61709708085403392
>>> df=50000;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
0.61707727784367095
0.61707727785345856
>>> df=2;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
0.66666666666666596
0.66666666666666652

I will make the change tonight.

Josef


From josef.pktd at gmail.com  Thu Dec  4 10:44:13 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 4 Dec 2008 10:44:13 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <gh8s5a$b0n$1@ger.gmane.org>
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
Message-ID: <1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>

Just to provide my comment on search:

Compared to a windows helpfile, and single page search, sphinx search
is very slow, especially when I'm on on online connection.
I, now, got used to np.lookfor because it is much faster than trying
to get through the online help.

I like the new Sphinx docs a lot if I have to look up a specific
topic, but when I don't know which function might be useful for my
case, then the overview and speed of looking around is pretty tedious,
compared to the windows help files for matlab or R.
(As an aside: I actually like the style sheet of the documentation
editor better than sphinx.)

I haven't tried yet to generate my own numpy, scipy docs, since they
are makefile based, and I didn't want to spend the time to figure out
how to use the build scripts on windows, but a windows helpfile is on
my wish/todo list, since sphinx makes it very easy to do.

Josef


From nwagner at iam.uni-stuttgart.de  Thu Dec  4 11:25:07 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 04 Dec 2008 17:25:07 +0100
Subject: [SciPy-dev] numpy/scipy docs and make html
Message-ID: <web-114761847@uni-stuttgart.de>

Hi all,

A make html in numpy/doc failed

(/usr/share/texmf/tex/latex/amsfonts/amssymb.sty
(/usr/share/texmf/tex/latex/amsfonts/amsfonts.sty))
(/usr/share/texmf/tex/latex/tools/bm.sty)

! LaTeX Error: File `preview.sty' not found.

Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)

Enter file name:
! Emergency stop.
<read *>

l.13 \begin
            {document}^^M
No pages of output.
Transcript written on math.log.

make: *** [html] Error 1
  
echo $TEXINPUT
/data/home/nwagner/texmf:/usr/share/texmf

find ~/texmf/ -name preview.sty
/data/home/nwagner/texmf/preview/preview.sty

Any idea ?

Nils



From bsouthey at gmail.com  Thu Dec  4 11:46:27 2008
From: bsouthey at gmail.com (Bruce Southey)
Date: Thu, 04 Dec 2008 10:46:27 -0600
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <1cd32cbb0812040729m9966855jf7b83e8b6cb21002@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>	<49370152.3010905@american.edu>	<1cd32cbb0812032045q3395bbc0ge10aa1efd73a4aa8@mail.gmail.com>	<4937E6F4.9000609@gmail.com>
	<1cd32cbb0812040729m9966855jf7b83e8b6cb21002@mail.gmail.com>
Message-ID: <49380963.1080304@gmail.com>

josef.pktd at gmail.com wrote:
>> Hi,
>> Could you also please replace the use of 'betai' with the appropriate
>> call to the t-distribution to the the probabilities?
>> Numerical Recipes uses betai and also it is more understandable to use
>> the actual t-distribution.
>>
>> Bruce
>>
>>     
>
> betai is used in several places in scipy.stats, until now I never
> touched scipy.special calls unless there was a real bug.
>
> There is some duplication in scipy.special, but I have no idea about
> the relative advantages and disadvantages of the different
> implementations. For some functions, I know that they don't work
> correctly over some parameter range. So, until now I'm reluctant to
> change any of the calls to special.
>
> Usually I don't change anything without having some tests first, and
> neither ttest_ind nor ttest_rand have any tests in the test suite and
> I'm running out of time to write tests before 0.70.
>
> but this replacement looks simple enough:
>
>   
>>>> df=5;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
>>>>         
> 0.6382988716409288
> 0.63829887164092902
>   
>>>> df=500;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
>>>>         
> 0.61729505009465102
> 0.61729505009466568
>   
>>>> df=5000;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
>>>>         
> 0.61709708085302761
> 0.61709708085403392
>   
>>>> df=50000;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
>>>>         
> 0.61707727784367095
> 0.61707727785345856
>   
>>>> df=2;t=0.5;stats.betai(0.5*df,0.5,float(df)/(df+t*t));stats.t.sf(t,df)*2
>>>>         
> 0.66666666666666596
> 0.66666666666666652
>
> I will make the change tonight.
>
> Josef
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>   
The probability density function can be written as an incomplete beta 
function see for example:
http://en.wikipedia.org/wiki/Student%27s_t-distribution

So it is probably just numerical error between division of two gamma 
functions and using the inverse of a beta function. While not an expert 
on this, it looks a like numerical precision of a double since the 
difference starts around 14 decimal place.

Bruce


From pav at iki.fi  Thu Dec  4 11:47:45 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 4 Dec 2008 16:47:45 +0000 (UTC)
Subject: [SciPy-dev] numpy/scipy docs and make html
References: <web-114761847@uni-stuttgart.de>
Message-ID: <gh91jh$3ik$1@ger.gmane.org>

Thu, 04 Dec 2008 17:25:07 +0100, Nils Wagner wrote:
> A make html in numpy/doc failed
[clip]
> ! LaTeX Error: File `preview.sty' not found.

You need the 'preview' Latex package so that Sphinx can get the baseline 
of the formulas right. See 

    http://sphinx.pocoo.org/ext/math.html#confval-pngmath_use_preview

or turn pngmath_use_preview off in source/conf.py.

-- 
Pauli Virtanen



From ellisonbg.net at gmail.com  Thu Dec  4 14:20:38 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 4 Dec 2008 11:20:38 -0800
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081204111531.82da715a.rbastian@free.fr>
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<20081204111531.82da715a.rbastian@free.fr>
Message-ID: <6ce0ac130812041120i27822402y8caa761dc1a02070@mail.gmail.com>

> Different pages are the best choice for treating the subjets, but
> a tutorial _is_ a subject. Anyway, everyone can choice
> the prefered form.

Yes, and most books I read are about a subject.  That doesn't mean
they are written on scrolls ;-)

More seriously, here is how I would use this (and I am not a SciPy
pro, so I consider myself a potential customer for the tutorial).

I would start reading it....

Then, in the middle of reading the Optimization section, I will
realize that I need to go back to the section on indexing to clear up
some things...

But, to get to the navigation column, I have to scroll to the top.
Once I find the indexing part, I read it...

 and then I have to scroll around again to find the Optimization
section again.

Bottom line, just because the tutorial is a single subject that is
designed to be read through like a book, does not mean it should be
impossible to navigate.  Also, what if I only want to read the
tutorial on a specific set of topics.  The current design makes the
ensuing navigation difficult.

So I remain +1 on breaking into multiple files.  But, I am not the one
doing the work here,

Brian


From millman at berkeley.edu  Thu Dec  4 14:39:37 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 4 Dec 2008 11:39:37 -0800
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <6ce0ac130812041120i27822402y8caa761dc1a02070@mail.gmail.com>
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<20081204111531.82da715a.rbastian@free.fr>
	<6ce0ac130812041120i27822402y8caa761dc1a02070@mail.gmail.com>
Message-ID: <c7009a550812041139s7ca34ba6k9e28a6b3b83c0aec@mail.gmail.com>

On Thu, Dec 4, 2008 at 11:20 AM, Brian Granger <ellisonbg.net at gmail.com> wrote:
> So I remain +1 on breaking into multiple files.  But, I am not the one
> doing the work here,

+1
I was planning on sending an email asking for it to broken up into
multiple pages before this discussion started.  I find the one long
web page format annoying.

I would also prefer changing the name from "Tutorial" to "User Guide".

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From nwagner at iam.uni-stuttgart.de  Thu Dec  4 14:55:14 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 04 Dec 2008 20:55:14 +0100
Subject: [SciPy-dev] scipy.test() errors/failures
Message-ID: <web-114767489@uni-stuttgart.de>

Hi all,

Can someone reproduce these errors/failures ? I am using

>>> scipy.__version__
'0.7.0.dev5221'
>>> numpy.__version__
'1.3.0.dev6136'


======================================================================
ERROR: Testing that kmeans2 init methods work.
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/cluster/tests/test_vq.py", 
line 147, in test_kmeans2_init
     data = data.reshape((200, 2))
ValueError: total size of new array must be unchanged

======================================================================
ERROR: Testing simple call to kmeans2 with rank 1 data.
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/cluster/tests/test_vq.py", 
line 127, in test_kmeans2_rank1
     data = data.reshape((200, 2))
ValueError: total size of new array must be unchanged

======================================================================
ERROR: Testing simple call to kmeans2 with rank 1 data.
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/cluster/tests/test_vq.py", 
line 139, in test_kmeans2_rank1_2
     data = data.reshape((200, 2))
ValueError: total size of new array must be unchanged

======================================================================
ERROR: This will cause kmean to have a cluster with no 
points.
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/cluster/tests/test_vq.py", 
line 96, in test_kmeans_lost_cluster
     data = data.reshape((200, 2))
ValueError: total size of new array must be unchanged

======================================================================
ERROR: test_complex_write_read 
(test_mmio.TestMMIOCoordinate)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py", 
line 290, in test_complex_write_read
     b = mmread(fn).todense()
   File 
"/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line 
52, in mmread
     return MMFile().read(source)
   File 
"/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line 
273, in read
     return self._parse_body(stream)
   File 
"/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line 
407, in _parse_body
     flat_data = flat_data.reshape(-1,4)
ValueError: total size of new array must be unchanged

======================================================================
ERROR: read a hermitian matrix
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py", 
line 213, in test_read_hermitian
     b = mmread(fn).todense()
   File 
"/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line 
52, in mmread
     return MMFile().read(source)
   File 
"/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line 
273, in read
     return self._parse_body(stream)
   File 
"/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line 
407, in _parse_body
     flat_data = flat_data.reshape(-1,4)
ValueError: total size of new array must be unchanged

======================================================================
FAIL: read a general matrix
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py", 
line 199, in test_read_geneal
     assert_array_almost_equal(a,b)
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 310, in assert_array_almost_equal
     header='Arrays are not almost equal')
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 295, in assert_array_compare
     raise AssertionError(msg)
AssertionError:
Arrays are not almost equal

(mismatch 28.0%)
  x: array([[  1.00000000e+00,   0.00000000e+00, 
  0.00000000e+00,
           6.00000000e+00,   0.00000000e+00],
        [  0.00000000e+00,   1.05000000e+01, 
  0.00000000e+00,...
  y: array([[ 1.,  0.,  0.,  0.,  0.],
        [ 0.,  0.,  0.,  0.,  0.],
        [ 0.,  0.,  0.,  0.,  0.],...

======================================================================
FAIL: read a skew-symmetric matrix
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py", 
line 229, in test_read_skew
     assert_array_almost_equal(a,b)
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 310, in assert_array_almost_equal
     header='Arrays are not almost equal')
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 295, in assert_array_compare
     raise AssertionError(msg)
AssertionError:
Arrays are not almost equal

(mismatch 24.0%)
  x: array([[  1.00000000e+00,   0.00000000e+00, 
  0.00000000e+00,
           0.00000000e+00,   0.00000000e+00],
        [  0.00000000e+00,   1.05000000e+01, 
  0.00000000e+00,...
  y: array([[ 1.,  0.,  0.,  0.,  0.],
        [ 0.,  0.,  0.,  0.,  0.],
        [ 0.,  0.,  0.,  0.,  0.],...

======================================================================
FAIL: test_real_write_read (test_mmio.TestMMIOCoordinate)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py", 
line 275, in test_real_write_read
     assert_array_almost_equal(a,b)
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 310, in assert_array_almost_equal
     header='Arrays are not almost equal')
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 295, in assert_array_compare
     raise AssertionError(msg)
AssertionError:
Arrays are not almost equal

(mismatch 24.0%)
  x: array([[  1.00000000e+00,   0.00000000e+00, 
  0.00000000e+00,
           6.00000000e+00,   0.00000000e+00],
        [  0.00000000e+00,   1.05000000e+01, 
  0.00000000e+00,...
  y: array([[  1.,   0.,   0.,   6.,   0.],
        [  0.,  10.,   0.,   0.,   0.],
        [  0.,   0.,   0.,   0.,   0.],...

======================================================================
FAIL: test_sparse_formats (test_mmio.TestMMIOCoordinate)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py", 
line 313, in test_sparse_formats
     assert_array_almost_equal(result, expected)
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 310, in assert_array_almost_equal
     header='Arrays are not almost equal')
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 295, in assert_array_compare
     raise AssertionError(msg)
AssertionError:
Arrays are not almost equal

(mismatch 24.0%)
  x: array([[  1.,   0.,   0.,   6.,   0.],
        [  0.,  10.,   0.,   0.,   0.],
        [  0.,   0.,   0.,   0.,   0.],...
  y: array([[  1.00000000e+00,   0.00000000e+00, 
  0.00000000e+00,
           6.00000000e+00,   0.00000000e+00],
        [  0.00000000e+00,   1.05000000e+01, 
  0.00000000e+00,...

======================================================================
FAIL: test_imresize (test_pilutil.TestPILUtil)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/decorators.py", 
line 82, in skipper
     return f(*args, **kwargs)
   File 
"/usr/lib/python2.4/site-packages/scipy/misc/tests/test_pilutil.py", 
line 24, in test_imresize
     assert_equal(im1.shape,(11,22))
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 174, in assert_equal
     assert_equal(len(actual),len(desired),err_msg,verbose)
   File 
"/usr/lib/python2.4/site-packages/numpy/testing/utils.py", 
line 183, in assert_equal
     raise AssertionError(msg)
AssertionError:
Items are not equal:
  ACTUAL: 0
  DESIRED: 2

----------------------------------------------------------------------
Ran 3460 tests in 159.021s

FAILED (KNOWNFAIL=2, SKIP=17, errors=6, failures=5)
<nose.result.TextTestResult run=3460 errors=6 failures=5>
  

Nils


From aisaac at american.edu  Thu Dec  4 15:27:16 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 04 Dec 2008 15:27:16 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <6ce0ac130812041120i27822402y8caa761dc1a02070@mail.gmail.com>
References: <gh754p$n6q$2@ger.gmane.org>	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>	<20081204102536.5faad9d5.rbastian@free.fr>	<20081204093944.GB18992@phare.normalesup.org>	<20081204111531.82da715a.rbastian@free.fr>
	<6ce0ac130812041120i27822402y8caa761dc1a02070@mail.gmail.com>
Message-ID: <49383D24.4070102@american.edu>

Brian Granger wrote:
> But, to get to the navigation column, I have to scroll to the top.

But that is just a design error,
simply fixed by suitable styling.
Alan Isaac


From pav at iki.fi  Thu Dec  4 15:36:46 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 4 Dec 2008 20:36:46 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
Message-ID: <gh9f0u$h5t$1@ger.gmane.org>

Thu, 04 Dec 2008 10:44:13 -0500, josef.pktd wrote:
>
> Just to provide my comment on search:
> 
> Compared to a windows helpfile, and single page search, sphinx search is
> very slow, especially when I'm on on online connection. I, now, got used
> to np.lookfor because it is much faster than trying to get through the
> online help.
> 
> I like the new Sphinx docs a lot if I have to look up a specific topic,
> but when I don't know which function might be useful for my case, then
> the overview and speed of looking around is pretty tedious, compared to
> the windows help files for matlab or R. (As an aside: I actually like
> the style sheet of the documentation editor better than sphinx.)

Can you be more specific: which elements of the page would you like to 
see changed?

> I haven't tried yet to generate my own numpy, scipy docs, since they are
> makefile based, and I didn't want to spend the time to figure out how to
> use the build scripts on windows, but a windows helpfile is on my
> wish/todo list, since sphinx makes it very easy to do.

Sure, HTML help workshop was easy to install on Linux too, so we can also 
build those.

-- 
Pauli Virtanen



From josef.pktd at gmail.com  Thu Dec  4 16:23:36 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 4 Dec 2008 16:23:36 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <gh9f0u$h5t$1@ger.gmane.org>
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gh9f0u$h5t$1@ger.gmane.org>
Message-ID: <1cd32cbb0812041323s37bdeb92v32b63097b73888ba@mail.gmail.com>

>>
>> I like the new Sphinx docs a lot if I have to look up a specific topic,
>> but when I don't know which function might be useful for my case, then
>> the overview and speed of looking around is pretty tedious, compared to
>> the windows help files for matlab or R. (As an aside: I actually like
>> the style sheet of the documentation editor better than sphinx.)
>
> Can you be more specific: which elements of the page would you like to
> see changed?

This might be mostly a taste question, I don't like the font size and
line spacing in sphinx css. I find smaller fonts and smaller line
spacing much easier to follow and much faster to skim for relevant
parts.

compare
http://docs.scipy.org/scipy/docs/scipy-docs/stats.rst/#stats

with sphinx rendered
http://docs.scipy.org/doc/scipy/reference/stats.html

also I like the overview tables of functions and classes that you have
in the editor
http://docs.scipy.org/scipy/docs/scipy.stats/#scipy-stats

As I mentioned I really like the windows help file format, with
search, tree and lists on the left navigation pane, together with the
"see also" it is very fast to check out different related functions.
In the stats front page, which is pretty long the toc tree on the left
does not have enough depth for navigation around.

A question to the new docs: what's the relationship between the front
rst page and the info.py of a sub package, the two files above, both
have currently roughly the same content. Should it stay that way or do
they have different purpose?

One problem for editing the doc string, I found:
stats.info.py is rendered here
http://docs.scipy.org/scipy/docs/scipy.stats/#scipy-stats
put clicking on source, leads to  __init__.py instead of info.py
http://docs.scipy.org/scipy/source/scipy/dist/lib64/python2.4/site-packages/scipy/stats/__init__.py#0


I will have to sign up for editing, since immediate feedback on rst
errors, is much better than only editing the source without feedback,
for example http://docs.scipy.org/scipy/docs/scipy.stats.stats.kstest/

Thanks,

Josef


From pav at iki.fi  Thu Dec  4 17:45:04 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 4 Dec 2008 22:45:04 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gh9f0u$h5t$1@ger.gmane.org>
	<1cd32cbb0812041323s37bdeb92v32b63097b73888ba@mail.gmail.com>
Message-ID: <gh9mhg$g4j$1@ger.gmane.org>

Thu, 04 Dec 2008 16:23:36 -0500, josef.pktd wrote:
[clip]
>> Can you be more specific: which elements of the page would you like to
>> see changed?
> 
> This might be mostly a taste question, I don't like the font size and
> line spacing in sphinx css. I find smaller fonts and smaller line
> spacing much easier to follow and much faster to skim for relevant
> parts.
> 
> compare
> http://docs.scipy.org/scipy/docs/scipy-docs/stats.rst/#stats
> 
> with sphinx rendered
> http://docs.scipy.org/doc/scipy/reference/stats.html

The font size in the Sphinx docs is indeed a bit large: it's the 
browser's default size which may vary across browsers. This should 
probably be fixed.

I'd believe the line spacing is the same for both, but it scales with 
font size.

> also I like the overview tables of functions and classes that you have
> in the editor
> http://docs.scipy.org/scipy/docs/scipy.stats/#scipy-stats

The summary listings try to accomplish this.

> As I mentioned I really like the windows help file format, with search,
> tree and lists on the left navigation pane, together with the "see also"
> it is very fast to check out different related functions. In the stats
> front page, which is pretty long the toc tree on the left does not have
> enough depth for navigation around.

It can be made to stay in the screen, but this requires some more serious 
CSS wizardry.

> A question to the new docs: what's the relationship between the front
> rst page and the info.py of a sub package, the two files above, both
> have currently roughly the same content. Should it stay that way or do
> they have different purpose?

In my opinion the documentation should go only in one of these two, and 
the other should be nearly empty. If it goes to info.py, then the 
corresponding rst file should only contain an automodule:: directive. 

The only reason why documentation couldn't go to info.py is that a part 
of the numpy-specific machinery doesn't handle the autosummary:: 
directives properly in docstrings.

Before the fix, I'd guess it's best to concentrate on the .rst docs. We 
can easily copy and paste the whole file contents to info.py, if we 
decide to change this.

[clip]
> One problem for editing the doc string, I found: stats.info.py is
> rendered here
> http://docs.scipy.org/scipy/docs/scipy.stats/#scipy-stats put clicking
> on source, leads to  __init__.py instead of info.py
> http://docs.scipy.org/scipy/source/scipy/dist/lib64/python2.4/site-
packages/scipy/stats/__init__.py#0

Hmm yes. There's no programmatic way of determining that __doc__ comes 
actually from info.py, apart from AST parsing. I'll have to think about 
whether to rewrite the docstring extraction code.

-- 
Pauli Virtanen



From gael.varoquaux at normalesup.org  Thu Dec  4 17:53:49 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 4 Dec 2008 23:53:49 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <gh9mhg$g4j$1@ger.gmane.org>
References: <20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gh9f0u$h5t$1@ger.gmane.org>
	<1cd32cbb0812041323s37bdeb92v32b63097b73888ba@mail.gmail.com>
	<gh9mhg$g4j$1@ger.gmane.org>
Message-ID: <20081204225349.GA16302@phare.normalesup.org>

On Thu, Dec 04, 2008 at 10:45:04PM +0000, Pauli Virtanen wrote:
> > This might be mostly a taste question, I don't like the font size and
> > line spacing in sphinx css. I find smaller fonts and smaller line
> > spacing much easier to follow and much faster to skim for relevant
> > parts.

> > compare
> > http://docs.scipy.org/scipy/docs/scipy-docs/stats.rst/#stats

> > with sphinx rendered
> > http://docs.scipy.org/doc/scipy/reference/stats.html

> The font size in the Sphinx docs is indeed a bit large: it's the 
> browser's default size which may vary across browsers. This should 
> probably be fixed.

Please don't change it. Screens are getting big nowadays and larger font
are easier to read. These fonts are not especially large. They are just
easy on the eye. I find myself browsing the web with the fonts blown up
all the time, and it screws the rendering of websites. For people with
small screens (eg laptops), they can reduce the font size (Ctrl-). Just
make it good for the majority of people.

Ga?l


From pav at iki.fi  Thu Dec  4 18:11:14 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 4 Dec 2008 23:11:14 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gh9f0u$h5t$1@ger.gmane.org>
	<1cd32cbb0812041323s37bdeb92v32b63097b73888ba@mail.gmail.com>
	<gh9mhg$g4j$1@ger.gmane.org>
Message-ID: <gh9o2i$g4j$2@ger.gmane.org>

Thu, 04 Dec 2008 22:45:04 +0000, Pauli Virtanen wrote:
[clip]
>> A question to the new docs: what's the relationship between the front
>> rst page and the info.py of a sub package, the two files above, both
>> have currently roughly the same content. Should it stay that way or do
>> they have different purpose?
> 
> In my opinion the documentation should go only in one of these two, and
> the other should be nearly empty. If it goes to info.py, then the
> corresponding rst file should only contain an automodule:: directive.
> 
> The only reason why documentation couldn't go to info.py is that a part
> of the numpy-specific machinery doesn't handle the autosummary::
> directives properly in docstrings.

This is fixed now, autosummary:: works in module docstrings.

I'd suggest moving the module reference documentation that is now in 
the .rst files to info.pys, in the long run, so that they are available 
via help(). Thought maybe it's better first wait for 0.7.0 to come out 
before making invasive changes.

-- 
Pauli Virtanen



From ellisonbg.net at gmail.com  Thu Dec  4 18:14:15 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 4 Dec 2008 15:14:15 -0800
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081204225349.GA16302@phare.normalesup.org>
References: <20081204102536.5faad9d5.rbastian@free.fr>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gh9f0u$h5t$1@ger.gmane.org>
	<1cd32cbb0812041323s37bdeb92v32b63097b73888ba@mail.gmail.com>
	<gh9mhg$g4j$1@ger.gmane.org>
	<20081204225349.GA16302@phare.normalesup.org>
Message-ID: <6ce0ac130812041514o59a2f376ndb6c9ff2f0577615@mail.gmail.com>

>> The font size in the Sphinx docs is indeed a bit large: it's the
>> browser's default size which may vary across browsers. This should
>> probably be fixed.
>
> Please don't change it. Screens are getting big nowadays and larger font
> are easier to read. These fonts are not especially large. They are just
> easy on the eye.

+1


From jh at physics.ucf.edu  Thu Dec  4 20:16:24 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Thu, 04 Dec 2008 20:16:24 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
Message-ID: <wl8skp3qv4n.fsf@glup.physics.ucf.edu>

Jarrod wrote:

> I would also prefer changing the name from "Tutorial" to "User Guide".

I disagree, for two reasons.  First, we are badly diluting our doc
effort and will fail to complete *any* documents to a user's
satisfaction with this approach.  We need to close the scipy doc wiki
(maintainers can edit in SVN), pause on tutorials, and focus on the
task at hand.  First document the core software.  Then document its
general use.  Of course, I can't force anyone to do this, but I hope
you all will.  Together, we are more than the sum of our parts.
Separately, we cannot get this job done.

Second, a (good) user manual takes a level of planning and commitment
that has not gone into any of our documents so far.  This tutorial
document cannot grow into a good user manual, because documents
*never* improve with organic growth.  They must be planned and written
with scope in mind from the outset.  We should reserve the name "User
Manual" for something we plan and execute to be good with that scope.

To expand (only a little, I promise!) on the User Guide, it needs
chapters on specific numpy topics, starting with the text in the new
numpy.doc module, and it needs chapters on general topics like
plotting, 3D visualization, handling datasets, scientific data
formats, etc.  Heavy-duty programming chapters belong in a separate
document (NumPy Programmer's Guide?).  Advanced tricks need to be
limited in the User Guide or it will be far too long.  It could go in
its own guide, or might better go in a web cookbook.

We are a grass-roots community, but good books are not, and have never
been, written by a haphazard grass-roots effort, with everyone writing
whatever they feel like writing.  The best you can hope for is to
scope and plan as a community, set goals for specific docs and
sections of docs, form small groups of writers, parcel out pieces with
specific goals to different groups, and then have a small editing team
work with the chapter teams to unify the pieces.  This is a standard
process in academic publishing and it works well.  Other than the
small-team-does-it-all model, I'm not aware of any other successful
authoring modes.  Grass roots works on the doc wiki only because we've
provided strict guidelines and process, and each page is done by just
one or a few people.

If we don't plan as a community and focus our efforts, the scattered
docs will reflect the interests of the writers rather than the
readers.  The quality will be quite variable and key items will never
be finished.

Please everyone,
new things are fun, 
but let's first finish
what we've begun.

Go write numpy docstrings.

--jh--


From david at ar.media.kyoto-u.ac.jp  Fri Dec  5 00:04:29 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 05 Dec 2008 14:04:29 +0900
Subject: [SciPy-dev] scipy.io.wav, data format convention
Message-ID: <4938B65D.2060603@ar.media.kyoto-u.ac.jp>

Hi,

    I wondered what would be the best data representation for audio
data-related functionalities in scipy. Generally, since numpy uses
"applying operation on last axis first", I thought it would make sense
to use the representation of one channel / row. OTOH, I've just found
out that scipy.io.wav does the contrary, and there is also the problem
on how to deal with mono signals (column vector vs array) - as well as
an efficiency question: most audio signals are interleaved (N
consecutive samples in memory for signal with N channels), so the format
of one column/channel is C-contiguous.
    I would like to keep audiolab as close as possible to whatever scipy
will use, but I wondered if anyone thought about the issue at all ?

cheers,

David



From gael.varoquaux at normalesup.org  Fri Dec  5 01:09:27 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 5 Dec 2008 07:09:27 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <wl8skp3qv4n.fsf@glup.physics.ucf.edu>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
Message-ID: <20081205060927.GA25416@phare.normalesup.org>

On Thu, Dec 04, 2008 at 08:16:24PM -0500, jh at physics.ucf.edu wrote:
> To expand (only a little, I promise!) on the User Guide, it needs
> chapters on specific numpy topics, starting with the text in the new
> numpy.doc module, and it needs chapters on general topics like
> plotting, 3D visualization, handling datasets, scientific data
> formats, etc.

I think you are talking of a very worthy goal. But I think this is not
the scipy user guide, this would be a book (maybe called "Python in
Science") that would give an introduction to the whole stack, and a
task-oriented approach with an overview of all the core packages.

This book cannot indeed be written by the community, as it requires a
consistent, well-thought, effort. On the other side, the scipy User Guide
is a bit more wikipedia-like. It is a searchable, and organised,
collection of technical documentation of each feature of SciPy, the
library. This can only be written and maintained by the community that
maintains SciPy. Especially since no one person knows SciPy well-enough
to do that.

My 2 cents,

Ga?l


From gael.varoquaux at normalesup.org  Fri Dec  5 01:11:41 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 5 Dec 2008 07:11:41 +0100
Subject: [SciPy-dev] scipy.io.wav, data format convention
In-Reply-To: <4938B65D.2060603@ar.media.kyoto-u.ac.jp>
References: <4938B65D.2060603@ar.media.kyoto-u.ac.jp>
Message-ID: <20081205061141.GB25416@phare.normalesup.org>

On Fri, Dec 05, 2008 at 02:04:29PM +0900, David Cournapeau wrote:
>     I would like to keep audiolab as close as possible to whatever scipy
> will use, but I wondered if anyone thought about the issue at all ?

Given that in the signal processing that I usually do, the last axis is
most often time, I tend to agree with you.

Ga?l


From scott.sinclair.za at gmail.com  Fri Dec  5 02:12:34 2008
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Fri, 5 Dec 2008 09:12:34 +0200
Subject: [SciPy-dev] scipy.test() errors/failures
In-Reply-To: <web-114767489@uni-stuttgart.de>
References: <web-114767489@uni-stuttgart.de>
Message-ID: <6a17e9ee0812042312i7fe89bc7s3594a0d115f5ad04@mail.gmail.com>

2008/12/4 Nils Wagner <nwagner at iam.uni-stuttgart.de>:
> Hi all,
>
> Can someone reproduce these errors/failures ? I am using
>
>>>> scipy.__version__
> '0.7.0.dev5221'
>>>> numpy.__version__
> '1.3.0.dev6136'
>
>
> ======================================================================
> ERROR: Testing that kmeans2 init methods work.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/cluster/tests/test_vq.py",
> line 147, in test_kmeans2_init
>     data = data.reshape((200, 2))
> ValueError: total size of new array must be unchanged
>
> ======================================================================
> ERROR: Testing simple call to kmeans2 with rank 1 data.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/cluster/tests/test_vq.py",
> line 127, in test_kmeans2_rank1
>     data = data.reshape((200, 2))
> ValueError: total size of new array must be unchanged
>
> ======================================================================
> ERROR: Testing simple call to kmeans2 with rank 1 data.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/cluster/tests/test_vq.py",
> line 139, in test_kmeans2_rank1_2
>     data = data.reshape((200, 2))
> ValueError: total size of new array must be unchanged
>
> ======================================================================
> ERROR: This will cause kmean to have a cluster with no
> points.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/cluster/tests/test_vq.py",
> line 96, in test_kmeans_lost_cluster
>     data = data.reshape((200, 2))
> ValueError: total size of new array must be unchanged
>
> ======================================================================
> ERROR: test_complex_write_read
> (test_mmio.TestMMIOCoordinate)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py",
> line 290, in test_complex_write_read
>     b = mmread(fn).todense()
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line
> 52, in mmread
>     return MMFile().read(source)
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line
> 273, in read
>     return self._parse_body(stream)
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line
> 407, in _parse_body
>     flat_data = flat_data.reshape(-1,4)
> ValueError: total size of new array must be unchanged
>
> ======================================================================
> ERROR: read a hermitian matrix
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py",
> line 213, in test_read_hermitian
>     b = mmread(fn).todense()
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line
> 52, in mmread
>     return MMFile().read(source)
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line
> 273, in read
>     return self._parse_body(stream)
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/mmio.py", line
> 407, in _parse_body
>     flat_data = flat_data.reshape(-1,4)
> ValueError: total size of new array must be unchanged
>
> ======================================================================
> FAIL: read a general matrix
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py",
> line 199, in test_read_geneal
>     assert_array_almost_equal(a,b)
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 310, in assert_array_almost_equal
>     header='Arrays are not almost equal')
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 295, in assert_array_compare
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal
>
> (mismatch 28.0%)
>  x: array([[  1.00000000e+00,   0.00000000e+00,
>  0.00000000e+00,
>           6.00000000e+00,   0.00000000e+00],
>        [  0.00000000e+00,   1.05000000e+01,
>  0.00000000e+00,...
>  y: array([[ 1.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.],...
>
> ======================================================================
> FAIL: read a skew-symmetric matrix
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py",
> line 229, in test_read_skew
>     assert_array_almost_equal(a,b)
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 310, in assert_array_almost_equal
>     header='Arrays are not almost equal')
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 295, in assert_array_compare
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal
>
> (mismatch 24.0%)
>  x: array([[  1.00000000e+00,   0.00000000e+00,
>  0.00000000e+00,
>           0.00000000e+00,   0.00000000e+00],
>        [  0.00000000e+00,   1.05000000e+01,
>  0.00000000e+00,...
>  y: array([[ 1.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.],...
>
> ======================================================================
> FAIL: test_real_write_read (test_mmio.TestMMIOCoordinate)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py",
> line 275, in test_real_write_read
>     assert_array_almost_equal(a,b)
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 310, in assert_array_almost_equal
>     header='Arrays are not almost equal')
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 295, in assert_array_compare
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal
>
> (mismatch 24.0%)
>  x: array([[  1.00000000e+00,   0.00000000e+00,
>  0.00000000e+00,
>           6.00000000e+00,   0.00000000e+00],
>        [  0.00000000e+00,   1.05000000e+01,
>  0.00000000e+00,...
>  y: array([[  1.,   0.,   0.,   6.,   0.],
>        [  0.,  10.,   0.,   0.,   0.],
>        [  0.,   0.,   0.,   0.,   0.],...
>
> ======================================================================
> FAIL: test_sparse_formats (test_mmio.TestMMIOCoordinate)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/scipy/io/tests/test_mmio.py",
> line 313, in test_sparse_formats
>     assert_array_almost_equal(result, expected)
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 310, in assert_array_almost_equal
>     header='Arrays are not almost equal')
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 295, in assert_array_compare
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal
>
> (mismatch 24.0%)
>  x: array([[  1.,   0.,   0.,   6.,   0.],
>        [  0.,  10.,   0.,   0.,   0.],
>        [  0.,   0.,   0.,   0.,   0.],...
>  y: array([[  1.00000000e+00,   0.00000000e+00,
>  0.00000000e+00,
>           6.00000000e+00,   0.00000000e+00],
>        [  0.00000000e+00,   1.05000000e+01,
>  0.00000000e+00,...
>
> ======================================================================
> FAIL: test_imresize (test_pilutil.TestPILUtil)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/decorators.py",
> line 82, in skipper
>     return f(*args, **kwargs)
>   File
> "/usr/lib/python2.4/site-packages/scipy/misc/tests/test_pilutil.py",
> line 24, in test_imresize
>     assert_equal(im1.shape,(11,22))
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 174, in assert_equal
>     assert_equal(len(actual),len(desired),err_msg,verbose)
>   File
> "/usr/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 183, in assert_equal
>     raise AssertionError(msg)
> AssertionError:
> Items are not equal:
>  ACTUAL: 0
>  DESIRED: 2
>
> ----------------------------------------------------------------------
> Ran 3460 tests in 159.021s
>
> FAILED (KNOWNFAIL=2, SKIP=17, errors=6, failures=5)
> <nose.result.TextTestResult run=3460 errors=6 failures=5>

I don't see these on 32 bit Ubuntu Intrepid with newer svn versions:

Running unit tests for scipy
NumPy version 1.3.0.dev6137
NumPy is installed in
/home/scott/.virtualenvs/scipy-dev/lib/python2.5/site-packages/numpy
SciPy version 0.7.0.dev5226
SciPy is installed in
/home/scott/.virtualenvs/scipy-dev/lib/python2.5/site-packages/scipy
Python version 2.5.2 (r252:60911, Oct  5 2008, 19:24:49) [GCC 4.3.2]
nose version 0.10.3

<snip>

Ran 3308 tests in 100.146s

OK (KNOWNFAIL=2, SKIP=28)

Cheers,
Scott


From cournape at gmail.com  Fri Dec  5 02:20:30 2008
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 5 Dec 2008 16:20:30 +0900
Subject: [SciPy-dev] scipy.io.wav, data format convention
In-Reply-To: <20081205061141.GB25416@phare.normalesup.org>
References: <4938B65D.2060603@ar.media.kyoto-u.ac.jp>
	<20081205061141.GB25416@phare.normalesup.org>
Message-ID: <5b8d13220812042320p3903d077k11ddfd7d82fb6018@mail.gmail.com>

On Fri, Dec 5, 2008 at 3:11 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Fri, Dec 05, 2008 at 02:04:29PM +0900, David Cournapeau wrote:
>>     I would like to keep audiolab as close as possible to whatever scipy
>> will use, but I wondered if anyone thought about the issue at all ?
>
> Given that in the signal processing that I usually do, the last axis is
> most often time, I tend to agree with you.

Agree to what ? I don't even agree with myself here :) audiolab used
the one column/channel convention, and during the conversion to one
row / channel, I realized that it is not as clear as I first thought -
if there is no clear consensus, I will keep the current audiolab
convention.

David


From nwagner at iam.uni-stuttgart.de  Fri Dec  5 02:30:19 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 05 Dec 2008 08:30:19 +0100
Subject: [SciPy-dev] numpy/scipy docs and make html
In-Reply-To: <gh91jh$3ik$1@ger.gmane.org>
References: <web-114761847@uni-stuttgart.de>
 <gh91jh$3ik$1@ger.gmane.org>
Message-ID: <web-114777395@uni-stuttgart.de>

On Thu, 4 Dec 2008 16:47:45 +0000 (UTC)
  Pauli Virtanen <pav at iki.fi> wrote:
> Thu, 04 Dec 2008 17:25:07 +0100, Nils Wagner wrote:
>> A make html in numpy/doc failed
> [clip]
>> ! LaTeX Error: File `preview.sty' not found.
> 
> You need the 'preview' Latex package so that Sphinx can 
>get the baseline 
> of the formulas right. See 
> 
>    http://sphinx.pocoo.org/ext/math.html#confval-pngmath_use_preview
> 
> or turn pngmath_use_preview off in source/conf.py.
> 
> -- 
> Pauli Virtanen
> 
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev

Hi Pauli,

I have disabled pngmath_use_preview.
Now a make html yields

spatial spatial.distance Math extension error:
latex exited with error:
[stderr]

[stdout]
This is pdfeTeX, Version 3.141592-1.21a-2.2 (Web2C 7.5.4)
entering extended mode
(./math.tex
LaTeX2e <2003/12/01>
Babel <v3.8d> and hyphenation patterns for american, 
french, german, ngerman, b
ahasa, basque, bulgarian, catalan, croatian, czech, 
danish, dutch, esperanto, e
stonian, finnish, greek, icelandic, irish, italian, latin, 
magyar, norsk, polis
h, portuges, romanian, russian, serbian, slovak, slovene, 
spanish, swedish, tur
kish, ukrainian, nohyphenation, loaded.
(/usr/share/texmf/tex/latex/base/article.cls
Document Class: article 2004/02/16 v1.4f Standard LaTeX 
document class
(/usr/share/texmf/tex/latex/base/size12.clo))
(/usr/share/texmf/tex/latex/base/inputenc.sty
(/usr/share/texmf/tex/latex/base/utf8.def
(/usr/share/texmf/tex/latex/base/t1enc.dfu)
(/usr/share/texmf/tex/latex/base/ot1enc.dfu)
(/usr/share/texmf/tex/latex/base/omsenc.dfu)))
(/usr/share/texmf/tex/latex/amsmath/amsmath.sty
For additional information on amsmath, use the `?' option.
(/usr/share/texmf/tex/latex/amsmath/amstext.sty
(/usr/share/texmf/tex/latex/amsmath/amsgen.sty))
(/usr/share/texmf/tex/latex/amsmath/amsbsy.sty)
(/usr/share/texmf/tex/latex/amsmath/amsopn.sty))
(/usr/share/texmf/tex/latex/amscls/amsthm.sty)
(/usr/share/texmf/tex/latex/amsfonts/amssymb.sty
(/usr/share/texmf/tex/latex/amsfonts/amsfonts.sty))
(/usr/share/texmf/tex/latex/tools/bm.sty) (./math.aux)
(/usr/share/texmf/tex/latex/amsfonts/umsa.fd)
(/usr/share/texmf/tex/latex/amsfonts/umsb.fd)
! Missing } inserted.
<inserted text>
                 }
l.15 \end{gather}

! Missing { inserted.
<inserted text>
                 {
l.15 \end{gather}

! Missing } inserted.
<inserted text>
                 }
l.15 \end{gather}

! Missing { inserted.
<inserted text>
                 {
l.15 \end{gather}

[1] (./math.aux) )
(see the transcript file for additional information)
Output written on math.dvi (1 page, 580 bytes).
Transcript written on math.log.

make: *** [html] Error 1
  
Nils


From gael.varoquaux at normalesup.org  Fri Dec  5 03:08:37 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 5 Dec 2008 09:08:37 +0100
Subject: [SciPy-dev] scipy.io.wav, data format convention
In-Reply-To: <5b8d13220812042320p3903d077k11ddfd7d82fb6018@mail.gmail.com>
References: <4938B65D.2060603@ar.media.kyoto-u.ac.jp>
	<20081205061141.GB25416@phare.normalesup.org>
	<5b8d13220812042320p3903d077k11ddfd7d82fb6018@mail.gmail.com>
Message-ID: <20081205080837.GB1780@phare.normalesup.org>

On Fri, Dec 05, 2008 at 04:20:30PM +0900, David Cournapeau wrote:
> > Given that in the signal processing that I usually do, the last axis is
> > most often time, I tend to agree with you.

> Agree to what ? I don't even agree with myself here :) audiolab used
> the one column/channel convention, and during the conversion to one
> row / channel, I realized that it is not as clear as I first thought -
> if there is no clear consensus, I will keep the current audiolab
> convention.

I meant, "one row = one channel". But anyhow, I don't think it matters
that much. I check the shape of everything before I do anything.

Ga?l


From pav at iki.fi  Fri Dec  5 04:00:26 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 5 Dec 2008 09:00:26 +0000 (UTC)
Subject: [SciPy-dev] scipy.test() errors/failures
References: <web-114767489@uni-stuttgart.de>
Message-ID: <ghaqja$loa$1@ger.gmane.org>

Thu, 04 Dec 2008 20:55:14 +0100, Nils Wagner wrote:

> Hi all,
> 
> Can someone reproduce these errors/failures ? I am using
> 
>>>> scipy.__version__
> '0.7.0.dev5221'
>>>> numpy.__version__
> '1.3.0.dev6136'
> 
> 
> ======================================================================
> ERROR: Testing that kmeans2 init methods work.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File
> "/usr/lib/python2.4/site-packages/scipy/cluster/tests/test_vq.py", line
> 147, in test_kmeans2_init
>      data = data.reshape((200, 2))
> ValueError: total size of new array must be unchanged
[clip]

I think this is a locale issue. Try running

	LC_ALL=C python -c 'import scipy; scipy.test()'

and checking if the errors go away.

Relevant tickets:

	http://scipy.org/scipy/numpy/ticket/884
	http://scipy.org/scipy/numpy/ticket/909

and a duplicate (of 884):

	http://scipy.org/scipy/numpy/ticket/902

-- 
Pauli Virtanen



From Per.Brodtkorb at ffi.no  Fri Dec  5 04:03:59 2008
From: Per.Brodtkorb at ffi.no (Per.Brodtkorb at ffi.no)
Date: Fri, 5 Dec 2008 10:03:59 +0100
Subject: [SciPy-dev] I would like to contribute to the scipy documentation.
Message-ID: <1ED225FF18AA8B48AC192F7E1D032C6E0C5C4C@hbu-posten.ffi.no>

Could someone give me edit rights?

 

My user name is pbrod.

 

Best regards Per A

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081205/7af10a9e/attachment.html>

From gael.varoquaux at normalesup.org  Fri Dec  5 04:27:58 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 5 Dec 2008 10:27:58 +0100
Subject: [SciPy-dev] I would like to contribute to the
	scipy	documentation.
In-Reply-To: <1ED225FF18AA8B48AC192F7E1D032C6E0C5C4C@hbu-posten.ffi.no>
References: <1ED225FF18AA8B48AC192F7E1D032C6E0C5C4C@hbu-posten.ffi.no>
Message-ID: <20081205092758.GD1780@phare.normalesup.org>

On Fri, Dec 05, 2008 at 10:03:59AM +0100, Per.Brodtkorb at ffi.no wrote:
>    Could someone give me edit rights?

You should be good to go now.

Thanks for your contributions,

Ga?l


From pav at iki.fi  Fri Dec  5 05:56:42 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 5 Dec 2008 10:56:42 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
Message-ID: <ghb1d9$loa$2@ger.gmane.org>

Thu, 04 Dec 2008 20:16:24 -0500, jh wrote:
> Jarrod wrote:
> 
>> I would also prefer changing the name from "Tutorial" to "User Guide".
> 
> I disagree, for two reasons.  First, we are badly diluting our doc
> effort and will fail to complete *any* documents to a user's
> satisfaction with this approach.  We need to close the scipy doc wiki

I do not believe that preventing people from editing scipy docstrings 
will improve numpy documentation in any way; it will only prevent point 
fixes to Scipy docstrings (eg. people becoming annoyed with a specific 
bad docstring, and fixing it). We can say instead that "concentrate on 
Numpy", strongly, on the front page, and remove scipy links from the 
table on the front page.

What in my opinion could help Numpy documentation now is coordination. 
For example via milestones:

- Compile a list of functions; sort it appropriately, and parcel it out
  to manageable chunks (say, tens of items).

- One possible way to parcel these could be to work in the order the
  functions currently appear in the current reference guide.

- I'd include a couple of "unimportant" functions to each milestone,
  so that we don't end up with dull milestones at the end of the run.

- Ask editors to work from top to bottom, one parcel at a time. Ditto for
  reviewers.

- I wouldn't necessarily include any deadlines, as these will slip,
  which is demotivating.

This should be fairly easy to compose, just requires putting up a new 
wiki page with appropriate links and sections for this. This could also 
speed up the review and QA that I feel is now not really progressing.

> Second, a (good) user manual takes a level of planning and commitment
> that has not gone into any of our documents so far.

I don't think it's fair to say that no planning has been done. The 
content of the Numpy reference guide is based on Travis's work, and I 
streamlined it by editing out the user-guidish parts.

Criticism is of course welcome, but it's more helpful to be specific.
I have seen little to no feedback on this.

> This tutorial
> document cannot grow into a good user manual, because documents *never*
> improve with organic growth.  They must be planned and written with
> scope in mind from the outset.  We should reserve the name "User Manual"
> for something we plan and execute to be good with that scope.
>
> To expand (only a little, I promise!) on the User Guide, it needs
> chapters on specific numpy topics, starting with the text in the new
> numpy.doc module, and it needs chapters on general topics like plotting,
> 3D visualization, handling datasets, scientific data formats, etc. 
> Heavy-duty programming chapters belong in a separate document (NumPy
> Programmer's Guide?).  Advanced tricks need to be limited in the User
> Guide or it will be far too long.  It could go in its own guide, or
> might better go in a web cookbook.

I believe, like Ga?l, that the "Scipy User Guide" (or Tutorial, whatever 
we choose to call it) can be more restricted in scope than what you 
describe above. Basically, we need only narrative documentation that 
explains background on what each of the parts does, gives motivation why 
and where it is useful, and gives examples how the functionality can be 
used that are more complete than those in the function docstrings.

It is probably much easier to write this type of a document for Scipy 
than for Numpy, since the functionality in Scipy splits naturally to 
parts along submodule boundaries, and each of the submodules concentrates 
on a single topic. What you say above about planning of course applies 
inside each of these topics, but the problem is already less difficult, 
at least for the more narrowly focused pieces (e.g. `scipy.odr`). For the 
larger modules, like interpolate, the problem seems to split according to 
different groups of functionality.

But yes, for each of the parts of Scipy, someone needs to have a vision 
of how the part of the document should be organized, and lay out the 
section structure. The current push for Scipy was to make use of material 
that was already written, in time for 0.7.0. We can maybe put off the 
push for tackling general documentation for the present, but I wouldn't 
object if someone already wants to stub out some of the Scipy submodules 
-- the fact that the writer must know the module well already limits the 
number of people to a small team, I'd believe.

> We are a grass-roots community, but good books are not, and have never
> been, written by a haphazard grass-roots effort, with everyone writing
> whatever they feel like writing.  The best you can hope for is to scope
> and plan as a community, set goals for specific docs and sections of
> docs, form small groups of writers, parcel out pieces with specific
> goals to different groups, and then have a small editing team work with
> the chapter teams to unify the pieces.  This is a standard process in
> academic publishing and it works well.  Other than the
> small-team-does-it-all model, I'm not aware of any other successful
> authoring modes.  Grass roots works on the doc wiki only because we've
> provided strict guidelines and process, and each page is done by just
> one or a few people.

I completely agree with this. Adding random snippets one by one to the 
wiki will not work well for the narrative documentation, so let's not do 
it like that. Maybe we can ask people to first stub out the complete 
structure for a part, and recommend point edits only after the structure 
is in place.

-- 
Pauli Virtanen



From stijn.deweirdt at ugent.be  Fri Dec  5 12:24:14 2008
From: stijn.deweirdt at ugent.be (Stijn De Weirdt)
Date: Fri, 05 Dec 2008 18:24:14 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
Message-ID: <1228497854.3733.32.camel@spike.ugent.be>

hi all,

i hope i can bother the dev list with this, as it's not a real user
problem imho.

i'm trying to compile python+numpy+scipy with intel toolkit (ictce
3.2.0.17, which has icc/ifc 11.0.069 and mkl 10.1.0.015


python 2.4.5 compiles and the make tests works fine
numpy 1.2.1 compiles and the tests are also fine
scipy 0.6.0 compiles, but one of the tests exits with a segmentation
fault. (i also tried 0.7.0b1, but it also gives segfualt, but haven't
verified if it's the same one)

now i'm trying to figure out which test causes the crash and i have some
questions related to that:
a. how can i skip/reorder tests? (just to check if it's an individual
issue or a broader one)
b. the segfault itself: 
python -v -c 'import scipy; scipy.test(level=10,verbosity=100)'
...

bench_random (scipy.fftpack.tests.test_basic.test_fft)
                 Fast Fourier Transform
=================================================
      |    real input     |   complex input    
-------------------------------------------------
 size |  scipy  |  numpy  |  scipy  |  numpy 
-------------------------------------------------
  100 |    0.07 |    0.06Segmentation fault

looking at the code in scipy/fftpack/tests/test_basic.py, it's the
complex part of bench_random that fails.

the basic code that triggers it, is (at least that how i read it)

x= random([size]).astype(cdouble)+random([size]).astype(cdou\
ble)*1j
y = direct_dft(x) ## size < 500
z=fft(x)

with fft loaded from scipy.fftpack (not from numpy)
(i'm sure it segfaults during the fft(x) call, because i added some
print statements to locate the exact code).

when i add some pprint statements to dump the x values that were used
during the segfault and write a small script doing just the above with
those values, it doesn't segfault. 
so doing the exact steps (except from the self.measure) in a separate
script, i'm unable to trigger the segfault, but the tests always fail.

could someone shine some light on why this seems not reproducable
outside the test environment?

many thanks,

stijn

-- 
The system will shutdown in 5 minutes.



From matthieu.brucher at gmail.com  Fri Dec  5 13:00:03 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 5 Dec 2008 19:00:03 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <1228497854.3733.32.camel@spike.ugent.be>
References: <1228497854.3733.32.camel@spike.ugent.be>
Message-ID: <e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>

Hi,

I think you are in trouble:
http://matt.eifelle.com/2008/11/03/i-used-the-latest-mkl-with-numpy-and.../
Unfortunately, I don't have a solution out of the box. It would need
some distutils hack to solve the issue. Use 10.0.2 instead.

Matthieu

2008/12/5 Stijn De Weirdt <stijn.deweirdt at ugent.be>:
> hi all,
>
> i hope i can bother the dev list with this, as it's not a real user
> problem imho.
>
> i'm trying to compile python+numpy+scipy with intel toolkit (ictce
> 3.2.0.17, which has icc/ifc 11.0.069 and mkl 10.1.0.015
>
>
> python 2.4.5 compiles and the make tests works fine
> numpy 1.2.1 compiles and the tests are also fine
> scipy 0.6.0 compiles, but one of the tests exits with a segmentation
> fault. (i also tried 0.7.0b1, but it also gives segfualt, but haven't
> verified if it's the same one)
>
> now i'm trying to figure out which test causes the crash and i have some
> questions related to that:
> a. how can i skip/reorder tests? (just to check if it's an individual
> issue or a broader one)
> b. the segfault itself:
> python -v -c 'import scipy; scipy.test(level=10,verbosity=100)'
> ...
>
> bench_random (scipy.fftpack.tests.test_basic.test_fft)
>                 Fast Fourier Transform
> =================================================
>      |    real input     |   complex input
> -------------------------------------------------
>  size |  scipy  |  numpy  |  scipy  |  numpy
> -------------------------------------------------
>  100 |    0.07 |    0.06Segmentation fault
>
> looking at the code in scipy/fftpack/tests/test_basic.py, it's the
> complex part of bench_random that fails.
>
> the basic code that triggers it, is (at least that how i read it)
>
> x= random([size]).astype(cdouble)+random([size]).astype(cdou\
> ble)*1j
> y = direct_dft(x) ## size < 500
> z=fft(x)
>
> with fft loaded from scipy.fftpack (not from numpy)
> (i'm sure it segfaults during the fft(x) call, because i added some
> print statements to locate the exact code).
>
> when i add some pprint statements to dump the x values that were used
> during the segfault and write a small script doing just the above with
> those values, it doesn't segfault.
> so doing the exact steps (except from the self.measure) in a separate
> script, i'm unable to trigger the segfault, but the tests always fail.
>
> could someone shine some light on why this seems not reproducable
> outside the test environment?
>
> many thanks,
>
> stijn
>
> --
> The system will shutdown in 5 minutes.
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From jh at physics.ucf.edu  Fri Dec  5 13:10:46 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Fri, 05 Dec 2008 13:10:46 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081205060927.GA25416@phare.normalesup.org> (message from Gael
	Varoquaux on Fri, 5 Dec 2008 07:09:27 +0100)
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
Message-ID: <wl8fxl2qyqh.fsf@glup.physics.ucf.edu>

Gael, what you describe below is just a collection of documents about
each of scipy's modules, written by the maintainers of those modules.
That is valuable to have, but it is not a user guide.  If you look at
the User Guide for any of our competing environments (IDL, Matlab,
Mathematica, etc.), or indeed for any large software package, you'll
find a document that is integrated across the areas of functionality.
It is oriented toward tasks the user wants to do, not necessarily the
structure of the software.  The "User Guide" name should be reserved
for a document that is planned from the start to Guide the User in
tasks of great interest to most users.  What you describe is one step
up from reference documentation, and it is useful to have, but it is
not integrative.

--jh--

Date: Fri, 5 Dec 2008 07:09:27 +0100
From: Gael Varoquaux <gael.varoquaux at normalesup.org>
To: jh at physics.ucf.edu, SciPy Developers List <scipy-dev at scipy.org>
Subject: Re:  [SciPy-dev] Scipy Tutorial (and updating it)
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
In-Reply-To: <wl8skp3qv4n.fsf at glup.physics.ucf.edu>

On Thu, Dec 04, 2008 at 08:16:24PM -0500, jh at physics.ucf.edu wrote:
> To expand (only a little, I promise!) on the User Guide, it needs
> chapters on specific numpy topics, starting with the text in the new
> numpy.doc module, and it needs chapters on general topics like
> plotting, 3D visualization, handling datasets, scientific data
> formats, etc.

I think you are talking of a very worthy goal. But I think this is not
the scipy user guide, this would be a book (maybe called "Python in
Science") that would give an introduction to the whole stack, and a
task-oriented approach with an overview of all the core packages.

This book cannot indeed be written by the community, as it requires a
consistent, well-thought, effort. On the other side, the scipy User Guide
is a bit more wikipedia-like. It is a searchable, and organised,
collection of technical documentation of each feature of SciPy, the
library. This can only be written and maintained by the community that
maintains SciPy. Especially since no one person knows SciPy well-enough
to do that.

My 2 cents,

Ga?l



From pav at iki.fi  Fri Dec  5 14:19:45 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 5 Dec 2008 19:19:45 +0000 (UTC)
Subject: [SciPy-dev] numpy/scipy docs and make html
References: <web-114761847@uni-stuttgart.de> <gh91jh$3ik$1@ger.gmane.org>
	<web-114777395@uni-stuttgart.de>
Message-ID: <ghbush$q58$1@ger.gmane.org>

Fri, 05 Dec 2008 08:30:19 +0100, Nils Wagner wrote:

[clip]
> I have disabled pngmath_use_preview.
> Now a make html yields
> 
> spatial spatial.distance Math extension error: latex exited with error:
> [stderr]
[clip]

Upgrade to Sphinx 0.5. Latex errors are no more fatal for it.

-- 
Pauli Virtanen



From stijn.deweirdt at ugent.be  Fri Dec  5 15:00:08 2008
From: stijn.deweirdt at ugent.be (Stijn De Weirdt)
Date: Fri, 05 Dec 2008 21:00:08 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>
Message-ID: <1228507208.8840.9.camel@spike.ugent.be>

hi,

just to be clear, i'm not asking any advice how to build scipy with icc,
i just want to now how to debug the problem and skip some of the tests.

to get back to the link blog: what is described there is a numpy issue
and i dealt with that using the advice the guy got from intel: link mkl
blas and lapack statically. 
numpy builds and works, it's scipy that causes the segfault (although
scipy uses the same build options as numpy. (i'm not saying that the
segfault has nothing to do with using icc/ifort/icpc, just that it's not
this specific problem).

i actually just found out that while running the scipy tests, gcc is
called. i always assumed that scipy takes whatever numpy uses. (it
seemed to work pretty well). maybe the tests don't?

stijn
 

> Hi,
> 
> I think you are in trouble:
> http://matt.eifelle.com/2008/11/03/i-used-the-latest-mkl-with-numpy-and.../
> Unfortunately, I don't have a solution out of the box. It would need
> some distutils hack to solve the issue. Use 10.0.2 instead.
> 
> Matthieu
> 
> 2008/12/5 Stijn De Weirdt <stijn.deweirdt at ugent.be>:
> > hi all,
> >
> > i hope i can bother the dev list with this, as it's not a real user
> > problem imho.
> >
> > i'm trying to compile python+numpy+scipy with intel toolkit (ictce
> > 3.2.0.17, which has icc/ifc 11.0.069 and mkl 10.1.0.015
> >
> >
> > python 2.4.5 compiles and the make tests works fine
> > numpy 1.2.1 compiles and the tests are also fine
> > scipy 0.6.0 compiles, but one of the tests exits with a segmentation
> > fault. (i also tried 0.7.0b1, but it also gives segfualt, but haven't
> > verified if it's the same one)
> >
> > now i'm trying to figure out which test causes the crash and i have some
> > questions related to that:
> > a. how can i skip/reorder tests? (just to check if it's an individual
> > issue or a broader one)
> > b. the segfault itself:
> > python -v -c 'import scipy; scipy.test(level=10,verbosity=100)'
> > ...
> >
> > bench_random (scipy.fftpack.tests.test_basic.test_fft)
> >                 Fast Fourier Transform
> > =================================================
> >      |    real input     |   complex input
> > -------------------------------------------------
> >  size |  scipy  |  numpy  |  scipy  |  numpy
> > -------------------------------------------------
> >  100 |    0.07 |    0.06Segmentation fault
> >
> > looking at the code in scipy/fftpack/tests/test_basic.py, it's the
> > complex part of bench_random that fails.
> >
> > the basic code that triggers it, is (at least that how i read it)
> >
> > x= random([size]).astype(cdouble)+random([size]).astype(cdou\
> > ble)*1j
> > y = direct_dft(x) ## size < 500
> > z=fft(x)
> >
> > with fft loaded from scipy.fftpack (not from numpy)
> > (i'm sure it segfaults during the fft(x) call, because i added some
> > print statements to locate the exact code).
> >
> > when i add some pprint statements to dump the x values that were used
> > during the segfault and write a small script doing just the above with
> > those values, it doesn't segfault.
> > so doing the exact steps (except from the self.measure) in a separate
> > script, i'm unable to trigger the segfault, but the tests always fail.
> >
> > could someone shine some light on why this seems not reproducable
> > outside the test environment?
> >
> > many thanks,
> >
> > stijn
> >
> > --
> > The system will shutdown in 5 minutes.
> >
> > _______________________________________________
> > Scipy-dev mailing list
> > Scipy-dev at scipy.org
> > http://projects.scipy.org/mailman/listinfo/scipy-dev
> >
> 
> 
> 
-- 
The system will shutdown in 5 minutes.



From matthieu.brucher at gmail.com  Fri Dec  5 15:46:14 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 5 Dec 2008 21:46:14 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <1228507208.8840.9.camel@spike.ugent.be>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>
	<1228507208.8840.9.camel@spike.ugent.be>
Message-ID: <e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>

2008/12/5 Stijn De Weirdt <stijn.deweirdt at ugent.be>:
> hi,
>
> just to be clear, i'm not asking any advice how to build scipy with icc,
> i just want to now how to debug the problem and skip some of the tests.
>
> to get back to the link blog: what is described there is a numpy issue

It's an MKL issue with every library that is linked against the
dynamic libraries. Thus Scipy.

> and i dealt with that using the advice the guy got from intel: link mkl
> blas and lapack statically.
> numpy builds and works,

Manually? Did you do the same for scipy ? Because the first time I ran
into this issue was with the FFT functions.
And the guy is me.

 it's scipy that causes the segfault (although
> scipy uses the same build options as numpy. (i'm not saying that the
> segfault has nothing to do with using icc/ifort/icpc, just that it's not
> this specific problem).

Another solution:
- make a simple Python script with the test inside
- call gdb python
- call r script.py

and tell us where it crashes ;)

> i actually just found out that while running the scipy tests, gcc is
> called. i always assumed that scipy takes whatever numpy uses. (it
> seemed to work pretty well). maybe the tests don't?

It may be weave that calls gcc.

-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From stijn.deweirdt at ugent.be  Fri Dec  5 16:04:20 2008
From: stijn.deweirdt at ugent.be (Stijn De Weirdt)
Date: Fri, 05 Dec 2008 22:04:20 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>
	<1228507208.8840.9.camel@spike.ugent.be>
	<e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>
Message-ID: <1228511060.11640.10.camel@spike.ugent.be>

salut matthieu,

> > and i dealt with that using the advice the guy got from intel: link mkl
> > blas and lapack statically.
> > numpy builds and works,
> 
> Manually? 
no. i modified the numpy distutils a little bit so it would accept the
-Bstatic and -Bdynamic switches, for everything else i used standard
numpy buld procedure (i also change intelccompiler.py a bit so numpy
would use icpc as cxx compiler)

> Did you do the same for scipy ? 
i could be completely wrong, but as far as i can tell, scipy reuses the
numpy distutils and site.cfg, and looking at the build logs, the same
options were used.

> Because the first time I ran
> into this issue was with the FFT functions.
when i say numpy works, i mean the unittests don't fail. (but i have set
the fftw3 intel interfaces in the numpy site.cfg. i haven't bothered yet
to check what scipy uses for fft, as i first want to reproduce the
segfault outside the test environment.)

> And the guy is me.
> 
ah. well, thanks for contacting intel for their advice. it saved me some
work ;)

>  it's scipy that causes the segfault (although
> > scipy uses the same build options as numpy. (i'm not saying that the
> > segfault has nothing to do with using icc/ifort/icpc, just that it's not
> > this specific problem).
> 
> Another solution:
> - make a simple Python script with the test inside
i tried but failed. and i don't want to figure out the whole nose test
environment just to do this ;)

> - call gdb python
> - call r script.py
> 
> and tell us where it crashes ;)
> 
> > i actually just found out that while running the scipy tests, gcc is
> > called. i always assumed that scipy takes whatever numpy uses. (it
> > seemed to work pretty well). maybe the tests don't?
> 
> It may be weave that calls gcc.
> 
but this should be 100% harmless? (i'm not familiar with the inner
working of scipy)


thanks

stijn
-- 
The system will shutdown in 5 minutes.



From matthieu.brucher at gmail.com  Fri Dec  5 16:29:43 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 5 Dec 2008 22:29:43 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <1228511060.11640.10.camel@spike.ugent.be>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>
	<1228507208.8840.9.camel@spike.ugent.be>
	<e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>
	<1228511060.11640.10.camel@spike.ugent.be>
Message-ID: <e76aa17f0812051329p2238c221kfc523a17f8c4d9f3@mail.gmail.com>

> no. i modified the numpy distutils a little bit so it would accept the
> -Bstatic and -Bdynamic switches, for everything else i used standard
> numpy buld procedure (i also change intelccompiler.py a bit so numpy
> would use icpc as cxx compiler)

I think a lot of people would love to what you did ;)

>> Did you do the same for scipy ?
> i could be completely wrong, but as far as i can tell, scipy reuses the
> numpy distutils and site.cfg, and looking at the build logs, the same
> options were used.

In this case, yes, it should ;)

>> Because the first time I ran
>> into this issue was with the FFT functions.
> when i say numpy works, i mean the unittests don't fail. (but i have set
> the fftw3 intel interfaces in the numpy site.cfg. i haven't bothered yet
> to check what scipy uses for fft, as i first want to reproduce the
> segfault outside the test environment.)
>
>> And the guy is me.
>>
> ah. well, thanks for contacting intel for their advice. it saved me some
> work ;)

No problem, I wanted to have an answer as well.

>>  it's scipy that causes the segfault (although
>> > scipy uses the same build options as numpy. (i'm not saying that the
>> > segfault has nothing to do with using icc/ifort/icpc, just that it's not
>> > this specific problem).
>>
>> Another solution:
>> - make a simple Python script with the test inside
> i tried but failed. and i don't want to figure out the whole nose test
> environment just to do this ;)

Then this is strange. I thought nose would help us in that matter.
I don't have icc and mkl on my own box now, so I cannot try myself. If
you give us the diff for numpy, we may eventually help you.

>> - call gdb python
>> - call r script.py
>>
>> and tell us where it crashes ;)
>>
>> > i actually just found out that while running the scipy tests, gcc is
>> > called. i always assumed that scipy takes whatever numpy uses. (it
>> > seemed to work pretty well). maybe the tests don't?
>>
>> It may be weave that calls gcc.
>>
> but this should be 100% harmless? (i'm not familiar with the inner
> working of scipy)

Completely. Weave is just a way to compile on the fly C++ code, and it
has no impact on fftpack.

Matthie
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From stijn.deweirdt at ugent.be  Fri Dec  5 16:58:50 2008
From: stijn.deweirdt at ugent.be (Stijn De Weirdt)
Date: Fri, 05 Dec 2008 22:58:50 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <e76aa17f0812051329p2238c221kfc523a17f8c4d9f3@mail.gmail.com>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>
	<1228507208.8840.9.camel@spike.ugent.be>
	<e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>
	<1228511060.11640.10.camel@spike.ugent.be>
	<e76aa17f0812051329p2238c221kfc523a17f8c4d9f3@mail.gmail.com>
Message-ID: <1228514330.13880.8.camel@spike.ugent.be>

> > no. i modified the numpy distutils a little bit so it would accept the
> > -Bstatic and -Bdynamic switches, for everything else i used standard
> > numpy buld procedure (i also change intelccompiler.py a bit so numpy
> > would use icpc as cxx compiler)
> 
> I think a lot of people would love to what you did ;)
> 
i paste my site.cfg and a patch here (i have no idea if the mailing list
accepts attachments ;)
keep in mind that -Bstatic and -Bdynamic are toggles and this has been
tested with ictce 3.2.0.017

(warning: ugly ;)

stijn


site.cfg (mkl related stuff, you get the idea)
-----------------------------------------------
[fftw]
libraries = -Bstatic,fftw3xc_intel,fftw3xf_intel,-Bdynamic
[mkl]
lapack_libs = -Bstatic,mkl_lapack,-Bdynamic
#mkl_libs = -Bstatic,mkl_intel_lp64, mkl_intel_thread,
mkl_core,-Bdynamic,iomp5, guide

mkl_libs = -Bstatic,mkl_em64t,-Bdynamic,iomp5, guide

-----------------------------------------------

patch for numpy, against 1.2.1

-----------------------------------------------
diff -ur --exclude='*pyc' --exclude='*~'
numpy-1.2.1.orig/numpy/distutils/fcompiler/intel.py
numpy-1.2.1/numpy/distutils/fcompiler/intel.py
--- numpy-1.2.1.orig/numpy/distutils/fcompiler/intel.py	2008-10-28
12:21:48.000000000 +0100
+++ numpy-1.2.1/numpy/distutils/fcompiler/intel.py	2008-12-04
22:40:19.000000000 +0100
@@ -152,9 +152,9 @@
 
     executables = {
         'version_cmd'  : None,
-        'compiler_f77' : [None, "-FI", "-w90", "-w95"],
-        'compiler_fix' : [None, "-FI"],
-        'compiler_f90' : [None],
+        'compiler_f77' : [None, "-FI", "-w90", "-w95","-v"],
+        'compiler_fix' : [None, "-FI","-v"],
+        'compiler_f90' : [None,"-v"],
         'linker_so'    : ['<F90>', "-shared"],
         'archiver'     : ["ar", "-cr"],
         'ranlib'       : ["ranlib"]
@@ -166,6 +166,13 @@
             opt.extend(['-tpp7', '-xW'])
         return opt
 
+    def library_option(self, lib):
+        if lib[0]=='-':
+            return lib
+        else:
+            return "-l" + lib
+        
+
 # Is there no difference in the version string between the above
compilers
 # and the Visual compilers?
 
diff -ur --exclude='*pyc' --exclude='*~'
numpy-1.2.1.orig/numpy/distutils/intelccompiler.py
numpy-1.2.1/numpy/distutils/intelccompiler.py
--- numpy-1.2.1.orig/numpy/distutils/intelccompiler.py	2008-10-28
12:21:49.000000000 +0100
+++ numpy-1.2.1/numpy/distutils/intelccompiler.py	2008-12-04
23:28:08.000000000 +0100
@@ -8,16 +8,30 @@
     """
 
     compiler_type = 'intel'
-    cc_exe = 'icc'
+    cc_exe = 'icc -v'
+    cxx_exe = 'icpc -v'
 
     def __init__ (self, verbose=0, dry_run=0, force=0):
         UnixCCompiler.__init__ (self, verbose,dry_run, force)
         compiler = self.cc_exe
-        self.set_executables(compiler=compiler,
-                             compiler_so=compiler,
-                             compiler_cxx=compiler,
-                             linker_exe=compiler,
-                             linker_so=compiler + ' -shared')
+        compiler_cxx = self.cxx_exe
+
+        opt=' -fPIC -O3 -unroll'
+        linkopt=' -shared -static-intel '
+
+        self.set_executables(compiler=compiler+opt,
+                             compiler_so=compiler+opt,
+                             compiler_cxx=compiler_cxx+opt,
+                             linker_exe=compiler+opt,
+                             linker_so=compiler+linkopt)
+
+    def library_option(self, lib):
+        if lib[0]=='-':
+            return lib
+        else:
+            return "-l" + lib
+
+                
 
 class IntelItaniumCCompiler(IntelCCompiler):
     compiler_type = 'intele'
diff -ur --exclude='*pyc' --exclude='*~'
numpy-1.2.1.orig/numpy/distutils/system_info.py
numpy-1.2.1/numpy/distutils/system_info.py
--- numpy-1.2.1.orig/numpy/distutils/system_info.py	2008-10-28
12:21:49.000000000 +0100
+++ numpy-1.2.1/numpy/distutils/system_info.py	2008-12-04
01:18:43.000000000 +0100
@@ -559,6 +559,9 @@
             lib_prefixes = ['lib']
         # for each library name, see if we can find a file for it.
         for l in libs:
+            if l[0]=='-':
+                liblist.append(l)
+                continue
             for ext in exts:
                 for prefix in lib_prefixes:
                     p = self.combine_paths(lib_dir, prefix+l+ext)
-----------------------------------------------------------------------



From ljosa at broad.mit.edu  Fri Dec  5 18:57:43 2008
From: ljosa at broad.mit.edu (Vebjorn Ljosa)
Date: Fri, 05 Dec 2008 18:57:43 -0500
Subject: [SciPy-dev] Patch for writing Matlab structs with long field names
Message-ID: <nq58wqunmg5.fsf@broad.mit.edu>

There is a bug in io/matlab/mio5.py: it can only write structs the file
names of which are shorter than 32 characters.  I am passing on a patch
from Lee Kamentsky <leek at broad.mit.edu> that extends this to the 64
characters, which is what Matlab can handle.

Thanks,
Vebjorn

Index: mio5.py
===================================================================
--- mio5.py	(revision 5227)
+++ mio5.py	(working copy)
@@ -808,9 +808,11 @@
     def write_fields(self):
         # write fieldnames
         fieldnames = [f[0] for f in self.arr.dtype.descr]
-        self.write_element(np.array([32], dtype='i4'))
-        self.write_element(np.array(fieldnames, dtype='S32'),
-                           mdtype=miINT8)
+        length = max([len(fieldname) for fieldname in fieldnames])+1
+        if length > 64:
+            raise ValueError, "Field names are restricted to 64 characters in Matlab"
+        self.write_element(np.array([length], dtype='i4'))
+        self.write_element(np.array(fieldnames, dtype='S%d'%(length)), mdtype=miINT8)
         A = np.atleast_2d(self.arr).flatten('F')
         MWG = Mat5WriterGetter(self.file_stream,
                                self.unicode_strings)


From matthew.brett at gmail.com  Fri Dec  5 21:08:53 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 5 Dec 2008 18:08:53 -0800
Subject: [SciPy-dev] Patch for writing Matlab structs with long field
	names
In-Reply-To: <nq58wqunmg5.fsf@broad.mit.edu>
References: <nq58wqunmg5.fsf@broad.mit.edu>
Message-ID: <1e2af89e0812051808j4ee24626r26e39e62a3001124@mail.gmail.com>

Hi,

On Fri, Dec 5, 2008 at 3:57 PM, Vebjorn Ljosa <ljosa at broad.mit.edu> wrote:
> There is a bug in io/matlab/mio5.py: it can only write structs the file
> names of which are shorter than 32 characters.  I am passing on a patch
> from Lee Kamentsky <leek at broad.mit.edu> that extends this to the 64
> characters, which is what Matlab can handle.

Thanks for the patch, it's good, but - what makes you think matlab can
handle 64 characters of fieldname?  The standard, here:

http://www.mathworks.com/access/helpdesk/help/pdf_doc/matlab/matfile_format.pdf

says:

"Field Name Length Subelement
This subelement specifies the maximum length of a Field Name. MATLAB
sets this limit to 32 (31 characters and a NULL terminator). In a MAT-file
created by MATLAB, this subelement always uses the compressed data
element format."

I take this to mean the maximum field name length is 31 characters.
Have you got other information somewhere?

Thanks again,

Matthew


From millman at berkeley.edu  Fri Dec  5 23:18:49 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Fri, 5 Dec 2008 20:18:49 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <49370152.3010905@american.edu>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<49370152.3010905@american.edu>
Message-ID: <c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>

On Wed, Dec 3, 2008 at 1:59 PM, Alan G Isaac <aisaac at american.edu> wrote:
> As for `ridder` and `brentq`, I hardly think NR
> would have a hope of copyrighting anything but
> their actual code implementing these.
> A classic reference should be given in any case:
>
> Ridders, C. F. J. "A New Algorithm for Computing a
> Single Root of a Real Continuous Function."
> IEEE Trans. Circuits Systems 26, 979-980, 1979.

Hello Alan,

Would you be willing to take care of `ridder` and `brentq`?  If you
are willing, I would appreciate it if you could verify that our
implementations are not derived from the Numerical Recipes code.  The
docstrings for these two functions seem to indicate that the code
differs from that in Numerical Recipes, so you may be able to find
references for our implementations.

Hopefully you could do somethings like Josef did for ks2_samp:
  http://projects.scipy.org/scipy/scipy/changeset/5223

Since we need to make sure we aren't violating the Numerical Recipes'
license (now that we have become aware of it) before releasing 0.7.0,
I hope that you will be able to take care of this ASAP.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From charlesr.harris at gmail.com  Sat Dec  6 03:19:26 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 6 Dec 2008 01:19:26 -0700
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<49370152.3010905@american.edu>
	<c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>
Message-ID: <e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>

On Fri, Dec 5, 2008 at 9:18 PM, Jarrod Millman <millman at berkeley.edu> wrote:

> On Wed, Dec 3, 2008 at 1:59 PM, Alan G Isaac <aisaac at american.edu> wrote:
> > As for `ridder` and `brentq`, I hardly think NR
> > would have a hope of copyrighting anything but
> > their actual code implementing these.
> > A classic reference should be given in any case:
> >
> > Ridders, C. F. J. "A New Algorithm for Computing a
> > Single Root of a Real Continuous Function."
> > IEEE Trans. Circuits Systems 26, 979-980, 1979.
>
> Hello Alan,
>
> Would you be willing to take care of `ridder` and `brentq`?  If you
> are willing, I would appreciate it if you could verify that our
> implementations are not derived from the Numerical Recipes code.  The
> docstrings for these two functions seem to indicate that the code
> differs from that in Numerical Recipes, so you may be able to find
> references for our implementations.
>
> Hopefully you could do somethings like Josef did for ks2_samp:
>  http://projects.scipy.org/scipy/scipy/changeset/5223
>
> Since we need to make sure we aren't violating the Numerical Recipes'
> license (now that we have become aware of it) before releasing 0.7.0,
> I hope that you will be able to take care of this ASAP.
>

Hi Jarrod,

I wrote the routines. They are similar too, but not the same as the NR
versions.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081206/75cd8337/attachment.html>

From scott.sinclair.za at gmail.com  Sat Dec  6 04:30:40 2008
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Sat, 6 Dec 2008 11:30:40 +0200
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
Message-ID: <6a17e9ee0812060130t5d9bec17l6f6551c8bc713f1b@mail.gmail.com>

> 2008/12/5  <jh at physics.ucf.edu>:
> The "User Guide" name should be reserved
> for a document that is planned from the start to Guide the User in
> tasks of great interest to most users.  What you describe is one step
> up from reference documentation, and it is useful to have, but it is
> not integrative.

Agreed.  However, I believe it makes most sense to aim at solid
reference documentation (module, class & function docstrings), with
instructive/tutorial narrative to make it more palatable wherever
possible.  This approach is most likely to produce consolidated (look
in one place) documentation that can be maintained, simply because it
is achievable using a model of many people contributing incrementally
and relatively independently.  The doc wiki is an excellent tool for
this, and for fleshing out the possible structure of Numpy and Scipy
user guides..

Cheers,
Scott


From scott.sinclair.za at gmail.com  Sat Dec  6 05:20:10 2008
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Sat, 6 Dec 2008 12:20:10 +0200
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <ghb1d9$loa$2@ger.gmane.org>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu> <ghb1d9$loa$2@ger.gmane.org>
Message-ID: <6a17e9ee0812060220g7690b865t325b52bfd2f1d4a2@mail.gmail.com>

> 2008/12/5 Pauli Virtanen <pav at iki.fi>:
> I do not believe that preventing people from editing scipy docstrings
> will improve numpy documentation in any way; it will only prevent point
> fixes to Scipy docstrings (eg. people becoming annoyed with a specific
> bad docstring, and fixing it). We can say instead that "concentrate on
> Numpy", strongly, on the front page, and remove scipy links from the
> table on the front page.

I don't think we should remove the Scipy links from the front page,
that just makes point fixes less likely.

> What in my opinion could help Numpy documentation now is coordination.
> For example via milestones:
>
> - Compile a list of functions; sort it appropriately, and parcel it out
>  to manageable chunks (say, tens of items).
>
> - One possible way to parcel these could be to work in the order the
>  functions currently appear in the current reference guide.
>
> - I'd include a couple of "unimportant" functions to each milestone,
>  so that we don't end up with dull milestones at the end of the run.
>
> - Ask editors to work from top to bottom, one parcel at a time. Ditto for
>  reviewers.
>
> - I wouldn't necessarily include any deadlines, as these will slip,
>  which is demotivating.
>
> This should be fairly easy to compose, just requires putting up a new
> wiki page with appropriate links and sections for this. This could also
> speed up the review and QA that I feel is now not really progressing.

I think that reviewing of docstrings is going to be tough to get going
because it relies on the input of people who are already heavily
involved in the project(s).  Any docstring that someone feels can be
marked "ready for review" is usually pretty good anyway, we should aim
to get as many docstrings as possible to this level before getting
overly concerned about review.

Here is a first attempt at some milestones achievable in the short term:

1. All functions in the numpy toplevel namespace to be "ready for
review" or better.
    - There are many docstrings marked as "being written", which are
almost done and not being worked on.

2. Make sure that ndarray and all sub-classes (Matrix, MaskedArray,
recarray etc.) have docstrings at "ready for review" level or better.

Cheers,
Scott


From gael.varoquaux at normalesup.org  Sat Dec  6 05:31:39 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 6 Dec 2008 11:31:39 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
Message-ID: <20081206103139.GA12839@phare.normalesup.org>

On Fri, Dec 05, 2008 at 01:10:46PM -0500, jh at physics.ucf.edu wrote:
> Gael, what you describe below is just a collection of documents about
> each of scipy's modules, written by the maintainers of those modules.
> That is valuable to have, but it is not a user guide.  If you look at
> the User Guide for any of our competing environments (IDL, Matlab,
> Mathematica, etc.), or indeed for any large software package, you'll
> find a document that is integrated across the areas of functionality.

Correct, and I agree that we need this. But this is not the SciPy user
guide. SciPy is only one of the packages. Unlike the other environments,
we have a separation of functionnality. From a software engineering point
of view, this is a good thing (eg it makes shipping pure numerical
algorithms easier, or it makes it possible to run them on embedded
systems). I agree that for a large classe of users, this separation makes
no sens. Thus I agree that an overall document is crucial. But the SciPy
User Guide isn't that.

Ga?l


From ljosa at broad.mit.edu  Sat Dec  6 06:14:51 2008
From: ljosa at broad.mit.edu (Vebjorn Ljosa)
Date: Sat, 06 Dec 2008 06:14:51 -0500
Subject: [SciPy-dev] Patch for writing Matlab structs with long field
	names
In-Reply-To: <1e2af89e0812051808j4ee24626r26e39e62a3001124@mail.gmail.com>
	(Matthew Brett's message of "Fri, 5 Dec 2008 18:08:53 -0800")
References: <nq58wqunmg5.fsf@broad.mit.edu>
	<1e2af89e0812051808j4ee24626r26e39e62a3001124@mail.gmail.com>
Message-ID: <nq54p1ho8r8.fsf@broad.mit.edu>

"Matthew Brett" <matthew.brett at gmail.com> writes:

> Thanks for the patch, it's good, but - what makes you think matlab can
> handle 64 characters of fieldname?

The actual behaviour of the current version of Matlab is to both read
and write fieldnames of up to 64 characters (63 characters + NUL).  We
discovered this because we encountered some .mat files in the wild that
broke our scipy scripts.

It is possible that some previous Matlab version can only handle 32
characters.  Even with the patch, this should not cause problems as long
as the actual fieldnames your write are no longer than 32 characters.

Vebjorn


From david at ar.media.kyoto-u.ac.jp  Sat Dec  6 06:04:41 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 06 Dec 2008 20:04:41 +0900
Subject: [SciPy-dev] Patch for writing Matlab structs with long field
 names
In-Reply-To: <nq54p1ho8r8.fsf@broad.mit.edu>
References: <nq58wqunmg5.fsf@broad.mit.edu>	<1e2af89e0812051808j4ee24626r26e39e62a3001124@mail.gmail.com>
	<nq54p1ho8r8.fsf@broad.mit.edu>
Message-ID: <493A5C49.4050605@ar.media.kyoto-u.ac.jp>

Vebjorn Ljosa wrote:
> "Matthew Brett" <matthew.brett at gmail.com> writes:
>
>   
>> Thanks for the patch, it's good, but - what makes you think matlab can
>> handle 64 characters of fieldname?
>>     
>
> The actual behaviour of the current version of Matlab is to both read
> and write fieldnames of up to 64 characters (63 characters + NUL).  We
> discovered this because we encountered some .mat files in the wild that
> broke our scipy scripts.
>
> It is possible that some previous Matlab version can only handle 32
> characters.  Even with the patch, this should not cause problems as long
> as the actual fieldnames your write are no longer than 32 characters.
>   

What about allowing reading up to 64 bytes strings but only writing up
to 32  - at least by default ?

David


From aisaac at american.edu  Sat Dec  6 11:14:37 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Sat, 06 Dec 2008 11:14:37 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>	<49370152.3010905@american.edu>	<c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>
	<e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>
Message-ID: <493AA4ED.3020800@american.edu>

> On Fri, Dec 5, 2008 at 9:18 PM, Jarrod Millman <millman at berkeley.edu wrote:
> Would you be willing to take care of `ridder` and 
> `brentq`?  If you are willing, I would appreciate it if 
> you could verify that our implementations are not derived 
> from the Numerical Recipes code.  The docstrings for these 
> two functions seem to indicate that the code
> differs from that in Numerical Recipes, so you may be able 
> to find references for our implementations.
> Hopefully you could do somethings like Josef did for 
> ks2_samp: 
> http://projects.scipy.org/scipy/scipy/changeset/5223


On 12/6/2008 3:19 AM Charles R Harris apparently wrote:
> I wrote the routines. They are similar too, but not the same as the NR 
> versions. 


So I take it then that is needed is a docstring revision?
I have attempted this for `ridder`:
http://docs.scipy.org/scipy/docs/scipy.optimize.zeros.ridder/
(Note that the parameter `disp` remains undocumented
as I do not know its function.)  If this is what is desired,
I am happy to change the brentq documentation similarly.
(http://docs.scipy.org/scipy/docs/scipy.optimize.zeros.brentq/)

Cheers,
Alan




From jh at physics.ucf.edu  Sat Dec  6 11:30:30 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Sat, 06 Dec 2008 11:30:30 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081206103139.GA12839@phare.normalesup.org> (message from Gael
	Varoquaux on Sat, 6 Dec 2008 11:31:39 +0100)
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
Message-ID: <wl8bpvpqna1.fsf@glup.physics.ucf.edu>

Here are the issues before us as I see them:

You would like:
  a document about each submodule in the SciPy package
    written by the developers/maintainers of those submodules, as experts
  to collect those together into one document
  to name that document "The SciPy User Manual"
  to work on this now

I would like:
  a document for the entire stack
    aimed at new and intermediate users
    using all components of the stack as needed in each chapter
    led by an editing team
    written to an agreed-upon style
    chapters to cover common areas of use
    chapters written by individuals or small teams
    chapter teams selected by an informal proposal process
    to name that document "The SciPy User Manual"
  not to start writing such documents until we have complete docstrings

I think both documents are valuable, as do you.  There are only two
differences:
  Who gets to call it "The SciPy User Manual"?
  When to do what?

On the latter point, I'll compare our situation to the US National
Debt.  We have a documentation deficit.  OVER THREE THOUSAND FUNCTIONS
in NumPy and SciPy DO NOT HAVE GOOD DOCSTRINGS (Gael, I know you know
this figure, I'm shouting for the community to wake up).  We pay
interest on that debt whenever we have to dig into source code or ask
on the mailing list to figure out what a function does.  Writing
higher-level manuals is misspending our resources until we have paid
our documentation debt.  We must get the backlog of documentation
cleared if we are ever to be considered a credible computing
environment.  There is a box of very cool T-shirts in my lab, waiting
to go out to whoever wants to write 1000 words!  (Who else gets to
wear frogs jumping off of sinusoids on a black background?)

Two points on naming.

1. "SciPy" is the only collective name we have, and it's how outsiders
refer to the software.  It's the name of the portal website, and we
considered this implication when the new site went up.  The software
engineering details of how we manage the stack should not push us into
poor choices about how we present the stack to the world.  "SciPy" is
the stack, and a package in the stack.  Not my decision, but we have
to live with it.  It is how outsiders see us, now.

2. "User Guide" has a specific meaning that a collection of submodule
manuals cannot serve.  It literally means an integrative document that
Guides the User in tasks of interest.  Users look for a document
called "User Guide" to learn how to use a suite of packages.  While
some chapters will focus on a specific module or set of modules,
others will not, and not all modules will get a chapter.

The document you describe is not integrative and omits critical things
like reading scientific data formats, plotting, 3D, and so forth that
are not in the SciPy package.  Much of it will also be written at the
expert level, etc.

So, I propose the following:

1. Reserve the name "The SciPy User Guide" for the document I describe
(whether or not I am involved in creating it).

2. Call what you describe something like "The SciPy Package Manual"
that clearly distinguishes it as just being about the package SciPy
and doesn't say "User Guide".

3. Close the SciPy docstring wiki, at least to non-maintainers.

4. Guide anyone who is not a maintainer of SciPy code to contribute
effort to the NumPy docstrings until they are done, then the SciPy
docstrings.

5. Encourage ALL developers and maintainers to write the docstrings
for their packages (in SVN for SciPy), before they do more than fix
bugs in the existing code.  Let's at least get the full function
signatures for ufuncs and the parameter and keyword descriptions done.
Others can fill in the notes and examples based on that.  If they then
want to go write a manual for their packages, great.

Of course, people get to make their own decisions about how to
volunteer their time.

--jh--

Date: Sat, 6 Dec 2008 11:31:39 +0100
From: Gael Varoquaux <gael.varoquaux at normalesup.org>
To: jh at physics.ucf.edu
Cc: scipy-dev at scipy.org
Subject: Re:  [SciPy-dev] Scipy Tutorial (and updating it)
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
In-Reply-To: <wl8fxl2qyqh.fsf at glup.physics.ucf.edu>

On Fri, Dec 05, 2008 at 01:10:46PM -0500, jh at physics.ucf.edu wrote:
> Gael, what you describe below is just a collection of documents about
> each of scipy's modules, written by the maintainers of those modules.
> That is valuable to have, but it is not a user guide.  If you look at
> the User Guide for any of our competing environments (IDL, Matlab,
> Mathematica, etc.), or indeed for any large software package, you'll
> find a document that is integrated across the areas of functionality.

Correct, and I agree that we need this. But this is not the SciPy user
guide. SciPy is only one of the packages. Unlike the other environments,
we have a separation of functionnality. From a software engineering point
of view, this is a good thing (eg it makes shipping pure numerical
algorithms easier, or it makes it possible to run them on embedded
systems). I agree that for a large classe of users, this separation makes
no sens. Thus I agree that an overall document is crucial. But the SciPy
User Guide isn't that.

Ga?l



From aisaac at american.edu  Sat Dec  6 11:54:16 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Sat, 06 Dec 2008 11:54:16 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <493AA4ED.3020800@american.edu>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>	<49370152.3010905@american.edu>	<c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>	<e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>
	<493AA4ED.3020800@american.edu>
Message-ID: <493AAE38.60608@american.edu>

On 12/6/2008 11:14 AM Alan G Isaac apparently wrote:
> I am happy to change the brentq documentation similarly.
> (http://docs.scipy.org/scipy/docs/scipy.optimize.zeros.brentq/)

Well anyway, I did this.
Let me know if other action is desired.
Alan



From jh at physics.ucf.edu  Sat Dec  6 12:01:27 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Sat, 06 Dec 2008 12:01:27 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <mailman.2698.1228580085.2878.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.2698.1228580085.2878.scipy-dev@scipy.org>
Message-ID: <wl8abb9qlug.fsf@glup.physics.ucf.edu>

> 2008/12/5 Pauli Virtanen <pav at iki.fi>:

> I do not believe that preventing people from editing scipy docstrings
> will improve numpy documentation in any way; it will only prevent point
> fixes to Scipy docstrings (eg. people becoming annoyed with a specific
> bad docstring, and fixing it). We can say instead that "concentrate on
> Numpy", strongly, on the front page, and remove scipy links from the
> table on the front page.

This seems like a good compromise.  If we see a lot of migration to
systematic scipy writing (as opposed to point fixes), we can revisit.
I do want the developers to get in there and write in the areas where
they are expert, and to see point fixes made.

> What in my opinion could help Numpy documentation now is coordination.
> For example via milestones:
>
> - Compile a list of functions; sort it appropriately, and parcel it out
>  to manageable chunks (say, tens of items).
>
> - One possible way to parcel these could be to work in the order the
>  functions currently appear in the current reference guide.
>
> - I'd include a couple of "unimportant" functions to each milestone,
>  so that we don't end up with dull milestones at the end of the run.
>
> - Ask editors to work from top to bottom, one parcel at a time. Ditto for
>  reviewers.
>
> - I wouldn't necessarily include any deadlines, as these will slip,
>  which is demotivating.
>
> This should be fairly easy to compose, just requires putting up a new
> wiki page with appropriate links and sections for this. This could also
> speed up the review and QA that I feel is now not really progressing.

Do we have a coherent-enough group of writers to do this?  If so, I
think it's a fantastic idea.  What we said a few months ago was that
writers looking for something to do should take the Functional List of
Routines and write a section, or part of a section.  That would give
coherence to the text in each area, and encourage cross-referencing.

> "Scott Sinclair" <scott.sinclair.za at gmail.com> writes:

> I think that reviewing of docstrings is going to be tough to get going
> because it relies on the input of people who are already heavily
> involved in the project(s).  Any docstring that someone feels can be
> marked "ready for review" is usually pretty good anyway, we should aim
> to get as many docstrings as possible to this level before getting
> overly concerned about review.

We've already said that our current goal is to get all functions to
the state of "needs review" before pushing hard on the review, for the
reasons you state.  At this point, everything, finished or not, is
included in the numpy releases, as it is an improvement on the past.
Once everything "needs review", we push on the review, and when that's
complete, we only include changes that pass review.

We do have a few contributors who are mainly reviewers, and most of
the reviewed functions were approved by them, I think.

I have come to recognize two forms of review: content and
presentation.  Some functions that are clearly not well written and
others that are incomplete have been marked as reviewed.  My guess is
that a reviewer expert in the other area missed that the presentation
or content of the given item was lacking.  So, when we do get to the
point of review, I think we should consider requiring both kinds of
review.  It will slow us but will improve the product a lot.  Some of
our experts are not great writers and some of our great writers are
not experts.  This is hardly surprising, and both have a lot to
contribute, but we should ensure our liabilities get caught.

--jh--


From jh at physics.ucf.edu  Sat Dec  6 12:28:13 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Sat, 06 Dec 2008 12:28:13 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <mailman.2629.1228499543.2878.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.2629.1228499543.2878.scipy-dev@scipy.org>
Message-ID: <wl88wqtqklu.fsf@glup.physics.ucf.edu>

> Pauli Virtanen <pav at iki.fi> writes:

> > Second, a (good) user manual takes a level of planning and commitment
> > that has not gone into any of our documents so far.

> I don't think it's fair to say that no planning has been done. 

I didn't!

> The 
> content of the Numpy reference guide is based on Travis's work, and I 
> streamlined it by editing out the user-guidish parts.

> Criticism is of course welcome, but it's more helpful to be specific.
> I have seen little to no feedback on this.

I'm not criticizing your work at all, Pauli.  I'm saying that the
document I call User Guide needs to follow a different path from any
we've taken so far.  I outlined this process, used widely in academic
publishing, in a prior post.  For example, we used it in writing this
book:

http://www.amazon.com/Jupiter-Satellites-Magnetosphere-Cambridge-Planetary/dp/0521035457/ref=pd_bbs_sr_3?ie=UTF8&s=books&qid=1228583542&sr=8-3

The organizer held a conference at which chapter topics were
discussed.  She got a publisher (CUP) and then put out an RFP for
additional editors and chapter teams.  Teams proposed to take on
chapters.  She selected two more editors and they selected among the
proposals for each chapter.  I think a few shotgun marriages were
arranged between proposal teams, etc.  We (I led chapter 8, on the
1994 Shoemaker-Levy 9 comet impact) got guidelines and a latex format,
and a set of (teflon-coated) deadlines.  We wrote.  They sent the
chapters out to three reviewers.  We revised, were re-reviewed,
resubmitted.  They sent out lists of consistency issues.  We fixed
them.  A professional indexer went through the whole book and did the
index.  We sent it to Cambridge Univ. Press, which printed it.  I am
quite proud of both my own chapter and the degree of integration we
were able to achieve across the whole volume.  This is now The Book on
Jupiter.

I don't expect we'll start on this volume until at least two years
from now, based on our rate of progress on the docstrings, but I
propose we use the academic process outlined here.  For our case, we
don't need to have a commercial publisher (but could, with appropriate
license), and we can put the chapter drafts online for a public review
in addition to getting expert reviews.

--jh--


From matthew.brett at gmail.com  Sat Dec  6 13:05:31 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 6 Dec 2008 10:05:31 -0800
Subject: [SciPy-dev] Patch for writing Matlab structs with long field
	names
In-Reply-To: <nq54p1ho8r8.fsf@broad.mit.edu>
References: <nq58wqunmg5.fsf@broad.mit.edu>
	<1e2af89e0812051808j4ee24626r26e39e62a3001124@mail.gmail.com>
	<nq54p1ho8r8.fsf@broad.mit.edu>
Message-ID: <1e2af89e0812061005x7f4ccff4la8d96913912de84@mail.gmail.com>

Hi,

> The actual behaviour of the current version of Matlab is to both read
> and write fieldnames of up to 64 characters (63 characters + NUL).  We
> discovered this because we encountered some .mat files in the wild that
> broke our scipy scripts.

I assume that scipy can read these, but not write them - is that correct?

> It is possible that some previous Matlab version can only handle 32
> characters.  Even with the patch, this should not cause problems as long
> as the actual fieldnames your write are no longer than 32 characters.

Yes, but I don't know what the effect will be of writing out 63
character names for versions of matlab that don't accept this.  It may
just generate an invalid .mat file, and that will seem to people like
a bug in scipy - which it would be.

We could try passing the maximum number of characters as an option
somehow, or we can encode this variable with the matlab file format -
now just '4' or '5', but in due course maybe '6', '7.0', '7.1', and so
on.

Best,

Matthew


From alan at ajackson.org  Sat Dec  6 13:12:23 2008
From: alan at ajackson.org (Alan Jackson)
Date: Sat, 6 Dec 2008 12:12:23 -0600
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <wl8bpvpqna1.fsf@glup.physics.ucf.edu>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
	<wl8bpvpqna1.fsf@glup.physics.ucf.edu>
Message-ID: <20081206121223.54568a61@ajackson.org>

On Sat, 06 Dec 2008 11:30:30 -0500
jh at physics.ucf.edu wrote:

> 
> So, I propose the following:
> 
> 1. Reserve the name "The SciPy User Guide" for the document I describe
> (whether or not I am involved in creating it).
> 
> 2. Call what you describe something like "The SciPy Package Manual"
> that clearly distinguishes it as just being about the package SciPy
> and doesn't say "User Guide".
> 

I agree with the names. To me, when I'm trying to decide which pdf to download
first, I expect a "User Guide" to be an overall exposition for new and
intermediate users, discussing the overall architecture and philosophy, filled
with examples, etc. A "Manual" on the other hand is more for digging into the
details of specific functions.

Alan
-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From alan at ajackson.org  Sat Dec  6 13:07:51 2008
From: alan at ajackson.org (Alan Jackson)
Date: Sat, 6 Dec 2008 12:07:51 -0600
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <wl88wqtqklu.fsf@glup.physics.ucf.edu>
References: <mailman.2629.1228499543.2878.scipy-dev@scipy.org>
	<wl88wqtqklu.fsf@glup.physics.ucf.edu>
Message-ID: <20081206120751.15445c07@ajackson.org>

On Sat, 06 Dec 2008 12:28:13 -0500
jh at physics.ucf.edu wrote:

> 
> I don't expect we'll start on this volume until at least two years
> from now, based on our rate of progress on the docstrings, but I
> propose we use the academic process outlined here.  For our case, we
> don't need to have a commercial publisher (but could, with appropriate
> license), and we can put the chapter drafts online for a public review
> in addition to getting expert reviews.
> 
> --jh--

I wonder if O'Reilly would have any interest in helping support such a
project at some level. I've recently been reading the Subversion manual,
which is published online for free (and in paperback) by O'Reilly.
http://svnbook.red-bean.com/

I know that O'Reilly has a long history of supporting open-source. Perhaps
a little professional editing could be contributed... There is also (at 
least in my mind) a huge degree of legitimacy a project gains when I see
an O'Reilly book. If there is an O'Reilly book it must be real. 8-)

BTW - the forward to the subversion book has some really good ideas in it about
how to write such a manual - in terms of mining user questions for a FAQ sort
of approach on top of the usual exposition.

-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From millman at berkeley.edu  Sun Dec  7 05:08:58 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 7 Dec 2008 02:08:58 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<49370152.3010905@american.edu>
	<c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>
	<e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>
Message-ID: <c7009a550812070208n58b774bh48ae7ebeea1dad80@mail.gmail.com>

On Sat, Dec 6, 2008 at 12:19 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> I wrote the routines. They are similar too, but not the same as the NR
> versions.

Hey Chuck,

That is good to know.  It is fine if the code is similar as long as it
isn't derived from their code.  I assume that the similarity is do to
the fact that the two codes are using similar algorithms.  Could you
confirm that this is indeed the case?

Also could you review the docstring edits that Alan made:
  http://docs.scipy.org/scipy/docs/scipy.optimize.zeros.ridder/
  http://docs.scipy.org/scipy/docs/scipy.optimize.zeros.brentq/

Finally, Alan wasn't able to document the  parameter `disp` for
ridder.  Could you please provide Alan with an explanation for `disp`?

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Sun Dec  7 06:51:59 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 7 Dec 2008 03:51:59 -0800
Subject: [SciPy-dev] #693 (Evaluating bivariate piecewise splines at
	arbitrary points)
In-Reply-To: <gh6ctg$3bd$1@ger.gmane.org>
References: <ggn14f$ibr$3@ger.gmane.org>
	<9457e7c80812030830q46a29babh61b4b7f2de827e44@mail.gmail.com>
	<gh6ctg$3bd$1@ger.gmane.org>
Message-ID: <c7009a550812070351r79e98ac2s1c1f9faa2499ef5f@mail.gmail.com>

On Wed, Dec 3, 2008 at 8:42 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Wed, 03 Dec 2008 18:30:12 +0200, St?fan van der Walt wrote:
>> 2008/11/27 Pauli Virtanen <pav at iki.fi>:
>>> Do we want to get this:
>>>
>>>        http://www.scipy.org/scipy/scipy/ticket/693
>>>
>>> to Scipy 0.7.0, or does it need improvements etc.?
>>
>> Did this make it into 0.7?  It looks like a sensible change.
>
> It's not committed.

Hello Pauli,

Is there any reason that this hasn't been committed?  It looks good to
me:  it is requested, small, useful, tested, and documented.  Stefan
has looked it over and given it his approval.  If there is a reason to
defer this, please update the ticket.  Otherwise, please commit the
patch and close the ticket.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Sun Dec  7 07:08:24 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 7 Dec 2008 04:08:24 -0800
Subject: [SciPy-dev] help needed with 0.7.0 release notes
In-Reply-To: <6c17e6f50811180959r7506f6d0u7586ac1906d3a1c7@mail.gmail.com>
References: <c7009a550811160005j18b55e4k9b42867877a21906@mail.gmail.com>
	<6c17e6f50811180959r7506f6d0u7586ac1906d3a1c7@mail.gmail.com>
Message-ID: <c7009a550812070408j5d57cce4u812dae4751133d7@mail.gmail.com>

On Tue, Nov 18, 2008 at 9:59 AM, Thouis (Ray) Jones
<thouis at broad.mit.edu> wrote:
> I don't have SVN write access, but here's a patch for the release
> notes adding a short description the matlab io changes (hopefully I've
> formatted it appropriately).

Hello Ray,

I committed your patch:  http://projects.scipy.org/scipy/scipy/changeset/5230

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From aisaac at american.edu  Sun Dec  7 09:11:03 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Sun, 07 Dec 2008 09:11:03 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812070208n58b774bh48ae7ebeea1dad80@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>	
	<49370152.3010905@american.edu>	
	<c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>	
	<e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>
	<c7009a550812070208n58b774bh48ae7ebeea1dad80@mail.gmail.com>
Message-ID: <493BD977.8030503@american.edu>

On 12/7/2008 5:08 AM Jarrod Millman apparently wrote:
> Finally, Alan wasn't able to document the  parameter `disp` for
> ridder.  Could you please provide Alan with an explanation for `disp`?

I would think it would be:

   disp : number
     non-zero to print convergence messages.

But in testing I did not see any effect ...

Alan



From pav at iki.fi  Sun Dec  7 12:40:31 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 7 Dec 2008 17:40:31 +0000 (UTC)
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<49370152.3010905@american.edu>
	<c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>
	<e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>
	<c7009a550812070208n58b774bh48ae7ebeea1dad80@mail.gmail.com>
	<493BD977.8030503@american.edu>
Message-ID: <ghh1qf$4jn$1@ger.gmane.org>

Sun, 07 Dec 2008 09:11:03 -0500, Alan G Isaac wrote:
> On 12/7/2008 5:08 AM Jarrod Millman apparently wrote:
>> Finally, Alan wasn't able to document the  parameter `disp` for ridder.
>>  Could you please provide Alan with an explanation for `disp`?
> 
> I would think it would be:
> 
>    disp : number
>      non-zero to print convergence messages.
> 
> But in testing I did not see any effect ...

Looking at the source code, output is printed only if it fails to 
converge.

-- 
Pauli Virtanen



From pav at iki.fi  Sun Dec  7 12:47:04 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 7 Dec 2008 17:47:04 +0000 (UTC)
Subject: [SciPy-dev] #693 (Evaluating bivariate piecewise splines
	at	arbitrary points)
References: <ggn14f$ibr$3@ger.gmane.org>
	<9457e7c80812030830q46a29babh61b4b7f2de827e44@mail.gmail.com>
	<gh6ctg$3bd$1@ger.gmane.org>
	<c7009a550812070351r79e98ac2s1c1f9faa2499ef5f@mail.gmail.com>
Message-ID: <ghh26o$4jn$2@ger.gmane.org>

Sun, 07 Dec 2008 03:51:59 -0800, Jarrod Millman wrote:

> On Wed, Dec 3, 2008 at 8:42 AM, Pauli Virtanen <pav at iki.fi> wrote:
>> Wed, 03 Dec 2008 18:30:12 +0200, St?fan van der Walt wrote:
>>> 2008/11/27 Pauli Virtanen <pav at iki.fi>:
>>>> Do we want to get this:
>>>>
>>>>        http://www.scipy.org/scipy/scipy/ticket/693
>>>>
>>>> to Scipy 0.7.0, or does it need improvements etc.?
>>>
>>> Did this make it into 0.7?  It looks like a sensible change.
>>
>> It's not committed.
> 
> Is there any reason that this hasn't been committed?  It looks good to
> me:  it is requested, small, useful, tested, and documented.  Stefan has
> looked it over and given it his approval.  If there is a reason to defer
> this, please update the ticket.  Otherwise, please commit the patch and
> close the ticket.

No reason to postpone AFAIK. Committed in r5232.

-- 
Pauli Virtanen



From pav at iki.fi  Sun Dec  7 13:21:55 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 7 Dec 2008 18:21:55 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <mailman.2698.1228580085.2878.scipy-dev@scipy.org>
	<wl8abb9qlug.fsf@glup.physics.ucf.edu>
Message-ID: <ghh483$9q8$1@ger.gmane.org>

Sat, 06 Dec 2008 12:01:27 -0500, jh wrote:
>> 2008/12/5 Pauli Virtanen <pav at iki.fi>:
[clip]
>> This should be fairly easy to compose, just requires putting up a new
>> wiki page with appropriate links and sections for this. This could also
>> speed up the review and QA that I feel is now not really progressing.
> 
> Do we have a coherent-enough group of writers to do this?  If so, I
> think it's a fantastic idea.  What we said a few months ago was that
> writers looking for something to do should take the Functional List of
> Routines and write a section, or part of a section.  That would give
> coherence to the text in each area, and encourage cross-referencing.

I took the categorized function lists from the reference docs, and joined 
the result on a single page:

	http://docs.scipy.org/numpy/Milestones/

It also shows the review status, so it's easy to see at a glance which 
groups of functions need more work.

Cheers,
Pauli



From charlesr.harris at gmail.com  Sun Dec  7 15:44:28 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 7 Dec 2008 13:44:28 -0700
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <ghh1qf$4jn$1@ger.gmane.org>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<49370152.3010905@american.edu>
	<c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>
	<e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>
	<c7009a550812070208n58b774bh48ae7ebeea1dad80@mail.gmail.com>
	<493BD977.8030503@american.edu> <ghh1qf$4jn$1@ger.gmane.org>
Message-ID: <e06186140812071244k5044095taed84e0a838685be@mail.gmail.com>

On Sun, Dec 7, 2008 at 10:40 AM, Pauli Virtanen <pav at iki.fi> wrote:

> Sun, 07 Dec 2008 09:11:03 -0500, Alan G Isaac wrote:
> > On 12/7/2008 5:08 AM Jarrod Millman apparently wrote:
> >> Finally, Alan wasn't able to document the  parameter `disp` for ridder.
> >>  Could you please provide Alan with an explanation for `disp`?
> >
> > I would think it would be:
> >
> >    disp : number
> >      non-zero to print convergence messages.
> >
> > But in testing I did not see any effect ...
>
> Looking at the source code, output is printed only if it fails to
> converge.
>

I'll fix it to raise an error. The question is whether is should go into the
current release...

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081207/1beae6bc/attachment.html>

From millman at berkeley.edu  Sun Dec  7 16:25:45 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 7 Dec 2008 13:25:45 -0800
Subject: [SciPy-dev] #693 (Evaluating bivariate piecewise splines at
	arbitrary points)
In-Reply-To: <ghh26o$4jn$2@ger.gmane.org>
References: <ggn14f$ibr$3@ger.gmane.org>
	<9457e7c80812030830q46a29babh61b4b7f2de827e44@mail.gmail.com>
	<gh6ctg$3bd$1@ger.gmane.org>
	<c7009a550812070351r79e98ac2s1c1f9faa2499ef5f@mail.gmail.com>
	<ghh26o$4jn$2@ger.gmane.org>
Message-ID: <c7009a550812071325k5a7dc4b1wac9648c13fe0ffe0@mail.gmail.com>

On Sun, Dec 7, 2008 at 9:47 AM, Pauli Virtanen <pav at iki.fi> wrote:
> No reason to postpone AFAIK. Committed in r5232.

Thanks.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From scott.sinclair.za at gmail.com  Mon Dec  8 00:19:03 2008
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Mon, 8 Dec 2008 07:19:03 +0200
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <ghh483$9q8$1@ger.gmane.org>
References: <mailman.2698.1228580085.2878.scipy-dev@scipy.org>
	<wl8abb9qlug.fsf@glup.physics.ucf.edu> <ghh483$9q8$1@ger.gmane.org>
Message-ID: <6a17e9ee0812072119k406a445lfcbd21922504ead@mail.gmail.com>

> 2008/12/7 Pauli Virtanen <pav at iki.fi>:
> I took the categorized function lists from the reference docs, and joined
> the result on a single page:
>
>        http://docs.scipy.org/numpy/Milestones/
>
> It also shows the review status, so it's easy to see at a glance which
> groups of functions need more work.

Very useful, I like it.

Cheers,
Scott


From tom.grydeland at gmail.com  Mon Dec  8 03:44:07 2008
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Mon, 8 Dec 2008 09:44:07 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <wl8bpvpqna1.fsf@glup.physics.ucf.edu>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
	<wl8bpvpqna1.fsf@glup.physics.ucf.edu>
Message-ID: <c21c0b2f0812080044n5d3bc951nfa8b9580b1a5c7c8@mail.gmail.com>

On Sat, Dec 6, 2008 at 5:30 PM,  <jh at physics.ucf.edu> wrote:

> OVER THREE THOUSAND FUNCTIONS
> in NumPy and SciPy DO NOT HAVE GOOD DOCSTRINGS

Right, I'll se what I can contribute.  I'm `togr` on the documentation wiki.

> --jh--

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From gael.varoquaux at normalesup.org  Mon Dec  8 03:49:29 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 8 Dec 2008 09:49:29 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <c21c0b2f0812080044n5d3bc951nfa8b9580b1a5c7c8@mail.gmail.com>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
	<wl8bpvpqna1.fsf@glup.physics.ucf.edu>
	<c21c0b2f0812080044n5d3bc951nfa8b9580b1a5c7c8@mail.gmail.com>
Message-ID: <20081208084929.GA32174@phare.normalesup.org>

On Mon, Dec 08, 2008 at 09:44:07AM +0100, Tom Grydeland wrote:
> On Sat, Dec 6, 2008 at 5:30 PM,  <jh at physics.ucf.edu> wrote:

> > OVER THREE THOUSAND FUNCTIONS
> > in NumPy and SciPy DO NOT HAVE GOOD DOCSTRINGS

> Right, I'll se what I can contribute.  I'm `togr` on the documentation wiki.

I gave you editors right.

Thanks a lot for contributing. I want to stress that the quality of the
documentation is really improving, and this is due to everybody's
contribution. It seems that we could achieve here a "wikipedia effect",
where the wisdom of the crowds gets synchronized to produce a document of
higher quality than individuals would have been able to.

This is fantastic. Let's keep the good work up (and I haven't done my
homework this week end, although I'll claim this was because I was busy
documenting Mayavi :)).

Ga?l


From ondrej at certik.cz  Mon Dec  8 11:31:01 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 8 Dec 2008 17:31:01 +0100
Subject: [SciPy-dev] make 0.7.0b1 lintian clean
Message-ID: <85b5c3130812080831j3bc1e78jf3c91c2be34e965f@mail.gmail.com>

Hi,

I am testing the debian packaging of 0.7.0b1 and lintian says:

$ lintian python-scipy_0.7.0b1-1_amd64.changes
W: python-scipy: unusual-interpreter
./usr/share/pyshared/scipy/sparse/linalg/eigen/lobpcg/tests/test_lobpcg.py
#!pytho
W: python-scipy: executable-not-elf-or-script
./usr/share/pyshared/scipy/io/matlab/tests/gen_mat5files.m
W: python-scipy: executable-not-elf-or-script
./usr/share/pyshared/scipy/io/matlab/tests/gen_mat4files.m
W: python-scipy: executable-not-elf-or-script
./usr/share/pyshared/scipy/io/matlab/tests/save_matfile.m

The attached patch fixes this in the scipy trunk. If you agree, please apply.

You can also pull from here:

http://github.com/certik/scipy/tree/ondrej

I am now compiling numpy 1.2.0 necessary to run scipy, I'll report
back. If I have time, I'll try to send svn di commits as well.

Ondrej
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Make-scipy-lintian-clean.patch
Type: text/x-diff
Size: 1307 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081208/769c462b/attachment.patch>

From ljosa at broad.mit.edu  Mon Dec  8 12:09:29 2008
From: ljosa at broad.mit.edu (Vebjorn Ljosa)
Date: Mon, 08 Dec 2008 12:09:29 -0500
Subject: [SciPy-dev] Patch for writing Matlab structs with long field
	names
In-Reply-To: <1e2af89e0812061005x7f4ccff4la8d96913912de84@mail.gmail.com>
	(Matthew Brett's message of "Sat, 6 Dec 2008 10:05:31 -0800")
References: <nq58wqunmg5.fsf@broad.mit.edu>
	<1e2af89e0812051808j4ee24626r26e39e62a3001124@mail.gmail.com>
	<nq54p1ho8r8.fsf@broad.mit.edu>
	<1e2af89e0812061005x7f4ccff4la8d96913912de84@mail.gmail.com>
Message-ID: <nq5myf61tme.fsf@gm3b6-dd5.broad.mit.edu>

"Matthew Brett" <matthew.brett at gmail.com> writes:

>> The actual behaviour of the current version of Matlab is to both read
>> and write fieldnames of up to 64 characters (63 characters + NUL).  We
>> discovered this because we encountered some .mat files in the wild that
>> broke our scipy scripts.
>
> I assume that scipy can read these, but not write them - is that
> correct?

Yes.

> Yes, but I don't know what the effect will be of writing out 63
> character names for versions of matlab that don't accept this.  It may
> just generate an invalid .mat file, and that will seem to people like
> a bug in scipy - which it would be.

You are right, it's probably prudent to refuse to write long field names
by default.

> We could try passing the maximum number of characters as an option
> somehow, 

The attached patch (passed on from Lee) gives savemat a new option,
"long_field_names".  This option is False by default, so scipy will by
default refuse to write structs with field names longer than 31
characters.  If long_field_names is true, up to 63 character will be
allowed (as supported by recent Matlab versions).

An error will be raised if format == '4' and long_field_names is true.

Thanks,
Vebjorn
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scipy-fieldname-length.diff
Type: text/x-patch
Size: 6386 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081208/8578c673/attachment.bin>

From ondrej at certik.cz  Mon Dec  8 12:23:52 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 8 Dec 2008 18:23:52 +0100
Subject: [SciPy-dev] make 0.7.0b1 lintian clean
In-Reply-To: <85b5c3130812080831j3bc1e78jf3c91c2be34e965f@mail.gmail.com>
References: <85b5c3130812080831j3bc1e78jf3c91c2be34e965f@mail.gmail.com>
Message-ID: <85b5c3130812080923m63d70c6fsd815c6350c9cd60a@mail.gmail.com>

On Mon, Dec 8, 2008 at 5:31 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I am testing the debian packaging of 0.7.0b1 and lintian says:
>
> $ lintian python-scipy_0.7.0b1-1_amd64.changes
> W: python-scipy: unusual-interpreter
> ./usr/share/pyshared/scipy/sparse/linalg/eigen/lobpcg/tests/test_lobpcg.py
> #!pytho
> W: python-scipy: executable-not-elf-or-script
> ./usr/share/pyshared/scipy/io/matlab/tests/gen_mat5files.m
> W: python-scipy: executable-not-elf-or-script
> ./usr/share/pyshared/scipy/io/matlab/tests/gen_mat4files.m
> W: python-scipy: executable-not-elf-or-script
> ./usr/share/pyshared/scipy/io/matlab/tests/save_matfile.m
>
> The attached patch fixes this in the scipy trunk. If you agree, please apply.
>
> You can also pull from here:
>
> http://github.com/certik/scipy/tree/ondrej
>
> I am now compiling numpy 1.2.0 necessary to run scipy, I'll report
> back. If I have time, I'll try to send svn di commits as well.

Ok, so all tests in numpy 1.2.1 pass and one test in scipy 0.7.0b1 fail:

======================================================================
FAIL: test_pbdv (test_basic.TestCephes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/scipy/special/tests/test_basic.py",
line 357, in test_pbdv
    assert_equal(cephes.pbdv(1,0),(0.0,0.0))
  File "/usr/lib/python2.4/site-packages/numpy/testing/utils.py", line
176, in assert_equal
    assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k,err_msg), verbose)
  File "/usr/lib/python2.4/site-packages/numpy/testing/utils.py", line
183, in assert_equal
    raise AssertionError(msg)
AssertionError:
Items are not equal:
item=1

 ACTUAL: 1.0
 DESIRED: 0.0

----------------------------------------------------------------------
Ran 2973 tests in 289.632s

FAILED (KNOWNFAIL=2, SKIP=12, failures=1)
Out[2]: <nose.result.TextTestResult run=2973 errors=0 failures=1>






Ondrej


From ondrej at certik.cz  Mon Dec  8 12:43:47 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 8 Dec 2008 18:43:47 +0100
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <1227987645.8264.200.camel@idol>
References: <1227987645.8264.200.camel@idol>
Message-ID: <85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>

Hi Pauli and all!

First let me apologize for taking me so long to reply. I wrote this
code in the first place and I am happy that Pauli has rewritten it. I
agree with the general direction, but I think this change should not
go into 0.7.0, as it changes the interface and it is not well tested
yet. Also, you renamed all the working broyden implementations that I
use as BadBroyden, so I suggest to name them GoodBroyden, more below.

On Sat, Nov 29, 2008 at 8:40 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Hi all,
>
> I spent some time rewriting the scipy.optimize.nonlin module:
>
>    http://github.com/pv/scipy/tree/ticket-791/scipy/optimize/nonlin.py
>
> The following things changed:
>
> - Support tolerance-based stopping conditions (cf. ticket #791)

+1

>
> - Improved handling of input and return values from the functions,
>  so that they are now easier to use.

+1

>
> - Don't use np.matrix at all.

+1

>
> - Jacobian approximations factored into classes; the iteration code is
>  now in only one place, so trust-region handling or other improvements
>  can be added to a single place later on.

+1

>
> - There's now a line search to the direction inverting the Jacobian
>  gave. (But there's no checking that it is an actual descent direction
>  for some merit function.)

+1

I would like this to test this on my codes how it behaves. If you
tested on your codes and as long as one can turn this off (as one
can), let's commit this and then improve.

>
> - Rewrote docstrings.

in general +1, but I don't understand what all the implementations
actually do. I suggest to also port my comments from the module
docstring into the respective classes. I can help you with that, after
we agree on other things below.

>
> The routines should produce the same output as previously. The tests are
> still not very strong, however.
>
>
> But I have now some questions:
>
> * Getting this to 0.7.0; this is a complete rewrite, so is it too late,
>  and is it better to wait for 0.8?

Yes, I think it should go into 0.8.

>
> * Some of the algorithms in there don't appear to work too well, and
>  some appear to be redundant. I'd like to clean up this a bit, leaving
>  only known-good stuff in.

+1

>
> * I'd like to remove `broyden2` as the actual Jacobian approximation in
>  this appears to be the same as in `broyden3`, and there does not
>  appear to be much difference in the work involved in the two.
>
>  Ondrej, since you wrote the original code, do you think there is
>  a reason to keep both?

I think it is, at least for tutorial purposes and also as an easy way
to check that all is fine. Because there may be some numericall
differences, also for a lot of iterations, the memory consumption of
broyden3 is not limited (is it?). Broyden2 just stores the NxN dense
matrix (that can of course be big), but that's it.

>
> * `broyden1_modified` appears to be, in the end if you work out the
>  matrix algebra, updating the inverse Jacobian in a way that
>  corresponds to
>
>      J := J + (y - J s / 2) s^T / ||s||^2
>
>  for the Jacobian (with s = dx, y = df). Apart from the factor
>  1/2, it's Broyden's good method. [1] One can also verify that the
>  updated inverse Jacobian does not satisfy the quasi-Newton condition
>  s = J^{-1} y, and that `broyden1_modified` doesn't generate the same
>  sequence as `broyden1`.
>
>  Hence, I'd like to remove this routine, unless there's some literature
>  that shows that the above works better than Broyden's method; Ondrej,
>  do you agree?

I agree.

>
>  .. [1] http://en.wikipedia.org/wiki/Broyden%27s_method
>         http://en.wikipedia.org/wiki/Sherman%E2%80%93Morrison_formula
>
> * Also, which articles were used as reference for the non-Quasi-Newton
>  algorithms:
>
>  - `broyden_modified`. This appears to be a bit exotic, and it uses
>    several magic constants (`w0`, `wl`) whose meaning is not clear to
>    me.
>
>    A reference would be helpful here, also for the user who has to
>    choose the parameters as appropriate for his/her specific problem.

You can use my master thesis:

http://ondrej.certik.cz/site_media/cookbook/master.pdf

pages 27-31. Everything is explained in there, plus references given
to the original literature.

>
>  - `broyden_generalized`, `anderson`, `anderson2`. These appear to be
>    variants of Anderson mixing, so probably we only want at most
>    one of these. Also, broyden_generalized() == anderson(w0=0), am I
>    correct?

Yes and no, see my master thesis for details and references. There are
some subtle differences. But those methods have never worked for me
well.

>
>    `anderson` and `anderson2` don't appear to function equivalently,
>    and I suspect the update formula in the latter is wrong, since this
>    algorithm can't solve any of the test problems. Do you have a
>    reference for this?

Yes, see my master thesis, pages above, it's the reference [4].

>
>    Is there a rule how `w0` should be chosen for some specific problem?

Yes, page 31 of my master thesis. But broyden2 or 3, or excitingmixing
works much better for my problems.

>
>  - `excitingmixing`. A reference would be useful, to clarify the
>    heuristics used.

master thesis, equation 3.28. It's the mixing used in the code exciting:

http://exciting.sourceforge.net/

I just noticed I forgot to put a reference to that code into my thesis. :(

>
>  - `linearmixing`. I'm not sure that Scipy should advocate this
>    method :)

Please don't remove this. Sometimes it's the method that works the
best in electronic structure calculations, because broyden2 and 3
sometimes fail. Also it's good to check that it converges to the same
thing as broyden (or other) methods.

>
>  Linearmixing and excitingmixing also seem to require something from
>  the objective function, possibly that the eigenvalues of its Jacobian
>  have the opposite sign than `alpha`. For example, neither of them can
>  find a solution for the equation ``I x = 0`` where ``I`` is the
>  identity matrix (test function F2). So, I'm a bit tempted to remove
>  also these routines, as it seems they probably are not too useful for
>  general problems.

Please don't removed those routines, they are essential for electronic
structure calculations, as they are very robust, fast and doesn't
require any memory. It just works for many problems.

>
> * The code in there is still a bit naive implementation of the inexact
>  (Quasi-)Newton method, and one could add eg. add trust-region handling
>  or try to guarantee that the line search direction is always a
>  decrease direction for a merit function. (I don't know if it's
>  possible to do the latter unless one has a matrix representation of
>  the Jacobian approximation.) So, I suspect there are problems for
>  which eg. MINPACK code will find solutions, but for which the
>  nonlin.py code fails.

Yes, maybe they could be improved. nonlin.py is made for cases, where
you cannot afford to store the full dense Jacobian NxN matrix in the
memory.

>
> * One could add more algorithms suitable for large-scale problems; for
>  example some limited-memory Broyden methods (eg. [1])

I think in the references in my thesis, the broyden3 is sometimes
called the limited-memory Broyden method.
However, I checked the equations in [1] in your email and they need to
do a singular-value-decomposition, so it seems there is yet another
approach to this. So yes.

> or Secant-Krylov
>  methods [2].

Indeed, that would be very helpful.

>
>  .. [1] http://www.math.leidenuniv.nl/~verduyn/publications/reports/equadiff.ps
>  .. [2] D. A. Knoll and D. E. Keyes. Jacobian free Newton-Krylov methods.
>         Journal of Computational Physics, 20(2):357?397, 2004.
>
>  I have implementations for both types of algorithms that could
>  possibly go in after some polishing.

Excellent, looking forward! I'll test it on my problems in electronic
structure if that works.


Let me know what you think about my comments. Basically, scipy should
have as many (working and well tested) algorithms as possible, with
the same interface, so that one can change the algorithms and see
which works for the particular problem. I am glad that you are also
using those methods, so that we can work on this together.

Ondrej

From pav at iki.fi  Mon Dec  8 13:11:35 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 8 Dec 2008 18:11:35 +0000 (UTC)
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
Message-ID: <ghjo0n$r9j$1@ger.gmane.org>

Hi Ondrej,

Mon, 08 Dec 2008 18:43:47 +0100, Ondrej Certik wrote:
> First let me apologize for taking me so long to reply. I wrote this code
> in the first place and I am happy that Pauli has rewritten it. I agree
> with the general direction, but I think this change should not go into
> 0.7.0, as it changes the interface and it is not well tested yet.
> Also, you renamed all the working broyden implementations that I use as
> BadBroyden, so I suggest to name them GoodBroyden, more below.

Quick comment (I'll post a more thorough reply later on). The "good" and 
"bad" Broyden's method are names referring to specific ways to update the 
Jacobian (at least these were the names I learned here in a univ. 
course), cf. also [1]; they do not really refer to goodness or badness of 
the methods, and definitely not to quality of implementation. (If you 
meant I had mislabeled one of these, please correct me.)

.. [1] http://en.wikipedia.org/wiki/Broyden%27s_method

Cheers,
Pauli



From ondrej at certik.cz  Mon Dec  8 13:42:29 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 8 Dec 2008 19:42:29 +0100
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <ghjo0n$r9j$1@ger.gmane.org>
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<ghjo0n$r9j$1@ger.gmane.org>
Message-ID: <85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>

On Mon, Dec 8, 2008 at 7:11 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Hi Ondrej,
>
> Mon, 08 Dec 2008 18:43:47 +0100, Ondrej Certik wrote:
>> First let me apologize for taking me so long to reply. I wrote this code
>> in the first place and I am happy that Pauli has rewritten it. I agree
>> with the general direction, but I think this change should not go into
>> 0.7.0, as it changes the interface and it is not well tested yet.
>> Also, you renamed all the working broyden implementations that I use as
>> BadBroyden, so I suggest to name them GoodBroyden, more below.
>
> Quick comment (I'll post a more thorough reply later on). The "good" and
> "bad" Broyden's method are names referring to specific ways to update the
> Jacobian (at least these were the names I learned here in a univ.
> course), cf. also [1]; they do not really refer to goodness or badness of
> the methods, and definitely not to quality of implementation. (If you
> meant I had mislabeled one of these, please correct me.)
>
> .. [1] http://en.wikipedia.org/wiki/Broyden%27s_method

Ah, ok ---  that wiki didn't exist yet when I wrote this. I only knew
first Broyden method and a second Broyden method. Well, still I think
it's weird to call something that works well by BadBrodyen, but if
that's what people are used to, then ok. Do you have some good
reference of this, besides wiki?

And in any case, all of this should be explained in the docstrings.

Ondrej


From robert.kern at gmail.com  Mon Dec  8 14:38:31 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 8 Dec 2008 13:38:31 -0600
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
Message-ID: <3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>

On Mon, Dec 8, 2008 at 12:42, Ondrej Certik <ondrej at certik.cz> wrote:
> On Mon, Dec 8, 2008 at 7:11 PM, Pauli Virtanen <pav at iki.fi> wrote:
>> Hi Ondrej,
>>
>> Mon, 08 Dec 2008 18:43:47 +0100, Ondrej Certik wrote:
>>> First let me apologize for taking me so long to reply. I wrote this code
>>> in the first place and I am happy that Pauli has rewritten it. I agree
>>> with the general direction, but I think this change should not go into
>>> 0.7.0, as it changes the interface and it is not well tested yet.
>>> Also, you renamed all the working broyden implementations that I use as
>>> BadBroyden, so I suggest to name them GoodBroyden, more below.
>>
>> Quick comment (I'll post a more thorough reply later on). The "good" and
>> "bad" Broyden's method are names referring to specific ways to update the
>> Jacobian (at least these were the names I learned here in a univ.
>> course), cf. also [1]; they do not really refer to goodness or badness of
>> the methods, and definitely not to quality of implementation. (If you
>> meant I had mislabeled one of these, please correct me.)
>>
>> .. [1] http://en.wikipedia.org/wiki/Broyden%27s_method
>
> Ah, ok ---  that wiki didn't exist yet when I wrote this. I only knew
> first Broyden method and a second Broyden method. Well, still I think
> it's weird to call something that works well by BadBrodyen, but if
> that's what people are used to, then ok. Do you have some good
> reference of this, besides wiki?
>
> And in any case, all of this should be explained in the docstrings.

If there's an alternate set of names, I would suggest going with
those. "Bad" and "Good" are simply going to confuse people.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From matthew.brett at gmail.com  Mon Dec  8 14:48:12 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 8 Dec 2008 11:48:12 -0800
Subject: [SciPy-dev] Patch for writing Matlab structs with long field
	names
In-Reply-To: <nq5myf61tme.fsf@gm3b6-dd5.broad.mit.edu>
References: <nq58wqunmg5.fsf@broad.mit.edu>
	<1e2af89e0812051808j4ee24626r26e39e62a3001124@mail.gmail.com>
	<nq54p1ho8r8.fsf@broad.mit.edu>
	<1e2af89e0812061005x7f4ccff4la8d96913912de84@mail.gmail.com>
	<nq5myf61tme.fsf@gm3b6-dd5.broad.mit.edu>
Message-ID: <1e2af89e0812081148v687dd5aemb4a1dc175ff7031a@mail.gmail.com>

Hi,

> You are right, it's probably prudent to refuse to write long field names
> by default.
>
>> We could try passing the maximum number of characters as an option
>> somehow,
>
> The attached patch (passed on from Lee) gives savemat a new option,
> "long_field_names".  This option is False by default, so scipy will by
> default refuse to write structs with field names longer than 31
> characters.  If long_field_names is true, up to 63 character will be
> allowed (as supported by recent Matlab versions).

Nice - thank you to you and to Lee, especially for writing the tests,
much appreciated.  I've committed that patch.

Matthew


From ondrej at certik.cz  Mon Dec  8 14:51:03 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 8 Dec 2008 20:51:03 +0100
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
	<3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
Message-ID: <85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>

On Mon, Dec 8, 2008 at 8:38 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Dec 8, 2008 at 12:42, Ondrej Certik <ondrej at certik.cz> wrote:
>> On Mon, Dec 8, 2008 at 7:11 PM, Pauli Virtanen <pav at iki.fi> wrote:
>>> Hi Ondrej,
>>>
>>> Mon, 08 Dec 2008 18:43:47 +0100, Ondrej Certik wrote:
>>>> First let me apologize for taking me so long to reply. I wrote this code
>>>> in the first place and I am happy that Pauli has rewritten it. I agree
>>>> with the general direction, but I think this change should not go into
>>>> 0.7.0, as it changes the interface and it is not well tested yet.
>>>> Also, you renamed all the working broyden implementations that I use as
>>>> BadBroyden, so I suggest to name them GoodBroyden, more below.
>>>
>>> Quick comment (I'll post a more thorough reply later on). The "good" and
>>> "bad" Broyden's method are names referring to specific ways to update the
>>> Jacobian (at least these were the names I learned here in a univ.
>>> course), cf. also [1]; they do not really refer to goodness or badness of
>>> the methods, and definitely not to quality of implementation. (If you
>>> meant I had mislabeled one of these, please correct me.)
>>>
>>> .. [1] http://en.wikipedia.org/wiki/Broyden%27s_method
>>
>> Ah, ok ---  that wiki didn't exist yet when I wrote this. I only knew
>> first Broyden method and a second Broyden method. Well, still I think
>> it's weird to call something that works well by BadBrodyen, but if
>> that's what people are used to, then ok. Do you have some good
>> reference of this, besides wiki?
>>
>> And in any case, all of this should be explained in the docstrings.
>
> If there's an alternate set of names, I would suggest going with
> those. "Bad" and "Good" are simply going to confuse people.

In fact, I would suggest to use or even invent alternate names, rather
than to use Bad and Good, it's really confusing.

Ondrej


From wbaxter at gmail.com  Mon Dec  8 15:01:40 2008
From: wbaxter at gmail.com (Bill Baxter)
Date: Tue, 9 Dec 2008 05:01:40 +0900
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
	<3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
	<85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>
Message-ID: <e86a5fd00812081201p34bc937ava0b5c2a76af7abca@mail.gmail.com>

On Tue, Dec 9, 2008 at 4:51 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Mon, Dec 8, 2008 at 8:38 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Mon, Dec 8, 2008 at 12:42, Ondrej Certik <ondrej at certik.cz> wrote:
>>> On Mon, Dec 8, 2008 at 7:11 PM, Pauli Virtanen <pav at iki.fi> wrote:
>>>> Hi Ondrej,
>>>>
>>>> Mon, 08 Dec 2008 18:43:47 +0100, Ondrej Certik wrote:
>>>>> First let me apologize for taking me so long to reply. I wrote this code
>>>>> in the first place and I am happy that Pauli has rewritten it. I agree
>>>>> with the general direction, but I think this change should not go into
>>>>> 0.7.0, as it changes the interface and it is not well tested yet.
>>>>> Also, you renamed all the working broyden implementations that I use as
>>>>> BadBroyden, so I suggest to name them GoodBroyden, more below.
>>>>
>>>> Quick comment (I'll post a more thorough reply later on). The "good" and
>>>> "bad" Broyden's method are names referring to specific ways to update the
>>>> Jacobian (at least these were the names I learned here in a univ.
>>>> course), cf. also [1]; they do not really refer to goodness or badness of
>>>> the methods, and definitely not to quality of implementation. (If you
>>>> meant I had mislabeled one of these, please correct me.)
>>>>
>>>> .. [1] http://en.wikipedia.org/wiki/Broyden%27s_method
>>>
>>> Ah, ok ---  that wiki didn't exist yet when I wrote this. I only knew
>>> first Broyden method and a second Broyden method. Well, still I think
>>> it's weird to call something that works well by BadBrodyen, but if
>>> that's what people are used to, then ok. Do you have some good
>>> reference of this, besides wiki?
>>>
>>> And in any case, all of this should be explained in the docstrings.
>>
>> If there's an alternate set of names, I would suggest going with
>> those. "Bad" and "Good" are simply going to confuse people.
>
> In fact, I would suggest to use or even invent alternate names, rather
> than to use Bad and Good, it's really confusing.

This page seems to suggest that "bad Broyden" is used to refer to the
inverse update using Sherman Morrison, which is what the Wikipedia
calls "good Broyden".
http://www.math.buffalo.edu/~pitman/courses/mth437/na3/node3.html

There's also a paper called "On The Discovery of the "Good Broyden"
method by Broyden himself which I presume would be the final authority
on which method is which:
http://www.springerlink.com/content/cvvygkeyv8blm96r/
(Needs Springer access to download ... which I don't have.)

--bb


From tom.grydeland at gmail.com  Mon Dec  8 15:29:29 2008
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Mon, 8 Dec 2008 21:29:29 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081208084929.GA32174@phare.normalesup.org>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
	<wl8bpvpqna1.fsf@glup.physics.ucf.edu>
	<c21c0b2f0812080044n5d3bc951nfa8b9580b1a5c7c8@mail.gmail.com>
	<20081208084929.GA32174@phare.normalesup.org>
Message-ID: <c21c0b2f0812081229m62901b29hc5b2d109664dc640@mail.gmail.com>

On Mon, Dec 8, 2008 at 9:49 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> I gave you editors right.

Thank you!

Just to get my hands dirty, I started on the `mgrid` and `ogrid`
objects from numpy.lib.index_tricks (both of which I find very handy,
and wish I had found earlier).  I am replicating text from the
`nd_grid` class here.

The first problem I ran into is that there is no way to tell the
documentation system that I am documenting a class instance, so the
"Parameters" and "Returns" sections are both mandatory, while only the
latter actually makes sense.  I see there is some specal handling of
class documentation, which might work for objects (instances) also,
perhaps with some modifications?

Another problem is that the "See Also" entries are not showing up as
cross-references (or even as a dictionary list).  What did I miss?

> Ga?l

Best regards,

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From millman at berkeley.edu  Mon Dec  8 21:13:14 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 8 Dec 2008 18:13:14 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <e06186140812071244k5044095taed84e0a838685be@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<49370152.3010905@american.edu>
	<c7009a550812052018s2a2fe625q922b77446122005e@mail.gmail.com>
	<e06186140812060019p32226e8en73a1e3fed5e84fc6@mail.gmail.com>
	<c7009a550812070208n58b774bh48ae7ebeea1dad80@mail.gmail.com>
	<493BD977.8030503@american.edu> <ghh1qf$4jn$1@ger.gmane.org>
	<e06186140812071244k5044095taed84e0a838685be@mail.gmail.com>
Message-ID: <c7009a550812081813u2b636327ida85e7ce258d0be0@mail.gmail.com>

On Sun, Dec 7, 2008 at 12:44 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
> I'll fix it to raise an error. The question is whether is should go into the
> current release...

Yes, please update the trunk so that this will make the upcoming
release.  It would be good if you could also add a test.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From scott.sinclair.za at gmail.com  Tue Dec  9 00:48:04 2008
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Tue, 9 Dec 2008 07:48:04 +0200
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <c21c0b2f0812081229m62901b29hc5b2d109664dc640@mail.gmail.com>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
	<wl8bpvpqna1.fsf@glup.physics.ucf.edu>
	<c21c0b2f0812080044n5d3bc951nfa8b9580b1a5c7c8@mail.gmail.com>
	<20081208084929.GA32174@phare.normalesup.org>
	<c21c0b2f0812081229m62901b29hc5b2d109664dc640@mail.gmail.com>
Message-ID: <6a17e9ee0812082148i7118700q4b16c99c980c236@mail.gmail.com>

Hi Tom,

> 2008/12/8 Tom Grydeland <tom.grydeland at gmail.com>:
> Just to get my hands dirty, I started on the `mgrid` and `ogrid`
> objects from numpy.lib.index_tricks (both of which I find very handy,
> and wish I had found earlier).  I am replicating text from the
> `nd_grid` class here.
>
> The first problem I ran into is that there is no way to tell the
> documentation system that I am documenting a class instance, so the
> "Parameters" and "Returns" sections are both mandatory, while only the
> latter actually makes sense.  I see there is some specal handling of
> class documentation, which might work for objects (instances) also,
> perhaps with some modifications?

The doc wiki might need to be changed to understand class
documentation better.  No doubt Pauli can comment on this.  In the
meantime, it is best to use the class documentation guidelines as
closely as you can:

http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines#docstring-standard

Have a look at the section "Documenting classes".

> Another problem is that the "See Also" entries are not showing up as
> cross-references (or even as a dictionary list).  What did I miss?

This looks like a bug, some cross-references aren't resolved. I
suggest using one of the "See Also" formats shown here:

http://projects.scipy.org/scipy/numpy/browser/trunk/doc/example.py#L37

and making sure it is easy to read in plain text.  Once the
cross-referencing issues are resolved, the docstring should just work.

Cheers,
Scott


From pav at iki.fi  Tue Dec  9 04:04:37 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 9 Dec 2008 09:04:37 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
	<wl8bpvpqna1.fsf@glup.physics.ucf.edu>
	<c21c0b2f0812080044n5d3bc951nfa8b9580b1a5c7c8@mail.gmail.com>
	<20081208084929.GA32174@phare.normalesup.org>
	<c21c0b2f0812081229m62901b29hc5b2d109664dc640@mail.gmail.com>
	<6a17e9ee0812082148i7118700q4b16c99c980c236@mail.gmail.com>
Message-ID: <ghlcb5$bh4$1@ger.gmane.org>

Tue, 09 Dec 2008 07:48:04 +0200, Scott Sinclair wrote:
>> 2008/12/8 Tom Grydeland <tom.grydeland at gmail.com>: Just to get my hands
[clip]
>> The first problem I ran into is that there is no way to tell the
>> documentation system that I am documenting a class instance, so the
>> "Parameters" and "Returns" sections are both mandatory, while only the
>> latter actually makes sense.  I see there is some specal handling of
>> class documentation, which might work for objects (instances) also,
>> perhaps with some modifications?
> 
> The doc wiki might need to be changed to understand class documentation
> better.  No doubt Pauli can comment on this.  In the meantime, it is
> best to use the class documentation guidelines as closely as you can:
> 
> http://projects.scipy.org/scipy/numpy/wiki/
CodingStyleGuidelines#docstring-standard
> 
> Have a look at the section "Documenting classes".

Note that `mgrid` is an object and not a class; class documentation is 
already handled. We don't have a standard for objects, so the parser 
assumes the format is the same as for functions.

>> Another problem is that the "See Also" entries are not showing up as
>> cross-references (or even as a dictionary list).  What did I miss?
> 
> This looks like a bug, some cross-references aren't resolved. I suggest
> using one of the "See Also" formats shown here:
> 
> http://projects.scipy.org/scipy/numpy/browser/trunk/doc/example.py#L37
> 
> and making sure it is easy to read in plain text.  Once the
> cross-referencing issues are resolved, the docstring should just work.

The See Also section won't work if the whole docstring is not formatted 
per the standard. Could be fixed, yes.

-- 
Pauli Virtanen



From matthieu.brucher at gmail.com  Tue Dec  9 04:25:45 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 9 Dec 2008 10:25:45 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <1228514330.13880.8.camel@spike.ugent.be>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>
	<1228507208.8840.9.camel@spike.ugent.be>
	<e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>
	<1228511060.11640.10.camel@spike.ugent.be>
	<e76aa17f0812051329p2238c221kfc523a17f8c4d9f3@mail.gmail.com>
	<1228514330.13880.8.camel@spike.ugent.be>
Message-ID: <e76aa17f0812090125m7763bad6lad5aa75e2e6f8f1c@mail.gmail.com>

Hi,

I'm downloading numpy and scipy to try to reproduce what you did. But
I saw something that I didn't expect: you are using fftw for the FFT.
Why don't you use the MKL, which should be faster ?

Matthieu

2008/12/5 Stijn De Weirdt <stijn.deweirdt at ugent.be>:
>> > no. i modified the numpy distutils a little bit so it would accept the
>> > -Bstatic and -Bdynamic switches, for everything else i used standard
>> > numpy buld procedure (i also change intelccompiler.py a bit so numpy
>> > would use icpc as cxx compiler)
>>
>> I think a lot of people would love to what you did ;)
>>
> i paste my site.cfg and a patch here (i have no idea if the mailing list
> accepts attachments ;)
> keep in mind that -Bstatic and -Bdynamic are toggles and this has been
> tested with ictce 3.2.0.017
>
> (warning: ugly ;)
>
> stijn
>
>
> site.cfg (mkl related stuff, you get the idea)
> -----------------------------------------------
> [fftw]
> libraries = -Bstatic,fftw3xc_intel,fftw3xf_intel,-Bdynamic
> [mkl]
> lapack_libs = -Bstatic,mkl_lapack,-Bdynamic
> #mkl_libs = -Bstatic,mkl_intel_lp64, mkl_intel_thread,
> mkl_core,-Bdynamic,iomp5, guide
>
> mkl_libs = -Bstatic,mkl_em64t,-Bdynamic,iomp5, guide
>
> -----------------------------------------------
>
> patch for numpy, against 1.2.1
>
> -----------------------------------------------
> diff -ur --exclude='*pyc' --exclude='*~'
> numpy-1.2.1.orig/numpy/distutils/fcompiler/intel.py
> numpy-1.2.1/numpy/distutils/fcompiler/intel.py
> --- numpy-1.2.1.orig/numpy/distutils/fcompiler/intel.py 2008-10-28
> 12:21:48.000000000 +0100
> +++ numpy-1.2.1/numpy/distutils/fcompiler/intel.py      2008-12-04
> 22:40:19.000000000 +0100
> @@ -152,9 +152,9 @@
>
>     executables = {
>         'version_cmd'  : None,
> -        'compiler_f77' : [None, "-FI", "-w90", "-w95"],
> -        'compiler_fix' : [None, "-FI"],
> -        'compiler_f90' : [None],
> +        'compiler_f77' : [None, "-FI", "-w90", "-w95","-v"],
> +        'compiler_fix' : [None, "-FI","-v"],
> +        'compiler_f90' : [None,"-v"],
>         'linker_so'    : ['<F90>', "-shared"],
>         'archiver'     : ["ar", "-cr"],
>         'ranlib'       : ["ranlib"]
> @@ -166,6 +166,13 @@
>             opt.extend(['-tpp7', '-xW'])
>         return opt
>
> +    def library_option(self, lib):
> +        if lib[0]=='-':
> +            return lib
> +        else:
> +            return "-l" + lib
> +
> +
>  # Is there no difference in the version string between the above
> compilers
>  # and the Visual compilers?
>
> diff -ur --exclude='*pyc' --exclude='*~'
> numpy-1.2.1.orig/numpy/distutils/intelccompiler.py
> numpy-1.2.1/numpy/distutils/intelccompiler.py
> --- numpy-1.2.1.orig/numpy/distutils/intelccompiler.py  2008-10-28
> 12:21:49.000000000 +0100
> +++ numpy-1.2.1/numpy/distutils/intelccompiler.py       2008-12-04
> 23:28:08.000000000 +0100
> @@ -8,16 +8,30 @@
>     """
>
>     compiler_type = 'intel'
> -    cc_exe = 'icc'
> +    cc_exe = 'icc -v'
> +    cxx_exe = 'icpc -v'
>
>     def __init__ (self, verbose=0, dry_run=0, force=0):
>         UnixCCompiler.__init__ (self, verbose,dry_run, force)
>         compiler = self.cc_exe
> -        self.set_executables(compiler=compiler,
> -                             compiler_so=compiler,
> -                             compiler_cxx=compiler,
> -                             linker_exe=compiler,
> -                             linker_so=compiler + ' -shared')
> +        compiler_cxx = self.cxx_exe
> +
> +        opt=' -fPIC -O3 -unroll'
> +        linkopt=' -shared -static-intel '
> +
> +        self.set_executables(compiler=compiler+opt,
> +                             compiler_so=compiler+opt,
> +                             compiler_cxx=compiler_cxx+opt,
> +                             linker_exe=compiler+opt,
> +                             linker_so=compiler+linkopt)
> +
> +    def library_option(self, lib):
> +        if lib[0]=='-':
> +            return lib
> +        else:
> +            return "-l" + lib
> +
> +
>
>  class IntelItaniumCCompiler(IntelCCompiler):
>     compiler_type = 'intele'
> diff -ur --exclude='*pyc' --exclude='*~'
> numpy-1.2.1.orig/numpy/distutils/system_info.py
> numpy-1.2.1/numpy/distutils/system_info.py
> --- numpy-1.2.1.orig/numpy/distutils/system_info.py     2008-10-28
> 12:21:49.000000000 +0100
> +++ numpy-1.2.1/numpy/distutils/system_info.py  2008-12-04
> 01:18:43.000000000 +0100
> @@ -559,6 +559,9 @@
>             lib_prefixes = ['lib']
>         # for each library name, see if we can find a file for it.
>         for l in libs:
> +            if l[0]=='-':
> +                liblist.append(l)
> +                continue
>             for ext in exts:
>                 for prefix in lib_prefixes:
>                     p = self.combine_paths(lib_dir, prefix+l+ext)
> -----------------------------------------------------------------------
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From stijn.deweirdt at ugent.be  Tue Dec  9 04:40:23 2008
From: stijn.deweirdt at ugent.be (Stijn De Weirdt)
Date: Tue, 09 Dec 2008 10:40:23 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <e76aa17f0812090125m7763bad6lad5aa75e2e6f8f1c@mail.gmail.com>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>
	<1228507208.8840.9.camel@spike.ugent.be>
	<e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>
	<1228511060.11640.10.camel@spike.ugent.be>
	<e76aa17f0812051329p2238c221kfc523a17f8c4d9f3@mail.gmail.com>
	<1228514330.13880.8.camel@spike.ugent.be>
	<e76aa17f0812090125m7763bad6lad5aa75e2e6f8f1c@mail.gmail.com>
Message-ID: <1228815623.19004.1.camel@spike.ugent.be>

> hi,

> I saw something that I didn't expect: you are using fftw for the FFT.
> Why don't you use the MKL, which should be faster ?
> 
euhm, i thought this was how one is supossed to use mkl fftw3
interfaces? how would you do that?

stijn

> Matthieu
> 
> 2008/12/5 Stijn De Weirdt <stijn.deweirdt at ugent.be>:
> >> > no. i modified the numpy distutils a little bit so it would accept the
> >> > -Bstatic and -Bdynamic switches, for everything else i used standard
> >> > numpy buld procedure (i also change intelccompiler.py a bit so numpy
> >> > would use icpc as cxx compiler)
> >>
> >> I think a lot of people would love to what you did ;)
> >>
> > i paste my site.cfg and a patch here (i have no idea if the mailing list
> > accepts attachments ;)
> > keep in mind that -Bstatic and -Bdynamic are toggles and this has been
> > tested with ictce 3.2.0.017
> >
> > (warning: ugly ;)
> >
> > stijn
> >
> >
> > site.cfg (mkl related stuff, you get the idea)
> > -----------------------------------------------
> > [fftw]
> > libraries = -Bstatic,fftw3xc_intel,fftw3xf_intel,-Bdynamic
> > [mkl]
> > lapack_libs = -Bstatic,mkl_lapack,-Bdynamic
> > #mkl_libs = -Bstatic,mkl_intel_lp64, mkl_intel_thread,
> > mkl_core,-Bdynamic,iomp5, guide
> >
> > mkl_libs = -Bstatic,mkl_em64t,-Bdynamic,iomp5, guide
> >
> > -----------------------------------------------
> >
> > patch for numpy, against 1.2.1
> >
> > -----------------------------------------------
> > diff -ur --exclude='*pyc' --exclude='*~'
> > numpy-1.2.1.orig/numpy/distutils/fcompiler/intel.py
> > numpy-1.2.1/numpy/distutils/fcompiler/intel.py
> > --- numpy-1.2.1.orig/numpy/distutils/fcompiler/intel.py 2008-10-28
> > 12:21:48.000000000 +0100
> > +++ numpy-1.2.1/numpy/distutils/fcompiler/intel.py      2008-12-04
> > 22:40:19.000000000 +0100
> > @@ -152,9 +152,9 @@
> >
> >     executables = {
> >         'version_cmd'  : None,
> > -        'compiler_f77' : [None, "-FI", "-w90", "-w95"],
> > -        'compiler_fix' : [None, "-FI"],
> > -        'compiler_f90' : [None],
> > +        'compiler_f77' : [None, "-FI", "-w90", "-w95","-v"],
> > +        'compiler_fix' : [None, "-FI","-v"],
> > +        'compiler_f90' : [None,"-v"],
> >         'linker_so'    : ['<F90>', "-shared"],
> >         'archiver'     : ["ar", "-cr"],
> >         'ranlib'       : ["ranlib"]
> > @@ -166,6 +166,13 @@
> >             opt.extend(['-tpp7', '-xW'])
> >         return opt
> >
> > +    def library_option(self, lib):
> > +        if lib[0]=='-':
> > +            return lib
> > +        else:
> > +            return "-l" + lib
> > +
> > +
> >  # Is there no difference in the version string between the above
> > compilers
> >  # and the Visual compilers?
> >
> > diff -ur --exclude='*pyc' --exclude='*~'
> > numpy-1.2.1.orig/numpy/distutils/intelccompiler.py
> > numpy-1.2.1/numpy/distutils/intelccompiler.py
> > --- numpy-1.2.1.orig/numpy/distutils/intelccompiler.py  2008-10-28
> > 12:21:49.000000000 +0100
> > +++ numpy-1.2.1/numpy/distutils/intelccompiler.py       2008-12-04
> > 23:28:08.000000000 +0100
> > @@ -8,16 +8,30 @@
> >     """
> >
> >     compiler_type = 'intel'
> > -    cc_exe = 'icc'
> > +    cc_exe = 'icc -v'
> > +    cxx_exe = 'icpc -v'
> >
> >     def __init__ (self, verbose=0, dry_run=0, force=0):
> >         UnixCCompiler.__init__ (self, verbose,dry_run, force)
> >         compiler = self.cc_exe
> > -        self.set_executables(compiler=compiler,
> > -                             compiler_so=compiler,
> > -                             compiler_cxx=compiler,
> > -                             linker_exe=compiler,
> > -                             linker_so=compiler + ' -shared')
> > +        compiler_cxx = self.cxx_exe
> > +
> > +        opt=' -fPIC -O3 -unroll'
> > +        linkopt=' -shared -static-intel '
> > +
> > +        self.set_executables(compiler=compiler+opt,
> > +                             compiler_so=compiler+opt,
> > +                             compiler_cxx=compiler_cxx+opt,
> > +                             linker_exe=compiler+opt,
> > +                             linker_so=compiler+linkopt)
> > +
> > +    def library_option(self, lib):
> > +        if lib[0]=='-':
> > +            return lib
> > +        else:
> > +            return "-l" + lib
> > +
> > +
> >
> >  class IntelItaniumCCompiler(IntelCCompiler):
> >     compiler_type = 'intele'
> > diff -ur --exclude='*pyc' --exclude='*~'
> > numpy-1.2.1.orig/numpy/distutils/system_info.py
> > numpy-1.2.1/numpy/distutils/system_info.py
> > --- numpy-1.2.1.orig/numpy/distutils/system_info.py     2008-10-28
> > 12:21:49.000000000 +0100
> > +++ numpy-1.2.1/numpy/distutils/system_info.py  2008-12-04
> > 01:18:43.000000000 +0100
> > @@ -559,6 +559,9 @@
> >             lib_prefixes = ['lib']
> >         # for each library name, see if we can find a file for it.
> >         for l in libs:
> > +            if l[0]=='-':
> > +                liblist.append(l)
> > +                continue
> >             for ext in exts:
> >                 for prefix in lib_prefixes:
> >                     p = self.combine_paths(lib_dir, prefix+l+ext)
> > -----------------------------------------------------------------------
> >
> > _______________________________________________
> > Scipy-dev mailing list
> > Scipy-dev at scipy.org
> > http://projects.scipy.org/mailman/listinfo/scipy-dev
> >
> 
> 
> 
-- 
The system will shutdown in 5 minutes.



From matthieu.brucher at gmail.com  Tue Dec  9 04:43:26 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 9 Dec 2008 10:43:26 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <1228815623.19004.1.camel@spike.ugent.be>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>
	<1228507208.8840.9.camel@spike.ugent.be>
	<e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>
	<1228511060.11640.10.camel@spike.ugent.be>
	<e76aa17f0812051329p2238c221kfc523a17f8c4d9f3@mail.gmail.com>
	<1228514330.13880.8.camel@spike.ugent.be>
	<e76aa17f0812090125m7763bad6lad5aa75e2e6f8f1c@mail.gmail.com>
	<1228815623.19004.1.camel@spike.ugent.be>
Message-ID: <e76aa17f0812090143n2e0bff1fr114a45e18c20ec0e@mail.gmail.com>

OK, I may be confused with David's Cournapeau attempt to unify the FFT's ;)

2008/12/9 Stijn De Weirdt <stijn.deweirdt at ugent.be>:
>> hi,
>
>> I saw something that I didn't expect: you are using fftw for the FFT.
>> Why don't you use the MKL, which should be faster ?
>>
> euhm, i thought this was how one is supossed to use mkl fftw3
> interfaces? how would you do that?
>
> stijn
>
>> Matthieu
>>
>> 2008/12/5 Stijn De Weirdt <stijn.deweirdt at ugent.be>:
>> >> > no. i modified the numpy distutils a little bit so it would accept the
>> >> > -Bstatic and -Bdynamic switches, for everything else i used standard
>> >> > numpy buld procedure (i also change intelccompiler.py a bit so numpy
>> >> > would use icpc as cxx compiler)
>> >>
>> >> I think a lot of people would love to what you did ;)
>> >>
>> > i paste my site.cfg and a patch here (i have no idea if the mailing list
>> > accepts attachments ;)
>> > keep in mind that -Bstatic and -Bdynamic are toggles and this has been
>> > tested with ictce 3.2.0.017
>> >
>> > (warning: ugly ;)
>> >
>> > stijn
>> >
>> >
>> > site.cfg (mkl related stuff, you get the idea)
>> > -----------------------------------------------
>> > [fftw]
>> > libraries = -Bstatic,fftw3xc_intel,fftw3xf_intel,-Bdynamic
>> > [mkl]
>> > lapack_libs = -Bstatic,mkl_lapack,-Bdynamic
>> > #mkl_libs = -Bstatic,mkl_intel_lp64, mkl_intel_thread,
>> > mkl_core,-Bdynamic,iomp5, guide
>> >
>> > mkl_libs = -Bstatic,mkl_em64t,-Bdynamic,iomp5, guide
>> >
>> > -----------------------------------------------
>> >
>> > patch for numpy, against 1.2.1
>> >
>> > -----------------------------------------------
>> > diff -ur --exclude='*pyc' --exclude='*~'
>> > numpy-1.2.1.orig/numpy/distutils/fcompiler/intel.py
>> > numpy-1.2.1/numpy/distutils/fcompiler/intel.py
>> > --- numpy-1.2.1.orig/numpy/distutils/fcompiler/intel.py 2008-10-28
>> > 12:21:48.000000000 +0100
>> > +++ numpy-1.2.1/numpy/distutils/fcompiler/intel.py      2008-12-04
>> > 22:40:19.000000000 +0100
>> > @@ -152,9 +152,9 @@
>> >
>> >     executables = {
>> >         'version_cmd'  : None,
>> > -        'compiler_f77' : [None, "-FI", "-w90", "-w95"],
>> > -        'compiler_fix' : [None, "-FI"],
>> > -        'compiler_f90' : [None],
>> > +        'compiler_f77' : [None, "-FI", "-w90", "-w95","-v"],
>> > +        'compiler_fix' : [None, "-FI","-v"],
>> > +        'compiler_f90' : [None,"-v"],
>> >         'linker_so'    : ['<F90>', "-shared"],
>> >         'archiver'     : ["ar", "-cr"],
>> >         'ranlib'       : ["ranlib"]
>> > @@ -166,6 +166,13 @@
>> >             opt.extend(['-tpp7', '-xW'])
>> >         return opt
>> >
>> > +    def library_option(self, lib):
>> > +        if lib[0]=='-':
>> > +            return lib
>> > +        else:
>> > +            return "-l" + lib
>> > +
>> > +
>> >  # Is there no difference in the version string between the above
>> > compilers
>> >  # and the Visual compilers?
>> >
>> > diff -ur --exclude='*pyc' --exclude='*~'
>> > numpy-1.2.1.orig/numpy/distutils/intelccompiler.py
>> > numpy-1.2.1/numpy/distutils/intelccompiler.py
>> > --- numpy-1.2.1.orig/numpy/distutils/intelccompiler.py  2008-10-28
>> > 12:21:49.000000000 +0100
>> > +++ numpy-1.2.1/numpy/distutils/intelccompiler.py       2008-12-04
>> > 23:28:08.000000000 +0100
>> > @@ -8,16 +8,30 @@
>> >     """
>> >
>> >     compiler_type = 'intel'
>> > -    cc_exe = 'icc'
>> > +    cc_exe = 'icc -v'
>> > +    cxx_exe = 'icpc -v'
>> >
>> >     def __init__ (self, verbose=0, dry_run=0, force=0):
>> >         UnixCCompiler.__init__ (self, verbose,dry_run, force)
>> >         compiler = self.cc_exe
>> > -        self.set_executables(compiler=compiler,
>> > -                             compiler_so=compiler,
>> > -                             compiler_cxx=compiler,
>> > -                             linker_exe=compiler,
>> > -                             linker_so=compiler + ' -shared')
>> > +        compiler_cxx = self.cxx_exe
>> > +
>> > +        opt=' -fPIC -O3 -unroll'
>> > +        linkopt=' -shared -static-intel '
>> > +
>> > +        self.set_executables(compiler=compiler+opt,
>> > +                             compiler_so=compiler+opt,
>> > +                             compiler_cxx=compiler_cxx+opt,
>> > +                             linker_exe=compiler+opt,
>> > +                             linker_so=compiler+linkopt)
>> > +
>> > +    def library_option(self, lib):
>> > +        if lib[0]=='-':
>> > +            return lib
>> > +        else:
>> > +            return "-l" + lib
>> > +
>> > +
>> >
>> >  class IntelItaniumCCompiler(IntelCCompiler):
>> >     compiler_type = 'intele'
>> > diff -ur --exclude='*pyc' --exclude='*~'
>> > numpy-1.2.1.orig/numpy/distutils/system_info.py
>> > numpy-1.2.1/numpy/distutils/system_info.py
>> > --- numpy-1.2.1.orig/numpy/distutils/system_info.py     2008-10-28
>> > 12:21:49.000000000 +0100
>> > +++ numpy-1.2.1/numpy/distutils/system_info.py  2008-12-04
>> > 01:18:43.000000000 +0100
>> > @@ -559,6 +559,9 @@
>> >             lib_prefixes = ['lib']
>> >         # for each library name, see if we can find a file for it.
>> >         for l in libs:
>> > +            if l[0]=='-':
>> > +                liblist.append(l)
>> > +                continue
>> >             for ext in exts:
>> >                 for prefix in lib_prefixes:
>> >                     p = self.combine_paths(lib_dir, prefix+l+ext)
>> > -----------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > Scipy-dev mailing list
>> > Scipy-dev at scipy.org
>> > http://projects.scipy.org/mailman/listinfo/scipy-dev
>> >
>>
>>
>>
> --
> The system will shutdown in 5 minutes.
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>



-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From scott.sinclair.za at gmail.com  Tue Dec  9 04:50:37 2008
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Tue, 9 Dec 2008 11:50:37 +0200
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <ghlcb5$bh4$1@ger.gmane.org>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
	<wl8bpvpqna1.fsf@glup.physics.ucf.edu>
	<c21c0b2f0812080044n5d3bc951nfa8b9580b1a5c7c8@mail.gmail.com>
	<20081208084929.GA32174@phare.normalesup.org>
	<c21c0b2f0812081229m62901b29hc5b2d109664dc640@mail.gmail.com>
	<6a17e9ee0812082148i7118700q4b16c99c980c236@mail.gmail.com>
	<ghlcb5$bh4$1@ger.gmane.org>
Message-ID: <6a17e9ee0812090150x5774e223v58d44da23da08ef8@mail.gmail.com>

> 2008/12/9 Pauli Virtanen <pav at iki.fi>:
>> 2008/12/8 Tom Grydeland <tom.grydeland at gmail.com>: Just to get my hands
> [clip]
>> The first problem I ran into is that there is no way to tell the
>> documentation system that I am documenting a class instance, so the
>> "Parameters" and "Returns" sections are both mandatory, while only the
>> latter actually makes sense.  I see there is some specal handling of
>> class documentation, which might work for objects (instances) also,
>> perhaps with some modifications?
>
> Note that `mgrid` is an object and not a class; class documentation is
> already handled. We don't have a standard for objects, so the parser
> assumes the format is the same as for functions.

My mistake, I missed that.

>> Another problem is that the "See Also" entries are not showing up as
>> cross-references (or even as a dictionary list).  What did I miss?
>
> The See Also section won't work if the whole docstring is not formatted
> per the standard. Could be fixed, yes.

It would be nice if it worked. It causes problems in a few other
places e.g. http://docs.scipy.org/numpy/docs/numpy.core.records.recarray/
where the "Other Parameters" section isn't recognized.

Cheers,
Scott


From millman at berkeley.edu  Tue Dec  9 05:11:18 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 9 Dec 2008 02:11:18 -0800
Subject: [SciPy-dev] Please help prepare the 0.7 release notes
Message-ID: <c7009a550812090211p384bb155t2580290c143fd45d@mail.gmail.com>

We are almost ready for 0.7.0rc1 (we just need to sort out the
Numerical Recipes issues and I haven't had time to look though them
yet).  So I wanted to ask once more for help with preparing the
release notes:
http://projects.scipy.org/scipy/scipy/browser/trunk/doc/release/0.7.0-notes.rst

There have been numerous improvements and changes.  As always I would
appreciate any feedback about mistakes or omissions.  It would also be
nice to know how many tests were in the last release and how many are
there now.  Highlighting major bug fixes or pointing out know issues
would be very useful.

I would also like to ask if anyone would be interested in stepping
forward to work on something like Andrew Kuchling's "What's New in
Python ....":  http://docs.python.org/whatsnew/2.6.html

This would be a great area to contribute.  The release notes provide
visibility for our developers' immense contributions of time and
effort.  They help provide an atmosphere of momentum, maturity, and
excitement to a project.  It is also a great service to users who
haven't been following the trunk closely as well as other developer's
who have missed what is happening in other areas of the code.  It is
also becomes a nice historical artifact for the future.

It would be great if someone wanted to contribute in this way.
Ideally, I would like to have someone who be interested in doing this
for several releases of scipy and numpy.  Such a person could develop
a standard template for this and write some scripts to gather specific
statistics (e.g., how many lines of code have changed, how many unit
tests were added, what is the test coverage, what is the docstring
coverage, who were the top contributors, who has increased their code
contributions the most, how many new developers, etc.)

Just a thought.  Figure it won't happen, if I don't ask.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From ondrej at certik.cz  Tue Dec  9 07:49:54 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 9 Dec 2008 13:49:54 +0100
Subject: [SciPy-dev] Please help prepare the 0.7 release notes
In-Reply-To: <c7009a550812090211p384bb155t2580290c143fd45d@mail.gmail.com>
References: <c7009a550812090211p384bb155t2580290c143fd45d@mail.gmail.com>
Message-ID: <85b5c3130812090449i7ba9ab14x149bd98d9a8cd1c1@mail.gmail.com>

On Tue, Dec 9, 2008 at 11:11 AM, Jarrod Millman <millman at berkeley.edu> wrote:
> We are almost ready for 0.7.0rc1 (we just need to sort out the
> Numerical Recipes issues and I haven't had time to look though them
> yet).  So I wanted to ask once more for help with preparing the

If possible, also apply this patch please:

http://projects.scipy.org/pipermail/scipy-dev/2008-December/010540.html


Ondrej


From tom.grydeland at gmail.com  Tue Dec  9 09:07:40 2008
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Tue, 9 Dec 2008 15:07:40 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <ghlcb5$bh4$1@ger.gmane.org>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<20081205060927.GA25416@phare.normalesup.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
	<wl8bpvpqna1.fsf@glup.physics.ucf.edu>
	<c21c0b2f0812080044n5d3bc951nfa8b9580b1a5c7c8@mail.gmail.com>
	<20081208084929.GA32174@phare.normalesup.org>
	<c21c0b2f0812081229m62901b29hc5b2d109664dc640@mail.gmail.com>
	<6a17e9ee0812082148i7118700q4b16c99c980c236@mail.gmail.com>
	<ghlcb5$bh4$1@ger.gmane.org>
Message-ID: <c21c0b2f0812090607v1a9c3001q16c2a8524f1c7e52@mail.gmail.com>

On Tue, Dec 9, 2008 at 10:04 AM, Pauli Virtanen <pav at iki.fi> wrote:

> Note that `mgrid` is an object and not a class; class documentation is
> already handled. We don't have a standard for objects, so the parser
> assumes the format is the same as for functions.

That is what I gathered, yes.

> The See Also section won't work if the whole docstring is not formatted
> per the standard. Could be fixed, yes.

Okay, I won't worry about that for the time being then.

I've started on the numpy.fft.* family of functions now.  I've
finished fft, ifft, fftn, ifftn, and would appreciate an indication
whether it appears to move in the right direction or not.  If people
are happy with the way I wrote these, I'll complete the rest of the
family in the same style.

> Pauli Virtanen

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From josef.pktd at gmail.com  Tue Dec  9 10:18:10 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 9 Dec 2008 10:18:10 -0500
Subject: [SciPy-dev] Please help prepare the 0.7 release notes
In-Reply-To: <85b5c3130812090449i7ba9ab14x149bd98d9a8cd1c1@mail.gmail.com>
References: <c7009a550812090211p384bb155t2580290c143fd45d@mail.gmail.com>
	<85b5c3130812090449i7ba9ab14x149bd98d9a8cd1c1@mail.gmail.com>
Message-ID: <1cd32cbb0812090718u55c828dk302a264d5d489ae5@mail.gmail.com>

I added a section about the changes in scipy stats to the release
notes, see below.

I'm not sure how stable the API for the masked methods in
scipy.stats.mstats are. Also, I mentioned the remaining issues with
distributions and indicated further changes for the next release. I
don't know if you want to have these comments in the release notes or
not.

Please review and make changes if necessary.

Josef

http://projects.scipy.org/scipy/scipy/browser/trunk/doc/release/0.7.0-notes.rst

Bug fixes in the stats package
--------------------------------------------

Statistical functions for masked arrays have been added and are
accessible through scipy.stats.mstats. The functions are similar to
their counterparts in scipy.stats but they have not yet been verified
for identical interfaces and algorithms.

Several bugs were fixed for statistical functions, of those, kstest
and percentileofscore gained new keyword arguments.

Added deprecation warning for mean, median, var, std, cov and
corrcoef. These functions should be replaced by their numpy
counterparts. Note, however, that some of the default options differ
between the scipy.stats and numpy versions of these functions.

Numerous bug fixes to stats.distributions: all generic methods work
now correctly, several methods in individual distributions were
corrected. However, a few issues remain with higher moments (skew,
kurtosis) and entropy. The maximum likelihood estimator, fit, does not
work out-of-the-box for some distributions, in some cases, starting
values have to be carefully chosen, in other cases, the generic
implementation of the maximum likelihood method might not be the
numerically appropriate estimation method.

We expect more bugfixes, increases in numerical precision and
enhancements in the next release of scipy.


From pgmdevlist at gmail.com  Tue Dec  9 11:30:04 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 9 Dec 2008 11:30:04 -0500
Subject: [SciPy-dev] Please help prepare the 0.7 release notes
In-Reply-To: <1cd32cbb0812090718u55c828dk302a264d5d489ae5@mail.gmail.com>
References: <c7009a550812090211p384bb155t2580290c143fd45d@mail.gmail.com>
	<85b5c3130812090449i7ba9ab14x149bd98d9a8cd1c1@mail.gmail.com>
	<1cd32cbb0812090718u55c828dk302a264d5d489ae5@mail.gmail.com>
Message-ID: <6DDF4987-BA30-4F67-A7CF-73295F799F60@gmail.com>


On Dec 9, 2008, at 10:18 AM, josef.pktd at gmail.com wrote:

> I added a section about the changes in scipy stats to the release
> notes, see below.
>
> I'm not sure how stable the API for the masked methods in
> scipy.stats.mstats are.

Quick answer: relatively stable.
Longer answer: I developed those routines because I direly needed them  
at that point. I tried to followed the original (ie, non-masked)  
interface, but never tried to make sure it was completely  
compatiblel.  An example of potential problem is kendall_seasonal: I  
used a dictionary to store the results, it may not be the best (most  
intuitive) kind of results, but keep in mind that without user  
feedback, I could only use what *I* needed. I'd be more than happy to  
switch it to something that would make more sense to the community.
For the algorithms: here again, I followed the basic (non-masked)  
ones. Quite often, the masked adaptation consists in getting rid of  
the masked data before process. In some other cases, the masked data  
are taken into account (for example for ranking). In yet some other  
cases, I was unhappy with the way the functions were actually  
implemented and tried to come up with an alternative, irrespectively  
of the presence of masked data (eg: mquantiles. R offers several  
options to compute experimental quantiles, in hydrology we have the  
need for several estimators, and the original stats version was, let's  
put it diplomatically, a bit to simple for my taste). In all cases, I  
double-triple checked results with stats books I had access to.
So yes, work is needed. I'm kinda swamped for the next couple of  
weeks, unfortunately, but I must also warn you that there'd be no way  
I can do that alone, once again because of the lack of feedback. I  
haven't heard anything about this module since I put it in scipy,  
about 8-9 months ago... So I assumed that there was no real problem  
with it.
Josef, thanks a million for getting into it, don't hesitate to contact  
me offlist if you have specific questions, and we can go back to the  
list for community feedback.





From josef.pktd at gmail.com  Tue Dec  9 13:13:23 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 9 Dec 2008 13:13:23 -0500
Subject: [SciPy-dev] Please help prepare the 0.7 release notes
In-Reply-To: <6DDF4987-BA30-4F67-A7CF-73295F799F60@gmail.com>
References: <c7009a550812090211p384bb155t2580290c143fd45d@mail.gmail.com>
	<85b5c3130812090449i7ba9ab14x149bd98d9a8cd1c1@mail.gmail.com>
	<1cd32cbb0812090718u55c828dk302a264d5d489ae5@mail.gmail.com>
	<6DDF4987-BA30-4F67-A7CF-73295F799F60@gmail.com>
Message-ID: <1cd32cbb0812091013jb137dbbs834631f40e63bb3c@mail.gmail.com>

On Tue, Dec 9, 2008 at 11:30 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> On Dec 9, 2008, at 10:18 AM, josef.pktd at gmail.com wrote:
>
>> I added a section about the changes in scipy stats to the release
>> notes, see below.
>>
>> I'm not sure how stable the API for the masked methods in
>> scipy.stats.mstats are.
>
> Quick answer: relatively stable.
> Longer answer: I developed those routines because I direly needed them
> at that point. I tried to followed the original (ie, non-masked)
> interface, but never tried to make sure it was completely
> compatiblel.  An example of potential problem is kendall_seasonal: I
> used a dictionary to store the results, it may not be the best (most
> intuitive) kind of results, but keep in mind that without user
> feedback, I could only use what *I* needed. I'd be more than happy to
> switch it to something that would make more sense to the community.
> For the algorithms: here again, I followed the basic (non-masked)
> ones. Quite often, the masked adaptation consists in getting rid of
> the masked data before process. In some other cases, the masked data
> are taken into account (for example for ranking). In yet some other
> cases, I was unhappy with the way the functions were actually
> implemented and tried to come up with an alternative, irrespectively
> of the presence of masked data (eg: mquantiles. R offers several
> options to compute experimental quantiles, in hydrology we have the
> need for several estimators, and the original stats version was, let's
> put it diplomatically, a bit to simple for my taste). In all cases, I
> double-triple checked results with stats books I had access to.
> So yes, work is needed. I'm kinda swamped for the next couple of
> weeks, unfortunately, but I must also warn you that there'd be no way
> I can do that alone, once again because of the lack of feedback. I
> haven't heard anything about this module since I put it in scipy,
> about 8-9 months ago... So I assumed that there was no real problem
> with it.
> Josef, thanks a million for getting into it, don't hesitate to contact
> me offlist if you have specific questions, and we can go back to the
> list for community feedback.
>
>
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>

> I
> haven't heard anything about this module since I put it in scipy,
> about 8-9 months ago... So I assumed that there was no real problem
> with it.

I don't know how many users actually knew that it was there since,
without looking at the source, it was not visible, not imported in any
__all__.

My main concern right now was whether we should warn users, for
example in the release notes, about possible coming changes in the
API, if or when we try to make stats and mstats more consistent. Once
mstats is advertised and will get more users, API cleanup will be more
difficult.

I haven't started to compare stats and mstats more closely, but some
improvements should be ported to the other version, e.g. quantiles and
scoreatpercentiles, also I got one test failure in
mstats.friedmanchisquare, that I didn't look into.

I just looked briefly at mstats_basic:
There are also many doc strings that can be ported to stats.stats, you
even implemented an improved version of ks_twosamp. I should have
looked there first, before making my changes. You chose a different
name for the 2 sample KS test mstats.ks_twosamp versus stats.ks_2samp.
For the one-sided versus two-sided option you chose the terminology
from R, alternative : {'two_sided', 'less', 'greater'} , which I
didn't like so much, but since I made the change very recently, I will
adjust the keyword arguments in kstest.

Do we want to keep any such name differences?

My time is also pretty limited right now, but, I think, we should do a
more careful comparison before the next scipy release. But in that
case, some indication, that the API might come under review, would be
appropriate.

Josef


From lists at onerussian.com  Tue Dec  9 13:32:14 2008
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Tue, 9 Dec 2008 13:32:14 -0500
Subject: [SciPy-dev] Warning about remaining issues
	in	stats.distributions ?
In-Reply-To: <1cd32cbb0812010806x6eb5bdcdt684c404e4e5e8027@mail.gmail.com>
References: <1cd32cbb0812010806x6eb5bdcdt684c404e4e5e8027@mail.gmail.com>
Message-ID: <20081209183213.GL25994@washoe.rutgers.edu>

> * distributions that have problems for some range of parameters
so a good (imho) piece to add to unittests for the 'issues' to be fixed:

scipy.stats.rdist(1.32, 0, 1).cdf(-1.0+numpy.finfo(float).eps)

(tried on the SVN trunk to verify that it fails... discover the reason on
your own ;-))

For myself I resolved it with

    __eps = N.sqrt(N.finfo(float).eps)
    rdist = rdist_gen(a=-1.0+__eps, b=1.0-__eps, ....

but I am not sure if that is the cleanest way... and may be some other
distributions would need such tweakery to make them more stable.

-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]




From pgmdevlist at gmail.com  Tue Dec  9 14:30:27 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 9 Dec 2008 14:30:27 -0500
Subject: [SciPy-dev] Please help prepare the 0.7 release notes
In-Reply-To: <1cd32cbb0812091013jb137dbbs834631f40e63bb3c@mail.gmail.com>
References: <c7009a550812090211p384bb155t2580290c143fd45d@mail.gmail.com>
	<85b5c3130812090449i7ba9ab14x149bd98d9a8cd1c1@mail.gmail.com>
	<1cd32cbb0812090718u55c828dk302a264d5d489ae5@mail.gmail.com>
	<6DDF4987-BA30-4F67-A7CF-73295F799F60@gmail.com>
	<1cd32cbb0812091013jb137dbbs834631f40e63bb3c@mail.gmail.com>
Message-ID: <93D81914-87D5-4EAB-8083-186105B50F4C@gmail.com>


On Dec 9, 2008, at 1:13 PM, josef.pktd at gmail.com wrote:
>>
>
> I don't know how many users actually knew that it was there since,
> without looking at the source, it was not visible, not imported in any
> __all__.

Ah, yes. The only advertisment was a post on the mailing list...

>
> My main concern right now was whether we should warn users, for
> example in the release notes, about possible coming changes in the
> API, if or when we try to make stats and mstats more consistent. Once
> mstats is advertised and will get more users, API cleanup will be more
> difficult.
>

We can indeed warn that it's a beta version that will stabilize in the  
next version, just like you did.


> I haven't started to compare stats and mstats more closely, but some
> improvements should be ported to the other version, e.g. quantiles and
> scoreatpercentiles, also I got one test failure in
> mstats.friedmanchisquare, that I didn't look into.





>
> I just looked briefly at mstats_basic:
> There are also many doc strings that can be ported to stats.stats, you
> even implemented an improved version of ks_twosamp. I should have
> looked there first, before making my changes.

I could have remembered and let you know... AFAIR, I checked my  
implementation w/ examples from stats books. Of course, I didn't  
document it at the time and now can't remmbr the details. (Let it be  
an example to younger generations of things not to do...)


> You chose a different
> name for the 2 sample KS test mstats.ks_twosamp versus stats.ks_2samp.

Did you put the alias ks_2samp to ks_twosamp, or did I ? Anyway, both  
should work.

>
> For the one-sided versus two-sided option you chose the terminology
> from R, alternative : {'two_sided', 'less', 'greater'} , which I
> didn't like so much, but since I made the change very recently, I will
> adjust the keyword arguments in kstest.
>
> Do we want to keep any such name differences?

Nope, the 2 methods should indeed have the same inputs. Using R's  
might make things easier for people coming from R...

>
> My time is also pretty limited right now, but, I think, we should do a
> more careful comparison before the next scipy release. But in that
> case, some indication, that the API might come under review, would be
> appropriate.


Sounds good, go for it.




From thouis at broad.mit.edu  Tue Dec  9 15:52:00 2008
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Tue, 9 Dec 2008 15:52:00 -0500
Subject: [SciPy-dev] patch: writing 1x1 cell arrays in matlab
Message-ID: <6c17e6f50812091252x352679aeu28e7a74015d22afb@mail.gmail.com>

Lee Kamnetsky discovered a possible bug in mio5.py.  Diff and test
case attached.

The second test is the one that fails and is addressed by this patch,
and this is the gist of it:

    cells = np.ndarray((1,1),dtype=object)
    cells[0,0]='Hello, world'
    savemat(StringIO(), {'x': cells}, format='5')


Ray Jones
-------------- next part --------------
A non-text attachment was scrubbed...
Name: regression_11cell.py
Type: text/x-python
Size: 583 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081209/8482bc44/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mio5.py.diff
Type: application/octet-stream
Size: 913 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081209/8482bc44/attachment.obj>

From josef.pktd at gmail.com  Tue Dec  9 16:28:09 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 9 Dec 2008 16:28:09 -0500
Subject: [SciPy-dev] Warning about remaining issues in
	stats.distributions ?
In-Reply-To: <20081209183213.GL25994@washoe.rutgers.edu>
References: <1cd32cbb0812010806x6eb5bdcdt684c404e4e5e8027@mail.gmail.com>
	<20081209183213.GL25994@washoe.rutgers.edu>
Message-ID: <1cd32cbb0812091328u533d7a1ex60539ed62952f500@mail.gmail.com>

On Tue, Dec 9, 2008 at 1:32 PM, Yaroslav Halchenko <lists at onerussian.com> wrote:
>> * distributions that have problems for some range of parameters
> so a good (imho) piece to add to unittests for the 'issues' to be fixed:
>
> scipy.stats.rdist(1.32, 0, 1).cdf(-1.0+numpy.finfo(float).eps)
>

In know that there are several problems where the calculations are not
very precise at the boundaries. But I didn't know about this case and
I hope to get rid of the exceptions. So please file tickets for any of
these cases.

In my testing so far, I was quite happy if I got results like these.

stats.rdist(1.32, 0, 1).cdf(-1.0+1e-13)
1.2060822041379395e-009

But I guess there might still be a systematic problem with the
treatment of open parameter spaces or open domains, c>0, x>0, which I
have not tried out at all.

Per Brodtkorb has proposed some improvements in numerical precision,
which we will comitt after 0.7 is out.

For some numerical methods e.g. ppf,  it won't be possible to solve it
arbitrarily close to 0 or 1 for every distribution, given the way the
generic method finds the inverse function, in the test suite, I think,
I used 0.001 and 0.999.

In my fuzz testing there were some cases, where, for example, the
shape parameter has the restriction c>0, but seems to work only for
c>2, but I postponed these bugs for now, since I have to go over each
distribution individually to try to figure out what the problem is.
Additionally, for many distributions I don't know if anyone would ever
need the distribution for that parameter range.

Back to the rdist example
In my bugfixes, I temporarily removed the rdist._cdf, since the
generic method works also for large parameters, while
special.hyp2f1(0.5,1.0-c/2.0,1.5,x*x) does not work for large c. Once
I know over which domain the special functions are reliable,
dispatching to the generic methods only for some part of the parameter
space will be an improvement, but this requires some time consuming
testing.

So any help, reporting and patches are very welcome, especially from
users who actually use the specific distribution.

Josef


From lists at onerussian.com  Tue Dec  9 17:18:45 2008
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Tue, 9 Dec 2008 17:18:45 -0500
Subject: [SciPy-dev] Warning about remaining issues
	in	stats.distributions ?
In-Reply-To: <1cd32cbb0812091328u533d7a1ex60539ed62952f500@mail.gmail.com>
References: <1cd32cbb0812010806x6eb5bdcdt684c404e4e5e8027@mail.gmail.com>
	<20081209183213.GL25994@washoe.rutgers.edu>
	<1cd32cbb0812091328u533d7a1ex60539ed62952f500@mail.gmail.com>
Message-ID: <20081209221844.GM25994@washoe.rutgers.edu>

> In my bugfixes, I temporarily removed the rdist._cdf, since the
> generic method works also for large parameters, while
> special.hyp2f1(0.5,1.0-c/2.0,1.5,x*x) does not work for large c. Once

I can only confirm that, and that is why I had to manually redefine
rdist_gen in the same fashion to be used with elderly scipy 0.5.2-0.1
(with def cdf being removed, and also rdist.veccdf.nin = 2),
otherwise I was getting arbitrary numbers in the output of cdf for some
large values of c (10000) and various values of x in the tails.

> I know over which domain the special functions are reliable,
> dispatching to the generic methods only for some part of the parameter
> space will be an improvement, but this requires some time consuming
> testing.

aren't CPU time is relatively cheap these days? Creating a simple test
battery which would sample the space of arguments with emphasis
on boundaries might prove to be informative to hunt down buried
problems. Alternatively, indeed, it is possible just to wait for someone
to hit the issue on his data / favorite distribution ;-)


-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]




From david at ar.media.kyoto-u.ac.jp  Wed Dec 10 01:02:31 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 10 Dec 2008 15:02:31 +0900
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <e76aa17f0812090143n2e0bff1fr114a45e18c20ec0e@mail.gmail.com>
References: <1228497854.3733.32.camel@spike.ugent.be>	<e76aa17f0812051000q4a663b6fy771e0b1a70d66a6c@mail.gmail.com>	<1228507208.8840.9.camel@spike.ugent.be>	<e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>	<1228511060.11640.10.camel@spike.ugent.be>	<e76aa17f0812051329p2238c221kfc523a17f8c4d9f3@mail.gmail.com>	<1228514330.13880.8.camel@spike.ugent.be>	<e76aa17f0812090125m7763bad6lad5aa75e2e6f8f1c@mail.gmail.com>	<1228815623.19004.1.camel@spike.ugent.be>
	<e76aa17f0812090143n2e0bff1fr114a45e18c20ec0e@mail.gmail.com>
Message-ID: <493F5B77.8060903@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
> OK, I may be confused with David's Cournapeau attempt to unify the FFT's ;)
>   

Well, now, there is no mkl interface anymore (nor fftw for that matter)
in scipy, so this should not be a problem anymore.

David


From matthieu.brucher at gmail.com  Wed Dec 10 02:00:59 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 10 Dec 2008 08:00:59 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <493F5B77.8060903@ar.media.kyoto-u.ac.jp>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<1228507208.8840.9.camel@spike.ugent.be>
	<e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>
	<1228511060.11640.10.camel@spike.ugent.be>
	<e76aa17f0812051329p2238c221kfc523a17f8c4d9f3@mail.gmail.com>
	<1228514330.13880.8.camel@spike.ugent.be>
	<e76aa17f0812090125m7763bad6lad5aa75e2e6f8f1c@mail.gmail.com>
	<1228815623.19004.1.camel@spike.ugent.be>
	<e76aa17f0812090143n2e0bff1fr114a45e18c20ec0e@mail.gmail.com>
	<493F5B77.8060903@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0812092300s31a3c0a5r6a4590d65f5dd5c@mail.gmail.com>

2008/12/10 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> Matthieu Brucher wrote:
>> OK, I may be confused with David's Cournapeau attempt to unify the FFT's ;)
>>
>
> Well, now, there is no mkl interface anymore (nor fftw for that matter)
> in scipy, so this should not be a problem anymore.
>
> David

Of course ;)

Stijn > I didn't manage to compile Numpy with the MKL, even with your
patch, at least for the moment :|

-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From leek at broad.mit.edu  Wed Dec 10 10:05:49 2008
From: leek at broad.mit.edu (Lee Kamentsky)
Date: Wed, 10 Dec 2008 10:05:49 -0500
Subject: [SciPy-dev] Writing Matlab 5 cells containing structures with long
	field names
Message-ID: <493FDACD.6050605@broad.mit.edu>

Hi all,
Unfortunately, the patch that Vebjorn made on my behalf earlier this 
week (checked in as 5237 I believe) had a bug. I failed to change calls 
to instantiate Mat5WriterGetter to pass the long_field_names flag. This 
makes the code fail when a cell contains a structure that has long field 
names because the cell code does not create a Mat5WriterGetter with the 
long_field_names flag set. Enclosed is a diff of scipy/io/matlab/mio5.py 
against revision 5240 that fixes the problem and a diff of 
scipy/io/matlab/tests/test_mio.py that contains regression tests that 
demonstrate the problem. In addition, test_mio.py contains a regression 
test that demonstrates the bug that Ray submitted yesterday on my behalf.

Thanks for your timely action over the last week.

--Lee
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mio5.py.diff
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081210/a3b16d17/attachment.ksh>

From dpeterson at enthought.com  Wed Dec 10 12:28:45 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Wed, 10 Dec 2008 11:28:45 -0600
Subject: [SciPy-dev] ANNOUNCE: ETS 3.1.0 released!
Message-ID: <493FFC4D.60500@enthought.com>

I'm pleased to announce that the Enthought Tool Suite (ETS) 3.1.0 has
been tagged, released, and uploaded to PyPi[1]!

Both source distributions (.tar.gz) and binary (.egg) for Windows have
been built and uploaded to PyPi.

You can update an existing ETS install to v3.1.0 like so:
    easy_install -U ETS==3.1.0


What is ETS?
------------------
The Enthought Tool Suite (ETS) is a collection of projects developed
by members of the OSS community, including Enthought employees, which we
use every day to construct custom scientific applications. It includes a
wide variety of components, including:

    * an extensible application framework
    * application building blocks
    * 2-D and 3-D graphics libraries
    * scientific and math libraries
    * developer tools

The cornerstone on which these tools rest is the Traits project, which
provides explicit type declarations in Python; its features include
initialization, validation, delegation, notification, and visualization
of typed attributes.

More information is available for all these packages from the Enthought
Tool Suite development home page:
    http://code.enthought.com/projects/index.php


-- Dave





From spacey-scipy-dev at lenin.net  Thu Dec 11 01:20:56 2008
From: spacey-scipy-dev at lenin.net (Peter Norton)
Date: Thu, 11 Dec 2008 01:20:56 -0500
Subject: [SciPy-dev] scipy.test() abruptly quits on scipy svn 0.7.0.dev5241
Message-ID: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>

I've had the same result for the last few checkouts. I've been
building with numscons, and as far as I can tell no severe errors are
coming up in the build. However I get this:

....zswap:n=4
..zswap:n=3
....
****************************************************************
WARNING: clapack module is empty
-----------
See scipy/INSTALL.txt for troubleshooting.
Notes:
* If atlas library is not found by numpy/distutils/system_info.py,
  then scipy uses flapack instead of clapack.
****************************************************************

...Result may be inaccurate, approximate err = 1.91669384639e-08
........SSS.................................................................................................................................................................................................................................................................................................................................................................................................................................................ESSSSSSSSSSS.
** On entry to CLASCL parameter number  4 had an illegal value
$ echo $?
0

This looks like it's bailing out, but it doesn't appear to be exiting
with any errors. This was invoked as:

$ python -c 'import scipy,sys; sys.exit(scipy.test())'

The same error occurs whether I build with numscons or distutils
(though numscons is a lot easier).

Is this expected behavior?

Thanks,

-Peter


From david at ar.media.kyoto-u.ac.jp  Thu Dec 11 01:13:34 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 11 Dec 2008 15:13:34 +0900
Subject: [SciPy-dev] scipy.test() abruptly quits on scipy svn
	0.7.0.dev5241
In-Reply-To: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>
References: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>
Message-ID: <4940AF8E.9040605@ar.media.kyoto-u.ac.jp>

Peter Norton wrote:
> I've had the same result for the last few checkouts. I've been
> building with numscons, and as far as I can tell no severe errors are
> coming up in the build. However I get this:
>
> ....zswap:n=4
> ..zswap:n=3
> ....
> ****************************************************************
> WARNING: clapack module is empty
> -----------
> See scipy/INSTALL.txt for troubleshooting.
> Notes:
> * If atlas library is not found by numpy/distutils/system_info.py,
>   then scipy uses flapack instead of clapack.
> ****************************************************************
>
> ...Result may be inaccurate, approximate err = 1.91669384639e-08
> ........SSS.................................................................................................................................................................................................................................................................................................................................................................................................................................................ESSSSSSSSSSS.
> ** On entry to CLASCL parameter number  4 had an illegal value
> $ echo $?
> 0
>
> This looks like it's bailing out, but it doesn't appear to be exiting
> with any errors. This was invoked as:
>   

Yes, this is basically a library fatal error (illegal value means that
the sunperf will call exit I believe, assuming you are using sunperf),
and I don't think we can catch it in numpy/scipy (it is a bug that the
error happens at all, though). IOW, it is a crash.

David


From tom.grydeland at gmail.com  Thu Dec 11 17:28:40 2008
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Thu, 11 Dec 2008 23:28:40 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <c21c0b2f0812090607v1a9c3001q16c2a8524f1c7e52@mail.gmail.com>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8fxl2qyqh.fsf@glup.physics.ucf.edu>
	<20081206103139.GA12839@phare.normalesup.org>
	<wl8bpvpqna1.fsf@glup.physics.ucf.edu>
	<c21c0b2f0812080044n5d3bc951nfa8b9580b1a5c7c8@mail.gmail.com>
	<20081208084929.GA32174@phare.normalesup.org>
	<c21c0b2f0812081229m62901b29hc5b2d109664dc640@mail.gmail.com>
	<6a17e9ee0812082148i7118700q4b16c99c980c236@mail.gmail.com>
	<ghlcb5$bh4$1@ger.gmane.org>
	<c21c0b2f0812090607v1a9c3001q16c2a8524f1c7e52@mail.gmail.com>
Message-ID: <c21c0b2f0812111428o498c5df9y851036448ab1637b@mail.gmail.com>

On Tue, Dec 9, 2008 at 3:07 PM, Tom Grydeland <tom.grydeland at gmail.com> wrote:

> I've started on the numpy.fft.* family of functions now.  I've
> finished fft, ifft, fftn, ifftn, and would appreciate an indication
> whether it appears to move in the right direction or not.  If people
> are happy with the way I wrote these, I'll complete the rest of the
> family in the same style.

Thanks to Per Brodtkorb for suggestions, I think these are ready for review now.

I also did fft2 and ifft2, which became less clear than I had hoped,
as these accept (and handle) instructions for n-dimensional transforms
without complaint.  If I document fft2 as a 2-D transform it becomes
more clear, but less accurate.  Opinions, anyone?

I've gone on to rfft, irfft, fftn and irfftn, and I am struggling a
bit with these, especially with the description of the length/shape
argument.  I know perfectly well what they should tell the reader, but
I am not 100% convinced the current prose actually tells it clearly.


-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From ndbecker2 at gmail.com  Fri Dec 12 12:55:44 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 12 Dec 2008 12:55:44 -0500
Subject: [SciPy-dev] error installing matplotlib-0.98.5
Message-ID: <ghu8j0$b0j$1@ger.gmane.org>


sudo easy_install -U matplotlib
Searching for matplotlib
Reading http://pypi.python.org/simple/matplotlib/
Reading http://matplotlib.sourceforge.net
Reading https://sourceforge.net/project/showfiles.php?group_id=80706&package_id=278194
Reading https://sourceforge.net/project/showfiles.php?group_id=80706&package_id=82474
Reading http://sourceforge.net/project/showfiles.php?group_id=80706
Best match: matplotlib 0.98.5
Downloading http://downloads.sourceforge.net/matplotlib/matplotlib-0.98.5.tar.gz?modtime=1229034572&big_mirror=0
Processing matplotlib-0.98.5.tar.gz
Running matplotlib-0.98.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-CC1jw7/matplotlib-0.98.5/egg-dist-tmp-NSNvcC
============================================================================
BUILDING MATPLOTLIB
            matplotlib: 0.98.5
                python: 2.5.2 (r252:60911, Sep 30 2008, 15:42:03)  [GCC
                        4.3.2 20080917 (Red Hat 4.3.2-4)]
              platform: linux2

REQUIRED DEPENDENCIES
                 numpy: 1.2.0
             freetype2: 9.18.3

OPTIONAL BACKEND DEPENDENCIES
                libpng: 1.2.33
               Tkinter: Tkinter: 50704, Tk: 8.5, Tcl: 8.5
                        * Guessing the library and include directories for
                        * Tcl and Tk because the tclConfig.sh and
                        * tkConfig.sh could not be found and/or parsed.
              wxPython: 2.8.9.1
                        * WxAgg extension not required for wxPython >= 2.8
                  Gtk+: gtk+: 2.14.4, glib: 2.18.3, pygtk: 2.13.0,
                        pygobject: 2.15.4
       Mac OS X native: no
                    Qt: Qt: 3.3.8, PyQt: 3.17.4
                   Qt4: Qt: 4.4.1, PyQt4: 4.4.3
                 Cairo: 1.4.12

OPTIONAL DATE/TIMEZONE DEPENDENCIES
              datetime: present, version unknown
              dateutil: 1.4
                  pytz: 2006p

OPTIONAL USETEX DEPENDENCIES
                dvipng: 1.11
           ghostscript: 8.63
                 latex: 3.141592

EXPERIMENTAL CONFIG PACKAGE DEPENDENCIES
             configobj: 4.5.2
      enthought.traits: no

[Edit setup.cfg to suppress the above messages]
============================================================================
error: lib/matplotlib/mpl-data/matplotlib.conf.template: No such file or directory
Exception exceptions.OSError: (2, 'No such file or directory', 'src/image.cpp') in <bound method CleanUpFile.__del__ of <setupext.CleanUpFile instance at 0x7fc85b3c51b8>> ignored
Exception exceptions.OSError: (2, 'No such file or directory', 'src/path.cpp') in <bound method CleanUpFile.__del__ of <setupext.CleanUpFile instance at 0x7fc85b3c4638>> ignored
Exception exceptions.OSError: (2, 'No such file or directory', 'src/backend_gdk.c') in <bound method CleanUpFile.__del__ of <setupext.CleanUpFile instance at 0x7fc85978c7a0>> ignored
Exception exceptions.OSError: (2, 'No such file or directory', 'src/backend_agg.cpp') in <bound method CleanUpFile.__del__ of <setupext.CleanUpFile instance at 0x7fc85b3c4fc8>> ignored




From ndbecker2 at gmail.com  Fri Dec 12 14:40:15 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 12 Dec 2008 14:40:15 -0500
Subject: [SciPy-dev] error installing matplotlib-0.98.5
References: <ghu8j0$b0j$1@ger.gmane.org>
Message-ID: <ghuemv$ufm$2@ger.gmane.org>

Sorry, wrong list.




From jh at physics.ucf.edu  Fri Dec 12 15:19:05 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Fri, 12 Dec 2008 15:19:05 -0500
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
	it))
In-Reply-To: <mailman.13.1229104806.4123.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.13.1229104806.4123.scipy-dev@scipy.org>
Message-ID: <wl8wse5kuyu.fsf@glup.physics.ucf.edu>

Tom Grydeland <tom.grydeland at gmail.com> wrote:

> I've gone on to rfft, irfft, fftn and irfftn, and I am struggling a
> bit with these, especially with the description of the length/shape
> argument.  I know perfectly well what they should tell the reader, but
> I am not 100% convinced the current prose actually tells it clearly.

Thanks for your work, Tom!  I think the text so far is fine.  I
tweaked some of the pages a little.

There's room and need for a bit more, since FFTs are among the most
used and abused methods out there.  Recall that our audience is one
level below likely users.  There's need for a short intro to the
whole concept on the module page (I added one), which each routine
page should refer to (but I haven't put those references in).

Also, the current routine examples satisfy a mathematician but do not
give insight into what the FT actually *does*, i.e., extract the
periodic content of a signal.  Also, an example should show the most
basic use, i.e., taking the modulus and the arctan of the inputs to
get the amplitude and phase of the contributing sinusoids.  Too many
people take the real part of an FFT and think they've got the
amplitude.  If you did a sine wave of amplitude 2 in 8 points, you
could show in a simple example how to pull that out of the fft.  For
the complex case, point out that you have to fold (or double) the
first half of the output, etc., since the negative frequencies get
half the signal for real inputs.  Then refer to rfft.  Note that many
users will not at first understand why (or even that) we have rfft,
since most packages do not provide it, which is why it is worth some
explanation.

Doing all this consistently across the many routines would be very
valuable.  The rfft vs. fft explanation might best belong on the
module page, since there are several routines for each.  Then the
routines themselves can have relatively spare pages.  The formulae
themselves might best go on the module page, for the same reason.

Thanks again for your contributions.  Send me your postal address and
your T-shirt size by private email and we'll send you one right away!

--jh--


From gael.varoquaux at normalesup.org  Fri Dec 12 17:11:26 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 12 Dec 2008 23:11:26 +0100
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and
	updating	it))
In-Reply-To: <wl8wse5kuyu.fsf@glup.physics.ucf.edu>
References: <mailman.13.1229104806.4123.scipy-dev@scipy.org>
	<wl8wse5kuyu.fsf@glup.physics.ucf.edu>
Message-ID: <20081212221126.GA21207@phare.normalesup.org>

On Fri, Dec 12, 2008 at 03:19:05PM -0500, jh at physics.ucf.edu wrote:
> Also, the current routine examples satisfy a mathematician but do not
> give insight into what the FT actually *does*, i.e., extract the
> periodic content of a signal.

Yes, I was thinking similarly. This is exactly why I brought up the
question about plotting.

Ga?l


From jh at physics.ucf.edu  Sat Dec 13 13:48:51 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Sat, 13 Dec 2008 13:48:51 -0500
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
 it))
In-Reply-To: <mailman.13.1229191205.14185.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.13.1229191205.14185.scipy-dev@scipy.org>
Message-ID: <wl863lnlxm4.fsf@glup.physics.ucf.edu>

On Fri, Dec 12, 2008 at 03:19:05PM -0500, jh at physics.ucf.edu wrote:
> Also, the current routine examples satisfy a mathematician but do not
> give insight into what the FT actually *does*, i.e., extract the
> periodic content of a signal.

Gael responded:
> Yes, I was thinking similarly. This is exactly why I brought up the
> question about plotting.

We avoid plots when possible, of course, but I think that if ever
there were a case for a line plot in numpy, it would be fft.  The
constraints are that we want a plot in the docs but can't have one in
the most basic doc reader, help().  A compromise might be to put it in
the package docstring and not the routine docstrings.  We might also
include the full filename of the plot, so someone working in text
could view the plot separately.

Looking ahead, would it be possible to get Python to use a module's
help() method if it exists?  Then we could replace help() with a
full-functioning reader that defaults to the current help() behavior
but could be configured by a state variable to run a browser for the
PDF or HTML docs.  I assume this would require a PEP and providing
some code.

--jh--


From pav at iki.fi  Sat Dec 13 14:42:21 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 13 Dec 2008 19:42:21 +0000 (UTC)
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
	it))
References: <mailman.13.1229191205.14185.scipy-dev@scipy.org>
	<wl863lnlxm4.fsf@glup.physics.ucf.edu>
Message-ID: <gi136t$7h4$1@ger.gmane.org>

Sat, 13 Dec 2008 13:48:51 -0500, jh wrote:
> On Fri, Dec 12, 2008 at 03:19:05PM -0500, jh at physics.ucf.edu wrote:
[clip]
> Gael responded:
>> Yes, I was thinking similarly. This is exactly why I brought up the
>> question about plotting.
> 
> We avoid plots when possible, of course, but I think that if ever there
> were a case for a line plot in numpy, it would be fft.  The constraints
> are that we want a plot in the docs but can't have one in the most basic
> doc reader, help().  A compromise might be to put it in the package
> docstring and not the routine docstrings.  We might also include the
> full filename of the plot, so someone working in text could view the
> plot separately.

I'd think there are plenty of possibilities for plots in Numpy 
documentation. For example, polyfit, etc. I'd also want to avoid putting 
any file names to the docstrings, as the base directory for these is 
ambiguous, and having a file name in the documentation is not really 
helpful for the reader. Moreover, filenames may easily clash, eg. 
plot-1.jpg, plot-2.jpg, etc.

Instead, we can put in code that produces the plot -- this helps both 
people reading the documentation via help (they can reproduce the plot 
themselves), and those reading the HTML or PDF versions (they can see 
both the code and the output). So, I don't think there's any special need 
to avoid plots. The code supporting plots in the documentation already 
exists (borrowed from matplotlib), we only need to decide how we want to 
use the feature. There's a thread on the numpy-discussion list about this.

> Looking ahead, would it be possible to get Python to use a module's
> help() method if it exists?  Then we could replace help() with a
> full-functioning reader that defaults to the current help() behavior but
> could be configured by a state variable to run a browser for the PDF or
> HTML docs.  I assume this would require a PEP and providing some code.

The help command can be overridden by replacing __builtin__.help, but I 
don't think doing this kind of monkeypatching is the proper thing to do 
on "import numpy". Rather, there could be a separate function that opens 
the correct point in the documentation in the browser. But I'm unsure how 
big convenience this actually is, as it's pretty quick to look up 
documentation for functions and modules using search in the HTML/PDF docs.

-- 
Pauli Virtanen



From pav at iki.fi  Sat Dec 13 16:18:23 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 13 Dec 2008 21:18:23 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
Message-ID: <gi18qv$nt4$1@ger.gmane.org>

Thu, 04 Dec 2008 10:44:13 -0500, josef.pktd wrote:
[clip]
> I haven't tried yet to generate my own numpy, scipy docs, since they are
> makefile based, and I didn't want to spend the time to figure out how to
> use the build scripts on windows, but a windows helpfile is on my
> wish/todo list, since sphinx makes it very easy to do.

Windows help files (CHM) are now available at docs.scipy.org.

-- 
Pauli Virtanen



From gael.varoquaux at normalesup.org  Sat Dec 13 16:21:43 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 13 Dec 2008 22:21:43 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <gi18qv$nt4$1@ger.gmane.org>
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gi18qv$nt4$1@ger.gmane.org>
Message-ID: <20081213212143.GJ19210@phare.normalesup.org>

On Sat, Dec 13, 2008 at 09:18:23PM +0000, Pauli Virtanen wrote:
> Thu, 04 Dec 2008 10:44:13 -0500, josef.pktd wrote:
> [clip]
> > I haven't tried yet to generate my own numpy, scipy docs, since they are
> > makefile based, and I didn't want to spend the time to figure out how to
> > use the build scripts on windows, but a windows helpfile is on my
> > wish/todo list, since sphinx makes it very easy to do.

> Windows help files (CHM) are now available at docs.scipy.org.

Pauli, you are impressive!

Ga?l


From pav at iki.fi  Sat Dec 13 16:26:45 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 13 Dec 2008 21:26:45 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <gh754p$n6q$2@ger.gmane.org>
	<6ce0ac130812031735k1eea6515t6c867999a1feb76c@mail.gmail.com>
	<20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gi18qv$nt4$1@ger.gmane.org>
	<20081213212143.GJ19210@phare.normalesup.org>
Message-ID: <gi19al$rca$1@ger.gmane.org>

Sat, 13 Dec 2008 22:21:43 +0100, Gael Varoquaux wrote:

> On Sat, Dec 13, 2008 at 09:18:23PM +0000, Pauli Virtanen wrote:
>> Thu, 04 Dec 2008 10:44:13 -0500, josef.pktd wrote: [clip]
>> > I haven't tried yet to generate my own numpy, scipy docs, since they
>> > are makefile based, and I didn't want to spend the time to figure out
>> > how to use the build scripts on windows, but a windows helpfile is on
>> > my wish/todo list, since sphinx makes it very easy to do.
> 
>> Windows help files (CHM) are now available at docs.scipy.org.
> 
> Pauli, you are impressive!

Well, Sphinx supports it out of the box, so one just needs to install 
MS's HTML help workshop.

-- 
Pauli Virtanen



From gael.varoquaux at normalesup.org  Sat Dec 13 16:27:43 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 13 Dec 2008 22:27:43 +0100
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <gi19al$rca$1@ger.gmane.org>
References: <20081204102536.5faad9d5.rbastian@free.fr>
	<20081204093944.GB18992@phare.normalesup.org>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gi18qv$nt4$1@ger.gmane.org>
	<20081213212143.GJ19210@phare.normalesup.org>
	<gi19al$rca$1@ger.gmane.org>
Message-ID: <20081213212743.GK19210@phare.normalesup.org>

On Sat, Dec 13, 2008 at 09:26:45PM +0000, Pauli Virtanen wrote:
> Sat, 13 Dec 2008 22:21:43 +0100, Gael Varoquaux wrote:

> > On Sat, Dec 13, 2008 at 09:18:23PM +0000, Pauli Virtanen wrote:
> >> Thu, 04 Dec 2008 10:44:13 -0500, josef.pktd wrote: [clip]
> >> > I haven't tried yet to generate my own numpy, scipy docs, since they
> >> > are makefile based, and I didn't want to spend the time to figure out
> >> > how to use the build scripts on windows, but a windows helpfile is on
> >> > my wish/todo list, since sphinx makes it very easy to do.

> >> Windows help files (CHM) are now available at docs.scipy.org.

> > Pauli, you are impressive!

> Well, Sphinx supports it out of the box, so one just needs to install 
> MS's HTML help workshop.

I know. But you really get things done.

Ga?l


From josef.pktd at gmail.com  Sat Dec 13 18:02:50 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 13 Dec 2008 18:02:50 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <20081213212743.GK19210@phare.normalesup.org>
References: <20081204102536.5faad9d5.rbastian@free.fr>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gi18qv$nt4$1@ger.gmane.org>
	<20081213212143.GJ19210@phare.normalesup.org>
	<gi19al$rca$1@ger.gmane.org>
	<20081213212743.GK19210@phare.normalesup.org>
Message-ID: <1cd32cbb0812131502s9838a46xae4688dae0cf9db5@mail.gmail.com>

On Sat, Dec 13, 2008 at 4:27 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sat, Dec 13, 2008 at 09:26:45PM +0000, Pauli Virtanen wrote:
>> Sat, 13 Dec 2008 22:21:43 +0100, Gael Varoquaux wrote:
>
>> > On Sat, Dec 13, 2008 at 09:18:23PM +0000, Pauli Virtanen wrote:
>> >> Thu, 04 Dec 2008 10:44:13 -0500, josef.pktd wrote: [clip]
>> >> > I haven't tried yet to generate my own numpy, scipy docs, since they
>> >> > are makefile based, and I didn't want to spend the time to figure out
>> >> > how to use the build scripts on windows, but a windows helpfile is on
>> >> > my wish/todo list, since sphinx makes it very easy to do.
>
>> >> Windows help files (CHM) are now available at docs.scipy.org.
>
>> > Pauli, you are impressive!
>
>> Well, Sphinx supports it out of the box, so one just needs to install
>> MS's HTML help workshop.
>
> I know. But you really get things done.
>
> Ga?l
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>

In both numpy  and scipy chm files I get:
The page cannot be displayed
The page you are looking for is currently unavailable. The Web site
might be experiencing technical difficulties, or you may need to
adjust your browser settings.

contents,search, index work correctly, but the page content is not found.

I decompiled and recompiled in with the html help workshop and it
looks pretty good. Search and Index are very fast and helpful.

Thank you from a Windows user.

Josef


From matthew.brett at gmail.com  Sat Dec 13 21:33:35 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 13 Dec 2008 18:33:35 -0800
Subject: [SciPy-dev] Writing Matlab 5 cells containing structures with
	long field names
In-Reply-To: <493FDACD.6050605@broad.mit.edu>
References: <493FDACD.6050605@broad.mit.edu>
Message-ID: <1e2af89e0812131833q4982c6c8k8b771efcb73c13b4@mail.gmail.com>

Hi,

> Unfortunately, the patch that Vebjorn made on my behalf earlier this week
> (checked in as 5237 I believe) had a bug.

Many thanks for this - and again - many thanks for the tests, they are
very helpful.  I took the fact that I also missed the problem with
option passing to mean that it was too complicated, and I've
simplified it, thus implementing your fix in a slightly different way.
 Please let me know if you have any problems with current SVN.

Thanks a lot,

Matthew


From jh at physics.ucf.edu  Sun Dec 14 14:34:06 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Sun, 14 Dec 2008 14:34:06 -0500
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial
	(and	updating	it))
In-Reply-To: <mailman.13.1229277606.7922.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.13.1229277606.7922.scipy-dev@scipy.org>
Message-ID: <wl8skoqk0up.fsf@glup.physics.ucf.edu>

> Pauli says:
> I'd think there are plenty of possibilities for plots in Numpy 
> documentation. For example, polyfit, etc. I'd also want to avoid putting 
> any file names to the docstrings, as the base directory for these is 
> ambiguous, and having a file name in the documentation is not really 
> helpful for the reader. Moreover, filenames may easily clash, eg. 
> plot-1.jpg, plot-2.jpg, etc.

Well, we've had this discussion before, last spring when we discussed
adding the "image" directive.  Some felt strongly that any sort of
image in the help() docs should be avoided since some people may be
reading in a text-only mode or may not have matplotlib installed
(e.g., OLPC users).

We could have decided to write slightly different docs for help() and
for the PDF/HTML version.  Sadly, we do not control help(): whatever
is in the docstring, directives and all, will be seen by the reader.
But, we agreed on the principle of keeping the docstrings as the
ultimate source of the reference docs, because a more removed source
that produced the docstring would be even less likely to be kept
updated by developers than docstrings, and we are very far behind on
even the docstrings.  So, putting a lot of images into our paper docs
means putting them in our help() docs as well, and the more that are
there the less happy text-only readers will be, particularly if the
images/plots are critical to the text.

Because matplotlib might not be available, we need to provide a copy
of the image or plot.  That requires a naming standard, and an obvious
one would be the full name of the object followed by -plot1.png,
-image1.png, etc.  We need to identify a directory in the release to
put these in (I think Stefan might have, already).  Then the help()
text gives the inclusion directive, the pathname for those using plain
help(), and (usually) the commands to make it, set as an example.

I will agree with those who say that help() text will be a little
cluttered.  But, given the constraints, we have the option not to
serve OLPC users and maybe get removed from their release, to avoid
images altogether, or to move to a higher-level source file that
produces a cleaner help() text, and convince the developers to update
that rather than ever touching the docstring.  I prefer living with a
little clutter, or mainly using the PDF or HTML versions, and I think
that was also how others were leaning last spring.

We need some regular input from some people who actually work in the
OLPC environment, or others for whom PDF/HTML don't work for some
reason, or who read only using help() for some reason other than
religion.  I'm defending their interests by applying theory rather
than actually knowing what they care about and can/can't tolerate.

> The help command can be overridden by replacing __builtin__.help, but I 
> don't think doing this kind of monkeypatching is the proper thing to do 
> on "import numpy".

I do not advocate monkeypatching help()!  That would have our help
reading other packages() docstrings.  Rather, I would feed back a
help() that looks for packagename.help() and calls that if it exists,
else just calls the current help().  We *could* monkeypatch that new
help() temporarily, but I'd rather leave that to the user's
discretion.

--jh--




From millman at berkeley.edu  Sun Dec 14 15:04:44 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 14 Dec 2008 12:04:44 -0800
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
	it))
In-Reply-To: <gi136t$7h4$1@ger.gmane.org>
References: <mailman.13.1229191205.14185.scipy-dev@scipy.org>
	<wl863lnlxm4.fsf@glup.physics.ucf.edu> <gi136t$7h4$1@ger.gmane.org>
Message-ID: <c7009a550812141204s51de7a32p38a0a77901b52f02@mail.gmail.com>

On Sat, Dec 13, 2008 at 11:42 AM, Pauli Virtanen <pav at iki.fi> wrote:
> I'd think there are plenty of possibilities for plots in Numpy
> documentation. For example, polyfit, etc. I'd also want to avoid putting
> any file names to the docstrings, as the base directory for these is
> ambiguous, and having a file name in the documentation is not really
> helpful for the reader. Moreover, filenames may easily clash, eg.
> plot-1.jpg, plot-2.jpg, etc.
>
> Instead, we can put in code that produces the plot -- this helps both
> people reading the documentation via help (they can reproduce the plot
> themselves), and those reading the HTML or PDF versions (they can see
> both the code and the output). So, I don't think there's any special need
> to avoid plots. The code supporting plots in the documentation already
> exists (borrowed from matplotlib), we only need to decide how we want to
> use the feature. There's a thread on the numpy-discussion list about this.

+1
I agree.  We should avoid filenames (more trouble than its worth).  I
also like the idea of potentially including plot-generating code as
long as we don't include copies of the plots/image.  Including
image-generating docstrings (but not copies of the images) would
provide several advantages:
  - a mini-tutorial on how to plot the results or generate images
using the NumPy/SciPy functionality
  - allow users to quickly generate their own plots (by providing them
with code) if they are interested in exploring
  - be used to generate figures in out HTML/PDF documentation
We should be judicious in our use of image-generating docstrings,
because we need to avoid cluttering up the docstrings to the point
they aren't helpful anymore.  I would also be happy adopting some
rule-of-thumb that image-generating docstrings have to be no more than
XX number of lines.

> The help command can be overridden by replacing __builtin__.help, but I
> don't think doing this kind of monkeypatching is the proper thing to do
> on "import numpy". Rather, there could be a separate function that opens
> the correct point in the documentation in the browser. But I'm unsure how
> big convenience this actually is, as it's pretty quick to look up
> documentation for functions and modules using search in the HTML/PDF docs.

I agree that we should avoid monkeypatching __builtin__.help.  We
should avoid adding functions to open up a web-browser.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From jdh2358 at gmail.com  Sun Dec 14 16:31:18 2008
From: jdh2358 at gmail.com (John Hunter)
Date: Sun, 14 Dec 2008 22:31:18 +0100
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
	it))
In-Reply-To: <gi136t$7h4$1@ger.gmane.org>
References: <mailman.13.1229191205.14185.scipy-dev@scipy.org>
	<wl863lnlxm4.fsf@glup.physics.ucf.edu> <gi136t$7h4$1@ger.gmane.org>
Message-ID: <88e473830812141331y4afe61b6i118be558d5fe383b@mail.gmail.com>

On Sat, Dec 13, 2008 at 8:42 PM, Pauli Virtanen <pav at iki.fi> wrote:

> The help command can be overridden by replacing __builtin__.help, but I
> don't think doing this kind of monkeypatching is the proper thing to do
> on "import numpy". Rather, there could be a separate function that opens
> the correct point in the documentation in the browser. But I'm unsure how
> big convenience this actually is, as it's pretty quick to look up
> documentation for functions and modules using search in the HTML/PDF docs.

It is possible to write doc strings that look one way in the
interactive help, and another way when building the html/pdf.  For
example, in mpl svn, we have a function which builds property tables
for all our artists, so when you see the help for "fill" for example,
you see the kwargs with nice links

  http://matplotlib.sourceforge.net/api/axes_api.html#matplotlib.axes.Axes.fill

but when you say "help fill" from the python shell all the link markup
is left out.  In the fill docstring, we do
        """
        kwargs control the Polygon properties:

        %(Polygon)s

        **Example:**

        .. plot:: mpl_examples/pylab_examples/fill_demo.py

        """

and interpolate the table into %(Polygon)s .  The table is
autogenerated, and will either generate ReST minus the heay link
markup or ReST plus the heavy markup depending on the rc setting

  docstring.hardcopy : True  # set this when you want to generate
hardcopy docstring

Basically, our docstrings are generated using python string
interpolation at runtime so that they look the best for the
environment they are in.  docstring.hardcopy defaults to false, but we
turn it on locally for our doc builds.

JDH


From alan at ajackson.org  Sun Dec 14 19:24:35 2008
From: alan at ajackson.org (Alan Jackson)
Date: Sun, 14 Dec 2008 18:24:35 -0600
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <6a17e9ee0812060220g7690b865t325b52bfd2f1d4a2@mail.gmail.com>
References: <mailman.2590.1228430728.2878.scipy-dev@scipy.org>
	<wl8skp3qv4n.fsf@glup.physics.ucf.edu> <ghb1d9$loa$2@ger.gmane.org>
	<6a17e9ee0812060220g7690b865t325b52bfd2f1d4a2@mail.gmail.com>
Message-ID: <20081214182435.0321e745@ajackson.org>

On Sat, 6 Dec 2008 12:20:10 +0200
"Scott Sinclair" <scott.sinclair.za at gmail.com> wrote:

> 
> I think that reviewing of docstrings is going to be tough to get going
> because it relies on the input of people who are already heavily
> involved in the project(s).  Any docstring that someone feels can be
> marked "ready for review" is usually pretty good anyway, we should aim
> to get as many docstrings as possible to this level before getting
> overly concerned about review.

I've been slowly working through strings and marking them "ready for review",
sometimes with no edits, sometimes with minor edits, if they look ready and have
been "being written" for over 6 weeks.


-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From alan at ajackson.org  Sun Dec 14 19:28:11 2008
From: alan at ajackson.org (Alan Jackson)
Date: Sun, 14 Dec 2008 18:28:11 -0600
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
 it))
In-Reply-To: <wl863lnlxm4.fsf@glup.physics.ucf.edu>
References: <mailman.13.1229191205.14185.scipy-dev@scipy.org>
	<wl863lnlxm4.fsf@glup.physics.ucf.edu>
Message-ID: <20081214182811.1a33e4f6@ajackson.org>

On Sat, 13 Dec 2008 13:48:51 -0500
jh at physics.ucf.edu wrote:

> On Fri, Dec 12, 2008 at 03:19:05PM -0500, jh at physics.ucf.edu wrote:
> > Also, the current routine examples satisfy a mathematician but do not
> > give insight into what the FT actually *does*, i.e., extract the
> > periodic content of a signal.
> 
> Gael responded:
> > Yes, I was thinking similarly. This is exactly why I brought up the
> > question about plotting.
> 
> We avoid plots when possible, of course, but I think that if ever
> there were a case for a line plot in numpy, it would be fft.  The
> constraints are that we want a plot in the docs but can't have one in
> the most basic doc reader, help().  A compromise might be to put it in
> the package docstring and not the routine docstrings.  We might also
> include the full filename of the plot, so someone working in text
> could view the plot separately.
> 

For many of the fuctions I documented, I put code in the Example that would
produce a (hopefully) enlightening plot.


-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From stefan at sun.ac.za  Mon Dec 15 04:07:54 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 15 Dec 2008 11:07:54 +0200
Subject: [SciPy-dev] Hankel transform
In-Reply-To: <c21c0b2f0812022357p61a32c87j8bd4dde244caf309@mail.gmail.com>
References: <c21c0b2f0812022357p61a32c87j8bd4dde244caf309@mail.gmail.com>
Message-ID: <9457e7c80812150107k5d03f7c1yd120bb501dacbbe1@mail.gmail.com>

Hi all,

Could someone familiar with the Hankel transform please comment on Tom's offer?

Thanks
St?fan

2008/12/3 Tom Grydeland <tom.grydeland at gmail.com>:
> Hi developers,
>
> When I recently needed a Hankel transform I was unable to find one in
> Scipy.  I found one in MATLAB though[1], written by Prof. Brian
> Borchers at New Mexico Tech. The code is based on a previous FORTRAN
> implementation by W. L. Anderson [2], and the MATLAB code is not
> marked with any copyright statements.  Hankel transforms of the first
> and second order can be computed through digital filtering.
>
> I have rewritten the code in Python/numpy, complete with tests and
> acknowledgements of the origins, and my employer has agreed that I can
> donate the code to Scipy.  I believe this should be of interest.
> Hankel transforms arise often in acoustics and other systems with
> cylinder symmetry.  If you want it, I would like a suggestion where it
> belongs (with other integral transforms, probably) and how I should
> shape the tests to make them suitable for inclusion.
>
> The tests I currently have use Hankel transforms of five different
> functions with known analytical transforms and compares the
> transformed values to the numerically evaluated analytical
> expressions.  Currently plots are generated, but for automated testing
> I suppose something else would be better.  Pointing me at an example
> is sufficient.
>
> [1] http://infohost.nmt.edu/~borchers/hankel.html
> [2] Anderson, W. L., 1979, Computer Program Numerical Integration of
> Related Hankel Transforms of Orders 0 and 1 by Adaptive Digital
> Filtering. Geophysics, 44(7):1287-1305.
>
> Best regards,
>
> --
> Tom Grydeland


From gelston at doosanbabcock.com  Mon Dec 15 04:48:46 2008
From: gelston at doosanbabcock.com (Elston, Gareth R)
Date: Mon, 15 Dec 2008 09:48:46 -0000
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <mailman.13.1229277606.7922.scipy-dev@scipy.org>
Message-ID: <9D4464CAAAB788439D66EE2432F9B5F104ECDB9C@00001EXCH.uk.mitsuibabcock.com>

>On Sat, Dec 13, 2008 at 4:27 PM, Gael Varoquaux
><gael.varoquaux at normalesup.org> wrote:
>> On Sat, Dec 13, 2008 at 09:26:45PM +0000, Pauli Virtanen wrote:
>>> Sat, 13 Dec 2008 22:21:43 +0100, Gael Varoquaux wrote:
>>
>>> > On Sat, Dec 13, 2008 at 09:18:23PM +0000, Pauli Virtanen wrote:
>>> >> Thu, 04 Dec 2008 10:44:13 -0500, josef.pktd wrote: [clip]
>>> >> > I haven't tried yet to generate my own numpy, scipy docs, since
they
>>> >> > are makefile based, and I didn't want to spend the time to
figure out
>>> >> > how to use the build scripts on windows, but a windows helpfile
is on
>>> >> > my wish/todo list, since sphinx makes it very easy to do.
>>
>>> >> Windows help files (CHM) are now available at docs.scipy.org.
>>
>>> > Pauli, you are impressive!
>>
>>> Well, Sphinx supports it out of the box, so one just needs to
install
>>> MS's HTML help workshop.
>>
>> I know. But you really get things done.
>>
>> Ga?l
>> _______________________________________________
>> Scipy-dev mailing list
>> Scipy-dev at scipy.org
>> http://projects.scipy.org/mailman/listinfo/scipy-dev
>>
> 
> In both numpy  and scipy chm files I get:
> The page cannot be displayed
> The page you are looking for is currently unavailable. The Web site
> might be experiencing technical difficulties, or you may need to
> adjust your browser settings.
> 
> contents,search, index work correctly, but the page content is not
found.
> 
> I decompiled and recompiled in with the html help workshop and it
> looks pretty good. Search and Index are very fast and helpful.
> 
> Thank you from a Windows user.
> 
> Josef

I am able to see all the help pages in the .chm after clicking on the
"Unblock" button in the "Properties" dialogue box. See also
http://support.microsoft.com/kb/902225/ 

Thanks again from another Windows user.

Gareth.


-------------------------------------------------------------
IMPORTANT NOTICE.

This E-Mail and any files transmitted with it, are  confidential and may be
privileged and are for the exclusive use of the intended recipient(s).
If you are not the intended recipient(s) please note that any form 
of distribution, copying or use of this communication or the
information in it, is strictly prohibited and may be unlawful.

If you have received this E-Mail in error please return it to the sender.
We should be grateful if you would also copy the communication to
postmaster at doosanbabcock.com the delete the E-Mail and destroy any
copies of it.

It is your responsibility to scan any attachments for viruses.

For further information, visit us at WWW.DOOSANBABCOCK.COM


From spacey-scipy-dev at lenin.net  Mon Dec 15 06:43:29 2008
From: spacey-scipy-dev at lenin.net (Peter Norton)
Date: Mon, 15 Dec 2008 06:43:29 -0500
Subject: [SciPy-dev] scipy.test() abruptly quits on scipy svn
	0.7.0.dev5241
In-Reply-To: <4940AF8E.9040605@ar.media.kyoto-u.ac.jp>
References: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>
	<4940AF8E.9040605@ar.media.kyoto-u.ac.jp>
Message-ID: <eeb997dc0812150343w6aa438cbv9139142d94261e8c@mail.gmail.com>

On Thu, Dec 11, 2008 at 1:13 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Peter Norton wrote:
>> I've had the same result for the last few checkouts. I've been
>> building with numscons, and as far as I can tell no severe errors are
>> coming up in the build. However I get this:
>> [...]
>> This looks like it's bailing out, but it doesn't appear to be exiting
>> with any errors. This was invoked as:
>>
>
> Yes, this is basically a library fatal error (illegal value means that
> the sunperf will call exit I believe, assuming you are using sunperf),
> and I don't think we can catch it in numpy/scipy (it is a bug that the
> error happens at all, though). IOW, it is a crash.
>
> David

I was doing this with ATLAS+Lapack, but I realized that I was having
some other difficulties with ATLAS+gcc (it wasn't obvious that gnu as
wasn't supported for ia32 due to "/" being treated as a comment
character on solaris x86. It's only fixed for 64-bit assembler!).

So I've gone back to scipy + numpy being built with sun's CC and using
libsunperf and numscons. And David, kudos to you because in the past
this was quite difficult. The current SVN head built with only one
minor error, that is the dreaded "o" command being invoked instead of
"CC -o". I've written a wrapper, and this is easier because it now
only seems to get invoked for c++ code.

Anyway the same essential crash is still happening:

...Result may be inaccurate, approximate err = 1.78418260944e-08
...Result may be inaccurate, approximate err = 7.27595761418e-12
.....SSS.........................................................................................................................................................................................EEEFFEEE..........................................................................................................................................................................................................................................................ESSSSSSSSSSSB->ncol
= 1  Bstore->lda = 5  A->nrow = 5
B->Stype = 5  B->Dtype = 0  B->Mtype = 0
** On entry to sgssv, parameter number 7 has an illegal value.
Segmentation Fault (core dumped)

The top of the stack looks like this:

#0  0xfe57da76 in ___pl_Destroy_SuperNode_Matrix () from
/usr/local/python-2.5.1/lib/libsunperf.so.3
#1  0xfb8832c4 in Py_sgssv (self=0x0, args=0x9c11c6c, kwdict=0x0) at
build/scons/scipy/sparse/linalg/dsolve/_ssuperlumodule.c:100
#2  0xfecd6bf2 in PyCFunction_Call (func=0x89cae4c, arg=0x9c11c6c,
kw=0x0) at Objects/methodobject.c:77
#3  0xfed20548 in PyEval_EvalFrameEx (f=0x9bf0cec, throwflag=0) at
Python/ceval.c:3564

Is there any more information I can gather to help resolve this crash?

Aside from this, there seem to be many test errors that come up, most
of them in the *fourier tests. I'm not sure if these are expected.

Thanks,

-Peter


From cournape at gmail.com  Mon Dec 15 07:08:04 2008
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 15 Dec 2008 21:08:04 +0900
Subject: [SciPy-dev] scipy.test() abruptly quits on scipy svn
	0.7.0.dev5241
In-Reply-To: <eeb997dc0812150343w6aa438cbv9139142d94261e8c@mail.gmail.com>
References: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>
	<4940AF8E.9040605@ar.media.kyoto-u.ac.jp>
	<eeb997dc0812150343w6aa438cbv9139142d94261e8c@mail.gmail.com>
Message-ID: <5b8d13220812150408y67d31b8ev24c7b8954e34510e@mail.gmail.com>

On Mon, Dec 15, 2008 at 8:43 PM, Peter Norton
<spacey-scipy-dev at lenin.net> wrote:

>
> So I've gone back to scipy + numpy being built with sun's CC and using
> libsunperf and numscons. And David, kudos to you because in the past
> this was quite difficult. The current SVN head built with only one
> minor error, that is the dreaded "o" command being invoked instead of
> "CC -o". I've written a wrapper, and this is easier because it now
> only seems to get invoked for c++ code.

Grr, I thought I fixed this problem for good. This is a really
annoying deficiency of scons (calling non existing tool).

>
> Anyway the same essential crash is still happening:

The fact that is happens with both ATLAS and libsunperf is good news
in a way: it would indicate that there is an problem in the scipy
code, and not with the build :) I think valgrind complains a few times
during the test suite run in the sparse code, so this may be related.
Could you rerun the test suite with test(verbose=10) so that we know
which test crashes ?

David


From spacey-scipy-dev at lenin.net  Mon Dec 15 09:15:45 2008
From: spacey-scipy-dev at lenin.net (Peter Norton)
Date: Mon, 15 Dec 2008 09:15:45 -0500
Subject: [SciPy-dev] scipy.test() abruptly quits on scipy svn
	0.7.0.dev5241
In-Reply-To: <5b8d13220812150408y67d31b8ev24c7b8954e34510e@mail.gmail.com>
References: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>
	<4940AF8E.9040605@ar.media.kyoto-u.ac.jp>
	<eeb997dc0812150343w6aa438cbv9139142d94261e8c@mail.gmail.com>
	<5b8d13220812150408y67d31b8ev24c7b8954e34510e@mail.gmail.com>
Message-ID: <eeb997dc0812150615l76ad0335yf424cc11a46dd9e9@mail.gmail.com>

On Mon, Dec 15, 2008 at 7:08 AM, David Cournapeau <cournape at gmail.com> wrote:
> The fact that is happens with both ATLAS and libsunperf is good news
> in a way: it would indicate that there is an problem in the scipy
> code, and not with the build :) I think valgrind complains a few times
> during the test suite run in the sparse code, so this may be related.
> Could you rerun the test suite with test(verbose=10) so that we know
> which test crashes ?

With verbose=10 nose says:


Prefactorize matrix for solving with multiple rhs ... SKIP: UMFPACK
appears not to be compiled
Solve with UMFPACK: double precision complex ... SKIP: UMFPACK appears
not to be compiled
Solve: single precision complex ... SKIP: UMFPACK appears not to be compiled
Solve with UMFPACK: double precision, sparse rhs ... SKIP: UMFPACK
appears not to be compiled
Solve with UMFPACK: double precision ... SKIP: UMFPACK appears not to
be compiled
Solve: single precision ... SKIP: UMFPACK appears not to be compiled
test_twodiags (test_linsolve.TestLinsolve) ... B->ncol = 1
Bstore->lda = 5  A->nrow = 5
B->Stype = 5  B->Dtype = 0  B->Mtype = 0
** On entry to sgssv, parameter number 7 has an illegal value.
Segmentation Fault (core dumped)

-Peter


From leek at broad.mit.edu  Mon Dec 15 09:20:07 2008
From: leek at broad.mit.edu (Lee Kamentsky)
Date: Mon, 15 Dec 2008 09:20:07 -0500
Subject: [SciPy-dev] Writing Matlab 5 cells containing structures with
 long field names
In-Reply-To: <1e2af89e0812131833q4982c6c8k8b771efcb73c13b4@mail.gmail.com>
References: <493FDACD.6050605@broad.mit.edu>
	<1e2af89e0812131833q4982c6c8k8b771efcb73c13b4@mail.gmail.com>
Message-ID: <49466797.1050606@broad.mit.edu>

Thanks - it works great and your change is much cleaner.
--Lee
Matthew Brett wrote:
> Hi,
>
>   
>> Unfortunately, the patch that Vebjorn made on my behalf earlier this week
>> (checked in as 5237 I believe) had a bug.
>>     
>
> Many thanks for this - and again - many thanks for the tests, they are
> very helpful.  I took the fact that I also missed the problem with
> option passing to mean that it was too complicated, and I've
> simplified it, thus implementing your fix in a slightly different way.
>  Please let me know if you have any problems with current SVN.
>
> Thanks a lot,
>
> Matthew
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>   



From ondrej at certik.cz  Mon Dec 15 10:19:29 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 15 Dec 2008 16:19:29 +0100
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <e86a5fd00812081201p34bc937ava0b5c2a76af7abca@mail.gmail.com>
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
	<3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
	<85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>
	<e86a5fd00812081201p34bc937ava0b5c2a76af7abca@mail.gmail.com>
Message-ID: <85b5c3130812150719l2b3af06credba9a92c15e145e@mail.gmail.com>

> This page seems to suggest that "bad Broyden" is used to refer to the
> inverse update using Sherman Morrison, which is what the Wikipedia
> calls "good Broyden".
> http://www.math.buffalo.edu/~pitman/courses/mth437/na3/node3.html
>
> There's also a paper called "On The Discovery of the "Good Broyden"
> method by Broyden himself which I presume would be the final authority
> on which method is which:
> http://www.springerlink.com/content/cvvygkeyv8blm96r/
> (Needs Springer access to download ... which I don't have.)

Bill, thanks for the tip. I sent you the article offlist.

Pauli --- do you have any comments to the discussion?  After we agree
on a better naming, I'll then try to use your branch at github and
help you polish it so that both my and your codes run well and then we
can push it to scipy. What do you think?

Ondrej


From dmitrey15 at ukr.net  Mon Dec 15 17:44:03 2008
From: dmitrey15 at ukr.net (Dmitrey)
Date: Tue, 16 Dec 2008 00:44:03 +0200
Subject: [SciPy-dev] openopt has to move out from scikits
Message-ID: <4946DDB3.10500@ukr.net>

Hi all,
openopt v. 0.21 has been released. However, old svn location is no 
longer valid (my chiefs have forced me to move my code into other 
location). All details here

http://openopt.blogspot.com/2008/12/openopt-release-021.html

Regards, Dmitrey.



From tom.grydeland at gmail.com  Mon Dec 15 18:13:49 2008
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Tue, 16 Dec 2008 00:13:49 +0100
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
	it))
In-Reply-To: <wl8wse5kuyu.fsf@glup.physics.ucf.edu>
References: <mailman.13.1229104806.4123.scipy-dev@scipy.org>
	<wl8wse5kuyu.fsf@glup.physics.ucf.edu>
Message-ID: <c21c0b2f0812151513mb593c06m3d05fac2a11b53f4@mail.gmail.com>

On Fri, Dec 12, 2008 at 9:19 PM,  <jh at physics.ucf.edu> wrote:

> There's room and need for a bit more, since FFTs are among the most
> used and abused methods out there.  Recall that our audience is one
> level below likely users.  There's need for a short intro to the
> whole concept on the module page (I added one), which each routine
> page should refer to (but I haven't put those references in).
>
> Also, the current routine examples satisfy a mathematician but do not
> give insight into what the FT actually *does*, i.e., extract the
> periodic content of a signal.  Also, an example should show the most
> basic use, i.e., taking the modulus and the arctan of the inputs to
> get the amplitude and phase of the contributing sinusoids.  Too many
> people take the real part of an FFT and think they've got the
> amplitude.  If you did a sine wave of amplitude 2 in 8 points, you
> could show in a simple example how to pull that out of the fft.  For
> the complex case, point out that you have to fold (or double) the
> first half of the output, etc., since the negative frequencies get
> half the signal for real inputs.  Then refer to rfft.  Note that many
> users will not at first understand why (or even that) we have rfft,
> since most packages do not provide it, which is why it is worth some
> explanation.
>
> Doing all this consistently across the many routines would be very
> valuable.  The rfft vs. fft explanation might best belong on the
> module page, since there are several routines for each.  Then the
> routines themselves can have relatively spare pages.  The formulae
> themselves might best go on the module page, for the same reason.

I've started on the module page, along the direction you outlined
above, not complete and no examples quite yet, but if people think it
looks okay, I can try to trim down the specific routines a bit as
well.

How can I make cross-references to `fft` in other functions refer to
numpy.fft[.fftpack].fft, instead of numpy.fft?  I believe the current
package should be searched before parents for cross-references.

> --jh--

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From david at ar.media.kyoto-u.ac.jp  Mon Dec 15 22:42:07 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 16 Dec 2008 12:42:07 +0900
Subject: [SciPy-dev] openopt has to move out from scikits
In-Reply-To: <4946DDB3.10500@ukr.net>
References: <4946DDB3.10500@ukr.net>
Message-ID: <4947238F.6010309@ar.media.kyoto-u.ac.jp>

Dmitrey wrote:
> Hi all,
> openopt v. 0.21 has been released. However, old svn location is no 
> longer valid (my chiefs have forced me to move my code into other 
> location). All details here
>   

Hi Dmitrey,

    Why do you have to make openopt a non scikit ? Your code can be
hosted anywhere your chiefs deem necessary, and still be kept as a scikit.

cheers,

David


From spacey-scipy-dev at lenin.net  Tue Dec 16 01:31:08 2008
From: spacey-scipy-dev at lenin.net (Peter Norton)
Date: Tue, 16 Dec 2008 01:31:08 -0500
Subject: [SciPy-dev] scipy.test() abruptly quits on scipy svn
	0.7.0.dev5241
In-Reply-To: <eeb997dc0812150615l76ad0335yf424cc11a46dd9e9@mail.gmail.com>
References: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>
	<4940AF8E.9040605@ar.media.kyoto-u.ac.jp>
	<eeb997dc0812150343w6aa438cbv9139142d94261e8c@mail.gmail.com>
	<5b8d13220812150408y67d31b8ev24c7b8954e34510e@mail.gmail.com>
	<eeb997dc0812150615l76ad0335yf424cc11a46dd9e9@mail.gmail.com>
Message-ID: <eeb997dc0812152231h5191ef00wda199ac01f7dc547@mail.gmail.com>

To update my prior problem it seems that I didn't re-build cleanly
enough. I'm not sure I'm getting to the same test now. When I build
the current head with numscons and studio 12 w/ libsunperf, I get a
few linkage problems. I don't believe I get the opportunity to ru
test_linsolve.TestLinsolve this time around. It looks like the C++
compiler isn't linking in libCstd.so:

$ LD_PRELOAD=/usr/local/python/lib/libpython2.5.so /lib/ld.so.1
/usr/local/python-2.5.1/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so
ld.so.1: _csr.so: fatal: relocation error: file
/usr/local/python-2.5.1/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so:
symbol __1cDstdJbad_allocG__vtbl_: referenced symbol not found

$ LD_PRELOAD=/usr/local/python/lib/libpython2.5.so:/usr/local/lib/32/libCstd.so
/lib/ld.so.1 /usr/local/python-2.5.1/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so
Segmentation Fault (core dumped)
(I'm not sure what to make of that core dump under the circumstances,
though). The lack of the mangled alloc call should be manually fixable
with an invocation of CC -G for _csr.so, _csc.so and _coo.so (I think
that' it, though that may change).

In a slightly more problematic case, I don't seem to be getting
dfitpack.so built properly. The symbol "bispeu_" isn't resolved. If
this is just a library ordering issue, can anyone suggest a proper
ordering?

$ objdump -x /usr/local/python/lib/python2.5/site-packages/scipy/interpolate/dfitpack.so
| grep bispeu
00008a4c l     F .text  000005c5 f2py_rout_dfitpack_bispeu
00038770 g     O .data  000001ae .XAV2UeWzL0RJE3G.doc_f2py_rout_dfitpack_bispeu
00038750 g     O .data  00000020
.XBV2UeWzL0RJE3G.f2py_rout_dfitpack_bispeu.capi_kwlist
00000000       F *UND*  00000000 bispeu_

I'm happy to provide most of the details of the build environment
including site.cfg, compiler.cfg, etc. It's possible they are
problematic, however since the build completes successfully I'm not in
a position to declare that it *doesn't* work. Assuming this works
sufficiently, this should be enough for me to test with for now in
spite of the numerous failures in the self tests. It involves fewer
library re-builds than using distutils (libraries get linked without
-G, creating exceutables), though I need to test that out, too.

Thanks,

-Peter

P.S. ATLAS doesn't build on solaris x86 32-bit, which is what I was
hoping I'd be able to do this time to avoid the uncertainty  of
building with libsunperf.


From david at ar.media.kyoto-u.ac.jp  Tue Dec 16 01:32:50 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 16 Dec 2008 15:32:50 +0900
Subject: [SciPy-dev] scipy.test() abruptly quits on scipy
	svn	0.7.0.dev5241
In-Reply-To: <eeb997dc0812152231h5191ef00wda199ac01f7dc547@mail.gmail.com>
References: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>	<4940AF8E.9040605@ar.media.kyoto-u.ac.jp>	<eeb997dc0812150343w6aa438cbv9139142d94261e8c@mail.gmail.com>	<5b8d13220812150408y67d31b8ev24c7b8954e34510e@mail.gmail.com>	<eeb997dc0812150615l76ad0335yf424cc11a46dd9e9@mail.gmail.com>
	<eeb997dc0812152231h5191ef00wda199ac01f7dc547@mail.gmail.com>
Message-ID: <49474B92.4060502@ar.media.kyoto-u.ac.jp>

Peter Norton wrote:

> $ LD_PRELOAD=/usr/local/python/lib/libpython2.5.so /lib/ld.so.1
> /usr/local/python-2.5.1/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so
> ld.so.1: _csr.so: fatal: relocation error: file
> /usr/local/python-2.5.1/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so:
> symbol __1cDstdJbad_allocG__vtbl_: referenced symbol not found
>
>   

Ahh, that rings a bell. IIRC, I could not find anything wrong with
either scipy or numscons; I think it is a bug in the sunstudio compiler
when linking C++ as shared library.

> In a slightly more problematic case, I don't seem to be getting
> dfitpack.so built properly. The symbol "bispeu_" isn't resolved. If
> this is just a library ordering issue, can anyone suggest a proper
> ordering?
>   

The interpolate code has changed recently, it may just be that I have
not kept up the scons scripts.

>
> I'm happy to provide most of the details of the build environment
> including site.cfg, compiler.cfg, etc. It's possible they are
> problematic, however since the build completes successfully I'm not in
> a position to declare that it *doesn't* work.

As far as I am concerned, the C++ problem is a sunstudio bug, so there
is nothing I can do about it (there are several problems with sun
compilers with shared libraries, at least on the machine I have been
using it on: some flags are ignored, or not set up correctly when the -G
flag is used - I find surprising that something as basic as linking the
sunperf to a shared library does not work, for example).

cheers,

David


From millman at berkeley.edu  Tue Dec 16 01:59:51 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 15 Dec 2008 22:59:51 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
Message-ID: <c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>

We are in better shape regarding the NR issue.  3 out of the 7 issues
have been resolved.  Before branching and tagging the release
candidate for 0.7.0, we have to deal with the remaining 4 issues.

Unfortunately, I don't have time to even look into this right now.  We
are really close and I hope someone will pitch in and resolve the
remaining issues.  Unless someone can say I wrote this code and I
didn't derive it from the NR code.  Since no one has stepped up to say
this, I assume we are going to have to review the code and make a
determination about this.  Basically, someone needs compare our code
with the NR code.  Check to see if the code is identical.  If the code
isn't identical, does it look like it is the NR code with
modifications (e.g., the same call signature, the same loops, the same
variable names, etc.).  If the code is derived from NR, we need to
remove it and if we want to include the functionality someone would
need to rewrite the code from scratch.  If our code isn't derived from
NR code, we need to correctly document this fact so that we don't
revisit this issue in a year.  Despite how annoying this may be, the
NR copyright specifically states that we don't have the right to port
their code to another language.  So having taken their code and
rewritten it in Python isn't allowed.  We are allowed to read their
descriptions of the algorithms and write code based on that
description; we just can't use their code to write the code.

Here is a summary of where I think we are now (please let me know if I
am incorrect):

1.  2 functions (ridder, brentq) in scipy/optimize/zeros.py

Resolved.

2.  quickselect in scipy/signal/medianfilter.c

Unresolved.

3.  scipy/special/orthogonal.py

Unresolved.

4.  3 functions in scipy/stats/stats.py.

Partially resolved.  One down (Josef rewrote ks_2samp), two left
(ttest_ind and ttest_rel).

On Wed, Dec 3, 2008 at 12:35 PM, Jarrod Millman <millman at berkeley.edu> wrote:
> Given the discussion about Numerical Recipes derived-code, I did a
> little grepping and found 7 instances where code is referencing
> Numerical Recipes.  Before branching for 0.7, I would like to see this
> resolved.  I am not sure what we should do, but the license for using
> Numerical Recipes derived code clearly does not permit us to include
> it in scipy:
>  http://www.nr.com/licenses/redistribute.html
>
> Below I include all the instances I could easily find.  Thoughts?
>
> --------------------------------------------------
>
> In scipy/optimize/zeros.py:
>
> def ridder(f, a, b, args=(),
>           xtol=_xtol, rtol=_rtol, maxiter=_iter,
>           full_output=False, disp=False):
>    """Find root of f in [a,b]
>
>    Ridder routine to find a zero of the function
>    f between the arguments a and b. f(a) and f(b) can not
>    have the same signs. Faster than bisection, but not
>    generaly as fast as the brent rountines. A description
>    may be found in a recent edition of Numerical Recipes.
>    The routine here is modified a bit to be more careful
>    of tolerance.
>
> def brentq(f, a, b, args=(),
>           xtol=_xtol, rtol=_rtol, maxiter=_iter,
>           full_output=False, disp=False):
>    """Find root of f in [a,b]
>
>    The classic Brent routine to find a zero of the function
>    f between the arguments a and b. f(a) and f(b) can not
>    have the same signs. Generally the best of the routines here.
>    It is a safe version of the secant method that uses inverse
>    quadratic extrapolation. The version here is a slight
>    modification that uses a different formula in the extrapolation
>    step. A description may be found in Numerical Recipes, but the
>    code here is probably easier to understand.
>
> In scipy/signal/medianfilter.c:
>
> /*
>  *  This Quickselect routine is based on the algorithm described in
>  *  "Numerical recipies in C", Second Edition,
>  *  Cambridge University Press, 1992, Section 8.5, ISBN 0-521-43108-5
>  */
>
> In  scipy/special/orthogonal.py:
>
> """
> A collection of functions to find the weights and abscissas for
> Gaussian Quadrature.
>
> These calculations are done by finding the eigenvalues of a
> tridiagonal matrix whose entries are dependent on the coefficients
> in the recursion formula for the orthogonal polynomials with the
> corresponding weighting function over the interval.
>
> Many recursion relations for orthogonal polynomials are given:
>
> a1n f_n+1 (x) = (a2n + a3n x ) f_n (x) - a4n f_n-1 (x)
>
> The recursion relation of interest is
>
> P_n+1 (x) = (x - A_n) P_n (x) - B_n P_n-1 (x)
>
> where P has a different normalization than f.
>
> The coefficients can be found as:
>
> A_n = -a2n / a3n
>
> B_n = ( a4n / a3n sqrt(h_n-1 / h_n))**2
>
>     where
>             h_n = int_a^b w(x) f_n(x)^2
> assume:
> P_0(x) = 1
> P_-1(x) == 0
>
> See Numerical Recipies in C, page 156 and
> Abramowitz and Stegun p. 774, 782
>
> In scipy/stats/stats.py:
>
> def ttest_ind(a, b, axis=0):
>    """Calculates the t-obtained T-test on TWO INDEPENDENT samples of scores
>    a, and b.  From Numerical Recipies, p.483. Axis can equal None (ravel
>    array first), or an integer (the axis over which to operate on a and b).
>
> def ttest_rel(a,b,axis=None):
>    """Calculates the t-obtained T-test on TWO RELATED samples of scores, a
>    and b.  From Numerical Recipies, p.483. Axis can equal None (ravel array
>    first), or an integer (the axis over which to operate on a and b).
>
> def ks_2samp(data1, data2):
>    """ Computes the Kolmogorov-Smirnof statistic on 2 samples.  Modified
>    from Numerical Recipies in C, page 493.  Returns KS D-value, prob.  Not
>    ufunc- like.


From charlesr.harris at gmail.com  Tue Dec 16 02:53:28 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 16 Dec 2008 00:53:28 -0700
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
Message-ID: <e06186140812152353g17dbedfdw6355a2021a154764@mail.gmail.com>

On Mon, Dec 15, 2008 at 11:59 PM, Jarrod Millman <millman at berkeley.edu>wrote:

> We are in better shape regarding the NR issue.  3 out of the 7 issues
> have been resolved.  Before branching and tagging the release
> candidate for 0.7.0, we have to deal with the remaining 4 issues.
>
> Unfortunately, I don't have time to even look into this right now.  We
> are really close and I hope someone will pitch in and resolve the
> remaining issues.  Unless someone can say I wrote this code and I
> didn't derive it from the NR code.  Since no one has stepped up to say
> this, I assume we are going to have to review the code and make a
> determination about this.  Basically, someone needs compare our code
> with the NR code.  Check to see if the code is identical.  If the code
> isn't identical, does it look like it is the NR code with
> modifications (e.g., the same call signature, the same loops, the same
> variable names, etc.).  If the code is derived from NR, we need to
> remove it and if we want to include the functionality someone would
> need to rewrite the code from scratch.  If our code isn't derived from
> NR code, we need to correctly document this fact so that we don't
> revisit this issue in a year.  Despite how annoying this may be, the
> NR copyright specifically states that we don't have the right to port
> their code to another language.  So having taken their code and
> rewritten it in Python isn't allowed.  We are allowed to read their
> descriptions of the algorithms and write code based on that
> description; we just can't use their code to write the code.
>
> Here is a summary of where I think we are now (please let me know if I
> am incorrect):
>
> 1.  2 functions (ridder, brentq) in scipy/optimize/zeros.py
>
> Resolved.
>
> 2.  quickselect in scipy/signal/medianfilter.c
>
> Unresolved.
>
> 3.  scipy/special/orthogonal.py
>

The first mention I see of this method of calculating the sample points and
weights for Gaussian quadrature is in this 1967 technical report:
ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/67/81/CS-TR-67-81.pdf. It
doesn't look to me like the NR version of the Golub-Welsch algorithm, or
numpy's for that matter, differs in any significant way. Note that the
original either misspells Welsch or a citation mutation occurred down the
line and the name has been spelled Welsh by all those who copied it without
looking up the original.

A more official publication is G.H. Golub and J.H.Welsch, Computation of
Gauss quadrature rules, Math. Comp., 23(1969), pp. 221-230

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081216/9083946e/attachment.html>

From timmichelsen at gmx-topmail.de  Tue Dec 16 07:06:13 2008
From: timmichelsen at gmx-topmail.de (timmichelsen at gmx-topmail.de)
Date: Tue, 16 Dec 2008 13:06:13 +0100
Subject: [SciPy-dev] doc review & sphinx extension help
Message-ID: <20081216120613.23120@gmx.net>

Dear Scipy developers,
would you please enable my account "timmichelsen"
for editing scipy and numpy documentation?

I would like to add examples and review.

Furthermore, I started with a bibtex extension for
Sphinx.
But I have problems with extracting citations from
docstrings.

Maybe the creators of the sphinx extension have a
bright idea.

I have a description of my approach here:
* http://groups.google.com/group/sphinx-dev/browse_thread/thread/16ba98801e8ea041

I hope to paste an example within this week to
http://www.bitbucket.org/birkenfeld/sphinx/issue/63/make-sphinx-read-bibtex-files-for

Thanks in advance,
Timmie
-- 
Psssst! Schon vom neuen GMX MultiMessenger geh?rt? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger


From aisaac at american.edu  Tue Dec 16 09:38:10 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 16 Dec 2008 09:38:10 -0500
Subject: [SciPy-dev] openopt has to move out from scikits
In-Reply-To: <4947238F.6010309@ar.media.kyoto-u.ac.jp>
References: <4946DDB3.10500@ukr.net> <4947238F.6010309@ar.media.kyoto-u.ac.jp>
Message-ID: <4947BD52.6010208@american.edu>

> Dmitrey wrote:
>> Hi all,
>> openopt v. 0.21 has been released. However, old svn location is no 
>> longer valid (my chiefs have forced me to move my code into other 
>> location). All details here


On 12/15/2008 10:42 PM David Cournapeau apparently wrote:
>     Why do you have to make openopt a non scikit ? Your code can be
> hosted anywhere your chiefs deem necessary, and still be kept as a scikit.


I think Dmitrey's concern is that SciPy svn can be
modified by anyone with commit privileges, while his
"chiefs" want to limit commit rights to OpenOpt
(with which they intend to be reputationally linked).

You seem to be proposing a way around these concerns.
Can you please elaborate?

Thanks,
Alan Isaac



From david at ar.media.kyoto-u.ac.jp  Tue Dec 16 09:34:36 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 16 Dec 2008 23:34:36 +0900
Subject: [SciPy-dev] openopt has to move out from scikits
In-Reply-To: <4947BD52.6010208@american.edu>
References: <4946DDB3.10500@ukr.net> <4947238F.6010309@ar.media.kyoto-u.ac.jp>
	<4947BD52.6010208@american.edu>
Message-ID: <4947BC7C.2060709@ar.media.kyoto-u.ac.jp>

Alan G Isaac wrote:
>
>
> I think Dmitrey's concern is that SciPy svn can be
> modified by anyone with commit privileges, while his
> "chiefs" want to limit commit rights to OpenOpt
> (with which they intend to be reputationally linked).

What I am saying is that openopt can be developed in a different svn
repository (or any other mean) as decided by Dmitrey's "chiefs", while
still being a scikit project, that is sharing the scikit namespace.

Keeping the namespace has many advantages:
    - it does not break backward compatibility
    - it can use all the other resources for a scikit: bug tracker -
although this may not be that useful because you lose the integration
trac/svn since trac cannot refer to 'external' respository AFAIK, being
available through the new webapp St?fan has been working on with one of
his student (so that people will see every new release, for example).

The commit right problem will be a problem if scikit grows
significantly, BTW. For now, there has not been any problem I am aware
of, though.

cheers,

David


From xavier.gnata at gmail.com  Tue Dec 16 10:15:56 2008
From: xavier.gnata at gmail.com (Xavier Gnata)
Date: Tue, 16 Dec 2008 16:15:56 +0100
Subject: [SciPy-dev] Ticket #803  test_pbdv failed
Message-ID: <4947C62C.7020509@gmail.com>

Hi,

Any idea to try to debug that issue (the only one I have with the
current svn scipy)?

My fortran residual knowledge is now so poor that I cannot really debug
this issue.
What I would first like to do is to run a small fortran routine calling
the pbdv procedure from cephes to see what I get (but with my poor
fortran knowledge it will take time :( )

My second point is the following:
If my understanding of pbdv is supposed to do, (0.0, 1.0) is the
correct answer : http://mathworld.wolfram.com/ParabolicCylinderFunction.html
I must be missing something here.

Xavier


From aisaac at american.edu  Tue Dec 16 10:34:36 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 16 Dec 2008 10:34:36 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
Message-ID: <4947CA8C.40008@american.edu>

On 12/16/2008 1:59 AM Jarrod Millman apparently wrote:
> 3.  scipy/special/orthogonal.py
> 
> Unresolved.


>> The coefficients can be found as:
>>
>> A_n = -a2n / a3n
>>
>> B_n = ( a4n / a3n sqrt(h_n-1 / h_n))**2
>>
>>     where
>>             h_n = int_a^b w(x) f_n(x)^2
>> assume:
>> P_0(x) = 1
>> P_-1(x) == 0
>>
>> See Numerical Recipies in C, page 156 and
>> Abramowitz and Stegun p. 774, 782



Travis O. is listed as writing this orthogonal.py in 2000
and doing bug fixes in 2003.  Note that the
reference to Numerical Recipes is one of two
references documenting an algebraic recursion
relation and is not referring to an implementation.

http://www.fizyka.umk.pl/nrbook/c4-5.pdf

http://www-lsp.ujf-grenoble.fr/recherche/a3t2/a3t2a2/bahram/biblio/abramowitz_and_stegun/page_774.htm

I do not see any problem with the code.
I think this one should also be considered resolved.

Alan



From jh at physics.ucf.edu  Tue Dec 16 10:53:50 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Tue, 16 Dec 2008 10:53:50 -0500
Subject: [SciPy-dev] usage of "scipy", "Scipy", "SciPy"
Message-ID: <wl8vdtki0a9.fsf@glup.physics.ucf.edu>

Damian Eads wrote in a thread on scipy-user (ANN: SciKits Portal):

> 2. Replace "Scipy" and "scipy" with "SciPy" when referring to the
> SciPy tool suite. Use "scipy" (all lowercase) in a fixed-width font
> when referring to the Python package name "scipy".

David Cournapeau <david at ar.media.kyoto-u.ac.jp> responded:

> I think it would be nice to have this noted somewhere for everything
> around scipy, actually.

+1, mostly.

I don't think the fixed-width font is crucial; we don't consistently
do this for "numpy" except in actual code examples and I for one find
it jarring to read prose in manuals where you know perfectly well what
is being discussed but fonts are changing all the time as though
you're an idiot and can't remember.  I do find it helpful to change
fonts in syntax usages, etc., just not in paragraphs of prose when the
context is clear (e.g., you're just talking about one item).  Others
will differ.  So, I'd make the suggestion and leave it to the writer
to decide on fonts appropriate to the document.

Also, what about numpy/Numpy/NumPy?  See below.

We have a set of doc guidelines.  I don't think we (yet) have a set of
style guidelines for web sites, but we should.  This could be its
first point.

----------------------------------------------------------------------
Use "SciPy" when referring to the SciPy tool suite.  When referring to
the single Python package named "scipy", use "scipy" (all lowercase),
preferably in a fixed-width font when using fonts to distinguish
syntactic elements.  At or very near first use, and in titles, etc.,
say "SciPy platform" when referring to the tool suite and "scipy
package" when referring to the package alone, to ensure that readers
not familiar with the distinction are not confused.  When
capitalization is required, "scipy" becomes "Scipy" and should be
followed by "package" unless the context is already well established.

Use "NumPy" when referring in general to the syntax, mechanics, and
coding approach of the numpy package and/or the ndarray class, as used
in other packages.  When referring to the single Python package named
"numpy", use "numpy" (all lowercase), preferably in a fixed-width font
when using fonts to distinguish syntactic elements.  Writers should
clarify the context if there is ambiguity.  When capitalization is
required, "numpy" becomes "Numpy".

Examples:

The SciPy platform includes the numpy, scipy, and matplotlib packages.
The SciPy universe is extended by the stand-alone packages called
SciKits, and many other packages distributed separately from the
SciKits.

The fnord package's routines follow NumPy style.  They use ndarrays
and many accept "axis=" and other NumPy keywords.

The fnord package is based on numpy.  It also uses matplotlib, but it
does not import the scipy package.

(Title) The SciPy Platform User Manual
(Title) The Scipy Package User Manual

Incorrect:

(Title) The SciPy User Manual (requires new reader to know our nuances)
(Title) Implementation of the SciPy "axis=" Keyword (should be NumPy)

In the NumPy package,... (should be numpy)
In numpy/scipy, this IDL expression shortens to... (should be NumPy
  for generic expression syntax, SciPy if specific routines outside
  of numpy are involved, or numpy or scipy if routines from those
  packages are involved)
-----------------------------------------------------------------------

Thoughts?  I'm open to calling the syntactic elements "SciPy" rather
than "NumPy", and never allowing "NumPy" or allowing it optionally as
a synonym for "numpy" the package, but I think there is a place for
"NumPy" as described above.

If people like the idea, Stefan, can you put this guideline in our doc
standards and examples?  I'll hang it off of Developer_Zone as the
start of our web style guidelines, once it's had some discussion here
(and pending the outcome).

--jh--


From jh at physics.ucf.edu  Tue Dec 16 12:05:23 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Tue, 16 Dec 2008 12:05:23 -0500
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial
	(and	updating	it))
In-Reply-To: <mailman.3246.1229414019.2878.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
Message-ID: <wl8tz94hwz0.fsf@glup.physics.ucf.edu>

Tom Grydeland <Tom.Grydeland@(gmail.com)> writes:
> I've started on the module page, along the direction you outlined
> above, not complete and no examples quite yet, but if people think it
> looks okay, I can try to trim down the specific routines a bit as
> well.

I like it!  A lot!

I added the helper routines and defined the time and frequency domains
near the top (and took out similar text near the bottom), fixed some
commas, etc.

I think all it needs now is examples.

One point: (I'm not sure it's worth putting these distinctions on the
page, I'm just justifying a change I made.)  Power is energy per unit
time; the energy spectrum and power spectrum are colloquially the
same.  Both are proportional to the square of the amplitude spectrum.
In areas where it matters, np.abs(A)**2 is the energy spectrum and
varies with the signal length (longer signals have more energy).  The
power spectrum is the energy spectrum divided by the time span (in
physical units) of the dataset, and is independent of the signal
length.  Both these must be adjusted by a constant to make the
sampling rate not affect the output.  The zero channel of np.fft(a) is
the mean, not the energy.  The zero channel of np.abs(A)**2 is the
total energy or power, each to within different constants.

--jh--


From gael.varoquaux at normalesup.org  Tue Dec 16 13:28:42 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 16 Dec 2008 19:28:42 +0100
Subject: [SciPy-dev] doc review & sphinx extension help
In-Reply-To: <20081216120613.23120@gmx.net>
References: <20081216120613.23120@gmx.net>
Message-ID: <20081216182842.GA21764@phare.normalesup.org>

On Tue, Dec 16, 2008 at 01:06:13PM +0100, timmichelsen at gmx-topmail.de wrote:
> Dear Scipy developers,
> would you please enable my account "timmichelsen"
> for editing scipy and numpy documentation?

Done. Thanks for your interest.

Ga?l


From spacey-scipy-dev at lenin.net  Tue Dec 16 15:23:23 2008
From: spacey-scipy-dev at lenin.net (Peter Norton)
Date: Tue, 16 Dec 2008 15:23:23 -0500
Subject: [SciPy-dev] scipy.test() abruptly quits on scipy svn
	0.7.0.dev5241
In-Reply-To: <49474B92.4060502@ar.media.kyoto-u.ac.jp>
References: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>
	<4940AF8E.9040605@ar.media.kyoto-u.ac.jp>
	<eeb997dc0812150343w6aa438cbv9139142d94261e8c@mail.gmail.com>
	<5b8d13220812150408y67d31b8ev24c7b8954e34510e@mail.gmail.com>
	<eeb997dc0812150615l76ad0335yf424cc11a46dd9e9@mail.gmail.com>
	<eeb997dc0812152231h5191ef00wda199ac01f7dc547@mail.gmail.com>
	<49474B92.4060502@ar.media.kyoto-u.ac.jp>
Message-ID: <eeb997dc0812161223s19e0a2feob834fd58419e046@mail.gmail.com>

On Tue, Dec 16, 2008 at 1:32 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Peter Norton wrote:
>
>> $ LD_PRELOAD=/usr/local/python/lib/libpython2.5.so /lib/ld.so.1
>> /usr/local/python-2.5.1/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so
>> ld.so.1: _csr.so: fatal: relocation error: file
>> /usr/local/python-2.5.1/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so:
>> symbol __1cDstdJbad_allocG__vtbl_: referenced symbol not found
>>
>>
>
> Ahh, that rings a bell. IIRC, I could not find anything wrong with
> either scipy or numscons; I think it is a bug in the sunstudio compiler
> when linking C++ as shared library.
>
>> In a slightly more problematic case, I don't seem to be getting
>> dfitpack.so built properly. The symbol "bispeu_" isn't resolved. If
>> this is just a library ordering issue, can anyone suggest a proper
>> ordering?
>>
>
> The interpolate code has changed recently, it may just be that I have
> not kept up the scons scripts.

Is there a place where I can find the suggested method of updating this?

>> I'm happy to provide most of the details of the build environment
>> including site.cfg, compiler.cfg, etc. It's possible they are
>> problematic, however since the build completes successfully I'm not in
>> a position to declare that it *doesn't* work.
>
> As far as I am concerned, the C++ problem is a sunstudio bug, so there
> is nothing I can do about it (there are several problems with sun
> compilers with shared libraries, at least on the machine I have been
> using it on: some flags are ignored, or not set up correctly when the -G
> flag is used - I find surprising that something as basic as linking the
> sunperf to a shared library does not work, for example).

It seems that sun doesn't default to automatically linking in the
library for C++, as you say.  I'd prefer to fix this by setting C++
only link flags, but

It looks like scons-local could benefit from having a distinction
between the C linker and the C++ linker, or at least augmenting the CC
linker. In sunlink.py. Adding -lCrun -lCstd to env['SHLINKFLAGS']
doesn't seem to do anything. Is there any chance that scons has a
separate way to pull the flags for a C++ compiler, eg
env['CXXSHLINKFLAGS']?

In the mean time, I'm going to just add -lCrun and -lCstd to LDFLAGS
for everything :(

Thanks,

-Peter


From spacey-scipy-dev at lenin.net  Tue Dec 16 15:35:38 2008
From: spacey-scipy-dev at lenin.net (Peter Norton)
Date: Tue, 16 Dec 2008 15:35:38 -0500
Subject: [SciPy-dev] scipy.test() abruptly quits on scipy svn
	0.7.0.dev5241
In-Reply-To: <eeb997dc0812161223s19e0a2feob834fd58419e046@mail.gmail.com>
References: <eeb997dc0812102220n525e8d67x787d2f081e144f47@mail.gmail.com>
	<4940AF8E.9040605@ar.media.kyoto-u.ac.jp>
	<eeb997dc0812150343w6aa438cbv9139142d94261e8c@mail.gmail.com>
	<5b8d13220812150408y67d31b8ev24c7b8954e34510e@mail.gmail.com>
	<eeb997dc0812150615l76ad0335yf424cc11a46dd9e9@mail.gmail.com>
	<eeb997dc0812152231h5191ef00wda199ac01f7dc547@mail.gmail.com>
	<49474B92.4060502@ar.media.kyoto-u.ac.jp>
	<eeb997dc0812161223s19e0a2feob834fd58419e046@mail.gmail.com>
Message-ID: <eeb997dc0812161235k753a6b56wd0a4322a90bfc761@mail.gmail.com>

After cleaning up my build, I do end up with the same crash:

............................................................................................................................ESSSSSSSSSSSB->ncol
= 1  Bstore->lda = 5  A->nrow = 5
B->Stype = 5  B->Dtype = 0  B->Mtype = 0
** On entry to sgssv, parameter number 7 has an illegal value.
Segmentation Fault (core dumped)

With verbose=10:

Solve with UMFPACK: double precision ... SKIP: UMFPACK appears not to
be compiled
Solve: single precision ... SKIP: UMFPACK appears not to be compiled
test_twodiags (test_linsolve.TestLinsolve) ... B->ncol = 1
Bstore->lda = 5  A->nrow = 5
B->Stype = 5  B->Dtype = 0  B->Mtype = 0
** On entry to sgssv, parameter number 7 has an illegal value.
Segmentation Fault (core dumped)

-Peter




On Tue, Dec 16, 2008 at 3:23 PM, Peter Norton
<spacey-scipy-dev at lenin.net> wrote:
> On Tue, Dec 16, 2008 at 1:32 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>> Peter Norton wrote:
>>
>>> $ LD_PRELOAD=/usr/local/python/lib/libpython2.5.so /lib/ld.so.1
>>> /usr/local/python-2.5.1/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so
>>> ld.so.1: _csr.so: fatal: relocation error: file
>>> /usr/local/python-2.5.1/lib/python2.5/site-packages/scipy/sparse/sparsetools/_csr.so:
>>> symbol __1cDstdJbad_allocG__vtbl_: referenced symbol not found
>>>
>>>
>>
>> Ahh, that rings a bell. IIRC, I could not find anything wrong with
>> either scipy or numscons; I think it is a bug in the sunstudio compiler
>> when linking C++ as shared library.
>>
>>> In a slightly more problematic case, I don't seem to be getting
>>> dfitpack.so built properly. The symbol "bispeu_" isn't resolved. If
>>> this is just a library ordering issue, can anyone suggest a proper
>>> ordering?
>>>
>>
>> The interpolate code has changed recently, it may just be that I have
>> not kept up the scons scripts.
>
> Is there a place where I can find the suggested method of updating this?
>
>>> I'm happy to provide most of the details of the build environment
>>> including site.cfg, compiler.cfg, etc. It's possible they are
>>> problematic, however since the build completes successfully I'm not in
>>> a position to declare that it *doesn't* work.
>>
>> As far as I am concerned, the C++ problem is a sunstudio bug, so there
>> is nothing I can do about it (there are several problems with sun
>> compilers with shared libraries, at least on the machine I have been
>> using it on: some flags are ignored, or not set up correctly when the -G
>> flag is used - I find surprising that something as basic as linking the
>> sunperf to a shared library does not work, for example).
>
> It seems that sun doesn't default to automatically linking in the
> library for C++, as you say.  I'd prefer to fix this by setting C++
> only link flags, but
>
> It looks like scons-local could benefit from having a distinction
> between the C linker and the C++ linker, or at least augmenting the CC
> linker. In sunlink.py. Adding -lCrun -lCstd to env['SHLINKFLAGS']
> doesn't seem to do anything. Is there any chance that scons has a
> separate way to pull the flags for a C++ compiler, eg
> env['CXXSHLINKFLAGS']?
>
> In the mean time, I'm going to just add -lCrun and -lCstd to LDFLAGS
> for everything :(
>
> Thanks,
>
> -Peter
>


From thouis at broad.mit.edu  Tue Dec 16 15:41:43 2008
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Tue, 16 Dec 2008 15:41:43 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
Message-ID: <6c17e6f50812161241k61344a0bx25c3bb8147e5b961@mail.gmail.com>

On Tue, Dec 16, 2008 at 01:59, Jarrod Millman <millman at berkeley.edu> wrote:

> 2.  quickselect in scipy/signal/medianfilter.c
>
> Unresolved.

I looked at the NR code for select (2nd ed, page 342).  It was perhaps
a little too close for comfort, but someone else might disagree.

I rewrote the select algorithm(s) from scratch, referring to the
Wikipedia page describing quickselect, and tested it against the
existing functions on a large number of random cases (small and large
buffers, all the same values, etc.).  It is perhaps a little faster
than the current version.  It's also written as a single macro that
expands for the float/double/byte cases, rather than replicating the
code.

Someone should review this code, and if it looks good, I'll submit a
patch for medianfilter.c.

For 0.8, perhaps scipy should use a variant of constant time median
filtering (http://nomis80.org/ctmf.html)

Ray Jones
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: median_select_new.c
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081216/f2e6c68c/attachment.c>

From millman at berkeley.edu  Tue Dec 16 16:40:12 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 16 Dec 2008 13:40:12 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <6c17e6f50812161241k61344a0bx25c3bb8147e5b961@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<6c17e6f50812161241k61344a0bx25c3bb8147e5b961@mail.gmail.com>
Message-ID: <c7009a550812161340r777c3470w23d3842c7318aff6@mail.gmail.com>

On Tue, Dec 16, 2008 at 12:41 PM, Thouis (Ray) Jones
<thouis at broad.mit.edu> wrote:
> On Tue, Dec 16, 2008 at 01:59, Jarrod Millman <millman at berkeley.edu> wrote:
>
>> 2.  quickselect in scipy/signal/medianfilter.c
>>
>> Unresolved.
>
> I looked at the NR code for select (2nd ed, page 342).  It was perhaps
> a little too close for comfort, but someone else might disagree.
>
> I rewrote the select algorithm(s) from scratch, referring to the
> Wikipedia page describing quickselect, and tested it against the
> existing functions on a large number of random cases (small and large
> buffers, all the same values, etc.).  It is perhaps a little faster
> than the current version.  It's also written as a single macro that
> expands for the float/double/byte cases, rather than replicating the
> code.

Thanks very much.  This was the code I was most concerned with as
well.  Hopefully, someone will have a chance to review your code over
the next few days.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From gilles.rochefort at gmail.com  Tue Dec 16 19:59:58 2008
From: gilles.rochefort at gmail.com (Gilles Rochefort)
Date: Wed, 17 Dec 2008 01:59:58 +0100
Subject: [SciPy-dev] Patching fmin_l_bfgs_b in scipy.optimize
Message-ID: <49484F0E.4000409@gmail.com>


Hi,

I had some troubles getting a stable version of fmin_l_bfgs_b. I got a 
segmentation fault
as soon as I turned a different value than -1 to iprint argument.

As far as I understood, this iprint parameter is directly related to 
some fortran printing/writing logs.
It prints iteration, norm of gradient on screen and also in a file 
called iterate.dat .. Even recompiling
the whole stuff from lapack, atlas etc.. (using gfortran only) up to 
scipy doesn't solve the problem.

Because I definitely gave up the idea to fix the bug by recompiling,  I 
decided to patch the way python
interacts with lbfgsb fortran procedure. So, I disabled the fortran 
iprint and add some iprint functionnality
in pure python code directly in lbfgsb.py (scipy/optimize).  Doing so, I 
get rid of the useless iterate.dat file too.

While patching, I followed by adding some few things :
- an optional stopping rule based on maximum number of iterations. I 
think this is more usefull than
 the number of function evaluations because the fortran code performs a 
linesearch procedure inside the lbfgsb ones.
- a callback procedure which allows to inspect solution, criterion, and 
gradient at current iterate.

Maybe my patch should be usefull for someone,
so I decided to share it with the scipy community.

Gilles.

PS:
I provide two files  :
lbfgsb.py -- the full python script to put in scipy/optimize
scipy-optimize-lbfgsb-12162008.patch -- a patch file

-------------- next part --------------
A non-text attachment was scrubbed...
Name: lbfgsb.py
Type: text/x-python
Size: 10287 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081217/411427ef/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scipy-optimize-lbfgs-12162008.patch
Type: text/x-patch
Size: 3466 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081217/411427ef/attachment.bin>

From jason-sage at creativetrax.com  Tue Dec 16 20:35:49 2008
From: jason-sage at creativetrax.com (jason-sage at creativetrax.com)
Date: Tue, 16 Dec 2008 19:35:49 -0600
Subject: [SciPy-dev] Patching fmin_l_bfgs_b in scipy.optimize
In-Reply-To: <49484F0E.4000409@gmail.com>
References: <49484F0E.4000409@gmail.com>
Message-ID: <49485775.9010506@creativetrax.com>

Gilles Rochefort wrote:
>
> Hi,
>
> I had some troubles getting a stable version of fmin_l_bfgs_b. I got a 
> segmentation fault
> as soon as I turned a different value than -1 to iprint argument.
>
> As far as I understood, this iprint parameter is directly related to 
> some fortran printing/writing logs.
> It prints iteration, norm of gradient on screen and also in a file 
> called iterate.dat .. Even recompiling
> the whole stuff from lapack, atlas etc.. (using gfortran only) up to 
> scipy doesn't solve the problem.
>
> Because I definitely gave up the idea to fix the bug by recompiling,  
> I decided to patch the way python
> interacts with lbfgsb fortran procedure. So, I disabled the fortran 
> iprint and add some iprint functionnality
> in pure python code directly in lbfgsb.py (scipy/optimize).  Doing so, 
> I get rid of the useless iterate.dat file too.
>
> While patching, I followed by adding some few things :
> - an optional stopping rule based on maximum number of iterations. I 
> think this is more usefull than
> the number of function evaluations because the fortran code performs a 
> linesearch procedure inside the lbfgsb ones.
> - a callback procedure which allows to inspect solution, criterion, 
> and gradient at current iterate.
>
> Maybe my patch should be usefull for someone,
> so I decided to share it with the scipy community.
>

Thanks for sharing.  As a general rule, I like having callback functions
inside of these types of procedures, if only for educational purposes
(as long as the callback check is negligible if no callback is
provided).  Along these lines, I notice that several times you use
"!=None" in an if statement.  It is faster to use "is not None" versus
"!= None".  I think this has to do with there being only one None
object, so a memory reference comparison (done with "is") is sufficient
and faster.

If a python wizard knows better, please correct me!

In [1]: a=lambda x: x

In [2]: %timeit a is not None
10000000 loops, best of 3: 113 ns per loop

In [3]: %timeit a != None
10000000 loops, best of 3: 146 ns per loop

In [4]: a=None

In [5]: %timeit a is not None
10000000 loops, best of 3: 99.2 ns per loop

In [6]: %timeit a != None
10000000 loops, best of 3: 147 ns per loop

Thanks,

Jason




From gilles.rochefort at gmail.com  Tue Dec 16 20:44:44 2008
From: gilles.rochefort at gmail.com (Gilles Rochefort)
Date: Wed, 17 Dec 2008 02:44:44 +0100
Subject: [SciPy-dev] Patching fmin_l_bfgs_b in scipy.optimize
In-Reply-To: <49485775.9010506@creativetrax.com>
References: <49484F0E.4000409@gmail.com> <49485775.9010506@creativetrax.com>
Message-ID: <4948598C.7050000@gmail.com>

Thanks for the advice,
here is my new patch !

Regards,
Gilles.

jason-sage at creativetrax.com a ?crit :
> Gilles Rochefort wrote:
>   
>> Hi,
>>
>> I had some troubles getting a stable version of fmin_l_bfgs_b. I got a 
>> segmentation fault
>> as soon as I turned a different value than -1 to iprint argument.
>>
>> As far as I understood, this iprint parameter is directly related to 
>> some fortran printing/writing logs.
>> It prints iteration, norm of gradient on screen and also in a file 
>> called iterate.dat .. Even recompiling
>> the whole stuff from lapack, atlas etc.. (using gfortran only) up to 
>> scipy doesn't solve the problem.
>>
>> Because I definitely gave up the idea to fix the bug by recompiling,  
>> I decided to patch the way python
>> interacts with lbfgsb fortran procedure. So, I disabled the fortran 
>> iprint and add some iprint functionnality
>> in pure python code directly in lbfgsb.py (scipy/optimize).  Doing so, 
>> I get rid of the useless iterate.dat file too.
>>
>> While patching, I followed by adding some few things :
>> - an optional stopping rule based on maximum number of iterations. I 
>> think this is more usefull than
>> the number of function evaluations because the fortran code performs a 
>> linesearch procedure inside the lbfgsb ones.
>> - a callback procedure which allows to inspect solution, criterion, 
>> and gradient at current iterate.
>>
>> Maybe my patch should be usefull for someone,
>> so I decided to share it with the scipy community.
>>
>>     
>
> Thanks for sharing.  As a general rule, I like having callback functions
> inside of these types of procedures, if only for educational purposes
> (as long as the callback check is negligible if no callback is
> provided).  Along these lines, I notice that several times you use
> "!=None" in an if statement.  It is faster to use "is not None" versus
> "!= None".  I think this has to do with there being only one None
> object, so a memory reference comparison (done with "is") is sufficient
> and faster.
>
> If a python wizard knows better, please correct me!
>
> In [1]: a=lambda x: x
>
> In [2]: %timeit a is not None
> 10000000 loops, best of 3: 113 ns per loop
>
> In [3]: %timeit a != None
> 10000000 loops, best of 3: 146 ns per loop
>
> In [4]: a=None
>
> In [5]: %timeit a is not None
> 10000000 loops, best of 3: 99.2 ns per loop
>
> In [6]: %timeit a != None
> 10000000 loops, best of 3: 147 ns per loop
>
> Thanks,
>
> Jason
>
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081217/e74b89f2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lbfgsb.py
Type: text/x-python
Size: 10295 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081217/e74b89f2/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lbfgs-optimize-12172008.patch
Type: text/x-patch
Size: 3474 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081217/e74b89f2/attachment.bin>

From thouis at broad.mit.edu  Tue Dec 16 21:50:18 2008
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Tue, 16 Dec 2008 21:50:18 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812161340r777c3470w23d3842c7318aff6@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<6c17e6f50812161241k61344a0bx25c3bb8147e5b961@mail.gmail.com>
	<c7009a550812161340r777c3470w23d3842c7318aff6@mail.gmail.com>
Message-ID: <6c17e6f50812161850y4f6042fva85680c7df597b3a@mail.gmail.com>

Here's a slightly better version (clearer, about 10% faster).

Ray Jones
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: median_select_new_v2.c
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081216/323bb660/attachment.c>

From josef.pktd at gmail.com  Wed Dec 17 00:14:44 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 17 Dec 2008 00:14:44 -0500
Subject: [SciPy-dev] hidden scipy.optimize.nnls
Message-ID: <1cd32cbb0812162114m1cb5b6c6i5e76e97a8065a746@mail.gmail.com>

Is scipy.optimize.nnls hidden on purpose?

It is not included in optimize.__init__.__all__ nor in info.py  and is
also not in the docs.

Or should we add to optimize.__init__?

from nnls import nnls

Josef


From tom.grydeland at gmail.com  Wed Dec 17 02:48:03 2008
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Wed, 17 Dec 2008 08:48:03 +0100
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
	it))
In-Reply-To: <wl8tz94hwz0.fsf@glup.physics.ucf.edu>
References: <mailman.3246.1229414019.2878.scipy-dev@scipy.org>
	<wl8tz94hwz0.fsf@glup.physics.ucf.edu>
Message-ID: <c21c0b2f0812162348q65858bbfta0cc8e2b09bfc0fe@mail.gmail.com>

On Tue, Dec 16, 2008 at 6:05 PM,  <jh at physics.ucf.edu> wrote:

> I like it!  A lot!

Excellent!  Thanks.

> I added the helper routines and defined the time and frequency domains
> near the top (and took out similar text near the bottom), fixed some
> commas, etc.

Looks good.  I didn't like the parentheses "(each gets exactly half
the spectrum) in the discussion on real transforms", I think it is too
easy to misunderstand (as I did on first reading).

> I think all it needs now is examples.

Indeed.  I'll see what I can do about that.

> One point: (I'm not sure it's worth putting these distinctions on the
> page, I'm just justifying a change I made.)  Power is energy per unit
> time; the energy spectrum and power spectrum are colloquially the
> same.

I was fairly sure what I wrote was not correct, but that it could be
fixed.  I hope what ends up in the docstring is correct, not just
colloquially.

>  Both are proportional to the square of the amplitude spectrum.
> In areas where it matters, np.abs(A)**2 is the energy spectrum and
> varies with the signal length (longer signals have more energy).  The
> power spectrum is the energy spectrum divided by the time span (in
> physical units) of the dataset, and is independent of the signal
> length.  Both these must be adjusted by a constant to make the
> sampling rate not affect the output.

I think perhaps this should be included, it is clarifying.

> The zero channel of np.fft(a) is
> the mean, not the energy.

Well -- A[0], according to the definition in the first equation, is
simply the sum of all the values in the input vector (since k == 0,
all the exponentials are == 1).  I would not call that the mean.

>  The zero channel of np.abs(A)**2 is the
> total energy or power, each to within different constants.

Agree.

> --jh--

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From jh at physics.ucf.edu  Wed Dec 17 13:18:11 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Wed, 17 Dec 2008 13:18:11 -0500
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
	it))
In-Reply-To: <c21c0b2f0812162348q65858bbfta0cc8e2b09bfc0fe@mail.gmail.com>
	(tom.grydeland@gmail.com)
References: <mailman.3246.1229414019.2878.scipy-dev@scipy.org>
	<wl8tz94hwz0.fsf@glup.physics.ucf.edu>
	<c21c0b2f0812162348q65858bbfta0cc8e2b09bfc0fe@mail.gmail.com>
Message-ID: <wl8wsdyhdi4.fsf@glup.physics.ucf.edu>

Joe: 

> I added the helper routines and defined the time and frequency domains
> near the top (and took out similar text near the bottom), fixed some
> commas, etc.

Tom:

> Looks good.  I didn't like the parentheses "(each gets exactly half
> the spectrum) in the discussion on real transforms", I think it is too
> easy to misunderstand (as I did on first reading).

Ok, but I think we need to tell them that the amplitudes from fft each
have half the value of the output of rfft, because the fft output
includes the set of redundant frequencies, which get summed in the
real case.  Maybe say just that.

Joe: 

> The zero channel of np.fft(a) is
> the mean, not the energy.

Tom:

> Well -- A[0], according to the definition in the first equation, is
> simply the sum of all the values in the input vector (since k == 0,
> all the exponentials are == 1).  I would not call that the mean.

You're right, it's the sum.  I was brought up on a 1/N normalization.

--jh--


From millman at berkeley.edu  Wed Dec 17 14:28:04 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 17 Dec 2008 11:28:04 -0800
Subject: [SciPy-dev] hidden scipy.optimize.nnls
In-Reply-To: <1cd32cbb0812162114m1cb5b6c6i5e76e97a8065a746@mail.gmail.com>
References: <1cd32cbb0812162114m1cb5b6c6i5e76e97a8065a746@mail.gmail.com>
Message-ID: <c7009a550812171128x6271f772wc0e312a38712179@mail.gmail.com>

On Tue, Dec 16, 2008 at 9:14 PM,  <josef.pktd at gmail.com> wrote:
> Is scipy.optimize.nnls hidden on purpose?

It was added a few months ago by Uwe Schmitt.  I don't think it was
hidden on purpose.

Please add:
  from nnls import nnls
to optimize.__init__.py and add a entry to info.py.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From pav at iki.fi  Wed Dec 17 15:55:57 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 17 Dec 2008 20:55:57 +0000 (UTC)
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
Message-ID: <gibp0s$7pe$1@ger.gmane.org>

Hi,

Sorry for the delay -- busy times...

First, a short summary of what I think it would be good to:

- Remove broyden1_modified

- Remove broyden_generalized; the code is identical to anderson
  with w0=0.

- Remove broyden_modified; if I understand Eyert [1] correctly, this method
  is expected to be always inferior to (extended) Anderson mixing. So I don't
  see a real need for retaining this in Scipy. Also, I don't know how to check
  that it works correctly.

- Remove anderson2, because it does not work for any of the test problems.
  Since `anderson` seems to work, I don't think it's wise to try to get
  this second implementation up to speed.

- Change broyden1 to update the inverse Jacobian using Sherman-Morrison.
  No need for inverting J on every iteration.

- Convert broyden3 to a simple limited-memory method.
  Rename it, too.

- Change the API so that there's a single nonlinear solver function,
  which takes an option that determines which Jacobian approximation to use.

- Maybe add a reverse-communication interface, see below.

.. [1] V. Eyert, J. Comp. Phys, 124, 271 (1996).


Here comes the rest:

Mon, 08 Dec 2008 18:43:47 +0100, Ondrej Certik wrote:
> On Sat, Nov 29, 2008 at 8:40 PM, Pauli Virtanen <pav at iki.fi> wrote:
[clip]
> First let me apologize for taking me so long to reply. I wrote this code
> in the first place and I am happy that Pauli has rewritten it. I agree
> with the general direction, but I think this change should not go into
> 0.7.0, as it changes the interface and it is not well tested yet. Also,
> you renamed all the working broyden implementations that I use as
> BadBroyden, so I suggest to name them GoodBroyden, more below.

This is probably not going to be in 0.7.0, even though the release was
postponed. The API might still need a bit of thinking, and it's probably best
to test this thoroughly.

The difference in Broyden's good/first and bad/second method is to my
understanding the one mentioned in Wikipedia: both methods can be written as
updates to the inverse Jacobian because the update is rank-1 so that the
Sherman-Morrison formula applies. The difference can be seen if one looks at
the domains (right-hand vector) of the rank-1 inverse updates: Broyden's "good"
method uses J_{n-1}^{-T} dx, whereas the "bad" method uses df. These two
vectors are not equal, even though both updates satisfy the quasi-Newton
condition.

[clip]
>> - Rewrote docstrings.
> 
> in general +1, but I don't understand what all the implementations
> actually do. I suggest to also port my comments from the module
> docstring into the respective classes. I can help you with that, after
> we agree on other things below.

I'm not sure how much we can say about relative efficiency of the various
methods. We can cite some references on this, but I'd rather not say much
based on a couple of test problems.

[clip]
>> * I'd like to remove `broyden2` as the actual Jacobian approximation in
>>  this appears to be the same as in `broyden3`, and there does not
>>  appear to be much difference in the work involved in the two.
>>
>>  Ondrej, since you wrote the original code, do you think there is a
>>  reason to keep both?
> 
> I think it is, at least for tutorial purposes and also as an easy way to
> check that all is fine. Because there may be some numericall
> differences, also for a lot of iterations, the memory consumption of
> broyden3 is not limited (is it?). Broyden2 just stores the NxN dense
> matrix (that can of course be big), but that's it.

Well, I think having two implementations of the same method does not really
make sense, and scipy.optimize should contain only production-quality code.

But yes, the memory consumption of broyden3 and broyden2 are not exactly the
same, although neither is a real limited-memory method. See below.

[clip]
>> * Also, which articles were used as reference for the non-Quasi-Newton
>>  algorithms:
>>
>>  - `broyden_modified`. This appears to be a bit exotic, and it uses
>>    several magic constants (`w0`, `wl`) whose meaning is not clear to
>>    me.
>>
>>    A reference would be helpful here, also for the user who has to
>>    choose the parameters as appropriate for his/her specific problem.
> 
> You can use my master thesis:
> 
> http://ondrej.certik.cz/site_media/cookbook/master.pdf
> 
> pages 27-31. Everything is explained in there, plus references given to
> the original literature.

Looking at Eyert's paper, it seems to me that there's no need to retain
this method.

>>  - `broyden_generalized`, `anderson`, `anderson2`. These appear to be
>>    variants of Anderson mixing, so probably we only want at most one of
>>    these. Also, broyden_generalized() == anderson(w0=0), am I correct?
> 
> Yes and no, see my master thesis for details and references. There are
> some subtle differences. But those methods have never worked for me
> well.
>
>>    `anderson` and `anderson2` don't appear to function equivalently,
>>    and I suspect the update formula in the latter is wrong, since this
>>    algorithm can't solve any of the test problems. Do you have a
>>    reference for this?
> 
> Yes, see my master thesis, pages above, it's the reference [4].

Ok, anderson2 and anderson should be exactly equivalent. This probably means
that there's a bug in anderson2, or maybe the default parameters are somehow
strange. Of the tree, I'd retain only `anderson`.

[clip]
>>  - `excitingmixing`. A reference would be useful, to clarify the
>>    heuristics used.
[clip]
>>  - `linearmixing`. I'm not sure that Scipy should advocate this
>>    method :)
[clip]
>>  Linearmixing and excitingmixing also seem to require something from
>>  the objective function, possibly that the eigenvalues of its Jacobian
>>  have the opposite sign than `alpha`. For example, neither of them can
>>  find a solution for the equation ``I x = 0`` where ``I`` is the
>>  identity matrix (test function F2). So, I'm a bit tempted to remove
>>  also these routines, as it seems they probably are not too useful for
>>  general problems.
> 
> Please don't removed those routines, they are essential for electronic
> structure calculations, as they are very robust, fast and doesn't
> require any memory. It just works for many problems.

These are pretty straightforward methods, so maybe they can stay.  But the
docstrings should clearly say that these are intended only for specialized
problems.

I wonder if electronic structure (DFT) calculations are a bit of a special
case. Ie. are the eigenvalues of the Jacobian of the Kohn-Sham functional F[n]
all negative (or less than 1)? 

[clip]
>> * One could add more algorithms suitable for large-scale problems; for
>>  example some limited-memory Broyden methods (eg. [1])
> 
> I think in the references in my thesis, the broyden3 is sometimes called
> the limited-memory Broyden method. However, I checked the equations in
> [1] in your email and they need to do a singular-value-decomposition, so
> it seems there is yet another approach to this. So yes.

It's a part of a limited-memory method, but not the whole story, because the
present implementation does not actually throw any data away.  So, it's
an "unlimited memory" Broyden method :)

But it would be very easy (= 3 lines of code) to make it a simple
limited-memory method, just by throwing away iterates older than some M.  The
singular value reduction can be implemented later. There are probably many
different schemes proposed in the literature how the reduction can be done...

[clip]
> Let me know what you think about my comments. Basically, scipy should
> have as many (working and well tested) algorithms as possible, with the
> same interface, so that one can change the algorithms and see which
> works for the particular problem.

Yes, but this must be balanced against maintenance + testing costs.  I want to
be sure that the code I contribute works, and I wouldn't like to spend too much
time working on unproven methods. So I'd rather include only methods for which
there is reason to believe that they work well.

> I am glad that you are also using those methods, so that we can work
> on this together.

It'd be very useful if you could test this code, including the line search,
on a real large-scale problem. At present, I'm not sure if it makes too many
iterations compared to its payoff.

Also, we might need to implement a reverse communication interface,
something on the lines of

    iterator = Solver(x0)
    for x in iterator:
        iterator.feed(func(x))
        if iterator.converged(): break
    else:
        raise RuntimeError("Didn't converge")

For expensive self-consistent iterations, this often feels the natural way
to use the nonlinear solver.

-- 
Pauli Virtanen



From pav at iki.fi  Wed Dec 17 16:06:25 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 17 Dec 2008 21:06:25 +0000 (UTC)
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
	<3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
	<85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>
	<e86a5fd00812081201p34bc937ava0b5c2a76af7abca@mail.gmail.com>
	<85b5c3130812150719l2b3af06credba9a92c15e145e@mail.gmail.com>
Message-ID: <gibpkh$7pe$2@ger.gmane.org>

Mon, 15 Dec 2008 16:19:29 +0100, Ondrej Certik wrote:
[clip]
> Pauli --- do you have any comments to the discussion?  After we agree on
> a better naming, I'll then try to use your branch at github and help you
> polish it so that both my and your codes run well and then we can push
> it to scipy. What do you think?

I'm not actually sure how this goes. Is Scipy trunk frozen until 0.7.0? 
At least there's no branch for 0.7, only the beta1 tag, so I'd assume the 
trunk is frozen for the moment.

Anyway, for the time being github will do, though I'm not sure how well 
git-svn with all its frequent rebasing will work in collaboration. We can 
of course hold off rebasing the branch (and only pulling) until the code 
gets in Scipy trunk, if git starts to get in the way.

Btw, I'm not sure how much I manage to work on this during the next few 
weeks, it's holidays soon, and I probably don't do much coding then. 
Unless I get extremely bored :) Anyway, we're not in a hurry with this.

Cheers,
Pauli



From millman at berkeley.edu  Wed Dec 17 16:25:07 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 17 Dec 2008 13:25:07 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <6c17e6f50812161850y4f6042fva85680c7df597b3a@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<6c17e6f50812161241k61344a0bx25c3bb8147e5b961@mail.gmail.com>
	<c7009a550812161340r777c3470w23d3842c7318aff6@mail.gmail.com>
	<6c17e6f50812161850y4f6042fva85680c7df597b3a@mail.gmail.com>
Message-ID: <c7009a550812171325h429068ebg77be51020d147ef7@mail.gmail.com>

On Tue, Dec 16, 2008 at 6:50 PM, Thouis (Ray) Jones
<thouis at broad.mit.edu> wrote:
> Here's a slightly better version (clearer, about 10% faster).

It looks good to me.  Did you take a look at whether medfilter2 was
from NR as well?  If not, I would appreciate it if you would take a
quick look.  It would also be nice if you would be willing to rewrite
the medfilter2s as a single macro which expands for the
float/double/byte cases like you did for the quick select.

Could you also provide some tests for medfilt2 in test_signaltools.py:
  trunk/scipy/signal/tests/test_signaltools.py

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Wed Dec 17 16:28:26 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 17 Dec 2008 13:28:26 -0800
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <gibpkh$7pe$2@ger.gmane.org>
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
	<3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
	<85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>
	<e86a5fd00812081201p34bc937ava0b5c2a76af7abca@mail.gmail.com>
	<85b5c3130812150719l2b3af06credba9a92c15e145e@mail.gmail.com>
	<gibpkh$7pe$2@ger.gmane.org>
Message-ID: <c7009a550812171328o2b1425dch1d31a972cbc828ab@mail.gmail.com>

On Wed, Dec 17, 2008 at 1:06 PM, Pauli Virtanen <pav at iki.fi> wrote:
> I'm not actually sure how this goes. Is Scipy trunk frozen until 0.7.0?
> At least there's no branch for 0.7, only the beta1 tag, so I'd assume the
> trunk is frozen for the moment.

Yes, the trunk is "frozen' until we branch for 0.70.  The only thing
that should be committed to the trunk at this point is documentation
improvements, new tests, or essential or trivial bug-fixes that have
been discussed on the list.  I hope that we can resolve the remaining
issues relating to NR licensing by the end of the weekend.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From pav at iki.fi  Wed Dec 17 16:29:01 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 17 Dec 2008 21:29:01 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <20081204102536.5faad9d5.rbastian@free.fr>
	<4937E2C5.5090001@american.edu>
	<20081204141114.GH18992@phare.normalesup.org>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gi18qv$nt4$1@ger.gmane.org>
	<20081213212143.GJ19210@phare.normalesup.org>
	<gi19al$rca$1@ger.gmane.org>
	<20081213212743.GK19210@phare.normalesup.org>
	<1cd32cbb0812131502s9838a46xae4688dae0cf9db5@mail.gmail.com>
Message-ID: <gibqut$gc8$1@ger.gmane.org>

Sat, 13 Dec 2008 18:02:50 -0500, josef.pktd wrote:
> In both numpy  and scipy chm files I get: The page cannot be displayed
> The page you are looking for is currently unavailable. The Web site
> might be experiencing technical difficulties, or you may need to adjust
> your browser settings.
> 
> contents,search, index work correctly, but the page content is not
> found.

Interesting. From Gareth Elston's mail it sounds like this is some 
internet explorer protection settings issue. The files work fine with 
Linux-based readers.

Did you save the file first on disk, or opened it directly from the 
browser? Looking at http://support.microsoft.com/kb/902225/ this might be 
an issue...

> I decompiled and recompiled in with the html help workshop and it looks
> pretty good. Search and Index are very fast and helpful.

I'll probably need to check the build process 

-- 
Pauli Virtanen



From pav at iki.fi  Wed Dec 17 16:39:20 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 17 Dec 2008 21:39:20 +0000 (UTC)
Subject: [SciPy-dev] r5268
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
	<3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
	<85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>
	<e86a5fd00812081201p34bc937ava0b5c2a76af7abca@mail.gmail.com>
	<85b5c3130812150719l2b3af06credba9a92c15e145e@mail.gmail.com>
	<gibpkh$7pe$2@ger.gmane.org>
	<c7009a550812171328o2b1425dch1d31a972cbc828ab@mail.gmail.com>
Message-ID: <gibri8$gc8$2@ger.gmane.org>

Wed, 17 Dec 2008 13:28:26 -0800, Jarrod Millman wrote:
[clip]
> Yes, the trunk is "frozen' until we branch for 0.70.  The only thing
> that should be committed to the trunk at this point is documentation
> improvements, new tests, or essential or trivial bug-fixes that have
> been discussed on the list.

Ok, I'll explain then also this: r5268

In the interp1d n-d cleanup I by mistake broke linear interpolation of 
complex values. This fixes it.

1D spline interpolation has never worked for complex-valued data, due to 
_fitpack.bisplev being written in C, so I changed interp1d to raise error 
in __init__ rather than later on in __call__.

Tests included, naturally.

-- 
Pauli Virtanen



From millman at berkeley.edu  Wed Dec 17 16:41:09 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 17 Dec 2008 13:41:09 -0800
Subject: [SciPy-dev] r5268
In-Reply-To: <gibri8$gc8$2@ger.gmane.org>
References: <1227987645.8264.200.camel@idol> <ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
	<3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
	<85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>
	<e86a5fd00812081201p34bc937ava0b5c2a76af7abca@mail.gmail.com>
	<85b5c3130812150719l2b3af06credba9a92c15e145e@mail.gmail.com>
	<gibpkh$7pe$2@ger.gmane.org>
	<c7009a550812171328o2b1425dch1d31a972cbc828ab@mail.gmail.com>
	<gibri8$gc8$2@ger.gmane.org>
Message-ID: <c7009a550812171341w46438978ta32349acac631348@mail.gmail.com>

On Wed, Dec 17, 2008 at 1:39 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Ok, I'll explain then also this: r5268
>
> In the interp1d n-d cleanup I by mistake broke linear interpolation of
> complex values. This fixes it.
>
> 1D spline interpolation has never worked for complex-valued data, due to
> _fitpack.bisplev being written in C, so I changed interp1d to raise error
> in __init__ rather than later on in __call__.
>
> Tests included, naturally.

Sounds good.  Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From ondrej at certik.cz  Wed Dec 17 16:43:07 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 17 Dec 2008 22:43:07 +0100
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <gibp0s$7pe$1@ger.gmane.org>
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<gibp0s$7pe$1@ger.gmane.org>
Message-ID: <85b5c3130812171343x60151bdaja45a8847c8830e74@mail.gmail.com>

On Wed, Dec 17, 2008 at 9:55 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Hi,
>
> Sorry for the delay -- busy times...
>
> First, a short summary of what I think it would be good to:
>
> - Remove broyden1_modified
>
> - Remove broyden_generalized; the code is identical to anderson
>  with w0=0.
>
> - Remove broyden_modified; if I understand Eyert [1] correctly, this method
>  is expected to be always inferior to (extended) Anderson mixing. So I don't
>  see a real need for retaining this in Scipy. Also, I don't know how to check
>  that it works correctly.
>
> - Remove anderson2, because it does not work for any of the test problems.
>  Since `anderson` seems to work, I don't think it's wise to try to get
>  this second implementation up to speed.
>
> - Change broyden1 to update the inverse Jacobian using Sherman-Morrison.
>  No need for inverting J on every iteration.
>
> - Convert broyden3 to a simple limited-memory method.
>  Rename it, too.
>
> - Change the API so that there's a single nonlinear solver function,
>  which takes an option that determines which Jacobian approximation to use.
>
> - Maybe add a reverse-communication interface, see below.
>
> .. [1] V. Eyert, J. Comp. Phys, 124, 271 (1996).
>
>
> Here comes the rest:
>
> Mon, 08 Dec 2008 18:43:47 +0100, Ondrej Certik wrote:
>> On Sat, Nov 29, 2008 at 8:40 PM, Pauli Virtanen <pav at iki.fi> wrote:
> [clip]
>> First let me apologize for taking me so long to reply. I wrote this code
>> in the first place and I am happy that Pauli has rewritten it. I agree
>> with the general direction, but I think this change should not go into
>> 0.7.0, as it changes the interface and it is not well tested yet. Also,
>> you renamed all the working broyden implementations that I use as
>> BadBroyden, so I suggest to name them GoodBroyden, more below.
>
> This is probably not going to be in 0.7.0, even though the release was
> postponed. The API might still need a bit of thinking, and it's probably best
> to test this thoroughly.
>
> The difference in Broyden's good/first and bad/second method is to my
> understanding the one mentioned in Wikipedia: both methods can be written as
> updates to the inverse Jacobian because the update is rank-1 so that the
> Sherman-Morrison formula applies. The difference can be seen if one looks at
> the domains (right-hand vector) of the rank-1 inverse updates: Broyden's "good"
> method uses J_{n-1}^{-T} dx, whereas the "bad" method uses df. These two
> vectors are not equal, even though both updates satisfy the quasi-Newton
> condition.
>
> [clip]
>>> - Rewrote docstrings.
>>
>> in general +1, but I don't understand what all the implementations
>> actually do. I suggest to also port my comments from the module
>> docstring into the respective classes. I can help you with that, after
>> we agree on other things below.
>
> I'm not sure how much we can say about relative efficiency of the various
> methods. We can cite some references on this, but I'd rather not say much
> based on a couple of test problems.
>
> [clip]
>>> * I'd like to remove `broyden2` as the actual Jacobian approximation in
>>>  this appears to be the same as in `broyden3`, and there does not
>>>  appear to be much difference in the work involved in the two.
>>>
>>>  Ondrej, since you wrote the original code, do you think there is a
>>>  reason to keep both?
>>
>> I think it is, at least for tutorial purposes and also as an easy way to
>> check that all is fine. Because there may be some numericall
>> differences, also for a lot of iterations, the memory consumption of
>> broyden3 is not limited (is it?). Broyden2 just stores the NxN dense
>> matrix (that can of course be big), but that's it.
>
> Well, I think having two implementations of the same method does not really
> make sense, and scipy.optimize should contain only production-quality code.
>
> But yes, the memory consumption of broyden3 and broyden2 are not exactly the
> same, although neither is a real limited-memory method. See below.
>
> [clip]
>>> * Also, which articles were used as reference for the non-Quasi-Newton
>>>  algorithms:
>>>
>>>  - `broyden_modified`. This appears to be a bit exotic, and it uses
>>>    several magic constants (`w0`, `wl`) whose meaning is not clear to
>>>    me.
>>>
>>>    A reference would be helpful here, also for the user who has to
>>>    choose the parameters as appropriate for his/her specific problem.
>>
>> You can use my master thesis:
>>
>> http://ondrej.certik.cz/site_media/cookbook/master.pdf
>>
>> pages 27-31. Everything is explained in there, plus references given to
>> the original literature.
>
> Looking at Eyert's paper, it seems to me that there's no need to retain
> this method.
>
>>>  - `broyden_generalized`, `anderson`, `anderson2`. These appear to be
>>>    variants of Anderson mixing, so probably we only want at most one of
>>>    these. Also, broyden_generalized() == anderson(w0=0), am I correct?
>>
>> Yes and no, see my master thesis for details and references. There are
>> some subtle differences. But those methods have never worked for me
>> well.
>>
>>>    `anderson` and `anderson2` don't appear to function equivalently,
>>>    and I suspect the update formula in the latter is wrong, since this
>>>    algorithm can't solve any of the test problems. Do you have a
>>>    reference for this?
>>
>> Yes, see my master thesis, pages above, it's the reference [4].
>
> Ok, anderson2 and anderson should be exactly equivalent. This probably means
> that there's a bug in anderson2, or maybe the default parameters are somehow
> strange. Of the tree, I'd retain only `anderson`.
>
> [clip]
>>>  - `excitingmixing`. A reference would be useful, to clarify the
>>>    heuristics used.
> [clip]
>>>  - `linearmixing`. I'm not sure that Scipy should advocate this
>>>    method :)
> [clip]
>>>  Linearmixing and excitingmixing also seem to require something from
>>>  the objective function, possibly that the eigenvalues of its Jacobian
>>>  have the opposite sign than `alpha`. For example, neither of them can
>>>  find a solution for the equation ``I x = 0`` where ``I`` is the
>>>  identity matrix (test function F2). So, I'm a bit tempted to remove
>>>  also these routines, as it seems they probably are not too useful for
>>>  general problems.
>>
>> Please don't removed those routines, they are essential for electronic
>> structure calculations, as they are very robust, fast and doesn't
>> require any memory. It just works for many problems.
>
> These are pretty straightforward methods, so maybe they can stay.  But the
> docstrings should clearly say that these are intended only for specialized
> problems.
>
> I wonder if electronic structure (DFT) calculations are a bit of a special
> case. Ie. are the eigenvalues of the Jacobian of the Kohn-Sham functional F[n]
> all negative (or less than 1)?
>
> [clip]
>>> * One could add more algorithms suitable for large-scale problems; for
>>>  example some limited-memory Broyden methods (eg. [1])
>>
>> I think in the references in my thesis, the broyden3 is sometimes called
>> the limited-memory Broyden method. However, I checked the equations in
>> [1] in your email and they need to do a singular-value-decomposition, so
>> it seems there is yet another approach to this. So yes.
>
> It's a part of a limited-memory method, but not the whole story, because the
> present implementation does not actually throw any data away.  So, it's
> an "unlimited memory" Broyden method :)
>
> But it would be very easy (= 3 lines of code) to make it a simple
> limited-memory method, just by throwing away iterates older than some M.  The
> singular value reduction can be implemented later. There are probably many
> different schemes proposed in the literature how the reduction can be done...
>
> [clip]
>> Let me know what you think about my comments. Basically, scipy should
>> have as many (working and well tested) algorithms as possible, with the
>> same interface, so that one can change the algorithms and see which
>> works for the particular problem.
>
> Yes, but this must be balanced against maintenance + testing costs.  I want to
> be sure that the code I contribute works, and I wouldn't like to spend too much
> time working on unproven methods. So I'd rather include only methods for which
> there is reason to believe that they work well.

Yes, the only thing that I really care is that the methods that are
known to work well for me are there. And I am willing to maintain
them.

>
>> I am glad that you are also using those methods, so that we can work
>> on this together.
>
> It'd be very useful if you could test this code, including the line search,
> on a real large-scale problem. At present, I'm not sure if it makes too many
> iterations compared to its payoff.
>
> Also, we might need to implement a reverse communication interface,
> something on the lines of
>
>    iterator = Solver(x0)
>    for x in iterator:
>        iterator.feed(func(x))
>        if iterator.converged(): break
>    else:
>        raise RuntimeError("Didn't converge")
>
> For expensive self-consistent iterations, this often feels the natural way
> to use the nonlinear solver.

Ok, I'll test it. So far I only have a one atomic solver, so that's
not a large scale problem, but if I put for example 20000 points in
the radial mesh, then I can test the nonlin methods pretty well (e.g.
Jacobian being 20000x20000 if we stored it fully, e.g. something like
1.5GB).

Right now I am working on a 2D solver and here it will show clearly
which method works the best. You can follow my progress here:

http://spilka.math.unr.edu/projects/hermes2d/wiki/Projects/SchroedingerEquation

So since you'll be busy in the next couple weeks, I think I'll know
soon which methods I need and which not. Thanks for all the comments
above -- I'll think about it and probably ask you soon to help me
implement or at least test some of those methods. :)

Thanks,
Ondrej


From josef.pktd at gmail.com  Wed Dec 17 19:35:16 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 17 Dec 2008 19:35:16 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <gibqut$gc8$1@ger.gmane.org>
References: <20081204102536.5faad9d5.rbastian@free.fr>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gi18qv$nt4$1@ger.gmane.org>
	<20081213212143.GJ19210@phare.normalesup.org>
	<gi19al$rca$1@ger.gmane.org>
	<20081213212743.GK19210@phare.normalesup.org>
	<1cd32cbb0812131502s9838a46xae4688dae0cf9db5@mail.gmail.com>
	<gibqut$gc8$1@ger.gmane.org>
Message-ID: <1cd32cbb0812171635y4587edcap301e88a627cc8144@mail.gmail.com>

On Wed, Dec 17, 2008 at 4:29 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Sat, 13 Dec 2008 18:02:50 -0500, josef.pktd wrote:
>> In both numpy  and scipy chm files I get: The page cannot be displayed
>> The page you are looking for is currently unavailable. The Web site
>> might be experiencing technical difficulties, or you may need to adjust
>> your browser settings.
>>
>> contents,search, index work correctly, but the page content is not
>> found.
>
> Interesting. From Gareth Elston's mail it sounds like this is some
> internet explorer protection settings issue. The files work fine with
> Linux-based readers.
>
> Did you save the file first on disk, or opened it directly from the
> browser? Looking at http://support.microsoft.com/kb/902225/ this might be
> an issue...
>
>> I decompiled and recompiled in with the html help workshop and it looks
>> pretty good. Search and Index are very fast and helpful.
>
> I'll probably need to check the build process
>
> --
> Pauli Virtanen
>

Everything works fine after following the instructions to unblock the
security lock. Thanks again.

I tried again saving the file with firefox and IE6. It is blocked, but
both methods on the microsoft web page work:
Method 1

   1. Double-click the .chm file.
   2. In the Open File-Security Warning dialog box, click to clear the
Always ask before opening this file check box.
   3. Click Open.

Method 2

   1. Right-click the CHM file, and then click Properties.
   2. Click Unblock.
   3. Double-click the .chm file to open the file.

>From the description of the security issue, you won't be able to do
something in the build process, but a note on the download page about
this issue would be helpful and avoid questions and surprises like
mine. An alternative would be to put it inside a zip file. I never had
problems with chm files that I downloaded as part of a package, e.g.
python or R packages, so I guess an archive should work.

( I still prefer the smaller font size of the standard help files such
as the one included with a python release. It' easier for me to read
on my medium sized notebook.)

Josef


From tom.grydeland at gmail.com  Thu Dec 18 02:36:17 2008
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Thu, 18 Dec 2008 08:36:17 +0100
Subject: [SciPy-dev] r5268
In-Reply-To: <gibri8$gc8$2@ger.gmane.org>
References: <1227987645.8264.200.camel@idol> <ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
	<3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
	<85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>
	<e86a5fd00812081201p34bc937ava0b5c2a76af7abca@mail.gmail.com>
	<85b5c3130812150719l2b3af06credba9a92c15e145e@mail.gmail.com>
	<gibpkh$7pe$2@ger.gmane.org>
	<c7009a550812171328o2b1425dch1d31a972cbc828ab@mail.gmail.com>
	<gibri8$gc8$2@ger.gmane.org>
Message-ID: <c21c0b2f0812172336v78d1baf0x1e82c6f73dd52fda@mail.gmail.com>

On Wed, Dec 17, 2008 at 10:39 PM, Pauli Virtanen <pav at iki.fi> wrote:

> 1D spline interpolation has never worked for complex-valued data, due to
> _fitpack.bisplev being written in C, [...]

Could this be addressed by calling bisplev twice, once for the real
part and once for the imaginary part?

> Pauli Virtanen

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From dpeterson at enthought.com  Thu Dec 18 04:14:44 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Thu, 18 Dec 2008 03:14:44 -0600
Subject: [SciPy-dev] ANNOUNCE: EPD Py25 v4.1.30101_beta2 available for
	testing
Message-ID: <494A1484.5090705@enthought.com>

Hello,

The Enthought Python Distribution's (EPD) early access program website
is now hosting the beta 2 build of the upcoming EPD Py25 v4.1.301
release.  We would very much appreciate your assistance in making EPD as
stable and reliable as possible!  Please join us in our efforts by
downloading an installer for Windows, Mac OS X, or RedHat EL versions 3,
4, and 5 from the following website:

	http://www.enthought.com/products/epdearlyaccess.php

The release notes for the beta2 build are available here:
	https://svn.enthought.com/epd/wiki/Py25/4.1.301/Beta2


Please provide any comments, concerns, or bug reports via the EPD Trac
instance at https://svn.enthought.com/epd or via e-mail to
epd-support at enthought.com.


-- Dave





About EPD
---------
The Enthought Python Distribution (EPD) is a "kitchen-sink-included"
distribution of the Python? Programming Language, including over 60
additional tools and libraries. The EPD bundle includes NumPy, SciPy,
IPython, 2D and 3D visualization, database adapters, and a lot of
other tools right out of the box.

	http://www.enthought.com/products/epd.php

It is currently available as a single-click installer for Windows XP
(x86), Mac OS X (a universal binary for OS X 10.4 and above), and
RedHat 3, 4, and 5 (x86 and amd64).

EPD is free for academic use.  An annual subscription and installation
support are available for individual commercial use.  Various workgroup,
departmental, and enterprise level subscription options with support and
training are also available.  Contact us for more information!






From timmichelsen at gmx-topmail.de  Thu Dec 18 08:28:34 2008
From: timmichelsen at gmx-topmail.de (timmichelsen at gmx-topmail.de)
Date: Thu, 18 Dec 2008 14:28:34 +0100
Subject: [SciPy-dev] suggestion for Numpy documentation editor
In-Reply-To: <200812181357.46991.faltet@pytables.com>
References: <20081218110145.81560@gmx.net>
	<200812181357.46991.faltet@pytables.com>
Message-ID: <20081218132834.265460@gmx.net>

Hello.
I would like to ask whether the Numpy documentation editor
could be implemented with a password reminder/resend option.

Also, the follwoing links are dead:
http://docs.scipy.org/numpy/Front%20Page/

=> 
http://docs.scipy.org/doc/numpy.doc.reference.jargon
http://docs.scipy.org/doc/numpy.doc.reference.glossary

Thanks and regards,
Timmie
-- 
Psssst! Schon vom neuen GMX MultiMessenger geh?rt? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger


From tom.grydeland at gmail.com  Thu Dec 18 08:52:47 2008
From: tom.grydeland at gmail.com (Tom Grydeland)
Date: Thu, 18 Dec 2008 14:52:47 +0100
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
	it))
In-Reply-To: <wl8tz94hwz0.fsf@glup.physics.ucf.edu>
References: <mailman.3246.1229414019.2878.scipy-dev@scipy.org>
	<wl8tz94hwz0.fsf@glup.physics.ucf.edu>
Message-ID: <c21c0b2f0812180552n7ff151far9021916008c4f806@mail.gmail.com>

On Tue, Dec 16, 2008 at 6:05 PM,  <jh at physics.ucf.edu> wrote:

> I added the helper routines and defined the time and frequency domains
> near the top (and took out similar text near the bottom), fixed some
> commas, etc.

A comment was dropped on the `ifft` routine re the semantics of the
zero padding done when `n > len(a)`.  I wrote a note in the
documentation to address the issue, but at the same time I thought
that perhaps we could add a helper routine (I suggest the name
`ifftpad`) to implement the padding that the commenter would like to
see, e.g.,

ifftpad(a, m) -> _r[a[:n/2], zeros((m-n)/2), a[n/2:], zeros((m-n)/2) ]

where `n == len(a)` and care must be taken to get even and odd `m` and
`n` correct, etc.
It might also be capable of truncating only high-frequency components
if `m < len(a)`, e.g.

ifftpad(a, m) -> _r[ a[:m/2], a[n/2:(n+m)/2 ]

in this way, the resampling approach suggested for `irfft` would work
also for complex ffts, thus:

ifft(ifftpad(fft(a), m))


> --jh--

-- 
Tom Grydeland
  <Tom.Grydeland@(gmail.com)>


From stefan at sun.ac.za  Thu Dec 18 09:37:37 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 18 Dec 2008 16:37:37 +0200
Subject: [SciPy-dev] FFT docstrings (was: Scipy Tutorial (and updating
	it))
In-Reply-To: <c21c0b2f0812180552n7ff151far9021916008c4f806@mail.gmail.com>
References: <mailman.3246.1229414019.2878.scipy-dev@scipy.org>
	<wl8tz94hwz0.fsf@glup.physics.ucf.edu>
	<c21c0b2f0812180552n7ff151far9021916008c4f806@mail.gmail.com>
Message-ID: <9457e7c80812180637o54c1c42co8a2d6802ddec3d4d@mail.gmail.com>

Hi Tom

2008/12/18 Tom Grydeland <tom.grydeland at gmail.com>:
> A comment was dropped on the `ifft` routine re the semantics of the
> zero padding done when `n > len(a)`.  I wrote a note in the
> documentation to address the issue, but at the same time I thought
> that perhaps we could add a helper routine (I suggest the name
> `ifftpad`) to implement the padding that the commenter would like to
> see, e.g.,
>
> ifftpad(a, m) -> _r[a[:n/2], zeros((m-n)/2), a[n/2:], zeros((m-n)/2) ]

I've been wanting this for a while, so I'm all in favour.

Cheers
St?fan


From thouis at broad.mit.edu  Thu Dec 18 12:20:35 2008
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Thu, 18 Dec 2008 12:20:35 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812171325h429068ebg77be51020d147ef7@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<6c17e6f50812161241k61344a0bx25c3bb8147e5b961@mail.gmail.com>
	<c7009a550812161340r777c3470w23d3842c7318aff6@mail.gmail.com>
	<6c17e6f50812161850y4f6042fva85680c7df597b3a@mail.gmail.com>
	<c7009a550812171325h429068ebg77be51020d147ef7@mail.gmail.com>
Message-ID: <6c17e6f50812180920r453d456ue23618c3d21efcda@mail.gmail.com>

On Wed, Dec 17, 2008 at 16:25, Jarrod Millman <millman at berkeley.edu> wrote:
> On Tue, Dec 16, 2008 at 6:50 PM, Thouis (Ray) Jones
> <thouis at broad.mit.edu> wrote:
>> Here's a slightly better version (clearer, about 10% faster).
>
> It looks good to me.  Did you take a look at whether medfilter2 was
> from NR as well?  If not, I would appreciate it if you would take a
> quick look.  It would also be nice if you would be willing to rewrite
> the medfilter2s as a single macro which expands for the
> float/double/byte cases like you did for the quick select.
>
> Could you also provide some tests for medfilt2 in test_signaltools.py:
>  trunk/scipy/signal/tests/test_signaltools.py

The medfilter2 is not from NR (NR doesn't include median filtering, as
far as I can tell).

I've attached the patch against the latest SVN.  It includes the following:
* new quickselect (from scratch) in medianfilter.c
* new macros for {f,d,b}_medfilt2 in medianfilter.c
* modified test for medfilt and medfilt2 (larger array, non-square filter).


The N-d order filter (called by medfilt) works by filling a buffer,
then calling qsort, then extracting the desired index.  It could
easily take advantage of a qselect() function based on the one in
medianfilter.c (except without .  I'll file an enhancement request, as
I doubt you want such a change in 0.7.

Also, medfilt2d should fall back to medfilt (nD version) when handed a
type not in its special cases (float, double, unsigned char).

Ray Jones
-------------- next part --------------
A non-text attachment was scrubbed...
Name: medfilt.patch
Type: application/octet-stream
Size: 18411 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081218/8d605e5e/attachment.obj>

From pav at iki.fi  Thu Dec 18 14:41:30 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 18 Dec 2008 19:41:30 +0000 (UTC)
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
References: <20081204102536.5faad9d5.rbastian@free.fr>
	<4937F0C6.8000202@american.edu> <gh8s5a$b0n$1@ger.gmane.org>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gi18qv$nt4$1@ger.gmane.org>
	<20081213212143.GJ19210@phare.normalesup.org>
	<gi19al$rca$1@ger.gmane.org>
	<20081213212743.GK19210@phare.normalesup.org>
	<1cd32cbb0812131502s9838a46xae4688dae0cf9db5@mail.gmail.com>
	<gibqut$gc8$1@ger.gmane.org>
	<1cd32cbb0812171635y4587edcap301e88a627cc8144@mail.gmail.com>
Message-ID: <gie91a$pae$1@ger.gmane.org>

Wed, 17 Dec 2008 19:35:16 -0500, josef.pktd wrote:
[clip]
> Everything works fine after following the instructions to unblock the
> security lock. Thanks again.
[clip]
> From the description of the security issue, you won't be able to do
> something in the build process, but a note on the download page about
> this issue would be helpful and avoid questions and surprises like mine.
> An alternative would be to put it inside a zip file. I never had
> problems with chm files that I downloaded as part of a package, e.g.
> python or R packages, so I guess an archive should work.

I zipped the files. Can you try if this helps? If it does, I'll have to 
say that this is one interesting security feature.

If it doesn't help, maybe we'll just point to the knowledge base archive, 
and ask on the Sphinx ML for help.

-- 
Pauli Virtanen



From josef.pktd at gmail.com  Thu Dec 18 14:53:02 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Dec 2008 14:53:02 -0500
Subject: [SciPy-dev] Scipy Tutorial (and updating it)
In-Reply-To: <gie91a$pae$1@ger.gmane.org>
References: <20081204102536.5faad9d5.rbastian@free.fr>
	<1cd32cbb0812040744j4bf4ab52h1b69c5367e6a2315@mail.gmail.com>
	<gi18qv$nt4$1@ger.gmane.org>
	<20081213212143.GJ19210@phare.normalesup.org>
	<gi19al$rca$1@ger.gmane.org>
	<20081213212743.GK19210@phare.normalesup.org>
	<1cd32cbb0812131502s9838a46xae4688dae0cf9db5@mail.gmail.com>
	<gibqut$gc8$1@ger.gmane.org>
	<1cd32cbb0812171635y4587edcap301e88a627cc8144@mail.gmail.com>
	<gie91a$pae$1@ger.gmane.org>
Message-ID: <1cd32cbb0812181153w4d0abfdaqbbf2481cb6325ebe@mail.gmail.com>

On Thu, Dec 18, 2008 at 2:41 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Wed, 17 Dec 2008 19:35:16 -0500, josef.pktd wrote:
> [clip]
>> Everything works fine after following the instructions to unblock the
>> security lock. Thanks again.
> [clip]
>> From the description of the security issue, you won't be able to do
>> something in the build process, but a note on the download page about
>> this issue would be helpful and avoid questions and surprises like mine.
>> An alternative would be to put it inside a zip file. I never had
>> problems with chm files that I downloaded as part of a package, e.g.
>> python or R packages, so I guess an archive should work.
>
> I zipped the files. Can you try if this helps? If it does, I'll have to
> say that this is one interesting security feature.
>
> If it doesn't help, maybe we'll just point to the knowledge base archive,
> and ask on the Sphinx ML for help.
>

It works, I tried both, numpy and scipy .chm: unzip, doubleclick in
explorer and it opens without problems and not protected.

Thanks,

Josef


From pav at iki.fi  Thu Dec 18 15:33:54 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 18 Dec 2008 20:33:54 +0000 (UTC)
Subject: [SciPy-dev] r5268
References: <1227987645.8264.200.camel@idol> <ghjo0n$r9j$1@ger.gmane.org>
	<85b5c3130812081042j56c9abf7l6fcf7f23d3745b8f@mail.gmail.com>
	<3d375d730812081138i1a0a255dqad8954b30d264f2a@mail.gmail.com>
	<85b5c3130812081151k1849c75epdd0a0f0c06ecb2ca@mail.gmail.com>
	<e86a5fd00812081201p34bc937ava0b5c2a76af7abca@mail.gmail.com>
	<85b5c3130812150719l2b3af06credba9a92c15e145e@mail.gmail.com>
	<gibpkh$7pe$2@ger.gmane.org>
	<c7009a550812171328o2b1425dch1d31a972cbc828ab@mail.gmail.com>
	<gibri8$gc8$2@ger.gmane.org>
	<c21c0b2f0812172336v78d1baf0x1e82c6f73dd52fda@mail.gmail.com>
Message-ID: <giec3i$pae$2@ger.gmane.org>

Thu, 18 Dec 2008 08:36:17 +0100, Tom Grydeland wrote:

> On Wed, Dec 17, 2008 at 10:39 PM, Pauli Virtanen <pav at iki.fi> wrote:
> 
>> 1D spline interpolation has never worked for complex-valued data, due
>> to _fitpack.bisplev being written in C, [...]
> 
> Could this be addressed by calling bisplev twice, once for the real part
> and once for the imaginary part?

Excellent point, it's linear in the data, so we can do this.

Committed as r5275, with tests. This is a rather trivial change.

-- 
Pauli Virtanen



From pav at iki.fi  Thu Dec 18 15:43:10 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 18 Dec 2008 20:43:10 +0000 (UTC)
Subject: [SciPy-dev] hidden scipy.optimize.nnls
References: <1cd32cbb0812162114m1cb5b6c6i5e76e97a8065a746@mail.gmail.com>
	<c7009a550812171128x6271f772wc0e312a38712179@mail.gmail.com>
Message-ID: <giecku$pae$3@ger.gmane.org>

Wed, 17 Dec 2008 11:28:04 -0800, Jarrod Millman wrote:

> On Tue, Dec 16, 2008 at 9:14 PM,  <josef.pktd at gmail.com> wrote:
>> Is scipy.optimize.nnls hidden on purpose?
> 
> It was added a few months ago by Uwe Schmitt.  I don't think it was
> hidden on purpose.
> 
> Please add:
>   from nnls import nnls
> to optimize.__init__.py and add a entry to info.py.

This made nnls's tests to look for the function in the wrong place. Fixed 
in r5276.

-- 
Pauli Virtanen



From josef.pktd at gmail.com  Thu Dec 18 15:54:41 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 18 Dec 2008 15:54:41 -0500
Subject: [SciPy-dev] hidden scipy.optimize.nnls
In-Reply-To: <giecku$pae$3@ger.gmane.org>
References: <1cd32cbb0812162114m1cb5b6c6i5e76e97a8065a746@mail.gmail.com>
	<c7009a550812171128x6271f772wc0e312a38712179@mail.gmail.com>
	<giecku$pae$3@ger.gmane.org>
Message-ID: <1cd32cbb0812181254g2a7ab69eu7fe9ada557669335@mail.gmail.com>

On Thu, Dec 18, 2008 at 3:43 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Wed, 17 Dec 2008 11:28:04 -0800, Jarrod Millman wrote:
>
>> On Tue, Dec 16, 2008 at 9:14 PM,  <josef.pktd at gmail.com> wrote:
>>> Is scipy.optimize.nnls hidden on purpose?
>>
>> It was added a few months ago by Uwe Schmitt.  I don't think it was
>> hidden on purpose.
>>
>> Please add:
>>   from nnls import nnls
>> to optimize.__init__.py and add a entry to info.py.
>
> This made nnls's tests to look for the function in the wrong place. Fixed
> in r5276.
>

Sorry for committing without running the tests. I guess even for
trivial changes, running tests is necessary.

Josef


From bradford.n.cross at gmail.com  Thu Dec 18 21:27:12 2008
From: bradford.n.cross at gmail.com (Bradford Cross)
Date: Thu, 18 Dec 2008 18:27:12 -0800
Subject: [SciPy-dev] new incremental statistics project
Message-ID: <ea7d6a710812181827t34572516j981df31fbadb3f07@mail.gmail.com>

This is a new project I just released.

I know it is C#, but some of the design and idioms would be nice in
numpy/scipy for working with discrete event simulators, time series, and
event stream processing.

http://code.google.com/p/incremental-statistics/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081218/f776750a/attachment.html>

From sturla at molden.no  Fri Dec 19 10:59:45 2008
From: sturla at molden.no (Sturla Molden)
Date: Fri, 19 Dec 2008 16:59:45 +0100
Subject: [SciPy-dev] fast gaussian filtering
Message-ID: <494BC4F1.9010304@molden.no>


Gaussian filtering is a very useful method for smoothing signals and 
images, as well as for kernel density estimation. I have previously used 
it to estimate the firing rates of neurons.

I've noticed that SciPy implements Gaussian filtering (in the ndimage 
package) using convolution. It is implemented by loops in Python, not 
even using a vectorized numpy.convolve or an FFT. For the sake of speed, 
SciPy is using the worst thinkable implementation.

To make it fast:

Gaussian filtering can be implemented using recursion (Young & Vliet, 
1995; Signal Processing, 44: 139-151). The recursive approximation is 
very accurate and does not introduce ringing. It is anti-causal 
(forward-backward) and has zero phase response. The filtering can be 
done in C using scipy's signal.lfilter function.

I found a solution in plain C on the Matlab central. But there is really 
no benefit from doing this in C as lfilter, flipud and fliplr does all 
the hard work. An efficient Matlab version could have been constructed 
similarly, without resorting to C.

http://www.mathworks.com/matlabcentral/fileexchange/5087
http://www.ph.tn.tudelft.nl/~lucas/publications/papersLJvV.html

Note that Young & Vliet (1995) also provides recursive solutions for the 
first, second and third derivatives. They are equally easy to implement 
using lfilter.


Regards,
Sturla Molden




from numpy import array, zeros, ones, flipud, fliplr
from scipy.signal import lfilter
from math import sqrt

def __gausscoeff(s):
     if s < .5: raise ValueError, \
         'Sigma for Gaussian filter must be >0.5 samples'
     q = 0.98711*s - 0.96330 if s > 0.5 else 3.97156 \
         - 4.14554*sqrt(1.0 - 0.26891*s)
     b = zeros(4)
     b[0] = 1.5785 + 2.44413*q + 1.4281*q**2 + 0.422205*q**3
     b[1] = 2.44413*q + 2.85619*q**2 + 1.26661*q**3
     b[2] = -(1.4281*q**2 + 1.26661*q**3)
     b[3] = 0.422205*q**3
     B = 1.0 - ((b[1] + b[2] + b[3])/b[0])
     # convert to a format compatible with lfilter's
     # difference equation
     B = array([B])
     A = ones(4)
     A[1:] = -b[1:]/b[0]
     return B,A

def Gaussian1D(signal, sigma, padding=0):
     n = signal.shape[0]
     tmp = zeros(n + padding)
     if tmp.shape[0] < 4: raise ValueError, \
         'Signal and padding too short'
     tmp[:n] = signal
     B,A = __gausscoeff(sigma)
     tmp = lfilter(B, A, tmp)
     tmp = tmp[::-1]
     tmp = lfilter(B, A, tmp)
     tmp = tmp[::-1]
     return tmp[:n]

def Gaussian2D(image, sigma, padding=0):
     n,m = image.shape[0],image.shape[1]
     tmp = zeros((n + padding, m + padding))
     if tmp.shape[0] < 4: raise ValueError, \
         'Image and padding too small'
     if tmp.shape[1] < 4: raise ValueError, \
         'Image and padding too small'
     B,A = __gausscoeff(sigma)
     tmp[:n,:m] = image
     tmp = lfilter(B, A, tmp, axis=0)
     tmp = flipud(tmp)
     tmp = lfilter(B, A, tmp, axis=0)
     tmp = flipud(tmp)
     tmp = lfilter(B, A, tmp, axis=1)
     tmp = fliplr(tmp)
     tmp = lfilter(B, A, tmp, axis=1)
     tmp = fliplr(tmp)
     return tmp[:n,:m]











From zachary.pincus at yale.edu  Fri Dec 19 12:31:41 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Fri, 19 Dec 2008 12:31:41 -0500
Subject: [SciPy-dev] fast gaussian filtering
In-Reply-To: <494BC4F1.9010304@molden.no>
References: <494BC4F1.9010304@molden.no>
Message-ID: <1AECFB3C-5088-4F86-B5F5-9C25C5FCB42F@yale.edu>

Hi,

> I've noticed that SciPy implements Gaussian filtering (in the ndimage
> package) using convolution. It is implemented by loops in Python, not
> even using a vectorized numpy.convolve or an FFT. For the sake of  
> speed,
> SciPy is using the worst thinkable implementation.

What makes you think this?

A brief look at the implementation in scipy/ndimage/filters.py shows  
that scipy.ndimage.gaussian_filter, after doing some set-up, defers to  
gaussian_filter1d to do the filtering on each dimension (as Gaussian  
filtering is separable). Then gaussian_filter1d uses for-loops to do  
some further set-up, like calculating the filter kernel, and then  
defers to correlate1d for the actual filtering. This function does  
some error-checking, and then calls _nd_image.correlate1d. The  
_nd_image library is a C extension module: _nd_image.correlate1d  
corresponds to Py_Correlate1D, which is defined in scipy/ndimage/src/ 
nd_image.c, and which calls to NI_Correlate1D to do the lifting.  
NI_Correlate1D, in scipy/ndimage/src/ni_filters.c, implements the  
filtering in a straightforward loop, but also takes advantage where it  
can of filter-kernel symmetry or anti-symmetry.

Aside from the several thin layers of error-checking and set-up, the  
code does the filtering in C in about the fastest way possible that  
doesn't use recursion or FFT. (And even then, I don't know under what  
circumstances the direct-convolution method might still win --  
probably with small kernels.)


> To make it fast:
>
> Gaussian filtering can be implemented using recursion (Young & Vliet,
> 1995; Signal Processing, 44: 139-151). The recursive approximation is
> very accurate and does not introduce ringing. It is anti-causal
> (forward-backward) and has zero phase response. The filtering can be
> done in C using scipy's signal.lfilter function.

Your implementation looks pretty straightforward. Here are some  
timings (sm_gaussian is your Gaussian2D, nd_gaussian is  
scipy.ndimage.gaussian_filter):

In: img = numpy.random.rand(100, 100)

In: timeit sm_gaussian(img, 2)
1000 loops, best of 3: 1.54 ms per loop

In: timeit nd_gaussian(img, 2)
1000 loops, best of 3: 1.26 ms per loop

In: timeit sm_gaussian(img, 20)
1000 loops, best of 3: 1.55 ms per loop

In: timeit nd_gaussian(img, 20)
100 loops, best of 3: 7.41 ms per loop

In: img = numpy.random.rand(1000, 1000)

In: timeit sm_gaussian(img, 2)
10 loops, best of 3: 168 ms per loop

In: timeit nd_gaussian(img, 2)
10 loops, best of 3: 92.8 ms per loop

In: timeit sm_gaussian(img, 20)
10 loops, best of 3: 167 ms per loop

In: timeit nd_gaussian(img, 20)
10 loops, best of 3: 657 ms per loop

As expected, the recursive approach is a definite win with larger  
kernels, but with small kernels the constant factors dominate and the  
straightforward implementation is narrowly faster. At the very least,  
you should post this implementation on the cookbook page...

Zach






From sturla at molden.no  Fri Dec 19 12:45:59 2008
From: sturla at molden.no (Sturla Molden)
Date: Fri, 19 Dec 2008 18:45:59 +0100
Subject: [SciPy-dev] fast gaussian filtering
In-Reply-To: <1AECFB3C-5088-4F86-B5F5-9C25C5FCB42F@yale.edu>
References: <494BC4F1.9010304@molden.no>
	<1AECFB3C-5088-4F86-B5F5-9C25C5FCB42F@yale.edu>
Message-ID: <494BDDD7.9080602@molden.no>

On 12/19/2008 6:31 PM, Zachary Pincus wrote:

> A brief look at the implementation in scipy/ndimage/filters.py shows  
> that scipy.ndimage.gaussian_filter, after doing some set-up, defers to  
> gaussian_filter1d to do the filtering on each dimension (as Gaussian  
> filtering is separable). Then gaussian_filter1d uses for-loops to do  
> some further set-up, like calculating the filter kernel, and then  
> defers to correlate1d for the actual filtering.  This function does
> some error-checking, and then calls _nd_image.correlate1d. 

Ok, sorry, I did not read the code thoroughly then. :)


> Aside from the several thin layers of error-checking and set-up, the  
> code does the filtering in C in about the fastest way possible that  
> doesn't use recursion or FFT. (And even then, I don't know under what  
> circumstances the direct-convolution method might still win --  
> probably with small kernels.)

Yes, for kernels smaller than 10 or so it will be faster. But a Gaussian 
is actually not of finite size. So a finite sized Gaussian is actually a 
misnomer; it should be a truncated Gaussian. The recursion does not 
truncate the Gaussian kernel (but at some point the impulse response 
will be so small that it falls into the range of rounding error).


Sturla Molden





From zachary.pincus at yale.edu  Fri Dec 19 13:51:30 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Fri, 19 Dec 2008 13:51:30 -0500
Subject: [SciPy-dev] fast gaussian filtering
In-Reply-To: <494BDDD7.9080602@molden.no>
References: <494BC4F1.9010304@molden.no>
	<1AECFB3C-5088-4F86-B5F5-9C25C5FCB42F@yale.edu>
	<494BDDD7.9080602@molden.no>
Message-ID: <FED05A63-7411-4E25-B790-FCB057B3BB42@yale.edu>

> Yes, for kernels smaller than 10 or so it will be faster. But a  
> Gaussian
> is actually not of finite size. So a finite sized Gaussian is  
> actually a
> misnomer; it should be a truncated Gaussian. The recursion does not
> truncate the Gaussian kernel (but at some point the impulse response
> will be so small that it falls into the range of rounding error).

Aah, cool -- that's good to know. I'll definitely keep your  
implementation handy. It would definitely be great up on the cookbook!

Zach


From millman at berkeley.edu  Fri Dec 19 15:46:08 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Fri, 19 Dec 2008 12:46:08 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <6c17e6f50812180920r453d456ue23618c3d21efcda@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<6c17e6f50812161241k61344a0bx25c3bb8147e5b961@mail.gmail.com>
	<c7009a550812161340r777c3470w23d3842c7318aff6@mail.gmail.com>
	<6c17e6f50812161850y4f6042fva85680c7df597b3a@mail.gmail.com>
	<c7009a550812171325h429068ebg77be51020d147ef7@mail.gmail.com>
	<6c17e6f50812180920r453d456ue23618c3d21efcda@mail.gmail.com>
Message-ID: <c7009a550812191246s3a644522g2e889da4c13f69a9@mail.gmail.com>

On Thu, Dec 18, 2008 at 9:20 AM, Thouis (Ray) Jones
<thouis at broad.mit.edu> wrote:
> The medfilter2 is not from NR (NR doesn't include median filtering, as
> far as I can tell).
>
> I've attached the patch against the latest SVN.  It includes the following:
> * new quickselect (from scratch) in medianfilter.c
> * new macros for {f,d,b}_medfilt2 in medianfilter.c
> * modified test for medfilt and medfilt2 (larger array, non-square filter).

Thank you very much for looking into this and supplying a patch.  I
went ahead and applied the patch:
http://projects.scipy.org/scipy/scipy/changeset/5279

Thanks for the enhancement tickets as well:
  http://projects.scipy.org/scipy/scipy/ticket/818
  http://projects.scipy.org/scipy/scipy/ticket/819

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From stefan at sun.ac.za  Fri Dec 19 17:41:20 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 20 Dec 2008 00:41:20 +0200
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
Message-ID: <9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>

2008/12/16 Jarrod Millman <millman at berkeley.edu>:
> 4.  3 functions in scipy/stats/stats.py.
>
> Partially resolved.  One down (Josef rewrote ks_2samp), two left
> (ttest_ind and ttest_rel).

>From the NR license:

"""Copyright does not protect ideas, but only the expression of those
ideas in a particular form.  In the case of a computer program, the
ideas consist of the program's methodology and algorithm, including
the necessary sequence of steps adopted by the programmer.  The
expression of those ideas is the program source code (particularly any
arbitrary or stylistic choices embodied in it), its derived object
code, and any other derivative work.

If you analyze the ideas contained in a program, and then express
those ideas in your own completely different implementation, then that
new program implementation belongs to you.  That is what we have done
for those programs in this book that are not entirely of our own
devising.  When programs in this book are said to be "based" on
programs published in copyright sources, we mean that the ideas are
the same.  The expression of these ideas as source code is our own.
We believe that no material in this book infringes on an existing
copyright."""

I am not convinced that their interpretation of copyright law is
correct, but either way I think ttest_ind and ttest_rel are safe
(according to *their* rules).  The code makes use of constructs such
as "where" and broadcasting, that aren't available in C.  The author
therefore took an idea from NR and reimplemented it in a novel way.

Regards
St?fan


From alan at ajackson.org  Fri Dec 19 18:25:48 2008
From: alan at ajackson.org (Alan Jackson)
Date: Fri, 19 Dec 2008 17:25:48 -0600
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>
Message-ID: <20081219172548.68b73939@ajackson.org>

On Sat, 20 Dec 2008 00:41:20 +0200
"St?fan van der Walt" <stefan at sun.ac.za> wrote:

> 2008/12/16 Jarrod Millman <millman at berkeley.edu>:
> > 4.  3 functions in scipy/stats/stats.py.
> >
> > Partially resolved.  One down (Josef rewrote ks_2samp), two left
> > (ttest_ind and ttest_rel).
> 
> >From the NR license:
> 
> """Copyright does not protect ideas, but only the expression of those
> ideas in a particular form.  In the case of a computer program, the
> ideas consist of the program's methodology and algorithm, including
> the necessary sequence of steps adopted by the programmer.  The
> expression of those ideas is the program source code (particularly any
> arbitrary or stylistic choices embodied in it), its derived object
> code, and any other derivative work.
> 
> I am not convinced that their interpretation of copyright law is
> correct, but either way I think ttest_ind and ttest_rel are safe
> (according to *their* rules).  The code makes use of constructs such
> as "where" and broadcasting, that aren't available in C.  The author
> therefore took an idea from NR and reimplemented it in a novel way.

Most of what I know about copyright law is :
- it is complex and changes depending on the object being discussed
- publishers like to claim many more rights than they actually have

For example, for cookbooks, the recipe itself is explicitly not copyrightable,
only the layout in a book and the description. 

There is also a paper I read a few years ago about music copyright showing
examples of publishers trying to claim copyright for Bach sonatas!

I would not be surprised if their interpretation of copyright law for software
is substantially correct. That is probably why people try to patent software
now (one-click anyone?) - that way they get greater (though shorter term)
protection.

-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From aisaac at american.edu  Fri Dec 19 20:18:56 2008
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 19 Dec 2008 20:18:56 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>
Message-ID: <494C4800.8050907@american.edu>

Quoting from http://www.iusmentis.com/copyright/software/protection/

         An implementation of a standard algorithm may
         require very little creativity and if so, is likely
         not protected by copyright. The function of a work
         can only be protected by a patent.

         Even when providing a largely functional
         implementation, creativity can still be found in
         things like function and variable names, code
         optimizations and even the comments accompanying the
         code. Copying such an implementation wholesale then
         still infringes on copyright.

fwiw,
Alan




From wbaxter at gmail.com  Fri Dec 19 20:49:20 2008
From: wbaxter at gmail.com (Bill Baxter)
Date: Sat, 20 Dec 2008 10:49:20 +0900
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <494C4800.8050907@american.edu>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>
	<494C4800.8050907@american.edu>
Message-ID: <e86a5fd00812191749l25844682u89188f76687935f3@mail.gmail.com>

On Sat, Dec 20, 2008 at 10:18 AM, Alan G Isaac <aisaac at american.edu> wrote:
> Quoting from http://www.iusmentis.com/copyright/software/protection/
>
>         An implementation of a standard algorithm may
>         require very little creativity and if so, is likely
>         not protected by copyright. The function of a work
>         can only be protected by a patent.
>
>         Even when providing a largely functional
>         implementation, creativity can still be found in
>         things like function and variable names, code
>         optimizations and even the comments accompanying the
>         code. Copying such an implementation wholesale then
>         still infringes on copyright.
>

The problem, though, is that terms like "requires little creativity"
are subjective.  You may think the code required no creativity, but
the publisher of NR may disagree.  And then it takes a court to decide
who is right.   And at that point even if SciPy is vindicated, the
SciPy community still loses because of all the lost time an money that
has to go into such a fight.  It's easier to just make sure you steer
well clear of any potential infringement from the beginning.

IANAL, but I think that especially once the issue is raised in a
public forum like this -- now you've got a record that the developers
were aware of a potential problem.  If nothing is done about it, it
might be regarded as "willful infringement".  But if an honest attempt
is made to replace the code, then you've at least got a public record
that you were attempting to do your "due dilligence" to eliminate
infringements.

So, silly as it may seem, I think that Jarrod's plan of action here is
correct.   The NR guys may be wrong about copyright law, but being
right is not a prerequisite for filing a lawsuit.  Merely thinking you
are is generally enough.  And sometimes not even that...

--bb


From sturla at molden.no  Fri Dec 19 22:40:36 2008
From: sturla at molden.no (Sturla Molden)
Date: Sat, 20 Dec 2008 04:40:36 +0100 (CET)
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>
Message-ID: <cb12d88bd56b8d136cbcf5d7026815be.squirrel@webmail.uio.no>

> 2008/12/16 Jarrod Millman <millman at berkeley.edu>:


> I am not convinced that their interpretation of copyright law is
> correct, but either way I think ttest_ind and ttest_rel are safe
> (according to *their* rules).  The code makes use of constructs such
> as "where" and broadcasting, that aren't available in C.  The author
> therefore took an idea from NR and reimplemented it in a novel way.

I haven't looked at the actual code in question here. Just beware that
there are Fortran 90 code on the NR CD-ROM and web site. In 1996 the "NR
in Fortran 90" book was published. Their Fortran 90 code does make use of
where constructs, broadcasting and array slices.


Sturla Molden



From josef.pktd at gmail.com  Sat Dec 20 00:12:57 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 20 Dec 2008 00:12:57 -0500
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <e86a5fd00812191749l25844682u89188f76687935f3@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>
	<494C4800.8050907@american.edu>
	<e86a5fd00812191749l25844682u89188f76687935f3@mail.gmail.com>
Message-ID: <1cd32cbb0812192112v2ad6a847qd200fc0dd9fafb89@mail.gmail.com>

On Fri, Dec 19, 2008 at 8:49 PM, Bill Baxter <wbaxter at gmail.com> wrote:
> On Sat, Dec 20, 2008 at 10:18 AM, Alan G Isaac <aisaac at american.edu> wrote:
>> Quoting from http://www.iusmentis.com/copyright/software/protection/
>>
>>         An implementation of a standard algorithm may
>>         require very little creativity and if so, is likely
>>         not protected by copyright. The function of a work
>>         can only be protected by a patent.
>>
>>         Even when providing a largely functional
>>         implementation, creativity can still be found in
>>         things like function and variable names, code
>>         optimizations and even the comments accompanying the
>>         code. Copying such an implementation wholesale then
>>         still infringes on copyright.
>>
>
> The problem, though, is that terms like "requires little creativity"
> are subjective.  You may think the code required no creativity, but
> the publisher of NR may disagree.  And then it takes a court to decide
> who is right.   And at that point even if SciPy is vindicated, the
> SciPy community still loses because of all the lost time an money that
> has to go into such a fight.  It's easier to just make sure you steer
> well clear of any potential infringement from the beginning.
>
> IANAL, but I think that especially once the issue is raised in a
> public forum like this -- now you've got a record that the developers
> were aware of a potential problem.  If nothing is done about it, it
> might be regarded as "willful infringement".  But if an honest attempt
> is made to replace the code, then you've at least got a public record
> that you were attempting to do your "due dilligence" to eliminate
> infringements.
>
> So, silly as it may seem, I think that Jarrod's plan of action here is
> correct.   The NR guys may be wrong about copyright law, but being
> right is not a prerequisite for filing a lawsuit.  Merely thinking you
> are is generally enough.  And sometimes not even that...
>
> --bb
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>


In these cases, I don't see any copyrightable creativity. I replaced
the use of betai with the t-distribution on recommendation of Bruce
Southey. The formula for `ttest_ind` is exactly the same as wikipedia
and I guess in any number of textbooks,
http://en.wikipedia.org/wiki/T-test#Independent_two-sample_t-test
section: Unequal sample sizes, equal variance

Variable names are not very creative and the shape and axis handling
is standard numpy style. The handling of the zero div problem seems to
be added later and is not in the current version of Gary Strangmans
code

So I don't see what can be changed here, except for cosmetic changes.

def ttest_ind(a, b, axis=0):
    a, b, axis = _chk2_asarray(a, b, axis)
    x1 = mean(a,axis)
    x2 = mean(b,axis)
    v1 = var(a,axis)
    v2 = var(b,axis)
    n1 = a.shape[axis]
    n2 = b.shape[axis]
    df = n1+n2-2
    svar = ((n1-1)*v1+(n2-1)*v2) / float(df)
    zerodivproblem = svar == 0
    t = (x1-x2)/np.sqrt(svar*(1.0/n1 + 1.0/n2))  # N-D COMPUTATION HERE!!!!!!
    t = np.where(zerodivproblem, 1.0, t)           # replace NaN
t-values with 1.0
    #probs = betai(0.5*df,0.5,float(df)/(df+t*t))
    probs = distributions.t.sf(np.abs(t),df)*2

    if not np.isscalar(t):
        probs = probs.reshape(t.shape)
    if not np.isscalar(probs) and len(probs) == 1:
        probs = probs[0]
    return t, probs

The second test, looks a little bit more complicated in the current
version, for example it defines several variables that are never used.
I am currently rewriting the main formula to follow exactly the
wikipedia formula, which is much easier to read.
For a Monte Carlo with 10000 replication and sample size 500, I get at
maximum difference of the new version to the old version of
3.9968028886505635e-015  which should be close enough. Rejection rates
for alpha = 1%, 5% and 10% are pretty close, so the test is correctly
sized.

This is the new code:

def ttest_rel(a,b,axis=None):
    a, b, axis = _chk2_asarray(a, b, axis)
    if len(a)!=len(b):
        raise ValueError, 'unequal length arrays'
    n = a.shape[axis]
    df = float(n-1)
    d = (a-b).astype('d')

    #denom is just sqrt(var(d)/df), Note: denominator for denom is 1/n * 1/(n-1)
    denom = np.sqrt(np.var(d,axis,ddof=1)/float(n))
    zerodivproblem = denom == 0
    t = np.mean(d, axis) / denom
    t = np.where(zerodivproblem, 1.0, t)    # replace NaN t-values with 1.0
    probs = distributions.t.sf(np.abs(t),df)*2
    if not np.isscalar(t):
        probs = np.reshape(probs, t.shape)
    if not np.isscalar(probs) and len(probs) == 1:
        probs = probs[0]
    return t, probs


As an aside, is there any difference between np.sum() and
np.add.reduce(), or are they equivalent or exactly the same?

Josef


From david at ar.media.kyoto-u.ac.jp  Sat Dec 20 00:45:55 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 20 Dec 2008 14:45:55 +0900
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <e86a5fd00812191749l25844682u89188f76687935f3@mail.gmail.com>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>	<9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>	<494C4800.8050907@american.edu>
	<e86a5fd00812191749l25844682u89188f76687935f3@mail.gmail.com>
Message-ID: <494C8693.6060808@ar.media.kyoto-u.ac.jp>

Bill Baxter wrote:
> So, silly as it may seem, I think that Jarrod's plan of action here is
> correct.   The NR guys may be wrong about copyright law, but being
> right is not a prerequisite for filing a lawsuit.  Merely thinking you
> are is generally enough.  And sometimes not even that...
>   

I think there is a very simple argument to be made here: we are all
(hopefully) better at programming than at understanding law. So we
should rather spent our time reimplementing the code rather than trying
to understand how the various countries implement copyrights :)

David


From wbaxter at gmail.com  Sat Dec 20 05:10:10 2008
From: wbaxter at gmail.com (Bill Baxter)
Date: Sat, 20 Dec 2008 19:10:10 +0900
Subject: [SciPy-dev] scipy.optimize.nonlin rewrite
In-Reply-To: <gibp0s$7pe$1@ger.gmane.org>
References: <1227987645.8264.200.camel@idol>
	<85b5c3130812080943m52902438jb3e79f96ad34f37c@mail.gmail.com>
	<gibp0s$7pe$1@ger.gmane.org>
Message-ID: <e86a5fd00812200210v34eadf33i74dc605237a89b06@mail.gmail.com>

> The difference in Broyden's good/first and bad/second method is to my
> understanding the one mentioned in Wikipedia: both methods can be written as
> updates to the inverse Jacobian because the update is rank-1 so that the
> Sherman-Morrison formula applies. The difference can be seen if one looks at
> the domains (right-hand vector) of the rank-1 inverse updates: Broyden's "good"
> method uses J_{n-1}^{-T} dx, whereas the "bad" method uses df. These two
> vectors are not equal, even though both updates satisfy the quasi-Newton
> condition.

Here's what it says in that paper "On the discovery of the 'good
Broyden' method" by Broyden

B[i] = B[i-1] + (y[i-1] - B[i-1]s[i-1]) *  ( s[i-1].T / ( s[i-1].T * s[i-1] ))
"""
The formula is referred to as "good" due to its better numerical
performance relative to another formula
that I also presented in [3], which has become to be known as the "bad
Broyden" update. See [5] for a discussion
of these two updates and their relations to the DFP and BFGS updates.
"""
Selected references:
[3] Broyden, C.G. (1965): A class of methods for solving nonlinear
simultaneous equations. Math. Comp.
19, 577?593
[5] Dennis, J.E., Jr., Schnabel, R.B. (1981): A new derivation of
symmetric positive definite secant updates.
In: Mangasarian, O.L., Meyer, R.R., Robinson, S.M., eds., Nonlinear
Programming 4, 1980. Academic
Press, New York, NY, 167?199

--bb


From nwagner at iam.uni-stuttgart.de  Sat Dec 20 07:39:25 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 20 Dec 2008 13:39:25 +0100
Subject: [SciPy-dev] python2.6 and scipy
Message-ID: <web-115324533@uni-stuttgart.de>

Hi all,

I just installed scipy using python2.6 on
openSuSE 11.1

Ran 3353 tests in 53.293s

FAILED (KNOWNFAIL=2, SKIP=28, errors=124, failures=1)
<nose.result.TextTestResult run=3353 errors=124 
failures=1>

Is it recommended to use python2.5 ?

Nils


From david at ar.media.kyoto-u.ac.jp  Sat Dec 20 07:30:00 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 20 Dec 2008 21:30:00 +0900
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <web-115324533@uni-stuttgart.de>
References: <web-115324533@uni-stuttgart.de>
Message-ID: <494CE548.1060104@ar.media.kyoto-u.ac.jp>

Nils Wagner wrote:
> Hi all,
>
> I just installed scipy using python2.6 on
> openSuSE 11.1
>
> Ran 3353 tests in 53.293s
>
> FAILED (KNOWNFAIL=2, SKIP=28, errors=124, failures=1)
> <nose.result.TextTestResult run=3353 errors=124 
> failures=1>
>
> Is it recommended to use python2.5 ?
>   

Is python 2.6 the default version on openSuSE 11.1 ?

David


From michael.abshoff at googlemail.com  Sat Dec 20 07:53:46 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Sat, 20 Dec 2008 04:53:46 -0800
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <494CE548.1060104@ar.media.kyoto-u.ac.jp>
References: <web-115324533@uni-stuttgart.de>
	<494CE548.1060104@ar.media.kyoto-u.ac.jp>
Message-ID: <494CEADA.1050801@gmail.com>

David Cournapeau wrote:
> Nils Wagner wrote:
>> Hi all,

Hi David,

>> I just installed scipy using python2.6 on
>> openSuSE 11.1
>>
>> Ran 3353 tests in 53.293s
>>
>> FAILED (KNOWNFAIL=2, SKIP=28, errors=124, failures=1)
>> <nose.result.TextTestResult run=3353 errors=124 
>> failures=1>
>>
>> Is it recommended to use python2.5 ?
>>   
> 
> Is python 2.6 the default version on openSuSE 11.1 ?

Yes, they ship Python 2.6

> David

Cheers,

Michael

> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
> 



From nwagner at iam.uni-stuttgart.de  Sat Dec 20 09:26:37 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 20 Dec 2008 15:26:37 +0100
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <494CEADA.1050801@gmail.com>
References: <web-115324533@uni-stuttgart.de>
	<494CE548.1060104@ar.media.kyoto-u.ac.jp>
	<494CEADA.1050801@gmail.com>
Message-ID: <web-115325753@uni-stuttgart.de>

On Sat, 20 Dec 2008 04:53:46 -0800
  Michael Abshoff <michael.abshoff at googlemail.com> wrote:
> David Cournapeau wrote:
>> Nils Wagner wrote:
>>> Hi all,
> 
> Hi David,
> 
>>> I just installed scipy using python2.6 on
>>> openSuSE 11.1
>>>
>>> Ran 3353 tests in 53.293s
>>>
>>> FAILED (KNOWNFAIL=2, SKIP=28, errors=124, failures=1)
>>> <nose.result.TextTestResult run=3353 errors=124 
>>> failures=1>
>>>
>>> Is it recommended to use python2.5 ?
>>>   
>> 
>> Is python 2.6 the default version on openSuSE 11.1 ?
> 
> Yes, they ship Python 2.6
> 
>> David
> 
> Cheers,
> 
> Michael
> 
Hi David and Michael,

Should I post a complete log of scipy.test() to the list ?

Nils


From cournape at gmail.com  Sat Dec 20 09:31:42 2008
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 20 Dec 2008 23:31:42 +0900
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <494CEADA.1050801@gmail.com>
References: <web-115324533@uni-stuttgart.de>
	<494CE548.1060104@ar.media.kyoto-u.ac.jp> <494CEADA.1050801@gmail.com>
Message-ID: <5b8d13220812200631q560d7596i4270a48d16cb613f@mail.gmail.com>

On Sat, Dec 20, 2008 at 9:53 PM, Michael Abshoff
<michael.abshoff at googlemail.com> wrote:

> Yes, they ship Python 2.6
>

I am surprised it is the default interpreter. 11.1 is not a beta or
anything, right ?

David


From nwagner at iam.uni-stuttgart.de  Sat Dec 20 09:40:10 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 20 Dec 2008 15:40:10 +0100
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <5b8d13220812200631q560d7596i4270a48d16cb613f@mail.gmail.com>
References: <web-115324533@uni-stuttgart.de>
	<494CE548.1060104@ar.media.kyoto-u.ac.jp>
	<494CEADA.1050801@gmail.com>
	<5b8d13220812200631q560d7596i4270a48d16cb613f@mail.gmail.com>
Message-ID: <web-115325945@uni-stuttgart.de>

On Sat, 20 Dec 2008 23:31:42 +0900
  "David Cournapeau" <cournape at gmail.com> wrote:
> On Sat, Dec 20, 2008 at 9:53 PM, Michael Abshoff
> <michael.abshoff at googlemail.com> wrote:
> 
>> Yes, they ship Python 2.6
>>
> 
> I am surprised it is the default interpreter. 11.1 is 
>not a beta or
> anything, right ?
> 
> David

David,

It is an official release.

http://news.opensuse.org/2008/12/18/opensuse-111-released/
  
See the notes below

Under the Hood

openSUSE 11.1 also includes several changes ?under the 
hood,? including a new kernel release, updated Glibc, new 
version of PackageKit, Smolt integration, and many other 
updated applications and utilities:

     * Linux 2.6.27.7
     * Glibc 2.9
     * Python 2.6
     * Perl 5.10
     * Mono 2.0

Nils


From michael.abshoff at googlemail.com  Sat Dec 20 09:41:43 2008
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Sat, 20 Dec 2008 06:41:43 -0800
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <5b8d13220812200631q560d7596i4270a48d16cb613f@mail.gmail.com>
References: <web-115324533@uni-stuttgart.de>	<494CE548.1060104@ar.media.kyoto-u.ac.jp>
	<494CEADA.1050801@gmail.com>
	<5b8d13220812200631q560d7596i4270a48d16cb613f@mail.gmail.com>
Message-ID: <494D0427.2040409@gmail.com>

David Cournapeau wrote:
> On Sat, Dec 20, 2008 at 9:53 PM, Michael Abshoff
> <michael.abshoff at googlemail.com> wrote:

Hi,

>> Yes, they ship Python 2.6
>>
> 
> I am surprised it is the default interpreter. 11.1 is not a beta or
> anything, right ?

It has been release a couple days, but I did not poke around if they 
also offer python 2.5.x

> David

Cheers,

Michael

> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
> 



From nwagner at iam.uni-stuttgart.de  Sat Dec 20 09:45:31 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 20 Dec 2008 15:45:31 +0100
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <494D0427.2040409@gmail.com>
References: <web-115324533@uni-stuttgart.de>
	<494CE548.1060104@ar.media.kyoto-u.ac.jp>
	<494CEADA.1050801@gmail.com>
	<5b8d13220812200631q560d7596i4270a48d16cb613f@mail.gmail.com>
	<494D0427.2040409@gmail.com>
Message-ID: <web-115326004@uni-stuttgart.de>

On Sat, 20 Dec 2008 06:41:43 -0800
  Michael Abshoff <michael.abshoff at googlemail.com> wrote:
> David Cournapeau wrote:
>> On Sat, Dec 20, 2008 at 9:53 PM, Michael Abshoff
>> <michael.abshoff at googlemail.com> wrote:
> 
> Hi,
> 
>>> Yes, they ship Python 2.6
>>>
>> 
>> I am surprised it is the default interpreter. 11.1 is 
>>not a beta or
>> anything, right ?
> 
> It has been release a couple days, but I did not poke 
>around if they 
> also offer python 2.5.x
> 

Michael,

AFAIK,python2.5 is not available in the current distro,

http://packages.opensuse-community.org/index.jsp?searchTerm=python&distro=openSUSE_111
  

Nils


From pav at iki.fi  Sat Dec 20 13:59:57 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 20 Dec 2008 18:59:57 +0000 (UTC)
Subject: [SciPy-dev] python2.6 and scipy
References: <web-115324533@uni-stuttgart.de>
Message-ID: <gijfbd$3vd$1@ger.gmane.org>

Sat, 20 Dec 2008 13:39:25 +0100, Nils Wagner wrote:

> Hi all,
> 
> I just installed scipy using python2.6 on openSuSE 11.1
> 
> Ran 3353 tests in 53.293s
> 
> FAILED (KNOWNFAIL=2, SKIP=28, errors=124, failures=1)
> <nose.result.TextTestResult run=3353 errors=124 failures=1>
> 
> Is it recommended to use python2.5 ?

For me, all the errors seem to look like this:

---------------------
Traceback (most recent call last):
  File "/home/pauli/koodi/proj/scipy/scipy.git/dist/linux/lib/python2.6/
site-packages/scipy/stats/tests/test_mstats_basic.py", line 347, in 
test_mode
    assert_equal(mstats.mode(a1, axis=None), (3,4))
  File "dist/linux/lib/python2.6/site-packages/scipy/stats/
mstats_basic.py", line 267, in mode
  File "dist/linux/lib/python2.6/site-packages/scipy/stats/
mstats_basic.py", line 259, in _mode1D
  File "dist/linux/lib/python2.6/site-packages/scipy/stats/
mstats_basic.py", line 142, in find_repeats
RuntimeError: more argument specifiers than keyword list entries 
(remaining format:'|:futil.dfreps')
---------------------

Looks like a f2py issue.

-- 
Pauli Virtanen



From josef.pktd at gmail.com  Sat Dec 20 14:33:19 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 20 Dec 2008 14:33:19 -0500
Subject: [SciPy-dev] changes to stats t-tests
Message-ID: <1cd32cbb0812201133u24c4752bgc7da2b955370cc11@mail.gmail.com>

I finally looked at the t-test in more detail and I would like to make
three changes to the ttests

1) ttest_1samp: add axis argument, it is the only ttest and similar
function without an axis argument. Currently the retrun for multi
dimensional arrays is wrong, the array is flattened for the
calculation of mean and variance, but the number of observations is
taken along axis 0 (Test suite only check for 1-dim case).
One problem is the default axis, the usual default axis in statistics,
scipy.stats is zero, but it is not always consistently used
ttest_rel(a,b,axis=None), ttest_ind(a, b, axis=0), sem(a, axis=0)

2) return for t statistic: return number instead of 0-dimensional
array, no changes for higher dimensions
     Same for all three tests.

current return (t-statistic, p-value)
(array(0.81248591389165681), 0.41846234511362179)
proposed return
(0.81248591389165681, 0.41846234511362179)

3) clean up handling of zero division problem: current zero division
problem doesn't make sense, it return a t-statistic  arbitrarily set
to one. This only applies to cases with zero variance
    proposed change, return either inf (if numerator different from
zero) or zero (if numerator is also zero.
     Same for all three tests.

current:
>>> print st.ttest_rel( [0,0], [1,1],0)    #mean is different
(array(1.0), 0.49999999999999956)
>>> print st.ttest_rel( [0,0], [0,0],0)   #mean is the same
(array(1.0), 0.49999999999999956)

proposed:
>>> print ttest_rel( [0,0], [1,1],0)
(-1.#INF, 0.0)
>>> print ttest_rel( [0,0], [0,0],0)
(0.0, 1.0)

Since, we are just before release, I don't know if I can make these
changes now or after the release. Most of it fixes incorrect returns,
and I'm writing the tests for all cases.

Josef


From josef.pktd at gmail.com  Sat Dec 20 14:43:11 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 20 Dec 2008 14:43:11 -0500
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <gijfbd$3vd$1@ger.gmane.org>
References: <web-115324533@uni-stuttgart.de> <gijfbd$3vd$1@ger.gmane.org>
Message-ID: <1cd32cbb0812201143x2d6b6200p73e1e7830bf5674c@mail.gmail.com>

On Sat, Dec 20, 2008 at 1:59 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Sat, 20 Dec 2008 13:39:25 +0100, Nils Wagner wrote:
>
>> Hi all,
>>
>> I just installed scipy using python2.6 on openSuSE 11.1
>>
>> Ran 3353 tests in 53.293s
>>
>> FAILED (KNOWNFAIL=2, SKIP=28, errors=124, failures=1)
>> <nose.result.TextTestResult run=3353 errors=124 failures=1>
>>
>> Is it recommended to use python2.5 ?
>
> For me, all the errors seem to look like this:
>
> ---------------------
> Traceback (most recent call last):
>  File "/home/pauli/koodi/proj/scipy/scipy.git/dist/linux/lib/python2.6/
> site-packages/scipy/stats/tests/test_mstats_basic.py", line 347, in
> test_mode
>    assert_equal(mstats.mode(a1, axis=None), (3,4))
>  File "dist/linux/lib/python2.6/site-packages/scipy/stats/
> mstats_basic.py", line 267, in mode
>  File "dist/linux/lib/python2.6/site-packages/scipy/stats/
> mstats_basic.py", line 259, in _mode1D
>  File "dist/linux/lib/python2.6/site-packages/scipy/stats/
> mstats_basic.py", line 142, in find_repeats
> RuntimeError: more argument specifiers than keyword list entries
> (remaining format:'|:futil.dfreps')
> ---------------------
>
> Looks like a f2py issue.
>
> --
> Pauli Virtanen
>

I don't know much about f2py, but stats.futil.f  doesn't have a
corresponding futil.pyf in my scipy compilation, and the
stats/setup.py looks different for futil than for the other fortran
modules. Maybe its an incomplete setup in stats/setup.py?

Josef

Josef


From pav at iki.fi  Sat Dec 20 14:52:28 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 20 Dec 2008 19:52:28 +0000 (UTC)
Subject: [SciPy-dev] python2.6 and scipy
References: <web-115324533@uni-stuttgart.de> <gijfbd$3vd$1@ger.gmane.org>
	<1cd32cbb0812201143x2d6b6200p73e1e7830bf5674c@mail.gmail.com>
Message-ID: <gijids$ctd$1@ger.gmane.org>

Sat, 20 Dec 2008 14:43:11 -0500, josef.pktd wrote:
[clip]
> I don't know much about f2py, but stats.futil.f  doesn't have a
> corresponding futil.pyf in my scipy compilation, and the stats/setup.py
> looks different for futil than for the other fortran modules. Maybe its
> an incomplete setup in stats/setup.py?

f2py can generate the .pyf file on-the-fly (in memory), so generating one 
beforehand is not needed unless some customization is necessary.

-- 
Pauli Virtanen



From xavier.gnata at gmail.com  Sat Dec 20 15:13:57 2008
From: xavier.gnata at gmail.com (Xavier Gnata)
Date: Sat, 20 Dec 2008 21:13:57 +0100
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <cb12d88bd56b8d136cbcf5d7026815be.squirrel@webmail.uio.no>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>	<9457e7c80812191441o2e17a671ud2836bcdd6d5d667@mail.gmail.com>
	<cb12d88bd56b8d136cbcf5d7026815be.squirrel@webmail.uio.no>
Message-ID: <494D5205.9040206@gmail.com>


>> I am not convinced that their interpretation of copyright law is
>> correct, but either way I think ttest_ind and ttest_rel are safe
>> (according to *their* rules).  The code makes use of constructs such
>> as "where" and broadcasting, that aren't available in C.  The author
>> therefore took an idea from NR and reimplemented it in a novel way.
> 
> I haven't looked at the actual code in question here. Just beware that
> there are Fortran 90 code on the NR CD-ROM and web site. In 1996 the "NR
> in Fortran 90" book was published. Their Fortran 90 code does make use of
> where constructs, broadcasting and array slices.
> 
> 
> Sturla Molden

Come on ;)
All these algo are very standard one.
You can refer to this great book to *explain* these pieces of math but
that's it.
If you want to code let's say a heapshort or a golden ratio based
search, you may refer to one another great book to explain how is it
supposed to work. Fine.
That said, there is no many different ways to code such algo in fortran
(it is of course the same whatever the language could be).
If you think there is a need to obfuscate standard pieces of math well
....that is your choice.

If someone try to claim something about maths, you can just remove all
the ref to this strange guy and put a ref to any classical paper written
 by someone reasonable (or event to wikipedia...)

Xavier


From matthew.brett at gmail.com  Sat Dec 20 20:23:11 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sun, 21 Dec 2008 01:23:11 +0000
Subject: [SciPy-dev] changes to stats t-tests
In-Reply-To: <1cd32cbb0812201133u24c4752bgc7da2b955370cc11@mail.gmail.com>
References: <1cd32cbb0812201133u24c4752bgc7da2b955370cc11@mail.gmail.com>
Message-ID: <1e2af89e0812201723u6190882cw80ed45eecdc3bc4b@mail.gmail.com>

Hi,

> I finally looked at the t-test in more detail and I would like to make
> three changes to the ttests

These changes look sensible to me.  Because they include bug-fixes, I
would tentatively vote for inclusion in 0.7

I would also like to say a very big thank you for all your work on the
stats package, it has been very encouraging to see all the excellent
and careful work you have been doing.

Best,

Matthew


From david at ar.media.kyoto-u.ac.jp  Sun Dec 21 00:48:14 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 21 Dec 2008 14:48:14 +0900
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <web-115326004@uni-stuttgart.de>
References: <web-115324533@uni-stuttgart.de>	<494CE548.1060104@ar.media.kyoto-u.ac.jp>	<494CEADA.1050801@gmail.com>	<5b8d13220812200631q560d7596i4270a48d16cb613f@mail.gmail.com>	<494D0427.2040409@gmail.com>
	<web-115326004@uni-stuttgart.de>
Message-ID: <494DD89E.3000503@ar.media.kyoto-u.ac.jp>

Nils Wagner wrote:
> Michael,
>
> AFAIK,python2.5 is not available in the current distro,
>
> http://packages.opensuse-community.org/index.jsp?searchTerm=python&distro=openSUSE_111
>   
>   

I am a bit surprised, because many projects do not work flawlessly on
python 2.6. OTOH, it is a pretty good intensive for people to update
their packages :)

David


From millman at berkeley.edu  Sun Dec 21 03:03:53 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 21 Dec 2008 00:03:53 -0800
Subject: [SciPy-dev] changes to stats t-tests
In-Reply-To: <1cd32cbb0812201133u24c4752bgc7da2b955370cc11@mail.gmail.com>
References: <1cd32cbb0812201133u24c4752bgc7da2b955370cc11@mail.gmail.com>
Message-ID: <c7009a550812210003k4863b9f9wfee89d4e83b1bf66@mail.gmail.com>

On Sat, Dec 20, 2008 at 11:33 AM,  <josef.pktd at gmail.com> wrote:
> I finally looked at the t-test in more detail and I would like to make
> three changes to the ttests

Thanks for looking into this.

> 1) ttest_1samp: add axis argument, it is the only ttest and similar
> function without an axis argument. Currently the retrun for multi
> dimensional arrays is wrong, the array is flattened for the
> calculation of mean and variance, but the number of observations is
> taken along axis 0 (Test suite only check for 1-dim case).
> One problem is the default axis, the usual default axis in statistics,
> scipy.stats is zero, but it is not always consistently used
> ttest_rel(a,b,axis=None), ttest_ind(a, b, axis=0), sem(a, axis=0)

+1 to adding axis arg to ttest_1samp.  I would set the default axis to
0 in all cases (i.e., I would also like to see ttest_rel default to
axis=0).

> 2) return for t statistic: return number instead of 0-dimensional
> array, no changes for higher dimensions
>     Same for all three tests.
>
> current return (t-statistic, p-value)
> (array(0.81248591389165681), 0.41846234511362179)
> proposed return
> (0.81248591389165681, 0.41846234511362179)

+1

> 3) clean up handling of zero division problem: current zero division
> problem doesn't make sense, it return a t-statistic  arbitrarily set
> to one. This only applies to cases with zero variance
>    proposed change, return either inf (if numerator different from
> zero) or zero (if numerator is also zero.

I would think that for a/0=x:
  a > 0  then x is +inf
  a < 0  then x is -inf
  a = 0 then x is NaN

Why would x be 0 when a is 0?  Regardless, I agree that the current
behavior is incorrect and needs to be fixed.

> Since, we are just before release, I don't know if I can make these
> changes now or after the release. Most of it fixes incorrect returns,
> and I'm writing the tests for all cases.

I would like to see these changes make it into the 0.7 release.
Especially if you write tests and document the changes.  Can you make
the changes before the end of the weekend?

Thanks,
Jarrod


From millman at berkeley.edu  Sun Dec 21 03:20:14 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 21 Dec 2008 00:20:14 -0800
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <494DD89E.3000503@ar.media.kyoto-u.ac.jp>
References: <web-115324533@uni-stuttgart.de>
	<494CE548.1060104@ar.media.kyoto-u.ac.jp> <494CEADA.1050801@gmail.com>
	<5b8d13220812200631q560d7596i4270a48d16cb613f@mail.gmail.com>
	<494D0427.2040409@gmail.com> <web-115326004@uni-stuttgart.de>
	<494DD89E.3000503@ar.media.kyoto-u.ac.jp>
Message-ID: <c7009a550812210020t1fc47952h549d1b9343236ae7@mail.gmail.com>

On Sat, Dec 20, 2008 at 9:48 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Nils Wagner wrote:
>> AFAIK,python2.5 is not available in the current distro,
>>
>> http://packages.opensuse-community.org/index.jsp?searchTerm=python&distro=openSUSE_111
>
> I am a bit surprised, because many projects do not work flawlessly on
> python 2.6. OTOH, it is a pretty good intensive for people to update
> their packages :)

FYI, Fedora 11 will also have Python 2.6:
https://fedoraproject.org/wiki/Releases/11/FeatureList

It doesn't come out for another 6 months:
https://fedoraproject.org/wiki/Releases/11/Schedule

There is also an interesting thread about how Fedora will handle the
transition to Python 3.0:
https://www.redhat.com/archives/fedora-devel-list/2008-December/msg00379.html


From josef.pktd at gmail.com  Sun Dec 21 10:56:12 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 21 Dec 2008 10:56:12 -0500
Subject: [SciPy-dev] changes to stats t-tests
In-Reply-To: <c7009a550812210003k4863b9f9wfee89d4e83b1bf66@mail.gmail.com>
References: <1cd32cbb0812201133u24c4752bgc7da2b955370cc11@mail.gmail.com>
	<c7009a550812210003k4863b9f9wfee89d4e83b1bf66@mail.gmail.com>
Message-ID: <1cd32cbb0812210756x4dbbe77fme778f1867b2b3dbf@mail.gmail.com>

On Sun, Dec 21, 2008 at 3:03 AM, Jarrod Millman <millman at berkeley.edu> wrote:
> On Sat, Dec 20, 2008 at 11:33 AM,  <josef.pktd at gmail.com> wrote:
>> I finally looked at the t-test in more detail and I would like to make
>> three changes to the ttests
>
> Thanks for looking into this.
>
>> 1) ttest_1samp: add axis argument, it is the only ttest and similar
>> function without an axis argument. Currently the retrun for multi
>> dimensional arrays is wrong, the array is flattened for the
>> calculation of mean and variance, but the number of observations is
>> taken along axis 0 (Test suite only check for 1-dim case).
>> One problem is the default axis, the usual default axis in statistics,
>> scipy.stats is zero, but it is not always consistently used
>> ttest_rel(a,b,axis=None), ttest_ind(a, b, axis=0), sem(a, axis=0)
>
> +1 to adding axis arg to ttest_1samp.  I would set the default axis to
> 0 in all cases (i.e., I would also like to see ttest_rel default to
> axis=0).



>> 2) return for t statistic: return number instead of 0-dimensional
>> array, no changes for higher dimensions
>>     Same for all three tests.
>>
>> current return (t-statistic, p-value)
>> (array(0.81248591389165681), 0.41846234511362179)
>> proposed return
>> (0.81248591389165681, 0.41846234511362179)
>
> +1
>
>> 3) clean up handling of zero division problem: current zero division
>> problem doesn't make sense, it return a t-statistic  arbitrarily set
>> to one. This only applies to cases with zero variance
>>    proposed change, return either inf (if numerator different from
>> zero) or zero (if numerator is also zero.
>
> I would think that for a/0=x:
>  a > 0  then x is +inf
>  a < 0  then x is -inf
>  a = 0 then x is NaN
>
> Why would x be 0 when a is 0?  Regardless, I agree that the current
> behavior is incorrect and needs to be fixed.

The +inf, -inf I have implemented this way.
The case when a=0 is a/0=x is not so obvious
a = 0 and the denominator =0, means that the variance is zero and the
null hypothesis is exactly satisfied, e.g. in the independent case,
this means we are comparing two identical constant arrays. So, I
thought the t-statistic should be zero and the pvalue is one, meaning
the probability that the two random variables have the same mean is
one.

that's what I currently have:
>>> ttest_ind([0,0,0], [0,0,0])
(0.0, 1.0)
>>> ttest_ind([0,0,0], [1,1,1])
(-1.#INF, 0.0)
>>> ttest_1samp([0,0,0], 0)
(0.0, 1.0)
>>> ttest_1samp([0,0,0], 1)
(-1.#INF, 0.0)
>>> ttest_1samp([0,0,0], -1)
(1.#INF, 0.0)

However after looking at the limiting case for a=0 some more,  it is
not clear to me anymore whether my initial intuition is correct. I
tried it out for a normal distribution with variance going to zero,
but I don't have time to find what the correct theoretical limit for
the t-statistic is for a degenerate normal distribution. But it seems
that it might be one, as defined in the initial zero division problem.
This appears correct to me, since the t-statistic is normalized in
such a way that it is prevented from collapsing to a degenerate random
variable.

The rejection rates for the ttest_1samp with a normal distribution
with scale 1e-8, are very close to the theoretical values. The power
to reject incorrect null hypothesis is huge. So, for any
non-degenerate case the ttest functions work correctly.

Now I think that defining t=0/0=1 might be more appropriate. For both
cases t=0/0=1 or t=0/0=0, we would not reject the null hypothesis at
any of the usual confidence levels, which is the correct answer. I
wouldn't set 0/0 = nan in this case, because it may show up in some
edge cases, and not rejecting the null is the correct answer.

t=0/0=1 is also the numerical limiting case for simple examples with
small variance (I hadn't checked this before), from current version
>>> st.ttest_rel([1e-4,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0])
(array(0.99999999999999967), 0.34659350708733427)
>>> st.ttest_ind([1e-4,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0])
(array(0.99999999999999967), 0.33219498465297947)
>>> st.ttest_1samp([1e-4,0,0,0,0,0,0,0,0], 0)
(0.99999999999999967, 0.34659350708733427)
>>> st.ttest_ind([1e-4,0,0,0,0,0,0,0,0], [1e-4,0,0,0,0,0,0,0,0])
(array(0.0), 1.0)

>
>> Since, we are just before release, I don't know if I can make these
>> changes now or after the release. Most of it fixes incorrect returns,
>> and I'm writing the tests for all cases.
>
> I would like to see these changes make it into the 0.7 release.
> Especially if you write tests and document the changes.  Can you make
> the changes before the end of the weekend?
>

I should be able to commit my rewrite in a few hours, I will use
default axis zero and  t=0/0=1, if there are no objections.

Josef


From rkramer at systemsbiology.org  Sun Dec 21 13:19:02 2008
From: rkramer at systemsbiology.org (Roger Kramer)
Date: Sun, 21 Dec 2008 10:19:02 -0800
Subject: [SciPy-dev] Non-deterministic test failures in scipy-0.7.0b1
Message-ID: <c5f179dd0812211019r56be8b13u2ffba8b34fa82e26@mail.gmail.com>

I'm still looking into this, but in case anyone has ideas that could save me
some time...
The following occurs non-deterministically:

======================================================================
FAIL: test_x_stride (test_fblas.TestCgemv)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/rk/lib/python2.5/site-packages/scipy/lib/blas/tests/test_fblas.py",
line 345, in test_x_stride
    assert_array_almost_equal(desired_y,y)
  File "/home/rk/lib/python2.5/site-packages/numpy/testing/utils.py", line
311, in assert_array_almost_equal
    header='Arrays are not almost equal')
  File "/home/rk/lib/python2.5/site-packages/numpy/testing/utils.py", line
296, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not almost equal

(mismatch 33.3333333333%)
 x: array([ -2.91183448 +2.91183448j, -14.95274162+16.95274162j,
         3.63382101 +0.36617899j], dtype=complex64)
 y: array([ -2.91183448 +2.91183448j, -14.95274162+16.95274353j,
         3.63382101 +0.36617899j], dtype=complex64)
------------------------------------------------------------------------

...and test_y_stride occasionally fails, too. What's odder than the fact
that the arrays don't really look 33% mismatched, is that on successive runs
of scipy.test(), the failure may or may not occur, and I'm changing/touching
-nothing- between the test runs. Normally, non-deterministic failures would
scream race conditions, but since these are the only test failures it seems
a stretch to implicate threading in general.

In case it's relevant, I compiled (gcc 4.3.2) Netlib CBLAS and LAPACK 3.2
sources directly as shared libraries. Have had considerable trouble with
LAPACK. Compiled code only passes the lapack test suite if the
machine-dependent bits in INSTALL are compiled without optimization.
Everything else (in LAPACK) is compiled with -O2 -march=native
-mtune=native, and I'm on a Dell machine with an E2160 processor. Intel's
"issues list" on this processor family is a fright.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081221/9a623507/attachment.html>

From cournape at gmail.com  Sun Dec 21 20:22:35 2008
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 22 Dec 2008 10:22:35 +0900
Subject: [SciPy-dev] Non-deterministic test failures in scipy-0.7.0b1
In-Reply-To: <c5f179dd0812211019r56be8b13u2ffba8b34fa82e26@mail.gmail.com>
References: <c5f179dd0812211019r56be8b13u2ffba8b34fa82e26@mail.gmail.com>
Message-ID: <5b8d13220812211722s5b62b882o49a2f73a8980d343@mail.gmail.com>

On Mon, Dec 22, 2008 at 3:19 AM, Roger Kramer
<rkramer at systemsbiology.org> wrote:
> I'm still looking into this, but in case anyone has ideas that could save me
> some time...
> The following occurs non-deterministically:
>
> ======================================================================
> FAIL: test_x_stride (test_fblas.TestCgemv)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/home/rk/lib/python2.5/site-packages/scipy/lib/blas/tests/test_fblas.py",
> line 345, in test_x_stride
>     assert_array_almost_equal(desired_y,y)
>   File "/home/rk/lib/python2.5/site-packages/numpy/testing/utils.py", line
> 311, in assert_array_almost_equal
>     header='Arrays are not almost equal')
>   File "/home/rk/lib/python2.5/site-packages/numpy/testing/utils.py", line
> 296, in assert_array_compare
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal
>
> (mismatch 33.3333333333%)
>  x: array([ -2.91183448 +2.91183448j, -14.95274162+16.95274162j,
>          3.63382101 +0.36617899j], dtype=complex64)
>  y: array([ -2.91183448 +2.91183448j, -14.95274162+16.95274353j,
>          3.63382101 +0.36617899j], dtype=complex64)
> ------------------------------------------------------------------------
>

the test is buggy, because with complex64, you have single precision
in each dimension, with 7 significant digits, and as you can see, the
difference in the second element appear at the 8th digit

>, but since these are the only test failures it seems
> a stretch to implicate threading in general.

Yes, specially since there is almost no explicit threading in scipy.

>
> In case it's relevant, I compiled (gcc 4.3.2) Netlib CBLAS and LAPACK 3.2
> sources directly as shared libraries. Have had considerable trouble with
> LAPACK.

Yes, LAPACK is a pain, because of fortran. You are better off with the
ones packaged by your OS vendor if you can have some.

> Compiled code only passes the lapack test suite if the
> machine-dependent bits in INSTALL are compiled without optimization.
> Everything else (in LAPACK) is compiled with -O2 -march=native
> -mtune=native, and I'm on a Dell machine with an E2160 processor. Intel's
> "issues list" on this processor family is a fright.

march tells gcc to use arch specific instructions, so the results may
well depend on your CPU.

David


From stijn.deweirdt at ugent.be  Mon Dec 22 02:56:29 2008
From: stijn.deweirdt at ugent.be (Stijn De Weirdt)
Date: Mon, 22 Dec 2008 08:56:29 +0100
Subject: [SciPy-dev] segmentation with scipy+icc during test
In-Reply-To: <e76aa17f0812092300s31a3c0a5r6a4590d65f5dd5c@mail.gmail.com>
References: <1228497854.3733.32.camel@spike.ugent.be>
	<1228507208.8840.9.camel@spike.ugent.be>
	<e76aa17f0812051246g691a4330pc8a3c3aaae2f52ff@mail.gmail.com>
	<1228511060.11640.10.camel@spike.ugent.be>
	<e76aa17f0812051329p2238c221kfc523a17f8c4d9f3@mail.gmail.com>
	<1228514330.13880.8.camel@spike.ugent.be>
	<e76aa17f0812090125m7763bad6lad5aa75e2e6f8f1c@mail.gmail.com>
	<1228815623.19004.1.camel@spike.ugent.be>
	<e76aa17f0812090143n2e0bff1fr114a45e18c20ec0e@mail.gmail.com>
	<493F5B77.8060903@ar.media.kyoto-u.ac.jp>
	<e76aa17f0812092300s31a3c0a5r6a4590d65f5dd5c@mail.gmail.com>
Message-ID: <1229932589.3174.11.camel@spike.ugent.be>

hi all,

i found some time again to look at this.
i compiled python 2.5.2 with icc and then numpy 1.2.1 and the i tried
scipy 0.7b1 (and also latest svn).

the test that now segfaults is from scipy/lib/lapack/tests/test_esv.py
(it starts with the test_dsyev).
i get the segfault with python -c 'import
numpy;scipy.test(level=10,verbosity=100)'

but when i try to reproduce it, i copy the test_esv.py and merge it with
the common.py from same location. i also add run_module_suite to the
'from numpy.testing import' so it can find eveything it seems to need.
if i then do python ./test_esv.py i get

SSSSSS......
----------------------------------------------------------------------
Ran 12 tests in 0.014s

OK (SKIP=6)

so all seems ok.

so who can help with the debugging ;)

stijn
-- 
The system will shutdown in 5 minutes.



From nwagner at iam.uni-stuttgart.de  Mon Dec 22 09:24:17 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 22 Dec 2008 15:24:17 +0100
Subject: [SciPy-dev] ValueError: Lack of convergence in obrientransform.
Message-ID: <web-115366795@uni-stuttgart.de>

>>> scipy.__version__
'0.7.0.dev5286'


======================================================================
ERROR: Tests Obrien transform
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/stats/tests/test_mstats_basic.py", 
line 469, in test_obrientransform
     assert_almost_equal(np.round(mstats.obrientransform(*args).T,4),
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/stats/mstats_basic.py", 
line 1732, in obrientransform
     raise ValueError("Lack of convergence in 
obrientransform.")
ValueError: Lack of convergence in obrientransform.
  


From josef.pktd at gmail.com  Mon Dec 22 10:47:44 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 22 Dec 2008 10:47:44 -0500
Subject: [SciPy-dev] ValueError: Lack of convergence in obrientransform.
In-Reply-To: <web-115366795@uni-stuttgart.de>
References: <web-115366795@uni-stuttgart.de>
Message-ID: <1cd32cbb0812220747j17f43b45v4a443f6862ce546d@mail.gmail.com>

On Mon, Dec 22, 2008 at 9:24 AM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
>>>> scipy.__version__
> '0.7.0.dev5286'
>
>
> ======================================================================
> ERROR: Tests Obrien transform
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/usr/local/lib64/python2.5/site-packages/scipy/stats/tests/test_mstats_basic.py",
> line 469, in test_obrientransform
>     assert_almost_equal(np.round(mstats.obrientransform(*args).T,4),
>   File
> "/usr/local/lib64/python2.5/site-packages/scipy/stats/mstats_basic.py",
> line 1732, in obrientransform
>     raise ValueError("Lack of convergence in
> obrientransform.")
> ValueError: Lack of convergence in obrientransform.
>

I don't see any changes in scipy.stats that could be related.
mstats.obrientransform only uses methods of
numpy.ma.core.MaskedArray. Which numpy version are you using?

Josef


From nwagner at iam.uni-stuttgart.de  Mon Dec 22 11:28:27 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 22 Dec 2008 17:28:27 +0100
Subject: [SciPy-dev] ValueError: Lack of convergence in obrientransform.
In-Reply-To: <1cd32cbb0812220747j17f43b45v4a443f6862ce546d@mail.gmail.com>
References: <web-115366795@uni-stuttgart.de>
	<1cd32cbb0812220747j17f43b45v4a443f6862ce546d@mail.gmail.com>
Message-ID: <web-115370864@uni-stuttgart.de>

On Mon, 22 Dec 2008 10:47:44 -0500
  josef.pktd at gmail.com wrote:
> On Mon, Dec 22, 2008 at 9:24 AM, Nils Wagner
> <nwagner at iam.uni-stuttgart.de> wrote:
>>>>> scipy.__version__
>> '0.7.0.dev5286'
>>
>>
>> ======================================================================
>> ERROR: Tests Obrien transform
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File
>> "/usr/local/lib64/python2.5/site-packages/scipy/stats/tests/test_mstats_basic.py",
>> line 469, in test_obrientransform
>>     assert_almost_equal(np.round(mstats.obrientransform(*args).T,4),
>>   File
>> "/usr/local/lib64/python2.5/site-packages/scipy/stats/mstats_basic.py",
>> line 1732, in obrientransform
>>     raise ValueError("Lack of convergence in
>> obrientransform.")
>> ValueError: Lack of convergence in obrientransform.
>>
> 
> I don't see any changes in scipy.stats that could be 
>related.
> mstats.obrientransform only uses methods of
> numpy.ma.core.MaskedArray. Which numpy version are you 
>using?
> 

>>> numpy.__version__
'1.3.0.dev6188'
  
Nils


From josef.pktd at gmail.com  Mon Dec 22 11:37:52 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 22 Dec 2008 11:37:52 -0500
Subject: [SciPy-dev] build problem error: file 'ARPACK\FWRAPPERS\dummy.f'
	does not exist
Message-ID: <1cd32cbb0812220837ofcf4f6etd8d4ede456b0cfe2@mail.gmail.com>

I was trying to build the current scipy trunk, r5286, and got stuck with arpack

this is the last part of the build log

building 'arpack' library
compiling Fortran sources
Fortran f77 compiler: C:\Programs\MinGW\bin\g77.exe -g -Wall -fno-second-undersc
ore -mno-cygwin -O3 -funroll-loops
error: file 'ARPACK\FWRAPPERS\dummy.f' does not exist

WindowsXP, mingw 3.4,
svn export
python25 setup.py bdist
>>> np.version.version
'1.3.0.dev6139'

I don't see any ARPACK\FWRAPPERS directory anywhere.

Josef


From josef.pktd at gmail.com  Mon Dec 22 13:51:13 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 22 Dec 2008 13:51:13 -0500
Subject: [SciPy-dev] build problem error: file
	'ARPACK\FWRAPPERS\dummy.f' does not exist
In-Reply-To: <1cd32cbb0812220837ofcf4f6etd8d4ede456b0cfe2@mail.gmail.com>
References: <1cd32cbb0812220837ofcf4f6etd8d4ede456b0cfe2@mail.gmail.com>
Message-ID: <1cd32cbb0812221051s6d566f11o6d08ddd6264724c5@mail.gmail.com>

On Mon, Dec 22, 2008 at 11:37 AM,  <josef.pktd at gmail.com> wrote:
> I was trying to build the current scipy trunk, r5286, and got stuck with arpack
>
> this is the last part of the build log
>
> building 'arpack' library
> compiling Fortran sources
> Fortran f77 compiler: C:\Programs\MinGW\bin\g77.exe -g -Wall -fno-second-undersc
> ore -mno-cygwin -O3 -funroll-loops
> error: file 'ARPACK\FWRAPPERS\dummy.f' does not exist
>
> WindowsXP, mingw 3.4,
> svn export
> python25 setup.py bdist
>>>> np.version.version
> '1.3.0.dev6139'
>
> I don't see any ARPACK\FWRAPPERS directory anywhere.
>
> Josef
>

All clear,

`svn update` didn't download the fwrappers directory, I had to
explicitely specify the missing directory for update.

Josef


From ccasey at enthought.com  Mon Dec 22 14:11:48 2008
From: ccasey at enthought.com (Chris Casey)
Date: Mon, 22 Dec 2008 13:11:48 -0600
Subject: [SciPy-dev] EPD Py2.5 v4.1.30101 Released
Message-ID: <1229973108.5867.0.camel@linux-8ej9.site>

Greetings,

Enthought, Inc. is very pleased to announce the newest release of the 
Enthought Python Distribution (EPD) Py2.5 v4.1.30101:

http://www.enthought.com/epd

The size of the installer has be reduced by about half. Also, this is
the first release to include a 3.1.0 version of the Enthought Tool Suite
(http://code.enthought.com/), featuring Mayavi 3.1.0. This is also the
first release to use Enthought's enhanced version of setuptools,
Enstaller (http://code.enthought.com/projects/enstaller/). Windows
installation enhancements, matplotlib and wx issues, and menu
consistency accross platforms are among notable fixes.


The full release notes for this release can be found here:
https://svn.enthought.com/epd/wiki/Py25/4.1.30101/RelNotes

Many thanks to the EPD team for putting this release together, and to 
the community of folks who have provided all of the valuable tools 
bundled here.

Best Regards,

Chris

---------
About EPD
---------
The Enthought Python Distribution (EPD) is a "kitchen-sink-included" 
distribution of the Python? Programming Language, including over 80 
additional tools and libraries. The EPD bundle includes NumPy, SciPy, 
IPython, 2D and 3D visualization, database adapters, and a lot of 
other tools right out of the box.

http://www.enthought.com/products/epd.php

It is currently available as an easy, single-click installer for 
Windows XP (x86), Mac OS X (a universal binary for Intel 10.4 and 
above) and RedHat EL3 (x86 and amd64).

EPD is free for 30-day trial use and for use in degree-granting 
academic institutions. An annual Subscription and installation 
support are available for commercial use
(http://www.enthought.com/products/epddownload.php 
) including an Enterprise Subscription with support for particular 
deployment environments
(http://www.enthought.com/products/enterprise.php 
).

_______________________________________________
Enthought-dev mailing list
Enthought-dev at mail.enthought.com
https://mail.enthought.com/mailman/listinfo/enthought-dev




From josef.pktd at gmail.com  Mon Dec 22 16:30:06 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 22 Dec 2008 16:30:06 -0500
Subject: [SciPy-dev] import scipy.xplt as xplt
Message-ID: <1cd32cbb0812221330u2ab5cdb2t9df26ec69aad1438@mail.gmail.com>

in scipy.stats.probplot there is a
`import scipy.xplt as xplt`

Is this a historical artifact that can be removed?

Josef


From robert.kern at gmail.com  Mon Dec 22 16:35:17 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 22 Dec 2008 15:35:17 -0600
Subject: [SciPy-dev] import scipy.xplt as xplt
In-Reply-To: <1cd32cbb0812221330u2ab5cdb2t9df26ec69aad1438@mail.gmail.com>
References: <1cd32cbb0812221330u2ab5cdb2t9df26ec69aad1438@mail.gmail.com>
Message-ID: <3d375d730812221335k28f6308arb3a145a22ebf90d2@mail.gmail.com>

On Mon, Dec 22, 2008 at 15:30,  <josef.pktd at gmail.com> wrote:
> in scipy.stats.probplot there is a
> `import scipy.xplt as xplt`
>
> Is this a historical artifact that can be removed?

Yes.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Mon Dec 22 21:09:17 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 23 Dec 2008 11:09:17 +0900
Subject: [SciPy-dev] build problem error:
 file	'ARPACK\FWRAPPERS\dummy.f' does not exist
In-Reply-To: <1cd32cbb0812221051s6d566f11o6d08ddd6264724c5@mail.gmail.com>
References: <1cd32cbb0812220837ofcf4f6etd8d4ede456b0cfe2@mail.gmail.com>
	<1cd32cbb0812221051s6d566f11o6d08ddd6264724c5@mail.gmail.com>
Message-ID: <4950484D.5030402@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
> On Mon, Dec 22, 2008 at 11:37 AM,  <josef.pktd at gmail.com> wrote:
>   
>> I was trying to build the current scipy trunk, r5286, and got stuck with arpack
>>
>> this is the last part of the build log
>>
>> building 'arpack' library
>> compiling Fortran sources
>> Fortran f77 compiler: C:\Programs\MinGW\bin\g77.exe -g -Wall -fno-second-undersc
>> ore -mno-cygwin -O3 -funroll-loops
>> error: file 'ARPACK\FWRAPPERS\dummy.f' does not exist
>>
>> WindowsXP, mingw 3.4,
>> svn export
>> python25 setup.py bdist
>>     
>>>>> np.version.version
>>>>>           
>> '1.3.0.dev6139'
>>
>> I don't see any ARPACK\FWRAPPERS directory anywhere.
>>
>> Josef
>>
>>     
>
> All clear,
>
> `svn update` didn't download the fwrappers directory, I had to
> explicitely specify the missing directory for update.
>   

That's strange. Generally, the problem is with removed directories - I
never had problems updating new directories. Which version of subversion
are you using ?

David


From nwagner at iam.uni-stuttgart.de  Tue Dec 23 07:21:57 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 23 Dec 2008 13:21:57 +0100
Subject: [SciPy-dev] python2.6 and scipy
In-Reply-To: <c7009a550812210020t1fc47952h549d1b9343236ae7@mail.gmail.com>
References: <web-115324533@uni-stuttgart.de>
	<494CE548.1060104@ar.media.kyoto-u.ac.jp>
	<494CEADA.1050801@gmail.com>
	<5b8d13220812200631q560d7596i4270a48d16cb613f@mail.gmail.com>
	<494D0427.2040409@gmail.com> <web-115326004@uni-stuttgart.de>
	<494DD89E.3000503@ar.media.kyoto-u.ac.jp>
	<c7009a550812210020t1fc47952h549d1b9343236ae7@mail.gmail.com>
Message-ID: <web-115394642@uni-stuttgart.de>

On Sun, 21 Dec 2008 00:20:14 -0800
  "Jarrod Millman" <millman at berkeley.edu> wrote:
> On Sat, Dec 20, 2008 at 9:48 PM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>> Nils Wagner wrote:
>>> AFAIK,python2.5 is not available in the current distro,
>>>
>>> http://packages.opensuse-community.org/index.jsp?searchTerm=python&distro=openSUSE_111
>>
>> I am a bit surprised, because many projects do not work 
>>flawlessly on
>> python 2.6. OTOH, it is a pretty good intensive for 
>>people to update
>> their packages :)
> 
>FYI, Fedora 11 will also have Python 2.6:
> https://fedoraproject.org/wiki/Releases/11/FeatureList
> 
> It doesn't come out for another 6 months:
> https://fedoraproject.org/wiki/Releases/11/Schedule
> 
> There is also an interesting thread about how Fedora 
>will handle the
> transition to Python 3.0:
> https://www.redhat.com/archives/fedora-devel-list/2008-December/msg00379.html
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev

I have applied Pearu's patch for python2.6.

Here is the output of scipy.test()
  
======================================================================
ERROR: test_implicit (test_odr.TestODR)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/odr/tests/test_odr.py", 
line 88, in test_implicit
     out = implicit_odr.run()
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/odr/odrpack.py", 
line 1055, in run
     self.output = Output(apply(odr, args, kwds))
TypeError: y must be a sequence or integer (if model is 
implicit)

======================================================================
ERROR: Tests Obrien transform
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/tests/test_mstats_basic.py", 
line 469, in test_obrientransform
     assert_almost_equal(np.round(mstats.obrientransform(*args).T,4),
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/stats/mstats_basic.py", 
line 1732, in obrientransform
     raise ValueError("Lack of convergence in 
obrientransform.")
ValueError: Lack of convergence in obrientransform.

======================================================================
ERROR: test_add_function_ordered 
(test_catalog.TestCatalog)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/weave/tests/test_catalog.py", 
line 279, in test_add_function_ordered
     q.add_function('f',string.upper)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/weave/catalog.py", 
line 650, in add_function
     self.add_function_persistent(code,function)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/weave/catalog.py", 
line 685, in add_function_persistent
     function_list = function_list + cat.get(code,[])
   File "/usr/lib64/python2.6/shelve.py", line 113, in get
     if key in self.dict:
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/io/dumbdbm_patched.py", 
line 73, in __getitem__
     pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: Test persisting a function in the default catalog
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/weave/tests/test_catalog.py", 
line 267, in test_add_function_persistent1
     q.add_function_persistent('code',i)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/weave/catalog.py", 
line 685, in add_function_persistent
     function_list = function_list + cat.get(code,[])
   File "/usr/lib64/python2.6/shelve.py", line 113, in get
     if key in self.dict:
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/io/dumbdbm_patched.py", 
line 73, in __getitem__
     pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: Shouldn't get a single file from the temp dir.
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/weave/tests/test_catalog.py", 
line 195, in test_get_existing_files2
     q.add_function('code', os.getpid)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/weave/catalog.py", 
line 650, in add_function
     self.add_function_persistent(code,function)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/weave/catalog.py", 
line 685, in add_function_persistent
     function_list = function_list + cat.get(code,[])
   File "/usr/lib64/python2.6/shelve.py", line 113, in get
     if key in self.dict:
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/io/dumbdbm_patched.py", 
line 73, in __getitem__
     pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
FAIL: test_pbdv (test_basic.TestCephes)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/special/tests/test_basic.py", 
line 357, in test_pbdv
     assert_equal(cephes.pbdv(1,0),(0.0,0.0))
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", 
line 176, in assert_equal
     assert_equal(actual[k], desired[k], 'item=%r\n%s' % 
(k,err_msg), verbose)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", 
line 183, in assert_equal
     raise AssertionError(msg)
AssertionError:
Items are not equal:
item=1

  ACTUAL: 1.0
  DESIRED: 0.0

----------------------------------------------------------------------
Ran 3356 tests in 57.961s

FAILED (KNOWNFAIL=2, SKIP=28, errors=5, failures=1)
<nose.result.TextTestResult run=3356 errors=5 failures=1>

Nils


From josef.pktd at gmail.com  Tue Dec 23 10:10:57 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 23 Dec 2008 10:10:57 -0500
Subject: [SciPy-dev] build problem error: file
	'ARPACK\FWRAPPERS\dummy.f' does not exist
In-Reply-To: <4950484D.5030402@ar.media.kyoto-u.ac.jp>
References: <1cd32cbb0812220837ofcf4f6etd8d4ede456b0cfe2@mail.gmail.com>
	<1cd32cbb0812221051s6d566f11o6d08ddd6264724c5@mail.gmail.com>
	<4950484D.5030402@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0812230710j73eab71aw91486ed5751dd2f9@mail.gmail.com>

On Mon, Dec 22, 2008 at 9:09 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> josef.pktd at gmail.com wrote:
>> On Mon, Dec 22, 2008 at 11:37 AM,  <josef.pktd at gmail.com> wrote:
>>
>>> I was trying to build the current scipy trunk, r5286, and got stuck with arpack
>>>
>>> this is the last part of the build log
>>>
>>> building 'arpack' library
>>> compiling Fortran sources
>>> Fortran f77 compiler: C:\Programs\MinGW\bin\g77.exe -g -Wall -fno-second-undersc
>>> ore -mno-cygwin -O3 -funroll-loops
>>> error: file 'ARPACK\FWRAPPERS\dummy.f' does not exist
>>>
>>> WindowsXP, mingw 3.4,
>>> svn export
>>> python25 setup.py bdist
>>>
>>>>>> np.version.version
>>>>>>
>>> '1.3.0.dev6139'
>>>
>>> I don't see any ARPACK\FWRAPPERS directory anywhere.
>>>
>>> Josef
>>>
>>>
>>
>> All clear,
>>
>> `svn update` didn't download the fwrappers directory, I had to
>> explicitely specify the missing directory for update.
>>
>
> That's strange. Generally, the problem is with removed directories - I
> never had problems updating new directories. Which version of subversion
> are you using ?
>
> David
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>

I'm using svn 1.4.6, using both command line and pysvn workbench,  I
think, I also have a bugfix from bzr-svn.  I also had one broken
install of svn 1.5  beta, which I had to delete. Usually I don't have
any problems at all, but I will need to upgrade svn soon, (also to
upgrade to newer bzr) and get a new clean checkout.

Josef


From josef.pktd at gmail.com  Tue Dec 23 10:41:23 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 23 Dec 2008 10:41:23 -0500
Subject: [SciPy-dev] help with xplot->matplotlib translation
Message-ID: <1cd32cbb0812230741r6e587310xab80d5eb2ce1fa9d@mail.gmail.com>

stats.morestats contains several function creating plots with xplot

    if plot is not None:
        try:
            import scipy.xplt as xplt
            xplt.limits()
        except: pass
        plot.plot(svals, ppcc, 'x')
        plot.title('(%s) PPCC Plot' % dist)
        plot.xlabel('Prob Plot Corr. Coef.'),deltay=-0.01)
        plot.ylabel('Shape Values'),deltax=-0.01)
        try: plot.expand_limits(5)
        except: pass

I would like to translate this to using plot=matplotlib.pylab, but I
don't know anything about xplt and I am not very good with matplotlib.

So, if someone familiar with these, can provide a translation for
xplt.limits(), deltay=-0.01, deltax and expand_limits, then I could
fix these bugs without hunting documentation for hours. If I delete
deltay and deltax, then the function works, but I don't know if I get
the indended graph.

Thank you,

Josef


From oliphant at enthought.com  Tue Dec 23 16:02:56 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Tue, 23 Dec 2008 15:02:56 -0600
Subject: [SciPy-dev] help with xplot->matplotlib translation
In-Reply-To: <1cd32cbb0812230741r6e587310xab80d5eb2ce1fa9d@mail.gmail.com>
References: <1cd32cbb0812230741r6e587310xab80d5eb2ce1fa9d@mail.gmail.com>
Message-ID: <49515200.3030408@enthought.com>

josef.pktd at gmail.com wrote:
> stats.morestats contains several function creating plots with xplot
>
>     if plot is not None:
>         try:
>             import scipy.xplt as xplt
>             xplt.limits()
>         except: pass
>         plot.plot(svals, ppcc, 'x')
>         plot.title('(%s) PPCC Plot' % dist)
>         plot.xlabel('Prob Plot Corr. Coef.'),deltay=-0.01)
>         plot.ylabel('Shape Values'),deltax=-0.01)
>         try: plot.expand_limits(5)
>         except: pass
>
> I would like to translate this to using plot=matplotlib.pylab, but I
> don't know anything about xplt and I am not very good with matplotlib.
>
> So, if someone familiar with these, can provide a translation for
> xplt.limits(), deltay=-0.01, deltax and expand_limits, then I could
> fix these bugs without hunting documentation for hours. If I delete
> deltay and deltax, then the function works, but I don't know if I get
> the indended graph.
>
>   
deltax and deltay are just terms to move the placement of text in the 
horizontal and vertical direction.   You can delete them without any 
concern. 

The expand_limits function can also be removed without concern as it is 
another "formatting" piece that tries and makes the graph look more 
pleasing.

Thanks for fixing things.

-Travis



From fperez.net at gmail.com  Sat Dec 27 21:34:58 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Sat, 27 Dec 2008 18:34:58 -0800
Subject: [SciPy-dev] Numerical Recipes (was tagging 0.7rc1 this weekend?)
In-Reply-To: <4947CA8C.40008@american.edu>
References: <c7009a550812031235r2e82d7f1md9357d6472c1b9d6@mail.gmail.com>
	<c7009a550812152259h5aad0af6t6f508ec03fad6878@mail.gmail.com>
	<4947CA8C.40008@american.edu>
Message-ID: <db6b5ecc0812271834s658101l1bccd32c117ab912@mail.gmail.com>

Howdy,

On Tue, Dec 16, 2008 at 7:34 AM, Alan G Isaac <aisaac at american.edu> wrote:

>>> See Numerical Recipies in C, page 156 and
>>> Abramowitz and Stegun p. 774, 782
>
>
>
> Travis O. is listed as writing this orthogonal.py in 2000
> and doing bug fixes in 2003.  Note that the
> reference to Numerical Recipes is one of two
> references documenting an algebraic recursion
> relation and is not referring to an implementation.
>
> http://www.fizyka.umk.pl/nrbook/c4-5.pdf
>
> http://www-lsp.ujf-grenoble.fr/recherche/a3t2/a3t2a2/bahram/biblio/abramowitz_and_stegun/page_774.htm
>
> I do not see any problem with the code.
> I think this one should also be considered resolved.

I concur.  I also had a look at the Golub&Welsch reference pointed to
by Chuck, and I don't see any problems.  The basic algorithm may be
the same, but the actual implementation in those chapters of NR is
fairly different from what's in special/orthogonal.py.  The Scipy code
is structured as a bunch of routines that provide quite a few
different orthogonal polyinomial objects, while NR only lists code for
a few specific cases where the weights and abscissas are being
computed (gauleg, gaulag, gauher, gaujac), plus the gaucof() that
implements the Golub-Welsch algorithm for generic abscissas and
weights.

In orthogonal.py, the closest pairing is between  NR's gaucof() and
gen_roots_and_weights(), but their actual codes are fairly different
to the extent possible. That is, it's a short code that does a very
specific thing, but the scipy code does it in a very python/numpy
specific way, while the NR code is very 'C'-ish.

Other than the fact that these codes both use the same piece of
mathematics, I see no problem here whatsoever.  It might be worth
clarifying in the docstring that reads

"""
See Numerical Recipies in C, page 156 and
Abramowitz and Stegun p. 774, 782
"""

that the NR reference is to the formulas only,  not to any actual
implementation.  If nothing else, to prevent any questions in the
future...

Cheers,

f


From millman at berkeley.edu  Mon Dec 29 05:02:31 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 29 Dec 2008 02:02:31 -0800
Subject: [SciPy-dev] tagging 0.7.0rc1 this weekend? (was Numerical Recipes
	...)
Message-ID: <c7009a550812290202u57d571c1tf868efbdffd9e578@mail.gmail.com>

The NR code review is basically complete.  All the NR copyright issues
have been resolved (the only thing left is to verify that the
docstrings clearly explain the code's relation to NR).  A big thanks
to everyone who helped resolve the NR issues!

Since this was the only thing blocking the release candidate, I am
planning to create a 0.7.x branch and tag 0.7.0rc1 at the end of the
weekend.  Once I branch for 0.7.x, the trunk will be open for 0.8
development.

Cheers,
Jarrod


From millman at berkeley.edu  Wed Dec 31 01:28:38 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 30 Dec 2008 22:28:38 -0800
Subject: [SciPy-dev] almost doubled the number of tests
Message-ID: <c7009a550812302228lbfe9e9as5a602100a8c68ee4@mail.gmail.com>

In preparation for the upcoming release, I was comparing the number of
tests in 0.6.0 to the number on the trunk:
  0.6.0:  Ran 2266 tests in 203.503s
  Trunk: Ran 3990 tests in 392.357s

I knew that a large number of test had been written over the last
year, but I was very pleased to see that we nearly doubled the number.
 Thanks to everyone who contributed new tests over the last year.
Increasing our test coverage is extremely important.   Importantly, it
makes it easier to refactor the code without worrying about
regressions creeping in without being noticed.  Hopefully, this trend
will continue and we will add nearly 2000 new tests over the next
year.

Jarrod