From pav at iki.fi  Tue Sep  1 04:47:36 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 1 Sep 2009 08:47:36 +0000 (UTC)
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com>
Message-ID: <h7in38$ejr$1@ger.gmane.org>

Mon, 31 Aug 2009 23:03:28 -0400, josef.pktd kirjoitti:
> Is it possible to have a source repository that is not located in svn on
> scipy.org?
> 
> I haven't seen an example for non svn repositories, e.g. Davids audiolab
> and talkbox still link to the svn, other non-svn have the source
> repository link missing.
> 
> For example, I would like to have "bzr branch lp:statsmodels" instead of
> "svn checkout"

The "svn checkout" part seems to be hardcoded in the application, and 
only currently works for stuff in scipy.org Scikit SVN. Cf.

	http://code.google.com/p/scikits-index/source/checkout

This is probably possible to change with a bit of a work. I might be 
interested in taking a shot at improving the app at some point.

-- 
Pauli Virtanen



From josef.pktd at gmail.com  Tue Sep  1 09:41:03 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 1 Sep 2009 09:41:03 -0400
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
In-Reply-To: <h7in38$ejr$1@ger.gmane.org>
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com>
	<h7in38$ejr$1@ger.gmane.org>
Message-ID: <1cd32cbb0909010641p3375a6c0h1c22d800d1c03829@mail.gmail.com>

On Tue, Sep 1, 2009 at 4:47 AM, Pauli Virtanen<pav at iki.fi> wrote:
> Mon, 31 Aug 2009 23:03:28 -0400, josef.pktd kirjoitti:
>> Is it possible to have a source repository that is not located in svn on
>> scipy.org?
>>
>> I haven't seen an example for non svn repositories, e.g. Davids audiolab
>> and talkbox still link to the svn, other non-svn have the source
>> repository link missing.
>>
>> For example, I would like to have "bzr branch lp:statsmodels" instead of
>> "svn checkout"
>
> The "svn checkout" part seems to be hardcoded in the application, and
> only currently works for stuff in scipy.org Scikit SVN. Cf.
>
> ? ? ? ?http://code.google.com/p/scikits-index/source/checkout
>
> This is probably possible to change with a bit of a work. I might be
> interested in taking a shot at improving the app at some point.
>
> --
> Pauli Virtanen

Thanks,
I just wanted to know. Skipper received a comment about this, and I
wasn't sure whether I missed some category or option on pypi.

Josef


>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From nwagner at iam.uni-stuttgart.de  Tue Sep  1 13:41:15 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 01 Sep 2009 19:41:15 +0200
Subject: [SciPy-dev] scipy.test() segfaults
Message-ID: <web-124038375@uni-stuttgart.de>


Prefactorize (with UMFPACK) matrix for solving with 
multiple rhs ...
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 46980518117776 (LWP 11237)]
PyArray_INCREF (mp=0xc) at 
numpy/core/src/multiarray/refcount.c:103
103         if (!PyDataType_REFCHK(mp->descr)) {
(gdb) bt
#0  PyArray_INCREF (mp=0xc) at 
numpy/core/src/multiarray/refcount.c:103
#1  0x00002aba96b6a4df in helper_getCArrayObject 
(input=0x2b0c090, type=12, minDim=1, maxDim=1)
     at 
build/src.linux-x86_64-2.5/scikits/umfpack/_umfpack_wrap.c:2535
#2  0x00002aba96b6beab in _wrap_umfpack_di_defaults 
(self=<value optimized out>, args=<value optimized out>)
     at 
build/src.linux-x86_64-2.5/scikits/umfpack/_umfpack_wrap.c:3054
  
>>> import scipy
>>> scipy.__version__
'0.8.0.dev5838'
>>> import numpy
>>> numpy.__version__
'1.4.0.dev7353'

Linux diego 2.6.18.8-0.13-default #1 SMP Thu Nov 6 
13:35:13 UTC 2008 x86_64 x86_64 x86_64 GNU/Linux



From jantod at gmail.com  Wed Sep  2 20:37:14 2009
From: jantod at gmail.com (Janto Dreijer)
Date: Thu, 3 Sep 2009 02:37:14 +0200
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
In-Reply-To: <9457e7c80909021514x57fcd3eeu476a02e472b264c1@mail.gmail.com>
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com>
	<9457e7c80909021514x57fcd3eeu476a02e472b264c1@mail.gmail.com>
Message-ID: <cbc7eb430909021737l2554915cq90c202c741451eb4@mail.gmail.com>

Hi Josef!

It's almost 3am over here and about to fall asleep, so please excuse
any grammatical curiosities.

I wrote the scikits website and would just like to confirm some of
your suspicions.

As you probably noticed your package automatically appeared at
http://scikits.appspot.com/statsmodels because it is listed on PyPI.

My initial design thoughts were to have the scikits site piggy-back on
PyPI. It's currently configured to search PyPI for packages with
"scikits" in its name. It also scans
http://svn.scipy.org/svn/scikits/trunk for packages. And yes,
unfortunately, it does not seem as if PyPI has any field for
repository information.

Whether any of this should change, is in my opinion, debatable. The
scikits website is currently supposed to just be an index of available
packages. PyPI looks to be focused on listing "versioned" releases for
easy_install and not active repository code.

So I think the best place for repo checkout instructions are on the
package's own website. If you have different ideas on this, I'd be
happy to reconsider and change things.

Regards
Janto


From josef.pktd at gmail.com  Wed Sep  2 21:06:22 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 2 Sep 2009 21:06:22 -0400
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
In-Reply-To: <cbc7eb430909021737l2554915cq90c202c741451eb4@mail.gmail.com>
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com>
	<9457e7c80909021514x57fcd3eeu476a02e472b264c1@mail.gmail.com>
	<cbc7eb430909021737l2554915cq90c202c741451eb4@mail.gmail.com>
Message-ID: <1cd32cbb0909021806m54806c3ao97b11b7c3f7060fc@mail.gmail.com>

On Wed, Sep 2, 2009 at 8:37 PM, Janto Dreijer<jantod at gmail.com> wrote:
> Hi Josef!
>
> It's almost 3am over here and about to fall asleep, so please excuse
> any grammatical curiosities.
>
> I wrote the scikits website and would just like to confirm some of
> your suspicions.
>
> As you probably noticed your package automatically appeared at
> http://scikits.appspot.com/statsmodels because it is listed on PyPI.
>
> My initial design thoughts were to have the scikits site piggy-back on
> PyPI. It's currently configured to search PyPI for packages with
> "scikits" in its name. It also scans
> http://svn.scipy.org/svn/scikits/trunk for packages. And yes,
> unfortunately, it does not seem as if PyPI has any field for
> repository information.
>
> Whether any of this should change, is in my opinion, debatable. The
> scikits website is currently supposed to just be an index of available
> packages. PyPI looks to be focused on listing "versioned" releases for
> easy_install and not active repository code.
>
> So I think the best place for repo checkout instructions are on the
> package's own website. If you have different ideas on this, I'd be
> happy to reconsider and change things.
>
> Regards
> Janto

If there is no easy way of getting the repository information into it,
it's ok with me.
Since it is picking up the description from pypi it is possible to
include additional
information.

However, in the case of no repository information, removing "svn checkout" would
look better and would not raise the comment, that we received about this.

Source code
You can get the latest sources from the repository using
svn checkout

And maybe also remove conditionally the empty " Source Repository:  "
line so that
it doesn't look so unfinished.

Overall relying on a fixed pattern doesn't seem to work well, for
example pymat, which is only
in svn has a link to pypi :

You can download the latest distribution from PyPI here:
http://pypi.python.org/pypi/scikits.pymat

A scikit that is under development and doesn't have yet a pypi
release, and is not under
svn in scipy.org, doesn't seem to have a way to get listed.
This wasn't really relevant for us, because we only decided two weeks
ago, that we will distribute
statsmodels as a scikits.

So, if development pattern become more diverse, then a more flexible
way of getting the correct
information into scikits.appspot seems to be necessary.

On the other hand, being very quickly picked up after the pypi upload
without any additional work
was very nice.

For us this would be just cosmetic changes, so it doesn't bother me much.

Josef


From stefan at sun.ac.za  Thu Sep  3 01:57:33 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 3 Sep 2009 07:57:33 +0200
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
In-Reply-To: <1cd32cbb0909021806m54806c3ao97b11b7c3f7060fc@mail.gmail.com>
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com> 
	<9457e7c80909021514x57fcd3eeu476a02e472b264c1@mail.gmail.com> 
	<cbc7eb430909021737l2554915cq90c202c741451eb4@mail.gmail.com> 
	<1cd32cbb0909021806m54806c3ao97b11b7c3f7060fc@mail.gmail.com>
Message-ID: <9457e7c80909022257p1973cb5aj31b07181d41bbaaf@mail.gmail.com>

Hi Josef

2009/9/3 josef.pktd at gmail.com <josef.pktd at gmail.com>:
> If there is no easy way of getting the repository information into it,
> it's ok with me.

Would it be possible to use the download-url field to specify the
repository location?

Regards
St?fan


From david at ar.media.kyoto-u.ac.jp  Thu Sep  3 01:44:42 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 03 Sep 2009 14:44:42 +0900
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
In-Reply-To: <9457e7c80909022257p1973cb5aj31b07181d41bbaaf@mail.gmail.com>
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com>
	<9457e7c80909021514x57fcd3eeu476a02e472b264c1@mail.gmail.com>
	<cbc7eb430909021737l2554915cq90c202c741451eb4@mail.gmail.com>
	<1cd32cbb0909021806m54806c3ao97b11b7c3f7060fc@mail.gmail.com>
	<9457e7c80909022257p1973cb5aj31b07181d41bbaaf@mail.gmail.com>
Message-ID: <4A9F57CA.1020704@ar.media.kyoto-u.ac.jp>

St?fan van der Walt wrote:
> Hi Josef
>
> 2009/9/3 josef.pktd at gmail.com <josef.pktd at gmail.com>:
>   
>> If there is no easy way of getting the repository information into it,
>> it's ok with me.
>>     
>
> Would it be possible to use the download-url field to specify the
> repository location?
>   

I am afraid this may break easy-install, as easy-install looks for
tarballs at the download-url meta-data if the package is not found in pypi.

I wonder if we should not get our own pypi at some point, with our own
extended metadata ?

cheers,

David


From robert.kern at gmail.com  Thu Sep  3 02:13:50 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 3 Sep 2009 01:13:50 -0500
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
In-Reply-To: <4A9F57CA.1020704@ar.media.kyoto-u.ac.jp>
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com> 
	<9457e7c80909021514x57fcd3eeu476a02e472b264c1@mail.gmail.com> 
	<cbc7eb430909021737l2554915cq90c202c741451eb4@mail.gmail.com> 
	<1cd32cbb0909021806m54806c3ao97b11b7c3f7060fc@mail.gmail.com> 
	<9457e7c80909022257p1973cb5aj31b07181d41bbaaf@mail.gmail.com> 
	<4A9F57CA.1020704@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730909022313g2f45e028k714cb35f42db20d3@mail.gmail.com>

On Thu, Sep 3, 2009 at 00:44, David
Cournapeau<david at ar.media.kyoto-u.ac.jp> wrote:
> St?fan van der Walt wrote:
>> Hi Josef
>>
>> 2009/9/3 josef.pktd at gmail.com <josef.pktd at gmail.com>:
>>
>>> If there is no easy way of getting the repository information into it,
>>> it's ok with me.
>>>
>>
>> Would it be possible to use the download-url field to specify the
>> repository location?
>>
>
> I am afraid this may break easy-install, as easy-install looks for
> tarballs at the download-url meta-data if the package is not found in pypi.

http://peak.telecommunity.com/DevCenter/setuptools

"""
So, if your url or download_url point either directly to a
downloadable source distribution, or to HTML page(s) that have direct
links to such, then EasyInstall will be able to locate downloads
automatically. If you want to make Subversion checkouts available,
then you should create links with either #egg=project or
#egg=project-version added to the URL. You should replace project and
version with the values they would have in an egg filename. (Be sure
to actually generate an egg and then use the initial part of the
filename, rather than trying to guess what the escaped form of the
project name and version number will be.)
"""

It won't break anything. easy_install can be dumb at times, but it's
not that dumb. SVN repo download_urls existed on PyPI before
setuptools was written.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Thu Sep  3 02:34:29 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 3 Sep 2009 08:34:29 +0200
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
In-Reply-To: <4A9F57CA.1020704@ar.media.kyoto-u.ac.jp>
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com> 
	<9457e7c80909021514x57fcd3eeu476a02e472b264c1@mail.gmail.com> 
	<cbc7eb430909021737l2554915cq90c202c741451eb4@mail.gmail.com> 
	<1cd32cbb0909021806m54806c3ao97b11b7c3f7060fc@mail.gmail.com> 
	<9457e7c80909022257p1973cb5aj31b07181d41bbaaf@mail.gmail.com> 
	<4A9F57CA.1020704@ar.media.kyoto-u.ac.jp>
Message-ID: <9457e7c80909022334o7e91fe7cw634d66a3c9a31ea2@mail.gmail.com>

2009/9/3 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>> Would it be possible to use the download-url field to specify the
>> repository location?
>>
>
> I am afraid this may break easy-install, as easy-install looks for
> tarballs at the download-url meta-data if the package is not found in pypi.
>
> I wonder if we should not get our own pypi at some point, with our own
> extended metadata ?

That wouldn't be too hard, given that most of the work has been done:

http://chrisarndt.de/projects/eggbasket/

Unfortunately, the SciPy "Roll Your Own" and "Infrastructure
Maintenance" teams are understaffed and overworked.

Cheers
St?fan


From david at ar.media.kyoto-u.ac.jp  Thu Sep  3 02:21:26 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 03 Sep 2009 15:21:26 +0900
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
In-Reply-To: <3d375d730909022313g2f45e028k714cb35f42db20d3@mail.gmail.com>
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com>
	<9457e7c80909021514x57fcd3eeu476a02e472b264c1@mail.gmail.com>
	<cbc7eb430909021737l2554915cq90c202c741451eb4@mail.gmail.com>
	<1cd32cbb0909021806m54806c3ao97b11b7c3f7060fc@mail.gmail.com>
	<9457e7c80909022257p1973cb5aj31b07181d41bbaaf@mail.gmail.com>
	<4A9F57CA.1020704@ar.media.kyoto-u.ac.jp>
	<3d375d730909022313g2f45e028k714cb35f42db20d3@mail.gmail.com>
Message-ID: <4A9F6066.7040600@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> On Thu, Sep 3, 2009 at 00:44, David
> Cournapeau<david at ar.media.kyoto-u.ac.jp> wrote:
>   
>> St?fan van der Walt wrote:
>>     
>>> Hi Josef
>>>
>>> 2009/9/3 josef.pktd at gmail.com <josef.pktd at gmail.com>:
>>>
>>>       
>>>> If there is no easy way of getting the repository information into it,
>>>> it's ok with me.
>>>>
>>>>         
>>> Would it be possible to use the download-url field to specify the
>>> repository location?
>>>
>>>       
>> I am afraid this may break easy-install, as easy-install looks for
>> tarballs at the download-url meta-data if the package is not found in pypi.
>>     
>
> http://peak.telecommunity.com/DevCenter/setuptools
>
> """
> So, if your url or download_url point either directly to a
> downloadable source distribution, or to HTML page(s) that have direct
> links to such, then EasyInstall will be able to locate downloads
> automatically. If you want to make Subversion checkouts available,
> then you should create links with either #egg=project or
> #egg=project-version added to the URL. You should replace project and
> version with the values they would have in an egg filename. (Be sure
> to actually generate an egg and then use the initial part of the
> filename, rather than trying to guess what the escaped form of the
> project name and version number will be.)
> """
>   

I am afraid I don't quite understand the above paragraph. Does it mean
you can have both tarball and checkout made available through this
mechanism, as long as download_url contain link to both of them ?

cheers,

David


From robert.kern at gmail.com  Thu Sep  3 03:10:17 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 3 Sep 2009 02:10:17 -0500
Subject: [SciPy-dev] source repository in http://scikits.appspot.com/
In-Reply-To: <4A9F6066.7040600@ar.media.kyoto-u.ac.jp>
References: <1cd32cbb0908312003g5b89b99cn8394dd9c3970f2e0@mail.gmail.com> 
	<9457e7c80909021514x57fcd3eeu476a02e472b264c1@mail.gmail.com> 
	<cbc7eb430909021737l2554915cq90c202c741451eb4@mail.gmail.com> 
	<1cd32cbb0909021806m54806c3ao97b11b7c3f7060fc@mail.gmail.com> 
	<9457e7c80909022257p1973cb5aj31b07181d41bbaaf@mail.gmail.com> 
	<4A9F57CA.1020704@ar.media.kyoto-u.ac.jp>
	<3d375d730909022313g2f45e028k714cb35f42db20d3@mail.gmail.com> 
	<4A9F6066.7040600@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730909030010l4dc7bce3n21c0e3eafc816138@mail.gmail.com>

2009/9/3 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> Robert Kern wrote:
>> On Thu, Sep 3, 2009 at 00:44, David
>> Cournapeau<david at ar.media.kyoto-u.ac.jp> wrote:
>>
>>> St?fan van der Walt wrote:
>>>
>>>> Hi Josef
>>>>
>>>> 2009/9/3 josef.pktd at gmail.com <josef.pktd at gmail.com>:
>>>>
>>>>
>>>>> If there is no easy way of getting the repository information into it,
>>>>> it's ok with me.
>>>>>
>>>>>
>>>> Would it be possible to use the download-url field to specify the
>>>> repository location?
>>>>
>>>>
>>> I am afraid this may break easy-install, as easy-install looks for
>>> tarballs at the download-url meta-data if the package is not found in pypi.
>>>
>>
>> http://peak.telecommunity.com/DevCenter/setuptools
>>
>> """
>> So, if your url or download_url point either directly to a
>> downloadable source distribution, or to HTML page(s) that have direct
>> links to such, then EasyInstall will be able to locate downloads
>> automatically. If you want to make Subversion checkouts available,
>> then you should create links with either #egg=project or
>> #egg=project-version added to the URL. You should replace project and
>> version with the values they would have in an egg filename. (Be sure
>> to actually generate an egg and then use the initial part of the
>> filename, rather than trying to guess what the escaped form of the
>> project name and version number will be.)
>> """
>>
>
> I am afraid I don't quite understand the above paragraph. Does it mean
> you can have both tarball and checkout made available through this
> mechanism, as long as download_url contain link to both of them ?

http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api

As far as easy_install is concerned, it is only looking at the web
page and scraping links with the rel="download" attribute. It doesn't
care whether that is in the "Download URL" spot of the official PyPI
metadata or not.

The download_url metadata of distutils can only accept one URL,
though. One option is to have the Homepage have links to the release
tarballs and the Download URL point to the SVN repository, and also
upload release tarballs to PyPI where possible.

Here is one example of a project that has an SVN repo as its Download URL:

  http://pypi.python.org/pypi/grokproject

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Mon Sep  7 12:35:17 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 7 Sep 2009 12:35:17 -0400
Subject: [SciPy-dev] ANN: new google group: pystatsmodels
Message-ID: <1cd32cbb0909070935t4618e971uab966b142d857f1e@mail.gmail.com>

After the initial release of scikits.statsmodels, we started to have
email discussions about the design, implementation and extension of
statsmodels, as well as about datahandling and how to use statsmodels
for different types of data. Since this discussion seemed to
specialized to fill up scipy-dev with it, we decided to start a
dedicated google group.

The overall objective is to make python easier to use for statistical
and econometric analysis, and the discussion group is open to and
welcomes the discussion of related packages. The first example is
pandas, a package for handling panel data that uses statsmodels for
estimation.

Anyone interested, is welcome to join us at

http://groups.google.ca/group/pystatsmodels/topics?hl=en

Josef et al


From robert.kern at gmail.com  Mon Sep  7 23:44:48 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 7 Sep 2009 22:44:48 -0500
Subject: [SciPy-dev] ANN: new google group: pystatsmodels
In-Reply-To: <1cd32cbb0909070935t4618e971uab966b142d857f1e@mail.gmail.com>
References: <1cd32cbb0909070935t4618e971uab966b142d857f1e@mail.gmail.com>
Message-ID: <3d375d730909072044t33345b87t9e9a3683fb5a01f3@mail.gmail.com>

On Mon, Sep 7, 2009 at 11:35, <josef.pktd at gmail.com> wrote:
> After the initial release of scikits.statsmodels, we started to have
> email discussions about the design, implementation and extension of
> statsmodels, as well as about datahandling and how to use statsmodels
> for different types of data. Since this discussion seemed to
> specialized to fill up scipy-dev with it, we decided to start a
> dedicated google group.

Maybe it's too late, but this is not at all too specialized for
scipy-dev. We explicitly welcome all discussions about any scikit
here, especially those aimed at inclusion into scipy. Keeping the
discussion here may help people searching for information at a later
date, particularly with respect to design decisions made early on.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From bsouthey at gmail.com  Tue Sep  8 09:23:40 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Tue, 08 Sep 2009 08:23:40 -0500
Subject: [SciPy-dev] ANN: new google group: pystatsmodels
In-Reply-To: <3d375d730909072044t33345b87t9e9a3683fb5a01f3@mail.gmail.com>
References: <1cd32cbb0909070935t4618e971uab966b142d857f1e@mail.gmail.com>
	<3d375d730909072044t33345b87t9e9a3683fb5a01f3@mail.gmail.com>
Message-ID: <4AA65ADC.2070507@gmail.com>

On 09/07/2009 10:44 PM, Robert Kern wrote:
> On Mon, Sep 7, 2009 at 11:35,<josef.pktd at gmail.com>  wrote:
>    
>> After the initial release of scikits.statsmodels, we started to have
>> email discussions about the design, implementation and extension of
>> statsmodels, as well as about datahandling and how to use statsmodels
>> for different types of data. Since this discussion seemed to
>> specialized to fill up scipy-dev with it, we decided to start a
>> dedicated google group.
>>      
> Maybe it's too late, but this is not at all too specialized for
> scipy-dev. We explicitly welcome all discussions about any scikit
> here, especially those aimed at inclusion into scipy. Keeping the
> discussion here may help people searching for information at a later
> date, particularly with respect to design decisions made early on.
>
>    
I tend to agree that scipy-users and scipy-dev should be able to handle 
a large part of the context proposed. I find that design and  
implementation discussions help others develop their own software and 
datahandling almost is required by all areas. But I do recognize that 
certain aspects are very specific and that could be a basis for a 
separate list, perhaps along the lines of the separation between the 
scipy-users and scipy-dev lists.

Bruce


From josef.pktd at gmail.com  Tue Sep  8 10:27:19 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 8 Sep 2009 10:27:19 -0400
Subject: [SciPy-dev] ANN: new google group: pystatsmodels
In-Reply-To: <4AA65ADC.2070507@gmail.com>
References: <1cd32cbb0909070935t4618e971uab966b142d857f1e@mail.gmail.com>
	<3d375d730909072044t33345b87t9e9a3683fb5a01f3@mail.gmail.com>
	<4AA65ADC.2070507@gmail.com>
Message-ID: <1cd32cbb0909080727y43c9ed83i453b56274c111a6b@mail.gmail.com>

On Tue, Sep 8, 2009 at 9:23 AM, Bruce Southey<bsouthey at gmail.com> wrote:
> On 09/07/2009 10:44 PM, Robert Kern wrote:
>> On Mon, Sep 7, 2009 at 11:35,<josef.pktd at gmail.com> ?wrote:
>>
>>> After the initial release of scikits.statsmodels, we started to have
>>> email discussions about the design, implementation and extension of
>>> statsmodels, as well as about datahandling and how to use statsmodels
>>> for different types of data. Since this discussion seemed to
>>> specialized to fill up scipy-dev with it, we decided to start a
>>> dedicated google group.
>>>
>> Maybe it's too late, but this is not at all too specialized for
>> scipy-dev. We explicitly welcome all discussions about any scikit
>> here, especially those aimed at inclusion into scipy. Keeping the
>> discussion here may help people searching for information at a later
>> date, particularly with respect to design decisions made early on.
>>
>>
> I tend to agree that scipy-users and scipy-dev should be able to handle
> a large part of the context proposed. I find that design and
> implementation discussions help others develop their own software and
> datahandling almost is required by all areas. But I do recognize that
> certain aspects are very specific and that could be a basis for a
> separate list, perhaps along the lines of the separation between the
> scipy-users and scipy-dev lists.

At the end of the gsoc, in preparation for the first scikits release and
to a slightly lesser extend throughout the summer, Skipper and I had a
huge mail traffic. After the release, this has slowed down since we are
both busy with other things, but we started to have detailed discussion
about improvements to the design and extensions. We wanted to move
the 2 or 3 way mailing threads to a public forum, but I felt, that we have
more freedom in producing long, "boring" mail threads on a dedicated
list than on the general purpose scipy lists.

The second reason was, that we started to discuss pandas, which
looks like a good package for data handling, used mainly in finance,
and which will have its first release in the near future.
I'm also keeping an eye on some other packages that might
become useful for econometrics (also BSD or compatible).
So, we preferred a theme oriented mailing list that gives as more
freedom to also discuss packages that are not directly related to
the use and development of scipy, the package.

These were our initial intentions, how it will turn out, we will see.

For me, it's not always easy to tell which mailing list is the best
location for a discussion (which of course might not improve if
there is one more).

Josef

>
> Bruce
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From fperez.net at gmail.com  Tue Sep  8 13:38:36 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 8 Sep 2009 10:38:36 -0700
Subject: [SciPy-dev] ANN: new google group: pystatsmodels
In-Reply-To: <1cd32cbb0909080727y43c9ed83i453b56274c111a6b@mail.gmail.com>
References: <1cd32cbb0909070935t4618e971uab966b142d857f1e@mail.gmail.com> 
	<3d375d730909072044t33345b87t9e9a3683fb5a01f3@mail.gmail.com> 
	<4AA65ADC.2070507@gmail.com>
	<1cd32cbb0909080727y43c9ed83i453b56274c111a6b@mail.gmail.com>
Message-ID: <db6b5ecc0909081038i2ad9a3afw645b0f23f657094@mail.gmail.com>

On Tue, Sep 8, 2009 at 7:27 AM,  <josef.pktd at gmail.com> wrote:
> about improvements to the design and extensions. We wanted to move
> the 2 or 3 way mailing threads to a public forum, but I felt, that we have
> more freedom in producing long, "boring" mail threads on a dedicated
> list than on the general purpose scipy lists.
>
> The second reason was, that we started to discuss pandas, which
> looks like a good package for data handling, used mainly in finance,
> and which will have its first release in the near future.
> I'm also keeping an eye on some other packages that might
> become useful for econometrics (also BSD or compatible).

FWIW, I personally like the fact that other 'scipy ecosystem' packages
can and are discussed here, since it means I learn a lot without
having to track yet another separate mailing list.

I really appreciate your tact and courtesy in trying to not 'pollute'
the scipy lists with material that you feel may be somewhat off-topic.
 But I think that, given how all of us have become pretty good at just
ignoring threads we aren't keeping track of directly, in general you
would not find too much push-back from this list.

So at least from me, +1 for you guys keeping your discussions here,
where I might actually learn something  :)

Best regards,

f


From c.schmidt-hieber at ucl.ac.uk  Tue Sep  8 13:44:07 2009
From: c.schmidt-hieber at ucl.ac.uk (Christoph Schmidt-Hieber)
Date: Tue, 8 Sep 2009 18:44:07 +0100
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
Message-ID: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>

Dear all,
I've started a Google code project (http://code.google.com/p/pypraxis/) to provide a Python interface to Brent's principal axis algorithm. It's basically a wrapper around some Fortran code from http://www.netlib.org/opt/. Brent's algorithm minimizes a function of several variables without calculating derivatives - not to be mistaken for scipy.optimize.brent, that only performs single-variable minimization. The algorithm typically outperforms other derivative- and gradient-free algorithms (Brent, 2002; http://wwwmaths.anu.edu.au/~brent/pub/pub011.html). In my experience, it converges substantially faster than fmin and fmin_powell from scipy.optimize when fitting models with 5 to 15 free parameters to experimental data. Notably, Mathematica uses this algorithm for minimization without derivatives
(http://reference.wolfram.com/mathematica/tutorial/UnconstrainedOptimizationPrincipalAxisMethod.html).
I've provided some test cases and a wrapper that allow to compare it directly to the existing algorithms from scipy.optimize. Let me know if you think that the code could be a candidate for integration into scipy.optimize. It would obviously require some work to make it conform with the other functions that are already present.
Best
Christoph

-----WIBR-EMAIL-----
This email and any files transmitted with it
are confidential and intended solely for the
use of the individual or entity to whom they
are addressed. If you have received this 
email in error please notify the system 
management (it.support at wibr.ucl.ac.uk).
--------***---------




From robert.kern at gmail.com  Tue Sep  8 15:46:59 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 8 Sep 2009 14:46:59 -0500
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
In-Reply-To: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
References: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
Message-ID: <3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com>

On Tue, Sep 8, 2009 at 12:44, Christoph
Schmidt-Hieber<c.schmidt-hieber at ucl.ac.uk> wrote:
> Dear all,
> I've started a Google code project (http://code.google.com/p/pypraxis/) to provide a Python interface to Brent's principal axis algorithm. It's basically a wrapper around some Fortran code from http://www.netlib.org/opt/. Brent's algorithm minimizes a function of several variables without calculating derivatives - not to be mistaken for scipy.optimize.brent, that only performs single-variable minimization. The algorithm typically outperforms other derivative- and gradient-free algorithms (Brent, 2002; http://wwwmaths.anu.edu.au/~brent/pub/pub011.html). In my experience, it converges substantially faster than fmin and fmin_powell from scipy.optimize when fitting models with 5 to 15 free parameters to experimental data. Notably, Mathematica uses this algorithm for minimization without derivatives
> (http://reference.wolfram.com/mathematica/tutorial/UnconstrainedOptimizationPrincipalAxisMethod.html).
> I've provided some test cases and a wrapper that allow to compare it directly to the existing algorithms from scipy.optimize. Let me know if you think that the code could be a candidate for integration into scipy.optimize. It would obviously require some work to make it conform with the other functions that are already present.

That would be great! Unfortunately, there is no license attached to
praxis.f, so it cannot be integrated into scipy until we find a
suitably licensed implementation of the algorithm.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From efiring at hawaii.edu  Tue Sep  8 20:28:42 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Tue, 08 Sep 2009 14:28:42 -1000
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
In-Reply-To: <3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com>
References: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
	<3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com>
Message-ID: <4AA6F6BA.30402@hawaii.edu>

Robert Kern wrote:
> On Tue, Sep 8, 2009 at 12:44, Christoph
> Schmidt-Hieber<c.schmidt-hieber at ucl.ac.uk> wrote:
>> Dear all,
>> I've started a Google code project (http://code.google.com/p/pypraxis/) to provide a Python interface to Brent's principal axis algorithm. It's basically a wrapper around some Fortran code from http://www.netlib.org/opt/. Brent's algorithm minimizes a function of several variables without calculating derivatives - not to be mistaken for scipy.optimize.brent, that only performs single-variable minimization. The algorithm typically outperforms other derivative- and gradient-free algorithms (Brent, 2002; http://wwwmaths.anu.edu.au/~brent/pub/pub011.html). In my experience, it converges substantially faster than fmin and fmin_powell from scipy.optimize when fitting models with 5 to 15 free parameters to experimental data. Notably, Mathematica uses this algorithm for minimization without derivatives
>> (http://reference.wolfram.com/mathematica/tutorial/UnconstrainedOptimizationPrincipalAxisMethod.html).
>> I've provided some test cases and a wrapper that allow to compare it directly to the existing algorithms from scipy.optimize. Let me know if you think that the code could be a candidate for integration into scipy.optimize. It would obviously require some work to make it conform with the other functions that are already present.
> 
> That would be great! Unfortunately, there is no license attached to
> praxis.f, so it cannot be integrated into scipy until we find a
> suitably licensed implementation of the algorithm.
> 

http://wwwmaths.anu.edu.au/~brent/software.html

This seems to imply that the code may be used freely; but it wouldn't 
hurt to ask the author.

Eric


From charlesr.harris at gmail.com  Tue Sep  8 23:10:50 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 8 Sep 2009 22:10:50 -0500
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
In-Reply-To: <4AA6F6BA.30402@hawaii.edu>
References: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
	<3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com>
	<4AA6F6BA.30402@hawaii.edu>
Message-ID: <e06186140909082010g3a1a8563yd74815785a01f95d@mail.gmail.com>

On Tue, Sep 8, 2009 at 7:28 PM, Eric Firing <efiring at hawaii.edu> wrote:

> Robert Kern wrote:
> > On Tue, Sep 8, 2009 at 12:44, Christoph
> > Schmidt-Hieber<c.schmidt-hieber at ucl.ac.uk> wrote:
> >> Dear all,
> >> I've started a Google code project (http://code.google.com/p/pypraxis/)
> to provide a Python interface to Brent's principal axis algorithm. It's
> basically a wrapper around some Fortran code from
> http://www.netlib.org/opt/. Brent's algorithm minimizes a function of
> several variables without calculating derivatives - not to be mistaken for
> scipy.optimize.brent, that only performs single-variable minimization. The
> algorithm typically outperforms other derivative- and gradient-free
> algorithms (Brent, 2002; http://wwwmaths.anu.edu.au/~brent/pub/pub011.html).
> In my experience, it converges substantially faster than fmin and
> fmin_powell from scipy.optimize when fitting models with 5 to 15 free
> parameters to experimental data. Notably, Mathematica uses this algorithm
> for minimization without derivatives
> >> (
> http://reference.wolfram.com/mathematica/tutorial/UnconstrainedOptimizationPrincipalAxisMethod.html
> ).
> >> I've provided some test cases and a wrapper that allow to compare it
> directly to the existing algorithms from scipy.optimize. Let me know if you
> think that the code could be a candidate for integration into
> scipy.optimize. It would obviously require some work to make it conform with
> the other functions that are already present.
> >
> > That would be great! Unfortunately, there is no license attached to
> > praxis.f, so it cannot be integrated into scipy until we find a
> > suitably licensed implementation of the algorithm.
> >
>
> http://wwwmaths.anu.edu.au/~brent/software.html
>
> This seems to imply that the code may be used freely; but it wouldn't
> hurt to ask the author.
>
>

No it doesn't, it implies that most of the code is GPL. ISTR looking at the
random number generation code on Brent's site and coming to a stop after
that bit. It may be that Brent would be open to relicensing the code, or
that the code in guestion is not just free, but BSD free, but I think it
would be advisable to contact Brent and find out.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090908/7388b65b/attachment.html>

From sturla at molden.no  Wed Sep  9 00:34:24 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 09 Sep 2009 06:34:24 +0200
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
In-Reply-To: <3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com>
References: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
	<3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com>
Message-ID: <4AA73050.3080404@molden.no>

Robert Kern skrev:
> That would be great! Unfortunately, there is no license attached to
> praxis.f, so it cannot be integrated into scipy until we find a
> suitably licensed implementation of the algorithm.
>   
I am not a lawyer, but --- I'd say if someone puts his own code on the 
netlib repository for free download, without even making a copyright 
notice, it can be seen as public domain.

Sturla




From robert.kern at gmail.com  Wed Sep  9 00:38:42 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 8 Sep 2009 23:38:42 -0500
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
In-Reply-To: <4AA73050.3080404@molden.no>
References: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
	<3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com> 
	<4AA73050.3080404@molden.no>
Message-ID: <3d375d730909082138ob80a589l6bcf0f8348f4b2c6@mail.gmail.com>

On Tue, Sep 8, 2009 at 23:34, Sturla Molden<sturla at molden.no> wrote:
> Robert Kern skrev:
>> That would be great! Unfortunately, there is no license attached to
>> praxis.f, so it cannot be integrated into scipy until we find a
>> suitably licensed implementation of the algorithm.
>>
> I am not a lawyer, but --- I'd say if someone puts his own code on the
> netlib repository for free download, without even making a copyright
> notice, it can be seen as public domain.

That is entirely contrary to copyright law. Since the Berne Convetion
in 1973, works are copyrighted at the moment of creation. One does not
need to specify that it is copyrighted. In order for you to distribute
it, the author must explicitly give you a license. Silence does not
imply consent.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From pav+sp at iki.fi  Wed Sep  9 04:01:00 2009
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Wed, 9 Sep 2009 08:01:00 +0000 (UTC)
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
References: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
	<3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com>
	<4AA73050.3080404@molden.no>
Message-ID: <h87nbs$3vu$1@ger.gmane.org>

Wed, 09 Sep 2009 06:34:24 +0200, Sturla Molden wrote:

> Robert Kern skrev:
>> That would be great! Unfortunately, there is no license attached to
>> praxis.f, so it cannot be integrated into scipy until we find a
>> suitably licensed implementation of the algorithm.
>>   
> I am not a lawyer, but --- I'd say if someone puts his own code on the
> netlib repository for free download, without even making a copyright
> notice, it can be seen as public domain.

This is not true in practice (nor in theory as Robert notes). The netlib 
repository contains many pieces of code licensed for example under the 
ACM license (academic non-commercial use only, restrictions on 
modification), copyright residing with ACM. Netlib just fails to notify 
people about this.

These source files of course don't contain any mention of copyright, but 
you find about it by looking at the calgo.acm.org list. (Some of them may 
be in public domain, being works of US government employees, but IANAL.) 

-- 
Pauli Virtanen



From benny.malengier at gmail.com  Wed Sep  9 04:04:08 2009
From: benny.malengier at gmail.com (Benny Malengier)
Date: Wed, 9 Sep 2009 10:04:08 +0200
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
In-Reply-To: <3d375d730909082138ob80a589l6bcf0f8348f4b2c6@mail.gmail.com>
References: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
	<3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com>
	<4AA73050.3080404@molden.no>
	<3d375d730909082138ob80a589l6bcf0f8348f4b2c6@mail.gmail.com>
Message-ID: <bcafe8290909090104w57463df3rdf758452f15b0f95@mail.gmail.com>

2009/9/9 Robert Kern <robert.kern at gmail.com>:
> On Tue, Sep 8, 2009 at 23:34, Sturla Molden<sturla at molden.no> wrote:
>> Robert Kern skrev:
>>> That would be great! Unfortunately, there is no license attached to
>>> praxis.f, so it cannot be integrated into scipy until we find a
>>> suitably licensed implementation of the algorithm.
>>>
>> I am not a lawyer, but --- I'd say if someone puts his own code on the
>> netlib repository for free download, without even making a copyright
>> notice, it can be seen as public domain.
>
> That is entirely contrary to copyright law. Since the Berne Convetion
> in 1973, works are copyrighted at the moment of creation. One does not
> need to specify that it is copyrighted. In order for you to distribute
> it, the author must explicitly give you a license. Silence does not
> imply consent.

To return to the original question, yes, it would be great if this
could be included in scipy optimize.
If it needs to be rewritten, if somebody makes a clean description of
how to do it by looking at the code, I'm sure some people would be
willing to reimplement it in C for scipy.
If Brent indicates the code is GPL, you can write a scikit that
extends scipy.optimize to offer this, as inclusion in scipy is not
possible, but I'm not sure GPL code really means all should be cleanly
reimplemented so as to allow inclusion in scipy. (GPL can use BSD code
after all, so the scikit would be GPL and independant).

Benny


From sturla at molden.no  Wed Sep  9 04:54:33 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 09 Sep 2009 10:54:33 +0200
Subject: [SciPy-dev] recursive Gaussian filter in C
Message-ID: <4AA76D49.7060409@molden.no>

I have written a C version of the recursive Gaussian filter. It is more 
accurate than gaussian_filter in ndimage (less truncation error near 
edges, as shown before) and also faster.  Here are some timings on 
filtering the "lenna" test image on my computer:
sigma = 5
elapsed time (iir with openmp): 81.366640 ms
elapsed time (iir): 107.282360 ms
elapsed time (ndimage): 137.548760 ms

sigma = 9
elapsed time (iir with openmp): 44.403760 ms
elapsed time (iir): 75.285720 ms
elapsed time (ndimage): 163.750920 ms

sigma = 21
elapsed time (iir with openmp): 45.063040 ms
elapsed time (iir): 75.941400 ms
elapsed time (ndimage): 313.359080 ms

sigma = 101
elapsed time (iir with openmp): 56.134120 ms
elapsed time (iir): 87.622240 ms
elapsed time (ndimage): 1210.016680 ms


It is still only written for np.float64, but it would be easy to make 
optimized versions for various dtypes, including rgb images.

You don't really see the scaling effect of OpenMP here, as the only 
thing that changes with sigma is the amount of zero-padding. Anyhow, 
this beats ndimage on speed and accuracy, and scales much better with 
sigma.

It is not restricted to 2 dimensions. It can filter along any axis of an 
ndarray. Thus another use case is fast kernel density estimation in nd 
space.


Regards,
Sturla Molden







From sturla at molden.no  Wed Sep  9 05:01:59 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 09 Sep 2009 11:01:59 +0200
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
In-Reply-To: <bcafe8290909090104w57463df3rdf758452f15b0f95@mail.gmail.com>
References: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>	<3d375d730909081246x75b7ced8gaf43dcb31f626f99@mail.gmail.com>	<4AA73050.3080404@molden.no>	<3d375d730909082138ob80a589l6bcf0f8348f4b2c6@mail.gmail.com>
	<bcafe8290909090104w57463df3rdf758452f15b0f95@mail.gmail.com>
Message-ID: <4AA76F07.2060004@molden.no>

Benny Malengier skrev:
> To return to the original question, yes, it would be great if this
> could be included in scipy optimize.
> If it needs to be rewritten, if somebody makes a clean description of
> how to do it by looking at the code, I'm sure some people would be
> willing to reimplement it in C for scipy.
>   
I took a quick look at the code. It seems to optimize with Brent's 
method along one dimension, jump randomly to a point in the vicinity, 
optimize along conjugate dimensions, etc. I should not be too hard to 
reimplement with a version of brent for one dimension and a prng.

S.M.




From c.schmidt-hieber at ucl.ac.uk  Wed Sep  9 05:11:15 2009
From: c.schmidt-hieber at ucl.ac.uk (Christoph Schmidt-Hieber)
Date: Wed, 9 Sep 2009 10:11:15 +0100
Subject: [SciPy-dev] Brent's Principal Axis Algorithm
In-Reply-To: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
References: <3AAA5B1312E7B74AA3E4704349266A5C14AE019E@pc5-1.cruciform.wibr.ucl.ac.uk>
Message-ID: <3AAA5B1312E7B74AA3E4704349266A5C14AE01A0@pc5-1.cruciform.wibr.ucl.ac.uk>

Dear all,
thanks for all your comments. In the meantime, I've switched to a Fortran90 implementation by John Burkardt. I'm aware that this adds an additional layer of license issues, but I believe that there are some advantages outweighing this:
- The code is more portable than the F77 version; I've tested on GNU/Linux 32bit and 64 bit and on Mac OS X.
- The code is more readable, and it should be more straightforward to port it to C if anyone wants to follow Benny's suggestion.
- Most of the code on Burkardt's page is under the LGPL, so I suppose that this particular file is just not LGPL'd because the license status of the original F77 file is unclear.
Cheers
Christoph

P.S. The mails that I write with my default mail client (alpine) don't seem to get through to this mailing list. Any ideas what I might be doing wrong?

-----WIBR-EMAIL-----
This email and any files transmitted with it
are confidential and intended solely for the
use of the individual or entity to whom they
are addressed. If you have received this 
email in error please notify the system 
management (it.support at wibr.ucl.ac.uk).
--------***---------




From efiring at hawaii.edu  Wed Sep  9 13:50:35 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Wed, 09 Sep 2009 07:50:35 -1000
Subject: [SciPy-dev] license status of your code on netlib
Message-ID: <4AA7EAEB.3040804@hawaii.edu>

Prof. Brent,

Regarding your minimization code on netlib, a question has arisen as to 
whether it, or code derived from it, can be used in a project such as 
scipy (http://scipy.org) that has a BSD-style license 
(http://scipy.org/License_Compatibility).  The question arose in the 
following mailing list conversation:

http://thread.gmane.org/gmane.comp.python.scientific.devel/11726

Thank you for any clarification you can provide, and thank you for 
having provided the code in the first place.

Eric Firing


From efiring at hawaii.edu  Thu Sep 10 03:13:53 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Wed, 09 Sep 2009 21:13:53 -1000
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
Message-ID: <4AA8A731.8000101@hawaii.edu>

I realize that with the exploration of other codes for minimization, the 
question about the license status of R. P. Brent's netlib code may be 
moot, but regardless, attached is Prof. Brent's gracious reply to my 
inquiry.

Eric
-------------- next part --------------
An embedded message was scrubbed...
From: Richard Brent <rpbrent at gmail.com>
Subject: Re: license status of your code on netlib
Date: Thu, 10 Sep 2009 10:11:33 +1000
Size: 4655
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090909/124ecdbc/attachment.eml>

From gael.varoquaux at normalesup.org  Thu Sep 10 03:32:28 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 10 Sep 2009 09:32:28 +0200
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <4AA8A731.8000101@hawaii.edu>
References: <4AA8A731.8000101@hawaii.edu>
Message-ID: <20090910073228.GG2482@phare.normalesup.org>

On Wed, Sep 09, 2009 at 09:13:53PM -1000, Eric Firing wrote:
> I realize that with the exploration of other codes for minimization, the  
> question about the license status of R. P. Brent's netlib code may be  
> moot, but regardless, attached is Prof. Brent's gracious reply to my  
> inquiry.

That's a fantastic answer, focusing more on the science than on the legal
restrictions. I would really like more people to think in these terms.

Hats off to Prof. Brent!

Ga?l


> Date: Thu, 10 Sep 2009 10:11:33 +1000
> From: Richard Brent <rpbrent at gmail.com>
> Subject: Re: license status of your code on netlib
> To: Eric Firing <efiring at hawaii.edu>
> Reply-to: Richard.Brent at anu.edu.au

> Dear Eric,

> You certainly have my permission, as the author of the original Algol
> W code (in my PhD thesis, 1971) and the Fortran translation (in my
> book "Algorithms for Minimization without Derivatives", Prentice-Hall
> 1972 and Dover 2002), to use my code with a BSD-style licence. When
> the code was originally published there was no licence, only a
> copyright notice on the book (and Prentice-Hall relinquished any
> rights to this to allow Dover to republish the book).

> Other code from my thesis was used, for example, in Numerical Recipes,
> without any explicit permission from me.

> I'm happy (and somewhat surprised) to know that my old mininimization
> code is still regarded as competitive with more recent developments
> (e.g. by Powell), and would be disappointed if any legal issues
> prevented you from using it in the scipy project.  All I ask is that
> you give a reference, e.g. to my book and/or web page
> http://wwwmaths.anu.edu.au/~brent/pub/pub011.html

> Regards,
> Richard Brent

> 2009/9/10 Eric Firing <efiring at hawaii.edu>:
> > Prof. Brent,

> > Regarding your minimization code on netlib, a question has arisen as to
> > whether it, or code derived from it, can be used in a project such as scipy
> > (http://scipy.org) that has a BSD-style license
> > (http://scipy.org/License_Compatibility). ?The question arose in the
> > following mailing list conversation:

> > http://thread.gmane.org/gmane.comp.python.scientific.devel/11726

> > Thank you for any clarification you can provide, and thank you for having
> > provided the code in the first place.

> > Eric Firing
-- 
    Gael Varoquaux
    Research Fellow, INRIA
    Laboratoire de Neuro-Imagerie Assistee par Ordinateur
    NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
    ++ 33-1-69-08-78-35


From benny.malengier at gmail.com  Thu Sep 10 03:48:57 2009
From: benny.malengier at gmail.com (Benny Malengier)
Date: Thu, 10 Sep 2009 09:48:57 +0200
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <4AA8A731.8000101@hawaii.edu>
References: <4AA8A731.8000101@hawaii.edu>
Message-ID: <bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>

2009/9/10 Eric Firing <efiring at hawaii.edu>:
> I realize that with the exploration of other codes for minimization, the
> question about the license status of R. P. Brent's netlib code may be moot,
> but regardless, attached is Prof. Brent's gracious reply to my inquiry.

Great,

On a side note, due to this I have read
http://scipy.org/License_Compatibility and as a developer who writes
BSD and GPL licenced code, I find the section 'John Hunter's License
Pitch' not very suiting. I see good use cases for both kind of
licences, and think all this side choosing does a project as scipy no
good.
To read things like " 1) Go with GPL and lose the mind-share of the
private sector 2) Forgo GPL code and retain the contribution of the
private sector." is just plain stupid in a day and age that private
companies spend huge amounts in releasing and writing (L)GPL code.
The same shoe just does not fit all feet, also many companies fear BSD
for their own products they want to open source.

I write BSD code for things like algorithms as the intended use is as
a library and my users are tech knowledgeable. Not because GPL is
wrong in some ways.
Perhaps somebody can write a better explanation on the wiki why scipy
uses BSD, without the attacks against GPL.

Benny


From robert.kern at gmail.com  Thu Sep 10 11:29:41 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 10 Sep 2009 10:29:41 -0500
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
Message-ID: <3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com>

On Thu, Sep 10, 2009 at 02:48, Benny Malengier<benny.malengier at gmail.com> wrote:

> On a side note, due to this I have read
> http://scipy.org/License_Compatibility and as a developer who writes
> BSD and GPL licenced code, I find the section 'John Hunter's License
> Pitch' not very suiting.

I have deleted it. I too do not like having a polemic on a page that
should just be describing the facts of license compatibility.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Thu Sep 10 12:03:20 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 10 Sep 2009 12:03:20 -0400
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com>
Message-ID: <1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com>

On Thu, Sep 10, 2009 at 11:29 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Sep 10, 2009 at 02:48, Benny Malengier<benny.malengier at gmail.com> wrote:
>
>> On a side note, due to this I have read
>> http://scipy.org/License_Compatibility and as a developer who writes
>> BSD and GPL licenced code, I find the section 'John Hunter's License
>> Pitch' not very suiting.
>
> I have deleted it. I too do not like having a polemic on a page that
> should just be describing the facts of license compatibility.
>

I also read this for the first time, but it contains mostly the obvious
cases.

What is the compatibility for the Apache license ?

There was the discussion on the cython list and I know of some
packages, that I'm interested in, that use Apache.

Are there any other (semi-)popular licenses that would be compatible
with BSD?

Josef


> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From robert.kern at gmail.com  Thu Sep 10 12:22:18 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 10 Sep 2009 11:22:18 -0500
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com> 
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com> 
	<1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com>
Message-ID: <3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com>

On Thu, Sep 10, 2009 at 11:03, <josef.pktd at gmail.com> wrote:
> On Thu, Sep 10, 2009 at 11:29 AM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Thu, Sep 10, 2009 at 02:48, Benny Malengier<benny.malengier at gmail.com> wrote:
>>
>>> On a side note, due to this I have read
>>> http://scipy.org/License_Compatibility and as a developer who writes
>>> BSD and GPL licenced code, I find the section 'John Hunter's License
>>> Pitch' not very suiting.
>>
>> I have deleted it. I too do not like having a polemic on a page that
>> should just be describing the facts of license compatibility.
>>
>
> I also read this for the first time, but it contains mostly the obvious
> cases.
>
> What is the compatibility for the Apache license ?
>
> There was the discussion on the cython list and I know of some
> packages, that I'm interested in, that use Apache.

Slightly incompatible. It's mostly good, but it has a few provisions
in it, like the patent peace clause, that make it GPLv2-incompatible.
These are complications that would be good to avoid. The authors of
such code may be willing to relicense for scipy if they aren't "using"
those clauses.

> Are there any other (semi-)popular licenses that would be compatible
> with BSD?

Not really. There are many ad hoc license grants that people make that
are compatible, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Thu Sep 10 12:43:28 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 10 Sep 2009 12:43:28 -0400
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com>
	<1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com>
	<3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com>
Message-ID: <1cd32cbb0909100943q5ef78e49h1b56da098a5d93fd@mail.gmail.com>

On Thu, Sep 10, 2009 at 12:22 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Sep 10, 2009 at 11:03, <josef.pktd at gmail.com> wrote:
>> On Thu, Sep 10, 2009 at 11:29 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Thu, Sep 10, 2009 at 02:48, Benny Malengier<benny.malengier at gmail.com> wrote:
>>>
>>>> On a side note, due to this I have read
>>>> http://scipy.org/License_Compatibility and as a developer who writes
>>>> BSD and GPL licenced code, I find the section 'John Hunter's License
>>>> Pitch' not very suiting.
>>>
>>> I have deleted it. I too do not like having a polemic on a page that
>>> should just be describing the facts of license compatibility.
>>>
>>
>> I also read this for the first time, but it contains mostly the obvious
>> cases.
>>
>> What is the compatibility for the Apache license ?
>>
>> There was the discussion on the cython list and I know of some
>> packages, that I'm interested in, that use Apache.
>

> Slightly incompatible. It's mostly good, but it has a few provisions
> in it, like the patent peace clause, that make it GPLv2-incompatible.

If this is not a typo: I don't understand how GPLv2 compatibility of
the Apache license affects the compatibility with BSD

Thanks,
Josef

> These are complications that would be good to avoid. The authors of
> such code may be willing to relicense for scipy if they aren't "using"
> those clauses.
>
>> Are there any other (semi-)popular licenses that would be compatible
>> with BSD?
>
> Not really. There are many ad hoc license grants that people make that
> are compatible, though.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From robert.kern at gmail.com  Thu Sep 10 12:53:06 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 10 Sep 2009 11:53:06 -0500
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <1cd32cbb0909100943q5ef78e49h1b56da098a5d93fd@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com> 
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com> 
	<1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com> 
	<3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com> 
	<1cd32cbb0909100943q5ef78e49h1b56da098a5d93fd@mail.gmail.com>
Message-ID: <3d375d730909100953y2a51bd8bld5ce6b6addd96854@mail.gmail.com>

On Thu, Sep 10, 2009 at 11:43, <josef.pktd at gmail.com> wrote:
> On Thu, Sep 10, 2009 at 12:22 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Thu, Sep 10, 2009 at 11:03, <josef.pktd at gmail.com> wrote:
>>> On Thu, Sep 10, 2009 at 11:29 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>>> On Thu, Sep 10, 2009 at 02:48, Benny Malengier<benny.malengier at gmail.com> wrote:
>>>>
>>>>> On a side note, due to this I have read
>>>>> http://scipy.org/License_Compatibility and as a developer who writes
>>>>> BSD and GPL licenced code, I find the section 'John Hunter's License
>>>>> Pitch' not very suiting.
>>>>
>>>> I have deleted it. I too do not like having a polemic on a page that
>>>> should just be describing the facts of license compatibility.
>>>>
>>>
>>> I also read this for the first time, but it contains mostly the obvious
>>> cases.
>>>
>>> What is the compatibility for the Apache license ?
>>>
>>> There was the discussion on the cython list and I know of some
>>> packages, that I'm interested in, that use Apache.
>>
>
>> Slightly incompatible. It's mostly good, but it has a few provisions
>> in it, like the patent peace clause, that make it GPLv2-incompatible.
>
> If this is not a typo: I don't understand how GPLv2 compatibility of
> the Apache license affects the compatibility with BSD

Well, compatibility may not be the right word. One can take BSD code
and Apache code and combine them into a program. One can take BSD code
and GPLv2 code and combine them into a program. One cannot take an
Apache code and a GPLv2 code and combine them into a program. This is
frequently what people refer to as "license compatibility".

However, per our BSD *policy* we should not take in Apache code
because then our project will contain code that has additional
restrictions beyond our advertised "BSD license". In particular, users
can mix our BSD code with their GPLv2 code freely. If we had an
Apache-licensed component, they could not. Or at least we would have
to change things significantly to allow people to "avoid" certain
components. That's a pain in the ass; the problem is better resolved
by having the differently-licensed code packaged separately.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Thu Sep 10 13:07:45 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 10 Sep 2009 13:07:45 -0400
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <3d375d730909100953y2a51bd8bld5ce6b6addd96854@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com>
	<1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com>
	<3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com>
	<1cd32cbb0909100943q5ef78e49h1b56da098a5d93fd@mail.gmail.com>
	<3d375d730909100953y2a51bd8bld5ce6b6addd96854@mail.gmail.com>
Message-ID: <1cd32cbb0909101007h6882e952j1c1275855e3b0c90@mail.gmail.com>

On Thu, Sep 10, 2009 at 12:53 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Sep 10, 2009 at 11:43, <josef.pktd at gmail.com> wrote:
>> On Thu, Sep 10, 2009 at 12:22 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Thu, Sep 10, 2009 at 11:03, <josef.pktd at gmail.com> wrote:
>>>> On Thu, Sep 10, 2009 at 11:29 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>>>> On Thu, Sep 10, 2009 at 02:48, Benny Malengier<benny.malengier at gmail.com> wrote:
>>>>>
>>>>>> On a side note, due to this I have read
>>>>>> http://scipy.org/License_Compatibility and as a developer who writes
>>>>>> BSD and GPL licenced code, I find the section 'John Hunter's License
>>>>>> Pitch' not very suiting.
>>>>>
>>>>> I have deleted it. I too do not like having a polemic on a page that
>>>>> should just be describing the facts of license compatibility.
>>>>>
>>>>
>>>> I also read this for the first time, but it contains mostly the obvious
>>>> cases.
>>>>
>>>> What is the compatibility for the Apache license ?
>>>>
>>>> There was the discussion on the cython list and I know of some
>>>> packages, that I'm interested in, that use Apache.
>>>
>>
>>> Slightly incompatible. It's mostly good, but it has a few provisions
>>> in it, like the patent peace clause, that make it GPLv2-incompatible.
>>
>> If this is not a typo: I don't understand how GPLv2 compatibility of
>> the Apache license affects the compatibility with BSD
>
> Well, compatibility may not be the right word. One can take BSD code
> and Apache code and combine them into a program. One can take BSD code
> and GPLv2 code and combine them into a program. One cannot take an
> Apache code and a GPLv2 code and combine them into a program. This is
> frequently what people refer to as "license compatibility".
>
> However, per our BSD *policy* we should not take in Apache code
> because then our project will contain code that has additional
> restrictions beyond our advertised "BSD license". In particular, users
> can mix our BSD code with their GPLv2 code freely. If we had an
> Apache-licensed component, they could not. Or at least we would have
> to change things significantly to allow people to "avoid" certain
> components. That's a pain in the ass; the problem is better resolved
> by having the differently-licensed code packaged separately.

Thanks for the clarification, this means essentially that BSD and MIT
are the only
non-infectious licenses (of the popular ones).

Josef


>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From robert.kern at gmail.com  Thu Sep 10 13:22:57 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 10 Sep 2009 12:22:57 -0500
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <1cd32cbb0909101007h6882e952j1c1275855e3b0c90@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com> 
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com> 
	<1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com> 
	<3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com> 
	<1cd32cbb0909100943q5ef78e49h1b56da098a5d93fd@mail.gmail.com> 
	<3d375d730909100953y2a51bd8bld5ce6b6addd96854@mail.gmail.com> 
	<1cd32cbb0909101007h6882e952j1c1275855e3b0c90@mail.gmail.com>
Message-ID: <3d375d730909101022v2821c927t6fc7afa304276d3c@mail.gmail.com>

On Thu, Sep 10, 2009 at 12:07, <josef.pktd at gmail.com> wrote:
> On Thu, Sep 10, 2009 at 12:53 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Thu, Sep 10, 2009 at 11:43, <josef.pktd at gmail.com> wrote:
>>> On Thu, Sep 10, 2009 at 12:22 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>>> On Thu, Sep 10, 2009 at 11:03, <josef.pktd at gmail.com> wrote:
>>>>> On Thu, Sep 10, 2009 at 11:29 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>>>>> On Thu, Sep 10, 2009 at 02:48, Benny Malengier<benny.malengier at gmail.com> wrote:
>>>>>>
>>>>>>> On a side note, due to this I have read
>>>>>>> http://scipy.org/License_Compatibility and as a developer who writes
>>>>>>> BSD and GPL licenced code, I find the section 'John Hunter's License
>>>>>>> Pitch' not very suiting.
>>>>>>
>>>>>> I have deleted it. I too do not like having a polemic on a page that
>>>>>> should just be describing the facts of license compatibility.
>>>>>>
>>>>>
>>>>> I also read this for the first time, but it contains mostly the obvious
>>>>> cases.
>>>>>
>>>>> What is the compatibility for the Apache license ?
>>>>>
>>>>> There was the discussion on the cython list and I know of some
>>>>> packages, that I'm interested in, that use Apache.
>>>>
>>>
>>>> Slightly incompatible. It's mostly good, but it has a few provisions
>>>> in it, like the patent peace clause, that make it GPLv2-incompatible.
>>>
>>> If this is not a typo: I don't understand how GPLv2 compatibility of
>>> the Apache license affects the compatibility with BSD
>>
>> Well, compatibility may not be the right word. One can take BSD code
>> and Apache code and combine them into a program. One can take BSD code
>> and GPLv2 code and combine them into a program. One cannot take an
>> Apache code and a GPLv2 code and combine them into a program. This is
>> frequently what people refer to as "license compatibility".
>>
>> However, per our BSD *policy* we should not take in Apache code
>> because then our project will contain code that has additional
>> restrictions beyond our advertised "BSD license". In particular, users
>> can mix our BSD code with their GPLv2 code freely. If we had an
>> Apache-licensed component, they could not. Or at least we would have
>> to change things significantly to allow people to "avoid" certain
>> components. That's a pain in the ass; the problem is better resolved
>> by having the differently-licensed code packaged separately.
>
> Thanks for the clarification, this means essentially that BSD and MIT
> are the only
> non-infectious licenses (of the popular ones).

Apache isn't really "infectious" as the term is usually used in
reference to the various copyleft licenses. It just has a few more
restrictions that we, as a matter of policy, do not want.

The copyleft licenses prevent you from adding more restrictions on top
of the license. You can't take a GPLed component and incorporate it
into a proprietary product that you forbid redistribution of. It
"infects" the product that way, if you wish to use such loaded
terminology (and I don't recommend it). That's a property of the
license.

I can take an Apache- or BSD-licensed component and incorporate it
into a proprietary product and load on any other restrictions I like.
However, I cannot *negate* any of the provisions of the original
licenses. I must still maintain the copyright notices and attribution.
With the Apache license, the patent peace provision extends to the
user of the proprietary product, too. This is true of all licenses. We
just happened to pick, as a matter of policy for the project, the BSD
set of restrictions and no more. Apache code would not "infect" us; we
just choose not to use it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jdh2358 at gmail.com  Thu Sep 10 13:26:14 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Thu, 10 Sep 2009 12:26:14 -0500
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <1cd32cbb0909101007h6882e952j1c1275855e3b0c90@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com>
	<1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com>
	<3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com>
	<1cd32cbb0909100943q5ef78e49h1b56da098a5d93fd@mail.gmail.com>
	<3d375d730909100953y2a51bd8bld5ce6b6addd96854@mail.gmail.com>
	<1cd32cbb0909101007h6882e952j1c1275855e3b0c90@mail.gmail.com>
Message-ID: <88e473830909101026w3c4be985g62220e81fdfedce5@mail.gmail.com>

On Thu, Sep 10, 2009 at 12:07 PM,  <josef.pktd at gmail.com> wrote:

>>> On a side note, due to this I have read
>>> http://scipy.org/License_Compatibility and as a developer who writes
>>> BSD and GPL licenced code, I find the section 'John Hunter's License
>>> Pitch' not very suiting.
>>
>> I have deleted it. I too do not like having a polemic on a page that
>> should just be describing the facts of license compatibility.

While I did not add that to the scipy page, I did write that some time
ago.  In the interim, I have significantly toned down the language in
the version on the mpl website

  http://matplotlib.sourceforge.net/devel/coding_guide.html#licenses

The polemic started it's life as a response to a question about
licenses on the mailing list, some people found it useful, and someone
posted it on the wiki.  But I agree, it is best to tone down any
rhetoric about licenses so the removal is a good thing.

JDH


From robert.kern at gmail.com  Thu Sep 10 13:36:05 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 10 Sep 2009 12:36:05 -0500
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <88e473830909101026w3c4be985g62220e81fdfedce5@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com> 
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com> 
	<1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com> 
	<3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com> 
	<1cd32cbb0909100943q5ef78e49h1b56da098a5d93fd@mail.gmail.com> 
	<3d375d730909100953y2a51bd8bld5ce6b6addd96854@mail.gmail.com> 
	<1cd32cbb0909101007h6882e952j1c1275855e3b0c90@mail.gmail.com> 
	<88e473830909101026w3c4be985g62220e81fdfedce5@mail.gmail.com>
Message-ID: <3d375d730909101036l209727a2s8c5c2273e3c26aa8@mail.gmail.com>

On Thu, Sep 10, 2009 at 12:26, John Hunter<jdh2358 at gmail.com> wrote:
> On Thu, Sep 10, 2009 at 12:07 PM, ?<josef.pktd at gmail.com> wrote:
>
>>>> On a side note, due to this I have read
>>>> http://scipy.org/License_Compatibility and as a developer who writes
>>>> BSD and GPL licenced code, I find the section 'John Hunter's License
>>>> Pitch' not very suiting.
>>>
>>> I have deleted it. I too do not like having a polemic on a page that
>>> should just be describing the facts of license compatibility.
>
> While I did not add that to the scipy page, I did write that some time
> ago. ?In the interim, I have significantly toned down the language in
> the version on the mpl website
>
> ?http://matplotlib.sourceforge.net/devel/coding_guide.html#licenses
>
> The polemic started it's life as a response to a question about
> licenses on the mailing list, some people found it useful, and someone
> posted it on the wiki. ?But I agree, it is best to tone down any
> rhetoric about licenses so the removal is a good thing.

It's not that I don't *like* polemics or disagree with yours, but
there's a time and a place for everything. :-)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From benny.malengier at gmail.com  Thu Sep 10 14:13:30 2009
From: benny.malengier at gmail.com (Benny Malengier)
Date: Thu, 10 Sep 2009 20:13:30 +0200
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <88e473830909101026w3c4be985g62220e81fdfedce5@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com>
	<1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com>
	<3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com>
	<1cd32cbb0909100943q5ef78e49h1b56da098a5d93fd@mail.gmail.com>
	<3d375d730909100953y2a51bd8bld5ce6b6addd96854@mail.gmail.com>
	<1cd32cbb0909101007h6882e952j1c1275855e3b0c90@mail.gmail.com>
	<88e473830909101026w3c4be985g62220e81fdfedce5@mail.gmail.com>
Message-ID: <bcafe8290909101113x6e4c9f81y542c8fc946cb98ba@mail.gmail.com>

2009/9/10 John Hunter <jdh2358 at gmail.com>:

> While I did not add that to the scipy page, I did write that some time
> ago. ?In the interim, I have significantly toned down the language in
> the version on the mpl website
>
> ?http://matplotlib.sourceforge.net/devel/coding_guide.html#licenses

It is nice to read this. I have a hard time understanding the reasons
for all the flamewars between BSD and GPL (too young probably).
Without the extremes I think we can all live perfectly together.
Especially in an academic environment where reason is king.

Benny


From d.l.goldsmith at gmail.com  Thu Sep 10 14:32:10 2009
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Thu, 10 Sep 2009 11:32:10 -0700
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <bcafe8290909101113x6e4c9f81y542c8fc946cb98ba@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<3d375d730909100829n231ae7a2s55cafa9b1b517485@mail.gmail.com>
	<1cd32cbb0909100903m395de207s586de2c76dc4ab63@mail.gmail.com>
	<3d375d730909100922y7809d41eh74f6e76d9462ab5d@mail.gmail.com>
	<1cd32cbb0909100943q5ef78e49h1b56da098a5d93fd@mail.gmail.com>
	<3d375d730909100953y2a51bd8bld5ce6b6addd96854@mail.gmail.com>
	<1cd32cbb0909101007h6882e952j1c1275855e3b0c90@mail.gmail.com>
	<88e473830909101026w3c4be985g62220e81fdfedce5@mail.gmail.com>
	<bcafe8290909101113x6e4c9f81y542c8fc946cb98ba@mail.gmail.com>
Message-ID: <45d1ab480909101132mbf18ca6t27070bbc3c5c8161@mail.gmail.com>

On Thu, Sep 10, 2009 at 11:13 AM, Benny Malengier <benny.malengier at gmail.com
> wrote:

> 2009/9/10 John Hunter <jdh2358 at gmail.com>:
>
> for all the flamewars between BSD and GPL (too young probably).
> Without the extremes I think we can all live perfectly together.
> Especially in an academic environment where reason is king.
>

My, you are young. ;-)

DG

Benny
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090910/16ede56d/attachment.html>

From alan at ajackson.org  Thu Sep 10 21:04:35 2009
From: alan at ajackson.org (alan at ajackson.org)
Date: Thu, 10 Sep 2009 20:04:35 -0500
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
Message-ID: <20090910200435.53ed9a5b@ajackson.org>

>2009/9/10 Eric Firing <efiring at hawaii.edu>:
>To read things like " 1) Go with GPL and lose the mind-share of the
>private sector 2) Forgo GPL code and retain the contribution of the
>private sector." is just plain stupid in a day and age that private
>companies spend huge amounts in releasing and writing (L)GPL code.
>The same shoe just does not fit all feet, also many companies fear BSD
>for their own products they want to open source.
>

Well I can assure you that at *my* company, we cannot use GPL code. The simple
reason is that we have subsidiaries that are not fully owned - some are 51%
owned by the local government, and so if we use GPL code in our internal
software, the viral provisions are triggered and we have to expose our
proprietary internal code. So we don't use it. At least that is what our
lawyers have told me. We like BSD. 8-)

So when we hire outside developers - they can't use GPL code. 

-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From david at ar.media.kyoto-u.ac.jp  Thu Sep 10 22:30:01 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 11 Sep 2009 11:30:01 +0900
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <20090910200435.53ed9a5b@ajackson.org>
References: <4AA8A731.8000101@hawaii.edu>	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<20090910200435.53ed9a5b@ajackson.org>
Message-ID: <4AA9B629.8090209@ar.media.kyoto-u.ac.jp>

alan at ajackson.org wrote:
>> 2009/9/10 Eric Firing <efiring at hawaii.edu>:
>> To read things like " 1) Go with GPL and lose the mind-share of the
>> private sector 2) Forgo GPL code and retain the contribution of the
>> private sector." is just plain stupid in a day and age that private
>> companies spend huge amounts in releasing and writing (L)GPL code.
>> The same shoe just does not fit all feet, also many companies fear BSD
>> for their own products they want to open source.
>>
>>     
>
> Well I can assure you that at *my* company, we cannot use GPL code.

I think that in the context of scipy, we should be very down to earth
and practical: if you want your code to be in scipy, you have to use a
BSD license. I much prefer the GPL to BSD due to the reciprocity idea,
and yet, 95 % of my open source contribution is under the BSD because I
mostly contribute to numpy, scipy and scons, which all fall under this
kind of license.

cheers,

David


From robince at gmail.com  Fri Sep 11 06:47:39 2009
From: robince at gmail.com (Robin)
Date: Fri, 11 Sep 2009 11:47:39 +0100
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <4AA9B629.8090209@ar.media.kyoto-u.ac.jp>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<20090910200435.53ed9a5b@ajackson.org>
	<4AA9B629.8090209@ar.media.kyoto-u.ac.jp>
Message-ID: <b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>

At the risk of deviating off topic - while we're talking about
licenses I was wondering if I could solicit some opinions.

In my academic area there is a move towards groups releasing toolkits
and packages for use by the wider community, and most of these refer
to it as 'open source', but they are usually distributed under strange
licenses. Some examples:

http://find.bccn.uni-freiburg.de/?n=Main.License
http://neurodatabase.org/src/license/
http://www.ibtb.org/Home/toolbox-downloads#currentversion
http://www.nest-initiative.org/index.php/Software:License

I guess I wondered what people thought about this. I'm looking for
convincing arguments for people to stick to standard open source
licenses where possible. I think the general impression from the above
is that in most cases GPL would be more suitable than BSD - since they
specifically want to prevent commercial exploitation. But what do
people suggest about the citation clauses and things like that?

What happens in other fields? (I assume this problem must also have
occurred there)

Cheers

Robin


From cournape at gmail.com  Fri Sep 11 07:30:02 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 11 Sep 2009 20:30:02 +0900
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<20090910200435.53ed9a5b@ajackson.org>
	<4AA9B629.8090209@ar.media.kyoto-u.ac.jp>
	<b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>
Message-ID: <5b8d13220909110430r70eadacgcef4e67471613330@mail.gmail.com>

On Fri, Sep 11, 2009 at 7:47 PM, Robin <robince at gmail.com> wrote:
> At the risk of deviating off topic - while we're talking about
> licenses I was wondering if I could solicit some opinions.
>
> In my academic area there is a move towards groups releasing toolkits
> and packages for use by the wider community, and most of these refer
> to it as 'open source', but they are usually distributed under strange
> licenses. Some examples:
>
> http://find.bccn.uni-freiburg.de/?n=Main.License
> http://neurodatabase.org/src/license/
> http://www.ibtb.org/Home/toolbox-downloads#currentversion
> http://www.nest-initiative.org/index.php/Software:License
>
> I guess I wondered what people thought about this.

This is bad in general, but people often have little choice, because
those licenses often are imposed by the legal dpt or similar from the
university/research institude in my experience.

> I'm looking for
> convincing arguments for people to stick to standard open source
> licenses where possible. I think the general impression from the above
> is that in most cases GPL would be more suitable than BSD - since they
> specifically want to prevent commercial exploitation.

There is this:

http://google-opensource.blogspot.com/2008/05/standing-against-license-proliferation.html

I think a quite strong argument, besides license compatibility, is
that few licenses are actually tested in court. I trust the GPL quite
a bit, since it has been written by people who knew about what they
were doing, and the BSD is well trusted as well. If someone violates
my license under the GPL, I can have some little hope that the FSF
will help me.

Writing your own license has nowhere near those guarantees.

> But what do
> people suggest about the citation clauses and things like that?

I don't think it is very useful - I doubt that people who are not
willing to cite your work will do so because you have a license in
your software, and it is not like you will sue someone because they
did not cite your work. I think it is a vain attempt at circumventing
a basic lack of ethics.

I also wonder whether a citation clause would not bring the same kind
of trouble as the advertisement clause in the old BSD.

cheers,

David


From benny.malengier at gmail.com  Fri Sep 11 08:26:14 2009
From: benny.malengier at gmail.com (Benny Malengier)
Date: Fri, 11 Sep 2009 14:26:14 +0200
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <5b8d13220909110430r70eadacgcef4e67471613330@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<20090910200435.53ed9a5b@ajackson.org>
	<4AA9B629.8090209@ar.media.kyoto-u.ac.jp>
	<b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>
	<5b8d13220909110430r70eadacgcef4e67471613330@mail.gmail.com>
Message-ID: <bcafe8290909110526l4c36d28fje922e54fbf4b1d26@mail.gmail.com>

This is actually the nice thing about the viral aspect of GPL. When
you start a new toolkit in your department by extending a GPL
component, you can tell the legal department that you don't have the
time to reimplement existing components (time pressure you know) so
you extend a GPL one, and now your component _must_ also be GPL. For
scipy this is more complicated, you must convince them you are better
of contributing to the project, and people who don't know the issue
have difficulty grasping the concept of this collaboration.

Hopefully you can later relax things to BSD, as within an academic
environment I see no reason why companies can not profit from the work
done. It is a reality however that when you are part of a project and
they see you will provide the modelling people are already asking if
it can 'exploited' later, and if so to add it to the part in the
project claiming what later can be done with the results.

It would be much easier if the European commission and other funding
organizations just require open source work under a set of approved
licences. After all, if it not distributed and just used internally by
the company the project is for (most likely) then GPL and BSD and
proprietary really don't matter as there is no distribution, but at
least a researcher has the right to take a copy of his code if he
obtains another position, or somebody else can grab it and do
something useful with it.

Benny

2009/9/11 David Cournapeau <cournape at gmail.com>:
> On Fri, Sep 11, 2009 at 7:47 PM, Robin <robince at gmail.com> wrote:
>> At the risk of deviating off topic - while we're talking about
>> licenses I was wondering if I could solicit some opinions.
>>
>> In my academic area there is a move towards groups releasing toolkits
>> and packages for use by the wider community, and most of these refer
>> to it as 'open source', but they are usually distributed under strange
>> licenses. Some examples:
>>
>> http://find.bccn.uni-freiburg.de/?n=Main.License
>> http://neurodatabase.org/src/license/
>> http://www.ibtb.org/Home/toolbox-downloads#currentversion
>> http://www.nest-initiative.org/index.php/Software:License
>>
>> I guess I wondered what people thought about this.
>
> This is bad in general, but people often have little choice, because
> those licenses often are imposed by the legal dpt or similar from the
> university/research institude in my experience.
>
>> I'm looking for
>> convincing arguments for people to stick to standard open source
>> licenses where possible. I think the general impression from the above
>> is that in most cases GPL would be more suitable than BSD - since they
>> specifically want to prevent commercial exploitation.
>
> There is this:
>
> http://google-opensource.blogspot.com/2008/05/standing-against-license-proliferation.html
>
> I think a quite strong argument, besides license compatibility, is
> that few licenses are actually tested in court. I trust the GPL quite
> a bit, since it has been written by people who knew about what they
> were doing, and the BSD is well trusted as well. If someone violates
> my license under the GPL, I can have some little hope that the FSF
> will help me.
>
> Writing your own license has nowhere near those guarantees.
>
>> But what do
>> people suggest about the citation clauses and things like that?
>
> I don't think it is very useful - I doubt that people who are not
> willing to cite your work will do so because you have a license in
> your software, and it is not like you will sue someone because they
> did not cite your work. I think it is a vain attempt at circumventing
> a basic lack of ethics.
>
> I also wonder whether a citation clause would not bring the same kind
> of trouble as the advertisement clause in the old BSD.
>
> cheers,
>
> David
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From bsouthey at gmail.com  Fri Sep 11 10:00:19 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 11 Sep 2009 09:00:19 -0500
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <5b8d13220909110430r70eadacgcef4e67471613330@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>	<20090910200435.53ed9a5b@ajackson.org>	<4AA9B629.8090209@ar.media.kyoto-u.ac.jp>	<b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>
	<5b8d13220909110430r70eadacgcef4e67471613330@mail.gmail.com>
Message-ID: <4AAA57F3.7040206@gmail.com>

On 09/11/2009 06:30 AM, David Cournapeau wrote:
> On Fri, Sep 11, 2009 at 7:47 PM, Robin<robince at gmail.com>  wrote:
>    
>> At the risk of deviating off topic - while we're talking about
>> licenses I was wondering if I could solicit some opinions.
>>
>> In my academic area there is a move towards groups releasing toolkits
>> and packages for use by the wider community, and most of these refer
>> to it as 'open source', but they are usually distributed under strange
>> licenses. Some examples:
>>
>> http://find.bccn.uni-freiburg.de/?n=Main.License
>> http://neurodatabase.org/src/license/
>> http://www.ibtb.org/Home/toolbox-downloads#currentversion
>> http://www.nest-initiative.org/index.php/Software:License
>>
>> I guess I wondered what people thought about this.
>>      
> This is bad in general, but people often have little choice, because
> those licenses often are imposed by the legal dpt or similar from the
> university/research institude in my experience.
>
>    
>> I'm looking for
>> convincing arguments for people to stick to standard open source
>> licenses where possible. I think the general impression from the above
>> is that in most cases GPL would be more suitable than BSD - since they
>> specifically want to prevent commercial exploitation.
>>      
> There is this:
>
> http://google-opensource.blogspot.com/2008/05/standing-against-license-proliferation.html
>
> I think a quite strong argument, besides license compatibility, is
> that few licenses are actually tested in court. I trust the GPL quite
> a bit, since it has been written by people who knew about what they
> were doing, and the BSD is well trusted as well. If someone violates
> my license under the GPL, I can have some little hope that the FSF
> will help me.
>
> Writing your own license has nowhere near those guarantees.
>
>    
>> But what do
>> people suggest about the citation clauses and things like that?
>>      
> I don't think it is very useful - I doubt that people who are not
> willing to cite your work will do so because you have a license in
> your software, and it is not like you will sue someone because they
> did not cite your work. I think it is a vain attempt at circumventing
> a basic lack of ethics.
>
> I also wonder whether a citation clause would not bring the same kind
> of trouble as the advertisement clause in the old BSD.
>
> cheers,
>
> David
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>    

If you require a citation clause then your license directly fits into 
the arguments against the BSD licenses with an advertising clause. But 
you can suggest that users cite your work provided that it no way 
restricts the usage of the code.

Personally I like that I can cite a package I use beyond referring just 
to the website. For example, the neurodatabase.org should at least 
appear in Nucleic Acid Research database or web server issues (too late 
this year for database but web server is still open).

Bruce


From pav+sp at iki.fi  Fri Sep 11 10:02:38 2009
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Fri, 11 Sep 2009 14:02:38 +0000 (UTC)
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<20090910200435.53ed9a5b@ajackson.org>
	<4AA9B629.8090209@ar.media.kyoto-u.ac.jp>
	<b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>
Message-ID: <h8dl9t$llc$1@ger.gmane.org>

(Straying well inside the fields of OT.)

Fri, 11 Sep 2009 11:47:39 +0100, Robin wrote:
[clip]
> I guess I wondered what people thought about this. I'm looking for
> convincing arguments for people to stick to standard open source
> licenses where possible. I think the general impression from the above
> is that in most cases GPL would be more suitable than BSD - since they
> specifically want to prevent commercial exploitation.

GPL doesn't really prevent all commercial exploitation, as there may be 
no redistribution -- just someone using your code to do computations, in 
a for-profit company. Or used in a commercially supported GPLed 
application.

> But what do people suggest about the citation clauses and things like
> that?

This probably depends. If you trust people enough, you can just include a 
strong citation recommendation and how-to instructions, but not a license 
clause. I don't have experience on real-world consequences of either 
alternative.

OTOH, citations (in addition to the grapevine) are nearly the only metric 
that matters for academic software, so maybe you would like to safeguard 
this "income". It may be difficult and unproductive to argue against 
this. But I suppose the drawbacks might be that

(i) If you don't keep manually track of who have your software,
    and someone is rude enough to disregard a strong recommendation,
    they'd also just disregard the license. Or maybe not read it at all.

(ii) The advertisment requirement may annoy some users away, especially
     if your code is a component, not a complete analysis package.

(iii) They exclude the software from being "free software" in the sense
      some people (say, Debian) understand it. Stuff with such clauses
      won't land into Debian/Ubuntu main.

In a sense, also the drawbacks of the advertisement clause start to apply 
-- suppose many basic numerical codes had citation clauses. Most of the 
papers in all fields would have a citation to LAPACK. In a sense a good 
thing, in a sense a bit silly. Maybe a less of an annoyance than the 
advertisement clause, though, as you'd need to do this only once per 
paper.

-- 
Pauli Virtanen



From answer at tnoo.net  Fri Sep 11 10:03:26 2009
From: answer at tnoo.net (Martin =?iso-8859-1?Q?L=FCthi?=)
Date: Fri, 11 Sep 2009 16:03:26 +0200
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<20090910200435.53ed9a5b@ajackson.org>
	<4AA9B629.8090209@ar.media.kyoto-u.ac.jp>
	<b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>
	<5b8d13220909110430r70eadacgcef4e67471613330@mail.gmail.com>
	<bcafe8290909110526l4c36d28fje922e54fbf4b1d26@mail.gmail.com>
Message-ID: <87bplhshn5.fsf@yak.ethz.ch>

Benny

Benny Malengier <benny.malengier at gmail.com> writes:

> This is actually the nice thing about the viral aspect of GPL. When
> you start a new toolkit in your department by extending a GPL
> component, you can tell the legal department that you don't have the
> time to reimplement existing components (time pressure you know) so
> you extend a GPL one, and now your component _must_ also be GPL. 

This is not viral, but inheritance. A virus injects it's DNA into
another body, so the linking clause could be called viral.

Best, tnoo



From d.l.goldsmith at gmail.com  Fri Sep 11 12:44:54 2009
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Fri, 11 Sep 2009 09:44:54 -0700
Subject: [SciPy-dev] SOT: Graphic parsing program
Message-ID: <45d1ab480909110944t29be4dd1w2770f64ea0c17a10@mail.gmail.com>

Hi!  Sorry for the somewhat OT post: can anyone refer me to a freeware Linux
or Windoze program for "parsing" graphics?  In particular, I want some thing
that can take the work-flow chart on the NumPy doc wiki "Front Page" and
reasonably successfully parse it into elemental objects that I can then
manipulate separately.  Thanks!

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090911/fd5ecc59/attachment.html>

From robert.kern at gmail.com  Fri Sep 11 12:54:11 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 11 Sep 2009 11:54:11 -0500
Subject: [SciPy-dev] SOT: Graphic parsing program
In-Reply-To: <45d1ab480909110944t29be4dd1w2770f64ea0c17a10@mail.gmail.com>
References: <45d1ab480909110944t29be4dd1w2770f64ea0c17a10@mail.gmail.com>
Message-ID: <3d375d730909110954n63bb6b5cr55e035ff89618085@mail.gmail.com>

On Fri, Sep 11, 2009 at 11:44, David Goldsmith<d.l.goldsmith at gmail.com> wrote:
> Hi!? Sorry for the somewhat OT post: can anyone refer me to a freeware Linux
> or Windoze program for "parsing" graphics?? In particular, I want some thing
> that can take the work-flow chart on the NumPy doc wiki "Front Page" and
> reasonably successfully parse it into elemental objects that I can then
> manipulate separately.? Thanks!

Not really, no. Ask the author for the source file that generated the
image. It might even be checked into the pydocweb repository.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From njs at pobox.com  Fri Sep 11 15:46:09 2009
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 11 Sep 2009 12:46:09 -0700
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <h8dl9t$llc$1@ger.gmane.org>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<20090910200435.53ed9a5b@ajackson.org>
	<4AA9B629.8090209@ar.media.kyoto-u.ac.jp>
	<b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>
	<h8dl9t$llc$1@ger.gmane.org>
Message-ID: <961fa2b40909111246k12b564eah7bfa8a9b915c5e8d@mail.gmail.com>

On Fri, Sep 11, 2009 at 7:02 AM, Pauli Virtanen <pav+sp at iki.fi> wrote:
> In a sense, also the drawbacks of the advertisement clause start to apply
> -- suppose many basic numerical codes had citation clauses. Most of the
> papers in all fields would have a citation to LAPACK. In a sense a good
> thing, in a sense a bit silly. Maybe a less of an annoyance than the
> advertisement clause, though, as you'd need to do this only once per
> paper.

It can also backfire hard -- if something as basic as LAPACK were GPL
incompatible (which a citation clause would make it), then we'd
probably all be using a rewrite under a more permissive license by
now, and LAPACK wouldn't get cited at all. (There's also the fun that
happens when every component of LAPACK has its own citation clause --
hope you weren't planning to publish in a journal with a # of
citations limit, like, oh, Science or Nature!)

My personal rule is that I don't touch code with GPL-incompatible
licenses. Life is too short, and too many of my basic tools are
GPL'ed. Your code is never more special and useful than all of the
GPL'ed code in the world put together. I mean, really.

No-one puts citation-requiring EULAs on their papers and yet life goes
on; I'd rather see software follow paper norms than papers start
following software norms.

-- Nathaniel


From josef.pktd at gmail.com  Fri Sep 11 16:19:33 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 11 Sep 2009 16:19:33 -0400
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <961fa2b40909111246k12b564eah7bfa8a9b915c5e8d@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<20090910200435.53ed9a5b@ajackson.org>
	<4AA9B629.8090209@ar.media.kyoto-u.ac.jp>
	<b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>
	<h8dl9t$llc$1@ger.gmane.org>
	<961fa2b40909111246k12b564eah7bfa8a9b915c5e8d@mail.gmail.com>
Message-ID: <1cd32cbb0909111319o252f9f9dwfe2441967393aeda@mail.gmail.com>

On Fri, Sep 11, 2009 at 3:46 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Sep 11, 2009 at 7:02 AM, Pauli Virtanen <pav+sp at iki.fi> wrote:
>> In a sense, also the drawbacks of the advertisement clause start to apply
>> -- suppose many basic numerical codes had citation clauses. Most of the
>> papers in all fields would have a citation to LAPACK. In a sense a good
>> thing, in a sense a bit silly. Maybe a less of an annoyance than the
>> advertisement clause, though, as you'd need to do this only once per
>> paper.
>
> It can also backfire hard -- if something as basic as LAPACK were GPL
> incompatible (which a citation clause would make it), then we'd
> probably all be using a rewrite under a more permissive license by
> now, and LAPACK wouldn't get cited at all. (There's also the fun that
> happens when every component of LAPACK has its own citation clause --
> hope you weren't planning to publish in a journal with a # of
> citations limit, like, oh, Science or Nature!)
>
> My personal rule is that I don't touch code with GPL-incompatible
> licenses. Life is too short, and too many of my basic tools are
> GPL'ed. Your code is never more special and useful than all of the
> GPL'ed code in the world put together. I mean, really.
>
> No-one puts citation-requiring EULAs on their papers and yet life goes
> on; I'd rather see software follow paper norms than papers start
> following software norms.
>
> -- Nathaniel

Many new statistical/econometrics procedures I have seen, have an
"academic/research use only" restriction, and there citation is part of
the professional ethics anyway (and can produce a large number of citation
for the author). But this is on a much "higher" level than LAPACK.

e.g. I looked for the license of this a few days ago
GAUSS GMM PACKAGE  Version 2.2 	 5/05/94
"This program package is free to all researchers, but please acknowledge
the National Science Foundation."

And these licenses are not so nice for the user if (s)he is not "academic".

Josef


From dwf at cs.toronto.edu  Sat Sep 12 03:42:04 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Sat, 12 Sep 2009 03:42:04 -0400
Subject: [SciPy-dev] [Fwd: Re: license status of your code on netlib]
In-Reply-To: <bcafe8290909110526l4c36d28fje922e54fbf4b1d26@mail.gmail.com>
References: <4AA8A731.8000101@hawaii.edu>
	<bcafe8290909100048u4bbd9e7dh2a4fba44e90e1297@mail.gmail.com>
	<20090910200435.53ed9a5b@ajackson.org>
	<4AA9B629.8090209@ar.media.kyoto-u.ac.jp>
	<b14b627d0909110347j33b8d321tc96abd5b8250d676@mail.gmail.com>
	<5b8d13220909110430r70eadacgcef4e67471613330@mail.gmail.com>
	<bcafe8290909110526l4c36d28fje922e54fbf4b1d26@mail.gmail.com>
Message-ID: <5C599279-3D43-4E59-A5B8-7E65129C3C50@cs.toronto.edu>

On 11-Sep-09, at 8:26 AM, Benny Malengier wrote:

> This is actually the nice thing about the viral aspect of GPL. When
> you start a new toolkit in your department by extending a GPL
> component, you can tell the legal department that you don't have the
> time to reimplement existing components (time pressure you know) so
> you extend a GPL one, and now your component _must_ also be GPL. For
> scipy this is more complicated, you must convince them you are better
> of contributing to the project, and people who don't know the issue
> have difficulty grasping the concept of this collaboration.

FWIW, I know the developers of several prominent BSD-licensed  
applications prefer BSD because they work for companies where their  
contracts stipulate that the company has a non-exclusive license to  
anything they create. Contributing to GPL software makes it much more  
complicated for their legal department, but the BSD license avoids  
that hurdle.

David


From d.l.goldsmith at gmail.com  Sun Sep 13 22:49:54 2009
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Sun, 13 Sep 2009 19:49:54 -0700
Subject: [SciPy-dev] K2C accepts negative valued input
Message-ID: <45d1ab480909131949w3c6856e3n23e8575aa1d18b5@mail.gmail.com>

Is the following behavior:

>>> from scipy.constants.constants import K2C
>>> K2C(-40)
-313.14999999999998

a bug or a feature?

(I know I've read about sub-absolute-zero temperature physics, e.g.,
http://cryo.gsfc.nasa.gov/introduction/neg_Kelvin.html, but, quoting that
site: "Physicists have defined a negative absolute temperature, but it's a
bit flakey...")


DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090913/bba9fcd0/attachment.html>

From charlesr.harris at gmail.com  Sun Sep 13 23:10:25 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 13 Sep 2009 21:10:25 -0600
Subject: [SciPy-dev] K2C accepts negative valued input
In-Reply-To: <45d1ab480909131949w3c6856e3n23e8575aa1d18b5@mail.gmail.com>
References: <45d1ab480909131949w3c6856e3n23e8575aa1d18b5@mail.gmail.com>
Message-ID: <e06186140909132010g5c56875esea7c34ad8758f47b@mail.gmail.com>

On Sun, Sep 13, 2009 at 8:49 PM, David Goldsmith <d.l.goldsmith at gmail.com>wrote:

> Is the following behavior:
>
> >>> from scipy.constants.constants import K2C
> >>> K2C(-40)
> -313.14999999999998
>
> a bug or a feature?
>
> (I know I've read about sub-absolute-zero temperature physics, e.g.,
> http://cryo.gsfc.nasa.gov/introduction/neg_Kelvin.html, but, quoting that
> site: "Physicists have defined a negative absolute temperature, but it's a
> bit flakey...")
>
>
It's a bit flakey. Raising an error would more likely indicate a problem
than not.

Chuck

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090913/82298bd2/attachment.html>

From d.l.goldsmith at gmail.com  Sun Sep 13 23:12:42 2009
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Sun, 13 Sep 2009 20:12:42 -0700
Subject: [SciPy-dev] K2C accepts negative valued input
In-Reply-To: <e06186140909132010g5c56875esea7c34ad8758f47b@mail.gmail.com>
References: <45d1ab480909131949w3c6856e3n23e8575aa1d18b5@mail.gmail.com>
	<e06186140909132010g5c56875esea7c34ad8758f47b@mail.gmail.com>
Message-ID: <45d1ab480909132012t1a88f342pc85a727112aed7c2@mail.gmail.com>

Are you "casting a vote" for "bug"?

DG

On Sun, Sep 13, 2009 at 8:10 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Sun, Sep 13, 2009 at 8:49 PM, David Goldsmith <d.l.goldsmith at gmail.com>wrote:
>
>> Is the following behavior:
>>
>> >>> from scipy.constants.constants import K2C
>> >>> K2C(-40)
>> -313.14999999999998
>>
>> a bug or a feature?
>>
>> (I know I've read about sub-absolute-zero temperature physics, e.g.,
>> http://cryo.gsfc.nasa.gov/introduction/neg_Kelvin.html, but, quoting that
>> site: "Physicists have defined a negative absolute temperature, but it's a
>> bit flakey...")
>>
>>
> It's a bit flakey. Raising an error would more likely indicate a problem
> than not.
>
> Chuck
>
>>
>>
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090913/78d94ab2/attachment.html>

From charlesr.harris at gmail.com  Sun Sep 13 23:35:17 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 13 Sep 2009 21:35:17 -0600
Subject: [SciPy-dev] K2C accepts negative valued input
In-Reply-To: <45d1ab480909132012t1a88f342pc85a727112aed7c2@mail.gmail.com>
References: <45d1ab480909131949w3c6856e3n23e8575aa1d18b5@mail.gmail.com>
	<e06186140909132010g5c56875esea7c34ad8758f47b@mail.gmail.com>
	<45d1ab480909132012t1a88f342pc85a727112aed7c2@mail.gmail.com>
Message-ID: <e06186140909132035ydcd75e7h5e90ea152c138959@mail.gmail.com>

On Sun, Sep 13, 2009 at 9:12 PM, David Goldsmith <d.l.goldsmith at gmail.com>wrote:

> Are you "casting a vote" for "bug"?
>
>
Yep.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090913/53f3957b/attachment.html>

From pgmdevlist at gmail.com  Mon Sep 14 18:52:57 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 14 Sep 2009 18:52:57 -0400
Subject: [SciPy-dev] API mismatch woes
Message-ID: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>

All,
I like to use
python -c "import setuptools; execfile('setup.py')" develop
to install numpy in a virtual environment, which simplifies testing  
local modifications. Unfortunately, I keep running into the API  
mismatch problem, even after getting rid of the build directory and  
the *.pyc files. I'm sure I must be missing some simple step, but  
which one ?

FYI,
OS : Mac 10.6
python : Python 2.6.2 (r262:71600, Sep 11 2009, 18:38:58) (installed  
in its own /usr/local/Cellar)
gcc : i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1

Note that the following command works like a charm.
pip install -E whatever -e svn+http://svn.scipy.org/svn/numpy/trunk/#egg=numpy



From robert.kern at gmail.com  Mon Sep 14 18:56:27 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 14 Sep 2009 17:56:27 -0500
Subject: [SciPy-dev] API mismatch woes
In-Reply-To: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>
References: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>
Message-ID: <3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>

On Mon, Sep 14, 2009 at 17:52, Pierre GM<pgmdevlist at gmail.com> wrote:
> All,
> I like to use
> python -c "import setuptools; execfile('setup.py')" develop
> to install numpy in a virtual environment, which simplifies testing
> local modifications. Unfortunately, I keep running into the API
> mismatch problem, even after getting rid of the build directory and
> the *.pyc files. I'm sure I must be missing some simple step, but
> which one ?

What API mismatch problem? Copy-and-paste full error messages, please.
I use "develop" all the time and do not run into this problem.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From d.l.goldsmith at gmail.com  Mon Sep 14 19:02:02 2009
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Mon, 14 Sep 2009 16:02:02 -0700
Subject: [SciPy-dev] Time for a reality check
Message-ID: <45d1ab480909141602j743d5be8qe09347860f2bc670@mail.gmail.com>

Hi, folks!  I'm having a heIluva time trying to get pydocweb to run without
"crashing" on my Kubuntu 9.0.4 install with Python2.6 and Django  1.0.2-1: I
can runserver, view the front page, create and login to an admin account,
view the control panel, enter and make changes in the admin site, but when I
try to Pull from sources, I get either a slough of permission denied errors
(if I didn't sudo manage.py) or (if I do sudo manage.py, which Pauli has
since told me I should not have to do) a

DjangoUnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
position 7504: ordinal not in range(128). You passed in
RuntimeError(<string>)

where <string> is longer than my Terminal's line retention buffer
(which is 1000 lines).

Pauli has me delving ever deeper into the pull scripts, but can't
confirm that there isn't something in pydocweb that makes it
incompatible w/ Python2.6, and is too busy
w/ his dissertation (which I certainly understand) to help me
substantially further.

So, before I agonize any more, I'm hoping to find out: 0) if _anyone_
is presently successfully running (including the Pull from sources
button/command)
pydocweb under Python2.6 and Django 1.0.2-1; and 1) even if someone(s)
is (are), if someone else running Python2.6 might be willing to take
the time to try
to install and run pydocweb (and all it's prereqs., of course) and let
me know if they can successfully Pull from sources.

Thanks!

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090914/20e956fa/attachment.html>

From pgmdevlist at gmail.com  Mon Sep 14 19:05:30 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 14 Sep 2009 19:05:30 -0400
Subject: [SciPy-dev] API mismatch woes
In-Reply-To: <3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>
References: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>
	<3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>
Message-ID: <BB7EF95A-4D4D-4088-9CB4-83A85A4756C4@gmail.com>


On Sep 14, 2009, at 6:56 PM, Robert Kern wrote:

> On Mon, Sep 14, 2009 at 17:52, Pierre GM<pgmdevlist at gmail.com> wrote:
>> All,
>> I like to use
>> python -c "import setuptools; execfile('setup.py')" develop
>> to install numpy in a virtual environment, which simplifies testing
>> local modifications. Unfortunately, I keep running into the API
>> mismatch problem, even after getting rid of the build directory and
>> the *.pyc files. I'm sure I must be missing some simple step, but
>> which one ?
>
> What API mismatch problem? Copy-and-paste full error messages, please.
> I use "develop" all the time and do not run into this problem.


Here's only the first 10 lines :

Running from numpy source directory.
non-existing path in 'numpy/distutils': 'site.cfg'
F2PY Version 2_7384
numpy/core/code_generators/genapi.py:9: DeprecationWarning: the md5  
module is deprecated; use hashlib instead
   import md5
numpy/core/setup_common.py:81: MismatchCAPIWarning: API mismatch  
detected, the C API version numbers have to be updated. Current C api  
version is 3, with checksum 4526bc5a07e51d13a2db642715fedca5, but  
recorded checksum for C API version 3 in codegen_dir/cversions.txt is  
bf22c0d05b31625d2a7015988d61ce5a. If functions were added in the C  
API, you have to update C_API_VERSION  in numpy/core/setup_common.py.
   MismatchCAPIWarning)
blas_opt_info:






From cournape at gmail.com  Mon Sep 14 19:21:14 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 15 Sep 2009 08:21:14 +0900
Subject: [SciPy-dev] API mismatch woes
In-Reply-To: <BB7EF95A-4D4D-4088-9CB4-83A85A4756C4@gmail.com>
References: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>
	<3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>
	<BB7EF95A-4D4D-4088-9CB4-83A85A4756C4@gmail.com>
Message-ID: <5b8d13220909141621p3b2d0c2ds314e75d31c6dad26@mail.gmail.com>

On Tue, Sep 15, 2009 at 8:05 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> numpy/core/setup_common.py:81: MismatchCAPIWarning: API mismatch
> detected, the C API version numbers have to be updated. Current C api
> version is 3, with checksum 4526bc5a07e51d13a2db642715fedca5, but
> recorded checksum for C API version 3 in codegen_dir/cversions.txt is
> bf22c0d05b31625d2a7015988d61ce5a. If functions were added in the C
> API, you have to update C_API_VERSION ?in numpy/core/setup_common.py.
> ? MismatchCAPIWarning)
> blas_opt_info:

This has nothing to do with setuptools or virtualenv - it is a
remainder to update the C API version before we release 1.4.0.

David


From pgmdevlist at gmail.com  Mon Sep 14 21:39:23 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 14 Sep 2009 21:39:23 -0400
Subject: [SciPy-dev] API mismatch woes
In-Reply-To: <5b8d13220909141621p3b2d0c2ds314e75d31c6dad26@mail.gmail.com>
References: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>
	<3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>
	<BB7EF95A-4D4D-4088-9CB4-83A85A4756C4@gmail.com>
	<5b8d13220909141621p3b2d0c2ds314e75d31c6dad26@mail.gmail.com>
Message-ID: <0AA14C64-BD2B-4222-8303-4D6D0234AC50@gmail.com>


On Sep 14, 2009, at 7:21 PM, David Cournapeau wrote:
>> This has nothing to do with setuptools or virtualenv - it is a
> remainder to update the C API version before we release 1.4.0.
>
> David

David, I got that. I still fail to understand why I get the message:  
I'm supposedly working from a clean rep, but it doesn't seem to be the  
case (else I wouldn't get the message). My guess is that some old file  
is accessed, but I can't find it. Any pointer would be deeply  
appreciated.
Cheers
P.


From cournape at gmail.com  Mon Sep 14 21:47:22 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 15 Sep 2009 10:47:22 +0900
Subject: [SciPy-dev] API mismatch woes
In-Reply-To: <0AA14C64-BD2B-4222-8303-4D6D0234AC50@gmail.com>
References: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>
	<3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>
	<BB7EF95A-4D4D-4088-9CB4-83A85A4756C4@gmail.com>
	<5b8d13220909141621p3b2d0c2ds314e75d31c6dad26@mail.gmail.com>
	<0AA14C64-BD2B-4222-8303-4D6D0234AC50@gmail.com>
Message-ID: <5b8d13220909141847t7487c920wf08cd76abf2acc68@mail.gmail.com>

On Tue, Sep 15, 2009 at 10:39 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> On Sep 14, 2009, at 7:21 PM, David Cournapeau wrote:
>>> This has nothing to do with setuptools or virtualenv - it is a
>> remainder to update the C API version before we release 1.4.0.
>>
>> David
>
> David, I got that. I still fail to understand why I get the message:
> I'm supposedly working from a clean rep, but it doesn't seem to be the
> case (else I wouldn't get the message). My guess is that some old file
> is accessed, but I can't find it. Any pointer would be deeply
> appreciated.

You should get the message, since the C api version has not been
updated, and there is an actual mismatch  (the C API has been changed
quite a bit since 1.3.0, and I don't want to update the version
several times - once the version will be updated in alpha phase, ther
won't be any message anymore).

David


From pgmdevlist at gmail.com  Mon Sep 14 22:23:21 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 14 Sep 2009 22:23:21 -0400
Subject: [SciPy-dev] API mismatch woes
In-Reply-To: <5b8d13220909141847t7487c920wf08cd76abf2acc68@mail.gmail.com>
References: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>
	<3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>
	<BB7EF95A-4D4D-4088-9CB4-83A85A4756C4@gmail.com>
	<5b8d13220909141621p3b2d0c2ds314e75d31c6dad26@mail.gmail.com>
	<0AA14C64-BD2B-4222-8303-4D6D0234AC50@gmail.com>
	<5b8d13220909141847t7487c920wf08cd76abf2acc68@mail.gmail.com>
Message-ID: <4EC329F7-6A65-4F7E-BFA1-C28D5381E492@gmail.com>


On Sep 14, 2009, at 9:47 PM, David Cournapeau wrote:
>
> You should get the message, since the C api version has not been
> updated, and there is an actual mismatch  (the C API has been changed
> quite a bit since 1.3.0, and I don't want to update the version
> several times - once the version will be updated in alpha phase, ther
> won't be any message anymore).

David,
Please be patient with me... Should I understand that it's normal that  
I get the message, and that it doesn't really matter at this point ?
Sorry for the noise...


From nmb at wartburg.edu  Mon Sep 14 22:30:00 2009
From: nmb at wartburg.edu (Neil Martinsen-Burrell)
Date: Mon, 14 Sep 2009 21:30:00 -0500
Subject: [SciPy-dev] API mismatch woes
In-Reply-To: <4EC329F7-6A65-4F7E-BFA1-C28D5381E492@gmail.com>
References: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>	<3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>	<BB7EF95A-4D4D-4088-9CB4-83A85A4756C4@gmail.com>	<5b8d13220909141621p3b2d0c2ds314e75d31c6dad26@mail.gmail.com>	<0AA14C64-BD2B-4222-8303-4D6D0234AC50@gmail.com>	<5b8d13220909141847t7487c920wf08cd76abf2acc68@mail.gmail.com>
	<4EC329F7-6A65-4F7E-BFA1-C28D5381E492@gmail.com>
Message-ID: <4AAEFC28.6060005@wartburg.edu>

On 2009-09-14 21:23 , Pierre GM wrote:
>
> On Sep 14, 2009, at 9:47 PM, David Cournapeau wrote:
>>
>> You should get the message, since the C api version has not been
>> updated, and there is an actual mismatch  (the C API has been changed
>> quite a bit since 1.3.0, and I don't want to update the version
>> several times - once the version will be updated in alpha phase, ther
>> won't be any message anymore).
>
> David,
> Please be patient with me... Should I understand that it's normal that
> I get the message, and that it doesn't really matter at this point ?
> Sorry for the noise...

I believe that David means that it is normal to get this message at this 
point in Numpy's development cycle.  If you are still getting it in the 
1.4.0a stage, then it would be a problem.  Just my reading of David's 
work on C API versioning.

-Neil


From cournape at gmail.com  Mon Sep 14 23:34:44 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 15 Sep 2009 12:34:44 +0900
Subject: [SciPy-dev] API mismatch woes
In-Reply-To: <4EC329F7-6A65-4F7E-BFA1-C28D5381E492@gmail.com>
References: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>
	<3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>
	<BB7EF95A-4D4D-4088-9CB4-83A85A4756C4@gmail.com>
	<5b8d13220909141621p3b2d0c2ds314e75d31c6dad26@mail.gmail.com>
	<0AA14C64-BD2B-4222-8303-4D6D0234AC50@gmail.com>
	<5b8d13220909141847t7487c920wf08cd76abf2acc68@mail.gmail.com>
	<4EC329F7-6A65-4F7E-BFA1-C28D5381E492@gmail.com>
Message-ID: <5b8d13220909142034mef10061v4a2166a33f797493@mail.gmail.com>

On Tue, Sep 15, 2009 at 11:23 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
>
> David,
> Please be patient with me... Should I understand that it's normal that
> I get the message, and that it doesn't really matter at this point ?

Yes, sorry if I was not very clear,

David


From pgmdevlist at gmail.com  Mon Sep 14 23:41:01 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 14 Sep 2009 23:41:01 -0400
Subject: [SciPy-dev] API mismatch woes
In-Reply-To: <5b8d13220909142034mef10061v4a2166a33f797493@mail.gmail.com>
References: <A6536BC3-DA7D-4FA1-B6D2-8A60D19EFC24@gmail.com>
	<3d375d730909141556o2d937130x8f08b52b8922f043@mail.gmail.com>
	<BB7EF95A-4D4D-4088-9CB4-83A85A4756C4@gmail.com>
	<5b8d13220909141621p3b2d0c2ds314e75d31c6dad26@mail.gmail.com>
	<0AA14C64-BD2B-4222-8303-4D6D0234AC50@gmail.com>
	<5b8d13220909141847t7487c920wf08cd76abf2acc68@mail.gmail.com>
	<4EC329F7-6A65-4F7E-BFA1-C28D5381E492@gmail.com>
	<5b8d13220909142034mef10061v4a2166a33f797493@mail.gmail.com>
Message-ID: <2A656475-6DE3-4F01-B8ED-2753EB410850@gmail.com>


On Sep 14, 2009, at 11:34 PM, David Cournapeau wrote:

> On Tue, Sep 15, 2009 at 11:23 AM, Pierre GM <pgmdevlist at gmail.com>  
> wrote:
>>
>>
>> David,
>> Please be patient with me... Should I understand that it's normal  
>> that
>> I get the message, and that it doesn't really matter at this point ?
>
> Yes, sorry if I was not very clear,

Fab'! Thanks again.
P.


From pav+sp at iki.fi  Tue Sep 15 03:18:27 2009
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Tue, 15 Sep 2009 07:18:27 +0000 (UTC)
Subject: [SciPy-dev] Time for a reality check
References: <45d1ab480909141602j743d5be8qe09347860f2bc670@mail.gmail.com>
Message-ID: <h8nf43$qpa$1@ger.gmane.org>

Mon, 14 Sep 2009 16:02:02 -0700, David Goldsmith wrote:
[clip]
> So, before I agonize any more, I'm hoping to find out: 0) if _anyone_ is
> presently successfully running (including the Pull from sources
> button/command)
> pydocweb under Python2.6 and Django 1.0.2-1; and 1) even if someone(s)
> is (are), if someone else running Python2.6 might be willing to take the
> time to try
> to install and run pydocweb (and all it's prereqs., of course) and let
> me know if they can successfully Pull from sources.

I can run it without problems on Python 2.5 and 2.6, so I think there 
should be no Python 2.6 incompatibilities. Version of Django does not 
matter here, as the part that is failing does not use Django.

I suspect there's something out-of-place in your setup, so the best 
advice I have is to check out the Pydocweb sources from bzr, to a 
different place where you have them now, and try again, not using sudo, 
and being careful when modifying the pull script.

-- 
Pauli Virtanen



From pgmdevlist at gmail.com  Tue Sep 15 12:38:55 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 15 Sep 2009 12:38:55 -0400
Subject: [SciPy-dev] SciPy SVN compilation on SnowLeopard
Message-ID: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>

All,
I'm banging my head on some configuration issues for Scipy on Mac OS  
10.6. I followed the recommendations from this site [http://blog.hyperjeff.net/?p=160 
], but there's obviously something very wrong in my system: _csr,  
spline, numpyio cannot be loaded because of a wrong architecture.
(for example, the SciPy test suite gives me ImportError: dlopen(/Users/ 
pierregm/Computing/workspace/scipy-dev/scipy/io/numpyio.so, 2): no  
suitable image found.  Did find:  /Users/pierregm/Computing/workspace/ 
scipy-dev/scipy/io/numpyio.so: mach-o, but wrong architecture)

Any hint would be welcome.

And as a side note, I have to confirm bug #1194 of numpy (I have to  
symlink libnpymath.a to somewhere it could be found).


Here are some relevant details

gcc --version : i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc.  
build 5646) (from Xcode 3.2, 64b)

gfortran --version : GNU Fortran (GCC) 4.2.3 (from

gfortran -m64 -v : Using built-in specs.
Target: i686-apple-darwin8
Configured with: /Builds/unix/gcc/gcc-4.2/configure --prefix=/usr/ 
local --mandir=/share/man --program-transform-name=/^[cg][^.-]*$/s/ 
$/-4.2/ --build=i686-apple-darwin8 --host=i686-apple-darwin8 -- 
target=i686-apple-darwin8 --enable-languages=fortran
Thread model: posix
gcc version 4.2.3

F77=?gfortran -m64?
FC=?gfortran -m64?

buildlog: http://pastebin.com/m11b9627d




From cournape at gmail.com  Wed Sep 16 01:35:43 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 16 Sep 2009 14:35:43 +0900
Subject: [SciPy-dev] SciPy SVN compilation on SnowLeopard
In-Reply-To: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>
References: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>
Message-ID: <5b8d13220909152235ve497e08mad4fb880fb676c1a@mail.gmail.com>

On Wed, Sep 16, 2009 at 1:38 AM, Pierre GM <pgmdevlist at gmail.com> wrote:
> All,
> I'm banging my head on some configuration issues for Scipy on Mac OS
> 10.6. I followed the recommendations from this site [http://blog.hyperjeff.net/?p=160
> ], but there's obviously something very wrong in my system: _csr,
> spline, numpyio cannot be loaded because of a wrong architecture.
> (for example, the SciPy test suite gives me ImportError: dlopen(/Users/
> pierregm/Computing/workspace/scipy-dev/scipy/io/numpyio.so, 2): no
> suitable image found. ?Did find: ?/Users/pierregm/Computing/workspace/
> scipy-dev/scipy/io/numpyio.so: mach-o, but wrong architecture)
>
> Any hint would be welcome.
>
> And as a side note, I have to confirm bug #1194 of numpy (I have to
> symlink libnpymath.a to somewhere it could be found).

Can you check whether r7387 fixes the problem ?

Concerning your first problem, I don't think the information given on
the link are really good. They are overly complicated, and using sudo
and modifying apple-installed numpy is a bad idea. Also, sudo does not
simply propagate environment variables.

If all you do is using the system python (the one from apple), then
you only need to set up FFLAGS because gfortran does not target x86_64
by default.

cheers,

David


From pgmdevlist at gmail.com  Wed Sep 16 02:06:08 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 16 Sep 2009 02:06:08 -0400
Subject: [SciPy-dev] SciPy SVN compilation on SnowLeopard
In-Reply-To: <5b8d13220909152235ve497e08mad4fb880fb676c1a@mail.gmail.com>
References: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>
	<5b8d13220909152235ve497e08mad4fb880fb676c1a@mail.gmail.com>
Message-ID: <A548608C-41E7-40FD-BD6D-94744929051B@gmail.com>


On Sep 16, 2009, at 1:35 AM, David Cournapeau wrote:

> On Wed, Sep 16, 2009 at 1:38 AM, Pierre GM <pgmdevlist at gmail.com>  
> wrote:
>
> Can you check whether r7387 fixes the problem ?

I didn't try to recompile scipy w/ numpy r7387 yet, but this latter  
already raises a second issue that maybe related to my scipy  
installation problems.
I tried to recompile some fortran libraries that I need for a project,  
and it fails (log available here: http://pastebin.com/m7d38b395).  
Looks like lipo can't figure the architecture of some of the temp  
files... Any pointer would be deeply appreciated.


> If all you do is using the system python (the one from apple), then
> you only need to set up FFLAGS because gfortran does not target x86_64
> by default.

Actually, I'm using Python2.6.2 compiled from sources (via homebrew,  
figured I could give it a try). What other flag should I set ?
Now, does the issue raised over here [http://www.nabble.com/-MacPorts---19397%3A-scipy-not-completely-universal-td23174931.html#a23174931 
] relevant to our problem ? (w/ gfortran failing to ouput anything  
when given 2 arch flags ?)


From david at ar.media.kyoto-u.ac.jp  Wed Sep 16 01:56:11 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 16 Sep 2009 14:56:11 +0900
Subject: [SciPy-dev] SciPy SVN compilation on SnowLeopard
In-Reply-To: <A548608C-41E7-40FD-BD6D-94744929051B@gmail.com>
References: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>	<5b8d13220909152235ve497e08mad4fb880fb676c1a@mail.gmail.com>
	<A548608C-41E7-40FD-BD6D-94744929051B@gmail.com>
Message-ID: <4AB07DFB.9070107@ar.media.kyoto-u.ac.jp>

Pierre GM wrote:
>
> I didn't try to recompile scipy w/ numpy r7387 yet, but this latter  
> already raises a second issue that maybe related to my scipy  
> installation problems.
> I tried to recompile some fortran libraries that I need for a project,  
> and it fails (log available here: http://pastebin.com/m7d38b395).  
> Looks like lipo can't figure the architecture of some of the temp  
> files... Any pointer would be deeply appreciated.
>   

I would try to fix the compilation problem first - the .o file, if it
exists at all (which I doubt), would certainly be garbage since the code
cannot be parsed by gcc.

>
>   
>> If all you do is using the system python (the one from apple), then
>> you only need to set up FFLAGS because gfortran does not target x86_64
>> by default.
>>     
>
> Actually, I'm using Python2.6.2 compiled from sources (via homebrew,  
> figured I could give it a try). What other flag should I set ?
>   

Then I don't know, it will depend on the flags you set yourself (or the
default ones). But python does not support 64 bits arch on mac os X yet
very well - the apple python uses some apple specific patches.

I think that for the time being, unless you really need it, you should
avoid using non-apple python if you care about 64 bits.

> Now, does the issue raised over here [http://www.nabble.com/-MacPorts---19397%3A-scipy-not-completely-universal-td23174931.html#a23174931 
> ] relevant to our problem ? 

No, because it was fixed some time ago in numpy.

cheers,

David



From pgmdevlist at gmail.com  Wed Sep 16 02:53:38 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 16 Sep 2009 02:53:38 -0400
Subject: [SciPy-dev] SciPy SVN compilation on SnowLeopard
In-Reply-To: <4AB07DFB.9070107@ar.media.kyoto-u.ac.jp>
References: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>	<5b8d13220909152235ve497e08mad4fb880fb676c1a@mail.gmail.com>
	<A548608C-41E7-40FD-BD6D-94744929051B@gmail.com>
	<4AB07DFB.9070107@ar.media.kyoto-u.ac.jp>
Message-ID: <BEC5BE45-A1D3-401E-95B3-238D755FD9F8@gmail.com>


On Sep 16, 2009, at 1:56 AM, David Cournapeau wrote:
> I would try to fix the compilation problem first - the .o file, if it
> exists at all (which I doubt), would certainly be garbage since the  
> code
> cannot be parsed by gcc.

And that's my problem, because I've no idea where to start...


> Then I don't know, it will depend on the flags you set yourself (or  
> the
> default ones). But python does not support 64 bits arch on mac os X  
> yet
> very well - the apple python uses some apple specific patches.
> I think that for the time being, unless you really need it, you should
> avoid using non-apple python if you care about 64 bits.

Oh my. What's the recommended safe procedure in that case ? Can  
packages be safely stored in /Library/Python/ or should I force  
storage to ~/Profile

>> Now, does the issue raised over here [http://www.nabble.com/- 
>> MacPorts---19397%3A-scipy-not-completely-universal- 
>> td23174931.html#a23174931
>> ] relevant to our problem ?
>
> No, because it was fixed some time ago in numpy.

I didn't expect less :)
Thx again


From david at ar.media.kyoto-u.ac.jp  Wed Sep 16 02:39:56 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 16 Sep 2009 15:39:56 +0900
Subject: [SciPy-dev] SciPy SVN compilation on SnowLeopard
In-Reply-To: <BEC5BE45-A1D3-401E-95B3-238D755FD9F8@gmail.com>
References: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>	<5b8d13220909152235ve497e08mad4fb880fb676c1a@mail.gmail.com>	<A548608C-41E7-40FD-BD6D-94744929051B@gmail.com>	<4AB07DFB.9070107@ar.media.kyoto-u.ac.jp>
	<BEC5BE45-A1D3-401E-95B3-238D755FD9F8@gmail.com>
Message-ID: <4AB0883C.7000600@ar.media.kyoto-u.ac.jp>

Pierre GM wrote:
> On Sep 16, 2009, at 1:56 AM, David Cournapeau wrote:
>   
>> I would try to fix the compilation problem first - the .o file, if it
>> exists at all (which I doubt), would certainly be garbage since the  
>> code
>> cannot be parsed by gcc.
>>     
>
> And that's my problem, because I've no idea where to start...
>
>
>   
>> Then I don't know, it will depend on the flags you set yourself (or  
>> the
>> default ones). But python does not support 64 bits arch on mac os X  
>> yet
>> very well - the apple python uses some apple specific patches.
>> I think that for the time being, unless you really need it, you should
>> avoid using non-apple python if you care about 64 bits.
>>     
>
> Oh my. What's the recommended safe procedure in that case ? Can  
> packages be safely stored in /Library/Python/ or should I force  
> storage to ~/Profile
>   

You can use the --user option of install (available in 2.6 and above).
This will install stuff into $HOME/.local, and
$HOME/.local/lib/python2.6/site-packages is automatically added to
sys.path by python.

Or use virtualenv, stow, whatever tool you may prefer to handle multiple
versions of those packages,

David


From dwf at cs.toronto.edu  Wed Sep 16 03:07:16 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 16 Sep 2009 03:07:16 -0400
Subject: [SciPy-dev] SciPy SVN compilation on SnowLeopard
In-Reply-To: <4AB0883C.7000600@ar.media.kyoto-u.ac.jp>
References: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>	<5b8d13220909152235ve497e08mad4fb880fb676c1a@mail.gmail.com>	<A548608C-41E7-40FD-BD6D-94744929051B@gmail.com>	<4AB07DFB.9070107@ar.media.kyoto-u.ac.jp>
	<BEC5BE45-A1D3-401E-95B3-238D755FD9F8@gmail.com>
	<4AB0883C.7000600@ar.media.kyoto-u.ac.jp>
Message-ID: <5D3A5443-3FFF-43C7-B0AF-4C83A3F18547@cs.toronto.edu>

On 16-Sep-09, at 2:39 AM, David Cournapeau wrote:

> You can use the --user option of install (available in 2.6 and above).
> This will install stuff into $HOME/.local, and
> $HOME/.local/lib/python2.6/site-packages is automatically added to
> sys.path by python.
>
> Or use virtualenv, stow, whatever tool you may prefer to handle  
> multiple
> versions of those packages,

I find the safest way is to install Python.org's binaries and modify  
your path to use /Library/Frameworks/Python.framework/Version/2.6/bin  
before the corresponding /System... path. Then you can keep this  
Python's site-packages clean (and use --user and virtualenv too, if  
you wish).

David


From pgmdevlist at gmail.com  Wed Sep 16 03:09:29 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 16 Sep 2009 03:09:29 -0400
Subject: [SciPy-dev] SciPy SVN compilation on SnowLeopard
In-Reply-To: <4AB0883C.7000600@ar.media.kyoto-u.ac.jp>
References: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>	<5b8d13220909152235ve497e08mad4fb880fb676c1a@mail.gmail.com>	<A548608C-41E7-40FD-BD6D-94744929051B@gmail.com>	<4AB07DFB.9070107@ar.media.kyoto-u.ac.jp>
	<BEC5BE45-A1D3-401E-95B3-238D755FD9F8@gmail.com>
	<4AB0883C.7000600@ar.media.kyoto-u.ac.jp>
Message-ID: <12AA90EF-56E9-4442-AFBD-1ABC762E60CC@gmail.com>


On Sep 16, 2009, at 2:39 AM, David Cournapeau wrote:
>> Oh my. What's the recommended safe procedure in that case ? Can
>> packages be safely stored in /Library/Python/ or should I force
>> storage to ~/Profile
>
> You can use the --user option of install (available in 2.6 and above).
> This will install stuff into $HOME/.local, and
> $HOME/.local/lib/python2.6/site-packages is automatically added to
> sys.path by python.
>
> Or use virtualenv, stow, whatever tool you may prefer to handle  
> multiple
> versions of those packages,

I'm a big fan of virtualenv already, I just wanted to make sure that I  
could install some packages (pip, sphinx...) in /Library/Python.
Thx again


From dwf at cs.toronto.edu  Wed Sep 16 14:50:23 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 16 Sep 2009 14:50:23 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
Message-ID: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>

Hi,

I just noticed that the way _sf() is implemented in the base class is  
particularly horrible for continuous distributions that are symmetric  
about the origin, where 1.0 - cdf(x) is prone to roundoff error and  
could just as easily be calculated as cdf(-x).

I wonder if this is a common enough case that that there ought to be  
an rv_continuous_zerosym class that candidate distributions inherit  
from, or if we should just override the method on a case by case basis.

David


From pgmdevlist at gmail.com  Wed Sep 16 15:05:20 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 16 Sep 2009 15:05:20 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
Message-ID: <C4C19175-BB9E-4BDF-9D6D-77CE19A1F41C@gmail.com>


On Sep 16, 2009, at 2:50 PM, David Warde-Farley wrote:
> I wonder if this is a common enough case that that there ought to be
> an rv_continuous_zerosym class that candidate distributions inherit
> from, or if we should just override the method on a case by case  
> basis.

+1 for 1: create a subclass.


From josef.pktd at gmail.com  Wed Sep 16 15:06:51 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 16 Sep 2009 15:06:51 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
Message-ID: <1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>

On Wed, Sep 16, 2009 at 2:50 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> Hi,
>
> I just noticed that the way _sf() is implemented in the base class is
> particularly horrible for continuous distributions that are symmetric
> about the origin, where 1.0 - cdf(x) is prone to roundoff error and
> could just as easily be calculated as cdf(-x).
>
> I wonder if this is a common enough case that that there ought to be
> an rv_continuous_zerosym class that candidate distributions inherit
> from, or if we should just override the method on a case by case basis.
>
> David

Without checking the different cases, I would say, add them to the individual
distributions, def _sf(self) ... return self._cdf(-x)

I'm not sure there are enough cases to separate them out for
numerical improvements, if it only needs the addition of one or two lines
for the relevant cases.

There are also other cases, where the generic procedure could be chosen
in a smarter way to make the calculations faster or more accurate.

Besides some improvements by Per, not much has been done to improve
numerical precision.

Josef


> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From pgmdevlist at gmail.com  Wed Sep 16 15:17:28 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 16 Sep 2009 15:17:28 -0400
Subject: [SciPy-dev] SciPy SVN compilation on SnowLeopard
In-Reply-To: <4AB07DFB.9070107@ar.media.kyoto-u.ac.jp>
References: <B8C845D9-F030-4915-87E1-66C7E7600901@gmail.com>	<5b8d13220909152235ve497e08mad4fb880fb676c1a@mail.gmail.com>
	<A548608C-41E7-40FD-BD6D-94744929051B@gmail.com>
	<4AB07DFB.9070107@ar.media.kyoto-u.ac.jp>
Message-ID: <9FF582E4-6599-45D4-8F0A-D6592F0DC92F@gmail.com>


On Sep 16, 2009, at 1:56 AM, David Cournapeau wrote:
>
> I think that for the time being, unless you really need it, you should
> avoid using non-apple python if you care about 64 bits.


Well, that was that. I recompiled numpy/scipy using the Apple python,  
and it seems to work: the fortran-based modules are properly imported,  
the test suite runs w/o too many problems: please find the output here  
[http://pastebin.com/m399993fb].
David, I'm in your debt, I'd still be running in circles without your  
help.


From josef.pktd at gmail.com  Wed Sep 16 15:17:48 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 16 Sep 2009 15:17:48 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
Message-ID: <1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>

On Wed, Sep 16, 2009 at 3:06 PM,  <josef.pktd at gmail.com> wrote:
> On Wed, Sep 16, 2009 at 2:50 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>> Hi,
>>
>> I just noticed that the way _sf() is implemented in the base class is
>> particularly horrible for continuous distributions that are symmetric
>> about the origin, where 1.0 - cdf(x) is prone to roundoff error and
>> could just as easily be calculated as cdf(-x).
>>
>> I wonder if this is a common enough case that that there ought to be
>> an rv_continuous_zerosym class that candidate distributions inherit
>> from, or if we should just override the method on a case by case basis.
>>
>> David
>
> Without checking the different cases, I would say, add them to the individual
> distributions, def _sf(self) ... return self._cdf(-x)
>
> I'm not sure there are enough cases to separate them out for
> numerical improvements, if it only needs the addition of one or two lines
> for the relevant cases.
>
> There are also other cases, where the generic procedure could be chosen
> in a smarter way to make the calculations faster or more accurate.
>
> Besides some improvements by Per, not much has been done to improve
> numerical precision.
>
> Josef
>

Did you check for which distributions this would apply?
I remember there was some problem that pymvpa had with the r
distribution because the numerical integration close to the lower
bound didn't work.
I thought for some time whether I should use to upper tail instead,
which would be the reverse of what you are proposing, but I didn't
manage to figure out which would be more accurate. I was also thinking
whether numerical integration should start at the center for symmetric
distributions. But I know too little about the precision of numerical
integration to fight for 1e-XX.

Josef

>
>> _______________________________________________
>> Scipy-dev mailing list
>> Scipy-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>


From dwf at cs.toronto.edu  Wed Sep 16 15:53:13 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 16 Sep 2009 15:53:13 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
Message-ID: <F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>

On 16-Sep-09, at 3:17 PM, josef.pktd at gmail.com wrote:

> Did you check for which distributions this would apply?
> I remember there was some problem that pymvpa had with the r
> distribution because the numerical integration close to the lower
> bound didn't work.

I haven't gone through the list exhaustively but I'm fairly certain it  
will make a positive difference for the t-distribution and the  
standard normal, especially since these are used in statistical tests  
where people care about their p-values.

I just committed a fix for these two distributions, we can weigh the  
worth of subclassing or not further (incidentally, this breaks two  
ttest tests because they differ in the 17th decimal place, how sure  
are you of those hard-coded numbers?).

David


From dwf at cs.toronto.edu  Wed Sep 16 17:01:47 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 16 Sep 2009 17:01:47 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
Message-ID: <E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>

On 16-Sep-09, at 3:53 PM, David Warde-Farley wrote:

> incidentally, this breaks two
> ttest tests because they differ in the 17th decimal place, how sure
> are you of those hard-coded numbers?).

I just did the integral with Sage and requested 100 bits of precision,  
more than we need, and got

sage: thepdf
1/4*sqrt(2)/(1/2*x^2 + 1)^(3/2)
sage: n(2 * integral(thepdf, x, -infinity, -1), prec=100)
0.42264973081037423549085121950


And in IPython:

In [92]: np.float64(0.42264973081037423549085121950)
Out[92]: 0.42264973081037421

So it seems that ...21 is the right answer and the hardcoded ...27 is  
actually the result of roundoff, it would seem.

I've fixed the tests.

David


From josef.pktd at gmail.com  Wed Sep 16 17:13:13 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 16 Sep 2009 17:13:13 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
Message-ID: <1cd32cbb0909161413n1c7241cdrd8a7999cea449ac2@mail.gmail.com>

On Wed, Sep 16, 2009 at 3:53 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> On 16-Sep-09, at 3:17 PM, josef.pktd at gmail.com wrote:
>
>> Did you check for which distributions this would apply?
>> I remember there was some problem that pymvpa had with the r
>> distribution because the numerical integration close to the lower
>> bound didn't work.
>
> I haven't gone through the list exhaustively but I'm fairly certain it
> will make a positive difference for the t-distribution and the
> standard normal, especially since these are used in statistical tests
> where people care about their p-values.

good, your changes are similar to what Per proposed in
http://projects.scipy.org/scipy/changeset/5602/trunk/scipy/stats/distributions.py

There might be a few more cases like these.


>
> I just committed a fix for these two distributions, we can weigh the
> worth of subclassing or not further (incidentally, this breaks two
> ttest tests because they differ in the 17th decimal place, how sure
> are you of those hard-coded numbers?).
>

I got busy and just saw your follow up.
If there is a request for higher than standard precision of
floating point calculationss, then it should be a bug in the
test. I guess, I missed adding decimal if it is in one of
my tests.

Which ttest test is it?

Josef


> David
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From pav at iki.fi  Wed Sep 16 17:25:21 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 17 Sep 2009 00:25:21 +0300
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
Message-ID: <1253136320.1990.184.camel@idol>

ke, 2009-09-16 kello 17:01 -0400, David Warde-Farley kirjoitti:
> On 16-Sep-09, at 3:53 PM, David Warde-Farley wrote:
> 
> > incidentally, this breaks two
> > ttest tests because they differ in the 17th decimal place, how sure
> > are you of those hard-coded numbers?).
> 
> I just did the integral with Sage and requested 100 bits of precision,  
> more than we need, and got
> 
> sage: thepdf
> 1/4*sqrt(2)/(1/2*x^2 + 1)^(3/2)
> sage: n(2 * integral(thepdf, x, -infinity, -1), prec=100)
> 0.42264973081037423549085121950

Btw, this integral evaluates to

	1 - 1/sqrt(3)

Now, depending on how you evaluate this in floating point, you get
different results:

	1. - 1./sqrt(3.)        = 0.42264973081037416
	(sqrt(3.) - 1)/sqrt(3.) = 0.42264973081037421
	(3. - sqrt(3.))/3.      = 0.42264973081037427

Perhaps the 0.###27 was obtained like this...

It's nice if you can actually maintain this level of accuracy in
scipy.stats in a way that works on all platforms -- doesn't work in
scipy.special as the boost tests show.

Cheers,
Pauli





From josef.pktd at gmail.com  Wed Sep 16 17:27:14 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 16 Sep 2009 17:27:14 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <1cd32cbb0909161413n1c7241cdrd8a7999cea449ac2@mail.gmail.com>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<1cd32cbb0909161413n1c7241cdrd8a7999cea449ac2@mail.gmail.com>
Message-ID: <1cd32cbb0909161427r40f8a28bn400508b09d428590@mail.gmail.com>

On Wed, Sep 16, 2009 at 5:13 PM,  <josef.pktd at gmail.com> wrote:
> On Wed, Sep 16, 2009 at 3:53 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>> On 16-Sep-09, at 3:17 PM, josef.pktd at gmail.com wrote:
>>
>>> Did you check for which distributions this would apply?
>>> I remember there was some problem that pymvpa had with the r
>>> distribution because the numerical integration close to the lower
>>> bound didn't work.
>>
>> I haven't gone through the list exhaustively but I'm fairly certain it
>> will make a positive difference for the t-distribution and the
>> standard normal, especially since these are used in statistical tests
>> where people care about their p-values.
>
> good, your changes are similar to what Per proposed in
> http://projects.scipy.org/scipy/changeset/5602/trunk/scipy/stats/distributions.py
>
> There might be a few more cases like these.
>
>
>>
>> I just committed a fix for these two distributions, we can weigh the
>> worth of subclassing or not further (incidentally, this breaks two
>> ttest tests because they differ in the 17th decimal place, how sure
>> are you of those hard-coded numbers?).
>>
>
> I got busy and just saw your follow up.
> If there is a request for higher than standard precision of
> floating point calculationss, then it should be a bug in the
> test. I guess, I missed adding decimal if it is in one of
> my tests.
>
> Which ttest test is it?

I saw the changes, Hooray we don't reject the null with
0.42264973081037421 instead of 0.42264973081037427 probability ;)

I think these tests should have used assert_almost_equal because of
the float. I might have been distracted because of the integer. And
these are some of my first tests in scipy, which might still be a bit
shaky.
These tests are just regression tests (as it says on top), so any
improvement in numerical precision might require some
adjustments.

Josef

>
> Josef
>
>
>> David
>> _______________________________________________
>> Scipy-dev mailing list
>> Scipy-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>


From dwf at cs.toronto.edu  Wed Sep 16 18:24:10 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 16 Sep 2009 18:24:10 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <1253136320.1990.184.camel@idol>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
	<1253136320.1990.184.camel@idol>
Message-ID: <62641D59-1D6B-4B85-A30F-DBEF517F7622@cs.toronto.edu>

On 16-Sep-09, at 5:25 PM, Pauli Virtanen wrote:

>> sage: thepdf
>> 1/4*sqrt(2)/(1/2*x^2 + 1)^(3/2)
>> sage: n(2 * integral(thepdf, x, -infinity, -1), prec=100)
>> 0.42264973081037423549085121950
>
> Btw, this integral evaluates to
>
> 	1 - 1/sqrt(3)
>
> Now, depending on how you evaluate this in floating point, you get
> different results:
>
> 	1. - 1./sqrt(3.)        = 0.42264973081037416
> 	(sqrt(3.) - 1)/sqrt(3.) = 0.42264973081037421
> 	(3. - sqrt(3.))/3.      = 0.42264973081037427
>
> Perhaps the 0.###27 was obtained like this...

It was obtained via 1 - scipy.special(2, 1),

> It's nice if you can actually maintain this level of accuracy in
> scipy.stats in a way that works on all platforms -- doesn't work in
> scipy.special as the boost tests show.

Well, I guess we're still in trouble (depending on the special  
function) but perhaps less trouble since we're not doing an  
unnecessary subtraction.

Anyway, the ###21 result is closer than ###27 so I'm taking pleasure  
in (*extremely*) small things.

David


From josef.pktd at gmail.com  Wed Sep 16 18:40:03 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 16 Sep 2009 18:40:03 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <62641D59-1D6B-4B85-A30F-DBEF517F7622@cs.toronto.edu>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
	<1253136320.1990.184.camel@idol>
	<62641D59-1D6B-4B85-A30F-DBEF517F7622@cs.toronto.edu>
Message-ID: <1cd32cbb0909161540k3300a928y59c174b2f2c5a1e2@mail.gmail.com>

On Wed, Sep 16, 2009 at 6:24 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> On 16-Sep-09, at 5:25 PM, Pauli Virtanen wrote:
>
>>> sage: thepdf
>>> 1/4*sqrt(2)/(1/2*x^2 + 1)^(3/2)
>>> sage: n(2 * integral(thepdf, x, -infinity, -1), prec=100)
>>> 0.42264973081037423549085121950
>>
>> Btw, this integral evaluates to
>>
>> ? ? ? 1 - 1/sqrt(3)
>>
>> Now, depending on how you evaluate this in floating point, you get
>> different results:
>>
>> ? ? ? 1. - 1./sqrt(3.) ? ? ? ?= 0.42264973081037416
>> ? ? ? (sqrt(3.) - 1)/sqrt(3.) = 0.42264973081037421
>> ? ? ? (3. - sqrt(3.))/3. ? ? ?= 0.42264973081037427
>>
>> Perhaps the 0.###27 was obtained like this...
>
> It was obtained via 1 - scipy.special(2, 1),
>
>> It's nice if you can actually maintain this level of accuracy in
>> scipy.stats in a way that works on all platforms -- doesn't work in
>> scipy.special as the boost tests show.
>
> Well, I guess we're still in trouble (depending on the special
> function) but perhaps less trouble since we're not doing an
> unnecessary subtraction.
>
> Anyway, the ###21 result is closer than ###27 so I'm taking pleasure
> in (*extremely*) small things.
>
> David
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>

David,
Since you have multi-precision available with sage, could you check
the precision of the normal cdf in the lower tail?

In http://projects.scipy.org/scipy/ticket/962 there was a discussion
about using the truncated normal in the tails. In the last comment
there, I mentioned that it would be possible to use the lower tail if
the precision of scipy.special in this case is high enough. But I
wasn't able to verify how good the normal cdf really is.

These numbers are in the range 1e-20 to 1e-80, so here precision really matters.

Thanks,

Josef


From dwf at cs.toronto.edu  Wed Sep 16 20:02:38 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 16 Sep 2009 20:02:38 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <1cd32cbb0909161540k3300a928y59c174b2f2c5a1e2@mail.gmail.com>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
	<1253136320.1990.184.camel@idol>
	<62641D59-1D6B-4B85-A30F-DBEF517F7622@cs.toronto.edu>
	<1cd32cbb0909161540k3300a928y59c174b2f2c5a1e2@mail.gmail.com>
Message-ID: <59940B61-CEB0-47EB-81B1-2E8CC053F399@cs.toronto.edu>


On 16-Sep-09, at 6:40 PM, josef.pktd at gmail.com wrote:

> David,
> Since you have multi-precision available with sage, could you check
> the precision of the normal cdf in the lower tail?
>
> In http://projects.scipy.org/scipy/ticket/962 there was a discussion
> about using the truncated normal in the tails. In the last comment
> there, I mentioned that it would be possible to use the lower tail if
> the precision of scipy.special in this case is high enough. But I
> wasn't able to verify how good the normal cdf really is.
>
> These numbers are in the range 1e-20 to 1e-80, so here precision  
> really matters.

Sadly, Sage works out the integral involving the erf() function, which  
is only implemented for double-precision.

I did however do it with numerical_integral which provides an error  
bound (not a very good one, it turns out):

sage: numerical_integral(1/sqrt(2*pi) * exp(-x^2/2), -infinity, -10.2)
(9.9136251237219932e-25, 6.0437083422386946e-28)
In [3]: norm.cdf(-10.2)
Out[3]: 9.9136251225599674e-25

sage: numerical_integral(1/sqrt(2*pi) * exp(-x^2/2), -infinity, -20)
(2.7536241192643532e-89, 1.2525306363698311e-92)
In [4]: norm.cdf(-20)
Out[4]: 2.7536241186061549e-89

sage: numerical_integral(1/sqrt(2*pi) * exp(-x^2/2), -infinity, -10)
(7.6198530279086935e-24, 5.1663456001768262e-27)
In [5]: norm.cdf(-10)
Out[5]: 7.619853024160474e-24

We seem to be getting about ten digits of agreement, but that could be  
an artifact.

sage: numerical_integral(1/sqrt(2*pi) * exp(-x^2/2), -infinity, -30)
(4.906713926779639e-198, 3.680071161187036e-199)
In [6]: norm.cdf(-30)
Out[6]: 4.906713927147908e-198

(gotta love that bound)

Maybe GSL has an arbitrary precision erf taht we could compare against?

David


From dwf at cs.toronto.edu  Wed Sep 16 20:14:59 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Wed, 16 Sep 2009 20:14:59 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <1cd32cbb0909161540k3300a928y59c174b2f2c5a1e2@mail.gmail.com>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
	<1253136320.1990.184.camel@idol>
	<62641D59-1D6B-4B85-A30F-DBEF517F7622@cs.toronto.edu>
	<1cd32cbb0909161540k3300a928y59c174b2f2c5a1e2@mail.gmail.com>
Message-ID: <565375EE-9352-4BF7-ADC3-55894157B704@cs.toronto.edu>


On 16-Sep-09, at 6:40 PM, josef.pktd at gmail.com wrote:

> David,
> Since you have multi-precision available with sage, could you check
> the precision of the normal cdf in the lower tail?
>
> In http://projects.scipy.org/scipy/ticket/962 there was a discussion
> about using the truncated normal in the tails. In the last comment
> there, I mentioned that it would be possible to use the lower tail if
> the precision of scipy.special in this case is high enough. But I
> wasn't able to verify how good the normal cdf really is.
>
> These numbers are in the range 1e-20 to 1e-80, so here precision  
> really matters.
>
Apparently the 'mpmath' package (easy_install'able) does arbitrary  
precision, and has a built-in normal cdf:

Out[34]:  
mpf 
('7.6198530241605260659733432515993083635040332779569605780353524e-24')

In [35]: mpmath.functions.ncdf(-10.2)
Out[35]:  
mpf 
('9.9136251225600715794942192071914050771727823887948971717481189e-25')

In [36]: mpmath.functions.ncdf(-20)
Out[36]:  
mpf 
('2.7536241186062336950756227808574653328074977347593305676993728e-89')

In [37]: mpmath.functions.npdf(-20.2) / mpmath.functions.ncdf(-20)
Out[37]:  
mpf('0.35995251388497550054560007939076980260015643935329865252047705')


Looks pretty good to me.

David


From josef.pktd at gmail.com  Wed Sep 16 20:46:47 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 16 Sep 2009 20:46:47 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <565375EE-9352-4BF7-ADC3-55894157B704@cs.toronto.edu>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
	<1253136320.1990.184.camel@idol>
	<62641D59-1D6B-4B85-A30F-DBEF517F7622@cs.toronto.edu>
	<1cd32cbb0909161540k3300a928y59c174b2f2c5a1e2@mail.gmail.com>
	<565375EE-9352-4BF7-ADC3-55894157B704@cs.toronto.edu>
Message-ID: <1cd32cbb0909161746h562eb3e5la13871e87de415c6@mail.gmail.com>

On Wed, Sep 16, 2009 at 8:14 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>
> On 16-Sep-09, at 6:40 PM, josef.pktd at gmail.com wrote:
>
>> David,
>> Since you have multi-precision available with sage, could you check
>> the precision of the normal cdf in the lower tail?
>>
>> In http://projects.scipy.org/scipy/ticket/962 there was a discussion
>> about using the truncated normal in the tails. In the last comment
>> there, I mentioned that it would be possible to use the lower tail if
>> the precision of scipy.special in this case is high enough. But I
>> wasn't able to verify how good the normal cdf really is.
>>
>> These numbers are in the range 1e-20 to 1e-80, so here precision
>> really matters.
>>
> Apparently the 'mpmath' package (easy_install'able) does arbitrary
> precision, and has a built-in normal cdf:
>
> Out[34]:
> mpf
> ('7.6198530241605260659733432515993083635040332779569605780353524e-24')
>
> In [35]: mpmath.functions.ncdf(-10.2)
> Out[35]:
> mpf
> ('9.9136251225600715794942192071914050771727823887948971717481189e-25')
>
> In [36]: mpmath.functions.ncdf(-20)
> Out[36]:
> mpf
> ('2.7536241186062336950756227808574653328074977347593305676993728e-89')
>
> In [37]: mpmath.functions.npdf(-20.2) / mpmath.functions.ncdf(-20)
> Out[37]:
> mpf('0.35995251388497550054560007939076980260015643935329865252047705')
>
>
> Looks pretty good to me.
>
> David

Thanks, I have mpmath installed for a while but haven't done much with
it. I thought initially it might be a good benchmark for testing
scipy.special.

The normal cdf at the lower tail in scipy.special looks really useful.
It would be possible to build a truncated normal tail distribution
with it. The conditional density looks precise enough. And it's
possible to use symmetry to build the upper tail.

more fun with distributions ?

Josef

>>> mpmath.functions.npdf(-20.2) / mpmath.functions.ncdf(-20)
mpf('0.3599525138849755005456000793907698026001564393532986525204771176091088952825768140613952325481008790977')
>>> from scipy import stats
>>> stats.norm.pdf(-20.2) / stats.norm.cdf(-20)
0.35995251388498711
>>> stats.norm.pdf(-30.2) / stats.norm.cdf(-20)
1.3002766730385739e-110
>>> mpmath.functions.npdf(-30.2) / mpmath.functions.ncdf(-20)
mpf('1.300276673038541314317767156123948362259642213373797768631623741891659293976807121550129087917318179814e-110')
>>> stats.norm.pdf(-21.2) / stats.norm.cdf(-20)
3.684252913218099e-010
>>> mpmath.functions.npdf(-21.2) / mpmath.functions.ncdf(-20)
mpf('0.0000000003684252913218045966433287833404555465054012445296205154822975165088200949391675713389170126995681295188')
>>> stats.norm.pdf(-20.02) / stats.norm.cdf(-20)
13.437063718396516
>>> mpmath.functions.npdf(-20.02) / mpmath.functions.ncdf(-20)
mpf('13.43706371839603326063511929219320799476797708512756936002002696323573503280240684590166610062394498302')


From josef.pktd at gmail.com  Wed Sep 16 20:59:54 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 16 Sep 2009 20:59:54 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <1cd32cbb0909161746h562eb3e5la13871e87de415c6@mail.gmail.com>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
	<1253136320.1990.184.camel@idol>
	<62641D59-1D6B-4B85-A30F-DBEF517F7622@cs.toronto.edu>
	<1cd32cbb0909161540k3300a928y59c174b2f2c5a1e2@mail.gmail.com>
	<565375EE-9352-4BF7-ADC3-55894157B704@cs.toronto.edu>
	<1cd32cbb0909161746h562eb3e5la13871e87de415c6@mail.gmail.com>
Message-ID: <1cd32cbb0909161759p4f6ebad6x96b48e3e6c9f5257@mail.gmail.com>

On Wed, Sep 16, 2009 at 8:46 PM,  <josef.pktd at gmail.com> wrote:
> On Wed, Sep 16, 2009 at 8:14 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>>
>> On 16-Sep-09, at 6:40 PM, josef.pktd at gmail.com wrote:
>>
>>> David,
>>> Since you have multi-precision available with sage, could you check
>>> the precision of the normal cdf in the lower tail?
>>>
>>> In http://projects.scipy.org/scipy/ticket/962 there was a discussion
>>> about using the truncated normal in the tails. In the last comment
>>> there, I mentioned that it would be possible to use the lower tail if
>>> the precision of scipy.special in this case is high enough. But I
>>> wasn't able to verify how good the normal cdf really is.
>>>
>>> These numbers are in the range 1e-20 to 1e-80, so here precision
>>> really matters.
>>>
>> Apparently the 'mpmath' package (easy_install'able) does arbitrary
>> precision, and has a built-in normal cdf:
>>
>> Out[34]:
>> mpf
>> ('7.6198530241605260659733432515993083635040332779569605780353524e-24')
>>
>> In [35]: mpmath.functions.ncdf(-10.2)
>> Out[35]:
>> mpf
>> ('9.9136251225600715794942192071914050771727823887948971717481189e-25')
>>
>> In [36]: mpmath.functions.ncdf(-20)
>> Out[36]:
>> mpf
>> ('2.7536241186062336950756227808574653328074977347593305676993728e-89')
>>
>> In [37]: mpmath.functions.npdf(-20.2) / mpmath.functions.ncdf(-20)
>> Out[37]:
>> mpf('0.35995251388497550054560007939076980260015643935329865252047705')
>>
>>
>> Looks pretty good to me.
>>
>> David
>
> Thanks, I have mpmath installed for a while but haven't done much with
> it. I thought initially it might be a good benchmark for testing
> scipy.special.
>
> The normal cdf at the lower tail in scipy.special looks really useful.
> It would be possible to build a truncated normal tail distribution
> with it. The conditional density looks precise enough. And it's
> possible to use symmetry to build the upper tail.
>
> more fun with distributions ?
>
> Josef
>
>>>> mpmath.functions.npdf(-20.2) / mpmath.functions.ncdf(-20)
> mpf('0.3599525138849755005456000793907698026001564393532986525204771176091088952825768140613952325481008790977')
>>>> from scipy import stats
>>>> stats.norm.pdf(-20.2) / stats.norm.cdf(-20)
> 0.35995251388498711
>>>> stats.norm.pdf(-30.2) / stats.norm.cdf(-20)
> 1.3002766730385739e-110
>>>> mpmath.functions.npdf(-30.2) / mpmath.functions.ncdf(-20)
> mpf('1.300276673038541314317767156123948362259642213373797768631623741891659293976807121550129087917318179814e-110')
>>>> stats.norm.pdf(-21.2) / stats.norm.cdf(-20)
> 3.684252913218099e-010
>>>> mpmath.functions.npdf(-21.2) / mpmath.functions.ncdf(-20)
> mpf('0.0000000003684252913218045966433287833404555465054012445296205154822975165088200949391675713389170126995681295188')
>>>> stats.norm.pdf(-20.02) / stats.norm.cdf(-20)
> 13.437063718396516
>>>> mpmath.functions.npdf(-20.02) / mpmath.functions.ncdf(-20)
> mpf('13.43706371839603326063511929219320799476797708512756936002002696323573503280240684590166610062394498302')
>

I checked the t distribution again. It is still missing the same trick for _isf.
David, could you check and, if correct, commit the following addition
to the t distribution.

2850	    def _ppf(self, q, df):
2851	        return special.stdtrit(df, q)

+	    def _isf(self, q, df):
+	        return -special.stdtrit(df, q)

Thanks,

Josef


From jsseabold at gmail.com  Wed Sep 16 21:02:41 2009
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 16 Sep 2009 21:02:41 -0400
Subject: [SciPy-dev] scipy.special.hyp2f1 bug?
Message-ID: <c048da1c0909161802k2b4f4a23ref52d016b44222d4@mail.gmail.com>

Since y'all are looking at the distributions vs. mpmath, it reminded
me of this.  I haven't filed a bug report yet, but I don't think one
existed when I found this.

I was working with the connection between the incomplete beta function
and the hypergeometric2f1 function in scipy.special for GLM in
statsmodels and I came across this blog post on mpmath about some
corner cases for the hypergeometric2f1.

<http://fredrik-j.blogspot.com/2009/06/hypergeometric-2f1-incomplete-beta.html>

Two examples of discrepancies.

In [1]: import mpmath

In [2]: from scipy import special

In [3]: mpmath.hyp2f1(3,-1,-1,0.5)
Out[3]: mpf('2.5')

In [4]: special.hyp2f1(3,-1,-1,.5)
Out[4]: 8.0

In [5]: mpmath.hyp2f1(1,1,4,3+4j)
Out[5]: mpc(real='0.49234384000963544', imag='0.60513406166123973')

In [6]: special.hyp2f1(1,1,4,3+4j)
Out[6]: (0.65925460474847919+1.3277887264473263j)

I will file a bug report shortly, if we think mpmath is correct vs. scipy.

-Skipper


From josef.pktd at gmail.com  Wed Sep 16 21:15:56 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 16 Sep 2009 21:15:56 -0400
Subject: [SciPy-dev] scipy.special.hyp2f1 bug?
In-Reply-To: <c048da1c0909161802k2b4f4a23ref52d016b44222d4@mail.gmail.com>
References: <c048da1c0909161802k2b4f4a23ref52d016b44222d4@mail.gmail.com>
Message-ID: <1cd32cbb0909161815x788fa53fw8e97f8d83a93544d@mail.gmail.com>

On Wed, Sep 16, 2009 at 9:02 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> Since y'all are looking at the distributions vs. mpmath, it reminded
> me of this. ?I haven't filed a bug report yet, but I don't think one
> existed when I found this.
>
> I was working with the connection between the incomplete beta function
> and the hypergeometric2f1 function in scipy.special for GLM in
> statsmodels and I came across this blog post on mpmath about some
> corner cases for the hypergeometric2f1.
>
> <http://fredrik-j.blogspot.com/2009/06/hypergeometric-2f1-incomplete-beta.html>
>
> Two examples of discrepancies.
>
> In [1]: import mpmath
>
> In [2]: from scipy import special
>
> In [3]: mpmath.hyp2f1(3,-1,-1,0.5)
> Out[3]: mpf('2.5')
>
> In [4]: special.hyp2f1(3,-1,-1,.5)
> Out[4]: 8.0
>
> In [5]: mpmath.hyp2f1(1,1,4,3+4j)
> Out[5]: mpc(real='0.49234384000963544', imag='0.60513406166123973')
>
> In [6]: special.hyp2f1(1,1,4,3+4j)
> Out[6]: (0.65925460474847919+1.3277887264473263j)
>
> I will file a bug report shortly, if we think mpmath is correct vs. scipy.

there is a ticket for special.hyp2f1  for values that are used in rdist:

http://projects.scipy.org/scipy/ticket/759

but the parameter values in your case look different.

Josef

>
> -Skipper
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From jsseabold at gmail.com  Wed Sep 16 21:22:20 2009
From: jsseabold at gmail.com (Skipper Seabold)
Date: Wed, 16 Sep 2009 21:22:20 -0400
Subject: [SciPy-dev] scipy.special.hyp2f1 bug?
In-Reply-To: <1cd32cbb0909161815x788fa53fw8e97f8d83a93544d@mail.gmail.com>
References: <c048da1c0909161802k2b4f4a23ref52d016b44222d4@mail.gmail.com> 
	<1cd32cbb0909161815x788fa53fw8e97f8d83a93544d@mail.gmail.com>
Message-ID: <c048da1c0909161822w8b62a40kfd80dd9087b1f514@mail.gmail.com>

On Wed, Sep 16, 2009 at 9:15 PM,  <josef.pktd at gmail.com> wrote:
> On Wed, Sep 16, 2009 at 9:02 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> Since y'all are looking at the distributions vs. mpmath, it reminded
>> me of this. ?I haven't filed a bug report yet, but I don't think one
>> existed when I found this.
>>
>> I was working with the connection between the incomplete beta function
>> and the hypergeometric2f1 function in scipy.special for GLM in
>> statsmodels and I came across this blog post on mpmath about some
>> corner cases for the hypergeometric2f1.
>>
>> <http://fredrik-j.blogspot.com/2009/06/hypergeometric-2f1-incomplete-beta.html>
>>
>> Two examples of discrepancies.
>>
>> In [1]: import mpmath
>>
>> In [2]: from scipy import special
>>
>> In [3]: mpmath.hyp2f1(3,-1,-1,0.5)
>> Out[3]: mpf('2.5')
>>
>> In [4]: special.hyp2f1(3,-1,-1,.5)
>> Out[4]: 8.0
>>
>> In [5]: mpmath.hyp2f1(1,1,4,3+4j)
>> Out[5]: mpc(real='0.49234384000963544', imag='0.60513406166123973')
>>
>> In [6]: special.hyp2f1(1,1,4,3+4j)
>> Out[6]: (0.65925460474847919+1.3277887264473263j)
>>
>> I will file a bug report shortly, if we think mpmath is correct vs. scipy.
>
> there is a ticket for special.hyp2f1 ?for values that are used in rdist:
>
> http://projects.scipy.org/scipy/ticket/759
>
> but the parameter values in your case look different.
>
> Josef
>

Ah, ok.  I don't think I saw that one.  I will just piggyback there then.

Skipper


From pav+sp at iki.fi  Thu Sep 17 07:02:10 2009
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Thu, 17 Sep 2009 11:02:10 +0000 (UTC)
Subject: [SciPy-dev] scipy.special.hyp2f1 bug?
References: <c048da1c0909161802k2b4f4a23ref52d016b44222d4@mail.gmail.com>
Message-ID: <h8t4vi$kp6$1@ger.gmane.org>

Wed, 16 Sep 2009 21:02:41 -0400, Skipper Seabold wrote:
> Since y'all are looking at the distributions vs. mpmath, it reminded me
> of this.  I haven't filed a bug report yet, but I don't think one
> existed when I found this.
> 
> I was working with the connection between the incomplete beta function
> and the hypergeometric2f1 function in scipy.special for GLM in
> statsmodels and I came across this blog post on mpmath about some corner
> cases for the hypergeometric2f1.
> 
> <http://fredrik-j.blogspot.com/2009/06/hypergeometric-2f1-incomplete-
beta.html>
> 
> Two examples of discrepancies.
[clip]

Yep, it seems that not only is the Cephes implementation of hyp2f1 buggy, 
but also the specfun.f implementation is. Oh joy.

If you want to help by writing patches adding test cases, please do! 
Possible ways to do testing are

1) Generating points and known-good values on-the-fly:

   a) mpmath (to be run only when mpmath is available, otherwise skipped)
   b) known-good expansions

2) Testing against tabulated known-good values:

   a) added directly to the unit tests
   b) tabulated to a file

Tests in 1), 2a) go in scipy/special/tests/test_basic.py.
For 2b), you can just make a text file that contains known-good values of
(a, b, c, z, hyp2f1(a,b,c,z)), one set per row.

The main task in writing the tests is in choosing representative values 
of (a,b,c,z). The hypergeometric functions have all sorts of 
cancellations and singularities, and the test points should be chosen 
with some knowledge of these -- this is the main work. Some tests can 
also be probably stolen from mpmath's test cases.

-- 
Pauli Virtanen



From fredrik.johansson at gmail.com  Thu Sep 17 14:54:14 2009
From: fredrik.johansson at gmail.com (Fredrik Johansson)
Date: Thu, 17 Sep 2009 20:54:14 +0200
Subject: [SciPy-dev] scipy.special.hyp2f1 bug?
In-Reply-To: <c048da1c0909161802k2b4f4a23ref52d016b44222d4@mail.gmail.com>
References: <c048da1c0909161802k2b4f4a23ref52d016b44222d4@mail.gmail.com>
Message-ID: <3d0cebfb0909171154t1fa1eb31u1a80a745247ddbb7@mail.gmail.com>

On Thu, Sep 17, 2009 at 3:02 AM, Skipper Seabold <jsseabold at gmail.com> wrote:
> Since y'all are looking at the distributions vs. mpmath, it reminded
> me of this. ?I haven't filed a bug report yet, but I don't think one
> existed when I found this.
>
> I was working with the connection between the incomplete beta function
> and the hypergeometric2f1 function in scipy.special for GLM in
> statsmodels and I came across this blog post on mpmath about some
> corner cases for the hypergeometric2f1.
>
> <http://fredrik-j.blogspot.com/2009/06/hypergeometric-2f1-incomplete-beta.html>
>
> Two examples of discrepancies.
>
> In [1]: import mpmath
>
> In [2]: from scipy import special
>
> In [3]: mpmath.hyp2f1(3,-1,-1,0.5)
> Out[3]: mpf('2.5')
>
> In [4]: special.hyp2f1(3,-1,-1,.5)
> Out[4]: 8.0

Hi, mpmath author here. Both these values are correct, depending on definition.

The answer returned by mpmath agrees with Mathematica (intentionally),
but I'm not entirely happy about this as Mathematica's definition here
is *weird*. Cephes' interpretation (if that is what special.hyp2f1
uses) is that the zero and singularity in each term cancel each other
out. Mathematica's (and mpmath's) interpretation is that the
hypergeometric series terminates on the first zero numerator --
despite the same term having a zero denominator. Mathematica's
conventions are not even self-consistent: Hypergeometric2F1[3,x,x,1/2]
= 8 (with a symbolic variable x) but it changes its mind as soon as
you plug in -1 instead.

With regard to the incomplete beta integral, it seems that the
interpretations give different endpoint values, but the result is the
same upon subtraction:

>>> from mpmath import hyp2f1, quad
>>> a=-2.0; b=2.0; z1=0.25; z2=0.75
>>> T2 = z2**a/a*hyp2f1(a,1-b,a+1,z2)
>>> T1 = z1**a/a*hyp2f1(a,1-b,a+1,z1)
>>> print T1, T2
-4.0 0.444444444444444
>>> print T2-T1
4.44444444444444
>>> print quad(lambda u: u**(a-1)*(1-u)**(b-1), [z1,z2])
4.44444444444444
>>>
>>> h = 1e-15    # emulate cancellation instead
>>> a=-2.0+h; b=2.0+h; z1=0.25; z2=0.75
>>> T2 = z2**a/a*hyp2f1(a,1-b,a+1,z2)
>>> T1 = z1**a/a*hyp2f1(a,1-b,a+1,z1)
>>> print T1, T2
-3.59999999999999 0.844444444444447   # different endpoint values
>>> print T2-T1
4.44444444444444   # same difference

Which convention makes more sense? I'd be happy to change mpmath
(though I need to check how many derived results are changed). It
would probably be easy to support both conventions in mpmath by adding
a flag.

Fredrik


From fredrik.johansson at gmail.com  Thu Sep 17 15:19:24 2009
From: fredrik.johansson at gmail.com (Fredrik Johansson)
Date: Thu, 17 Sep 2009 21:19:24 +0200
Subject: [SciPy-dev] scipy.special.hyp2f1 bug?
In-Reply-To: <3d0cebfb0909171154t1fa1eb31u1a80a745247ddbb7@mail.gmail.com>
References: <c048da1c0909161802k2b4f4a23ref52d016b44222d4@mail.gmail.com>
	<3d0cebfb0909171154t1fa1eb31u1a80a745247ddbb7@mail.gmail.com>
Message-ID: <3d0cebfb0909171219y17cd5b9p51990ab4a8f99ffd@mail.gmail.com>

On Thu, Sep 17, 2009 at 8:54 PM, Fredrik Johansson
<fredrik.johansson at gmail.com> wrote:
> Which convention makes more sense? I'd be happy to change mpmath
> (though I need to check how many derived results are changed). It
> would probably be easy to support both conventions in mpmath by adding
> a flag.

I checked, and the only tests in mpmath that break if this change is
made are explicit tests for the value of hyp2f1(a,-n,-n,z). I'm
tempted to simply change mpmath's behavior and add a documentation
note to the effect that Mathematica is wrong :-)

Fredrik


From dwf at cs.toronto.edu  Thu Sep 17 16:16:42 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Thu, 17 Sep 2009 16:16:42 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <1cd32cbb0909161759p4f6ebad6x96b48e3e6c9f5257@mail.gmail.com>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
	<1253136320.1990.184.camel@idol>
	<62641D59-1D6B-4B85-A30F-DBEF517F7622@cs.toronto.edu>
	<1cd32cbb0909161540k3300a928y59c174b2f2c5a1e2@mail.gmail.com>
	<565375EE-9352-4BF7-ADC3-55894157B704@cs.toronto.edu>
	<1cd32cbb0909161746h562eb3e5la13871e87de415c6@mail.gmail.com>
	<1cd32cbb0909161759p4f6ebad6x96b48e3e6c9f5257@mail.gmail.com>
Message-ID: <D65AB9A2-B897-4B2D-9BA3-1775AB9645D2@cs.toronto.edu>

On 16-Sep-09, at 8:59 PM, josef.pktd at gmail.com wrote:

> I checked the t distribution again. It is still missing the same  
> trick for _isf.
> David, could you check and, if correct, commit the following addition
> to the t distribution.
>
> 2850	    def _ppf(self, q, df):
> 2851	        return special.stdtrit(df, q)
>
> +	    def _isf(self, q, df):
> +	        return -special.stdtrit(df, q)

Looks good to me; in r5928.

David


From josef.pktd at gmail.com  Thu Sep 17 21:37:29 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 17 Sep 2009 21:37:29 -0400
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <D65AB9A2-B897-4B2D-9BA3-1775AB9645D2@cs.toronto.edu>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
	<1253136320.1990.184.camel@idol>
	<62641D59-1D6B-4B85-A30F-DBEF517F7622@cs.toronto.edu>
	<1cd32cbb0909161540k3300a928y59c174b2f2c5a1e2@mail.gmail.com>
	<565375EE-9352-4BF7-ADC3-55894157B704@cs.toronto.edu>
	<1cd32cbb0909161746h562eb3e5la13871e87de415c6@mail.gmail.com>
	<1cd32cbb0909161759p4f6ebad6x96b48e3e6c9f5257@mail.gmail.com>
	<D65AB9A2-B897-4B2D-9BA3-1775AB9645D2@cs.toronto.edu>
Message-ID: <1cd32cbb0909171837v689199f0xfe13b947b7d9624@mail.gmail.com>

On Thu, Sep 17, 2009 at 4:16 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> On 16-Sep-09, at 8:59 PM, josef.pktd at gmail.com wrote:
>
>> I checked the t distribution again. It is still missing the same
>> trick for _isf.
>> David, could you check and, if correct, commit the following addition
>> to the t distribution.
>>
>> 2850 ? ? ?def _ppf(self, q, df):
>> 2851 ? ? ? ? ?return special.stdtrit(df, q)
>>
>> + ? ? ? ? def _isf(self, q, df):
>> + ? ? ? ? ? ? return -special.stdtrit(df, q)
>
> Looks good to me; in r5928.
>
> David

Thanks,
Josef

> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From josef.pktd at gmail.com  Thu Sep 17 23:14:13 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 17 Sep 2009 23:14:13 -0400
Subject: [SciPy-dev] scikits.statsmodels, new source distribution
Message-ID: <1cd32cbb0909172014w508a8f12u373d5fca8df21a77@mail.gmail.com>

This week, Skipper and I fixed some bugs, did some cleaning (renaming
for consistency) and added more examples and additional code to the
sandbox.

I added a source distribution file at

http://code.launchpad.net/statsmodels/trunk/initial-release/+download/scikits.statsmodels-0.1.0b2dev_r1913.zip

which is the same as current trunk, but maybe more convenient for
those without bazaar. This file doesn't contain the html docs since
there were no changes since b1.

We still have a few more improvements in mind before we will get to a
release of scikits.statsmodels-0.1.0b2, but for those who want to try
some regression in python with the webinar, the trunk is an
improvement compared to b1 and passes the test suite.

We welcome any bug reports and any ideas for improvements (and any
comments, contributions, criticism,...)

Josef


From humongo.shi at gmail.com  Fri Sep 18 01:02:28 2009
From: humongo.shi at gmail.com (Hugo Shi)
Date: Fri, 18 Sep 2009 00:02:28 -0500
Subject: [SciPy-dev] fortran compiler arguments for scipy build?
Message-ID: <1253250148.5428.4.camel@gigoblob>

I'm trying to build a 32 bit version of scipy on a 64 bit RHEL5 box.

My build dies on fftpack, and it looks like it's cause gfortran needs to
be called with the -m32 argument to make it compile for 32 bit systems.
I've tried using config_fc, and distutils.cfg, and setup.cfg to make it
do so, but nothing seems to work, can someone help me?


I've tried doing this inthe command line - 
python setup.py config_fc --opt=m32 build


as well as  adding the following lines to configs

[config_fc]
opt=m32
in a local setup.cfg and in distutils.cfg in python2.6/distutil

nothing seems to work.

I've built numpy, lapack, BLAS and python all with a 32 bit target.


Thanks in advanced.



From jason-sage at creativetrax.com  Fri Sep 18 12:12:30 2009
From: jason-sage at creativetrax.com (Jason Grout)
Date: Fri, 18 Sep 2009 11:12:30 -0500
Subject: [SciPy-dev] scipy.stats.chisquare insists second parameter is a
	numpy array
Message-ID: <h90bhe$pia$1@ger.gmane.org>

In the code for scipy.stats.chisquare, the first parameter is 
automatically converted to a numpy array, but the second parameter 
isn't; it is just used as if it were a numpy array.  Can the code be 
changed so that it does:

f_obs = asarray(f_obs)
k = len(f_obs)
if f_exp is None:
    f_exp = array([np.sum(f_obs,axis=0)/float(k)] * len(f_obs),float)
else:
    f_exp = asarray(f_exp,float)
chisq = np.add.reduce((f_obs-f_exp)**2 / f_exp)
return chisq, chisqprob(chisq, k-1)

This will allow the user to specify a normal python list for f_exp, I 
believe.

Thanks,

Jason

-- 
Jason Grout



From charlesr.harris at gmail.com  Sun Sep 20 00:00:19 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 19 Sep 2009 22:00:19 -0600
Subject: [SciPy-dev] scipy.stats: sf for symmetric distributions.
In-Reply-To: <1253136320.1990.184.camel@idol>
References: <C2FFF60B-81C1-449C-905F-D6D9A8138024@cs.toronto.edu>
	<1cd32cbb0909161206u6883a72bl53ceaa4551bccfa3@mail.gmail.com>
	<1cd32cbb0909161217n38b2e4e9tf5157eb156afab00@mail.gmail.com>
	<F6EF9B53-1054-4598-B259-3A048CB90874@cs.toronto.edu>
	<E17EBCAA-7E42-4B11-96EE-198841415D00@cs.toronto.edu>
	<1253136320.1990.184.camel@idol>
Message-ID: <e06186140909192100n20874163uf4eaee6da0bc3d44@mail.gmail.com>

On Wed, Sep 16, 2009 at 3:25 PM, Pauli Virtanen <pav at iki.fi> wrote:

> ke, 2009-09-16 kello 17:01 -0400, David Warde-Farley kirjoitti:
> > On 16-Sep-09, at 3:53 PM, David Warde-Farley wrote:
> >
> > > incidentally, this breaks two
> > > ttest tests because they differ in the 17th decimal place, how sure
> > > are you of those hard-coded numbers?).
> >
> > I just did the integral with Sage and requested 100 bits of precision,
> > more than we need, and got
> >
> > sage: thepdf
> > 1/4*sqrt(2)/(1/2*x^2 + 1)^(3/2)
> > sage: n(2 * integral(thepdf, x, -infinity, -1), prec=100)
> > 0.42264973081037423549085121950
>
> Btw, this integral evaluates to
>
>        1 - 1/sqrt(3)
>
>
Playing tricks gives the form 2/(3 + sqrt(3)), which is less susceptible to
round off errors.

In [6]: 2/(3 + sqrt(3))
Out[6]: 0.42264973081037427

In [8]: 2/(3 + sqrt(float96(3)))
Out[8]: 0.42264973081037423547

Hmm, looks like the precision of the sqrt might be the limiting factor.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090919/e9cbf3f7/attachment.html>

From Norbert.Nemec.list at gmx.de  Sun Sep 20 04:24:05 2009
From: Norbert.Nemec.list at gmx.de (Norbert Nemec)
Date: Sun, 20 Sep 2009 09:24:05 +0100
Subject: [SciPy-dev] PATCH: Update blitz++ to upstream CVS
Message-ID: <4AB5E6A5.2020006@gmx.de>

Hi there,

the version of blitz++ in the repository is on the status of 2005 with 
just a number of patches applied. Here is a patch that updates the 
version to the latest CVS version, containing a large pile of bugfixes 
and minor extensions.

I checked carefully: all changes that happened in the scipy version have 
been merged upstream already. Apart from the headers, this patch makes 
both versions identical.

Is there anything I might have overlooked? If not - please, could 
someone with access commit this patch to the SVN?

Greetings,
Norbert
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Updated-blitz-to-the-latest-upstream-CVS-version.patch.gz
Type: application/gzip
Size: 63996 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090920/43908fbe/attachment.bin>

From josef.pktd at gmail.com  Sun Sep 20 15:18:45 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 20 Sep 2009 15:18:45 -0400
Subject: [SciPy-dev] updating statsmodels license
Message-ID: <1cd32cbb0909201218n66d5179dy4bf0520ba4ebc1c7@mail.gmail.com>

A formality/legal question

I want to update the official license file of scikits.statsmodels. Is
the following header for the BSD license ok?
partially infered from scipy svn history

Thanks,
Josef

Copyright (C) 2006, Jonathan E. Taylor
All rights reserved.

Copyright (c) 2006-2008 Scipy Developers.
All rights reserved.

Copyright (c) 2009 Statsmodels Developers.
All rights reserved.


From chanley at stsci.edu  Mon Sep 21 11:20:24 2009
From: chanley at stsci.edu (Christopher Hanley)
Date: Mon, 21 Sep 2009 11:20:24 -0400
Subject: [SciPy-dev] scipy trunk build issue
Message-ID: <C3A08F86-E4E0-4F59-95A2-1983BF33D2E3@stsci.edu>

Hi,

When attempting to build scipy from the trunk I receive the following  
error:

building extension "scipy.integrate.dop" sources
error: mismatch of extension names: scipy/integrate/dop.pyf provides  
'_dop' but expected 'dop'

I am running Python 2.5 on my fully patched MacBook Pro OS X 10.5  
system.

Thanks,
Chris


-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338



From Norbert.Nemec.list at gmx.de  Mon Sep 21 14:05:17 2009
From: Norbert.Nemec.list at gmx.de (Norbert Nemec)
Date: Mon, 21 Sep 2009 19:05:17 +0100
Subject: [SciPy-dev] PATCH2: Update blitz++ to upstream CVS
In-Reply-To: <4AB5E6A5.2020006@gmx.de>
References: <4AB5E6A5.2020006@gmx.de>
Message-ID: <4AB7C05D.5070301@gmx.de>

Sorry, missed a set of changes in the first patch. Here is a second 
patch to be applied on top of the first one.



Norbert Nemec wrote:
> Hi there,
>
> the version of blitz++ in the repository is on the status of 2005 with 
> just a number of patches applied. Here is a patch that updates the 
> version to the latest CVS version, containing a large pile of bugfixes 
> and minor extensions.
>
> I checked carefully: all changes that happened in the scipy version 
> have been merged upstream already. Apart from the headers, this patch 
> makes both versions identical.
>
> Is there anything I might have overlooked? If not - please, could 
> someone with access commit this patch to the SVN?
>
> Greetings,
> Norbert
> ------------------------------------------------------------------------
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Some-changes-missing-in-the-blitz-update.patch.gz
Type: application/gzip
Size: 8086 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090921/50236a14/attachment.bin>

From Norbert.Nemec.list at gmx.de  Mon Sep 21 14:21:09 2009
From: Norbert.Nemec.list at gmx.de (Norbert Nemec)
Date: Mon, 21 Sep 2009 19:21:09 +0100
Subject: [SciPy-dev] PATCH3: Update blitz++ to upstream CVS
In-Reply-To: <4AB7C05D.5070301@gmx.de>
References: <4AB5E6A5.2020006@gmx.de> <4AB7C05D.5070301@gmx.de>
Message-ID: <4AB7C415.8010802@gmx.de>

Grmblgrmpf!!!

So, this should now be the last missing piece to the update. Working 
with patch/diff, kdiff3 and git on top of svn and cvs certainly is a 
complex choice of tools...


Norbert Nemec wrote:
> Sorry, missed a set of changes in the first patch. Here is a second 
> patch to be applied on top of the first one.
>
>
>
> Norbert Nemec wrote:
>> Hi there,
>>
>> the version of blitz++ in the repository is on the status of 2005 
>> with just a number of patches applied. Here is a patch that updates 
>> the version to the latest CVS version, containing a large pile of 
>> bugfixes and minor extensions.
>>
>> I checked carefully: all changes that happened in the scipy version 
>> have been merged upstream already. Apart from the headers, this patch 
>> makes both versions identical.
>>
>> Is there anything I might have overlooked? If not - please, could 
>> someone with access commit this patch to the SVN?
>>
>> Greetings,
>> Norbert
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Scipy-dev mailing list
>> Scipy-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>   
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-One-hopefully-last-fix-to-the-blitz-update.patch.gz
Type: application/gzip
Size: 948 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090921/aa978ffb/attachment.bin>

From cournape at gmail.com  Mon Sep 21 20:59:32 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 22 Sep 2009 09:59:32 +0900
Subject: [SciPy-dev] scipy trunk build issue
In-Reply-To: <C3A08F86-E4E0-4F59-95A2-1983BF33D2E3@stsci.edu>
References: <C3A08F86-E4E0-4F59-95A2-1983BF33D2E3@stsci.edu>
Message-ID: <5b8d13220909211759v602d1b6leda081159269fee8@mail.gmail.com>

Christopher,

On Tue, Sep 22, 2009 at 12:20 AM, Christopher Hanley <chanley at stsci.edu> wrote:
> Hi,
>
> When attempting to build scipy from the trunk I receive the following
> error:
>
> building extension "scipy.integrate.dop" sources
> error: mismatch of extension names: scipy/integrate/dop.pyf provides
> '_dop' but expected 'dop'

Sorry about that, I was working on numscons build on windows which
required the rename, and forgot to update the distutils build as well.

Should be fixed in r5950.

cheers,

David


From chanley at stsci.edu  Mon Sep 21 21:31:54 2009
From: chanley at stsci.edu (Christopher Hanley)
Date: Mon, 21 Sep 2009 21:31:54 -0400
Subject: [SciPy-dev] scipy trunk build issue
In-Reply-To: <5b8d13220909211759v602d1b6leda081159269fee8@mail.gmail.com>
References: <C3A08F86-E4E0-4F59-95A2-1983BF33D2E3@stsci.edu>
	<5b8d13220909211759v602d1b6leda081159269fee8@mail.gmail.com>
Message-ID: <66FE583C-FBEE-4844-AA68-89EF1D41964A@stsci.edu>

On Sep 21, 2009, at 8:59 PM, David Cournapeau wrote:

> Christopher,
>
> On Tue, Sep 22, 2009 at 12:20 AM, Christopher Hanley <chanley at stsci.edu 
> > wrote:
>> Hi,
>>
>> When attempting to build scipy from the trunk I receive the following
>> error:
>>
>> building extension "scipy.integrate.dop" sources
>> error: mismatch of extension names: scipy/integrate/dop.pyf provides
>> '_dop' but expected 'dop'
>
> Sorry about that, I was working on numscons build on windows which
> required the rename, and forgot to update the distutils build as well.
>
> Should be fixed in r5950.
>
> cheers,
>
> David



Thanks David.  I grabbed the trunk and all is now right with the world.

Cheers,
Chris






From ondrej at certik.cz  Tue Sep 22 01:09:30 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 21 Sep 2009 22:09:30 -0700
Subject: [SciPy-dev] Gauss-Lobatto points
Message-ID: <85b5c3130909212209l4abedf6ao135220a0683750e0@mail.gmail.com>

Hi,

does anyone know a way to get Gauss-Lobatto points out of scipy? I
know how to get Gauss integration points, but that's different:

In [1]: from scipy.special.orthogonal import p_roots

In [2]: p_roots(2)
Out[2]: [array([-0.57735027,  0.57735027]), array([ 1.,  1.])]

In [3]: p_roots(3)
Out[3]:
[array([ -7.74596669e-01,  -4.78946310e-17,   7.74596669e-01]),
 array([ 0.55555556,  0.88888889,  0.55555556])]

In [4]: p_roots(4)
Out[4]:
[array([-0.86113631, -0.33998104,  0.33998104,  0.86113631]),
 array([ 0.34785485,  0.65214515,  0.65214515,  0.34785485])]

In [5]: p_roots(5)
Out[5]:
[array([ -9.06179846e-01,  -5.38469310e-01,  -1.25197490e-16,
        5.38469310e-01,   9.06179846e-01]),
 array([ 0.23692689,  0.47862867,  0.56888889,  0.47862867,  0.23692689])]

the Gauss-Lobatto points are used for interpolation.

Thanks!
Ondrej


From ondrej at certik.cz  Tue Sep 22 01:25:41 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 21 Sep 2009 22:25:41 -0700
Subject: [SciPy-dev] Gauss-Lobatto points
In-Reply-To: <85b5c3130909212209l4abedf6ao135220a0683750e0@mail.gmail.com>
References: <85b5c3130909212209l4abedf6ao135220a0683750e0@mail.gmail.com>
Message-ID: <85b5c3130909212225l3e44a8aejb5fba38c1b17008d@mail.gmail.com>

On Mon, Sep 21, 2009 at 10:09 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> does anyone know a way to get Gauss-Lobatto points out of scipy? I

P.S. Gauss-Lobatto points are the roots of
(1-x*x)L'(x) where L(x) are Legendre
polynomials.

Ondrej


From sturla at molden.no  Tue Sep 22 02:21:44 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 22 Sep 2009 08:21:44 +0200
Subject: [SciPy-dev] reimplementation of lfilter
Message-ID: <4AB86CF8.10606@molden.no>

Looking at this atrocious piece of C:

http://svn.scipy.org/svn/scipy/trunk/scipy/signal/lfilter.c.src

I got some inspiration for reusing some of my other C code to
reimplement scipy.signal.lfilter. I basically just had to write
this small C function + a Cython wrapper:


static void lfilter_1d(double *restrict b, double *restrict a,
    double *x, double *y, double *zi, npy_intp K, npy_intp N)
{
    double Z[K];
    double *restrict pz = zi, *restrict z = Z;
    register npy_intp n, k;
    for (n=0; n<N; n++) {
        register double yn, xn = x[n];
        yn = y[n] = b[0] * xn + pz[0];
        for (k=0; k<K-1; k++)
            z[k] = b[k+1] * xn + pz[k+1] - a[k+1]*yn;
        z[K-1] = b[K]*xn - a[K]*yn;
        double *tmp = pz;
        pz = z;
        z = tmp;
    }
    if (pz != zi)     
        memcpy(zi, pz, K*sizeof(double));       
}


Relative speed compared to scipy.signal.lfilter:

One processor:
1D array, 1000000 floats:              150%
2D array, 1000 x 1000 floats, axis=0:   80%
2D array, 1000 x 1000 floats, axis=1:  150%

It is faster than lfilter except for 2D with axis=0. For
2D with axis=1 it is probably faster than lfilter due to
copy-in copy-out optimization. For arrays with more than
one dimension, multithreading helps as well:

Four processors:
1D array, 1000000 floats:              150%
2D array, 1000 x 1000 floats, axis=0:  160%
2D array, 1000 x 1000 floats, axis=1:  250%

Multithreading obviously does not help for filtering single
vector, as there is no work to share.

Some other improvements over signal.lfilter, apart from readable
Cython code:

* GIL is released. The C code is not littered with Python C
  API calls that prevents this.

* Filtering can be done in reverse, i.e. no need for fliplr or
  flipud for filtering backwards.

* Arrays can be filtered in-place.

* Cleaned up docstring.

Still it only work as a "proof of concept" with double precision
floats. Adding support for other dtypes would be easy, just
replace double with any of:

float
float _Complex
double _Complex
long double
long double _Complex



Regards,
Sturla Molden







-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: linear_filter.pyx
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090922/f052db95/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: _linear_filter.c
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090922/f052db95/attachment.c>

From charlesr.harris at gmail.com  Tue Sep 22 09:51:08 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 22 Sep 2009 07:51:08 -0600
Subject: [SciPy-dev] Gauss-Lobatto points
In-Reply-To: <85b5c3130909212225l3e44a8aejb5fba38c1b17008d@mail.gmail.com>
References: <85b5c3130909212209l4abedf6ao135220a0683750e0@mail.gmail.com>
	<85b5c3130909212225l3e44a8aejb5fba38c1b17008d@mail.gmail.com>
Message-ID: <e06186140909220651y481ef05l477d5feb74f13884@mail.gmail.com>

On Mon, Sep 21, 2009 at 11:25 PM, Ondrej Certik <ondrej at certik.cz> wrote:

> On Mon, Sep 21, 2009 at 10:09 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> > Hi,
> >
> > does anyone know a way to get Gauss-Lobatto points out of scipy? I
>
> P.S. Gauss-Lobatto points are the roots of
> (1-x*x)L'(x) where L(x) are Legendre
> polynomials.
>
>
If you use the (1-x*x)^.5 instead you are looking for the roots of an
associated Legendre polynomial which satisfies a recurrence that can be use
to set up a companion matrix whose eigenvalues should be the roots. The
companion matrix is multiplication by x mod
{polynomial-whose-roots-you-want}. I'm not sure what having the factor will
do, but I suspect something like that should work.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090922/ea1e5cf4/attachment.html>

From ralf.gommers at googlemail.com  Tue Sep 22 10:23:55 2009
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 22 Sep 2009 10:23:55 -0400
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4AB86CF8.10606@molden.no>
References: <4AB86CF8.10606@molden.no>
Message-ID: <dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>

Looks like a very useful improvement.

Your docstring won't render well, and there is already a cleaned up version
of the old one here:
http://docs.scipy.org/scipy/docs/scipy.signal.signaltools.lfilter/ Could you
please integrate those changes with yours?

Cheers,
Ralf



On Tue, Sep 22, 2009 at 2:21 AM, Sturla Molden <sturla at molden.no> wrote:

> Looking at this atrocious piece of C:
>
> http://svn.scipy.org/svn/scipy/trunk/scipy/signal/lfilter.c.src
>
> I got some inspiration for reusing some of my other C code to
> reimplement scipy.signal.lfilter. I basically just had to write
> this small C function + a Cython wrapper:
>
>
> static void lfilter_1d(double *restrict b, double *restrict a,
>   double *x, double *y, double *zi, npy_intp K, npy_intp N)
> {
>   double Z[K];
>   double *restrict pz = zi, *restrict z = Z;
>   register npy_intp n, k;
>   for (n=0; n<N; n++) {
>       register double yn, xn = x[n];
>       yn = y[n] = b[0] * xn + pz[0];
>       for (k=0; k<K-1; k++)
>           z[k] = b[k+1] * xn + pz[k+1] - a[k+1]*yn;
>       z[K-1] = b[K]*xn - a[K]*yn;
>       double *tmp = pz;
>       pz = z;
>       z = tmp;
>   }
>   if (pz != zi)           memcpy(zi, pz, K*sizeof(double));       }
>
>
> Relative speed compared to scipy.signal.lfilter:
>
> One processor:
> 1D array, 1000000 floats:              150%
> 2D array, 1000 x 1000 floats, axis=0:   80%
> 2D array, 1000 x 1000 floats, axis=1:  150%
>
> It is faster than lfilter except for 2D with axis=0. For
> 2D with axis=1 it is probably faster than lfilter due to
> copy-in copy-out optimization. For arrays with more than
> one dimension, multithreading helps as well:
>
> Four processors:
> 1D array, 1000000 floats:              150%
> 2D array, 1000 x 1000 floats, axis=0:  160%
> 2D array, 1000 x 1000 floats, axis=1:  250%
>
> Multithreading obviously does not help for filtering single
> vector, as there is no work to share.
>
> Some other improvements over signal.lfilter, apart from readable
> Cython code:
>
> * GIL is released. The C code is not littered with Python C
>  API calls that prevents this.
>
> * Filtering can be done in reverse, i.e. no need for fliplr or
>  flipud for filtering backwards.
>
> * Arrays can be filtered in-place.
>
> * Cleaned up docstring.
>
> Still it only work as a "proof of concept" with double precision
> floats. Adding support for other dtypes would be easy, just
> replace double with any of:
>
> float
> float _Complex
> double _Complex
> long double
> long double _Complex
>
>
>
> Regards,
> Sturla Molden
>
>
>
>
>
>
>
>
> # Copyright (C) 2009 Sturla Molden
>
>
>
> import numpy as np
> cimport numpy as np
>
>
> cdef extern int _linear_filter( object a, object b,
>                                object x, object y, object z,
>                                int axis, int direction)
>
> def lfilter(object B, object A, object X,
>            object zi=None,
>            object axis = None,
>            int direction=1,
>            object inplace=False):
>
>    """
>    Filter data along one-dimension with an IIR or FIR filter.
>
>
>    Description
>    ===========
>
>      Filter a data sequence, x, using a digital filter.  This works for
> many
>      fundamental data types (including Object type).  The filter is a
> direct
>      form II transposed implementation of the standard difference equation
>      (see "Algorithm").
>
>
>    Inputs:
>    =======
>
>      b : vector-like
>          The numerator coefficient vector in a 1-D sequence.
>
>      a : vector-like
>          The denominator coefficient vector in a 1-D sequence. If a[0]
>          is not 1, then both a and b are normalized by a[0].
>
>      x : array-like
>          An N-dimensional input array.
>
>      axis : int or None
>          The axis of the input data array along which to apply the
>          linear filter. The filter is applied to each subarray along
>          this axis. If axis is None, the filter is applied to x
>          flattened. Default: axis=None.
>
>      zi : array-like
>          Initial conditions for the filter delays.  It is a vector
>          (or array of vectors for an N-dimensional input) of length
>          max(len(a),len(b)). If zi=None or is not given then initial
>          rest is assumed. If axis is None, zi should be a vector
>          of length K = max(M,N), where M=len(b)-1 and N=len(a)-1. If
>          an axis is not None, zi should either be a vector of length
>          K, or an array with same shape as x, except for zi.shape[axis]
>          which should be K. If zi is a vector, the same initial
>          conditions are applied to each vector along the specified
>          axis. Default: zi=None (initial rest).
>
>          See signal.lfiltic for more information.
>
>      direction : int
>          If negative, the filter is applied in reverse direction
>          along the axis. Default: direction=1 (filter forward).
>
>      inplace : bool
>          If True, allow x or zi to be modified inplace if possible.
>          Default: False.
>
>
>    Outputs: (y, {zf})
>    =======
>
>      y : ndarray
>          The output of the digital filter.
>
>      zf : ndarray
>          If zi is None, this is not returned, otherwise, zf holds the
>          final filter delay values.
>
>
>    Algorithm:
>    ==========
>      The filter function is implemented as a direct II transposed
> structure.
>      This means that the filter implements
>
>      a[0]*y[n] = b[0]*x[n] + b[1]*x[n-1] + ... + b[nb]*x[n-nb]
>                            - a[1]*y[n-1] - ... - a[na]*y[n-na]
>
>      using the following difference equations:
>
>      y[m] = b[0]*x[m] + z[0,m-1]
>      z[0,m] = b[1]*x[m] + z[1,m-1] - a[1]*y[m]
>      ...
>      z[n-3,m] = b[n-2]*x[m] + z[n-2,m-1] - a[n-2]*y[m]
>      z[n-2,m] = b[n-1]*x[m] - a[n-1]*y[m]
>
>      where m is the output sample number and n=max(len(a),len(b)) is the
>      model order.
>
>      The rational transfer function describing this filter in the
>      z-transform domain is
>                                  -1               -nb
>                      b[0] + b[1]z  + ... + b[nb] z
>              Y(z) = ---------------------------------- X(z)
>                                  -1               -na
>                      a[0] + a[1]z  + ... + a[na] z
>
>
>    """
>
>
>    cdef np.ndarray b, a, x, y, z
>    cdef np.npy_intp i, ierr
>    cdef double tmp
>    cdef object K, zshape, iaxis
>
>    # get filter coeffs b, a
>
>    A = np.asarray(A)
>    B = np.asarray(B)
>
>    if (A.ndim != 1) or (B.ndim != 1):
>        raise ValueError, 'a and b must be vectors'
>
>    K = max(len(A)-1,len(B)-1)
>
>    b = np.zeros(K+1)
>    a = np.zeros(K+1)
>    a[:A.shape[0]] = A[:]
>    b[:B.shape[0]] = B[:]
>
>
>    # normalize by a[0]
>    if a[0] != 1.0:
>        tmp = a[0]
>        for i in range(a.shape[0]):
>            a[i] /= tmp
>            b[i] /= tmp
>
>
>    # set up input signal
>    X = np.asarray(X, dtype=np.float64)
>    if axis is None:
>        X = X.ravel()
>    elif type(axis) != int:
>        raise ValueError, 'axis must be None or an integer >= 0'
>    elif axis < 0:
>        raise ValueError, 'axis must be None or an integer >= 0'
>    x = X
>
>
>    # set up output signal
>    if inplace:
>        y = X
>    else:
>        y = np.zeros(X.shape, dtype=np.float64)
>
>
>    # set up filter delay buffer
>
>    iaxis = 0 if (axis is None) else axis
>
>    if zi is None:
>
>        # zi not specified, assume initial rest
>
>        zshape = list(X.shape)
>        zshape[iaxis] = K
>        z = np.zeros(zshape, dtype=np.float64)
>
>    else:
>
>        zi = np.asarray(zi, dtype=np.float64)
>
>        # x and zi are both vectors
>
>        if (x.ndim == 1) and (zi.ndim == 1):
>
>            if (zi.shape[0] != K):
>                raise ValueError, 'length of zi must be K'
>
>            if inplace:
>                z = zi
>            else:
>                z = zi.copy()
>
>        # x is nd array, zi is vector
>        # broadcast zi (cannot modify inplace)
>
>        elif (x.ndim > 1) and (zi.ndim == 1):
>
>            if (zi.shape[0] != K):
>                raise ValueError, 'length of zi must be K'
>
>            zshape = list(X.shape)
>            zshape[iaxis] = K
>            z = np.zeros(zshape, dtype=np.float64)
>
>            zshape = [1] * X.ndim
>            zshape[iaxis] = K
>            z = z + zi.reshape(zshape)
>
>
>        # x and zi are both nd arrays
>
>        else:
>
>            zshape = list(X.shape)
>            zshape[iaxis] = K
>            zshape = tuple(zshape)
>
>            if (zi.shape != zshape):
>                raise ValueError, ('bad shape of zi: expected %r, got %r' %
> (zshape,zi.shape))
>
>            if inplace:
>                z = zi
>            else:
>                z = zi.copy()
>
>
>    # apply the linear filter
>
>    ierr = _linear_filter(a, b, x, y, z, <int> iaxis, direction)
>
>
>    # check for error conditions on return
>
>    if ierr == -1:
>        raise MemoryError, 'malloc returned NULL'
>    if ierr == -2:
>        raise ValueError, 'shape of x, y, and z did not match... should
> never happen, debug please.'
>    if ierr == -3:
>        raise ValueError, 'shape of a and b did not match... should never
> happen, debug please.'
>
>
>    # return y or (y, z) depending on zi
>
>    # return (y if (zi is None) else (y, z)) ## Cython fails on this
>
>    if (zi is None):
>       return y
>    else:
>       return (y,z)
>
>
>
>
>
>
>
>
> /* Copyright (C) 2009 Sturla Molden */
>
>
>
>
> #include <Python.h>
> #include <numpy/arrayobject.h>
> #include <string.h>
> #include <stdlib.h>
>
> typedef PyArrayObject ndarray;    /* PyArrayObject is an ugly name */
> typedef Py_ssize_t    integer;    /* Py_ssize_t of int for 64 bit support,
> npy_intp is typedef'ed incorrectly. */
>
>
> /* copy data to and from temporary work arrays  */
>
> static void copy_in(double *restrict y, double *restrict x, integer n,
> integer s)
> {
>    integer i;
>    char *tmp;
>
>    if (s == sizeof(double))
>        memcpy(y, x, n*sizeof(double));
>    else {
>        tmp = (char *)x;
>        for (i=0; i<n; i++) {
>            *y++ = *x;
>            tmp += s;
>            x = (double *)tmp;
>        }
>    }
> }
>
> static void copy_out(double *restrict y, double *restrict x, integer n,
> integer s)
> {
>    integer i;
>    char *tmp;
>    if (s == sizeof(double))
>        memcpy(y, x, n*sizeof(double));
>    else {
>        tmp = (char *)y;
>        for (i=0; i<n; i++) {
>            *y = *x++;
>            tmp += s;
>            y = (double *)tmp;
>        }
>    }
> }
>
>
> /* this computes the start adress for every vector along a dimension
>  (axis) of an ndarray */
>
> inline char *datapointer(ndarray *a, integer *indices)
> {
>    char *data = a->data;
>    int i;
>    integer j;
>    for (i=0; i < a->nd; i++) {
>        j = indices[i];
>        data += j * a->strides[i];
>    }
>    return data;
> }
>
> static double ** _get_axis_pointer(integer *indices, int axis,
>                           ndarray *a, double **axis_ptr_array, int dim)
> {
>    /* recursive axis pointer search for 4 dimensions or more */
>    integer i;
>    double *axis_ptr;
>    if (dim == a->nd) {
>        /* done recursing dimensions,
>        compute address of this axis */
>        axis_ptr = (double *) datapointer(a, indices);
>        *axis_ptr_array = axis_ptr;
>        return (axis_ptr_array + 1);
>    } else if (dim == axis) {
>        /* use first element only */
>        indices[dim] = 0;
>        axis_ptr_array = _get_axis_pointer(indices, axis, a, axis_ptr_array,
> dim+1);
>        return axis_ptr_array;
>    } else {
>        /* iterate range of indices */
>        integer length = a->dimensions[dim];
>        for (i=0; i < length; i++) {
>            indices[dim] = i;
>            axis_ptr_array = _get_axis_pointer(indices, axis, a,
> axis_ptr_array, dim+1);
>        }
>        return axis_ptr_array;
>    }
> }
>
>
> static double **get_axis_pointers(ndarray *a, int axis, integer *count)
> {
>    integer indices[a->nd];
>    double **out, **tmp;
>    integer i, j;
>    j = 1;
>    for (i=0; i < a->nd; i++) {
>        if (i != axis)
>            j *= a->dimensions[i];
>    }
>    *count = j;
>
>    out = (double **) malloc(*count * sizeof(double*));
>    if (out == NULL) {
>        *count = 0;
>        return NULL;
>    }
>    tmp = out;
>    switch (a->nd) {
>        /* for one dimension we just return the data pointer */
>        case 1:
>            *tmp = (double *)a->data;
>            break;
>        /* specialized for two dimensions */
>        case 2:
>            switch (axis) {
>                case 0:
>                    for (i=0; i<a->dimensions[1]; i++)
>                        *tmp++ = (double *)(a->data + i*a->strides[1]);
>                    break;
>
>                case 1:
>                    for (i=0; i<a->dimensions[0]; i++)
>                        *tmp++ = (double *)(a->data + i*a->strides[0]);
>                    break;
>            }
>            break;
>        /* specialized for three dimensions */
>        case 3:
>            switch (axis) {
>                case 0:
>                    for (i=0; i<a->dimensions[1]; i++)
>                        for (j=0; j<a->dimensions[2]; j++)
>                            *tmp++ = (double *)(a->data + i*a->strides[1] +
> j*a->strides[2]);
>                    break;
>                case 1:
>                    for (i=0; i<a->dimensions[0]; i++)
>                        for (j=0; j<a->dimensions[2]; j++)
>                            *tmp++ = (double *)(a->data + i*a->strides[0] +
> j*a->strides[2]);
>                    break;
>
>                case 2:
>                    for (i=0; i<a->dimensions[0]; i++)
>                        for (j=0; j<a->dimensions[1]; j++)
>                            *tmp++ = (double *)(a->data + i*a->strides[0] +
> j*a->strides[1]);
>
>            }
>            break;
>        /* four or more dimensions: use recursion */
>        default:
>            for (i=0; i<a->nd; i++) indices[i] = 0;
>            _get_axis_pointer(indices, axis, a, out, 0);
>
>    }
> done:
>    return out;
> }
>
>
>
> /* applies a linear filter along one dimension */
>
> static void lfilter_1d( double *restrict b, double *restrict a,
>                        double *x, double *y,
>                        double *zi,
>                        integer K, integer N)
> {
>    double Z[K];
>    double *restrict pz = zi, *restrict z = Z;
>    register integer n, k;
>
>
>    for (n=0; n<N; n++) {
>
>        register double yn, xn = x[n];
>
>        yn = y[n] = b[0] * xn + pz[0];
>
>        for (k=0; k<K-1; k++)
>            z[k] = b[k+1] * xn + pz[k+1] - a[k+1]*yn;
>
>        z[K-1] = b[K]*xn - a[K]*yn;
>
>        double *tmp = pz;
>        pz = z;
>        z = tmp;
>    }
>
>    if (pz != zi)
>        memcpy(zi, pz, K*sizeof(double));
> }
>
>
> static void rev_lfilter_1d( double *restrict b, double *restrict a,
>                            double *restrict x, double *restrict y,
>                            double *zi,
>                            integer K, integer N)
> {
>    double Z[K];
>    double *restrict pz = zi, *restrict z = Z;
>    register integer n, k;
>
>    for (n=N-1; n >= 0; n--) {
>
>        register double yn, xn = x[n];
>
>        yn = y[n] = b[0] * xn + pz[0];
>
>        for (k=0; k<K-1; k++)
>            z[k] = b[k+1]*xn + pz[k+1] - a[k+1]*yn;
>
>        z[K-1] = b[K]*xn - a[K]*yn;
>
>        double *tmp = pz;
>        pz = z;
>        z = tmp;
>    }
>
>    if (pz != zi)
>        memcpy(zi, pz, K*sizeof(double));
> }
>
>
>
> typedef void (*lfilterfun_t)(double *restrict b, double *restrict a,
>                            double *restrict x, double *restrict y,
>                            double *zi, integer K, integer N);
>
>
> int _linear_filter(ndarray *a, ndarray *b,
>                   ndarray *x, ndarray *y, ndarray *z,
>                   int axis,
>                   int direction)
> {
>    int retval = 0;
>    double *wx, *wy, *wz, *data_a, *data_b;
>    integer xcount, ycount, zcount;
>    integer xstride, ystride, zstride;
>    double *xdata, *ydata, *zdata;
>    double **x_axis_ptrs = NULL, **y_axis_ptrs = NULL, **z_axis_ptrs = NULL;
>    lfilterfun_t lfilter;
>    integer i, K, N;
>
>    /* release the GIL */
>    Py_BEGIN_ALLOW_THREADS
>
>    /* sanity check on a and b (these should never fail) */
>    if (a->nd != 1) goto error_ab;
>    if (b->nd != 1) goto error_ab;
>    if (b->dimensions[0] != a->dimensions[0]) goto error_ab;
>    data_a = (double *)a->data;
>    data_b = (double *)b->data;
>    K = a->dimensions[0] - 1;
>
>    /* sanity ckeck on x, y and z */
>    if ((x->nd != y->nd) || (y->nd != z->nd)) goto error_xyz;
>    for (int d=0; d < a->nd; d++) {
>        if (d == axis) continue;
>        if ((x->dimensions[d] != y->dimensions[d])
>            || (y->dimensions[d] != z->dimensions[d])) goto error_xyz;
>    }
>    N = x->dimensions[axis];
>
>    /* extract axis pointers */
>    x_axis_ptrs = get_axis_pointers(x, axis, &xcount);
>    y_axis_ptrs = get_axis_pointers(y, axis, &ycount);
>    z_axis_ptrs = get_axis_pointers(z, axis, &zcount);
>
>    /* sanity check */
>    if (!x_axis_ptrs) goto error_malloc;
>    if (!y_axis_ptrs) goto error_malloc;
>    if (!z_axis_ptrs) goto error_malloc;
>    if ((xcount != ycount)||(ycount != zcount)) goto error_xyz;
>
>    /* all ok, we can start ... */
>
>    lfilter = direction < 0 ? rev_lfilter_1d : lfilter_1d;
>
>    xstride = x->strides[axis];
>    ystride = y->strides[axis];
>    zstride = z->strides[axis];
>
>
>    #pragma omp parallel \
>        firstprivate(lfilter, N, K, xcount, x_axis_ptrs, y_axis_ptrs,
> z_axis_ptrs, \
>            xstride, ystride, zstride, data_b, data_a)  \
>        private(i, xdata, ydata, zdata, wx, wy, wz) \
>        shared(retval) \
>        default(none)
>    {
>        if ((xstride==sizeof(double))
>             && (ystride==sizeof(double))
>             && (xstride==sizeof(double)))
>        {
>
>            #pragma omp for schedule(static)
>            for (i=0; i < xcount; i++) {
>                xdata = x_axis_ptrs[i];
>                ydata = y_axis_ptrs[i];
>                zdata = z_axis_ptrs[i];
>                /* strictly speaking we could be aliasing
>                   xdata and ydata here, but it should not matter. */
>                lfilter(data_b, data_a, xdata, ydata, zdata, K, N);
>            }
>
>        } else {
>
>            /* axis is strided, use copy-in copy-out */
>
>            wx = (double *)malloc((2*N+K)*sizeof(double));
>            if (!wx) {
>                #pragma omp critical
>                retval = -1; /* error_malloc */
>            }
>            #pragma omp barrier
>            #pragma omp flush(retval)
>            if (retval < 0)
>                /* malloc failed in some thread */
>                goto filtering_complete;
>            wy = wx + N;
>            wz = wy + N;
>
>            #pragma omp for schedule(static)
>            for (i=0; i < xcount; i++) {
>
>                /* get pointe to the first element of the axis */
>                xdata = x_axis_ptrs[i];
>                ydata = y_axis_ptrs[i];
>                zdata = z_axis_ptrs[i];
>
>                /* copy to local contiguous buffers */
>                copy_in(wx, xdata, N, xstride);
>                copy_in(wz, zdata, K, zstride);
>
>                /* filter */
>                lfilter(data_b, data_a, wx, wy, wz, K, N);
>
>                /* copy data back */
>                copy_out(ydata, wy, N, ystride);
>                copy_out(zdata, wz, K, zstride);
>
>            }
> filtering_complete:
>            if (wx) free(wx);
>        }
>    }
>    /* we are done... */
>    goto done;
>
> /* error conditions */
> error_ab:
>    retval = -3;
>    goto done;
> error_xyz:
>    retval = -2;
>    goto done;
> error_malloc:
>    retval = -1;
>
> /* clean up and exit */
> done:
>
>    if (x_axis_ptrs) free(x_axis_ptrs);
>    if (y_axis_ptrs) free(y_axis_ptrs);
>    if (z_axis_ptrs) free(z_axis_ptrs);
>
>    /* get the GIL back and return */
>    Py_END_ALLOW_THREADS
>
>    return retval;
> }
>
>
>
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090922/62e79c1f/attachment.html>

From ondrej at certik.cz  Tue Sep 22 13:04:26 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 22 Sep 2009 10:04:26 -0700
Subject: [SciPy-dev] Gauss-Lobatto points
In-Reply-To: <e06186140909220651y481ef05l477d5feb74f13884@mail.gmail.com>
References: <85b5c3130909212209l4abedf6ao135220a0683750e0@mail.gmail.com>
	<85b5c3130909212225l3e44a8aejb5fba38c1b17008d@mail.gmail.com>
	<e06186140909220651y481ef05l477d5feb74f13884@mail.gmail.com>
Message-ID: <85b5c3130909221004h7c0c00e7o19e34d0e31a1ed5e@mail.gmail.com>

Hi Chuck!

On Tue, Sep 22, 2009 at 6:51 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Mon, Sep 21, 2009 at 11:25 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>>
>> On Mon, Sep 21, 2009 at 10:09 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>> > Hi,
>> >
>> > does anyone know a way to get Gauss-Lobatto points out of scipy? I
>>
>> P.S. Gauss-Lobatto points are the roots of
>> (1-x*x)L'(x) where L(x) are Legendre
>> polynomials.
>>
>
> If you use the (1-x*x)^.5 instead you are looking for the roots of an
> associated Legendre polynomial which satisfies a recurrence that can be use
> to set up a companion matrix whose eigenvalues should be the roots. The
> companion matrix is multiplication by x mod
> {polynomial-whose-roots-you-want}. I'm not sure what having the factor will
> do, but I suspect something like that should work.

Thanks for the help!

Ondrej


From kgdunn at gmail.com  Tue Sep 22 14:04:37 2009
From: kgdunn at gmail.com (Kevin Dunn)
Date: Tue, 22 Sep 2009 14:04:37 -0400
Subject: [SciPy-dev] Bug/Error with chi-squared distribution and df<1
In-Reply-To: <ad94c31e0909220759l501c96beu7e75f5d1d0842297@mail.gmail.com>
References: <ad94c31e0909220759l501c96beu7e75f5d1d0842297@mail.gmail.com>
Message-ID: <ad94c31e0909221104p5d999647xee8c14447dca2351@mail.gmail.com>

Thanks for the prompt reply Josef; please see my comments below.

> On Tue, Sep 22, 2009 at 10:59 AM, Kevin Dunn <kgdunn at gmail.com> wrote:
>> Hi there,
>>
>> I'm not an expert on distributions, but as far as I can tell, the chi2
>> distribution is defined for degrees of freedom >0. ?I'm getting "nan"
>> results however when df is less than one (but still greater than 0).
>> The chi2 CDF values agree between R, MATLAB and Scipy when the the
>> degrees of freedom are >= 1. ?For example:
>> * R: pchisq(0.95, 1)
>> * MATLAB: chi2cdf(0.95, 1)
>> * SciPy: scipy.stats.chi2.cdf(0.95, 1)
>>
>
>
>> However, changing the last argument to 0.5 returns a NaN in SciPy, but
>> gives a result (0.8392259) in R and MATLAB.
>
>>>> stats.chi2.veccdf(0.95, 0.5)
> array(0.83922587961194761)
>
>>
>> I'm suspecting there is something wrong with my SciPy installation,
>> because the example included with SciPy, (found using
>> scipy.stats.chi2? inside iPython) calls the chi2.cdf function with
>> df=0.9, yet when I run that example as follows:
>
> Where is this example? maybe it is an autogenerated example with wrong numbers?

The example was found when typing "scipy.stats.chi2?" in ipython after
importing scipy.stats.  The code below was taken from the example,
only I omitted the matplotlib plotting commands.

>> from scipy.stats import chi2
>> import numpy as np
>> numargs = chi2.numargs
>> [ df ] = [0.9,]*numargs
>> rv = chi2(df)
>> x = np.linspace(0,np.minimum(rv.dist.b,3))
>> prb = chi2.cdf(x,df)
>>
>> My result for prb is as follows; which I don't think would have been
>> used as an example if this is the expected output.
>> array([ ?0., ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN,
>> ? ? ? ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN,
>> ? ? ? ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN,
>> ? ? ? ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN,
>> ? ? ? ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN])
>>
>> Is my SciPy install messed up? ?I'm using MacOSX and downloaded
>> version 0.7.1 from SourceForge this morning. ?I just tried it in
>> Ubuntu Linux (version 0.7.0 though), and get the same results.
>>
>
> chi2 uses scipy.special
>
> class chi2_gen
>
> def _cdf(self, x, df):
> ? ? ? ?return special.chdtr(df, x)
>
> which obviously cannot handle df<1 , which I don't know if it would
> ever show up in the usual statistical tests.

I don't believe it is obvious why special.chdtr can't handle values df
< 1. Please see my additional comments below.

>
> pdf doesn't seem to have any problems
>
>>>> stats.chi2.pdf(np.arange(5),0.99)
> array([ ? ? ? ?Inf, ?0.24042373, ?0.10275665, ?0.05078513, ?0.02663761])
>>>> stats.chi2.pdf(np.arange(5),0.5)
> array([ ? ? ? ?Inf, ?0.14067411, ?0.05073346, ?0.02270277, ?0.01109756])
>>>> stats.chi2.pdf(np.arange(5),0.25)
> array([ ? ? ? ?Inf, ?0.07382471, ?0.02441481, ?0.01038547, ?0.00489731])
>
> so numerical integration should also work
>
>>>> stats.chi2.veccdf(np.arange(1,5),0.25)
> array([ 0.92605422, ?0.96918041, ?0.98539847, ?0.99265551])
>>>> stats.chi2.veccdf(np.linspace(0.01,10.0,11),0.25)
> array([ 0.54726537, ?0.92671456, ?0.96937499, ?0.98547096, ?0.99268483,
> ? ? ? ?0.99618434, ?0.99796201, ?0.99889274, ?0.99939058, ?0.99966116,
> ? ? ? ?0.99981006])
>>>> stats.chi2.veccdf(np.linspace(0.01,10.0,11),0.5)
> array([ 0.29308089, ?0.84774539, ?0.93248332, ?0.96674206, ?0.98278044,
> ? ? ? ?0.99081467, ?0.99500114, ?0.99723983, ?0.9984591 , ?0.99913231,
> ? ? ? ?0.99950797])
>
> since pdf at zero is inf, I don't know how good the numbers are if the
> cdf is calculated for points close to zero
> e.g.
>
>>>> stats.chi2.veccdf(1e-8,0.5)
> array(0.0092772960765327081)
>
>
> Since I don't think df<1 is a common case, I wouldn't want to switch
> to numerical integration by default. But veccdf although only a
> private function should work although it bypasses some of the checking
> and broadcasting.

Thanks for pointing out the numerical integration approach; 'll give
it a try.  As for df < 1, please see my other comments below.

>
> Somehow this sounds familiar, I need to check whether this or a
> similar case is already on record.
>
> Hope that helps,
>
> Josef

And regarding your other reply:

> I didn't find anything on the scipy trac nor on the mailinglist, since
> I'm subscribed to them. If you think that df<1 is an important case
> you could file a ticket for scipy.special. I have no idea how
> difficult it would be to extend the chi squared related functions for
> this or whether there exists another workaround.

I just downloaded the latest SVN code and found the code that does the
work is /scipy/special/cephes/chdtr.c

In that code it seems that when the degrees of freedom are less than
one, the code is (artificially) forced to return a NaN:
        if (df < 1.0)
        {
            mtherr( "chdtrc", DOMAIN );
            return(NPY_NAN);
        }
       return( igamc( df/2.0, x/2.0 ) );

This seems a somewhat artificial constraint to me, since the gamma
function can accept values of 0 < df < 1.  Does someone else on this
list know why that constraint is there for df<1?

As for a practical usage case, I'm computing CDF values for a variable
by matching moments, and the value for the degrees of freedom term is
a computed value.  This value is always non-integer, and can sometimes
be less than one.

Additional justification for df < 1 can be sought by looking at plots
of the chi-squared distribution (e.g.
http://en.wikipedia.org/wiki/Chi-square_distribution) when plotting
with a degrees of freedom parameter (they call it "k" in the Wikipedia
article).  There's no reason why df can't be less than one, anymore
than there's reason for df being non-integer.

I'll give it a go compiling SciPy with removing those constraints in
chdtr.c and see how it works.  I've never done this before, but I'm
busy installing the NumPy/SciPy installation requirements at the
moment, and I'll report how it went.

If things works out, I'll file a trac report.  I've also cc'd the
scipy-dev list on this reply.

Thanks for your help,
Kevin

> The only similar story was ?for the negative binomial for n<1,
> http://projects.scipy.org/scipy/ticket/978 . In that case
> scipy.special.nbdtr didn't allow the extension for n<1.
>
> Josef


From josef.pktd at gmail.com  Tue Sep 22 14:27:48 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 22 Sep 2009 14:27:48 -0400
Subject: [SciPy-dev] Bug/Error with chi-squared distribution and df<1
In-Reply-To: <ad94c31e0909221104p5d999647xee8c14447dca2351@mail.gmail.com>
References: <ad94c31e0909220759l501c96beu7e75f5d1d0842297@mail.gmail.com>
	<ad94c31e0909221104p5d999647xee8c14447dca2351@mail.gmail.com>
Message-ID: <1cd32cbb0909221127s379c06ccoc375ad005c8a87fd@mail.gmail.com>

On Tue, Sep 22, 2009 at 2:04 PM, Kevin Dunn <kgdunn at gmail.com> wrote:
> Thanks for the prompt reply Josef; please see my comments below.
>
>> On Tue, Sep 22, 2009 at 10:59 AM, Kevin Dunn <kgdunn at gmail.com> wrote:
>>> Hi there,
>>>
>>> I'm not an expert on distributions, but as far as I can tell, the chi2
>>> distribution is defined for degrees of freedom >0. ?I'm getting "nan"
>>> results however when df is less than one (but still greater than 0).
>>> The chi2 CDF values agree between R, MATLAB and Scipy when the the
>>> degrees of freedom are >= 1. ?For example:
>>> * R: pchisq(0.95, 1)
>>> * MATLAB: chi2cdf(0.95, 1)
>>> * SciPy: scipy.stats.chi2.cdf(0.95, 1)
>>>
>>
>>
>>> However, changing the last argument to 0.5 returns a NaN in SciPy, but
>>> gives a result (0.8392259) in R and MATLAB.
>>
>>>>> stats.chi2.veccdf(0.95, 0.5)
>> array(0.83922587961194761)
>>
>>>
>>> I'm suspecting there is something wrong with my SciPy installation,
>>> because the example included with SciPy, (found using
>>> scipy.stats.chi2? inside iPython) calls the chi2.cdf function with
>>> df=0.9, yet when I run that example as follows:
>>
>> Where is this example? maybe it is an autogenerated example with wrong numbers?
>
> The example was found when typing "scipy.stats.chi2?" in ipython after
> importing scipy.stats. ?The code below was taken from the example,
> only I omitted the matplotlib plotting commands.
>
>>> from scipy.stats import chi2
>>> import numpy as np
>>> numargs = chi2.numargs
>>> [ df ] = [0.9,]*numargs
>>> rv = chi2(df)
>>> x = np.linspace(0,np.minimum(rv.dist.b,3))
>>> prb = chi2.cdf(x,df)

Yes, this is a generic template with autofilled names, obviously not
tested if the numbers make sense in all cases.

>>>
>>> My result for prb is as follows; which I don't think would have been
>>> used as an example if this is the expected output.
>>> array([ ?0., ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN,
>>> ? ? ? ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN,
>>> ? ? ? ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN,
>>> ? ? ? ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN,
>>> ? ? ? ?NaN, ?NaN, ?NaN, ?NaN, ?NaN, ?NaN])
>>>
>>> Is my SciPy install messed up? ?I'm using MacOSX and downloaded
>>> version 0.7.1 from SourceForge this morning. ?I just tried it in
>>> Ubuntu Linux (version 0.7.0 though), and get the same results.
>>>
>>
>> chi2 uses scipy.special
>>
>> class chi2_gen
>>
>> def _cdf(self, x, df):
>> ? ? ? ?return special.chdtr(df, x)
>>
>> which obviously cannot handle df<1 , which I don't know if it would
>> ever show up in the usual statistical tests.
>
> I don't believe it is obvious why special.chdtr can't handle values df
> < 1. Please see my additional comments below.
>
>>
>> pdf doesn't seem to have any problems
>>
>>>>> stats.chi2.pdf(np.arange(5),0.99)
>> array([ ? ? ? ?Inf, ?0.24042373, ?0.10275665, ?0.05078513, ?0.02663761])
>>>>> stats.chi2.pdf(np.arange(5),0.5)
>> array([ ? ? ? ?Inf, ?0.14067411, ?0.05073346, ?0.02270277, ?0.01109756])
>>>>> stats.chi2.pdf(np.arange(5),0.25)
>> array([ ? ? ? ?Inf, ?0.07382471, ?0.02441481, ?0.01038547, ?0.00489731])
>>
>> so numerical integration should also work
>>
>>>>> stats.chi2.veccdf(np.arange(1,5),0.25)
>> array([ 0.92605422, ?0.96918041, ?0.98539847, ?0.99265551])
>>>>> stats.chi2.veccdf(np.linspace(0.01,10.0,11),0.25)
>> array([ 0.54726537, ?0.92671456, ?0.96937499, ?0.98547096, ?0.99268483,
>> ? ? ? ?0.99618434, ?0.99796201, ?0.99889274, ?0.99939058, ?0.99966116,
>> ? ? ? ?0.99981006])
>>>>> stats.chi2.veccdf(np.linspace(0.01,10.0,11),0.5)
>> array([ 0.29308089, ?0.84774539, ?0.93248332, ?0.96674206, ?0.98278044,
>> ? ? ? ?0.99081467, ?0.99500114, ?0.99723983, ?0.9984591 , ?0.99913231,
>> ? ? ? ?0.99950797])
>>
>> since pdf at zero is inf, I don't know how good the numbers are if the
>> cdf is calculated for points close to zero
>> e.g.
>>
>>>>> stats.chi2.veccdf(1e-8,0.5)
>> array(0.0092772960765327081)
>>
>>
>> Since I don't think df<1 is a common case, I wouldn't want to switch
>> to numerical integration by default. But veccdf although only a
>> private function should work although it bypasses some of the checking
>> and broadcasting.
>
> Thanks for pointing out the numerical integration approach; 'll give
> it a try. ?As for df < 1, please see my other comments below.
>
>>
>> Somehow this sounds familiar, I need to check whether this or a
>> similar case is already on record.
>>
>> Hope that helps,
>>
>> Josef
>
> And regarding your other reply:
>
>> I didn't find anything on the scipy trac nor on the mailinglist, since
>> I'm subscribed to them. If you think that df<1 is an important case
>> you could file a ticket for scipy.special. I have no idea how
>> difficult it would be to extend the chi squared related functions for
>> this or whether there exists another workaround.
>
> I just downloaded the latest SVN code and found the code that does the
> work is /scipy/special/cephes/chdtr.c
>
> In that code it seems that when the degrees of freedom are less than
> one, the code is (artificially) forced to return a NaN:
> ? ? ? ?if (df < 1.0)
> ? ? ? ?{
> ? ? ? ? ? ?mtherr( "chdtrc", DOMAIN );
> ? ? ? ? ? ?return(NPY_NAN);
> ? ? ? ?}
> ? ? ? return( igamc( df/2.0, x/2.0 ) );
>
> This seems a somewhat artificial constraint to me, since the gamma
> function can accept values of 0 < df < 1. ?Does someone else on this
> list know why that constraint is there for df<1?


I don't know the c code, but many of the statistical functions, have
duplicate ways of calculation with scipy special.

Taking the hint with incomplete gamma, the following looks good. This
would mean until Pauli fixes scipy.special if it your fix works, we
could also use gammainc directly. I don't know the differences between
the various implementations well enough to see whether we buy some
other problems with this

>>> df=2;x=1.5;special.gammainc(df/2., x/2.)
0.52763344725898531
>>> df=0.5;x=1.5;special.gammainc(df/2., x/2.)
0.89993651328449831

>>> stats.chi2.cdf(x,df)
nan
>>> stats.chi2.cdf(1.5,2)
0.52763344725898531
>>> stats.chi2.veccdf(1.5,2)
array(0.52763344725898531)
>>> stats.chi2.veccdf(1.5,0.5)
array(0.89993651328445579)
>>> stats.chi2.veccdf(1.5,0.5) - special.gammainc(0.5/2., 1.5/2.)
-4.2521541843143495e-014

I'm used to the chi square distribution as a statistical test
distribution, but of course if it is treated just as a distribution
that is matched to data, then these limitations (and nans) are not
very useful.

Thanks for looking into this.

Josef

>
> As for a practical usage case, I'm computing CDF values for a variable
> by matching moments, and the value for the degrees of freedom term is
> a computed value. ?This value is always non-integer, and can sometimes
> be less than one.
>
> Additional justification for df < 1 can be sought by looking at plots
> of the chi-squared distribution (e.g.
> http://en.wikipedia.org/wiki/Chi-square_distribution) when plotting
> with a degrees of freedom parameter (they call it "k" in the Wikipedia
> article). ?There's no reason why df can't be less than one, anymore
> than there's reason for df being non-integer.
>
> I'll give it a go compiling SciPy with removing those constraints in
> chdtr.c and see how it works. ?I've never done this before, but I'm
> busy installing the NumPy/SciPy installation requirements at the
> moment, and I'll report how it went.
>
> If things works out, I'll file a trac report. ?I've also cc'd the
> scipy-dev list on this reply.
>
> Thanks for your help,
> Kevin
>
>> The only similar story was ?for the negative binomial for n<1,
>> http://projects.scipy.org/scipy/ticket/978 . In that case
>> scipy.special.nbdtr didn't allow the extension for n<1.
>>
>> Josef
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From kgdunn at gmail.com  Tue Sep 22 15:07:18 2009
From: kgdunn at gmail.com (Kevin Dunn)
Date: Tue, 22 Sep 2009 15:07:18 -0400
Subject: [SciPy-dev] Building SciPy: breaks when looking for get_info
	function
Message-ID: <ad94c31e0909221207k69686872w11b109b40dbfb6f3@mail.gmail.com>

Hi there,

This is my first attempt at building SciPy; I'm doing this to solve a
problem described in another posting here today with subject
"Bug/Error with chi-squared distribution and df<1"

Background: I'm using Ubuntu 9.04, I've built and installed Numpy
successfully from SVN this afternoon.   Other details about my
platform, as requested by INSTALL.TXT, are available at
http://pastebin.com/m26f8d1d1

Now I'm trying to build and install SciPy from SVN as well, but I get
the following traceback regarding the `get_info` function when I call:
python setup.py build

When inspecting the ./scipy/special/setup.py file I notice it is
trying to import get_info, however, `get_info` is from
/numpy/distutils/misc_util.py where it exists only as a method of the
Configuration class, not as a stand-alone function.  I'm not sure how
to proceed from here.  Any advice?

Thanks,
Kevin
----
Traceback (most recent call last):
  File "setup.py", line 160, in <module>
    setup_package()
  File "setup.py", line 152, in setup_package
    configuration=configuration )
  File "/usr/lib/python2.6/dist-packages/numpy/distutils/core.py",
line 122, in setup
    return setup(**attr)
  File "/usr/lib/python2.6/dist-packages/numpy/distutils/core.py",
line 150, in setup
    config = configuration()
  File "setup.py", line 118, in configuration
    config.add_subpackage('scipy')
  File "/usr/lib/python2.6/dist-packages/numpy/distutils/misc_util.py",
line 851, in add_subpackage
    caller_level = 2)
  File "/usr/lib/python2.6/dist-packages/numpy/distutils/misc_util.py",
line 834, in get_subpackage
    caller_level = caller_level + 1)
  File "/usr/lib/python2.6/dist-packages/numpy/distutils/misc_util.py",
line 781, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "scipy/setup.py", line 20, in configuration
    config.add_subpackage('special')
  File "/usr/lib/python2.6/dist-packages/numpy/distutils/misc_util.py",
line 851, in add_subpackage
    caller_level = 2)
  File "/usr/lib/python2.6/dist-packages/numpy/distutils/misc_util.py",
line 834, in get_subpackage
    caller_level = caller_level + 1)
  File "/usr/lib/python2.6/dist-packages/numpy/distutils/misc_util.py",
line 766, in _get_configuration_from_setup_py
    ('.py', 'U', 1))
  File "scipy/special/setup.py", line 7, in <module>
    from numpy.distutils.misc_util import get_numpy_include_dirs, get_info
ImportError: cannot import name get_info


From sturla at molden.no  Tue Sep 22 15:19:27 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 22 Sep 2009 21:19:27 +0200
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>
References: <4AB86CF8.10606@molden.no>
	<dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>
Message-ID: <4AB9233F.4040704@molden.no>


Ralf Gommers skrev:
 > Looks like a very useful improvement.
 >
 > Your docstring won't render well, and there is already a cleaned up 
version of the old one here: 
http://docs.scipy.org/scipy/docs/scipy.signal.signaltools.lfilter/ Could 
you please integrate those changes with yours?

The code from SVN I linked is current version used in signaltools.

Also lost of the lfilter C code deal with complex numbers, which
is something an ISO compliant C or C++ compiler can do for us.

#ifdef __cplusplus
  #include <complex>
  typedef std::complex<float>       complex_float; 
  typedef std::complex<double>      complex_double; 
  typedef std::complex<long double> complex_long_double;
  #define restrict
#else
  #include <complex.h>
  typedef float       _Complex      complex_float;
  typedef double      _Complex      complex_double;
  typedef long double _Complex      complex_long_double;
#endif


Perhaps a better option is to use C++ only... In any case, it is
better to leave the complex number arithmetics to the compiler,
instead of bloating the source.

Sturla Molden


From pav at iki.fi  Tue Sep 22 15:47:30 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 22 Sep 2009 22:47:30 +0300
Subject: [SciPy-dev] Bug/Error with chi-squared distribution and df<1
In-Reply-To: <1cd32cbb0909221127s379c06ccoc375ad005c8a87fd@mail.gmail.com>
References: <ad94c31e0909220759l501c96beu7e75f5d1d0842297@mail.gmail.com>
	<ad94c31e0909221104p5d999647xee8c14447dca2351@mail.gmail.com>
	<1cd32cbb0909221127s379c06ccoc375ad005c8a87fd@mail.gmail.com>
Message-ID: <1253648849.3968.2.camel@idol>

ti, 2009-09-22 kello 14:27 -0400, josef.pktd at gmail.com kirjoitti:
[clip]
> Taking the hint with incomplete gamma, the following looks good. This
> would mean until Pauli fixes scipy.special if it your fix works, we
> could also use gammainc directly. I don't know the differences between
> the various implementations well enough to see whether we buy some
> other problems with this

The C function `igam` called by `chdtr` is the same as what is exposed
as `gammainc`.

-- 
Pauli Virtanen





From kgdunn at gmail.com  Tue Sep 22 17:48:57 2009
From: kgdunn at gmail.com (Kevin Dunn)
Date: Tue, 22 Sep 2009 17:48:57 -0400
Subject: [SciPy-dev] Bug/Error with chi-squared distribution and df<1
Message-ID: <ad94c31e0909221448k524ce0e0kf697b12cfc13fa49@mail.gmail.com>

> ti, 2009-09-22 kello 14:27 -0400, josef.pktd at gmail.com kirjoitti:
> [clip]
>> Taking the hint with incomplete gamma, the following looks good. This
>> would mean until Pauli fixes scipy.special if it your fix works, we
>> could also use gammainc directly. I don't know the differences between
>> the various implementations well enough to see whether we buy some
>> other problems with this
>
> The C function `igam` called by `chdtr` is the same as what is exposed
> as `gammainc`.

Thanks again to all the helpful replies on this list.  I'm using this
relationship between the chi-squared distribution and the gamma
function to evaluate the CDF and inverse CDF.

As an example for future readers of this list, let our value be x = 3
with 0.6 degrees of freedom,

Chi-square CDF
----------------
* In MATLAB: chi2cdf(3, 0.6)   % 0.957890536704110
* In R: pchisq(3, 0.6)  # 0.9578905
* And temporarily in SciPy:  scipy.special.gammainc(0.6/2.0, 3/2.0)
# note the order of parameters is flipped; result =
0.95789053670411062
* Or another temporary possibility: scipy.stats.gamma.cdf(3/2.0,
0.6/2.0) # this time the parameters are in the expected order

But ideally in SciPy: scipy.stats.chi2.cdf(3, 0.6)  # currently
returns a NaN, but should return the same result above

Inverse Chi-square CDF
----------------
* In MATLAB: chi2inv(0.95789, 0.6)  % 2.999980885343446
* In R: qchisq(0.95789, 0.6)    #  2.999981
* And temporarily in SciPy: scipy.special.gammaincinv(0.6/2.0,
0.95789) * 2.0  # again we reorder the parameters; result =
2.9999808853434109
* Or another temporary possibility: scipy.stats.gamma.ppf(0.95789,
0.6/2.0)*2 # 2.9999808853434109

But ideally in SciPy: scipy.stats.chi2.ppf(0.95789, 0.6) # currently
returns a NaN, but should return the same result above

Hope that helps in the mean time,
Kevin



> --
> Pauli Virtanen


From ralf.gommers at googlemail.com  Tue Sep 22 18:02:48 2009
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 22 Sep 2009 18:02:48 -0400
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4AB9233F.4040704@molden.no>
References: <4AB86CF8.10606@molden.no>
	<dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>
	<4AB9233F.4040704@molden.no>
Message-ID: <dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>

On Tue, Sep 22, 2009 at 3:19 PM, Sturla Molden <sturla at molden.no> wrote:

>
> Ralf Gommers skrev:
>  > Looks like a very useful improvement.
>  >
>  > Your docstring won't render well, and there is already a cleaned up
> version of the old one here:
> http://docs.scipy.org/scipy/docs/scipy.signal.signaltools.lfilter/ Could
> you please integrate those changes with yours?
>
> The code from SVN I linked is current version used in signaltools.
>
> Sure, I know. The docs I linked have improvements over svn. If you merge
your changes, there'll be a conflict and those docs will not get merged soon
- I was hoping to avoid that.

I also wanted to draw your attention to the doc wiki, since in this case you
redid work that was already done by someone else on the wiki before.
Hopefully this will save you some effort next time.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090922/db3d7515/attachment.html>

From sturla at molden.no  Tue Sep 22 20:25:13 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 23 Sep 2009 02:25:13 +0200
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>
References: <4AB86CF8.10606@molden.no>	<dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>	<4AB9233F.4040704@molden.no>
	<dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>
Message-ID: <4AB96AE9.1070103@molden.no>

Ralf Gommers skrev:
> Sure, I know. The docs I linked have improvements over svn. If you 
> merge your changes, there'll be a conflict and those docs will not get 
> merged soon - I was hoping to avoid that.
>
> I also wanted to draw your attention to the doc wiki, since in this 
> case you redid work that was already done by someone else on the wiki 
> before. Hopefully this will save you some effort next time.

Well, I changed all the code to C++, to use the std::complex type, 
std::vector instead of malloc, and templates for specializing the filter 
functions to all dtypes supported by lfilter. I'm quite happy with the 
way the C++ looks :-)

It seems to be faster than signal.lfilter in most circumstances, except 
one (annoyingly the most common). The most extreme case were complex64 
dtype with inplace filtering, for which I got 369% speed improvement.

Here are some timings (speed improvement over signal.lfilter in percent):

<type 'numpy.float32'>
axis=0, shape=(1000000,),   speed: 240
axis=0, shape=(1000, 1000), speed: 149
axis=1, shape=(1000, 1000), speed: 238

<type 'numpy.float64'>
axis=0, shape=(1000000,),   speed: 151
axis=0, shape=(1000, 1000), speed:  93
axis=1, shape=(1000, 1000), speed: 146

<type 'numpy.complex64'>
axis=0, shape=(1000000,),   speed: 297
axis=0, shape=(1000, 1000), speed: 204
axis=1, shape=(1000, 1000), speed: 292

<type 'numpy.complex128'>
axis=0, shape=(1000000,),   speed: 209
axis=0, shape=(1000, 1000), speed: 137
axis=1, shape=(1000, 1000), speed: 193


Inplace filter:

<type 'numpy.float32'>
axis=0, shape=(1000000,), speed:   291
axis=0, shape=(1000, 1000), speed: 182
axis=1, shape=(1000, 1000), speed: 300

<type 'numpy.float64'>
axis=0, shape=(1000000,), speed:   227
axis=0, shape=(1000, 1000), speed: 137
axis=1, shape=(1000, 1000), speed: 228

<type 'numpy.complex64'>
axis=0, shape=(1000000,), speed:   369
axis=0, shape=(1000, 1000), speed: 257
axis=1, shape=(1000, 1000), speed: 345

<type 'numpy.complex128'>
axis=0, shape=(1000000,), speed:   288
axis=0, shape=(1000, 1000), speed: 179
axis=1, shape=(1000, 1000), speed: 276


To build on Windows I did this, using GCC 4.4.1 from 
http://www.equation.com/servlet/equation.cmd?fa=fortran

g++ -c -O3 -ffast-math -msse3 -march=core2 -Ic:/Python26/include 
-Ic:/Python26/Lib/site-packages/numpy/core/include _linear_filter.cpp
cython.py linear_filter.pyx
gcc -c -O2 -ffast-math -msse3 -march=core2 -Ic:/Python26/include 
-Ic:/Python26/Lib/site-packages/numpy/core/include linear_filter.c
g++ -shared -o linear_filter.pyd -Lc:/Python26/libs _linear_filter.o 
linear_filter.o -lpython26 -lmsvcr90



Best regards,

Sturla Molden













-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: _linear_filter.cpp
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090923/2bdb4076/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: linear_filter.pyx
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090923/2bdb4076/attachment-0001.ksh>

From sturla at molden.no  Tue Sep 22 20:30:00 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 23 Sep 2009 02:30:00 +0200
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4AB96AE9.1070103@molden.no>
References: <4AB86CF8.10606@molden.no>	<dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>	<4AB9233F.4040704@molden.no>	<dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>
	<4AB96AE9.1070103@molden.no>
Message-ID: <4AB96C08.1030002@molden.no>

Sturla Molden skrev:
> It seems to be faster than signal.lfilter in most circumstances, 
> except one (annoyingly the most common). The most extreme case were 
> complex64 dtype with inplace filtering, for which I got 369% speed 
> improvement.
>
> Here are some timings (speed improvement over signal.lfilter in percent):

Sorry, my english were bad. The numbers are not speed "improvement", 
they are speed relative to signal.lfilter.

S.M.




From josef.pktd at gmail.com  Tue Sep 22 20:39:55 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 22 Sep 2009 20:39:55 -0400
Subject: [SciPy-dev] Bug/Error with chi-squared distribution and df<1
In-Reply-To: <ad94c31e0909221448k524ce0e0kf697b12cfc13fa49@mail.gmail.com>
References: <ad94c31e0909221448k524ce0e0kf697b12cfc13fa49@mail.gmail.com>
Message-ID: <1cd32cbb0909221739u4ba1b8b5lf920a1fefc4bc4d@mail.gmail.com>

On Tue, Sep 22, 2009 at 5:48 PM, Kevin Dunn <kgdunn at gmail.com> wrote:
>> ti, 2009-09-22 kello 14:27 -0400, josef.pktd at gmail.com kirjoitti:
>> [clip]
>>> Taking the hint with incomplete gamma, the following looks good. This
>>> would mean until Pauli fixes scipy.special if it your fix works, we
>>> could also use gammainc directly. I don't know the differences between
>>> the various implementations well enough to see whether we buy some
>>> other problems with this
>>
>> The C function `igam` called by `chdtr` is the same as what is exposed
>> as `gammainc`.
>
> Thanks again to all the helpful replies on this list. ?I'm using this
> relationship between the chi-squared distribution and the gamma
> function to evaluate the CDF and inverse CDF.
>
> As an example for future readers of this list, let our value be x = 3
> with 0.6 degrees of freedom,
>
> Chi-square CDF
> ----------------
> * In MATLAB: chi2cdf(3, 0.6) ? % 0.957890536704110
> * In R: pchisq(3, 0.6) ?# 0.9578905
> * And temporarily in SciPy: ?scipy.special.gammainc(0.6/2.0, 3/2.0)
> # note the order of parameters is flipped; result =
> 0.95789053670411062
> * Or another temporary possibility: scipy.stats.gamma.cdf(3/2.0,
> 0.6/2.0) # this time the parameters are in the expected order
>
> But ideally in SciPy: scipy.stats.chi2.cdf(3, 0.6) ?# currently
> returns a NaN, but should return the same result above
>
> Inverse Chi-square CDF
> ----------------
> * In MATLAB: chi2inv(0.95789, 0.6) ?% 2.999980885343446
> * In R: qchisq(0.95789, 0.6) ? ?# ?2.999981
> * And temporarily in SciPy: scipy.special.gammaincinv(0.6/2.0,
> 0.95789) * 2.0 ?# again we reorder the parameters; result =
> 2.9999808853434109
> * Or another temporary possibility: scipy.stats.gamma.ppf(0.95789,
> 0.6/2.0)*2 # 2.9999808853434109
>
> But ideally in SciPy: scipy.stats.chi2.ppf(0.95789, 0.6) # currently
> returns a NaN, but should return the same result above
>
> Hope that helps in the mean time,
> Kevin
>
>
>
>> --
>> Pauli Virtanen
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>

We got ticket 1000 : http://projects.scipy.org/scipy/ticket/1000

I added the information, especially for future reference about the
relationship between the chi squared distribution and gammainc and the
gamma distribution.

Josef


From sturla at molden.no  Tue Sep 22 21:01:53 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 23 Sep 2009 03:01:53 +0200
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>
References: <4AB86CF8.10606@molden.no>	<dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>	<4AB9233F.4040704@molden.no>
	<dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>
Message-ID: <4AB97381.20300@molden.no>

Ralf Gommers skrev:
> Sure, I know. The docs I linked have improvements over svn. If you 
> merge your changes, there'll be a conflict and those docs will not get 
> merged soon - I was hoping to avoid that.

Oh, the docs, not the code... Would something like this be better?

The Cython module doesn't need to have docs in it.



from linear_filter import lfilter as _lfilter

def lfilter(b, a, x, axis=-1, zi=None, direction=1, inplace=-1):
    """Filter data along one-dimension with an IIR or FIR filter.

  Description

    Filter a data sequence, x, using a digital filter.  This works for many
    fundamental data types (including Object type).  The filter is a direct
    form II transposed implementation of the standard difference equation
    (see "Algorithm").

  Inputs:

    b -- The numerator coefficient vector in a 1-D sequence.
    a -- The denominator coefficient vector in a 1-D sequence. If a[0]
         is not 1, then both a and b are normalized by a[0].
    x -- An N-dimensional input array.
    axis -- The axis of the input data array along which to apply the
            linear filter. The filter is applied to each subarray along
            this axis. If axis is -1, the filter is applied to x
            flattened. (*Default* = -1)
    zi -- Initial conditions for the filter delays.  It is a vector
          (or array of vectors for an N-dimensional input) of length
          max(len(a),len(b)). If zi=None or is not given then initial
          rest is assumed. If axis is None, zi should be a vector
          of length K = max(M,N), where M=len(b)-1 and N=len(a)-1. If
          an axis is not None, zi should either be a vector of length
          K, or an array with same shape as x, except for zi.shape[axis]
          which should be K. If zi is a vector, the same initial
          conditions are applied to each vector along the specified
          axis. Default: zi=None (initial rest).
    direction -- Filtered in the reverse direction if -1. (*Default* = 1)
    inplace -- If possible, modification to x is done implace. 
(*Default* = -1)

  Outputs: (y, {zf})

    y -- The output of the digital filter.
    zf -- If zi is None, this is not returned, otherwise, zf holds the
          final filter delay values.

  Algorithm:

    The filter function is implemented as a direct II transposed structure.
    This means that the filter implements

    a[0]*y[n] = b[0]*x[n] + b[1]*x[n-1] + ... + b[nb]*x[n-nb]
                          - a[1]*y[n-1] - ... - a[na]*y[n-na]

    using the following difference equations:

    y[m] = b[0]*x[m] + z[0,m-1]
    z[0,m] = b[1]*x[m] + z[1,m-1] - a[1]*y[m]
    ...
    z[n-3,m] = b[n-2]*x[m] + z[n-2,m-1] - a[n-2]*y[m]
    z[n-2,m] = b[n-1]*x[m] - a[n-1]*y[m]

    where m is the output sample number and n=max(len(a),len(b)) is the
    model order.

    The rational transfer function describing this filter in the
    z-transform domain is
                                -1               -nb
                    b[0] + b[1]z  + ... + b[nb] z
            Y(z) = ---------------------------------- X(z)
                                -1               -na
                    a[0] + a[1]z  + ... + a[na] z

    """
    if isscalar(a):
        a = [a]

    if axis < 0: axis = None
    if inplace < 0: inplace = None
   
    return _lfilter(b, a, x, axis=axis, zi=zi,
             inplace=inplace, direction=direction)



Regards,
S.M.


 



From ralf.gommers at googlemail.com  Tue Sep 22 21:40:48 2009
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Tue, 22 Sep 2009 21:40:48 -0400
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4AB97381.20300@molden.no>
References: <4AB86CF8.10606@molden.no>
	<dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>
	<4AB9233F.4040704@molden.no>
	<dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>
	<4AB97381.20300@molden.no>
Message-ID: <dde7764a0909221840u178f9712rf1c4834699e1a34d@mail.gmail.com>

On Tue, Sep 22, 2009 at 9:01 PM, Sturla Molden <sturla at molden.no> wrote:

> Ralf Gommers skrev:
> > Sure, I know. The docs I linked have improvements over svn. If you
> > merge your changes, there'll be a conflict and those docs will not get
> > merged soon - I was hoping to avoid that.
>
> Oh, the docs, not the code... Would something like this be better?
>
>
See attached for a new version which incorporates your changes, the latest
wiki version, as well as some more edits. It fixes indentation, type
descriptions and rest markup.

 I have one (very minor) comment on the options you added, "direction" and
"inplace". I think these should both be bools, because they select between
two options.

The C++ code looks clean, I do not know enough about that language to say
much more about it.

Cheers,
Ralf




> The Cython module doesn't need to have docs in it.
>
>
>
> from linear_filter import lfilter as _lfilter
>
> def lfilter(b, a, x, axis=-1, zi=None, direction=1, inplace=-1):
>     """Filter data along one-dimension with an IIR or FIR filter.
>
>  Description
>
>    Filter a data sequence, x, using a digital filter.  This works for many
>    fundamental data types (including Object type).  The filter is a direct
>    form II transposed implementation of the standard difference equation
>    (see "Algorithm").
>
>  Inputs:
>
>     b -- The numerator coefficient vector in a 1-D sequence.
>    a -- The denominator coefficient vector in a 1-D sequence. If a[0]
>          is not 1, then both a and b are normalized by a[0].
>     x -- An N-dimensional input array.
>    axis -- The axis of the input data array along which to apply the
>             linear filter. The filter is applied to each subarray along
>             this axis. If axis is -1, the filter is applied to x
>            flattened. (*Default* = -1)
>    zi -- Initial conditions for the filter delays.  It is a vector
>           (or array of vectors for an N-dimensional input) of length
>          max(len(a),len(b)). If zi=None or is not given then initial
>          rest is assumed. If axis is None, zi should be a vector
>          of length K = max(M,N), where M=len(b)-1 and N=len(a)-1. If
>          an axis is not None, zi should either be a vector of length
>          K, or an array with same shape as x, except for zi.shape[axis]
>          which should be K. If zi is a vector, the same initial
>          conditions are applied to each vector along the specified
>          axis. Default: zi=None (initial rest).
>     direction -- Filtered in the reverse direction if -1. (*Default* = 1)
>    inplace -- If possible, modification to x is done implace.
> (*Default* = -1)
>
>  Outputs: (y, {zf})
>
>    y -- The output of the digital filter.
>    zf -- If zi is None, this is not returned, otherwise, zf holds the
>           final filter delay values.
>
>  Algorithm:
>
>    The filter function is implemented as a direct II transposed structure.
>    This means that the filter implements
>
>    a[0]*y[n] = b[0]*x[n] + b[1]*x[n-1] + ... + b[nb]*x[n-nb]
>                          - a[1]*y[n-1] - ... - a[na]*y[n-na]
>
>    using the following difference equations:
>
>    y[m] = b[0]*x[m] + z[0,m-1]
>    z[0,m] = b[1]*x[m] + z[1,m-1] - a[1]*y[m]
>    ...
>    z[n-3,m] = b[n-2]*x[m] + z[n-2,m-1] - a[n-2]*y[m]
>    z[n-2,m] = b[n-1]*x[m] - a[n-1]*y[m]
>
>    where m is the output sample number and n=max(len(a),len(b)) is the
>    model order.
>
>    The rational transfer function describing this filter in the
>    z-transform domain is
>                                -1               -nb
>                    b[0] + b[1]z  + ... + b[nb] z
>            Y(z) = ---------------------------------- X(z)
>                                -1               -na
>                    a[0] + a[1]z  + ... + a[na] z
>
>    """
>     if isscalar(a):
>        a = [a]
>
>    if axis < 0: axis = None
>    if inplace < 0: inplace = None
>
>    return _lfilter(b, a, x, axis=axis, zi=zi,
>             inplace=inplace, direction=direction)
>
>
>
> Regards,
> S.M.
>
>
>
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090922/04b6e1f5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lfilterdoc_new.py
Type: application/octet-stream
Size: 2670 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090922/04b6e1f5/attachment.obj>

From sturla at molden.no  Wed Sep 23 02:10:39 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 23 Sep 2009 08:10:39 +0200
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4AB96AE9.1070103@molden.no>
References: <4AB86CF8.10606@molden.no>	<dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>	<4AB9233F.4040704@molden.no>	<dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>
	<4AB96AE9.1070103@molden.no>
Message-ID: <4AB9BBDF.4020606@molden.no>

Sturla Molden skrev:
> Well, I changed all the code to C++, to use the std::complex type, 
> std::vector instead of malloc, and templates for specializing the 
> filter functions to all dtypes supported by lfilter. I'm quite happy 
> with the way the C++ looks :-) 

I forgot to add support for the object dtype.

This also fixes a bug in the current signal.lfilter. This will crash the 
interpreter:

   import numpy as np
   import scipy
   from scipy.signal import butter, lfilter
   b,a = butter(4, .25)
   x = np.random.randn(1000000).astype(object).reshape((1000,1000))
   fx = lfilter(b,a,x, axis=1)

The C++ version here does not crash on this. :-)

I just ripped some code from current lfilter and cleaned it up slightly; 
I hope that is ok. (Cython was not happy about "cdef object *ptr", so I 
reluctantly had to put it all in C++.)  The reason my code does not 
crash like current lfilter, is copy-in copy-out for strided vectors. It 
seems lfilter messes up the strides and then segfaults.

<type 'object'>
axis=0, shape=(1000000,), speed: 102
axis=0, shape=(1000, 1000), speed: 105
axis=1, shape=(1000, 1000), scipy.signal.lfilter crashes the interpreter!!!

Building on Windows:

g++ -c -O3 -ffast-math -msse3 -march=core2 -Ic:/Python26/include 
-Ic:/Python26/Lib/site-packages/numpy/core/include _linear_filter.cpp
cython.py --cplus linear_filter.pyx
g++ -c -O2 -ffast-math -msse3 -march=core2 -Ic:/Python26/include 
-Ic:/Python26/Lib/site-packages/numpy/core/include linear_filter.cpp
g++ -shared -o linear_filter.pyd -Lc:/Python26/libs _linear_filter.o 
linear_filter.o -lpython26 -lmsvcr90


Regards,
Sturla Molden


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: _linear_filter.cpp
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090923/1811e1ff/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: linear_filter.pyx
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090923/1811e1ff/attachment-0001.ksh>

From cournape at gmail.com  Wed Sep 23 04:31:40 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 23 Sep 2009 17:31:40 +0900
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4AB9BBDF.4020606@molden.no>
References: <4AB86CF8.10606@molden.no>
	<dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>
	<4AB9233F.4040704@molden.no>
	<dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>
	<4AB96AE9.1070103@molden.no> <4AB9BBDF.4020606@molden.no>
Message-ID: <5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>

On Wed, Sep 23, 2009 at 3:10 PM, Sturla Molden <sturla at molden.no> wrote:
> Sturla Molden skrev:
>>
>> Well, I changed all the code to C++, to use the std::complex type,
>> std::vector instead of malloc, and templates for specializing the filter
>> functions to all dtypes supported by lfilter. I'm quite happy with the way
>> the C++ looks :-)
>
> I forgot to add support for the object dtype.
>
> This also fixes a bug in the current signal.lfilter. This will crash the
> interpreter:
>
> ?import numpy as np
> ?import scipy
> ?from scipy.signal import butter, lfilter
> ?b,a = butter(4, .25)
> ?x = np.random.randn(1000000).astype(object).reshape((1000,1000))
> ?fx = lfilter(b,a,x, axis=1)
>
> The C++ version here does not crash on this. :-)
>
> I just ripped some code from current lfilter and cleaned it up slightly; I
> hope that is ok. (Cython was not happy about "cdef object *ptr", so I
> reluctantly had to put it all in C++.) ?The reason my code does not crash
> like current lfilter, is copy-in copy-out for strided vectors. It seems
> lfilter messes up the strides and then segfaults.

I am all for improving the lfilter code, here a few comments:
 - it is much more likely that your improvements will be included if
you provide patches instead of rewrite of the full code - it makes
reviewing much easier.
 - I would also prefer having C instead of C++ as well - in this case,
C++ does not bring much since we have our "templating" system and you
don't use the STL much.
 - In any case, please do not use exception, it is not portable.
 - you cannot use Py_ssize_t, as it is python 2.5 >= feature - there
is nothing wrong with npy_intp, I don't understand your comment.
 - using cython is good - that could be a first patch, to replace the
existing manual wrapping by cython. You can declare pointer without
trouble in cython, you would have to be more explicit about the exact
problem you had.

cheers,

David


From sturla at molden.no  Wed Sep 23 07:24:41 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 23 Sep 2009 13:24:41 +0200
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>
References: <4AB86CF8.10606@molden.no>	<dde7764a0909220723t2039498cg87b608e98b9d3ce6@mail.gmail.com>	<4AB9233F.4040704@molden.no>	<dde7764a0909221502ia40e3b4x1b3aadadc9307b0d@mail.gmail.com>	<4AB96AE9.1070103@molden.no>
	<4AB9BBDF.4020606@molden.no>
	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>
Message-ID: <4ABA0579.6010405@molden.no>

David Cournapeau skrev:

>  - I would also prefer having C instead of C++ as well - in this case,
> C++ does not bring much since we have our "templating" system and you
> don't use the STL much.
>   
It was mainly for complex numbers, since MSVC does not support ISO C.


>  - In any case, please do not use exception, it is not portable.
>   
Ok, the STL containers like vector<> can throw exceptions like 
std::bad_alloc. That is why I did this.


>  - you cannot use Py_ssize_t, as it is python 2.5 >= feature - there
> is nothing wrong with npy_intp, I don't understand your comment.
>   
Yes there is, cf. PEP 353.

Using Py_intptr_t for indexing would depend on sizeof(void*) == 
sizeof(size_t), which the C standard does not mandate. It can differ on 
segment and offset architectures. Two examples are 16-bit x86 (cf. far 
and near pointers) and x86 with 36-bit PAE. It accidentally works for 
flat 32- and 64-bit address spaces. This is why Python uses Py_ssize_t 
instead of Py_intptr_t. And as it happens, npy_intp is typedef'ed to the 
latter. (This might be pedantic, but it is a formal error.)

>  - using cython is good - that could be a first patch, to replace the
> existing manual wrapping by cython. You can declare pointer without
> trouble in cython, 
No, you cannot create pointers to a variable declared object. This is 
illegal Cython:

cdef object *ptr   # would simliar to PyObject **ptr in C

So if we want to filter with dtype object, one could use Dag Sverre's 
numpy syntax and fake "cdef object *ptr" with "cdef np.ndarray[object] 
ptr", but it would not be efficient. We have to store two z-buffers and 
swap them for each filter step. The other option is to use "cdef 
PyObject**" instead of "cdef object *" in Cython, but then Cython will 
not do reference counting.


S.M.



From thomas.robitaille at gmail.com  Wed Sep 23 08:03:39 2009
From: thomas.robitaille at gmail.com (Thomas Robitaille)
Date: Wed, 23 Sep 2009 08:03:39 -0400
Subject: [SciPy-dev] scipy.weave crashes wth KeyError
Message-ID: <94e44680909230503q4c81e4f4odaccf55a9e8bb4e3@mail.gmail.com>

Hi,

I'm using a recent svn revision of scipy (5925). After installing it I
went to scipy/weave/examples and ran 'python array3d.py'. I get the
following error message (below). Can other people reproduce this
problem? If not, maybe it's some local installation issue. This
problem occurs with any code that uses weave.inline. I'm using python
2.6 64-bit, but I've also seen reports of the problem on python 2.6
32-bit.

I've tracked the problem down to the following file being empty

.python26_compiled/darwin26compiled_catalog.dat

However, it does look like there are some compiled shared object
libraries in there

air:.python26_compiled tom$ du -sk *
0	darwin26compiled_catalog.dat
192	m0
344	m1
1908	m2
580	m3

So it's not clear if compilation is the problem?

Thanks,

Thomas

---

numpy:
[[[ 0  1  2  3]
  [ 4  5  6  7]
  [ 8  9 10 11]]

 [[12 13 14 15]
  [16 17 18 19]
  [20 21 22 23]]]
Pure Inline:
Traceback (most recent call last):
  File "array3d.py", line 105, in <module>
    main()
  File "array3d.py", line 98, in main
    pure_inline(arr)
  File "array3d.py", line 57, in pure_inline
    weave.inline(code, ['arr'])
  File
"/Users/tom/Library/Python/2.6/site-packages/scipy/weave/inline_tools.py",
line 324, in inline
    results = attempt_function_call(code,local_dict,global_dict)
  File
"/Users/tom/Library/Python/2.6/site-packages/scipy/weave/inline_tools.py",
line 392, in attempt_function_call
    function_list = function_catalog.get_functions(code,module_dir)
  File "/Users/tom/Library/Python/2.6/site-packages/scipy/weave/catalog.py",
line 615, in get_functions
    function_list = self.get_cataloged_functions(code)
  File "/Users/tom/Library/Python/2.6/site-packages/scipy/weave/catalog.py",
line 529, in get_cataloged_functions
    if cat is not None and code in cat:
  File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/shelve.py",
line 110, in __contains__
    return key in self.dict
  File
"/Users/tom/Library/Python/2.6/site-packages/scipy/io/dumbdbm_patched.py",
line 73, in __getitem__
    pos, siz = self._index[key] # may raise KeyError
KeyError: 0


From lists_ravi at lavabit.com  Wed Sep 23 08:29:58 2009
From: lists_ravi at lavabit.com (Ravi)
Date: Wed, 23 Sep 2009 08:29:58 -0400
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>
References: <4AB86CF8.10606@molden.no> <4AB9BBDF.4020606@molden.no>
	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>
Message-ID: <200909230829.58471.lists_ravi@lavabit.com>

Hi,

On Wednesday 23 September 2009 04:31:40 David Cournapeau wrote:
>  - it is much more likely that your improvements will be included if
> you provide patches instead of rewrite of the full code - it makes
> reviewing much easier.

In this case, I respectfully disagree; the full rewrite actually makes sense 
when comparing the previous code to the current one.

>  - I would also prefer having C instead of C++ as well - in this case,
> C++ does not bring much since we have our "templating" system and you
> don't use the STL much.
>  - In any case, please do not use exception, it is not portable.

Are there any such compilers on which scipy can be compiled? The only compiler 
on which exceptions are still a problem (as far as I know) is OpenWatcom, but 
scipy cannot be compiled using that compiler anyway. For all major *nixes 
(Linux, Solaris, HPUX, AIX(!), *BSD), both the system C++ compiler and the 
major external compilers (usually gcc & icc) provide full exception support. 
On Windows, only versions upto MSVC6 have problems (but they cannot be used to 
compile python extensions). Even the proprietary ARM and MIPS compilers 
support exceptions. On the embedded side, the only compiler that I can think 
of that does not support real C++ is Keil, but is anyone running scipy on an 
8051?

Regards,
Ravi



From cournape at gmail.com  Wed Sep 23 09:58:40 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 23 Sep 2009 22:58:40 +0900
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <200909230829.58471.lists_ravi@lavabit.com>
References: <4AB86CF8.10606@molden.no> <4AB9BBDF.4020606@molden.no>
	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>
	<200909230829.58471.lists_ravi@lavabit.com>
Message-ID: <5b8d13220909230658s42845a6q5a5178e404dd9128@mail.gmail.com>

On Wed, Sep 23, 2009 at 9:29 PM, Ravi <lists_ravi at lavabit.com> wrote:
> Hi,
>
> On Wednesday 23 September 2009 04:31:40 David Cournapeau wrote:
>> ?- it is much more likely that your improvements will be included if
>> you provide patches instead of rewrite of the full code - it makes
>> reviewing much easier.
>
> In this case, I respectfully disagree; the full rewrite actually makes sense
> when comparing the previous code to the current one.

It can be a full rewrite, but still should be sent as patches. If I am
the one to review, I would prefer this way. That's especially
important to track regressions.

>
>> ?- I would also prefer having C instead of C++ as well - in this case,
>> C++ does not bring much since we have our "templating" system and you
>> don't use the STL much.
>> ?- In any case, please do not use exception, it is not portable.
>
> Are there any such compilers on which scipy can be compiled?

It is a fundamental problem of C++. Different compilers do not
propagate exceptions the same way, and that's a problem when different
compilers are involved (happens easily when the C and C++ compilers
are not the same, for example). That has been a problem on every new
platform I have tried to port numpy and scipy to.

That's the same rationale as why avoiding fortran runtime calls - if
you only use fortran, it is ok, but once you use the fortran runtime
and the C runtime in the same extension, you get random crashes which
are impossible to debug.

cheers,

David


From krunk7 at gmail.com  Wed Sep 23 10:02:11 2009
From: krunk7 at gmail.com (James Kyle)
Date: Wed, 23 Sep 2009 07:02:11 -0700
Subject: [SciPy-dev] compiling against custom atlas/lapack on os x
Message-ID: <675CB282-54ED-43B8-90D9-CD361093D214@gmail.com>

I posted this to scipy-users about a week ago, but after thinking  
about it the dev list seemed more appropriate.

My apologies for this cross post.

----------------------------

Morning,

I'm attempting to build scipy against a custom atlas/lapack library on  
Snow Leopard (this is for inclusion in macports).

I've set my build environment according to the documentation here:

> http://www.scipy.org/Installing_SciPy/BuildingGeneral

Here's my environment and build command:

CPATH='/Users/jkyle/Projects/macports/mports/include' LAPACK='/Users/ 
jkyle/Projects/macports/mports/lib' LIBRARY_PATH='/Users/jkyle/ 
Projects/macports/mports/lib' MACOSX_DEPLOYMENT_TARGET='10.6' ATLAS='/ 
Users/jkyle/Projects/macports/mports/lib' CC='/Users/jkyle/Projects/ 
macports/mports/bin/gcc-mp-4.3' BLAS='/Users/jkyle/Projects/macports/ 
mports/lib' CXX='/Users/jkyle/Projects/macports/mports/bin/g++-mp-4.3'  
CCFLAGS='-I/Users/jkyle/Projects/macports/mports/include -L/Users/ 
jkyle/Projects/macports/mports/lib'

python2.6 setup.py --no-user-cfg config_fc --fcompiler gnu95 -- 
f77exec /Users/jkyle/Projects/macports/mports/bin/gfortran-mp-4.3 -- 
f90exec /Users/jkyle/Projects/macports/mports/bin/gfortran-mp-4.3 build

And here is the configuration output:

http://pastie.org/622884


Now, everything seems to go ok, but when I check the resulting  
libraries I noticed that they are still linking against the system  
vecLib and Accelerate frameworks (the system os x lapack/blas) when  
building the cblas.so and fblas.so libraries.

The build also seems to create two sets of the cblas and fblas  
libraries. For example, it installs these linked against my libs:

> /Users/jkyle/Projects/macports/mports/Library/Frameworks/ 
> Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/ 
> linalg/cblas.so:
> 	/Users/jkyle/Projects/macports/mports/lib/liblapack.dylib  
> (compatibility version 0.0.0, current version 0.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/libptf77blas.dylib  
> (compatibility version 0.0.0, current version 0.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/libptcblas.dylib  
> (compatibility version 0.0.0, current version 0.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/libatlas.dylib  
> (compatibility version 0.0.0, current version 0.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/gcc43/libgfortran.3.dylib  
> (compatibility version 4.0.0, current version 4.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/gcc43/libgcc_s.1.dylib  
> (compatibility version 1.0.0, current version 1.0.0)
> 	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current  
> version 124.1.1)
>
> /Users/jkyle/Projects/macports/mports/Library/Frameworks/ 
> Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/ 
> linalg/fblas.so:
> 	/Users/jkyle/Projects/macports/mports/lib/liblapack.dylib  
> (compatibility version 0.0.0, current version 0.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/libptf77blas.dylib  
> (compatibility version 0.0.0, current version 0.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/libptcblas.dylib  
> (compatibility version 0.0.0, current version 0.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/libatlas.dylib  
> (compatibility version 0.0.0, current version 0.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/gcc43/libgfortran.3.dylib  
> (compatibility version 4.0.0, current version 4.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/gcc43/libgcc_s.1.dylib  
> (compatibility version 1.0.0, current version 1.0.0)
> 	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current  
> version 124.1.1)

And it installs these linked against the system libs:
>
> /Users/jkyle/Projects/macports/mports/Library/Frameworks/ 
> Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/lib/ 
> blas/cblas.so:
> 	/System/Library/Frameworks/Accelerate.framework/Versions/A/ 
> Accelerate (compatibility version 1.0.0, current version 4.0.0)
> 	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current  
> version 124.1.1)
> /Users/jkyle/Projects/macports/mports/Library/Frameworks/ 
> Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/lib/ 
> blas/fblas.so:
> 	/Users/jkyle/Projects/macports/mports/lib/gcc43/libgfortran.3.dylib  
> (compatibility version 4.0.0, current version 4.0.0)
> 	/System/Library/Frameworks/Accelerate.framework/Versions/A/ 
> Accelerate (compatibility version 1.0.0, current version 4.0.0)
> 	/Users/jkyle/Projects/macports/mports/lib/gcc43/libgcc_s.1.dylib  
> (compatibility version 1.0.0, current version 1.0.0)
> 	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current  
> version 124.1.1)


Lastly, I'm noticing that scipy is building with mixed compilers. I've  
set both CC and CXX in my build environment to 4.3. However, in the  
build logs I see that scipy is using gcc-4.2 for some things. For  
example:

> creating build/temp.macosx-10.6-i386-2.6/scipy/cluster
> creating build/temp.macosx-10.6-i386-2.6/scipy/cluster/src
> compile options: '-I/Users/jkyle/Projects/macports/mports/Library/ 
> Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ 
> numpy/core/include -I/Users/jkyle/Projects/macports/mports/Library/ 
> Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ 
> numpy/core/include -I/Users/jkyle/Projects/macports/mports/Library/ 
> Frameworks/Python.framework/Versions/2.6/include/python2.6 -c'
> gcc-mp-4.3: scipy/cluster/src/vq_module.c
> gcc-mp-4.3: scipy/cluster/src/vq.c
> /Users/jkyle/Projects/macports/mports/Library/Frameworks/ 
> Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/ 
> include/numpy/__multiarray_api.h:969: warning: '_import_array'  
> defined but not used
> /usr/bin/gcc-4.2 -L/Users/jkyle/Projects/macports/mports/lib -bundle  
> -undefined dynamic_lookup build/temp.macosx-10.6-i386-2.6/scipy/ 
> cluster/src/vq_module.o build/temp.macosx-10.6-i386-2.6/scipy/ 
> cluster/src/vq.o -Lbuild/temp.macosx-10.6-i386-2.6 -o build/ 
> lib.macosx-10.6-i386-2.6/scipy/cluster/_vq.so
> building 'scipy.cluster._hierarchy_wrap' extension
> compiling C sources
> C compiler: /Users/jkyle/Projects/macports/mports/bin/gcc-mp-4.3 - 
> fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 - 
> Wall -Wstrict-prototypes

Here, the C compiler is gcc-mp-4.3, but the one actually used is  
gcc-4.2.


Any tips are very welcome.

Cheers,

-james


From cournape at gmail.com  Wed Sep 23 10:16:02 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 23 Sep 2009 23:16:02 +0900
Subject: [SciPy-dev] compiling against custom atlas/lapack on os x
In-Reply-To: <675CB282-54ED-43B8-90D9-CD361093D214@gmail.com>
References: <675CB282-54ED-43B8-90D9-CD361093D214@gmail.com>
Message-ID: <5b8d13220909230716n34ee3980qfcd9b2b4fa8f89@mail.gmail.com>

On Wed, Sep 23, 2009 at 11:02 PM, James Kyle <krunk7 at gmail.com> wrote:
> I posted this to scipy-users about a week ago, but after thinking
> about it the dev list seemed more appropriate.

I don't know exactly what the problem is, except for gcc 4.3 vs 4.2:
the one which is unexpected is the linker, which you need to override
as well, through LD I believe.

I am not sure how you can easily bypass accelerate detection with distutils.

cheers,

David


From sturla at molden.no  Wed Sep 23 11:07:29 2009
From: sturla at molden.no (Sturla Molden)
Date: Wed, 23 Sep 2009 17:07:29 +0200
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <5b8d13220909230658s42845a6q5a5178e404dd9128@mail.gmail.com>
References: <4AB86CF8.10606@molden.no>
	<4AB9BBDF.4020606@molden.no>	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>	<200909230829.58471.lists_ravi@lavabit.com>
	<5b8d13220909230658s42845a6q5a5178e404dd9128@mail.gmail.com>
Message-ID: <4ABA39B1.3000405@molden.no>

David Cournapeau skrev:
> It can be a full rewrite, but still should be sent as patches. If I am
> the one to review, I would prefer this way. That's especially
> important to track regressions.
>
>   
Well.. I didn't intend this for inclusion in SciPy, at least not in the 
beginning. I needed it for some neuroscience software I am writing.
> It is a fundamental problem of C++. Different compilers do not
> propagate exceptions the same way, and that's a problem when different
> compilers are involved (happens easily when the C and C++ compilers
> are not the same, for example). That has been a problem on every new
> platform I have tried to port numpy and scipy to.
>   
Does this mean we cannot use g++ to compile extensions at all, when 
Python VM is compiled with MSVC? Or does the rule just apply to the 
particular extension, like C and Fortran runtimes?

I don't like the complexity of C++. But it has some advantages over C 
for scientific computing; notably STL containers, templates for 
generics, the std::complex<> type, and so on. The problem with C++ is 
that it encourages bloat ugly style, and OOP stuff is  etter left in Python.

I personally like exceptions because they remove the need for lot or 
error checking. In C, we can achieve almost the same effect using 
setjmp/longjmp. Is that bad style as well?



Sturla Molden


From charlesr.harris at gmail.com  Wed Sep 23 11:31:10 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 23 Sep 2009 09:31:10 -0600
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4ABA39B1.3000405@molden.no>
References: <4AB86CF8.10606@molden.no> <4AB9BBDF.4020606@molden.no>
	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>
	<200909230829.58471.lists_ravi@lavabit.com>
	<5b8d13220909230658s42845a6q5a5178e404dd9128@mail.gmail.com>
	<4ABA39B1.3000405@molden.no>
Message-ID: <e06186140909230831n6b9ddd82pfeb6d724d65588a1@mail.gmail.com>

On Wed, Sep 23, 2009 at 9:07 AM, Sturla Molden <sturla at molden.no> wrote:

> David Cournapeau skrev:
> > It can be a full rewrite, but still should be sent as patches. If I am
> > the one to review, I would prefer this way. That's especially
> > important to track regressions.
> >
> >
> Well.. I didn't intend this for inclusion in SciPy, at least not in the
> beginning. I needed it for some neuroscience software I am writing.
> > It is a fundamental problem of C++. Different compilers do not
> > propagate exceptions the same way, and that's a problem when different
> > compilers are involved (happens easily when the C and C++ compilers
> > are not the same, for example). That has been a problem on every new
> > platform I have tried to port numpy and scipy to.
> >
> Does this mean we cannot use g++ to compile extensions at all, when
> Python VM is compiled with MSVC? Or does the rule just apply to the
> particular extension, like C and Fortran runtimes?
>
> I don't like the complexity of C++. But it has some advantages over C
> for scientific computing; notably STL containers, templates for
> generics, the std::complex<> type, and so on. The problem with C++ is
> that it encourages bloat ugly style, and OOP stuff is  etter left in
> Python.
>
> I personally like exceptions because they remove the need for lot or
> error checking. In C, we can achieve almost the same effect using
> setjmp/longjmp. Is that bad style as well?
>
>
The setjmp/longjmp machinery has been used (zeros.c, for instance), but I
thinks it makes the C code less transparent and should be avoided unless the
traditional use of return values is so ugly you can't bear to look at the
code. C really doesn't have very good error handling, that may even be a
feature in a low level language.

It might be useful to think up some way that setjmp/longjmp could be used in
some sort of standard error handling template.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090923/0829c4c8/attachment.html>

From jameskyle at ucla.edu  Wed Sep 23 11:54:51 2009
From: jameskyle at ucla.edu (James Kyle)
Date: Wed, 23 Sep 2009 08:54:51 -0700
Subject: [SciPy-dev] compiling against custom atlas/lapack on os x
In-Reply-To: <5b8d13220909230716n34ee3980qfcd9b2b4fa8f89@mail.gmail.com>
References: <675CB282-54ED-43B8-90D9-CD361093D214@gmail.com>
	<5b8d13220909230716n34ee3980qfcd9b2b4fa8f89@mail.gmail.com>
Message-ID: <3B099F86-57AE-4F94-910F-80B18735D89A@ucla.edu>

> I don't know exactly what the problem is, except for gcc 4.3 vs 4.2:
> the one which is unexpected is the linker, which you need to override
> as well, through LD I believe.
>
> I am not sure how you can easily bypass accelerate detection with  
> distutils.

Thanks for the response, David.

Oh, good. I'll confirm how to override the linker.

But yes, I was looking at how to override the veclib/accellerate  
detection. I think I've run down the areas where it's set, I can just  
patch those files.

-james


From cournape at gmail.com  Wed Sep 23 12:25:46 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 24 Sep 2009 01:25:46 +0900
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4ABA39B1.3000405@molden.no>
References: <4AB86CF8.10606@molden.no> <4AB9BBDF.4020606@molden.no>
	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>
	<200909230829.58471.lists_ravi@lavabit.com>
	<5b8d13220909230658s42845a6q5a5178e404dd9128@mail.gmail.com>
	<4ABA39B1.3000405@molden.no>
Message-ID: <5b8d13220909230925v243f7508m906ae73c0e8ac531@mail.gmail.com>

On Thu, Sep 24, 2009 at 12:07 AM, Sturla Molden <sturla at molden.no> wrote:
> David Cournapeau skrev:
>> It can be a full rewrite, but still should be sent as patches. If I am
>> the one to review, I would prefer this way. That's especially
>> important to track regressions.
>>
>>
> Well.. I didn't intend this for inclusion in SciPy, at least not in the
> beginning. I needed it for some neuroscience software I am writing.
>> It is a fundamental problem of C++. Different compilers do not
>> propagate exceptions the same way, and that's a problem when different
>> compilers are involved (happens easily when the C and C++ compilers
>> are not the same, for example). That has been a problem on every new
>> platform I have tried to port numpy and scipy to.
>>
> Does this mean we cannot use g++ to compile extensions at all, when
> Python VM is compiled with MSVC?

Oh, you certainly *can*, as we can use gcc to compile extensions
against python built with MS compiler. But there are limitations -
with C, those limitations are known, with C++, they become atrocious
once you use the C++ runtime (RTTI, exception, etc...). I had to debug
some stuff in sparsetools on windows 64 bits a few days ago, and this
was horrible because the exceptions are lost between DLL.

> I don't like the complexity of C++. But it has some advantages over C
> for scientific computing; notably STL containers, templates for
> generics, the std::complex<> type, and so on.

For basic templates, the .src stuff is good enough. For complex, yes,
that's a bit of a pain, although you could use the C99 complex type if
you only care about modern C compilers in your own extension.

>
> I personally like exceptions because they remove the need for lot or
> error checking.

I call this a disadvantage :) And having exceptions in a language
without gc causes more trouble than it worths IMHO. Exception-safe C++
is incredibly hard to do right.

> In C, we can achieve almost the same effect using
> setjmp/longjmp. Is that bad style as well?

The standard way to handle errors in C code using the python API is
goto. It works well, and if you need to jump several calls, you can
always use python exception mechanism.

cheers,

David


From kgdunn at gmail.com  Wed Sep 23 16:36:10 2009
From: kgdunn at gmail.com (Kevin Dunn)
Date: Wed, 23 Sep 2009 16:36:10 -0400
Subject: [SciPy-dev] Any advice for building SciPy? LAPACK error
Message-ID: <ad94c31e0909231336j3394b2aeuad398c11aa956a6@mail.gmail.com>

Hi there,

Yesterday I asked about building Scipy
(http://article.gmane.org/gmane.comp.python.scientific.devel/11832) as
I was following the various instructions
* here: http://www.scipy.org/Installing_SciPy/BuildingGeneral
* and here: http://www.scipy.org/Installing_SciPy/Linux

Since then I discovered the SCons script, so first question: is using
the SCons script the preferred way to build SciPy now? If so, I think
those website should be updated.  If not, please help with the next
question.

Using the setupscons.py script (after cloning and installing numscons
from github, version 0.11.0), I encounter the error, "no lapack found,
necessary for lapack module" when running "python setupscons.py
build".  More details in the traceback below.

For reference:
* Ubuntu 9.04 (Jaunty)
* SciPy revision 5951
* other details about my platform: http://pastebin.com/m26f8d1d1
* and I followed the LAPACK instructions at the bottom of this page
http://www.scipy.org/Installing_SciPy/BuildingGeneral (i.e. I built
LAPACK from source, and I set the environment variable)

Any additional pointers to solve this ?  I thought ATLAS is used
instead of LAPACK; can someone help me understand this?
Thanks,
Kevin

Executing scons command (pkg is scipy.lib.lapack): /usr/bin/python
"/usr/local/lib/python2.6/dist-packages/numscons/scons-local/scons.py"
-f scipy/lib/lapack/SConstruct -I. scons_tool_path=""
src_dir="scipy/lib/lapack" pkg_name="scipy.lib.lapack" log_level=50
distutils_libdir="../../../../../build/lib.linux-i686-2.6"
distutils_clibdir="../../../../../build/temp.linux-i686-2.6"
distutils_install_prefix="/usr/local/lib/python2.6/dist-packages/scipy/lib/lapack"
cc_opt=gcc cc_opt_path="/usr/bin" debug=0 f77_opt=gfortran
f77_opt_path="/usr/bin" cxx_opt=g++ cxx_opt_path="/usr/bin"
include_bootstrap=/usr/local/lib/python2.6/dist-packages/numpy/core/include
bypass=0 import_env=0 silent=0 bootstrapping=0
scons: Reading SConscript files ...
Checking if gfortran needs dummy main - MAIN__.
Checking gfortran name mangling - '_', '', lower-case.
Checking for ATLAS ... yes
Checking for F77 LAPACK ... no
RuntimeError: no lapack found, necessary for lapack module:
  File "/home/connectmv/workspace/scipy/scipy/lib/lapack/SConstruct", line 2:
    GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript')
  File "/usr/local/lib/python2.6/dist-packages/numscons/core/numpyenv.py",
line 133:
    build_dir = '$build_dir', src_dir = '$src_dir')
  File "/usr/local/lib/python2.6/dist-packages/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py",
line 553:
    return apply(_SConscript, [self.fs,] + files, subst_kw)
  File "/usr/local/lib/python2.6/dist-packages/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py",
line 262:
    exec _file_ in call_stack[-1].globals
  File "/home/connectmv/workspace/scipy/build/scons/scipy/lib/lapack/SConscript",
line 37:
    raise RuntimeError("no lapack found, necessary for lapack module")
error: Error while executing scons command. See above for more information.
If you think it is a problem in numscons, you can also try executing the scons
command with --log-level option for more detailed output of what numscons is
doing, for example --log-level=0; the lowest the level is, the more detailed
the output it.


From cournape at gmail.com  Wed Sep 23 20:03:05 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 24 Sep 2009 09:03:05 +0900
Subject: [SciPy-dev] Any advice for building SciPy? LAPACK error
In-Reply-To: <ad94c31e0909231336j3394b2aeuad398c11aa956a6@mail.gmail.com>
References: <ad94c31e0909231336j3394b2aeuad398c11aa956a6@mail.gmail.com>
Message-ID: <5b8d13220909231703n78ec6c03q4c64042547dd45f9@mail.gmail.com>

On Thu, Sep 24, 2009 at 5:36 AM, Kevin Dunn <kgdunn at gmail.com> wrote:
> Hi there,
>
> Yesterday I asked about building Scipy
> (http://article.gmane.org/gmane.comp.python.scientific.devel/11832) as
> I was following the various instructions
> * here: http://www.scipy.org/Installing_SciPy/BuildingGeneral
> * and here: http://www.scipy.org/Installing_SciPy/Linux
>
> Since then I discovered the SCons script, so first question: is using
> the SCons script the preferred way to build SciPy now?

No, the default option is still to use distutils (the setup.py files).
But the scons scripts are maintained and should work on most platforms
(I almost exclusively use the scons build myself).

> Using the setupscons.py script (after cloning and installing numscons
> from github, version 0.11.0), I encounter the error, "no lapack found,
> necessary for lapack module" when running "python setupscons.py
> build". ?More details in the traceback below.

You downloaded a development version, so things may be broken. To get
the exact error, you can look at
build/scons/scipy/lib/lapack/config.log, and paste this output.

cheers,

David


From sturla at molden.no  Wed Sep 23 21:10:45 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 24 Sep 2009 03:10:45 +0200
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <e06186140909230831n6b9ddd82pfeb6d724d65588a1@mail.gmail.com>
References: <4AB86CF8.10606@molden.no>
	<4AB9BBDF.4020606@molden.no>	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>	<200909230829.58471.lists_ravi@lavabit.com>	<5b8d13220909230658s42845a6q5a5178e404dd9128@mail.gmail.com>	<4ABA39B1.3000405@molden.no>
	<e06186140909230831n6b9ddd82pfeb6d724d65588a1@mail.gmail.com>
Message-ID: <4ABAC715.6070507@molden.no>

Charles R Harris skrev:
> It might be useful to think up some way that setjmp/longjmp could be 
> used in some sort of standard error handling template.
>

One way is to use a resource pool (a la Apache), see Cython code below.

cdef int _foobar() nogil:
    cdef  mempool *mp
    cdef int rv
    mp = mempool_init()
    if (mp == NULL):
        return -1 # error
    else:
        some_C_function(mp)
        mempool_destroy(mp)
        return int(rv)

def foobar():
    if (_foobar() < 0):
        raise MemoryError, 'malloc failed'

This alleviates any need for error checking on the return value from 
mempool_malloc. It might not be obvious though. What happens is that a 
NULL return form malloc trigger a longjmp back to mempool_init, after 
which the memory error is raised. Anything left allocated by the pool is 
freed on mempool_destroy, so it cannot leak. Pools can be used to manage 
any type of resource, e.g. open file handles, not just memory. The 
advantage is that you get C code that don't have to do error checking 
everywhere.

The problem with Cython is that a longjmp can mess up refcounts, so it 
probably should only be used in pure C mode, i.e. in a function safe to 
call with nogil.

C++ exceptions are much cleaner than C resource pools though, for 
example because destructors are called automatically. And the fact that 
C++ can put objects on the stack, and references can be used instead of 
pointers, means it is very safe against resource leaks if used 
correctly. The problems people have with C++ comes from bad style, for 
example calling new outside a constructor, using new[] instead of 
std::vector, using pointers instead of references (a reference cannot be 
dangling), etc.


Sturla Molden




# memory pool
# Copyright (C) 2009 Sturla Molden

import numpy as np
cimport numpy

cdef extern from "setjmp.h":
    ctypedef struct jmp_buf:
        pass
    int setjmp(jmp_buf jb) nogil
    void longjmp(jmp_buf jb, int errcode) nogil
  
cdef extern from "stdlib.h":
    # Assume malloc, realloc and free are thread safe
    # We may need to change this, in which case a lock
    # is needed in the memory pool.
    ctypedef unsigned long size_t
    void free(void *ptr) nogil
    void *malloc(size_t size) nogil
    void *realloc(void *ptr, size_t size) nogil
  
cdef struct mempool:
    mempool *prev
    mempool *next
  
cdef public mempool *mempool_init() nogil:
    cdef jmp_buf *jb
    cdef mempool *p = <mempool *> malloc(sizeof(jmp_buf) + sizeof(mempool))
    if not p: return NULL
    jb = <jmp_buf*>(<np.npy_intp>p + sizeof(mempool))
    p.prev = NULL
    p.next = NULL
    if setjmp(jb[0]):
        # mempool_error has been called
        mempool_destroy(p)
        return NULL
    else:
        return p
  
cdef public void mempool_error(mempool *p) nogil:
    # rewind stack back to mempool_init()
    cdef jmp_buf *jb = <jmp_buf*>(<np.npy_intp>p + sizeof(mempool))
    longjmp(jb[0], 1)

cdef public void *mempool_destroy(mempool *p) nogil:
    # this releases all memory allocated to this pool
    cdef mempool *tmp
    while p:
        tmp = p
        p = p.next
        free(tmp)

cdef public void *mempool_malloc(mempool *p, size_t n) nogil:
    cdef mempool *block
    cdef void *m = malloc(sizeof(mempool) + n)
    if not m:
        mempool_error(p)
    block = <mempool*> m
    block.next = p.next
    if block.next:
        block.next.prev = block
    block.prev = p
    p.next = block
    return <void*>(<np.npy_intp>m + sizeof(mempool))

cdef public void *mempool_realloc(mempool *p, void *ptr, size_t n) nogil:
    cdef void *retval
    cdef mempool *block, *prev, *next, *new_addr
    if not n:
        # realloc(p, 0) is free(p)
        mempool_free(ptr)
        retval = NULL
    elif not ptr:
        # realloc(0, n) is malloc(n)
        retval = mempool_malloc(p,n)
    else:
        # realloc(p, n)
        block = <mempool *>(<np.npy_intp>ptr - sizeof(mempool))
        prev = block.prev
        next = block.next
        new_addr = <mempool*> realloc(block, n + sizeof(mempool))
        if not new_addr:
            mempool_error(p)
        if new_addr != block:
            prev.next = new_addr
            if next:
                next.prev = new_addr
            new_addr.prev = prev
            new_addr.next = next
        retval = <void *>(<np.npy_intp>new_addr + sizeof(mempool))
    return retval

cdef public void *mempool_free(void *ptr) nogil:
    cdef mempool *prev, *next, *cur
    cdef mempool *block = <mempool *>(<np.npy_intp>ptr - sizeof(mempool))
    prev = block.prev
    next = block.next
    if next:
        free(block)
        next.prev = prev
        prev.next = next
    else:
        free(block)
        prev.next = NULL







From sturla at molden.no  Wed Sep 23 21:43:30 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 24 Sep 2009 03:43:30 +0200
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <5b8d13220909230925v243f7508m906ae73c0e8ac531@mail.gmail.com>
References: <4AB86CF8.10606@molden.no>
	<4AB9BBDF.4020606@molden.no>	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>	<200909230829.58471.lists_ravi@lavabit.com>	<5b8d13220909230658s42845a6q5a5178e404dd9128@mail.gmail.com>	<4ABA39B1.3000405@molden.no>
	<5b8d13220909230925v243f7508m906ae73c0e8ac531@mail.gmail.com>
Message-ID: <4ABACEC2.6050300@molden.no>

David Cournapeau skrev:
> Oh, you certainly *can*, as we can use gcc to compile extensions
> against python built with MS compiler. But there are limitations -
> with C, those limitations are known, with C++, they become atrocious
> once you use the C++ runtime (RTTI, exception, etc...). I had to debug
> some stuff in sparsetools on windows 64 bits a few days ago, and this
> was horrible because the exceptions are lost between DLL.
>
>   
Mingw (g++) will statically link its own C++ runtime.  DLLs compiled 
with g++ do not share C++ runtime. This obviously is a problem if you 
need to propagate an exception from one DLL to another. MS Visual C++ 
does not have this issue.

You get the same issue in C if a DLL statically links its own C runtime, 
or two DLLs use different shared C runtimes.

> I call this a disadvantage :) And having exceptions in a language
> without gc causes more trouble than it worths IMHO. Exception-safe C++
> is incredibly hard to do right.
>   
I'd say exceptipn safe C++  is easy to do right, but C++ textbooks don't 
teach you how. They generally teach you all the mistakes you can do. And 
if a book has 'best practices' or 'effective' in the title, it is likely 
full of BS.

The main issue is to never -- ever -- open an allocated reource outside 
a constructor. That includes calling new or new[]. Always use 
std::vector instead of new[], unless you have an extremely good reason. 
Always deallocate resources in a destructor. Objects should always be 
put on the stack (unless allocated in a constructor), and references 
should be used instead of pointers.

For example, look at wxWidgets, that has an additional Destroy() method 
in the GUI classes. Destroy() must be called manually. This is a classic 
C++ mistake. Destroy() is not called automatically, so an exception will 
mess this up.

Sturla Molden






>   
>> In C, we can achieve almost the same effect using
>> setjmp/longjmp. Is that bad style as well?
>>     
>
> The standard way to handle errors in C code using the python API is
> goto. It works well, and if you need to jump several calls, you can
> always use python exception mechanism.
>
> cheers,
>
> David
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   



From david at ar.media.kyoto-u.ac.jp  Wed Sep 23 21:26:15 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 24 Sep 2009 10:26:15 +0900
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4ABAC715.6070507@molden.no>
References: <4AB86CF8.10606@molden.no>	<4AB9BBDF.4020606@molden.no>	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>	<200909230829.58471.lists_ravi@lavabit.com>	<5b8d13220909230658s42845a6q5a5178e404dd9128@mail.gmail.com>	<4ABA39B1.3000405@molden.no>	<e06186140909230831n6b9ddd82pfeb6d724d65588a1@mail.gmail.com>
	<4ABAC715.6070507@molden.no>
Message-ID: <4ABACAB7.3030102@ar.media.kyoto-u.ac.jp>

Sturla Molden wrote:
> The 
> advantage is that you get C code that don't have to do error checking 
> everywhere.
>   

Really, this is a big disadvantage. In C, you have to check error codes.
Anything which gives the illusion you can get away with it is wrong
IMHO. Technically,  setjmp/longjmp are also problematic for many reasons
(often the same as C++ exceptions: it is difficult to get right). At
least with goto you can often understand what's wrong - not so much with
longjmp/setjmp - if the error handling is too complicated to handle with
simple goto, then the code itself is too complicated. If you need to
handle deeply nested errors, then using the C API for python exception
is good enough.

I agree we are not as good as we should in numpy for error handling at
the C level, but I think it boils down to not having a unified error
handling (that is -3 may mean could not allocate in one function, and
could not convert the argument to something in another). But that can be
solved with a unified set of error codes and associated strings + a
simple log system (that's how sndfile does it for example, and I really
like it).

cheers,

David




From david at ar.media.kyoto-u.ac.jp  Wed Sep 23 21:37:55 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 24 Sep 2009 10:37:55 +0900
Subject: [SciPy-dev] reimplementation of lfilter
In-Reply-To: <4ABACEC2.6050300@molden.no>
References: <4AB86CF8.10606@molden.no>	<4AB9BBDF.4020606@molden.no>	<5b8d13220909230131x601efe41yf7c88f496fa4770b@mail.gmail.com>	<200909230829.58471.lists_ravi@lavabit.com>	<5b8d13220909230658s42845a6q5a5178e404dd9128@mail.gmail.com>	<4ABA39B1.3000405@molden.no>	<5b8d13220909230925v243f7508m906ae73c0e8ac531@mail.gmail.com>
	<4ABACEC2.6050300@molden.no>
Message-ID: <4ABACD73.4090303@ar.media.kyoto-u.ac.jp>

Sturla Molden wrote:
> Mingw (g++) will statically link its own C++ runtime.  DLLs compiled 
> with g++ do not share C++ runtime. This obviously is a problem if you 
> need to propagate an exception from one DLL to another. MS Visual C++ 
> does not have this issue.
>   

yes it does, when you mix C runtimes (/MD vs /MDd for example),
different C++ code is compiled with different incompatible options, or
when you mix mingw and msvc (Those are all concrete problems I had to
solve on windows at one point or the other BTW).

Those problems are not c++ specific, but they are much more manageable
in C than in C++. Interoperability with C++ from other languages is a
known problem - given that numpy supports so many compilers and
platforms, and that using 'weird' compilers is common in scientific
computing, I don't think it worths the hassle in most cases.

cheers,

David


From Emmanuel.Lambert at intec.ugent.be  Thu Sep 24 02:57:41 2009
From: Emmanuel.Lambert at intec.ugent.be (Emmanuel Lambert)
Date: Thu, 24 Sep 2009 08:57:41 +0200
Subject: [SciPy-dev] scipy-weave unit tests fail on KeyError
Message-ID: <1253775461.3836.3.camel@emmanuel-ubuntu>

Hi,

I compiled SciPy and Numpy on a machine with Scientific Linux (64bit).
We detected a problem with Weave and after
investigation, it turns out that some of the unit tests delivered
with Scipy-Weave also fail ! Below is a list of tests that fail in for
example the "test_c_spec" file. They all raise a KeyError.

This is with SciPy 0.7.1 on Python 2.6. I also downloaded the latest
Weave code again from the SVN repository (trunk), but the problem is not
resolved.

Any idea on how to tackle this problem? There are no posts that help
me further. 
It looks like the compilation done by Weave works fine (see sample
stdout also below), but afterwards, he cannot find the compiled code in
the list.

What could cause this? 

thanks for any help.
Emmanuel

******************* SAMPLE OF STDOUT ******************

-------------------- >> begin captured stdout << ---------------------
<weave: compiling>
running build_ext
running build_src
building extension "sc_d133102ab45193e072f8dbb5a1f6848513" sources
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
building 'sc_d133102ab45193e072f8dbb5a1f6848513' extension
compiling C++ sources
C compiler: g++ -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -
Wall -fPIC

compile options: '-I/user/home/gent/vsc401/vsc40157/scipy-runtime/
scipy/weave -I/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/
weave/scxx -I/user/home/gent/vsc401/vsc40157/numpy-runtime/numpy/core/
include -I/apps/gent/gengar/harpertown/software/Python/2.6.2-
gimkl-0.5.0/include/python2.6 -c'
g++: /user/home/gent/vsc401/vsc40157/.python26_compiled/
sc_d133102ab45193e072f8dbb5a1f6848513.cpp
g++ -pthread -shared /tmp/vsc40157/python26_intermediate/
compiler_c1b5f1b73f1ce7d0c836cdad4c7c5ded/user/home/gent/vsc401/
vsc40157/.python26_compiled/sc_d133102ab45193e072f8dbb5a1f6848513.o /
tmp/vsc40157/python26_intermediate/
compiler_c1b5f1b73f1ce7d0c836cdad4c7c5ded/user/home/gent/vsc401/
vsc40157/scipy-runtime/scipy/weave/scxx/weave_imp.o -o /user/home/gent/
vsc401/vsc40157/.python26_compiled/
sc_d133102ab45193e072f8dbb5a1f6848513.so
running scons

--------------------- >> end captured stdout << ----------------------


********************** TESTS THAT FAIL ***********************

-bash-3.2$ python ./test_c_spec.py
E..........EE.................EEEE......E..........EE.................EEEE..............
======================================================================
ERROR: test_call_function (test_c_spec.CallableConverter)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 296, in test_call_function
   compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_file_to_py (test_c_spec.FileConverter)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 262, in test_file_to_py
   force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_py_to_file (test_c_spec.FileConverter)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 246, in test_py_to_file
   inline_tools.inline(code,['file'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_convert_to_dict (test_c_spec.SequenceConverter)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 305, in test_convert_to_dict
   inline_tools.inline("",['d'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_convert_to_list (test_c_spec.SequenceConverter)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 309, in test_convert_to_list
   inline_tools.inline("",['l'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_convert_to_string (test_c_spec.SequenceConverter)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 313, in test_convert_to_string
   inline_tools.inline("",['s'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_convert_to_tuple (test_c_spec.SequenceConverter)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 317, in test_convert_to_tuple
   inline_tools.inline("",['t'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_call_function (test_c_spec.TestCallableConverterUnix)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 296, in test_call_function
   compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_file_to_py (test_c_spec.TestFileConverterUnix)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 262, in test_file_to_py
   force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_py_to_file (test_c_spec.TestFileConverterUnix)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 246, in test_py_to_file
   inline_tools.inline(code,['file'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_convert_to_dict (test_c_spec.TestSequenceConverterUnix)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 305, in test_convert_to_dict
   inline_tools.inline("",['d'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_convert_to_list (test_c_spec.TestSequenceConverterUnix)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 309, in test_convert_to_list
   inline_tools.inline("",['l'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_convert_to_string (test_c_spec.TestSequenceConverterUnix)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 313, in test_convert_to_string
   inline_tools.inline("",['s'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

======================================================================
ERROR: test_convert_to_tuple (test_c_spec.TestSequenceConverterUnix)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
tests/test_c_spec.py", line 317, in test_convert_to_tuple
   inline_tools.inline("",['t'],compiler=self.compiler,force=1)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
inline_tools.py", line 301, in inline
   function_catalog.add_function(code,func,module_dir)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 648, in add_function
   self.cache[code] = self.get_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 615, in get_functions
   function_list = self.get_cataloged_functions(code)
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/weave/
catalog.py", line 529, in get_cataloged_functions
   if cat is not None and code in cat:
 File "/apps/gent/gengar/harpertown/software/Python/2.6.2-gimkl-0.5.0/
lib/python2.6/shelve.py", line 110, in __contains__
   return key in self.dict
 File "/user/home/gent/vsc401/vsc40157/scipy-runtime/scipy/io/
dumbdbm_patched.py", line 73, in __getitem__
   pos, siz = self._index[key] # may raise KeyError
KeyError: 0

----------------------------------------------------------------------
Ran 88 tests in 32.581s

FAILED (errors=14)





From jsseabold at gmail.com  Thu Sep 24 13:50:43 2009
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 24 Sep 2009 13:50:43 -0400
Subject: [SciPy-dev] Installing Numpy/Scipy with custom ATLAS
Message-ID: <c048da1c0909241050w177117aakd96ddccf9a533114@mail.gmail.com>

For some reason I thought it would be a good idea to install a custom
ATLAS...again.

I think I've built lapack 3.1.1 and atlas 3.8.0 correctly following
the instructions here <http://www.scipy.org/Installing_SciPy/Linux>
and the ones included with atlas, but we shall see...the checks pass
at least.  I installed the atlas libraries to ~/atlas and copied over
the dynamic libraries by hand.

I then edit my site.cfg

[DEFAULT]
library_dirs = /usr/local/lib:/usr/lib:/home/skipper/atlas/lib
include_dirs = /usr/local/include:/usr/include:usr/include/suitesparse:/home/skipper/atlas/include
[blas_opt]
libraries = ptf77blas, ptcblas, atlas
[lapack_opt]
libraries = lapack, ptf77blas, ptcblas, atlas
[amd]
amd_libs = amd
[umfpack]
umfpack_libs = umfpack
[fftw]
libraries = fftw3

I then build and install numpy without error.  But when I try to
import numpy I get
[...]
/usr/local/lib/python2.6/dist-packages/numpy/linalg/linalg.py in <module>()
     20         isfinite, size
     21 from numpy.lib import triu
---> 22 from numpy.linalg import lapack_lite
     23 from numpy.matrixlib.defmatrix import matrix_power
     24

ImportError: /usr/local/lib/python2.6/dist-packages/numpy/linalg/lapack_lite.so:
undefined symbol: ATL_dptsyrk


Running ldd gives

$ ldd /usr/local/lib/python2.6/dist-packages/numpy/linalg/lapack_lite.so
        linux-vdso.so.1 =>  (0x00007fff129fe000)
        liblapack.so => /usr/lib/atlas/liblapack.so (0x00007f4609c57000)
        libatlas.so => /usr/lib/libatlas.so (0x00007f4609615000)
        libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00007f4609338000)
        libm.so.6 => /lib/libm.so.6 (0x00007f46090b3000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f4608e9b000)
        libc.so.6 => /lib/libc.so.6 (0x00007f4608b28000)
        libblas.so.3gf => /usr/lib/atlas/libblas.so.3gf (0x00007f46084ad000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f460a991000)

I currently have atlas installed from David's repo as well
(incidentally, David, where is your repo, I can't find it again, and
wanted to add it to the wiki.) and am wondering if there is a conflict
here.  When I look at the build log it says, for example,

  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/home/skipper/atlas/lib']
    language = c
    include_dirs = ['/usr/include']

Should include_dirs be '/home/skipper/atlas/include' ?  I don't know.
Any pointers would be appreciated, as I wasn't able to install numpy
without trying to use the threaded libraries either now...  I can
provide anymore output as needed.  Build log attached.

Cheers,

Skipper
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log
Type: application/octet-stream
Size: 12277 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090924/d8a62a77/attachment.obj>

From jsseabold at gmail.com  Thu Sep 24 14:10:24 2009
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 24 Sep 2009 14:10:24 -0400
Subject: [SciPy-dev] Installing Numpy/Scipy with custom ATLAS
In-Reply-To: <c048da1c0909241050w177117aakd96ddccf9a533114@mail.gmail.com>
References: <c048da1c0909241050w177117aakd96ddccf9a533114@mail.gmail.com>
Message-ID: <c048da1c0909241110q659a5f65i175db2827c397c92@mail.gmail.com>

On Thu, Sep 24, 2009 at 1:50 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> For some reason I thought it would be a good idea to install a custom
> ATLAS...again.
>
> I think I've built lapack 3.1.1 and atlas 3.8.0 correctly following
> the instructions here <http://www.scipy.org/Installing_SciPy/Linux>
> and the ones included with atlas, but we shall see...the checks pass
> at least. ?I installed the atlas libraries to ~/atlas and copied over
> the dynamic libraries by hand.
>
> I then edit my site.cfg
>
> [DEFAULT]
> library_dirs = /usr/local/lib:/usr/lib:/home/skipper/atlas/lib
> include_dirs = /usr/local/include:/usr/include:usr/include/suitesparse:/home/skipper/atlas/include
> [blas_opt]
> libraries = ptf77blas, ptcblas, atlas
> [lapack_opt]
> libraries = lapack, ptf77blas, ptcblas, atlas
> [amd]
> amd_libs = amd
> [umfpack]
> umfpack_libs = umfpack
> [fftw]
> libraries = fftw3
>
> I then build and install numpy without error. ?But when I try to
> import numpy I get
> [...]
> /usr/local/lib/python2.6/dist-packages/numpy/linalg/linalg.py in <module>()
> ? ? 20 ? ? ? ? isfinite, size
> ? ? 21 from numpy.lib import triu
> ---> 22 from numpy.linalg import lapack_lite
> ? ? 23 from numpy.matrixlib.defmatrix import matrix_power
> ? ? 24
>
> ImportError: /usr/local/lib/python2.6/dist-packages/numpy/linalg/lapack_lite.so:
> undefined symbol: ATL_dptsyrk
>
>
> Running ldd gives
>
> $ ldd /usr/local/lib/python2.6/dist-packages/numpy/linalg/lapack_lite.so
> ? ? ? ?linux-vdso.so.1 => ?(0x00007fff129fe000)
> ? ? ? ?liblapack.so => /usr/lib/atlas/liblapack.so (0x00007f4609c57000)
> ? ? ? ?libatlas.so => /usr/lib/libatlas.so (0x00007f4609615000)
> ? ? ? ?libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00007f4609338000)
> ? ? ? ?libm.so.6 => /lib/libm.so.6 (0x00007f46090b3000)
> ? ? ? ?libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f4608e9b000)
> ? ? ? ?libc.so.6 => /lib/libc.so.6 (0x00007f4608b28000)
> ? ? ? ?libblas.so.3gf => /usr/lib/atlas/libblas.so.3gf (0x00007f46084ad000)
> ? ? ? ?/lib64/ld-linux-x86-64.so.2 (0x00007f460a991000)
>
> I currently have atlas installed from David's repo as well
> (incidentally, David, where is your repo, I can't find it again, and
> wanted to add it to the wiki.) and am wondering if there is a conflict
> here. ?When I look at the build log it says, for example,
>
> ?FOUND:
> ? ?libraries = ['ptf77blas', 'ptcblas', 'atlas']
> ? ?library_dirs = ['/home/skipper/atlas/lib']
> ? ?language = c
> ? ?include_dirs = ['/usr/include']
>
> Should include_dirs be '/home/skipper/atlas/include' ? ?I don't know.
> Any pointers would be appreciated, as I wasn't able to install numpy
> without trying to use the threaded libraries either now... ?I can
> provide anymore output as needed. ?Build log attached.
>
> Cheers,
>
> Skipper
>

I figured the include directories in site.cfg were greedy, so I
changed the order to

library_dirs = /usr/local/lib:/usr/lib:/home/skipper/atlas/lib
include_dirs = /home/skipper/atlas/include:/usr/local/include:/usr/include:/usr/include/suitesparse

but I get the same import error in numpy.

The correct include directory is listed in the build log, but I'm
wondering why it says atlas 3.8.3 when I installed 3.8.0

New build log attached.

Skipper
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build3.log
Type: application/octet-stream
Size: 73650 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090924/c1125b5c/attachment.obj>

From jsseabold at gmail.com  Thu Sep 24 14:40:36 2009
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 24 Sep 2009 14:40:36 -0400
Subject: [SciPy-dev] Installing Numpy/Scipy with custom ATLAS [SOLVED]
Message-ID: <c048da1c0909241140x3308b362g5dd81499bbe2e566@mail.gmail.com>

On Thu, Sep 24, 2009 at 2:10 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Thu, Sep 24, 2009 at 1:50 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> For some reason I thought it would be a good idea to install a custom
>> ATLAS...again.
>>
>> I think I've built lapack 3.1.1 and atlas 3.8.0 correctly following
>> the instructions here <http://www.scipy.org/Installing_SciPy/Linux>
>> and the ones included with atlas, but we shall see...the checks pass
>> at least. ?I installed the atlas libraries to ~/atlas and copied over
>> the dynamic libraries by hand.
>>
>> I then edit my site.cfg
>>
>> [DEFAULT]
>> library_dirs = /usr/local/lib:/usr/lib:/home/skipper/atlas/lib
>> include_dirs = /usr/local/include:/usr/include:usr/include/suitesparse:/home/skipper/atlas/include
>> [blas_opt]
>> libraries = ptf77blas, ptcblas, atlas
>> [lapack_opt]
>> libraries = lapack, ptf77blas, ptcblas, atlas
>> [amd]
>> amd_libs = amd
>> [umfpack]
>> umfpack_libs = umfpack
>> [fftw]
>> libraries = fftw3
>>
>> I then build and install numpy without error. ?But when I try to
>> import numpy I get
>> [...]
>> /usr/local/lib/python2.6/dist-packages/numpy/linalg/linalg.py in <module>()
>> ? ? 20 ? ? ? ? isfinite, size
>> ? ? 21 from numpy.lib import triu
>> ---> 22 from numpy.linalg import lapack_lite
>> ? ? 23 from numpy.matrixlib.defmatrix import matrix_power
>> ? ? 24
>>
>> ImportError: /usr/local/lib/python2.6/dist-packages/numpy/linalg/lapack_lite.so:
>> undefined symbol: ATL_dptsyrk
>>
>>
>> Running ldd gives
>>
>> $ ldd /usr/local/lib/python2.6/dist-packages/numpy/linalg/lapack_lite.so
>> ? ? ? ?linux-vdso.so.1 => ?(0x00007fff129fe000)
>> ? ? ? ?liblapack.so => /usr/lib/atlas/liblapack.so (0x00007f4609c57000)
>> ? ? ? ?libatlas.so => /usr/lib/libatlas.so (0x00007f4609615000)
>> ? ? ? ?libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00007f4609338000)
>> ? ? ? ?libm.so.6 => /lib/libm.so.6 (0x00007f46090b3000)
>> ? ? ? ?libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f4608e9b000)
>> ? ? ? ?libc.so.6 => /lib/libc.so.6 (0x00007f4608b28000)
>> ? ? ? ?libblas.so.3gf => /usr/lib/atlas/libblas.so.3gf (0x00007f46084ad000)
>> ? ? ? ?/lib64/ld-linux-x86-64.so.2 (0x00007f460a991000)
>>
>> I currently have atlas installed from David's repo as well
>> (incidentally, David, where is your repo, I can't find it again, and
>> wanted to add it to the wiki.) and am wondering if there is a conflict
>> here. ?When I look at the build log it says, for example,
>>
>> ?FOUND:
>> ? ?libraries = ['ptf77blas', 'ptcblas', 'atlas']
>> ? ?library_dirs = ['/home/skipper/atlas/lib']
>> ? ?language = c
>> ? ?include_dirs = ['/usr/include']
>>
>> Should include_dirs be '/home/skipper/atlas/include' ? ?I don't know.
>> Any pointers would be appreciated, as I wasn't able to install numpy
>> without trying to use the threaded libraries either now... ?I can
>> provide anymore output as needed. ?Build log attached.
>>
>> Cheers,
>>
>> Skipper
>>
>
> I figured the include directories in site.cfg were greedy, so I
> changed the order to
>
> library_dirs = /usr/local/lib:/usr/lib:/home/skipper/atlas/lib
> include_dirs = /home/skipper/atlas/include:/usr/local/include:/usr/include:/usr/include/suitesparse
>
> but I get the same import error in numpy.
>
> The correct include directory is listed in the build log, but I'm
> wondering why it says atlas 3.8.3 when I installed 3.8.0
>
> New build log attached.
>
> Skipper
>

I can't believe it, but I figured it out.  I uninstalled the old
atlas, then had to copy libatlas.so from the directory where I
installed atlas to /usr/lib/ and everything went to plan.  I now see
all of my cores working when trying to invert a large matrix rather
than one core getting overloaded and it taking forever.

Sorry for the noise.

Skipper


From jdh2358 at gmail.com  Fri Sep 25 11:36:44 2009
From: jdh2358 at gmail.com (John Hunter)
Date: Fri, 25 Sep 2009 10:36:44 -0500
Subject: [SciPy-dev] CS causes compiler error on solaris in cluster hierarchy
Message-ID: <88e473830909250836m686a6551o2fabf6fdc433d8e4@mail.gmail.com>

When I try and compile scipy svn HEAD on solaris for python2.4 using
gcc 3.4.3, I get the following compiler error.  Apparently CS is
reserved as a numeric constant:

scipy/cluster/src/hierarchy.c:1082: error: subscripted value is
neither array nor pointer
scipy/cluster/src/hierarchy.c:1082: error: subscripted value is
neither array nor pointer
scipy/cluster/src/hierarchy.c:1086: error: subscripted value is
neither array nor pointer
scipy/cluster/src/hierarchy.c:1091: error: subscripted value is
neither array nor pointer
scipy/cluster/src/hierarchy.c:1091: error: subscripted value is
neither array nor pointer
scipy/cluster/src/hierarchy.c:1095: error: subscripted value is
neither array nor pointer
scipy/cluster/src/hierarchy.c: At top level:

The attached patch (replace CS by CS_) allows scipy to compile on this platform



johnh at udesktop191:scipy> gcc --version
gcc (GCC) 3.4.3 (csl-sol210-3_4-branch+sol_rpath)
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

johnh at udesktop191:scipy> python -V
Python 2.4.5
johnh at udesktop191:scipy> uname -a
SunOS udesktop191 5.10 Generic_137138-09 i86pc i386 i86pc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: diff.out
Type: application/octet-stream
Size: 4702 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090925/14903179/attachment.obj>

From david at ar.media.kyoto-u.ac.jp  Sat Sep 26 06:13:31 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 26 Sep 2009 19:13:31 +0900
Subject: [SciPy-dev] CS causes compiler error on solaris in cluster
	hierarchy
In-Reply-To: <88e473830909250836m686a6551o2fabf6fdc433d8e4@mail.gmail.com>
References: <88e473830909250836m686a6551o2fabf6fdc433d8e4@mail.gmail.com>
Message-ID: <4ABDE94B.1070205@ar.media.kyoto-u.ac.jp>

Hi John,

John Hunter wrote:
> When I try and compile scipy svn HEAD on solaris for python2.4 using
> gcc 3.4.3, I get the following compiler error.  Apparently CS is
> reserved as a numeric constant:
>   

Thanks for the report - I changes the CS in the C code in scipy.cluster,
tell me if you still see problems,

cheers,

David



From d.l.goldsmith at gmail.com  Mon Sep 28 15:26:45 2009
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Mon, 28 Sep 2009 12:26:45 -0700
Subject: [SciPy-dev] New Work Flow Diagram
Message-ID: <45d1ab480909281226lc3b26e8s5a319756eb01c8c8@mail.gmail.com>

Hi!  Can someone with the appropriate privilege please add the attached to
http://mentat.za.net/numpy/?  Thanks!

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090928/2543fdca/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: newdocflow.png
Type: image/png
Size: 99578 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090928/2543fdca/attachment.png>

From enzomich at gmail.com  Wed Sep 30 10:03:34 2009
From: enzomich at gmail.com (Enzo Michelangeli)
Date: Wed, 30 Sep 2009 22:03:34 +0800
Subject: [SciPy-dev] Fast (O(n log(n)) ) implementation of Kendall Tau
Message-ID: <8752305D07F74413BC569D657AD65551@EMLT>

Dear all,

A few days ago I posted to http://projects.scipy.org/scipy/ticket/999 a 
drop-in replacement for scipy.stats.kendalltau. My code implements the 
algorithm with complexity O(n log()) described by William R. Knight in a 
paper of 1966 archived at http://www.jstor.org/pss/2282833 , whereas the 
function currently in SciPy has complexity O(n^2), which makes it unusable 
with large data sets.

One potential issue here is that my code is in part derived from a Java 
implementation contained in the Java package called "Law" 
(http://law.dsi.unimi.it/software/law-1.4-src.tar.gz ).
That code is GPL'd, but its authors 
(http://law.dsi.unimi.it/index.php?option=com_contact&catid=4&Itemid=3 ) 
have already informally confirmed to me by e-mail that they have no 
objections to the release of my Python code under a BSD-style license. Now 
my question is: exactly which steps should I take in order to clear the path 
to a possible inclusion in SciPy?

All the best,

Enzo Michelangeli 



From mdroe at stsci.edu  Wed Sep 30 10:19:18 2009
From: mdroe at stsci.edu (Michael Droettboom)
Date: Wed, 30 Sep 2009 10:19:18 -0400
Subject: [SciPy-dev] Doc edit rights
Message-ID: <4AC368E6.9080705@stsci.edu>

I've been improving the chararray functionality (see this thread [1]).  
It would be helpful to get edit rights to the documentation editor so I 
can comment on and mark some docstrings for review.

(FWIW, I think Travis O. is giving me SVN commit rights, so that's taken 
care of.)

[1] 
http://www.nabble.com/Re%3A--SciPy-dev--Deprecate-chararray--was-Plea-for-help--p25667873.html

Cheers,
Mike

-- 
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA



From scott.sinclair.za at gmail.com  Wed Sep 30 10:41:05 2009
From: scott.sinclair.za at gmail.com (Scott Sinclair)
Date: Wed, 30 Sep 2009 16:41:05 +0200
Subject: [SciPy-dev] Fast (O(n log(n)) ) implementation of Kendall Tau
In-Reply-To: <8752305D07F74413BC569D657AD65551@EMLT>
References: <8752305D07F74413BC569D657AD65551@EMLT>
Message-ID: <6a17e9ee0909300741g632441b6y1d3463f3e9f47021@mail.gmail.com>

> 2009/9/30 Enzo Michelangeli <enzomich at gmail.com>:
> A few days ago I posted to http://projects.scipy.org/scipy/ticket/999 a
> drop-in replacement for scipy.stats.kendalltau.

It was a good idea to post this note on the mailing list. I don't
think Trac is sending out notifications of new tickets, so it's likely
that no-one has spotted your contribution yet.

Cheers,
Scott


From ralf.gommers at googlemail.com  Wed Sep 30 11:05:16 2009
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Wed, 30 Sep 2009 11:05:16 -0400
Subject: [SciPy-dev] Doc edit rights
In-Reply-To: <4AC368E6.9080705@stsci.edu>
References: <4AC368E6.9080705@stsci.edu>
Message-ID: <dde7764a0909300805l33602061y9a53276cf62dbdad@mail.gmail.com>

On Wed, Sep 30, 2009 at 10:19 AM, Michael Droettboom <mdroe at stsci.edu>wrote:

> I've been improving the chararray functionality (see this thread [1]).
> It would be helpful to get edit rights to the documentation editor so I
> can comment on and mark some docstrings for review.
>

Can you register a username here:
http://docs.scipy.org/numpy/accounts/register/ and send it to the list?
Then someone can give you edit permissions.

Cheers,
Ralf


>
> (FWIW, I think Travis O. is giving me SVN commit rights, so that's taken
> care of.)
>
> [1]
>
> http://www.nabble.com/Re%3A--SciPy-dev--Deprecate-chararray--was-Plea-for-help--p25667873.html
>
> Cheers,
> Mike
>
> --
> Michael Droettboom
> Science Software Branch
> Operations and Engineering Division
> Space Telescope Science Institute
> Operated by AURA for NASA
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090930/d509bcbb/attachment.html>

From mdroe at stsci.edu  Wed Sep 30 11:09:33 2009
From: mdroe at stsci.edu (Michael Droettboom)
Date: Wed, 30 Sep 2009 11:09:33 -0400
Subject: [SciPy-dev] Doc edit rights
In-Reply-To: <dde7764a0909300805l33602061y9a53276cf62dbdad@mail.gmail.com>
References: <4AC368E6.9080705@stsci.edu>
	<dde7764a0909300805l33602061y9a53276cf62dbdad@mail.gmail.com>
Message-ID: <4AC374AD.3090201@stsci.edu>

My username is mdroe.

Ralf Gommers wrote:
>
>
> On Wed, Sep 30, 2009 at 10:19 AM, Michael Droettboom <mdroe at stsci.edu 
> <mailto:mdroe at stsci.edu>> wrote:
>
>     I've been improving the chararray functionality (see this thread [1]).
>     It would be helpful to get edit rights to the documentation editor
>     so I
>     can comment on and mark some docstrings for review.
>
>
> Can you register a username here: 
> http://docs.scipy.org/numpy/accounts/register/ and send it to the list?
> Then someone can give you edit permissions.
>
> Cheers,
> Ralf
>  
>
>
>     (FWIW, I think Travis O. is giving me SVN commit rights, so that's
>     taken
>     care of.)
>
>     [1]
>     http://www.nabble.com/Re%3A--SciPy-dev--Deprecate-chararray--was-Plea-for-help--p25667873.html
>
>     Cheers,
>     Mike
>
>     --
>     Michael Droettboom
>     Science Software Branch
>     Operations and Engineering Division
>     Space Telescope Science Institute
>     Operated by AURA for NASA
>
>     _______________________________________________
>     Scipy-dev mailing list
>     Scipy-dev at scipy.org <mailto:Scipy-dev at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   

-- 
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA



From d.l.goldsmith at gmail.com  Wed Sep 30 17:56:10 2009
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Wed, 30 Sep 2009 14:56:10 -0700
Subject: [SciPy-dev] Viewing svn status (ready to merge, merged,
	etc.) at thenumpy trac 	site
Message-ID: <45d1ab480909301456n70728ceep23a3aa7071c2be96@mail.gmail.com>

How does one do it?  (Assuming it's possible.)

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090930/d6a74806/attachment.html>

From sturla at molden.no  Wed Sep 30 18:03:49 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 01 Oct 2009 00:03:49 +0200
Subject: [SciPy-dev] Fast (O(n log(n)) ) implementation of Kendall Tau
In-Reply-To: <8752305D07F74413BC569D657AD65551@EMLT>
References: <8752305D07F74413BC569D657AD65551@EMLT>
Message-ID: <4AC3D5C5.8030107@molden.no>

Enzo Michelangeli skrev:
> Dear all,
>
> A few days ago I posted to http://projects.scipy.org/scipy/ticket/999 a 
> drop-in replacement for scipy.stats.kendalltau. My code implements the 
> algorithm with complexity O(n log()) described by William R. Knight in a 
> paper of 1966 archived at http://www.jstor.org/pss/2282833 , whereas the 
> function currently in SciPy has complexity O(n^2), which makes it unusable 
>   
There is also:

http://projects.scipy.org/scipy/ticket/893

It has a contigency table version that would be fast for large data 
sets, in theory O(n).


Sturla Molden




From sturla at molden.no  Wed Sep 30 18:09:20 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 01 Oct 2009 00:09:20 +0200
Subject: [SciPy-dev] Fast (O(n log(n)) ) implementation of Kendall Tau
In-Reply-To: <8752305D07F74413BC569D657AD65551@EMLT>
References: <8752305D07F74413BC569D657AD65551@EMLT>
Message-ID: <4AC3D710.8090505@molden.no>

Enzo Michelangeli skrev:
> Dear all,
>
> A few days ago I posted to http://projects.scipy.org/scipy/ticket/999 a 
> drop-in replacement for scipy.stats.kendalltau. 
Why do you re-implement mergesort in pure Python?

ndarrays have a sort method that can use mergesort.

Python lists has the same (timsort is mergesort on steroids).