From dwf at cs.toronto.edu  Fri May  1 00:17:09 2009
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 1 May 2009 00:17:09 -0400
Subject: [SciPy-dev] Matlab Control System Toolbox for Python / SciPy
In-Reply-To: <99214b470904292107u7702cceagcb9b1274ba8a1a45@mail.gmail.com>
References: <99214b470904292107u7702cceagcb9b1274ba8a1a45@mail.gmail.com>
Message-ID: <E6A9FA14-39E5-4458-94B2-D3305483DF1D@cs.toronto.edu>


On 30-Apr-09, at 12:07 AM, Luke wrote:

> Does anybody know of any projects that aim to implement much of the
> functionality of the control system toolbox in Matlab?  There is a
> control system toolbox for Octave, with what looks like a large amount
> of similar tools, but I haven't really found anything for Python.
>
> Is anybody out there interested in using and/or developing such a  
> set of tools?

This has come up before:

http://mail.scipy.org/pipermail/scipy-dev/2005-April/002948.html
http://mail.scipy.org/pipermail/scipy-user/2008-January/thread.html#15059
http://mail.scipy.org/pipermail/scipy-user/2008-March/thread.html#15729

Cheers,

David


From jsseabold at gmail.com  Fri May  1 00:26:29 2009
From: jsseabold at gmail.com (Skipper Seabold)
Date: Fri, 1 May 2009 00:26:29 -0400
Subject: [SciPy-dev] How to handle code contribution with GSoC students ?
In-Reply-To: <9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
References: <49F81F75.5040102@ar.media.kyoto-u.ac.jp>
	<9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
Message-ID: <c048da1c0904302126x69d03146yd32f85fb1e9e595d@mail.gmail.com>

2009/4/29 St?fan van der Walt <stefan at sun.ac.za>:
> 2009/4/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>> ? ?I was wondering about the best way to handle code by students for
>> the upcoming GSoC. The year I participated, I already had write access,
>> so the question did not come up. I think I would prefer starting with
>> code reviews, before giving them svn write access later. Do people have
>> better suggestions ?
>
> It's a good time to learn about distributed revision control :) ?So
> yes, let's have review branches.
>
> Cheers
> St?fan

Then will we need SVN write access at some point I guess.  Also
wondering if I might be given edit rights for the docs wiki to work on
scipy.stats for a start.  My registered user name is jseabold.

Skipper


From gael.varoquaux at normalesup.org  Fri May  1 03:40:48 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 1 May 2009 09:40:48 +0200
Subject: [SciPy-dev] How to handle code contribution with GSoC	students ?
In-Reply-To: <c048da1c0904302126x69d03146yd32f85fb1e9e595d@mail.gmail.com>
References: <49F81F75.5040102@ar.media.kyoto-u.ac.jp>
	<9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
	<c048da1c0904302126x69d03146yd32f85fb1e9e595d@mail.gmail.com>
Message-ID: <20090501074048.GA30225@phare.normalesup.org>

On Fri, May 01, 2009 at 12:26:29AM -0400, Skipper Seabold wrote:
> Also wondering if I might be given edit rights for the docs wiki to
> work on scipy.stats for a start.  My registered user name is jseabold.

Sure thing. Done!

Ga?l


From pgmdevlist at gmail.com  Fri May  1 04:03:03 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 1 May 2009 04:03:03 -0400
Subject: [SciPy-dev] How to handle code contribution with GSoC students ?
In-Reply-To: <9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
References: <49F81F75.5040102@ar.media.kyoto-u.ac.jp>
	<9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
Message-ID: <67E6065A-E7AF-4D80-A7A3-43290ECECDCF@gmail.com>


On Apr 29, 2009, at 10:44 AM, St?fan van der Walt wrote:

> 2009/4/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>>    I was wondering about the best way to handle code by students for
>> the upcoming GSoC. The year I participated, I already had write  
>> access,
>> so the question did not come up. I think I would prefer starting with
>> code reviews, before giving them svn write access later. Do people  
>> have
>> better suggestions ?
>
> It's a good time to learn about distributed revision control :)  So
> yes, let's have review branches.

A whole branch for modifications to only one specific submodule ?  
Isn't it a bit too much (OK, I must admit that I'm not particularly  
familiar with the details of "distributed revision cntrol"). Wouldn't  
it be simpler to have something on git/mercurial/bazaar/whatever-is- 
recommended-by-the-community-at-large ? Which would simplifying  
reviewing and avoid potential mess ?
But once again, I don't know much about versioning...

From david at ar.media.kyoto-u.ac.jp  Fri May  1 04:06:09 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 01 May 2009 17:06:09 +0900
Subject: [SciPy-dev] How to handle code contribution with GSoC students ?
In-Reply-To: <67E6065A-E7AF-4D80-A7A3-43290ECECDCF@gmail.com>
References: <49F81F75.5040102@ar.media.kyoto-u.ac.jp>	<9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
	<67E6065A-E7AF-4D80-A7A3-43290ECECDCF@gmail.com>
Message-ID: <49FAAD71.4090107@ar.media.kyoto-u.ac.jp>

Pierre GM wrote:
> On Apr 29, 2009, at 10:44 AM, St?fan van der Walt wrote:
>
>   
>> 2009/4/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>>     
>>>    I was wondering about the best way to handle code by students for
>>> the upcoming GSoC. The year I participated, I already had write  
>>> access,
>>> so the question did not come up. I think I would prefer starting with
>>> code reviews, before giving them svn write access later. Do people  
>>> have
>>> better suggestions ?
>>>       
>> It's a good time to learn about distributed revision control :)  So
>> yes, let's have review branches.
>>     
>
> A whole branch for modifications to only one specific submodule ?
>   

One possibility is to have one sandbox / student, the student would only
commit in the sandbox, and the mentor would be responsible for merging
it into the trunk. The advantage of this technique is that the mentor
can do all the work if desired (updating the branch from the trunk,
merging, etc...).

Using something like git, bzr or hg is easier for the mentor, but this
assumes the student knows how to use them. I am not sure I want to
bother students with new tools if they don't know them - getting
familiar with the scipy code, how to build, run and test it is already
enough as a barrier of entry.

cheers,

David



From aahz at pythoncraft.com  Fri May  1 07:10:55 2009
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 1 May 2009 04:10:55 -0700
Subject: [SciPy-dev] Python goes Mercurial (was Re: How to handle code
	contribution	with GSoC students?)
In-Reply-To: <49FAAD71.4090107@ar.media.kyoto-u.ac.jp>
References: <49F81F75.5040102@ar.media.kyoto-u.ac.jp>
	<9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
	<67E6065A-E7AF-4D80-A7A3-43290ECECDCF@gmail.com>
	<49FAAD71.4090107@ar.media.kyoto-u.ac.jp>
Message-ID: <20090501111055.GA13273@panix.com>

On Fri, May 01, 2009, David Cournapeau wrote:
>
> Using something like git, bzr or hg is easier for the mentor, but this
> assumes the student knows how to use them. I am not sure I want to
> bother students with new tools if they don't know them - getting
> familiar with the scipy code, how to build, run and test it is already
> enough as a barrier of entry.

AFAICT from skimming Subject: lines in the archives, it hasn't yet been
mentioned that the Python project itself is switching to Mercurial:

http://www.python.org/dev/peps/pep-0374/

(This isn't particularly relevant to GSoC, but it's something to keep an
eye on if you want to maintain tool compatibility with core Python.)
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"Typing is cheap.  Thinking is expensive."  --Roy Smith


From josef.pktd at gmail.com  Fri May  1 09:16:10 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 1 May 2009 09:16:10 -0400
Subject: [SciPy-dev] How to handle code contribution with GSoC students ?
In-Reply-To: <49FAAD71.4090107@ar.media.kyoto-u.ac.jp>
References: <49F81F75.5040102@ar.media.kyoto-u.ac.jp>
	<9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
	<67E6065A-E7AF-4D80-A7A3-43290ECECDCF@gmail.com>
	<49FAAD71.4090107@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0905010616i456ab718p56de6eaa4e8e055@mail.gmail.com>

On Fri, May 1, 2009 at 4:06 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Pierre GM wrote:
>> On Apr 29, 2009, at 10:44 AM, St?fan van der Walt wrote:
>>
>>
>>> 2009/4/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>>>
>>>> ? ?I was wondering about the best way to handle code by students for
>>>> the upcoming GSoC. The year I participated, I already had write
>>>> access,
>>>> so the question did not come up. I think I would prefer starting with
>>>> code reviews, before giving them svn write access later. Do people
>>>> have
>>>> better suggestions ?
>>>>
>>> It's a good time to learn about distributed revision control :) ?So
>>> yes, let's have review branches.
>>>
>>
>> A whole branch for modifications to only one specific submodule ?
>>
>
> One possibility is to have one sandbox / student, the student would only
> commit in the sandbox, and the mentor would be responsible for merging
> it into the trunk. The advantage of this technique is that the mentor
> can do all the work if desired (updating the branch from the trunk,
> merging, etc...).
>
> Using something like git, bzr or hg is easier for the mentor, but this
> assumes the student knows how to use them. I am not sure I want to
> bother students with new tools if they don't know them - getting
> familiar with the scipy code, how to build, run and test it is already
> enough as a barrier of entry.
>

The new stats.models code can be developed relatively isolated from
scipy, and added back in when it or parts are ready, i.e. tested and
reviewed. So, it's pretty flexible what the revision control structure
can be.

Since the current models are in launchpad under bzr revision control,
working with bzr branches would be the easiest for this.
For my own use, I'm keeping now bzr branches of scipy, and it works
reasonably well. (I haven't tried the hg svn bridge yet, but because
google code will allow mercurial repositories, using hg will become a
more attractive alternative.)

I would also find it useful, if stats.models can be released as a
standalone package during summer, so that it can be tried out by users
that don't build scipy or nipy themselves, and hopefully get more
feedback. Currently only bspline is in c, and it looks pretty broken
(I'm only getting segfaults out of it). Other than that it is a pure
python package that can be installed very easily.

Josef


From bsouthey at gmail.com  Fri May  1 10:01:46 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 01 May 2009 09:01:46 -0500
Subject: [SciPy-dev] How to handle code contribution with GSoC students ?
In-Reply-To: <1cd32cbb0905010616i456ab718p56de6eaa4e8e055@mail.gmail.com>
References: <49F81F75.5040102@ar.media.kyoto-u.ac.jp>	<9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>	<67E6065A-E7AF-4D80-A7A3-43290ECECDCF@gmail.com>	<49FAAD71.4090107@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905010616i456ab718p56de6eaa4e8e055@mail.gmail.com>
Message-ID: <49FB00CA.8060603@gmail.com>

josef.pktd at gmail.com wrote:
> On Fri, May 1, 2009 at 4:06 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>   
>> Pierre GM wrote:
>>     
>>> On Apr 29, 2009, at 10:44 AM, St?fan van der Walt wrote:
>>>
>>>
>>>       
>>>> 2009/4/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>>>>
>>>>         
>>>>>    I was wondering about the best way to handle code by students for
>>>>> the upcoming GSoC. The year I participated, I already had write
>>>>> access,
>>>>> so the question did not come up. I think I would prefer starting with
>>>>> code reviews, before giving them svn write access later. Do people
>>>>> have
>>>>> better suggestions ?
>>>>>
>>>>>           
>>>> It's a good time to learn about distributed revision control :)  So
>>>> yes, let's have review branches.
>>>>
>>>>         
>>> A whole branch for modifications to only one specific submodule ?
>>>
>>>       
>> One possibility is to have one sandbox / student, the student would only
>> commit in the sandbox, and the mentor would be responsible for merging
>> it into the trunk. The advantage of this technique is that the mentor
>> can do all the work if desired (updating the branch from the trunk,
>> merging, etc...).
>>
>> Using something like git, bzr or hg is easier for the mentor, but this
>> assumes the student knows how to use them. I am not sure I want to
>> bother students with new tools if they don't know them - getting
>> familiar with the scipy code, how to build, run and test it is already
>> enough as a barrier of entry.
>>
>>     
>
> The new stats.models code can be developed relatively isolated from
> scipy, and added back in when it or parts are ready, i.e. tested and
> reviewed. So, it's pretty flexible what the revision control structure
> can be.
>
> Since the current models are in launchpad under bzr revision control,
> working with bzr branches would be the easiest for this.
> For my own use, I'm keeping now bzr branches of scipy, and it works
> reasonably well. (I haven't tried the hg svn bridge yet, but because
> google code will allow mercurial repositories, using hg will become a
> more attractive alternative.)
>
> I would also find it useful, if stats.models can be released as a
> standalone package during summer, so that it can be tried out by users
> that don't build scipy or nipy themselves, and hopefully get more
> feedback. Currently only bspline is in c, and it looks pretty broken
> (I'm only getting segfaults out of it). Other than that it is a pure
> python package that can be installed very easily.
>
> Josef
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   
Hi,
With regards to the splines as I have not looked at this, shouldn't we 
be using the various spline functions already in Scipy?

Actually I do think that getting the students (not just Gsoc ones) to 
learn a DVCS is rather important and for the students it would 'foster' 
experience that would be useful for them to work in other projects now 
that many projects use or are moving to DVCS. Also, doesn't using a DVCS 
remove the need for a sandbox?

Bruce



From thouis at broad.mit.edu  Fri May  1 11:13:06 2009
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Fri, 1 May 2009 11:13:06 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<18239520-DA02-4895-88C9-B5B21B533011@yale.edu>
	<6c17e6f50904181347u7fabd69cja24f7f14e6dce185@mail.gmail.com> 
	<6c17e6f50904181628w6a436f73y19ba43fc5c22badd@mail.gmail.com> 
	<35AD1AEA-77B3-498D-96EC-B6A017F15BEA@yale.edu>
	<6c17e6f50904181921l1e29b1ccsecb01903b04ecca2@mail.gmail.com> 
	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com> 
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com> 
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com>
Message-ID: <6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>

2009/4/19 St?fan van der Walt <stefan at sun.ac.za>:
> 2009/4/19  <josef.pktd at gmail.com>:
>> The long term intention is to rewrite ndimage in cython, but I don't think
>> this will happen anytime soon.
>
> I wish it could, though -- it would save us so much time, and I think
> it is doable in two months by a dedicated student.  Does anyone have
> some free time to spare, or some money to pay for a student?

We may have a student this summer that can work on it as a project.

In the meantime, I rewrote ndimage.measurements in pure python/numpy.
I'm still testing its speed, but I think it will provide a good base
for a cython version.  I've simplified the code quite a bit.
Everything goes through a single labeled_reduce function, which
handles all the label/index interactions without needing any type
conversions.

The changes are available via git at:
http://broad.mit.edu/~thouis/scipy.git ndimage_measurements_rewrite

Ray Jones


From josef.pktd at gmail.com  Fri May  1 12:07:27 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 1 May 2009 12:07:27 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com>
	<6c17e6f50904181347u7fabd69cja24f7f14e6dce185@mail.gmail.com>
	<6c17e6f50904181628w6a436f73y19ba43fc5c22badd@mail.gmail.com>
	<35AD1AEA-77B3-498D-96EC-B6A017F15BEA@yale.edu>
	<6c17e6f50904181921l1e29b1ccsecb01903b04ecca2@mail.gmail.com>
	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com>
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com>
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com>
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
Message-ID: <1cd32cbb0905010907n732be082y47f9ea2fe59af0a@mail.gmail.com>

On Fri, May 1, 2009 at 11:13 AM, Thouis (Ray) Jones
<thouis at broad.mit.edu> wrote:
> 2009/4/19 St?fan van der Walt <stefan at sun.ac.za>:
>> 2009/4/19 ?<josef.pktd at gmail.com>:
>>> The long term intention is to rewrite ndimage in cython, but I don't think
>>> this will happen anytime soon.
>>
>> I wish it could, though -- it would save us so much time, and I think
>> it is doable in two months by a dedicated student. ?Does anyone have
>> some free time to spare, or some money to pay for a student?
>
> We may have a student this summer that can work on it as a project.
>
> In the meantime, I rewrote ndimage.measurements in pure python/numpy.
> I'm still testing its speed, but I think it will provide a good base
> for a cython version. ?I've simplified the code quite a bit.
> Everything goes through a single labeled_reduce function, which
> handles all the label/index interactions without needing any type
> conversions.
>
> The changes are available via git at:
> http://broad.mit.edu/~thouis/scipy.git ndimage_measurements_rewrite
>

I cloned your git repository, but how do I find your changes?
`git log` only shows some old svn.scipy commits. And I didn't find any
other branch hiding inside.

(anything tricky with git and I'm lost)

Josef


From josef.pktd at gmail.com  Fri May  1 12:17:38 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 1 May 2009 12:17:38 -0400
Subject: [SciPy-dev] How to handle code contribution with GSoC students ?
In-Reply-To: <49FB00CA.8060603@gmail.com>
References: <49F81F75.5040102@ar.media.kyoto-u.ac.jp>
	<9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
	<67E6065A-E7AF-4D80-A7A3-43290ECECDCF@gmail.com>
	<49FAAD71.4090107@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905010616i456ab718p56de6eaa4e8e055@mail.gmail.com>
	<49FB00CA.8060603@gmail.com>
Message-ID: <1cd32cbb0905010917n5744069aweb985ce98a68ee87@mail.gmail.com>

On Fri, May 1, 2009 at 10:01 AM, Bruce Southey <bsouthey at gmail.com> wrote:
> josef.pktd at gmail.com wrote:
>> On Fri, May 1, 2009 at 4:06 AM, David Cournapeau
>> <david at ar.media.kyoto-u.ac.jp> wrote:
>>
>>> Pierre GM wrote:
>>>
>>>> On Apr 29, 2009, at 10:44 AM, St?fan van der Walt wrote:
>>>>
>>>>
>>>>
>>>>> 2009/4/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>>>>>
>>>>>
>>>>>> ? ?I was wondering about the best way to handle code by students for
>>>>>> the upcoming GSoC. The year I participated, I already had write
>>>>>> access,
>>>>>> so the question did not come up. I think I would prefer starting with
>>>>>> code reviews, before giving them svn write access later. Do people
>>>>>> have
>>>>>> better suggestions ?
>>>>>>
>>>>>>
>>>>> It's a good time to learn about distributed revision control :) ?So
>>>>> yes, let's have review branches.
>>>>>
>>>>>
>>>> A whole branch for modifications to only one specific submodule ?
>>>>
>>>>
>>> One possibility is to have one sandbox / student, the student would only
>>> commit in the sandbox, and the mentor would be responsible for merging
>>> it into the trunk. The advantage of this technique is that the mentor
>>> can do all the work if desired (updating the branch from the trunk,
>>> merging, etc...).
>>>
>>> Using something like git, bzr or hg is easier for the mentor, but this
>>> assumes the student knows how to use them. I am not sure I want to
>>> bother students with new tools if they don't know them - getting
>>> familiar with the scipy code, how to build, run and test it is already
>>> enough as a barrier of entry.
>>>
>>>
>>
>> The new stats.models code can be developed relatively isolated from
>> scipy, and added back in when it or parts are ready, i.e. tested and
>> reviewed. So, it's pretty flexible what the revision control structure
>> can be.
>>
>> Since the current models are in launchpad under bzr revision control,
>> working with bzr branches would be the easiest for this.
>> For my own use, I'm keeping now bzr branches of scipy, and it works
>> reasonably well. (I haven't tried the hg svn bridge yet, but because
>> google code will allow mercurial repositories, using hg will become a
>> more attractive alternative.)
>>
>> I would also find it useful, if stats.models can be released as a
>> standalone package during summer, so that it can be tried out by users
>> that don't build scipy or nipy themselves, and hopefully get more
>> feedback. Currently only bspline is in c, and it looks pretty broken
>> (I'm only getting segfaults out of it). Other than that it is a pure
>> python package that can be installed very easily.
>>
>> Josef
>> _______________________________________________
>> Scipy-dev mailing list
>> Scipy-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
> Hi,
> With regards to the splines as I have not looked at this, shouldn't we
> be using the various spline functions already in Scipy?

Yes, I was also thinking that this would be the right approach, but I
haven't looked at it in details.

The models.bspline seem to be specifically designed to be used for
generalized additive models. I'm not familiar enough with bsplines to
tell whether it is easy to replace them with scipy.interpolate.

my current notes on models, especially the problem parts of it, are here

http://bazaar.launchpad.net/~josef-pktd/%2Bjunk/nipy_stats_models2/annotate/head%3A/nipy/fixes/scipy/stats/models/commentJP.txt

Josef


>
> Actually I do think that getting the students (not just Gsoc ones) to
> learn a DVCS is rather important and for the students it would 'foster'
> experience that would be useful for them to work in other projects now
> that many projects use or are moving to DVCS. Also, doesn't using a DVCS
> remove the need for a sandbox?
>
> Bruce
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From strawman at astraw.com  Fri May  1 12:18:08 2009
From: strawman at astraw.com (Andrew Straw)
Date: Fri, 01 May 2009 09:18:08 -0700
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
 if index is a list?
In-Reply-To: <1cd32cbb0905010907n732be082y47f9ea2fe59af0a@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com>	<6c17e6f50904181347u7fabd69cja24f7f14e6dce185@mail.gmail.com>	<6c17e6f50904181628w6a436f73y19ba43fc5c22badd@mail.gmail.com>	<35AD1AEA-77B3-498D-96EC-B6A017F15BEA@yale.edu>	<6c17e6f50904181921l1e29b1ccsecb01903b04ecca2@mail.gmail.com>	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com>	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com>	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com>	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
	<1cd32cbb0905010907n732be082y47f9ea2fe59af0a@mail.gmail.com>
Message-ID: <49FB20C0.7060100@astraw.com>

josef.pktd at gmail.com wrote:
> On Fri, May 1, 2009 at 11:13 AM, Thouis (Ray) Jones

>> In the meantime, I rewrote ndimage.measurements in pure python/numpy.

Very cool.

>> The changes are available via git at:
>> http://broad.mit.edu/~thouis/scipy.git ndimage_measurements_rewrite
>>
> 
> I cloned your git repository, but how do I find your changes?

git checkout -b ndimage_measurements_rewrite \
  origin/ndimage_measurements_rewrite

> `git log` only shows some old svn.scipy commits. And I didn't find any
> other branch hiding inside.

Check out "gitk --all" or at least "git branch -a"



From josef.pktd at gmail.com  Fri May  1 12:34:40 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 1 May 2009 12:34:40 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <49FB20C0.7060100@astraw.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com>
	<35AD1AEA-77B3-498D-96EC-B6A017F15BEA@yale.edu>
	<6c17e6f50904181921l1e29b1ccsecb01903b04ecca2@mail.gmail.com>
	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com>
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com>
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com>
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
	<1cd32cbb0905010907n732be082y47f9ea2fe59af0a@mail.gmail.com>
	<49FB20C0.7060100@astraw.com>
Message-ID: <1cd32cbb0905010934x58d6c2c8n3dab8ad497d3d4af@mail.gmail.com>

On Fri, May 1, 2009 at 12:18 PM, Andrew Straw <strawman at astraw.com> wrote:
> josef.pktd at gmail.com wrote:
>> On Fri, May 1, 2009 at 11:13 AM, Thouis (Ray) Jones
>
>>> In the meantime, I rewrote ndimage.measurements in pure python/numpy.
>
> Very cool.
>
>>> The changes are available via git at:
>>> http://broad.mit.edu/~thouis/scipy.git ndimage_measurements_rewrite
>>>
>>
>> I cloned your git repository, but how do I find your changes?
>
> git checkout -b ndimage_measurements_rewrite \
> ?origin/ndimage_measurements_rewrite
>
>> `git log` only shows some old svn.scipy commits. And I didn't find any
>> other branch hiding inside.
>
> Check out "gitk --all" or at least "git branch -a"


Thanks, this worked.

I'm curious about the speed and scalability to a large number of
unique labels, since that's the main advantage of current
implementation.

Josef


From stefan at sun.ac.za  Fri May  1 13:21:13 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 1 May 2009 19:21:13 +0200
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<6c17e6f50904181347u7fabd69cja24f7f14e6dce185@mail.gmail.com> 
	<6c17e6f50904181628w6a436f73y19ba43fc5c22badd@mail.gmail.com> 
	<35AD1AEA-77B3-498D-96EC-B6A017F15BEA@yale.edu>
	<6c17e6f50904181921l1e29b1ccsecb01903b04ecca2@mail.gmail.com> 
	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com> 
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com> 
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com> 
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
Message-ID: <9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com>

Hi Ray

2009/5/1 Thouis (Ray) Jones <thouis at broad.mit.edu>:
> 2009/4/19 St?fan van der Walt <stefan at sun.ac.za>:
> We may have a student this summer that can work on it as a project.

Fantastic, that'd be wonderful.

> In the meantime, I rewrote ndimage.measurements in pure python/numpy.

What a great patch!

For the record, here are the git instructions to view the patch:

git fetch http://broad.mit.edu/~thouis/scipy.git ndimage_measurements_rewrite
git log -p ..FETCH_HEAD

> Everything goes through a single labeled_reduce function, which
> handles all the label/index interactions without needing any type
> conversions.

Looks like _labeled_reduce and _labeled_reduce2 can possibly be
combined, saving some duplication.  I see there is also a
labeled_reduce, but I haven't looked at that in too much detail.


I think this piece of code proves that it is well worth converting
ndimage to Python/Cython.  The code is *much* more readable, which
means better code review, more bug fixes, and happier maintainers.

Thanks for your effort, Ray!

Cheers
St?fan


From thouis at broad.mit.edu  Fri May  1 14:02:09 2009
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Fri, 1 May 2009 14:02:09 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<6c17e6f50904181628w6a436f73y19ba43fc5c22badd@mail.gmail.com> 
	<35AD1AEA-77B3-498D-96EC-B6A017F15BEA@yale.edu>
	<6c17e6f50904181921l1e29b1ccsecb01903b04ecca2@mail.gmail.com> 
	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com> 
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com> 
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com> 
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com> 
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com>
Message-ID: <6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com>

2009/5/1 St?fan van der Walt <stefan at sun.ac.za>:
> 2009/5/1 Thouis (Ray) Jones <thouis at broad.mit.edu>:
>> Everything goes through a single labeled_reduce function, which
>> handles all the label/index interactions without needing any type
>> conversions.
>
> Looks like _labeled_reduce and _labeled_reduce2 can possibly be
> combined, saving some duplication. ?I see there is also a
> labeled_reduce, but I haven't looked at that in too much detail.

Yes, the _labeled_reduce and _labeled_reduce2 could be combined.  I
wrote them separately because I thought they might end up more
different than they did, and because they might end up different when
translated to Cython.

labeled_reduce is intended as a user-exposed function.

The new code's about 5 times slower than the current tree, *not*
taking into account time spent in the _labeled_reduce function(s).
Most of that time is the call to argsort() and the subsequent
reorderings in labeled_reduce.  There might be some tricks to avoid
those.

The code is 10-20 times slower altogether.

Ray Jones


From stefan at sun.ac.za  Fri May  1 14:07:15 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 1 May 2009 20:07:15 +0200
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<35AD1AEA-77B3-498D-96EC-B6A017F15BEA@yale.edu>
	<6c17e6f50904181921l1e29b1ccsecb01903b04ecca2@mail.gmail.com> 
	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com> 
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com> 
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com> 
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com> 
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com> 
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com>
Message-ID: <9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com>

2009/5/1 Thouis (Ray) Jones <thouis at broad.mit.edu>:
> The new code's about 5 times slower than the current tree, *not*
> taking into account time spent in the _labeled_reduce function(s).
> Most of that time is the call to argsort() and the subsequent
> reorderings in labeled_reduce. ?There might be some tricks to avoid
> those.
>
> The code is 10-20 times slower altogether.

I don't think we should read too much into that.  After we've
optimised the code using Cython, we may very well be cose to where we
started.

Either way, in my opinion a 5 times slow-down in exchange for a
maintainable library is well worth it.

Regards
St?fan


From twood at doc.ic.ac.uk  Fri May  1 13:55:15 2009
From: twood at doc.ic.ac.uk (Tobias Wood)
Date: Fri, 01 May 2009 18:55:15 +0100
Subject: [SciPy-dev] erf, erfc and ndtr on OS X
Message-ID: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>

Hi,
I'm not sure whether this should go to the dev or users list, so sorry if  
it is in the wrong place. I'm using the lastest SVN Numpy/Scipy checkouts  
with Python 2.6 on Mac OS 10.5. I was trying to use the ndtr function in  
scipy.special and discovered that it returned 0.5 for all input values  
except +/-Inf. A little bit of searching pulled up the following bug  
ticket:
http://projects.scipy.org/scipy/ticket/899
which hasn't been closed yet (I can't find any mention of it on the  
mailing lists though). I'd just like to report that I did manage to get  
erf and ndtr to return correct values by forcing the compiler to use the  
floating point constant values defined between #ifdef/endif UNK in both  
ndtr.c and const.c in scipy/special/cephes. I'm not totally sure which  
branch of the #ifdefs my compiler was going down before - is there an easy  
way to check? - but the constants it was getting were clearly wrong. SQRTH  
(1 over root 2) was defined as 0 previously.
What is the likelihood of this being fixed in the SVN repository? I am  
happy to help in constructing a patch but don't understand the  
relationship between Cephes and Scipy.
Thanks,
Toby Wood



From stefan at sun.ac.za  Fri May  1 15:12:53 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 1 May 2009 21:12:53 +0200
Subject: [SciPy-dev] erf, erfc and ndtr on OS X
In-Reply-To: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>
References: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>
Message-ID: <9457e7c80905011212m78640892l34a2787660b7cf63@mail.gmail.com>

Hi Tobias

2009/5/1 Tobias Wood <twood at doc.ic.ac.uk>:
> I'm not sure whether this should go to the dev or users list, so sorry if
> it is in the wrong place. I'm using the lastest SVN Numpy/Scipy checkouts
> with Python 2.6 on Mac OS 10.5. I was trying to use the ndtr function in
> scipy.special and discovered that it returned 0.5 for all input values
> except +/-Inf. A little bit of searching pulled up the following bug
> ticket:
> http://projects.scipy.org/scipy/ticket/899

> What is the likelihood of this being fixed in the SVN repository?

Very good :)

Pauli and I had a look, and it seems that for some reason the
Endianess of your system was detected incorrectly.  This could be
because of a Python header built on a different system, I'm not sure.
Either way, we removed the Endianess detection, so you should no
longer experience problems.

Cheers
St?fan


From josef.pktd at gmail.com  Fri May  1 16:11:53 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 1 May 2009 16:11:53 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com>
	<6c17e6f50904181921l1e29b1ccsecb01903b04ecca2@mail.gmail.com>
	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com>
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com>
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com>
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com>
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com>
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com>
Message-ID: <1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com>

2009/5/1 St?fan van der Walt <stefan at sun.ac.za>:
> 2009/5/1 Thouis (Ray) Jones <thouis at broad.mit.edu>:
>> The new code's about 5 times slower than the current tree, *not*
>> taking into account time spent in the _labeled_reduce function(s).
>> Most of that time is the call to argsort() and the subsequent
>> reorderings in labeled_reduce. ?There might be some tricks to avoid
>> those.
>>
>> The code is 10-20 times slower altogether.
>
> I don't think we should read too much into that. ?After we've
> optimised the code using Cython, we may very well be cose to where we
> started.
>
> Either way, in my opinion a 5 times slow-down in exchange for a
> maintainable library is well worth it.
>

After a recent comment by Anne, I looked at the weights option in np.bincount.
It is as fast to calculate group/label means with np.bincount as with
the current ndimage and it scales the same, but it works only for all
indices and not for min, max.
weights can be calculated for any element wise function.
group labels can be anything that np.unique1d can handle, string
labels take twice as long

Josef

Here are some timeit results:

number of observations 100000, factors 10000
number of runs 100
group/label mean
np.bincount  0.0382392734272
ndimage.mean 0.0387848287981
group/label mean and variance
np.bincount mv 0.0417609612395
ndimage     mv 0.0501527009718
>>>
number of observations 100000, factors 1000
number of runs 100
np.bincount  0.0356244774126
ndimage.mean 0.0349412732624
np.bincount mv 0.0448310357881
ndimage     mv 0.041015631875
>>>
number of observations 100000, factors 100
number of runs 100
np.bincount  0.0296716174821
ndimage.mean 0.0300010113017
np.bincount mv 0.0360344764488
ndimage     mv 0.0352608138744

number of observations 1000000, factors 100000
number of runs 100
np.bincount  0.967738271205
ndimage.mean 0.83807955201
np.bincount mv 0.96804377372
ndimage     mv 1.08760137392
-------------- next part --------------

import numpy as np
from scipy import ndimage

def groupmeanbin(factors, values):
    '''uses np.bincount, assumes factors/labels are integers
    '''
    #n = len(factors)
    ix,rind = np.unique1d(factors, return_inverse=1)
    gcount = np.bincount(rind)
    gmean = np.bincount(rind, weights=values)/ (1.0*gcount)
    return gmean


def labelmean_str(factors, values):
    # works also for string labels in ys, but requires 1D
    # from mailing list scipy-user 2009-02-11
    unil, unilinv = np.unique1d(factors, return_index=False, return_inverse=True)
    labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=np.arange(len(unil)+1)))
    return labelmeans

def groupstatsbin(factors, values):
    '''uses np.bincount, assumes factors/labels are integers
    '''
    n = len(factors)
    ix,rind = np.unique1d(factors, return_inverse=1)
    gcount = np.bincount(rind)
    gmean = np.bincount(rind, weights=values)/ (1.0*gcount)
    meanarr = gmean[rind]
    withinvar = np.bincount(rind, weights=(values-meanarr)**2) / (1.0*gcount)
    #withinvararr = withinvar[rind]
    #return gcount, gmean , meanarr, withinvar, withinvararr
    return gmean, withinvar


def labelstats_str(factors, values):
    # works also for string labels in ys, but requires 1D
    # from mailing list scipy-user 2009-02-11
    unil, unilinv = np.unique1d(factors, return_index=False, return_inverse=True)
    labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=np.arange(len(unil)+1)))
    labelvars = np.array(ndimage.variance(values, labels=unilinv, index=np.arange(len(unil)+1)))
    return labelmeans, labelvars


nobs = 100000
nfact = 100
factors = np.random.randint(nfact,size=nobs)#.astype('S4')
values = np.random.randn(nobs)

if __name__ == "__main__":




    #print groupstatsbin(factors, values)
    #print labelmean_str(factors, values)

    setupstr = '''from __main__ import np, groupmeanbin, labelmean_str, \
                groupstatsbin, labelstats_str, factors, values'''

    from timeit import Timer

    n = 100
    t1 = Timer(setup=setupstr, stmt='groupmeanbin(factors, values)')
    t2 = Timer(setup=setupstr, stmt='labelmean_str(factors, values)')
    t3 = Timer(setup=setupstr, stmt='groupstatsbin(factors, values)')
    t4 = Timer(setup=setupstr, stmt='labelstats_str(factors, values)')

    print 'number of observations %d, factors %d' % (nobs, nfact)
    print 'number of runs %d' % n
    print 'np.bincount ', t1.timeit(n)/float(n)
    print 'ndimage.mean', t2.timeit(n)/float(n)
    print 'np.bincount mv', t3.timeit(n)/float(n)
    print 'ndimage     mv', t4.timeit(n)/float(n)

From thouis at broad.mit.edu  Fri May  1 16:37:42 2009
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Fri, 1 May 2009 16:37:42 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com> 
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com> 
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com> 
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com> 
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com> 
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com> 
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com> 
	<1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com>
Message-ID: <6c17e6f50905011337v944bc57n13ee8cfec170bfc0@mail.gmail.com>

On Fri, May 1, 2009 at 16:11,  <josef.pktd at gmail.com> wrote:
> After a recent comment by Anne, I looked at the weights option in np.bincount.
> It is as fast to calculate group/label means with np.bincount as with
> the current ndimage and it scales the same, but it works only for all
> indices and not for min, max.
> weights can be calculated for any element wise function.
> group labels can be anything that np.unique1d can handle, string
> labels take twice as long

I think you can get min/max by sorting by (label, value) and then
using diff on the sorted labels to find the indices where a label
changes.

Ray Jones


From charlesr.harris at gmail.com  Fri May  1 19:27:33 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 1 May 2009 17:27:33 -0600
Subject: [SciPy-dev] erf, erfc and ndtr on OS X
In-Reply-To: <9457e7c80905011212m78640892l34a2787660b7cf63@mail.gmail.com>
References: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>
	<9457e7c80905011212m78640892l34a2787660b7cf63@mail.gmail.com>
Message-ID: <e06186140905011627j4f6ee383pdf6bbd61f6dad2e5@mail.gmail.com>

2009/5/1 St?fan van der Walt <stefan at sun.ac.za>

> Hi Tobias
>
> 2009/5/1 Tobias Wood <twood at doc.ic.ac.uk>:
> > I'm not sure whether this should go to the dev or users list, so sorry if
> > it is in the wrong place. I'm using the lastest SVN Numpy/Scipy checkouts
> > with Python 2.6 on Mac OS 10.5. I was trying to use the ndtr function in
> > scipy.special and discovered that it returned 0.5 for all input values
> > except +/-Inf. A little bit of searching pulled up the following bug
> > ticket:
> > http://projects.scipy.org/scipy/ticket/899
>
> > What is the likelihood of this being fixed in the SVN repository?
>
> Very good :)
>
> Pauli and I had a look, and it seems that for some reason the
> Endianess of your system was detected incorrectly.  This could be
> because of a Python header built on a different system, I'm not sure.
> Either way, we removed the Endianess detection, so you should no
> longer experience problems.
>

There was a problem with the endianess in the python header for some
versions of python2.6 on Apple. I believe David got it the python folks to
fix it in the most recent versions. Numpy now detects incompatible endianess
assumptions.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090501/3a464770/attachment.html>

From josef.pktd at gmail.com  Sat May  2 02:14:39 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 2 May 2009 02:14:39 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <6c17e6f50905011337v944bc57n13ee8cfec170bfc0@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com>
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com>
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com>
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com>
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com>
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com>
	<1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com>
	<6c17e6f50905011337v944bc57n13ee8cfec170bfc0@mail.gmail.com>
Message-ID: <1cd32cbb0905012314w4fb49f44me5192626ebd8e028@mail.gmail.com>

On Fri, May 1, 2009 at 4:37 PM, Thouis (Ray) Jones <thouis at broad.mit.edu> wrote:
> On Fri, May 1, 2009 at 16:11, ?<josef.pktd at gmail.com> wrote:
>> After a recent comment by Anne, I looked at the weights option in np.bincount.
>> It is as fast to calculate group/label means with np.bincount as with
>> the current ndimage and it scales the same, but it works only for all
>> indices and not for min, max.
>> weights can be calculated for any element wise function.
>> group labels can be anything that np.unique1d can handle, string
>> labels take twice as long
>
> I think you can get min/max by sorting by (label, value) and then
> using diff on the sorted labels to find the indices where a label
> changes.

yes this works, it takes about twice as long for  min and max than the
current ndimage implementation, see below.

In my previous script, I didn't test against ndimage and needed two
corrections, index for ndimage had one element too many,
and ndimage.variance uses the unbiased estimator for the variance,
while I worked with the biased estimator. I added a ddof option to the
bincount version.

I had looked at this when I started to rewrite and generalize ANOVA in
stats and for manipulating "factors" as in R. Before I found that I
can (ab)use np.bincount, ndimage.measurement was by far the fastest
way of doing this for large arrays. There are currently similar uses
in stats._support that have a very slow implementation and could be
made faster this way.
ndimage.measurement is still more flexible than np.bincount, because
it allows to specify index separately, it works for subsets of labels,
and for supersets (indices that don't actually occur in the label
array).

One thing that neither bincount nor the current implementation allows
but would be very useful for stats, is an axis argument for 2d arrays,
e.g.  groupmeans(values,..., axis=0) similar to np.mean, np.var

Josef

    nobs = 100
    nfact = 5
    factors = np.random.randint(nfact,size=nobs)
    values = np.random.randn(nobs)

    # find minimum and maximum per factor
    sortind = np.lexsort((values, factors))
    fs = factors[sortind]
    vs = values[sortind]
    fsd = np.hstack((np.inf,np.diff(fs),np.inf))
    gmin = vs[fsd[:-1] != 0]
    gmax = vs[fsd[1:] != 0]
    print gmin
    print ndimage.minimum(values, labels=factors, index=np.arange(5))
    print gmax
    print ndimage.maximum(values, labels=factors, index=np.arange(5))


From stefan at sun.ac.za  Sat May  2 04:08:50 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 2 May 2009 10:08:50 +0200
Subject: [SciPy-dev] erf, erfc and ndtr on OS X
In-Reply-To: <e06186140905011627j4f6ee383pdf6bbd61f6dad2e5@mail.gmail.com>
References: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>
	<9457e7c80905011212m78640892l34a2787660b7cf63@mail.gmail.com> 
	<e06186140905011627j4f6ee383pdf6bbd61f6dad2e5@mail.gmail.com>
Message-ID: <9457e7c80905020108h1a482802raffca7f9b51d55cc@mail.gmail.com>

2009/5/2 Charles R Harris <charlesr.harris at gmail.com>:
> There was a problem with the endianess in the python header for some
> versions of python2.6 on Apple. I believe David got it the python folks to
> fix it in the most recent versions. Numpy now detects incompatible endianess
> assumptions.

Thanks for the reminder.

In this case, I think it's OK to use the Cephes-defined constants?  If
not, I'll go back and use the NumPy define instead.

Cheers
St?fan


From thouis at broad.mit.edu  Sat May  2 11:03:07 2009
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Sat, 2 May 2009 11:03:07 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <1cd32cbb0905012314w4fb49f44me5192626ebd8e028@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com> 
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com> 
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com> 
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com> 
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com> 
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com> 
	<1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com> 
	<6c17e6f50905011337v944bc57n13ee8cfec170bfc0@mail.gmail.com> 
	<1cd32cbb0905012314w4fb49f44me5192626ebd8e028@mail.gmail.com>
Message-ID: <6c17e6f50905020803nd21445fnd4ac05bdedf7ffc8@mail.gmail.com>

On Sat, May 2, 2009 at 02:14,  <josef.pktd at gmail.com> wrote:
> One thing that neither bincount nor the current implementation allows
> but would be very useful for stats, is an axis argument for 2d arrays,
> e.g. ?groupmeans(values,..., axis=0) similar to np.mean, np.var

You could get that by replacing the label_matrix with [label,
axis_position], and using the unique1d approach, then using the sparse
function to convert the result to a matrix LxN (L = labels, N = axis
size).

Ray


From matthew.brett at gmail.com  Sun May  3 00:59:28 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 2 May 2009 21:59:28 -0700
Subject: [SciPy-dev] Please hammer scipy.io.matlab routines
Message-ID: <1e2af89e0905022159t352b7bacj487540ab3b320997@mail.gmail.com>

Hi,

We are getting very close to a scipy point release, and I have just
been putting together fixes for the matlab routines.

I would be very grateful if anyone who was having problems, or is not
having problems, could update to latest scipy SVN and try on the worst
possible files to see if it is working and at reasonable speed.
Please let me know if not; thanks to all those who have posted files,
please keep doing so.

Thanks a lot,

Matthew


From pav at iki.fi  Sun May  3 09:56:08 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 3 May 2009 13:56:08 +0000 (UTC)
Subject: [SciPy-dev] RFR: New iterative sparse solver: LGMRES
Message-ID: <gtk7po$3c5$2@ger.gmane.org>

Hi,

I'd like to submit a new iterative (pure-Python) sparse solver to Scipy:

	http://codereview.appspot.com/53096/show
	http://github.com/pv/scipy-work/commits/lgmres

This is a part of a new Newton-Krylov solver in the scipy.optimize.nonlin 
rewrite. (For those interested, it's almost done,
http://github.com/pv/scipy-work/commits/ticket/791-optimize-nonlin-rewrite
but still needs tests for the line searches since I needed to refactor
them out of optimize.py.)

-- 
Pauli Virtanen



From stefan at sun.ac.za  Sun May  3 12:25:30 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 3 May 2009 18:25:30 +0200
Subject: [SciPy-dev] RFR: New iterative sparse solver: LGMRES
In-Reply-To: <gtk7po$3c5$2@ger.gmane.org>
References: <gtk7po$3c5$2@ger.gmane.org>
Message-ID: <9457e7c80905030925y479d6d7cl55ac181d6787cb38@mail.gmail.com>

2009/5/3 Pauli Virtanen <pav at iki.fi>:
> I'd like to submit a new iterative (pure-Python) sparse solver to Scipy:
>
> ? ? ? ?http://codereview.appspot.com/53096/show
> ? ? ? ?http://github.com/pv/scipy-work/commits/lgmres

Awesome -- a reallly nice patch!  +1 from me.

Cheers
St?fan


From wnbell at gmail.com  Sun May  3 15:10:49 2009
From: wnbell at gmail.com (Nathan Bell)
Date: Sun, 3 May 2009 15:10:49 -0400
Subject: [SciPy-dev] RFR: New iterative sparse solver: LGMRES
In-Reply-To: <9457e7c80905030925y479d6d7cl55ac181d6787cb38@mail.gmail.com>
References: <gtk7po$3c5$2@ger.gmane.org>
	<9457e7c80905030925y479d6d7cl55ac181d6787cb38@mail.gmail.com>
Message-ID: <d05265cb0905031210j3b1f6970n8f0f0ff9c60ec41e@mail.gmail.com>

2009/5/3 St?fan van der Walt <stefan at sun.ac.za>:
>
> Awesome -- a reallly nice patch! ?+1 from me.
>

Looks good to me too, +1

I would suggest including the rationale/explanation for LGMRES [1]
into the lgmres() docstring.  Perhaps a couple sentences before
"Parameters" and then the remaining details in a "Notes" section.

[1] http://codereview.appspot.com/53096/show

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From pav at iki.fi  Sun May  3 18:24:40 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 3 May 2009 22:24:40 +0000 (UTC)
Subject: [SciPy-dev] RFR: refactor line searches out of optimize.py
Message-ID: <gtl5j8$t7g$1@ger.gmane.org>

Hi,

I'd like to refactor line searches in optimize.py a bit, so that they can 
be easily used for solving nonlinear equations. Code review here:

	http://codereview.appspot.com/52096
	http://github.com/pv/scipy-work/commits/linesearch-refactor

This is a part of the scipy.optimize.nonlin rewrite:

	http://github.com/pv/scipy-work/commits/ticket/791-optimize-nonlin-rewrite

-- 
Pauli Virtanen



From skorpio11 at gmail.com  Mon May  4 14:00:59 2009
From: skorpio11 at gmail.com (Leon Adams)
Date: Mon, 4 May 2009 14:00:59 -0400
Subject: [SciPy-dev] Status: optimize.fmin_slsqp
Message-ID: <9e3090c0905041100q9f4a7e5p6b68bcdd6f0a8955@mail.gmail.com>

Hi,

I was wondering what the status is of the Slsqp extension to the optimize
package. I am currently in need of the ability to place an equality
constraint on some of my input variables, but the available documentation on
slsqp seems insufficient.

My Scenario:

I have an objective fn: Obfn(x1,x2,x3,x4,x5,x6) that I would like to place
the additional constrain of
x1 + x2 + x3 = 1
x4 + x5 + x6 = 1

If there is a good usage example I can be pointed to, it would be
appreciated

Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090504/67838112/attachment.html>

From thouis at broad.mit.edu  Mon May  4 16:00:16 2009
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Mon, 4 May 2009 16:00:16 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<038EAA37-600A-4249-B8FC-25F5A3DB54D2@yale.edu>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com> 
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com> 
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com> 
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com> 
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com> 
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com> 
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com> 
	<1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com>
Message-ID: <6c17e6f50905041300q25dca2c0m93ec9f6cd74f6391@mail.gmail.com>

On Fri, May 1, 2009 at 16:11,  <josef.pktd at gmail.com> wrote:
> After a recent comment by Anne, I looked at the weights option in np.bincount.
> It is as fast to calculate group/label means with np.bincount as with
> the current ndimage and it scales the same, but it works only for all
> indices and not for min, max.
> weights can be calculated for any element wise function.
> group labels can be anything that np.unique1d can handle, string
> labels take twice as long

I've attached a rewrite of mean() using this method.  It's still about
5-10 times slower than the existing ndimage code.  Perhaps I'm missing
some obvious optimization.

Ray Jones
-------------- next part --------------
A non-text attachment was scrubbed...
Name: newmean.py
Type: application/octet-stream
Size: 1581 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090504/8590aae5/attachment.obj>

From josef.pktd at gmail.com  Mon May  4 16:43:04 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 4 May 2009 16:43:04 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <6c17e6f50905041300q25dca2c0m93ec9f6cd74f6391@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com>
	<6c17e6f50904190731l1af4fe8dsd8570cbdbc16c602@mail.gmail.com>
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com>
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com>
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com>
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com>
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com>
	<1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com>
	<6c17e6f50905041300q25dca2c0m93ec9f6cd74f6391@mail.gmail.com>
Message-ID: <1cd32cbb0905041343u433fcf49o68ef89928a0a71f0@mail.gmail.com>

On Mon, May 4, 2009 at 4:00 PM, Thouis (Ray) Jones <thouis at broad.mit.edu> wrote:
> On Fri, May 1, 2009 at 16:11, ?<josef.pktd at gmail.com> wrote:
>> After a recent comment by Anne, I looked at the weights option in np.bincount.
>> It is as fast to calculate group/label means with np.bincount as with
>> the current ndimage and it scales the same, but it works only for all
>> indices and not for min, max.
>> weights can be calculated for any element wise function.
>> group labels can be anything that np.unique1d can handle, string
>> labels take twice as long
>
> I've attached a rewrite of mean() using this method. ?It's still about
> 5-10 times slower than the existing ndimage code. ?Perhaps I'm missing
> some obvious optimization.
>
> Ray Jones

The use case, I wrote my application for, also allowed for labels that
are strings. The return inverse of unique1d is very flexible since it
can handle many different types including structured arrays, but for
numeric labels, I just found out, that it is a large speed penalty. If
the labels are already integers or can be easily converted to
integers, then not using unique1d to create the integer labels should
be much faster.

In my timing comparison, I had compared both bincount and ndimage.mean
including unique1d.
bincount should be able to work directly with integer labels, which is
also the restriction with the current (broken) ndimage.measurement.
However, I haven't tried yet what the best way is to use bincount for
float or fixed decimal without using unique1d.

Josef


From arokem at berkeley.edu  Mon May  4 16:54:30 2009
From: arokem at berkeley.edu (Ariel Rokem)
Date: Mon, 4 May 2009 13:54:30 -0700
Subject: [SciPy-dev] Please hammer scipy.io.matlab routines
In-Reply-To: <1e2af89e0905022159t352b7bacj487540ab3b320997@mail.gmail.com>
References: <1e2af89e0905022159t352b7bacj487540ab3b320997@mail.gmail.com>
Message-ID: <43958ee60905041354k12989f24g2e59eb291051147d@mail.gmail.com>

Hi Matthew,

I ran this on a big file with loads of fMRI time-series in it.

Here's the comparison I made.

*Old(er) version:

*In [1]: import scipy

In [2]: scipy.__version__
Out[2]: '0.8.0.dev5606'

In [3]: import time

In [4]: import scipy.io as sio

In [5]: a = time.clock(); sio.loadmat('tSeries1.mat'); b = time.clock()

...

In [7]: b-a
Out[7]: 27.68

*New(er) version:
*

In [1]: import scipy

In [2]: scipy.__version__
Out[2]: '0.8.0.dev5697'

In [3]: import time

In [4]: import scipy.io as sio

In [5]: a = time.clock(); sio.loadmat('tSeries1.mat'); b = time.clock()

...

In [7]: b-a
Out[7]: 15.370000000000005

So yes - faster by a factor of 2 (What are the units of "time.clock"? It's
definitely not seconds and I am pretty sure that it's not minutes...)

Cheers,

Ariel

P.S. Where should I post the file?



On Sat, May 2, 2009 at 9:59 PM, Matthew Brett <matthew.brett at gmail.com>wrote:

> Hi,
>
> We are getting very close to a scipy point release, and I have just
> been putting together fixes for the matlab routines.
>
> I would be very grateful if anyone who was having problems, or is not
> having problems, could update to latest scipy SVN and try on the worst
> possible files to see if it is working and at reasonable speed.
> Please let me know if not; thanks to all those who have posted files,
> please keep doing so.
>
> Thanks a lot,
>
> Matthew
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>



-- 
Ariel Rokem
Helen Wills Neuroscience Institute
University of California, Berkeley
http://argentum.ucbso.berkeley.edu/ariel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090504/38433b7c/attachment.html>

From thouis at broad.mit.edu  Mon May  4 17:11:11 2009
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Mon, 4 May 2009 17:11:11 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <1cd32cbb0905041343u433fcf49o68ef89928a0a71f0@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<1cd32cbb0904190924i57a72727gaa22567125c23c78@mail.gmail.com> 
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com> 
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com> 
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com> 
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com> 
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com> 
	<1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com> 
	<6c17e6f50905041300q25dca2c0m93ec9f6cd74f6391@mail.gmail.com> 
	<1cd32cbb0905041343u433fcf49o68ef89928a0a71f0@mail.gmail.com>
Message-ID: <6c17e6f50905041411wdc519f7o5812618beab518f3@mail.gmail.com>

On Mon, May 4, 2009 at 16:43,  <josef.pktd at gmail.com> wrote:
> The use case, I wrote my application for, also allowed for labels that
> are strings. The return inverse of unique1d is very flexible since it
> can handle many different types including structured arrays, but for
> numeric labels, I just found out, that it is a large speed penalty. If
> the labels are already integers or can be easily converted to
> integers, then not using unique1d to create the integer labels should
> be much faster.

Using bincount probably requires calling unique1d (or some equivalent)
for this usage, since otherwise the call to bincount could create a
very large output (2^32 or even 2^64 bins).

Ray Jones


From twood at doc.ic.ac.uk  Mon May  4 18:03:40 2009
From: twood at doc.ic.ac.uk (Tobias Wood)
Date: Mon, 04 May 2009 23:03:40 +0100
Subject: [SciPy-dev] erf, erfc and ndtr on OS X
In-Reply-To: <9457e7c80905020108h1a482802raffca7f9b51d55cc@mail.gmail.com>
References: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>
	<9457e7c80905011212m78640892l34a2787660b7cf63@mail.gmail.com>
	<e06186140905011627j4f6ee383pdf6bbd61f6dad2e5@mail.gmail.com>
	<9457e7c80905020108h1a482802raffca7f9b51d55cc@mail.gmail.com>
Message-ID: <op.utfd8ewai8f5qd@zaphod.lan>

On Sat, 02 May 2009 09:08:50 +0100, St?fan van der Walt <stefan at sun.ac.za>  
wrote:

> 2009/5/2 Charles R Harris <charlesr.harris at gmail.com>:
>> There was a problem with the endianess in the python header for some
>> versions of python2.6 on Apple. I believe David got it the python folks  
>> to
>> fix it in the most recent versions. Numpy now detects incompatible  
>> endianess
>> assumptions.
>
> Thanks for the reminder.
>
> In this case, I think it's OK to use the Cephes-defined constants?  If
> not, I'll go back and use the NumPy define instead.
>
> Cheers
> St?fan

It would certainly appear that way. I downloaded the most recent Python  
(Python 2.6.2 (r262:71600, Apr 16 2009, 09:17:39)), recompiled Scipy, and  
ndtr(), erf() etc. have started working correctly without any modification  
 from the SVN checkout. Thanks for the heads up Charles.

On a side-note, a simple 'setup.py install' produces huge amounts of  
compiler warnings along the following lines on OS X:
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:1:1:,
                  from  
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/arrayobject.h:14,
                  from scipy/special/_cephesmodule.c:12:
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:21:1:  
warning: "NPY_SIZEOF_SHORT" redefined
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:1:1:  
warning: this is the location of the previous definition
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:22:1:  
warning: "NPY_SIZEOF_INT" redefined
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:2:1:  
warning: this is the location of the previous definition
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:23:1:  
warning: "NPY_SIZEOF_LONG" redefined
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:3:1:  
warning: this is the location of the previous definition
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:24:1:  
warning: "NPY_SIZEOF_FLOAT" redefined
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:4:1:  
warning: this is the location of the previous definition
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:25:1:  
warning: "NPY_SIZEOF_DOUBLE" redefined
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:5:1:  
warning: this is the location of the previous definition
when compiling Scipy (but not Numpy). Anyone know how I could fix this?
Thanks,
Toby


From charlesr.harris at gmail.com  Mon May  4 19:49:38 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 May 2009 17:49:38 -0600
Subject: [SciPy-dev] erf, erfc and ndtr on OS X
In-Reply-To: <op.utfd8ewai8f5qd@zaphod.lan>
References: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>
	<9457e7c80905011212m78640892l34a2787660b7cf63@mail.gmail.com>
	<e06186140905011627j4f6ee383pdf6bbd61f6dad2e5@mail.gmail.com>
	<9457e7c80905020108h1a482802raffca7f9b51d55cc@mail.gmail.com>
	<op.utfd8ewai8f5qd@zaphod.lan>
Message-ID: <e06186140905041649v25120c30y280fe5e71ecd57c6@mail.gmail.com>

On Mon, May 4, 2009 at 4:03 PM, Tobias Wood <twood at doc.ic.ac.uk> wrote:

> On Sat, 02 May 2009 09:08:50 +0100, St?fan van der Walt <stefan at sun.ac.za>
> wrote:
>
> > 2009/5/2 Charles R Harris <charlesr.harris at gmail.com>:
> >> There was a problem with the endianess in the python header for some
> >> versions of python2.6 on Apple. I believe David got it the python folks
> >> to
> >> fix it in the most recent versions. Numpy now detects incompatible
> >> endianess
> >> assumptions.
> >
> > Thanks for the reminder.
> >
> > In this case, I think it's OK to use the Cephes-defined constants?  If
> > not, I'll go back and use the NumPy define instead.
> >
> > Cheers
> > St?fan
>
> It would certainly appear that way. I downloaded the most recent Python
> (Python 2.6.2 (r262:71600, Apr 16 2009, 09:17:39)), recompiled Scipy, and
> ndtr(), erf() etc. have started working correctly without any modification
>  from the SVN checkout. Thanks for the heads up Charles.
>
> On a side-note, a simple 'setup.py install' produces huge amounts of
> compiler warnings along the following lines on OS X:
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:1:1:,
>                  from
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/arrayobject.h:14,
>                  from scipy/special/_cephesmodule.c:12:
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:21:1:
> warning: "NPY_SIZEOF_SHORT" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:1:1:
> warning: this is the location of the previous definition
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:22:1:
> warning: "NPY_SIZEOF_INT" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:2:1:
> warning: this is the location of the previous definition
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:23:1:
> warning: "NPY_SIZEOF_LONG" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:3:1:
> warning: this is the location of the previous definition
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:24:1:
> warning: "NPY_SIZEOF_FLOAT" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:4:1:
> warning: this is the location of the previous definition
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:25:1:
> warning: "NPY_SIZEOF_DOUBLE" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:5:1:
> warning: this is the location of the previous definition
> when compiling Scipy (but not Numpy). Anyone know how I could fix this?
> Thanks,
> Toby
> ___
>

Is that from the latest svn? Or does it happen with the last official
release. There have been some changes to the build in svn.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090504/5bba272b/attachment.html>

From d_l_goldsmith at yahoo.com  Mon May  4 23:40:14 2009
From: d_l_goldsmith at yahoo.com (David Goldsmith)
Date: Mon, 4 May 2009 20:40:14 -0700 (PDT)
Subject: [SciPy-dev] NumPy Doc edit rights request
Message-ID: <465516.2449.qm@web52101.mail.re2.yahoo.com>


To whom it may concern: As per the instructions on the "Front Page," I'm emailing my NumPy Doc Editor registered username - dgoldsmith - and requesting edit rights.  Thanks!

David Goldsmith
Olympia, WA


      


From charlesr.harris at gmail.com  Tue May  5 00:45:45 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 4 May 2009 22:45:45 -0600
Subject: [SciPy-dev] erf, erfc and ndtr on OS X
In-Reply-To: <op.utfd8ewai8f5qd@zaphod.lan>
References: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>
	<9457e7c80905011212m78640892l34a2787660b7cf63@mail.gmail.com>
	<e06186140905011627j4f6ee383pdf6bbd61f6dad2e5@mail.gmail.com>
	<9457e7c80905020108h1a482802raffca7f9b51d55cc@mail.gmail.com>
	<op.utfd8ewai8f5qd@zaphod.lan>
Message-ID: <e06186140905042145h3b19276q348737489a64a4b3@mail.gmail.com>

On Mon, May 4, 2009 at 4:03 PM, Tobias Wood <twood at doc.ic.ac.uk> wrote:

> On Sat, 02 May 2009 09:08:50 +0100, St?fan van der Walt <stefan at sun.ac.za>
> wrote:
>
> > 2009/5/2 Charles R Harris <charlesr.harris at gmail.com>:
> >> There was a problem with the endianess in the python header for some
> >> versions of python2.6 on Apple. I believe David got it the python folks
> >> to
> >> fix it in the most recent versions. Numpy now detects incompatible
> >> endianess
> >> assumptions.
> >
> > Thanks for the reminder.
> >
> > In this case, I think it's OK to use the Cephes-defined constants?  If
> > not, I'll go back and use the NumPy define instead.
> >
> > Cheers
> > St?fan
>
> It would certainly appear that way. I downloaded the most recent Python
> (Python 2.6.2 (r262:71600, Apr 16 2009, 09:17:39)), recompiled Scipy, and
> ndtr(), erf() etc. have started working correctly without any modification
>  from the SVN checkout. Thanks for the heads up Charles.
>
> On a side-note, a simple 'setup.py install' produces huge amounts of
> compiler warnings along the following lines on OS X:
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:1:1:,
>                  from
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/arrayobject.h:14,
>                  from scipy/special/_cephesmodule.c:12:
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:21:1:
> warning: "NPY_SIZEOF_SHORT" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:1:1:
> warning: this is the location of the previous definition
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:22:1:
> warning: "NPY_SIZEOF_INT" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:2:1:
> warning: this is the location of the previous definition
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:23:1:
> warning: "NPY_SIZEOF_LONG" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:3:1:
> warning: this is the location of the previous definition
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:24:1:
> warning: "NPY_SIZEOF_FLOAT" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:4:1:
> warning: this is the location of the previous definition
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:25:1:
> warning: "NPY_SIZEOF_DOUBLE" redefined
>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/numpy/numpyconfig.h:5:1:
> warning: this is the location of the previous definition
> when compiling Scipy (but not Numpy). Anyone know how I could fix this?
> Thanks,
> Toby
> _
>

I removed my build directory before the build and these warnings went away.
Seems to be a left over file thing.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090504/421386eb/attachment.html>

From cournape at gmail.com  Tue May  5 01:07:39 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 5 May 2009 14:07:39 +0900
Subject: [SciPy-dev] erf, erfc and ndtr on OS X
In-Reply-To: <e06186140905042145h3b19276q348737489a64a4b3@mail.gmail.com>
References: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>
	<9457e7c80905011212m78640892l34a2787660b7cf63@mail.gmail.com>
	<e06186140905011627j4f6ee383pdf6bbd61f6dad2e5@mail.gmail.com>
	<9457e7c80905020108h1a482802raffca7f9b51d55cc@mail.gmail.com>
	<op.utfd8ewai8f5qd@zaphod.lan>
	<e06186140905042145h3b19276q348737489a64a4b3@mail.gmail.com>
Message-ID: <5b8d13220905042207l458ca204l7362158802b2fe54@mail.gmail.com>

On Tue, May 5, 2009 at 1:45 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> I removed my build directory before the build and these warnings went away.
> Seems to be a left over file thing.

Yes it is. I think there should be a way to regenerate those files
correctly, even though removing the build directory is always a good
idea with distutils. I will look into it,

David


From pgmdevlist at gmail.com  Tue May  5 03:12:35 2009
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 5 May 2009 03:12:35 -0400
Subject: [SciPy-dev] How to handle code contribution with GSoC students ?
In-Reply-To: <1cd32cbb0905010616i456ab718p56de6eaa4e8e055@mail.gmail.com>
References: <49F81F75.5040102@ar.media.kyoto-u.ac.jp>
	<9457e7c80904290744y1bbb3750scaa1e53bf8d4cf28@mail.gmail.com>
	<67E6065A-E7AF-4D80-A7A3-43290ECECDCF@gmail.com>
	<49FAAD71.4090107@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905010616i456ab718p56de6eaa4e8e055@mail.gmail.com>
Message-ID: <ABB7CDDA-4864-4DF2-86E6-59891CD634EA@gmail.com>

All,
I hope you won't mind my bumping this thread.
So, what's the news ? Any consensus yet ? SVN on scipy.org w/ one  
sandbox per student ? hg on external servers ? Yet another option ?

Cheers
P.
</bump>


On May 1, 2009, at 9:16 AM, josef.pktd at gmail.com wrote:

> On Fri, May 1, 2009 at 4:06 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>> Pierre GM wrote:
>>> On Apr 29, 2009, at 10:44 AM, St?fan van der Walt wrote:
>>>
>>>
>>>> 2009/4/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>>>>
>>>>>    I was wondering about the best way to handle code by students  
>>>>> for
>>>>> the upcoming GSoC. The year I participated, I already had write
>>>>> access,
>>>>> so the question did not come up. I think I would prefer starting  
>>>>> with
>>>>> code reviews, before giving them svn write access later. Do people
>>>>> have
>>>>> better suggestions ?
>>>>>
>>>> It's a good time to learn about distributed revision control :)  So
>>>> yes, let's have review branches.
>>>>
>>>
>>> A whole branch for modifications to only one specific submodule ?
>>>
>>
>> One possibility is to have one sandbox / student, the student would  
>> only
>> commit in the sandbox, and the mentor would be responsible for  
>> merging
>> it into the trunk. The advantage of this technique is that the mentor
>> can do all the work if desired (updating the branch from the trunk,
>> merging, etc...).
>>
>> Using something like git, bzr or hg is easier for the mentor, but  
>> this
>> assumes the student knows how to use them. I am not sure I want to
>> bother students with new tools if they don't know them - getting
>> familiar with the scipy code, how to build, run and test it is  
>> already
>> enough as a barrier of entry.
>>
>
> The new stats.models code can be developed relatively isolated from
> scipy, and added back in when it or parts are ready, i.e. tested and
> reviewed. So, it's pretty flexible what the revision control structure
> can be.
>
> Since the current models are in launchpad under bzr revision control,
> working with bzr branches would be the easiest for this.
> For my own use, I'm keeping now bzr branches of scipy, and it works
> reasonably well. (I haven't tried the hg svn bridge yet, but because
> google code will allow mercurial repositories, using hg will become a
> more attractive alternative.)
>
> I would also find it useful, if stats.models can be released as a
> standalone package during summer, so that it can be tried out by users
> that don't build scipy or nipy themselves, and hopefully get more
> feedback. Currently only bspline is in c, and it looks pretty broken
> (I'm only getting segfaults out of it). Other than that it is a pure
> python package that can be installed very easily.
>
> Josef
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev



From stefan at sun.ac.za  Tue May  5 05:30:26 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 5 May 2009 11:30:26 +0200
Subject: [SciPy-dev] NumPy Doc edit rights request
In-Reply-To: <465516.2449.qm@web52101.mail.re2.yahoo.com>
References: <465516.2449.qm@web52101.mail.re2.yahoo.com>
Message-ID: <9457e7c80905050230w39067f6fna40041f6593e40fb@mail.gmail.com>

2009/5/5 David Goldsmith <d_l_goldsmith at yahoo.com>:
>
> To whom it may concern: As per the instructions on the "Front Page," I'm emailing my NumPy Doc Editor registered username - dgoldsmith - and requesting edit rights. ?Thanks!

Thanks, David!  I've updated your permissions.

Regards
St?fan


From twood at doc.ic.ac.uk  Tue May  5 05:34:13 2009
From: twood at doc.ic.ac.uk (Tobias Wood)
Date: Tue, 05 May 2009 10:34:13 +0100
Subject: [SciPy-dev] erf, erfc and ndtr on OS X
In-Reply-To: <5b8d13220905042207l458ca204l7362158802b2fe54@mail.gmail.com>
References: <op.us9iqdyhi8f5qd@dyn1215-13.wlan.ic.ac.uk>
	<9457e7c80905011212m78640892l34a2787660b7cf63@mail.gmail.com>
	<e06186140905011627j4f6ee383pdf6bbd61f6dad2e5@mail.gmail.com>
	<9457e7c80905020108h1a482802raffca7f9b51d55cc@mail.gmail.com>
	<op.utfd8ewai8f5qd@zaphod.lan>
	<e06186140905042145h3b19276q348737489a64a4b3@mail.gmail.com>
	<5b8d13220905042207l458ca204l7362158802b2fe54@mail.gmail.com>
Message-ID: <op.utf97bexi8f5qd@dyn1213-88.wlan.ic.ac.uk>

On Tue, 05 May 2009 06:07:39 +0100, David Cournapeau <cournape at gmail.com>  
wrote:

> On Tue, May 5, 2009 at 1:45 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>>
>> I removed my build directory before the build and these warnings went  
>> away.
>> Seems to be a left over file thing.
>
> Yes it is. I think there should be a way to regenerate those files
> correctly, even though removing the build directory is always a good
> idea with distutils. I will look into it,
>
> David

Thanks, I removed the build directory and recompiled, those warnings have  
gone away.

Toby


From robfalck at gmail.com  Tue May  5 08:06:44 2009
From: robfalck at gmail.com (Rob Falck)
Date: Tue, 5 May 2009 08:06:44 -0400
Subject: [SciPy-dev] Status: optimize.fmin_slsqp
In-Reply-To: <9e3090c0905041100q9f4a7e5p6b68bcdd6f0a8955@mail.gmail.com>
References: <9e3090c0905041100q9f4a7e5p6b68bcdd6f0a8955@mail.gmail.com>
Message-ID: <ccb1b5850905050506h12eda63coc4c783459842b079@mail.gmail.com>

The argument you're interested in is eqcons, which is a list of functions
whose value should be zero in a successfully optimized problem.

See the fmin_slsqp test script at
http://projects.scipy.org/scipy/attachment/ticket/570/slsqp_test.py

In particular, your case will be something like this

x = fmin_slsqp(testfunc,[-1.0,1.0], args = (-1.0,), eqcons = [lambda x, y:
x[0]-x[1] ], iprint = 2, full_output = 1)

In your case, eqcons will be:
[lambda x, y: x[0]+x[1]+x[2]-1, lambda x, y: x[3]+x[4]+x[5]-1 ]

Alternatively, you can write a Python function that returns those two values
in a sequence as pass it to fmin_slsqp as  f_eqcons.



On Mon, May 4, 2009 at 2:00 PM, Leon Adams <skorpio11 at gmail.com> wrote:

> Hi,
>
> I was wondering what the status is of the Slsqp extension to the optimize
> package. I am currently in need of the ability to place an equality
> constraint on some of my input variables, but the available documentation on
> slsqp seems insufficient.
>
> My Scenario:
>
> I have an objective fn: Obfn(x1,x2,x3,x4,x5,x6) that I would like to place
> the additional constrain of
> x1 + x2 + x3 = 1
> x4 + x5 + x6 = 1
>
> If there is a good usage example I can be pointed to, it would be
> appreciated
>
> Thanks in advance.
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>


-- 
- Rob Falck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090505/a1489f42/attachment.html>

From josef.pktd at gmail.com  Tue May  5 11:33:34 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 5 May 2009 11:33:34 -0400
Subject: [SciPy-dev] Status: optimize.fmin_slsqp
In-Reply-To: <ccb1b5850905050506h12eda63coc4c783459842b079@mail.gmail.com>
References: <9e3090c0905041100q9f4a7e5p6b68bcdd6f0a8955@mail.gmail.com>
	<ccb1b5850905050506h12eda63coc4c783459842b079@mail.gmail.com>
Message-ID: <1cd32cbb0905050833x35bbb640r2c62d43753fa4993@mail.gmail.com>

On Tue, May 5, 2009 at 8:06 AM, Rob Falck <robfalck at gmail.com> wrote:
> The argument you're interested in is eqcons, which is a list of functions
> whose value should be zero in a successfully optimized problem.
>
> See the fmin_slsqp test script at
> http://projects.scipy.org/scipy/attachment/ticket/570/slsqp_test.py
>
> In particular, your case will be something like this
>
> x = fmin_slsqp(testfunc,[-1.0,1.0], args = (-1.0,), eqcons = [lambda x, y:
> x[0]-x[1] ], iprint = 2, full_output = 1)
>
> In your case, eqcons will be:
> [lambda x, y: x[0]+x[1]+x[2]-1, lambda x, y: x[3]+x[4]+x[5]-1 ]
>
> Alternatively, you can write a Python function that returns those two values
> in a sequence as pass it to fmin_slsqp as? f_eqcons.
>

Thanks for the pointer. This hasn't made it's way into the docs and
help file yet.

Josef


From josef.pktd at gmail.com  Tue May  5 14:54:15 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 5 May 2009 14:54:15 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <6c17e6f50905041411wdc519f7o5812618beab518f3@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com>
	<9457e7c80904191640rf05528cve0edfd8228a0bdb7@mail.gmail.com>
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com>
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com>
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com>
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com>
	<1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com>
	<6c17e6f50905041300q25dca2c0m93ec9f6cd74f6391@mail.gmail.com>
	<1cd32cbb0905041343u433fcf49o68ef89928a0a71f0@mail.gmail.com>
	<6c17e6f50905041411wdc519f7o5812618beab518f3@mail.gmail.com>
Message-ID: <1cd32cbb0905051154t689ad8cdw898ce5e97b9872fc@mail.gmail.com>

On Mon, May 4, 2009 at 5:11 PM, Thouis (Ray) Jones <thouis at broad.mit.edu> wrote:
> On Mon, May 4, 2009 at 16:43, ?<josef.pktd at gmail.com> wrote:
>> The use case, I wrote my application for, also allowed for labels that
>> are strings. The return inverse of unique1d is very flexible since it
>> can handle many different types including structured arrays, but for
>> numeric labels, I just found out, that it is a large speed penalty. If
>> the labels are already integers or can be easily converted to
>> integers, then not using unique1d to create the integer labels should
>> be much faster.
>
> Using bincount probably requires calling unique1d (or some equivalent)
> for this usage, since otherwise the call to bincount could create a
> very large output (2^32 or even 2^64 bins).
>
> Ray Jones

In my version, I started to use a conditional call to unique1d, if the
type of the labels is not int (to handle float and strings). This
could be extended to a very large max(labels).  If the label type is
int then, I just use bincount without unique1d.

I tried it with 50% unused numbers, and the speed of bincount is
comparable to ndimage, labels are generated with
factors = (100+2*np.random.randint(nfact,size=nobs))
#.astype('S6')#.astype(float)
I tried with up to 4 million observations and 10000 unique labels.

I'm starting to get confused with the many different versions, that I
have now for this, the two typical examples are below. ndimage looks
very picky which arrays it accepts, every once in a while I get a not
implemented type exception or even a segfault.

I also attach my full timing and test script.

Josef


def labelmean_ndi(factors, values, index=None):
    # works also for string labels in ys, but requires 1D
    # from mailing list scipy-user 2009-02-11
    unilinv = np.asanyarray(factors)

    if not np.issubdtype(unilinv.dtype, int):
        unil, unilinv = np.unique1d(factors, return_inverse=1)
        if index is None:
            index = np.arange(len(unil)) #unil
    elif index is None:
        index = np.arange(np.max(unilinv)+ 1)

    labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=index))
    return labelmeans

def groupstatsbin(factors, values, ddof=0, stat='mvnx', drop=True):
    '''uses np.bincount, assumes factors/labels are integers,
    create labels with unique1d return_inverse if string labels
    '''
    #n = len(factors)

    rind = np.asanyarray(factors)
    #note: bincount uses complete list of integers, maybe range(max(factor)+1)
    if not np.issubdtype(rind.dtype, int):
        ix, rind = np.unique1d(factors, return_inverse=1)
##    else:
##        rind = factors
    #print rind.shape
    res = []
    if 'c' in stat or 'm' in stat or 'v' in stat:
        gcount = np.bincount(rind)
        if drop:
            keepidx = np.nonzero(gcount)
        if 'c' in stat:
            if drop:
                res.append(gcount[keepidx])
            else:
                res.append(gcount)

    if 'm' in stat or 'v' in stat:
        gmean = np.bincount(rind, weights=values)/ (1.0*gcount)
        if np.max(np.abs(gmean)) > 1e6:
            # improve numerical precision if means are large
            # (to "cheat" on NIST anova test examples)
            meanarr = gmean[rind]
            gmean += np.bincount(rind, weights=(values-meanarr)) / (1.0*gcount)
        if 'm' in stat:
            if drop:
                res.append(gmean[keepidx])
            else:
                res.append(gmean)

    if 'v' in stat:
        meanarr = gmean[rind]
        withinvar = np.bincount(rind, weights=(values-meanarr)**2) /
(1.0*gcount-ddof)
        if drop:
            res.append(withinvar[keepidx])
        else:
            res.append(withinvar)

    if 'n' in stat or 'x' in stat:
        #calculate min, max per factor
        sortind = np.lexsort((values, rind))
        fs = rind[sortind]
        vs = values[sortind]
        fsd = np.hstack((np.inf,np.diff(fs),np.inf))
    if 'n' in stat:
        #minimum
        res.append(vs[fsd[:-1] != 0])
    if 'x' in stat:
        #maximum
        res.append(vs[fsd[1:] != 0])
    return res
-------------- next part --------------

import numpy as np
from numpy.testing import assert_array_almost_equal
from scipy import ndimage

def groupmeanbin(factors, values):
    '''uses np.bincount, assumes factors/labels are integers
    '''
    #n = len(factors)
    ix,rind = np.unique1d(factors, return_inverse=1)
    gcount = np.bincount(rind)
    gmean = np.bincount(rind, weights=values)/ (1.0*gcount)
    return gmean


def labelmean_str(factors, values):
    # works also for string labels in ys, but requires 1D
    # from mailing list scipy-user 2009-02-11
    # check mistake: returns one element to much
    unil,unilinv = np.unique1d(factors, return_index=False, return_inverse=True)

    #labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=np.arange(len(unil)+1)))
    labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=np.arange(len(unil))))
    return labelmeans

def labelmean_dig(factors, values):
    # works also for string labels in ys, but requires 1D
    # from mailing list scipy-user 2009-02-11
    # check mistake: returns one element to much
    #unil = np.unique1d(factors, return_index=False, return_inverse=False)
    unil = np.unique1d(factors)
    unilinv = np.digitize(factors, unil).astype('int64')
    #print unilinv.shape
    #print unilinv.dtype
    #labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=np.arange(len(unil)+1)))
    labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=np.arange(len(unil))))
    return labelmeans

def labelmean_ndi(factors, values, index=None):
    # works also for string labels in ys, but requires 1D
    # from mailing list scipy-user 2009-02-11
    # check mistake: returns one element to much
    #unil,unilinv = np.unique1d(factors, return_index=False, return_inverse=True)
    unilinv = np.asanyarray(factors)

    #note: bincount uses complete list of integers, maybe range(max(factor)+1)
    if not np.issubdtype(unilinv.dtype, int):
        unil, unilinv = np.unique1d(factors, return_inverse=1)
        if index is None:
            index = np.arange(len(unil)) #unil
    elif index is None:
        index = np.arange(np.max(unilinv)+ 1)

    labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=index))
    return labelmeans

def bincount2d(factors):
    # array check copied from np.histogramdd
    try:
        # Sample is an ND-array.
        N, D = factors.shape
        sample = factors
    except (AttributeError, ValueError):
        # Sample is a sequence of 1D arrays.
        sample = np.atleast_2d(factors).T
        N, D = sample.shape
    tmp = np.ascontiguousarray(sample)
    factarr = tmp.view([('',tmp.dtype)]*tmp.shape[-1])
    uni, rind = np.unique1d(factarr, return_inverse=1)
    return np.bincount(rind), uni.view(tmp.dtype).reshape(-1,D)
    #or return unique values as structured array:
    #return np.bincount(rind), uni


def groupstatsbin(factors, values, ddof=0, stat='mvnx', drop=True):
    '''uses np.bincount, assumes factors/labels are integers,
    create labels with unique1d return_inverse if string labels
    '''
    #n = len(factors)

    rind = np.asanyarray(factors)
    #note: bincount uses complete list of integers, maybe range(max(factor)+1)
    if not np.issubdtype(rind.dtype, int):
        ix, rind = np.unique1d(factors, return_inverse=1)
##    else:
##        rind = factors
    #print rind.shape
    res = []
    if 'c' in stat or 'm' in stat or 'v' in stat:
        gcount = np.bincount(rind)
        if drop:
            keepidx = np.nonzero(gcount)
        if 'c' in stat:
            if drop:
                res.append(gcount[keepidx])
            else:
                res.append(gcount)

    if 'm' in stat or 'v' in stat:
        #gcount = np.bincount(rind)
        gmean = np.bincount(rind, weights=values)/ (1.0*gcount)
        if np.max(np.abs(gmean)) > 1e6:
            # improve numerical precision if means are large
            meanarr = gmean[rind]
            gmean += np.bincount(rind, weights=(values-meanarr)) / (1.0*gcount)
        if 'm' in stat:
            if drop:
                res.append(gmean[keepidx])
            else:
                res.append(gmean)

    if 'v' in stat:
        meanarr = gmean[rind]
        withinvar = np.bincount(rind, weights=(values-meanarr)**2) / (1.0*gcount-ddof)
        if drop:
            res.append(withinvar[keepidx])
        else:
            res.append(withinvar)

    #withinvararr = withinvar[rind]
    #return gcount, gmean , meanarr, withinvar, withinvararr

    if 'n' in stat or 'x' in stat:
        #calculate min, max per factor
        sortind = np.lexsort((values, rind))
        fs = rind[sortind]
        vs = values[sortind]
        fsd = np.hstack((np.inf,np.diff(fs),np.inf))
    if 'n' in stat:
        #minimum
        res.append(vs[fsd[:-1] != 0])
    if 'x' in stat:
        #maximum
        res.append(vs[fsd[1:] != 0])
    return res


def labelstats_str(factors, values, stat='mvnx'):
    # works also for string labels in ys, but requires 1D
    # from mailing list scipy-user 2009-02-11
    unil, unilinv = np.unique1d(factors, return_index=False, return_inverse=True)
    res = []
    if 'm' in stat:
        labelmeans = np.array(ndimage.mean(values, labels=unilinv,
                                index=np.arange(len(unil))))
        res.append(labelmeans)
    if 'v' in stat:
        labelvars = np.array(ndimage.variance(values, labels=unilinv,
                                index=np.arange(len(unil))))
        res.append(labelvars)
    if 'n' in stat:
        labelmin = np.array(ndimage.minimum(values, labels=unilinv,
                                index=np.arange(len(unil))))
        res.append(labelmin)
    if 'x' in stat:
        labelmax = np.array(ndimage.maximum(values, labels=unilinv,
                                index=np.arange(len(unil))))
        res.append(labelmax)
    return res


nobs = 40000
nfact = 1000
factors = (100+2*np.random.randint(nfact,size=nobs))#.astype('S6')#.astype(float)#copy()#
#factors = np.random.randint(nfact,size=nobs)
#factors = np.array(factors.tolist(),int)
print factors.dtype
values = np.random.randn(nobs)
indexs = 100 + 2*np.arange(nfact)

def test_compare(nrun):
    nobs = 100
    nfact = 5
    for i in range(nrun):
        #factors = (100*np.random.randint(nfact,size=nobs)).astype('S2')
        factors = (np.random.randint(nfact,size=nobs))#.astype('S2')
        values = 1e0 + 1.e1 * np.random.randn(nobs)
        m,v,n,x = groupstatsbin(factors, values, ddof=1)
        #print factors.shape, values.shape, m.shape, v.shape, n.shape, x.shape
        assert_array_almost_equal(groupstatsbin(factors, values, ddof=1), \
            labelstats_str(factors, values), decimal=10, err_msg = \
            repr(np.asarray(groupstatsbin(factors, values, ddof=1, stat='mvnx')) - \
            np.asarray(labelstats_str(factors, values))) + str(i))

if __name__ == "__main__":

    #print groupstatsbin(factors, values)
    #print labelmean_str(factors, values)

    setupstr = '''from __main__ import np, groupmeanbin, labelmean_str, \
               labelmean_dig, labelmean_ndi, groupstatsbin, labelstats_str, \
               factors, values, indexs'''

    from timeit import Timer

    n = 10
    t1 = Timer(setup=setupstr, stmt='groupmeanbin(factors, values)')
    t2 = Timer(setup=setupstr, stmt='labelmean_str(factors, values)')
    t2a = Timer(setup=setupstr, stmt='labelmean_dig(factors, values)')
    t2b = Timer(setup=setupstr+'; factors=factors.tolist()', \
                                stmt='labelmean_ndi(factors, values,indexs)')
    t2c = Timer(setup=setupstr+'; factors2=np.array(factors.tolist()); from scipy import ndimage', \
                                stmt='np.array(ndimage.mean(values, labels=factors2, index=indexs))')
    t3a = Timer(setup=setupstr, stmt="groupstatsbin(factors, values, stat='m')")
    t3 = Timer(setup=setupstr, stmt="groupstatsbin(factors, values, stat='mv')")
    t4 = Timer(setup=setupstr, stmt="labelstats_str(factors, values, stat='mv')")
    t5 = Timer(setup=setupstr, stmt="groupstatsbin(factors, values, stat='nx')")
    t6 = Timer(setup=setupstr, stmt="labelstats_str(factors, values, stat='nx')")

    print 'number of observations %d, factors %d' % (nobs, nfact)
    print 'number of runs %d' % n
    print 'np.bincount ', t1.timeit(n)/float(n)
    print 'ndimage.mean', t2.timeit(n)/float(n)
    #digitize is slow, drop it
    #print 'ndimage dig ', t2a.timeit(n)/float(n)
    print 'ndimage ndi ', t2b.timeit(n)/float(n)
    if not np.issubdtype(factors.dtype, str):
        print 'ndimage ndipl', t2c.timeit(n)/float(n)
    print 'np.bincount m', t3a.timeit(n)/float(n)
    print 'np.bincount mv', t3.timeit(n)/float(n)
    print 'ndimage     mv', t4.timeit(n)/float(n)
    print 'np.bincount nx', t5.timeit(n)/float(n)
    print 'ndimage     nx', t6.timeit(n)/float(n)

    test_compare(10)

    debug = 0
    if debug:
        nobs = 100
        nfact = 5
        for i in range(1):
            factors = 10+ np.random.randint(nfact,size=nobs)#.astype('S2')
            values = 1e6 + np.random.randn(nobs)
            print np.array(groupstatsbin(factors, values,ddof=1)) - \
                           np.array(labelstats_str(factors, values))#[:,:-1]

        sortind = np.lexsort((values, factors))
        fs = factors[sortind]
        vs = values[sortind]
        #fsd = np.inf*np.ones(len(fs)+)
        fsd = np.hstack((np.inf,np.diff(fs),np.inf)) #fs[:-1]-fs[1:]
        gmin = vs[fsd[:-1] != 0]
        gmax = vs[fsd[1:] != 0]
        print gmin
        print ndimage.minimum(values, labels=factors.tolist(), index=np.arange(15))
        print gmax
        print ndimage.maximum(values, labels=factors.tolist(), index=np.arange(15))

        m,v,n,x = groupstatsbin(factors, values,ddof=1)
        (values[factors==0]-1e6).mean() - ((values[factors==0]).mean()-1e6)
        m1=values[factors==0].mean();
        m1+(values[factors==0]-m1).mean() - (m[0])
        m1=1e6;
        m1+(values[factors==0]-m1).mean() - (m[0])
        m1=0;
        m1+(values[factors==0]-m1).mean() - (m[0])

    debug2 = 0
    if debug2:
        print bincount2d(factors)
        print np.bincount(factors)
        factors = np.random.randint(5,size=(100,3))
        print bincount2d(factors)
        nbins = 2
        nbinsp1 = nbins + 1
        digitizedarr1 = (factors - factors % nbinsp1) / nbinsp1
        digitizedarr = np.mod(factors,nbins)
        #assert np.all(digitizedarr1==digitizedarr)
        cb, bb = bincount2d(digitizedarr1)
        ch, eh = np.histogramdd(digitizedarr1,nbins)
        print cb
        print ch
        assert np.all(cb==ch.ravel())
        print bb

From skorpio11 at gmail.com  Wed May  6 00:21:45 2009
From: skorpio11 at gmail.com (Leon Adams)
Date: Wed, 6 May 2009 00:21:45 -0400
Subject: [SciPy-dev] Status: optimize.fmin_slsqp
In-Reply-To: <1cd32cbb0905050833x35bbb640r2c62d43753fa4993@mail.gmail.com>
References: <9e3090c0905041100q9f4a7e5p6b68bcdd6f0a8955@mail.gmail.com>
	<ccb1b5850905050506h12eda63coc4c783459842b079@mail.gmail.com>
	<1cd32cbb0905050833x35bbb640r2c62d43753fa4993@mail.gmail.com>
Message-ID: <9e3090c0905052121m36756842ydef0208055a1e861@mail.gmail.com>

Thanks for the response and thanks for the work on slsqp as well

Leon

On Tue, May 5, 2009 at 11:33 AM, <josef.pktd at gmail.com> wrote:

> On Tue, May 5, 2009 at 8:06 AM, Rob Falck <robfalck at gmail.com> wrote:
> > The argument you're interested in is eqcons, which is a list of functions
> > whose value should be zero in a successfully optimized problem.
> >
> > See the fmin_slsqp test script at
> > http://projects.scipy.org/scipy/attachment/ticket/570/slsqp_test.py
> >
> > In particular, your case will be something like this
> >
> > x = fmin_slsqp(testfunc,[-1.0,1.0], args = (-1.0,), eqcons = [lambda x,
> y:
> > x[0]-x[1] ], iprint = 2, full_output = 1)
> >
> > In your case, eqcons will be:
> > [lambda x, y: x[0]+x[1]+x[2]-1, lambda x, y: x[3]+x[4]+x[5]-1 ]
> >
> > Alternatively, you can write a Python function that returns those two
> values
> > in a sequence as pass it to fmin_slsqp as  f_eqcons.
> >
>
> Thanks for the pointer. This hasn't made it's way into the docs and
> help file yet.
>
> Josef
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>



-- 
Leon Adams
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090506/bbf609c7/attachment.html>

From dpeterson at enthought.com  Wed May  6 12:36:08 2009
From: dpeterson at enthought.com (Dave Peterson)
Date: Wed, 06 May 2009 11:36:08 -0500
Subject: [SciPy-dev] Undefined NAN in amos_wrappers.c errors on Solaris amd64
Message-ID: <4A01BC78.30107@enthought.com>

When trying to build rev 5698 of SciPy's trunk on a Solaris amd64 
system, I'm getting the a bunch of 'NAN undeclared' type errors in 
building the 'scipy.special._cephes' extension. Is this a known error by 
any chance? Or does anyone have an idea of what is going on?

-- Dave


Here's the GCC command being run:

(build)-bash-3.00$ /home/dpeterson/local64/bin/gcc -m64 -DNDEBUG -O2 
-DHAVE_NCURSES_H -I/home/dpeterson/py/build/include 
-I/home/dpeterson/py/build/include/ncurses -I/usr/include 
-I/home/dpeterson/py/build/include -m64 -g -O3 -pipe 
-fomit-frame-pointer -fPIC 
-I/home/dpeterson/py/build/trunk/recipes/install/numpy-1.3.0n1-py2.5-solaris-2.10-i86pc.egg/numpy/core/include 
-I/home/dpeterson/py/build/include/python2.5 -c 
scipy/special/amos_wrappers.c -o
build/temp.solaris-2.10-i86pc-2.5/scipy/special/amos_wrappers.o


And here's the error output:

scipy/special/amos_wrappers.c: In function ?set_nan_if_no_computation_done?:
scipy/special/amos_wrappers.c:67: error: ?NAN? undeclared (first use in 
this function)
scipy/special/amos_wrappers.c:67: error: (Each undeclared identifier is 
reported only once
scipy/special/amos_wrappers.c:67: error: for each function it appears in.)
scipy/special/amos_wrappers.c: In function ?cairy_wrap_e_real?:
scipy/special/amos_wrappers.c:177: error: ?NAN? undeclared (first use in 
this function)
scipy/special/amos_wrappers.c: In function ?cbesi_wrap?:
scipy/special/amos_wrappers.c:218: error: ?INFINITY? undeclared (first 
use in this function)
scipy/special/amos_wrappers.c: In function ?cbesi_wrap_e_real?:
scipy/special/amos_wrappers.c:275: error: ?NAN? undeclared (first use in 
this function)
scipy/special/amos_wrappers.c: In function ?cbesj_wrap?:
scipy/special/amos_wrappers.c:300: error: ?INFINITY? undeclared (first 
use in this function)
scipy/special/amos_wrappers.c: In function ?cbesj_wrap_e_real?:
scipy/special/amos_wrappers.c:340: error: ?NAN? undeclared (first use in 
this function)
scipy/special/amos_wrappers.c: In function ?cbesy_wrap?:
scipy/special/amos_wrappers.c:365: error: ?INFINITY? undeclared (first 
use in this function)
scipy/special/amos_wrappers.c: In function ?cbesy_wrap_e?:
scipy/special/amos_wrappers.c:396: error: ?INFINITY? undeclared (first 
use in this function)
scipy/special/amos_wrappers.c: In function ?cbesy_wrap_e_real?:
scipy/special/amos_wrappers.c:414: error: ?NAN? undeclared (first use in 
this function)
scipy/special/amos_wrappers.c: In function ?cbesk_wrap?:
scipy/special/amos_wrappers.c:438: error: ?INFINITY? undeclared (first 
use in this function)
scipy/special/amos_wrappers.c: In function ?cbesk_wrap_e?:
scipy/special/amos_wrappers.c:461: error: ?INFINITY? undeclared (first 
use in this function)
scipy/special/amos_wrappers.c: In function ?cbesk_wrap_real?:
scipy/special/amos_wrappers.c:472: error: ?NAN? undeclared (first use in 
this function)
scipy/special/amos_wrappers.c: In function ?cbesk_wrap_e_real?:
scipy/special/amos_wrappers.c:484: error: ?NAN? undeclared (first use in 
this function)



From dpeterson at enthought.com  Wed May  6 12:54:12 2009
From: dpeterson at enthought.com (Dave Peterson)
Date: Wed, 06 May 2009 11:54:12 -0500
Subject: [SciPy-dev] Undefined NAN in amos_wrappers.c errors on Solaris
 amd64
In-Reply-To: <4A01BC78.30107@enthought.com>
References: <4A01BC78.30107@enthought.com>
Message-ID: <4A01C0B4.1050003@enthought.com>

Finding that many of the other .c files in scipy/special/cephes 
contained an extern NAN definition, I ended up inserting the following 
in amos_wrappers.c. The file now compiles and 'python setup.py build' 
finishes, but is this the "right" fix? I would have thought there was a 
common header where this should be declared?

extern double INFINITY, NAN;


-- Dave


Dave Peterson wrote:
> When trying to build rev 5698 of SciPy's trunk on a Solaris amd64 
> system, I'm getting the a bunch of 'NAN undeclared' type errors in 
> building the 'scipy.special._cephes' extension. Is this a known error by 
> any chance? Or does anyone have an idea of what is going on?
>
> -- Dave
>
>
> Here's the GCC command being run:
>
> (build)-bash-3.00$ /home/dpeterson/local64/bin/gcc -m64 -DNDEBUG -O2 
> -DHAVE_NCURSES_H -I/home/dpeterson/py/build/include 
> -I/home/dpeterson/py/build/include/ncurses -I/usr/include 
> -I/home/dpeterson/py/build/include -m64 -g -O3 -pipe 
> -fomit-frame-pointer -fPIC 
> -I/home/dpeterson/py/build/trunk/recipes/install/numpy-1.3.0n1-py2.5-solaris-2.10-i86pc.egg/numpy/core/include 
> -I/home/dpeterson/py/build/include/python2.5 -c 
> scipy/special/amos_wrappers.c -o
> build/temp.solaris-2.10-i86pc-2.5/scipy/special/amos_wrappers.o
>
>
> And here's the error output:
>
> scipy/special/amos_wrappers.c: In function ?set_nan_if_no_computation_done?:
> scipy/special/amos_wrappers.c:67: error: ?NAN? undeclared (first use in 
> this function)
> scipy/special/amos_wrappers.c:67: error: (Each undeclared identifier is 
> reported only once
> scipy/special/amos_wrappers.c:67: error: for each function it appears in.)
> scipy/special/amos_wrappers.c: In function ?cairy_wrap_e_real?:
> scipy/special/amos_wrappers.c:177: error: ?NAN? undeclared (first use in 
> this function)
> scipy/special/amos_wrappers.c: In function ?cbesi_wrap?:
> scipy/special/amos_wrappers.c:218: error: ?INFINITY? undeclared (first 
> use in this function)
> scipy/special/amos_wrappers.c: In function ?cbesi_wrap_e_real?:
> scipy/special/amos_wrappers.c:275: error: ?NAN? undeclared (first use in 
> this function)
> scipy/special/amos_wrappers.c: In function ?cbesj_wrap?:
> scipy/special/amos_wrappers.c:300: error: ?INFINITY? undeclared (first 
> use in this function)
> scipy/special/amos_wrappers.c: In function ?cbesj_wrap_e_real?:
> scipy/special/amos_wrappers.c:340: error: ?NAN? undeclared (first use in 
> this function)
> scipy/special/amos_wrappers.c: In function ?cbesy_wrap?:
> scipy/special/amos_wrappers.c:365: error: ?INFINITY? undeclared (first 
> use in this function)
> scipy/special/amos_wrappers.c: In function ?cbesy_wrap_e?:
> scipy/special/amos_wrappers.c:396: error: ?INFINITY? undeclared (first 
> use in this function)
> scipy/special/amos_wrappers.c: In function ?cbesy_wrap_e_real?:
> scipy/special/amos_wrappers.c:414: error: ?NAN? undeclared (first use in 
> this function)
> scipy/special/amos_wrappers.c: In function ?cbesk_wrap?:
> scipy/special/amos_wrappers.c:438: error: ?INFINITY? undeclared (first 
> use in this function)
> scipy/special/amos_wrappers.c: In function ?cbesk_wrap_e?:
> scipy/special/amos_wrappers.c:461: error: ?INFINITY? undeclared (first 
> use in this function)
> scipy/special/amos_wrappers.c: In function ?cbesk_wrap_real?:
> scipy/special/amos_wrappers.c:472: error: ?NAN? undeclared (first use in 
> this function)
> scipy/special/amos_wrappers.c: In function ?cbesk_wrap_e_real?:
> scipy/special/amos_wrappers.c:484: error: ?NAN? undeclared (first use in 
> this function)
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   


From pav at iki.fi  Wed May  6 14:31:48 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 6 May 2009 18:31:48 +0000 (UTC)
Subject: [SciPy-dev] Undefined NAN in amos_wrappers.c errors on Solaris
	amd64
References: <4A01BC78.30107@enthought.com> <4A01C0B4.1050003@enthought.com>
Message-ID: <gtsl2k$8qg$1@ger.gmane.org>

Wed, 06 May 2009 11:54:12 -0500, Dave Peterson wrote:

> Finding that many of the other .c files in scipy/special/cephes
> contained an extern NAN definition, I ended up inserting the following
> in amos_wrappers.c. The file now compiles and 'python setup.py build'
> finishes, but is this the "right" fix?

Good catch! My bad, you're perhaps the first one to compile Scipy SVN 
with a non-C99 compiler since January :)

> I would have thought there was a common header where this should be
> declared?

We have a lot of technical debt in scipy.special, so NAN etc. are not 
collected to a single header.

-- 
Pauli Virtanen



From michael.abshoff at googlemail.com  Wed May  6 17:57:14 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Wed, 06 May 2009 14:57:14 -0700
Subject: [SciPy-dev] Undefined NAN in amos_wrappers.c errors on Solaris
 amd64
In-Reply-To: <4A01C0B4.1050003@enthought.com>
References: <4A01BC78.30107@enthought.com> <4A01C0B4.1050003@enthought.com>
Message-ID: <4A0207BA.20100@gmail.com>

Dave Peterson wrote:

Hi,

> Finding that many of the other .c files in scipy/special/cephes 
> contained an extern NAN definition, I ended up inserting the following 
> in amos_wrappers.c. The file now compiles and 'python setup.py build' 
> finishes, but is this the "right" fix? I would have thought there was a 
> common header where this should be declared?
> 
> extern double INFINITY, NAN;

I don't believe this is the "right" fix. It will work, but that doesn't 
mean it is the way it is supposed to be fixed. Unlike many other 
operating systems the C99 bits of the math standard are guarded on 
Solaris by C99 checks, i.e. if you add "-std=c99" to your compiler flags 
it should work out of the box. Note that this might open up a potential 
can of worms since all the sudden some other header bits aren't there 
any more since they are conditionally used only when C99 is not defined.

Anyway, the flag setting might be too invasive for some people, but I 
just wanted to point out that technically the fix is a hacky (but quite 
common) workaround. As long as it works I don't care too much :)

> 
> -- Dave

Cheers,

Michael


From david at ar.media.kyoto-u.ac.jp  Wed May  6 23:05:57 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 07 May 2009 12:05:57 +0900
Subject: [SciPy-dev] Undefined NAN in amos_wrappers.c errors on Solaris
 amd64
In-Reply-To: <4A0207BA.20100@gmail.com>
References: <4A01BC78.30107@enthought.com> <4A01C0B4.1050003@enthought.com>
	<4A0207BA.20100@gmail.com>
Message-ID: <4A025015.1020800@ar.media.kyoto-u.ac.jp>

Michael Abshoff wrote:
> Dave Peterson wrote:
>
> Hi,
>
>   
>> Finding that many of the other .c files in scipy/special/cephes 
>> contained an extern NAN definition, I ended up inserting the following 
>> in amos_wrappers.c. The file now compiles and 'python setup.py build' 
>> finishes, but is this the "right" fix? I would have thought there was a 
>> common header where this should be declared?
>>
>> extern double INFINITY, NAN;
>>     
>
> I don't believe this is the "right" fix.

No it is not :) We have the right fix (our own portable NPY_NAN macro)
in numpy, but I have not yet found a way to use numpy C libraries into
scipy proper. Once this is done, we will be able to depend on quite a
useful set of portable math functionalities instead of those ugly hacks.
Although technically, for this exact macro, only the header is needed.

cheers,

David


From charlesr.harris at gmail.com  Thu May  7 03:40:25 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 7 May 2009 01:40:25 -0600
Subject: [SciPy-dev] Lagrange interpolation.
Message-ID: <e06186140905070040l52713715xfdee9ec96c2d4b59@mail.gmail.com>

Hi All,

I was just looking for Lagrange interpolation in scipy and didn't find it.
So I wrote my own, then found this thread
http://mail.scipy.org/pipermail/scipy-user/2008-April/016407.html. Anne's
version of barycentric interpolation is almost the same as the one I just
wrote, and I have a fortran version of Krogh that I wrote for scipy about,
oh, seven years ago ;) Anyway, I don't see Anne's routines in scipy, what
happened to them?

I've attached my version for comparison. BTW, the current lagrange in scipy
is the interpolating polynomial, which is pretty much useless for numerical
work. And the barycentric form can be differentiated, but it is work. Note
that with the proper grid the barycentric form provides a good way to
interpolate with chebyschev polynomials and can also be adapted for
trigonometric and rational interpolation, all of which are useful.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/061e6843/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lagrange.py
Type: text/x-python
Size: 2134 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/061e6843/attachment.py>

From andrew at andrewschein.com  Thu May  7 10:32:54 2009
From: andrew at andrewschein.com (Andrew Schein)
Date: Thu, 7 May 2009 10:32:54 -0400
Subject: [SciPy-dev] fast sorting of most numpy array types
Message-ID: <4e963a650905070732h2edabe72ue8e2797f45bfff5@mail.gmail.com>

Hi developers -

I have recently release highly optimized type-specific C routines for
sorting arrays of numeric types:
**http://bitbucket.org/ais/usort/wiki/Home

The code was motivated in part by noting that the numpy implementation of
sort appears to be a template-ized (over comparison) quicksort... and radix
sort is often the better choice.  Actually, it has been a few months since I
looked at the numpy source, so don't quote me.  The usort routines use a
variety of strategies depending on the size of the array: radix sort, quick
sort, insertion sort.

I think the code could easily be incorporated in to numpy... and if there is
interest in accepting such a change, I might find time to do it myself.

Cheers,

Andy

-- 
Andrew I. Schein
www.andrewschein.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/34f90a4c/attachment.html>

From josef.pktd at gmail.com  Thu May  7 11:02:25 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 7 May 2009 11:02:25 -0400
Subject: [SciPy-dev] docstrings in class.__init__
Message-ID: <1cd32cbb0905070802r2b9b8591iedd10400ade9c53f@mail.gmail.com>

I just looked at the spline classes in scipy.interpolate.

The docstring for the constructor is in the __init__ method, however
this is currently not captured by the sphinx documentation.
example: http://docs.scipy.org/scipy/docs/scipy.interpolate.fitpack2.UnivariateSpline/#scipy-interpolate-univariatespline

Is it possible to automatically add the __init__ docstring to the
class docstring, or should we copy or move them over, or add __init__
method explicitly to the .rst files to be scanned by sphinx?

help(interpolate.LSQUnivariateSpline) doesn't have this problem, as
discussed last year.

Josef


From josef.pktd at gmail.com  Thu May  7 11:08:03 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 7 May 2009 11:08:03 -0400
Subject: [SciPy-dev] docstrings in class.__init__
In-Reply-To: <1cd32cbb0905070802r2b9b8591iedd10400ade9c53f@mail.gmail.com>
References: <1cd32cbb0905070802r2b9b8591iedd10400ade9c53f@mail.gmail.com>
Message-ID: <1cd32cbb0905070808l1bcdb111saea1ce2cea05bff5@mail.gmail.com>

On Thu, May 7, 2009 at 11:02 AM,  <josef.pktd at gmail.com> wrote:
> I just looked at the spline classes in scipy.interpolate.
>
> The docstring for the constructor is in the __init__ method, however
> this is currently not captured by the sphinx documentation.
> example: http://docs.scipy.org/scipy/docs/scipy.interpolate.fitpack2.UnivariateSpline/#scipy-interpolate-univariatespline
>
> Is it possible to automatically add the __init__ docstring to the
> class docstring, or should we copy or move them over, or add __init__
> method explicitly to the .rst files to be scanned by sphinx?
>
> help(interpolate.LSQUnivariateSpline) doesn't have this problem, as
> discussed last year.
>

or can we use

.. autoclass:: Noodle
   :members: eat, slurp

in the automatic summary

Josef


From aahz at pythoncraft.com  Thu May  7 12:46:20 2009
From: aahz at pythoncraft.com (Aahz)
Date: Thu, 7 May 2009 09:46:20 -0700
Subject: [SciPy-dev] fast sorting of most numpy array types
In-Reply-To: <4e963a650905070732h2edabe72ue8e2797f45bfff5@mail.gmail.com>
References: <4e963a650905070732h2edabe72ue8e2797f45bfff5@mail.gmail.com>
Message-ID: <20090507164620.GA24583@panix.com>

On Thu, May 07, 2009, Andrew Schein wrote:
> 
> I have recently release highly optimized type-specific C routines for
> sorting arrays of numeric types:
> **http://bitbucket.org/ais/usort/wiki/Home
> 
> The code was motivated in part by noting that the numpy implementation of
> sort appears to be a template-ized (over comparison) quicksort... and radix
> sort is often the better choice.  Actually, it has been a few months since I
> looked at the numpy source, so don't quote me.  The usort routines use a
> variety of strategies depending on the size of the array: radix sort, quick
> sort, insertion sort.

Speaking as someone who doesn't know anything about NumPy, have you
looked at the timsort used in Python itself?  One of the advantages of
timsort is that it's guaranteed to be stable (elements that compare equal
retain their relative ordering from before sorting).
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"It is easier to optimize correct code than to correct optimized code."
--Bill Harlan


From robert.kern at gmail.com  Thu May  7 13:46:37 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 7 May 2009 13:46:37 -0400
Subject: [SciPy-dev] fast sorting of most numpy array types
In-Reply-To: <20090507164620.GA24583@panix.com>
References: <4e963a650905070732h2edabe72ue8e2797f45bfff5@mail.gmail.com> 
	<20090507164620.GA24583@panix.com>
Message-ID: <3d375d730905071046i399ed59ev3ae06f9d6e639a08@mail.gmail.com>

On Thu, May 7, 2009 at 12:46, Aahz <aahz at pythoncraft.com> wrote:
> On Thu, May 07, 2009, Andrew Schein wrote:
>>
>> I have recently release highly optimized type-specific C routines for
>> sorting arrays of numeric types:
>> **http://bitbucket.org/ais/usort/wiki/Home
>>
>> The code was motivated in part by noting that the numpy implementation of
>> sort appears to be a template-ized (over comparison) quicksort... and radix
>> sort is often the better choice. ?Actually, it has been a few months since I
>> looked at the numpy source, so don't quote me. ?The usort routines use a
>> variety of strategies depending on the size of the array: radix sort, quick
>> sort, insertion sort.
>
> Speaking as someone who doesn't know anything about NumPy, have you
> looked at the timsort used in Python itself? ?One of the advantages of
> timsort is that it's guaranteed to be stable (elements that compare equal
> retain their relative ordering from before sorting).

We actually have a few sorting algorithms, including a stable mergesort:

  http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html

The last time we looked at this, we decided that implementing timsort
probably wouldn't be much better than mergesort. timsort is designed
for the case of expensive compare operations, and we have relatively
cheap compares.

Happy to be proven wrong, of course.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Thu May  7 13:49:08 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 7 May 2009 11:49:08 -0600
Subject: [SciPy-dev] fast sorting of most numpy array types
In-Reply-To: <4e963a650905070732h2edabe72ue8e2797f45bfff5@mail.gmail.com>
References: <4e963a650905070732h2edabe72ue8e2797f45bfff5@mail.gmail.com>
Message-ID: <e06186140905071049x29ae30f4x43e74480549779af@mail.gmail.com>

On Thu, May 7, 2009 at 8:32 AM, Andrew Schein <andrew at andrewschein.com>wrote:

> Hi developers -
>
> I have recently release highly optimized type-specific C routines for
> sorting arrays of numeric types:
> **http://bitbucket.org/ais/usort/wiki/Home
>
> The code was motivated in part by noting that the numpy implementation of
> sort appears to be a template-ized (over comparison) quicksort... and radix
> sort is often the better choice.  Actually, it has been a few months since I
> looked at the numpy source, so don't quote me.  The usort routines use a
> variety of strategies depending on the size of the array: radix sort, quick
> sort, insertion sort.
>
> I think the code could easily be incorporated in to numpy... and if there
> is interest in accepting such a change, I might find time to do it myself.
>

Numpy has quicksort, mergesort, and heapsort and now and then someone on the
list suggests adding radix sort. No doubt radix is suitable for certain
types, char arrays seem a natural. But there are problems with byte order
for the different integer types and floats can become complicated. And
unlike quicksort radix sort isn't done inplace. So it isn't clear that a
radix sort is the way to go for sorting in general but it could be useful in
some places.

Do you have some time results?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/373af14e/attachment.html>

From william.ratcliff at gmail.com  Thu May  7 14:08:04 2009
From: william.ratcliff at gmail.com (william ratcliff)
Date: Thu, 7 May 2009 14:08:04 -0400
Subject: [SciPy-dev] Adding mpfit scipy optimize package,
	koposov@mpia-hd.mpg.de, 
Message-ID: <827183970905071108i6ce55a8fj7efa224bfeecd926@mail.gmail.com>

Hi!  For a long time, there has been a standard package used by IDL users
for fitting functions to data called MPFIT:
http://cow.physics.wisc.edu/~craigm/idl/fitting.html

It allows the user to limit the domain (simple box limits), fix parameters,
etc.  It also returns errors on the parameters based on the covariance
matrix.
Essentially, it translates MINPACK to IDL and adds some nice features (like
the ability to fix parameters).  Some time later, Mark Rivers translated
this to
python for use with Numeric, and Sergei Koposov migrated the code to numpy
and added several bug fixes from the Numeric version of the code.  We've
been working with the author of the IDL package, Craig Markwardt to include
the additional bug fixes which he has added.

I have written a nose-test compatible test.  Could someone look at it and
tell me if it meets the scipy style before I continue adding more tests from
Craig's test-suite for the C version of his program?  Also, what else would
we need to do to include this in scipy?  Mark Rivers, Craig, and Sergey have
agreed to a BSD license for their contributions.


Thanks,
William
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/c52bd128/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpfit.zip
Type: application/zip
Size: 28971 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/c52bd128/attachment.zip>

From andrew at andrewschein.com  Thu May  7 14:30:59 2009
From: andrew at andrewschein.com (Andrew Schein)
Date: Thu, 7 May 2009 14:30:59 -0400
Subject: [SciPy-dev] Scipy-dev Digest, Vol 67, Issue 12
In-Reply-To: <mailman.876.1241719692.2410.scipy-dev@scipy.org>
References: <mailman.876.1241719692.2410.scipy-dev@scipy.org>
Message-ID: <4e963a650905071130v2c9a312agbd05b055a856a419@mail.gmail.com>

>
> Date: Thu, 7 May 2009 11:49:08 -0600
> From: Charles R Harris <charlesr.harris at gmail.com>
> Subject: Re: [SciPy-dev] fast sorting of most numpy array types
> To: SciPy Developers List <scipy-dev at scipy.org>
>
> Numpy has quicksort, mergesort, and heapsort and now and then someone on
> the
> list suggests adding radix sort. No doubt radix is suitable for certain
> types, char arrays seem a natural. But there are problems with byte order
> for the different integer types and floats can become complicated. And
> unlike quicksort radix sort isn't done inplace. So it isn't clear that a
> radix sort is the way to go for sorting in general but it could be useful
> in
> some places.

Do you have some time results?
>

 Hi Charles -

There are time results against glibc qsort() in the original post. The
distribution of usort also provides an optimized quicksort with inlined
comparisons and the capacity to test this against glibc.  The "win" from
inlining comparions is not as great as using radix sort.

I have not timed against heap/merge sorts... since I believe these are
slower on average on randomized inputs.

By byte order were you referring to endianess of different architectures?
This could be a deal breaker as the code is written.

The usort library already consists of in-place routines.

-Andy

-- 
Andrew I. Schein
www.andrewschein.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/4242028d/attachment.html>

From charlesr.harris at gmail.com  Thu May  7 14:52:47 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 7 May 2009 12:52:47 -0600
Subject: [SciPy-dev] fast sorting of most numpy array types
In-Reply-To: <e06186140905071049x29ae30f4x43e74480549779af@mail.gmail.com>
References: <4e963a650905070732h2edabe72ue8e2797f45bfff5@mail.gmail.com>
	<e06186140905071049x29ae30f4x43e74480549779af@mail.gmail.com>
Message-ID: <e06186140905071152j1442f291l6376da8b34e3e4e5@mail.gmail.com>

On Thu, May 7, 2009 at 11:49 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Thu, May 7, 2009 at 8:32 AM, Andrew Schein <andrew at andrewschein.com>wrote:
>
>> Hi developers -
>>
>> I have recently release highly optimized type-specific C routines for
>> sorting arrays of numeric types:
>> **http://bitbucket.org/ais/usort/wiki/Home
>>
>> The code was motivated in part by noting that the numpy implementation of
>> sort appears to be a template-ized (over comparison) quicksort... and radix
>> sort is often the better choice.  Actually, it has been a few months since I
>> looked at the numpy source, so don't quote me.  The usort routines use a
>> variety of strategies depending on the size of the array: radix sort, quick
>> sort, insertion sort.
>>
>> I think the code could easily be incorporated in to numpy... and if there
>> is interest in accepting such a change, I might find time to do it myself.
>>
>
> Numpy has quicksort, mergesort, and heapsort and now and then someone on
> the list suggests adding radix sort. No doubt radix is suitable for certain
> types, char arrays seem a natural. But there are problems with byte order
> for the different integer types and floats can become complicated. And
> unlike quicksort radix sort isn't done inplace. So it isn't clear that a
> radix sort is the way to go for sorting in general but it could be useful in
> some places.
>
> Do you have some time results?
>

It isn't too hard to add sorts to numpy, so if you want to experiment with
code I can help you get started.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/38d01733/attachment.html>

From charlesr.harris at gmail.com  Thu May  7 15:30:03 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 7 May 2009 13:30:03 -0600
Subject: [SciPy-dev] Scipy-dev Digest, Vol 67, Issue 12
In-Reply-To: <4e963a650905071130v2c9a312agbd05b055a856a419@mail.gmail.com>
References: <mailman.876.1241719692.2410.scipy-dev@scipy.org>
	<4e963a650905071130v2c9a312agbd05b055a856a419@mail.gmail.com>
Message-ID: <e06186140905071230g30381bbclda61d77704948759@mail.gmail.com>

On Thu, May 7, 2009 at 12:30 PM, Andrew Schein <andrew at andrewschein.com>wrote:

> Date: Thu, 7 May 2009 11:49:08 -0600
>> From: Charles R Harris <charlesr.harris at gmail.com>
>> Subject: Re: [SciPy-dev] fast sorting of most numpy array types
>> To: SciPy Developers List <scipy-dev at scipy.org>
>>
>> Numpy has quicksort, mergesort, and heapsort and now and then someone on
>> the
>> list suggests adding radix sort. No doubt radix is suitable for certain
>> types, char arrays seem a natural. But there are problems with byte order
>> for the different integer types and floats can become complicated. And
>> unlike quicksort radix sort isn't done inplace. So it isn't clear that a
>> radix sort is the way to go for sorting in general but it could be useful
>> in
>> some places.
>
> Do you have some time results?
>>
>
>  Hi Charles -
>
> There are time results against glibc qsort() in the original post. The
> distribution of usort also provides an optimized quicksort with inlined
> comparisons and the capacity to test this against glibc.  The "win" from
> inlining comparions is not as great as using radix sort.
>
> I have not timed against heap/merge sorts... since I believe these are
> slower on average on randomized inputs.
>

The three sorts cover three different needs: fast (quicksort),
guaranteed/stable (mergesort), guaranteed/inplace (heapsort).


>
> By byte order were you referring to endianess of different architectures?
> This could be a deal breaker as the code is written.
>

Yep. Note that the actual code is generated when numpy is built so you might
be able to work around this.


>
> The usort library already consists of in-place routines.
>

Good.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/80f3b124/attachment.html>

From pav at iki.fi  Thu May  7 15:47:05 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 7 May 2009 19:47:05 +0000 (UTC)
Subject: [SciPy-dev] Lagrange interpolation.
References: <e06186140905070040l52713715xfdee9ec96c2d4b59@mail.gmail.com>
Message-ID: <gtvdrp$oln$1@ger.gmane.org>

Thu, 07 May 2009 01:40:25 -0600, Charles R Harris wrote:
> I was just looking for Lagrange interpolation in scipy and didn't find
> it. So I wrote my own, then found this thread
> http://mail.scipy.org/pipermail/scipy-user/2008-April/016407.html.
> Anne's version of barycentric interpolation is almost the same as the
> one I just wrote, and I have a fortran version of Krogh that I wrote for
> scipy about, oh, seven years ago ;) Anyway, I don't see Anne's routines
> in scipy, what happened to them?

Umm, I believe they got into Scipy, see scipy/interpolate/polyint.py

-- 
Pauli Virtanen



From charlesr.harris at gmail.com  Thu May  7 15:56:02 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 7 May 2009 13:56:02 -0600
Subject: [SciPy-dev] Lagrange interpolation
Message-ID: <e06186140905071256t3e14f38s930d543f7230329e@mail.gmail.com>

Hi All,

I am going to add Anne's barycentric Lagrange interpolation with some
modifications to scale the weights and separate out the weight computations,
which depend only on the grid, from the interpolator. I have also made the
interpolation handle more general inputs. Any objections?

In my experience Krogh interpolation needs to be in C to be efficient, there
are too many decisions involved in finding the right path to make it fast
when written in python. Barycentric interpolation also tends to be more
stable than divided difference based approaches, especially with complex
interpolation points. But I will take a closer look at Anne's
implementation.

It is somewhat scandalous that Scipy doesn't have decent polynomial
interpolation after all these years.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/d9b94cbd/attachment.html>

From pav at iki.fi  Thu May  7 16:04:28 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 7 May 2009 20:04:28 +0000 (UTC)
Subject: [SciPy-dev] Lagrange interpolation
References: <e06186140905071256t3e14f38s930d543f7230329e@mail.gmail.com>
Message-ID: <gtvesc$oln$3@ger.gmane.org>

Hi,

Thu, 07 May 2009 13:56:02 -0600, Charles R Harris wrote:
> I am going to add Anne's barycentric Lagrange interpolation with some
> modifications to scale the weights and separate out the weight
> computations, which depend only on the grid, from the interpolator. I
> have also made the interpolation handle more general inputs. Any
> objections?

I'm now a bit confused: wasn't Anne's polynomial interpolation code added 
as scipy.interpolate.polyint April last year? Or is the question now 
about some other code?

But I guess if you have improvements, those are welcome to go in if they 
don't break the API.

-- 
Pauli Virtanen



From charlesr.harris at gmail.com  Thu May  7 17:17:21 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 7 May 2009 15:17:21 -0600
Subject: [SciPy-dev] Lagrange interpolation.
In-Reply-To: <gtvdrp$oln$1@ger.gmane.org>
References: <e06186140905070040l52713715xfdee9ec96c2d4b59@mail.gmail.com>
	<gtvdrp$oln$1@ger.gmane.org>
Message-ID: <e06186140905071417je5965erf7b175bdd9306899@mail.gmail.com>

On Thu, May 7, 2009 at 1:47 PM, Pauli Virtanen <pav at iki.fi> wrote:

> Thu, 07 May 2009 01:40:25 -0600, Charles R Harris wrote:
> > I was just looking for Lagrange interpolation in scipy and didn't find
> > it. So I wrote my own, then found this thread
> > http://mail.scipy.org/pipermail/scipy-user/2008-April/016407.html.
> > Anne's version of barycentric interpolation is almost the same as the
> > one I just wrote, and I have a fortran version of Krogh that I wrote for
> > scipy about, oh, seven years ago ;) Anyway, I don't see Anne's routines
> > in scipy, what happened to them?
>
> Umm, I believe they got into Scipy, see scipy/interpolate/polyint.py
>

Hmm, so it is. I'll do some vectorization and scaling fixes and maybe add a
derived class.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090507/08d6cdfa/attachment.html>

From pav at iki.fi  Thu May  7 17:23:50 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 7 May 2009 21:23:50 +0000 (UTC)
Subject: [SciPy-dev] RFR: Nonlinear solver rewrite
Message-ID: <gtvjh6$oln$4@ger.gmane.org>

Hi,

The large-scale nonlinear equation solver (nonlin.py) rewrite is nearing 
completion.

Git branch is here:

    http://github.com/pv/scipy-work/commits/ticket/791-optimize-nonlin-rewrite

Review ticket is here:

    http://codereview.appspot.com/62059

Trac ticket is here:

    http://projects.scipy.org/scipy/ticket/791

You may want to try out the examples in the docstring of nonlin.py


The following new features are added

    - Proper termination conditions
    - Generic framework for tolerance-controlled inexact Newton methods
    - Backtracking and line searches
    - Limited-memory Broyden methods, including SVD rank reduction
      in addition to more basic options
    - Newton-Krylov solver, using any of the solvers from 
      scipy.sparse.linalg

The new code API is mostly backward-compatible to the old one. Some
methods that were not very useful are deprecated. All of the code is
rewritten and refactored, aiming for easier maintenance.

The API may need some tuning still; should the method_name wrappers be
discarded, and the nonlin_solve(..., jacobian='foo') be the preferred
interface?


Note that this patchset includes some work in other parts of Scipy; these
are in review separately (so don't comment on those in this ticket):

    - LGMRES sparse solver
      http://codereview.appspot.com/53096
      http://github.com/pv/scipy-work/commits/lgmres

    - Line search refactoring
      http://codereview.appspot.com/52096
      http://github.com/pv/scipy-work/commits/linesearch-refactor


Cheers,
Pauli



From josef.pktd at gmail.com  Fri May  8 13:35:41 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 8 May 2009 13:35:41 -0400
Subject: [SciPy-dev] 2 trac instances?
	http://scipy.org/scipy/scikits/timeline vs
	http://projects.scipy.org/scikits/timeline
Message-ID: <1cd32cbb0905081035q41103405ka5cb233c7a905709@mail.gmail.com>

I was using an old bookmark to access the scikits trac and was
surprised to see no recent updates.

Are there two trac instances running? Is it possible that the old one
redirects to the new?

It's confusing until I figured out that I was on the wrong address.

Josef


From stefan at sun.ac.za  Fri May  8 14:32:59 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 8 May 2009 20:32:59 +0200
Subject: [SciPy-dev] 2 trac instances?
	http://scipy.org/scipy/scikits/timeline vs
	http://projects.scipy.org/scikits/timeline
In-Reply-To: <1cd32cbb0905081035q41103405ka5cb233c7a905709@mail.gmail.com>
References: <1cd32cbb0905081035q41103405ka5cb233c7a905709@mail.gmail.com>
Message-ID: <9457e7c80905081132s540a03ffxf0186b8903cd06d0@mail.gmail.com>

Hi Josef

Which one are they currently using?  The site is supposed to be at

http://projects.scipy.org/scikits

If that is not the case, I'll have to re-migrate the data and then
redirect the old site.

Thanks for the heads up,

Cheers
St?fan

2009/5/8  <josef.pktd at gmail.com>:
> I was using an old bookmark to access the scikits trac and was
> surprised to see no recent updates.
>
> Are there two trac instances running? Is it possible that the old one
> redirects to the new?
>
> It's confusing until I figured out that I was on the wrong address.
>
> Josef


From pav at iki.fi  Fri May  8 14:36:52 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 8 May 2009 18:36:52 +0000 (UTC)
Subject: [SciPy-dev] 2 trac
	instances?	http://scipy.org/scipy/scikits/timeline
	vs	http://projects.scipy.org/scikits/timeline
References: <1cd32cbb0905081035q41103405ka5cb233c7a905709@mail.gmail.com>
Message-ID: <gu1u44$2lc$1@ger.gmane.org>

Fri, 08 May 2009 13:35:41 -0400, josef.pktd wrote:
> I was using an old bookmark to access the scikits trac and was surprised
> to see no recent updates.
> 
> Are there two trac instances running? Is it possible that the old one
> redirects to the new?
> 
> It's confusing until I figured out that I was on the wrong address.

IIRC the scikits Trac was moved to the new server only after the other 
two Tracs, and I guess disabling the old instance was forgotten. Someone 
with access to old.scipy.org could disable the old Trac and replace it 
with a redirect.

I doubt that this has to with the redirection loop issues...

-- 
Pauli Virtanen



From thouis at broad.mit.edu  Fri May  8 14:43:15 2009
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Fri, 8 May 2009 14:43:15 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <1cd32cbb0905051154t689ad8cdw898ce5e97b9872fc@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<6c17e6f50905010813g31c58b7ehe932f15d8494dd6f@mail.gmail.com> 
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com> 
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com> 
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com> 
	<1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com> 
	<6c17e6f50905041300q25dca2c0m93ec9f6cd74f6391@mail.gmail.com> 
	<1cd32cbb0905041343u433fcf49o68ef89928a0a71f0@mail.gmail.com> 
	<6c17e6f50905041411wdc519f7o5812618beab518f3@mail.gmail.com> 
	<1cd32cbb0905051154t689ad8cdw898ce5e97b9872fc@mail.gmail.com>
Message-ID: <6c17e6f50905081143t61917ed0ub9e0e9e5326eafd1@mail.gmail.com>

I've just pushed a new version to
http://broad.mit.edu/~thouis/scipy.git ndimage_measurements_rewrite

This includes rewrites of:  sum, mean, variance, standard_deviation,
minimum, maximum, minimum_position, maximum_position, center_of_mass,
and histogram in pure python.  It also adds a labeled_comprehension()
function.

The histogram() code could be much improved, speed-wise.  It uses
labeled_comprehension and numpy.histogram, but could use similar
bincount tricks as have been done for sum et al.

Some timing comparisons below, with test script attached.

Ray


Timing information (1K x 1K image), secs per call.
New:
variance 0.0620911860466
minimum 0.48847284317
maximum_position 0.588409199715
extrema 0.813310542107
center_of_mass 0.220837605
histogram 5.70109256506

Old:
variance 0.0640315508842
minimum 0.0331222701073
maximum_position 1.07506107569
extrema 2.27282817364
center_of_mass 0.0554035758972
histogram 0.0865430688858




On Tue, May 5, 2009 at 14:54,  <josef.pktd at gmail.com> wrote:
> On Mon, May 4, 2009 at 5:11 PM, Thouis (Ray) Jones <thouis at broad.mit.edu> wrote:
>> On Mon, May 4, 2009 at 16:43, ?<josef.pktd at gmail.com> wrote:
>>> The use case, I wrote my application for, also allowed for labels that
>>> are strings. The return inverse of unique1d is very flexible since it
>>> can handle many different types including structured arrays, but for
>>> numeric labels, I just found out, that it is a large speed penalty. If
>>> the labels are already integers or can be easily converted to
>>> integers, then not using unique1d to create the integer labels should
>>> be much faster.
>>
>> Using bincount probably requires calling unique1d (or some equivalent)
>> for this usage, since otherwise the call to bincount could create a
>> very large output (2^32 or even 2^64 bins).
>>
>> Ray Jones
>
> In my version, I started to use a conditional call to unique1d, if the
> type of the labels is not int (to handle float and strings). This
> could be extended to a very large max(labels). ?If the label type is
> int then, I just use bincount without unique1d.
>
> I tried it with 50% unused numbers, and the speed of bincount is
> comparable to ndimage, labels are generated with
> factors = (100+2*np.random.randint(nfact,size=nobs))
> #.astype('S6')#.astype(float)
> I tried with up to 4 million observations and 10000 unique labels.
>
> I'm starting to get confused with the many different versions, that I
> have now for this, the two typical examples are below. ndimage looks
> very picky which arrays it accepts, every once in a while I get a not
> implemented type exception or even a segfault.
>
> I also attach my full timing and test script.
>
> Josef
>
>
> def labelmean_ndi(factors, values, index=None):
> ? ?# works also for string labels in ys, but requires 1D
> ? ?# from mailing list scipy-user 2009-02-11
> ? ?unilinv = np.asanyarray(factors)
>
> ? ?if not np.issubdtype(unilinv.dtype, int):
> ? ? ? ?unil, unilinv = np.unique1d(factors, return_inverse=1)
> ? ? ? ?if index is None:
> ? ? ? ? ? ?index = np.arange(len(unil)) #unil
> ? ?elif index is None:
> ? ? ? ?index = np.arange(np.max(unilinv)+ 1)
>
> ? ?labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=index))
> ? ?return labelmeans
>
> def groupstatsbin(factors, values, ddof=0, stat='mvnx', drop=True):
> ? ?'''uses np.bincount, assumes factors/labels are integers,
> ? ?create labels with unique1d return_inverse if string labels
> ? ?'''
> ? ?#n = len(factors)
>
> ? ?rind = np.asanyarray(factors)
> ? ?#note: bincount uses complete list of integers, maybe range(max(factor)+1)
> ? ?if not np.issubdtype(rind.dtype, int):
> ? ? ? ?ix, rind = np.unique1d(factors, return_inverse=1)
> ## ? ?else:
> ## ? ? ? ?rind = factors
> ? ?#print rind.shape
> ? ?res = []
> ? ?if 'c' in stat or 'm' in stat or 'v' in stat:
> ? ? ? ?gcount = np.bincount(rind)
> ? ? ? ?if drop:
> ? ? ? ? ? ?keepidx = np.nonzero(gcount)
> ? ? ? ?if 'c' in stat:
> ? ? ? ? ? ?if drop:
> ? ? ? ? ? ? ? ?res.append(gcount[keepidx])
> ? ? ? ? ? ?else:
> ? ? ? ? ? ? ? ?res.append(gcount)
>
> ? ?if 'm' in stat or 'v' in stat:
> ? ? ? ?gmean = np.bincount(rind, weights=values)/ (1.0*gcount)
> ? ? ? ?if np.max(np.abs(gmean)) > 1e6:
> ? ? ? ? ? ?# improve numerical precision if means are large
> ? ? ? ? ? ?# (to "cheat" on NIST anova test examples)
> ? ? ? ? ? ?meanarr = gmean[rind]
> ? ? ? ? ? ?gmean += np.bincount(rind, weights=(values-meanarr)) / (1.0*gcount)
> ? ? ? ?if 'm' in stat:
> ? ? ? ? ? ?if drop:
> ? ? ? ? ? ? ? ?res.append(gmean[keepidx])
> ? ? ? ? ? ?else:
> ? ? ? ? ? ? ? ?res.append(gmean)
>
> ? ?if 'v' in stat:
> ? ? ? ?meanarr = gmean[rind]
> ? ? ? ?withinvar = np.bincount(rind, weights=(values-meanarr)**2) /
> (1.0*gcount-ddof)
> ? ? ? ?if drop:
> ? ? ? ? ? ?res.append(withinvar[keepidx])
> ? ? ? ?else:
> ? ? ? ? ? ?res.append(withinvar)
>
> ? ?if 'n' in stat or 'x' in stat:
> ? ? ? ?#calculate min, max per factor
> ? ? ? ?sortind = np.lexsort((values, rind))
> ? ? ? ?fs = rind[sortind]
> ? ? ? ?vs = values[sortind]
> ? ? ? ?fsd = np.hstack((np.inf,np.diff(fs),np.inf))
> ? ?if 'n' in stat:
> ? ? ? ?#minimum
> ? ? ? ?res.append(vs[fsd[:-1] != 0])
> ? ?if 'x' in stat:
> ? ? ? ?#maximum
> ? ? ? ?res.append(vs[fsd[1:] != 0])
> ? ?return res
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ndimage_test.py
Type: application/octet-stream
Size: 1124 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090508/ee6b6398/attachment.obj>

From josef.pktd at gmail.com  Fri May  8 15:11:23 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 8 May 2009 15:11:23 -0400
Subject: [SciPy-dev] 2 trac instances?
	http://scipy.org/scipy/scikits/timeline vs
	http://projects.scipy.org/scikits/timeline
In-Reply-To: <gu1u44$2lc$1@ger.gmane.org>
References: <1cd32cbb0905081035q41103405ka5cb233c7a905709@mail.gmail.com>
	<gu1u44$2lc$1@ger.gmane.org>
Message-ID: <1cd32cbb0905081211i370552e9kb072c21d91756d80@mail.gmail.com>

On Fri, May 8, 2009 at 2:36 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Fri, 08 May 2009 13:35:41 -0400, josef.pktd wrote:
>> I was using an old bookmark to access the scikits trac and was surprised
>> to see no recent updates.
>>
>> Are there two trac instances running? Is it possible that the old one
>> redirects to the new?
>>
>> It's confusing until I figured out that I was on the wrong address.
>
> IIRC the scikits Trac was moved to the new server only after the other
> two Tracs, and I guess disabling the old instance was forgotten. Someone
> with access to old.scipy.org could disable the old Trac and replace it
> with a redirect.
>
> I doubt that this has to with the redirection loop issues...
>
> --
> Pauli Virtanen
>

the old trac had some ticket activity, ticket numbering diverges at
#89 according to timeline.

Josef


From william.ratcliff at gmail.com  Fri May  8 16:09:35 2009
From: william.ratcliff at gmail.com (william ratcliff)
Date: Fri, 8 May 2009 16:09:35 -0400
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
Message-ID: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>

Hi!  For a long time, there has been a standard package used by IDL users
for fitting functions to data called MPFIT:
http://cow.physics.wisc.edu/~craigm/idl/fitting.html
<http://cow.physics.wisc.edu/%7Ecraigm/idl/fitting.html>

It allows the user to limit the domain (simple box limits), fix parameters,
etc.  It also returns errors on the parameters based on the covariance
matrix.
Essentially, it translates MINPACK to IDL and adds some nice features (like
the ability to fix parameters).  Some time later, Mark Rivers translated
this to python for use with Numeric, and Sergei Koposov migrated the
code to numpy
and added several bug fixes from the Numeric version of the code.  We've
been working with the author of the IDL package, Craig Markwardt to include
the additional bug fixes which he has added.

I have written a nose-test compatible test.  Could someone look at it and
tell me if it meets the scipy style before I continue adding more tests from
Craig's test-suite for the C version of his program?  Also, what else would
we need to do to include this in scipy?  Mark Rivers, Craig, and Sergey have
agreed to a BSD license for their contributions.


Thanks,
William





I realize that in my previous email, the attachment with the files were
scrubbed.  They can be found at:
http://drop.io/mpfitdrop



Thanks,
William
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090508/29a7c0c5/attachment.html>

From d_l_goldsmith at yahoo.com  Sat May  9 02:34:34 2009
From: d_l_goldsmith at yahoo.com (David Goldsmith)
Date: Fri, 8 May 2009 23:34:34 -0700 (PDT)
Subject: [SciPy-dev] Question about doc editing - here the right place?
Message-ID: <567397.78020.qm@web52101.mail.re2.yahoo.com>


Hi, all.  First, is this the right place to ask technical questions about editing the doc, or is there a sub-list for that?  Second, for now, I'll ask my present question here: the "template" states "refer to variables using back-ticks, e.g. `var`," but I find that when I do so for variables that are also numpy function names (e.g., pv, nper, pmt, etc. in fv), the resulting formatting is a link, not the italics present in the function declaration; so, which takes precedence: the dictum "refer to variables using back-ticks," or the desire to have the variables appear in the same font as they do in the function declaration?  Thanks!

DG


      


From pav at iki.fi  Sat May  9 11:55:49 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 9 May 2009 15:55:49 +0000 (UTC)
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
Message-ID: <gu4925$hmn$2@ger.gmane.org>

Fri, 08 May 2009 16:09:35 -0400, william ratcliff wrote:

> Hi!  For a long time, there has been a standard package used by IDL
> users for fitting functions to data called MPFIT:
> http://cow.physics.wisc.edu/~craigm/idl/fitting.html
> <http://cow.physics.wisc.edu/%7Ecraigm/idl/fitting.html>
[clip]
> http://drop.io/mpfitdrop

Nice!

However, I see some points where the code could be improved for better 
reusability and maintainability.

If we lived in an ideal world with infinite time to polish everything, 
I'd like to see all of the points below addressed before landing this to 
Scipy. But since this would be lots of error-prone work, it's probably 
best to try to reach some compromise.

Given these constraints, I'd still like to see at least the coding style 
and error handling fixed (which probably are not too difficult to 
change), in addition to having better test coverage. The rest could come 
later, even if we accrue yet more technical debt with this...


First, broad maintenance concerns:

- We already have `leastsq` from MINPACK. Having two MINPACK-derived
  least squares fitting routines is not good.

  So, I'd perhaps like to see the `leastsq` core part extracted out of
  MPFIT, and the MPFIT interface implemented on top of it as a thin
  wrapper, or the other way around.

  Maybe, if the modifications made on MINPACK are small, they can be
  backported to the Fortran code and MPFIT can be reimplemented on top
  of `leastsq`.

  Any thoughts on this?

- What is the performance of the Python implementation as compared to the
  Fortran code? For small data sets, the Python code is probably much
  slower, but for large datasets is the performance is comparable?

- Much of the code is Fortran written in Python: long routines, 
  goto-equivalents, 6-letter variable names.

  Good commenting offset this, though.


Then more specific points of refactoring:

- The code implements QR factorization with column pivoting from scratch.

  Perhaps some of this could be replaced with suitable LAPACK routines,
  or with stuff from scipy.linalg. (Cf. DGEQPF, DGEQP3)

  I'm not sure whether there's something suitable for qrsolve in LAPACK;
  the triangular system solver could be replaced with DTRTRS.

  Anyway, it might be useful to refactor qrfac and qrsolve out of MPFIT;
  there may be other applications when it's useful to be able to solve
  ||(A + I lambda) x - b||_2 = min! efficiently for multiple different
  `lambda` in sequence.

- fdjac2 should probably be refactored out of MPFIT; other optimization 
  algorithms that need Jacobians computed by forward differences can then
  use it. Do we already have something similar in Scipy already?

- `enorm` should be replaced with BLAS xNRM2; it does appropriate scaling
  automatically and is probably much faster.

  enorm = scipy.lib.blas.get_blas_funcs(['nrm2'], [some_array])

- The long mpfit.__init__ routine should be split into smaller parts,
  the function body is too long. I'm not sure exactly what parts, though.

  Perhaps at least the covariance matrix computation and input argument
  parsing should be separated.

- "self.errmsg = 'ERROR ...; return'".
  Probably should raise exceptions instead, at least for errors in input
  parameters.

  In general, I think the error handling should make better use of the
  Python exception and warning system; using `print` is not a correct
  way to signal an error in library code.

- I'm not sure about implementing everything in a class. This tends to
  couple tightly parts of the code that wouldn't really need such a strong
  coupling.

- Does it work with complex-valued inputs?


Numpy issues:

- Use numpy.finfo instead of machar

- Lots of nonzero/put/take combos, probably dates from Numeric.

  I think these should be replaced with boolean array indexing, to
  enhance readability and performance.

- numpy.min/max -> amin/amax


Minor stylistic issues in the code:

- `catch_msg` is not used for anything

- The original fortran documentation could be moved into the method
  docstrings.

- "if foo:", not "if (foo):"

- if foo:
     bar

  not

  if foo: bar

- "return foo", not "return(foo)"

- "# comment", not "## comment"

- It's not necessary to "import types"; you can just use 
  isinstance(x, float)


[clip]
> I have written a nose-test compatible test.  Could someone look at it
> and tell me if it meets the scipy style before I continue adding more
> tests from Craig's test-suite for the C version of his program?

The test style is OK; if it works with nosetests, it should be OK. 
Suggest using "import numpy as np", though, to stay in line with the rest 
of the code.

-- 
Pauli Virtanen



From charlesr.harris at gmail.com  Sat May  9 12:07:55 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 9 May 2009 10:07:55 -0600
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
In-Reply-To: <gu4925$hmn$2@ger.gmane.org>
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
	<gu4925$hmn$2@ger.gmane.org>
Message-ID: <e06186140905090907w181e966cw5d2b8012ad114026@mail.gmail.com>

On Sat, May 9, 2009 at 9:55 AM, Pauli Virtanen <pav at iki.fi> wrote:

> Fri, 08 May 2009 16:09:35 -0400, william ratcliff wrote:
>
> > Hi!  For a long time, there has been a standard package used by IDL
> > users for fitting functions to data called MPFIT:
> > http://cow.physics.wisc.edu/~craigm/idl/fitting.html<http://cow.physics.wisc.edu/%7Ecraigm/idl/fitting.html>
> > <http://cow.physics.wisc.edu/%7Ecraigm/idl/fitting.html>
> [clip]
> > http://drop.io/mpfitdrop
>
> Nice!
>
> However, I see some points where the code could be improved for better
> reusability and maintainability.
>
> If we lived in an ideal world with infinite time to polish everything,
> I'd like to see all of the points below addressed before landing this to
> Scipy. But since this would be lots of error-prone work, it's probably
> best to try to reach some compromise.
>
> Given these constraints, I'd still like to see at least the coding style
> and error handling fixed (which probably are not too difficult to
> change), in addition to having better test coverage. The rest could come
> later, even if we accrue yet more technical debt with this...
>
>
> First, broad maintenance concerns:
>
> - We already have `leastsq` from MINPACK. Having two MINPACK-derived
>  least squares fitting routines is not good.
>
>  So, I'd perhaps like to see the `leastsq` core part extracted out of
>  MPFIT, and the MPFIT interface implemented on top of it as a thin
>  wrapper, or the other way around.
>
>  Maybe, if the modifications made on MINPACK are small, they can be
>  backported to the Fortran code and MPFIT can be reimplemented on top
>  of `leastsq`.
>
>  Any thoughts on this?
>
> - What is the performance of the Python implementation as compared to the
>  Fortran code? For small data sets, the Python code is probably much
>  slower, but for large datasets is the performance is comparable?
>
> - Much of the code is Fortran written in Python: long routines,
>  goto-equivalents, 6-letter variable names.
>
>  Good commenting offset this, though.
>
>
> Then more specific points of refactoring:
>
> - The code implements QR factorization with column pivoting from scratch.
>
>  Perhaps some of this could be replaced with suitable LAPACK routines,
>  or with stuff from scipy.linalg. (Cf. DGEQPF, DGEQP3)
>
>  I'm not sure whether there's something suitable for qrsolve in LAPACK;
>  the triangular system solver could be replaced with DTRTRS.
>
>  Anyway, it might be useful to refactor qrfac and qrsolve out of MPFIT;
>  there may be other applications when it's useful to be able to solve
>  ||(A + I lambda) x - b||_2 = min! efficiently for multiple different
>  `lambda` in sequence.
>

This looks like Levenberg-Marquardt. There is a version already in MINPACK.

<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090509/5e2a4234/attachment.html>

From charlesr.harris at gmail.com  Sat May  9 12:17:18 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 9 May 2009 10:17:18 -0600
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
In-Reply-To: <e06186140905090907w181e966cw5d2b8012ad114026@mail.gmail.com>
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
	<gu4925$hmn$2@ger.gmane.org>
	<e06186140905090907w181e966cw5d2b8012ad114026@mail.gmail.com>
Message-ID: <e06186140905090917r3238f98ayf2ff042da70afcde@mail.gmail.com>

On Sat, May 9, 2009 at 10:07 AM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Sat, May 9, 2009 at 9:55 AM, Pauli Virtanen <pav at iki.fi> wrote:
>
>> Fri, 08 May 2009 16:09:35 -0400, william ratcliff wrote:
>>
>> > Hi!  For a long time, there has been a standard package used by IDL
>> > users for fitting functions to data called MPFIT:
>> > http://cow.physics.wisc.edu/~craigm/idl/fitting.html<http://cow.physics.wisc.edu/%7Ecraigm/idl/fitting.html>
>> > <http://cow.physics.wisc.edu/%7Ecraigm/idl/fitting.html>
>> [clip]
>> > http://drop.io/mpfitdrop
>>
>> Nice!
>>
>> However, I see some points where the code could be improved for better
>> reusability and maintainability.
>>
>> If we lived in an ideal world with infinite time to polish everything,
>> I'd like to see all of the points below addressed before landing this to
>> Scipy. But since this would be lots of error-prone work, it's probably
>> best to try to reach some compromise.
>>
>> Given these constraints, I'd still like to see at least the coding style
>> and error handling fixed (which probably are not too difficult to
>> change), in addition to having better test coverage. The rest could come
>> later, even if we accrue yet more technical debt with this...
>>
>>
>> First, broad maintenance concerns:
>>
>> - We already have `leastsq` from MINPACK. Having two MINPACK-derived
>>  least squares fitting routines is not good.
>>
>>  So, I'd perhaps like to see the `leastsq` core part extracted out of
>>  MPFIT, and the MPFIT interface implemented on top of it as a thin
>>  wrapper, or the other way around.
>>
>>  Maybe, if the modifications made on MINPACK are small, they can be
>>  backported to the Fortran code and MPFIT can be reimplemented on top
>>  of `leastsq`.
>>
>>  Any thoughts on this?
>>
>> - What is the performance of the Python implementation as compared to the
>>  Fortran code? For small data sets, the Python code is probably much
>>  slower, but for large datasets is the performance is comparable?
>>
>> - Much of the code is Fortran written in Python: long routines,
>>  goto-equivalents, 6-letter variable names.
>>
>>  Good commenting offset this, though.
>>
>>
>> Then more specific points of refactoring:
>>
>> - The code implements QR factorization with column pivoting from scratch.
>>
>>  Perhaps some of this could be replaced with suitable LAPACK routines,
>>  or with stuff from scipy.linalg. (Cf. DGEQPF, DGEQP3)
>>
>>  I'm not sure whether there's something suitable for qrsolve in LAPACK;
>>  the triangular system solver could be replaced with DTRTRS.
>>
>>  Anyway, it might be useful to refactor qrfac and qrsolve out of MPFIT;
>>  there may be other applications when it's useful to be able to solve
>>  ||(A + I lambda) x - b||_2 = min! efficiently for multiple different
>>  `lambda` in sequence.
>>
>
> This looks like Levenberg-Marquardt. There is a version already in MINPACK.
>

Not exactly though, the version quoted depends on the column ordering of A.
That doesn't look right.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090509/9b5301e8/attachment.html>

From pav at iki.fi  Sat May  9 14:01:03 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 9 May 2009 18:01:03 +0000 (UTC)
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
	<gu4925$hmn$2@ger.gmane.org>
	<e06186140905090907w181e966cw5d2b8012ad114026@mail.gmail.com>
Message-ID: <gu4gcv$qtg$1@ger.gmane.org>

Sat, 09 May 2009 10:07:55 -0600, Charles R Harris wrote:
> On Sat, May 9, 2009 at 9:55 AM, Pauli Virtanen <pav at iki.fi> wrote:
>>  Anyway, it might be useful to refactor qrfac and qrsolve out of PFIT;
>>  there may be other applications when it's useful to be able to solve
>>  ||(A + I lambda) x - b||_2 = min! efficiently for multiple different
>>  `lambda` in sequence.
>
> This looks like Levenberg-Marquardt. There is a version already in
> MINPACK.

I'm not sure I understand what you mean by the above.

Yes, there's an implementation of Levenberg-Marquardt in MINPACK, but the 
Python code discussed here is exactly a Python port of the MINPACK code, 
with some additional features bolted on.

For the LSQ problem, course there's numpy.linalg.lstsq, but the point in 
the Minpack version is that you can avoid repeated QR-factorization of 
the matrix A if you want to find the result for many values of the 
parameter lambda. I don't think there's anything like this exposed in 
Scipy; of course the MINPACK codes are there, but are they accessible 
from Python?

[clip]
> Not exactly though, the version quoted depends on the column ordering of
> A. That doesn't look right.

Ah, I put lambda in the wrong place in the LSQ problem I wrote there, it 
should have been

	|| (A; sqrt(lambda) I) x - (b; 0)||_2 = min!

which is equivalent to the Levenberg-Marquardt step computation
(A^T A + I lambda) x = A^T b. This should make more sense...

-- 
Pauli Virtanen



From charlesr.harris at gmail.com  Sat May  9 15:54:28 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 9 May 2009 13:54:28 -0600
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
In-Reply-To: <gu4gcv$qtg$1@ger.gmane.org>
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
	<gu4925$hmn$2@ger.gmane.org>
	<e06186140905090907w181e966cw5d2b8012ad114026@mail.gmail.com>
	<gu4gcv$qtg$1@ger.gmane.org>
Message-ID: <e06186140905091254q771aca91yb4de482d0996f2cf@mail.gmail.com>

On Sat, May 9, 2009 at 12:01 PM, Pauli Virtanen <pav at iki.fi> wrote:

> Sat, 09 May 2009 10:07:55 -0600, Charles R Harris wrote:
> > On Sat, May 9, 2009 at 9:55 AM, Pauli Virtanen <pav at iki.fi> wrote:
> >>  Anyway, it might be useful to refactor qrfac and qrsolve out of PFIT;
> >>  there may be other applications when it's useful to be able to solve
> >>  ||(A + I lambda) x - b||_2 = min! efficiently for multiple different
> >>  `lambda` in sequence.
> >
> > This looks like Levenberg-Marquardt. There is a version already in
> > MINPACK.
>
> I'm not sure I understand what you mean by the above.
>
> Yes, there's an implementation of Levenberg-Marquardt in MINPACK, but the
> Python code discussed here is exactly a Python port of the MINPACK code,
> with some additional features bolted on.
>
> For the LSQ problem, course there's numpy.linalg.lstsq, but the point in
> the Minpack version is that you can avoid repeated QR-factorization of
> the matrix A if you want to find the result for many values of the
> parameter lambda. I don't think there's anything like this exposed in
> Scipy; of course the MINPACK codes are there, but are they accessible
> from Python?
>
> [clip]
> > Not exactly though, the version quoted depends on the column ordering of
> > A. That doesn't look right.
>
> Ah, I put lambda in the wrong place in the LSQ problem I wrote there, it
> should have been
>
>        || (A; sqrt(lambda) I) x - (b; 0)||_2 = min!
>
> which is equivalent to the Levenberg-Marquardt step computation
> (A^T A + I lambda) x = A^T b. This should make more sense...
>

And in that case the svd of A will do what you need lambda-wise.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090509/44f062e9/attachment.html>

From pav at iki.fi  Sat May  9 16:09:40 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 9 May 2009 20:09:40 +0000 (UTC)
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
	<gu4925$hmn$2@ger.gmane.org>
	<e06186140905090907w181e966cw5d2b8012ad114026@mail.gmail.com>
	<gu4gcv$qtg$1@ger.gmane.org>
	<e06186140905091254q771aca91yb4de482d0996f2cf@mail.gmail.com>
Message-ID: <gu4nu4$qtg$2@ger.gmane.org>

Sat, 09 May 2009 13:54:28 -0600, Charles R Harris wrote:
[clip]
> And in that case the svd of A will do what you need lambda-wise.

Yes, but perhaps with more cost? I guess what you are saying boils down 
to that it's not worth the trouble to try to refactor this part of the 
converted Minpack code.

-- 
Pauli Virtanen



From charlesr.harris at gmail.com  Sat May  9 16:28:08 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 9 May 2009 14:28:08 -0600
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
In-Reply-To: <gu4nu4$qtg$2@ger.gmane.org>
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
	<gu4925$hmn$2@ger.gmane.org>
	<e06186140905090907w181e966cw5d2b8012ad114026@mail.gmail.com>
	<gu4gcv$qtg$1@ger.gmane.org>
	<e06186140905091254q771aca91yb4de482d0996f2cf@mail.gmail.com>
	<gu4nu4$qtg$2@ger.gmane.org>
Message-ID: <e06186140905091328p187b69b4me3cfbccbdf776c5b@mail.gmail.com>

On Sat, May 9, 2009 at 2:09 PM, Pauli Virtanen <pav at iki.fi> wrote:

> Sat, 09 May 2009 13:54:28 -0600, Charles R Harris wrote:
> [clip]
> > And in that case the svd of A will do what you need lambda-wise.
>
> Yes, but perhaps with more cost? I guess what you are saying boils down
> to that it's not worth the trouble to try to refactor this part of the
> converted Minpack code.


I couldn't say without actually doing a good look over the code. But I
expect the Levenberg-Marquardt step has been studied and most reasonable
methods explored. IIRC, some routines do use the svd for stability, but it's
suitability will depend on the number of variables as the first step is
reduction to the upper triangular form which is then decomposed. That first
step is the QR reduction, although it can also be done with a Gram-Schmidt
procedure.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090509/57ebbc1e/attachment.html>

From william.ratcliff at gmail.com  Sun May 10 01:21:32 2009
From: william.ratcliff at gmail.com (william ratcliff)
Date: Sun, 10 May 2009 01:21:32 -0400
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
In-Reply-To: <gu4925$hmn$2@ger.gmane.org>
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
	<gu4925$hmn$2@ger.gmane.org>
Message-ID: <827183970905092221o61ab7e18tf4be51d6d262cc5c@mail.gmail.com>

Thanks for the feedback!   Next week is rather busy for me, but I will try
to improve the exception handling and look into calling portions of minpack
directly (ex qr factorization) --but are they currently exposed to python?

Thanks,
William

On Sat, May 9, 2009 at 11:55 AM, Pauli Virtanen <pav at iki.fi> wrote:

> Fri, 08 May 2009 16:09:35 -0400, william ratcliff wrote:
>
> > Hi!  For a long time, there has been a standard package used by IDL
> > users for fitting functions to data called MPFIT:
> > http://cow.physics.wisc.edu/~craigm/idl/fitting.html<http://cow.physics.wisc.edu/%7Ecraigm/idl/fitting.html>
> > <http://cow.physics.wisc.edu/%7Ecraigm/idl/fitting.html>
> [clip]
> > http://drop.io/mpfitdrop
>
> Nice!
>
> However, I see some points where the code could be improved for better
> reusability and maintainability.
>
> If we lived in an ideal world with infinite time to polish everything,
> I'd like to see all of the points below addressed before landing this to
> Scipy. But since this would be lots of error-prone work, it's probably
> best to try to reach some compromise.
>
> Given these constraints, I'd still like to see at least the coding style
> and error handling fixed (which probably are not too difficult to
> change), in addition to having better test coverage. The rest could come
> later, even if we accrue yet more technical debt with this...
>
>
> First, broad maintenance concerns:
>
> - We already have `leastsq` from MINPACK. Having two MINPACK-derived
>  least squares fitting routines is not good.
>
>  So, I'd perhaps like to see the `leastsq` core part extracted out of
>  MPFIT, and the MPFIT interface implemented on top of it as a thin
>  wrapper, or the other way around.
>
>  Maybe, if the modifications made on MINPACK are small, they can be
>  backported to the Fortran code and MPFIT can be reimplemented on top
>  of `leastsq`.
>
>  Any thoughts on this?
>
> - What is the performance of the Python implementation as compared to the
>  Fortran code? For small data sets, the Python code is probably much
>  slower, but for large datasets is the performance is comparable?
>
> - Much of the code is Fortran written in Python: long routines,
>  goto-equivalents, 6-letter variable names.
>
>  Good commenting offset this, though.
>
>
> Then more specific points of refactoring:
>
> - The code implements QR factorization with column pivoting from scratch.
>
>  Perhaps some of this could be replaced with suitable LAPACK routines,
>  or with stuff from scipy.linalg. (Cf. DGEQPF, DGEQP3)
>
>  I'm not sure whether there's something suitable for qrsolve in LAPACK;
>  the triangular system solver could be replaced with DTRTRS.
>
>  Anyway, it might be useful to refactor qrfac and qrsolve out of MPFIT;
>  there may be other applications when it's useful to be able to solve
>  ||(A + I lambda) x - b||_2 = min! efficiently for multiple different
>  `lambda` in sequence.
>
> - fdjac2 should probably be refactored out of MPFIT; other optimization
>  algorithms that need Jacobians computed by forward differences can then
>  use it. Do we already have something similar in Scipy already?
>
> - `enorm` should be replaced with BLAS xNRM2; it does appropriate scaling
>  automatically and is probably much faster.
>
>  enorm = scipy.lib.blas.get_blas_funcs(['nrm2'], [some_array])
>
> - The long mpfit.__init__ routine should be split into smaller parts,
>  the function body is too long. I'm not sure exactly what parts, though.
>
>  Perhaps at least the covariance matrix computation and input argument
>  parsing should be separated.
>
> - "self.errmsg = 'ERROR ...; return'".
>  Probably should raise exceptions instead, at least for errors in input
>  parameters.
>
>  In general, I think the error handling should make better use of the
>  Python exception and warning system; using `print` is not a correct
>  way to signal an error in library code.
>
> - I'm not sure about implementing everything in a class. This tends to
>  couple tightly parts of the code that wouldn't really need such a strong
>  coupling.
>
> - Does it work with complex-valued inputs?
>
>
> Numpy issues:
>
> - Use numpy.finfo instead of machar
>
> - Lots of nonzero/put/take combos, probably dates from Numeric.
>
>  I think these should be replaced with boolean array indexing, to
>  enhance readability and performance.
>
> - numpy.min/max -> amin/amax
>
>
> Minor stylistic issues in the code:
>
> - `catch_msg` is not used for anything
>
> - The original fortran documentation could be moved into the method
>  docstrings.
>
> - "if foo:", not "if (foo):"
>
> - if foo:
>     bar
>
>  not
>
>  if foo: bar
>
> - "return foo", not "return(foo)"
>
> - "# comment", not "## comment"
>
> - It's not necessary to "import types"; you can just use
>  isinstance(x, float)
>
>
> [clip]
> > I have written a nose-test compatible test.  Could someone look at it
> > and tell me if it meets the scipy style before I continue adding more
> > tests from Craig's test-suite for the C version of his program?
>
> The test style is OK; if it works with nosetests, it should be OK.
> Suggest using "import numpy as np", though, to stay in line with the rest
> of the code.
>
> --
> Pauli Virtanen
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090510/dc08e674/attachment.html>

From charlesr.harris at gmail.com  Sun May 10 01:33:13 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 9 May 2009 23:33:13 -0600
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
In-Reply-To: <827183970905092221o61ab7e18tf4be51d6d262cc5c@mail.gmail.com>
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
	<gu4925$hmn$2@ger.gmane.org>
	<827183970905092221o61ab7e18tf4be51d6d262cc5c@mail.gmail.com>
Message-ID: <e06186140905092233u3b514d4lb55fbc39e75ed982@mail.gmail.com>

On Sat, May 9, 2009 at 11:21 PM, william ratcliff <
william.ratcliff at gmail.com> wrote:

> Thanks for the feedback!   Next week is rather busy for me, but I will try
> to improve the exception handling and look into calling portions of minpack
> directly (ex qr factorization) --but are they currently exposed to python?
>

qr is available in numpy, try help np.linalg.qr .

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090509/a3f18e5e/attachment.html>

From pav at iki.fi  Sun May 10 04:57:52 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 10 May 2009 08:57:52 +0000 (UTC)
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com>
	<gu4925$hmn$2@ger.gmane.org>
	<827183970905092221o61ab7e18tf4be51d6d262cc5c@mail.gmail.com>
	<e06186140905092233u3b514d4lb55fbc39e75ed982@mail.gmail.com>
Message-ID: <gu64ug$u4o$1@ger.gmane.org>

Sat, 09 May 2009 23:33:13 -0600, Charles R Harris wrote:

> On Sat, May 9, 2009 at 11:21 PM, william ratcliff <
> william.ratcliff at gmail.com> wrote:
> 
>> Thanks for the feedback!   Next week is rather busy for me, but I will
>> try to improve the exception handling and look into calling portions of
>> minpack directly (ex qr factorization) --but are they currently exposed
>> to python?
>>
>>
> qr is available in numpy, try help np.linalg.qr .

However, column-pivoted QR, which the Minpack code uses is not exposed 
currently, AFAIK. The LAPACK routines xGEQPF/xGEQP3 doing this could be 
easily exposed in scipy.lib.lapack by modifying	scipy/lib/lapack/
flapack_llsc.pyf.src

-- 
Pauli Virtanen



From stefan at sun.ac.za  Sun May 10 07:20:51 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 10 May 2009 13:20:51 +0200
Subject: [SciPy-dev] adding mpfit to scipy optimize (Please respond)
In-Reply-To: <gu64ug$u4o$1@ger.gmane.org>
References: <827183970905081309v37c46490j902a2663d9455ba@mail.gmail.com> 
	<gu4925$hmn$2@ger.gmane.org>
	<827183970905092221o61ab7e18tf4be51d6d262cc5c@mail.gmail.com> 
	<e06186140905092233u3b514d4lb55fbc39e75ed982@mail.gmail.com> 
	<gu64ug$u4o$1@ger.gmane.org>
Message-ID: <9457e7c80905100420o1980fa27y8485e18fb2f70820@mail.gmail.com>

Hi Pauli

2009/5/10 Pauli Virtanen <pav at iki.fi>:
> However, column-pivoted QR, which the Minpack code uses is not exposed
> currently, AFAIK. The LAPACK routines xGEQPF/xGEQP3 doing this could be
> easily exposed in scipy.lib.lapack by modifying scipy/lib/lapack/
> flapack_llsc.pyf.src

How would we go about exposing DGESVJ in SciPy (this algorithm is only
available in LAPACK 3.2, but it would be a very good addition!).

Regards
St?fan


From stefan at sun.ac.za  Sun May 10 10:49:05 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 10 May 2009 16:49:05 +0200
Subject: [SciPy-dev] Question about doc editing - here the right place?
In-Reply-To: <567397.78020.qm@web52101.mail.re2.yahoo.com>
References: <567397.78020.qm@web52101.mail.re2.yahoo.com>
Message-ID: <9457e7c80905100749x3458a0d9u3c303b520a0a1574@mail.gmail.com>

Hi David

2009/5/9 David Goldsmith <d_l_goldsmith at yahoo.com>:
> Hi, all. ?First, is this the right place to ask technical questions about editing the doc, or is there a sub-list for that?

Sure, either here or on numpy-discussion.


>?Second, for now, I'll ask my present question here: the "template" states "refer to variables using back-ticks, e.g. `var`," but I find that when I do so for variables that are also numpy function names (e.g., pv, nper, pmt, etc. in fv), the resulting formatting is a link, not the italics present in the function declaration;

Please keep variable names in `backticks`.  If there is a problem with
the presentation, we can fix it later, but it is very difficult to
estimate the author's intended meaning!

Regards
St?fan


From tpk at kraussfamily.org  Sun May 10 22:41:40 2009
From: tpk at kraussfamily.org (Tom K.)
Date: Mon, 11 May 2009 02:41:40 +0000 (UTC)
Subject: [SciPy-dev] RFR: #902 firwin upgrades
Message-ID: <loom.20090511T023202-872@post.gmane.org>

(I sent this to scipy-dev at scipy.org, thinking it would appear here, 
I am horribly confused - sorry for the duplication :-)

SIMPLE REVIEW NEEDED - SINGLE FUNCTION

Link to Trac ticket:  http://projects.scipy.org/scipy/ticket/902

This is adding a highpass, bandpass, bandstop, and multiband capability to 
signal.firwin.  I submitted a patch attached to the ticket to filter_design.py.
I also added a test class in tests/test_filter_design.py.

Let me know if there are any problems with the patch as this is my first patch 
to scipy and I am new to SVN.  

To do:
  - consider accepting a list of tuples of band edges for cutoff, since that 
    seems more natural for describing multiband filters.
  - figure out what to do about even length filters which are currently not 
    linear-phase (that is symmetric).

Also, I seem to be having trouble with the trac system, it's saying that the 
data base is locked.  It won't let me add notes or change the status to 
"needs review"...

Cheers,
  Tom K. 





From d_l_goldsmith at yahoo.com  Sun May 10 23:41:43 2009
From: d_l_goldsmith at yahoo.com (David Goldsmith)
Date: Sun, 10 May 2009 20:41:43 -0700 (PDT)
Subject: [SciPy-dev] Question about doc editing - here the right place?
In-Reply-To: <9457e7c80905100749x3458a0d9u3c303b520a0a1574@mail.gmail.com>
Message-ID: <630588.76352.qm@web52104.mail.re2.yahoo.com>


--- On Sun, 5/10/09, St?fan van der Walt <stefan at sun.ac.za> wrote:

> Please keep variable names in `backticks`.  If there is a
> problem with
> the presentation, we can fix it later, but it is very
> difficult to
> estimate the author's intended meaning!

OK, thanks!

DG

> 
> Regards
> St?fan


      


From charlesr.harris at gmail.com  Mon May 11 00:09:48 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sun, 10 May 2009 22:09:48 -0600
Subject: [SciPy-dev] RFR: #902 firwin upgrades
In-Reply-To: <loom.20090511T023202-872@post.gmane.org>
References: <loom.20090511T023202-872@post.gmane.org>
Message-ID: <e06186140905102109o7e9fb26brb6d279422b5d3bc1@mail.gmail.com>

On Sun, May 10, 2009 at 8:41 PM, Tom K. <tpk at kraussfamily.org> wrote:

> (I sent this to scipy-dev at scipy.org, thinking it would appear here,
> I am horribly confused - sorry for the duplication :-)
>
> SIMPLE REVIEW NEEDED - SINGLE FUNCTION
>
> Link to Trac ticket:  http://projects.scipy.org/scipy/ticket/902
>
> This is adding a highpass, bandpass, bandstop, and multiband capability to
> signal.firwin.  I submitted a patch attached to the ticket to
> filter_design.py.
> I also added a test class in tests/test_filter_design.py.
>
> Let me know if there are any problems with the patch as this is my first
> patch
> to scipy and I am new to SVN.
>

I think class BandPass is too general a name for a method based on
windowing. Have you looked at the remez filter design? You can design the
type of filters you are talking about using it also.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090510/9c3529aa/attachment.html>

From stefan at sun.ac.za  Mon May 11 16:54:56 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 11 May 2009 22:54:56 +0200
Subject: [SciPy-dev] RFR: Nonlinear solver rewrite
In-Reply-To: <gtvjh6$oln$4@ger.gmane.org>
References: <gtvjh6$oln$4@ger.gmane.org>
Message-ID: <9457e7c80905111354u43df2cf7ha3150549b20d2d07@mail.gmail.com>

Hi Pauli

2009/5/7 Pauli Virtanen <pav at iki.fi>:
> The large-scale nonlinear equation solver (nonlin.py) rewrite is nearing
> completion.
>
> Git branch is here:
>
> ? ?http://github.com/pv/scipy-work/commits/ticket/791-optimize-nonlin-rewrite

A big thumbs up and a *super* big thank you!  This is a big piece of work.

Cheers
St?fan


From tpk at kraussfamily.org  Mon May 11 21:44:32 2009
From: tpk at kraussfamily.org (Tom K.)
Date: Tue, 12 May 2009 01:44:32 +0000 (UTC)
Subject: [SciPy-dev] RFR: #902 firwin upgrades
References: <loom.20090511T023202-872@post.gmane.org>
	<e06186140905102109o7e9fb26brb6d279422b5d3bc1@mail.gmail.com>
Message-ID: <loom.20090512T011204-100@post.gmane.org>

Charles R Harris <charlesr.harris <at> gmail.com> writes:

> 
> 
> On Sun, May 10, 2009 at 8:41 PM, Tom K. <tpk <at> kraussfamily.org> wrote:
> (I sent this to scipy-dev <at> scipy.org, thinking it would appear here,
> I am horribly confused - sorry for the duplication 
> SIMPLE REVIEW NEEDED - SINGLE FUNCTION
> Link to Trac ticket:  http://projects.scipy.org/scipy/ticket/902
> This is adding a highpass, bandpass, bandstop, and multiband capability to
> signal.firwin.  I submitted a patch attached to the ticket to 
> filter_design.py.
> I also added a test class in tests/test_filter_design.py.
> Let me know if there are any problems with the patch as this is my first patch
> to scipy and I am new to SVN.
> 
> 
> I think class BandPass is too general a name for a method based on windowing.
> Have you looked at the remez filter design? You can design the type of 
> filters you are talking about using it also.
> Chuck 

For PassBand, which of these options sounds best:
 1) rename to _PassBand, or FirwinPassBand
 2) make local to firwin function
 3) remove class altogether and just go with a list of (left, right) tuples

I hadn't seen remez as the wrapper is in signaltools, not filter_design, but
thanks for the pointer.  See this thread 
http://www.nabble.com/firwin-upgrades-td23246480.html for discussion about
the interface to firwin.

The question I have, what is the right way to provide a convenient shorthand
for describing a multiband filter with just a vector?  The Signal Processing
Toolbox has this divergence which maybe we want to avoid.  E.g. fir1 takes a 
list of band edges and you leave 0 and 1 off (they are present implicitly)
and the magnitudes - which are not required to be specified by the user - 
alternate between 1 and 0; the stop, high, and DC-0 options
change the magnitude to start at 0 instead (whereas pass, low, DC-1 start
with the default magnitude of 1).  Remez and firls take only band edges
in pairs and require the same for the magnitude - this allows for 
a piece-wise linear desired function with unspecified regions in between.
I haven't looked at the newly submitted fir2 (which is sitting in a trac issue)
but the Signal Processing Toolbox has a list of edges and magnitudes with
0 and 1 implicit again IIRC.  fir2 doesn't require paired bands, you just 
specify a piecewise linear function and that gets interpolated over the entire 
band and inverted to the impulse domain.

The more I think about it, the functions seem slightly different in the
ways that the algorithms work and so it makes some sense that they each
accept arguments that are tailored to be the most natural for that
algorithm.

It looks like remez in scipy takes only constant bands.  Hence it may make
sense to unify it with firwin.  firwin would then allow varying amplitudes
(the feature I added has them all the same).  The cost is some simplicity,
but perhaps some "smarts" could be added that make the magnitude vector
implicit if unspecified (which could work for remez in the same way).
Is that what you would propose? 





From charlesr.harris at gmail.com  Mon May 11 23:34:50 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 11 May 2009 21:34:50 -0600
Subject: [SciPy-dev] RFR: #902 firwin upgrades
In-Reply-To: <loom.20090512T011204-100@post.gmane.org>
References: <loom.20090511T023202-872@post.gmane.org>
	<e06186140905102109o7e9fb26brb6d279422b5d3bc1@mail.gmail.com>
	<loom.20090512T011204-100@post.gmane.org>
Message-ID: <e06186140905112034y4fb2401emc770315cc11dcfe5@mail.gmail.com>

On Mon, May 11, 2009 at 7:44 PM, Tom K. <tpk at kraussfamily.org> wrote:

> Charles R Harris <charlesr.harris <at> gmail.com> writes:
>
> >
> >
> > On Sun, May 10, 2009 at 8:41 PM, Tom K. <tpk <at> kraussfamily.org>
> wrote:
> > (I sent this to scipy-dev <at> scipy.org, thinking it would appear here,
> > I am horribly confused - sorry for the duplication
> > SIMPLE REVIEW NEEDED - SINGLE FUNCTION
> > Link to Trac ticket:  http://projects.scipy.org/scipy/ticket/902
> > This is adding a highpass, bandpass, bandstop, and multiband capability
> to
> > signal.firwin.  I submitted a patch attached to the ticket to
> > filter_design.py.
> > I also added a test class in tests/test_filter_design.py.
> > Let me know if there are any problems with the patch as this is my first
> patch
> > to scipy and I am new to SVN.
> >
> >
> > I think class BandPass is too general a name for a method based on
> windowing.
> > Have you looked at the remez filter design? You can design the type of
> > filters you are talking about using it also.
> > Chuck
>
> For PassBand, which of these options sounds best:
>  1) rename to _PassBand, or FirwinPassBand
>  2) make local to firwin function
>  3) remove class altogether and just go with a list of (left, right) tuples
>

I can't decide at the moment ;) We could put all that stuff in a submodule
with win in it somewhere. If we go that route the name is less important.
Too bad something like FIRPassBandFilterDesignedByWindowingImpulseResponse
is a bit on the lengthy side...


>
> I hadn't seen remez as the wrapper is in signaltools, not filter_design,
> but
> thanks for the pointer.  See this thread
> http://www.nabble.com/firwin-upgrades-td23246480.html for discussion about
> the interface to firwin.
>

On the Nyquist problem with even length filters I'd be inclined to raise an
error.


> The question I have, what is the right way to provide a convenient
> shorthand
> for describing a multiband filter with just a vector?  The Signal
> Processing
> Toolbox has this divergence which maybe we want to avoid.  E.g. fir1 takes
> a
> list of band edges and you leave 0 and 1 off (they are present implicitly)
> and the magnitudes - which are not required to be specified by the user -
> alternate between 1 and 0; the stop, high, and DC-0 options
> change the magnitude to start at 0 instead (whereas pass, low, DC-1 start
> with the default magnitude of 1).  Remez and firls take only band edges
> in pairs and require the same for the magnitude - this allows for
> a piece-wise linear desired function with unspecified regions in between.
> I haven't looked at the newly submitted fir2 (which is sitting in a trac
> issue)
> but the Signal Processing Toolbox has a list of edges and magnitudes with
> 0 and 1 implicit again IIRC.  fir2 doesn't require paired bands, you just
> specify a piecewise linear function and that gets interpolated over the
> entire
> band and inverted to the impulse domain.
>

I think the lower level stuff should be quite general. Upper level
interfaces can simplify things for folks who just want quick and dirty. I
don't know much about the signal processing toolbox, if that is the
functionality we are shooting for.


>
> The more I think about it, the functions seem slightly different in the
> ways that the algorithms work and so it makes some sense that they each
> accept arguments that are tailored to be the most natural for that
> algorithm.
>
> It looks like remez in scipy takes only constant bands.  Hence it may make
> sense to unify it with firwin.  firwin would then allow varying amplitudes
> (the feature I added has them all the same).  The cost is some simplicity,
> but perhaps some "smarts" could be added that make the magnitude vector
> implicit if unspecified (which could work for remez in the same way).
> Is that what you would propose?
>

Hmm, the lower level remez simply takes a list of desired function values
and weights over a defined set of points in the transform domain, so in that
sense it is quite general as long as the desired frequency response is real.
It is a bit crippled at the moment, however, and only does
symmetric/antisymmetric filter designs whereas it could theoretically do
complex hermitean designs with a few small mods. I see that the sigtools
interface does do differentiators or passbands with specified gain and
relative ripple. I really don't see why it isn't in with the filter design
stuff where you would have noticed it. Maybe we need to submodules, one for
remez designs, the other for win designs, something like fir_windowed and
fir_equiripple.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090511/092b4079/attachment.html>

From cimrman3 at ntc.zcu.cz  Tue May 12 04:24:50 2009
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Tue, 12 May 2009 10:24:50 +0200
Subject: [SciPy-dev] ANN: SfePy 2009.2
Message-ID: <4A093252.4070706@ntc.zcu.cz>

I am pleased to announce the release of SfePy 2009.2.

SfePy (simple finite elements in Python) is a software, distributed 
under the BSD license, for solving systems of coupled partial 
differential equations by the finite element method. The code is based 
on NumPy and SciPy packages.

Mailing lists, issue tracking, git repository: http://sfepy.org
Home page: http://sfepy.kme.zcu.cz

People who contributed to this release: Vladimir Lukes.

Major improvements:
- new scripts:
   - isfepy (interactive sfepy) - customized IPython shell for quick 
"compute and visualize" work
   - postproc.py - a script to visualize (via mayavi2) results saved in 
result files
   - probe.py - a script to probe and plot results along geometrical 
objects (e.g. lines, rays) intersecting the mesh
- automatic html documentation generation via doxygen
- extended syntax of equations to allow boundary traces of variables
- fixed live plotting via multiprocessing for multi-core machines
- short input syntax for LCBC conditions, fields, integrals, materials 
and solvers
- new solvers:
    - Broyden and Anderson nonlinear solvers (SciPy implementation)
- new mesh readers:
   - Nastran (.bdf) format
   - Abaqus ascii (.inp)
- new example problems, tests and terms

Applications:
- phononic materials:
   - plotting improved
   - caching of eigen-problem solution and Christoffel acoustic tensor
- schroedinger.py:
   - choose and call DFT solver via solver interface

For more information on this release, see
http://sfepy.googlecode.com/svn/web/releases/2009.2_RELEASE_NOTES.txt

Best regards,
Robert Cimrman


From chanley at stsci.edu  Tue May 12 14:41:37 2009
From: chanley at stsci.edu (Christopher Hanley)
Date: Tue, 12 May 2009 14:41:37 -0400
Subject: [SciPy-dev] TABLES/STSDAS v3.10 and STScI_Python v2.8 now available]
Message-ID: <4A09C2E1.2080509@stsci.edu>

For those that may be interested.

Chris


-------- Original Message --------
Subject: [AstroPy] TABLES/STSDAS v3.10 and STScI_Python v2.8 now available
Date: Tue, 12 May 2009 14:38:25 -0400
From: Warren J. Hack <hack at stsci.edu>
To: iraf-announce at tucana.tuc.noao.edu, astropy at scipy.net, 
pyraf-regist at stsci.edu

    *******************************************************

     TABLES/STSDAS v3.10 and STSCI_PYTHON v2.8 NOW AVAILABLE

     *******************************************************


The Science Software Branch of the Space Telescope
Science Institute wishes to announce the availability of version
3.10 of the Space Telescope Science Data Analysis Software (STSDAS).
Concurrent with the STSDAS release, we have also released v3.10 of
the TABLES external package. Both of these packages are layered upon
IRAF as external packages; it is therefore necessary to obtain and
install IRAF (from http://iraf.noao.edu) before installing TABLES
and STSDAS.  STSDAS is linked against libraries in TABLES, so TABLES
must be installed before STSDAS.

This release of STSDAS contains changes to NICMOS, ACS, STIS,
WFPC/CALWP2, COS, WFC3, and the Dither packages. This release
includes the same versions of the calibration software used in
the pipeline to support all active HST instruments after Servicing
Mission 4, including the new instruments Wide-Field Camera 3 (WFC3)
and the Cosmic Origins Spectrograph (COS).

STSDAS also includes some Python-based tasks; running these requires the
STScI Python Libraries to be installed (all the non-Python tasks
can be used in the traditional way and do not require Python
or the STScI Python libraries). These tasks can only be run from
PyRAF using standard IRAF CL task syntax, or within Python scripts
using Python syntax.

The Science Software Branch of STScI also wishes to announce the
availability of version 2.8 of STSCI_PYTHON.  Stsci_python is a collection
of Python modules, packages and C extensions that has been developed to
provide a general astronomical data analysis infrastructure. They can
be used for Python tasks that are accessible from within STSDAS when
running under PyRAF or standalone from within Python.

All tasks in this release of STScI_Python rely on the use of numpy
for array operations. All references to numarray and Numeric have been
removed from the code, with support no longer being provided for use of
numarray or Numeric.

This distribution continues to develop packages with tasks specific
to each HST instrument, with updates being made to code for NICMOS
and COS.  This release also contains an updated version of Multidrizzle
and PyDrizzle as used by the HST ACS and WFPC2 pipelines, along with
significantly updated versions of PyFITS and PyRAF.  Use of the Python
tasks requires upgrading to this version of STSCI_PYTHON.

No More support for numarray
============================
Numarray is no longer used by tasks within STScI_Python. Compatibility
with numarray still exists in PyFITS and PyRAF in this release, but will
be eliminated entirely after this release. As a result, you are advised
to migrate to numpy as soon as possible, as no further support will be
provided for the use of numarray.

This switch to numpy represents a step towards unifying the Python
scientific community in the use of a common array package.  This is
a very desirable outcome despite the inconvenience in making the
switch. Information on making the switch to numpy from numarray can
be found in this document:

http://www.stsci.edu/resources/software_hardware/numarray/numarray2numpy.pdf


STSCI_Python on WINDOWS
=======================
This release includes a port of stsci_python that runs natively under
Windows.
Although IRAF is now available for running under Cygwin, PyRAF is not yet
available for use under Windows. This means that, in this release, all
the tasks in the STScI_Python distribution will be run under Python
(or preferably, IPython) on Windows using Python syntax rather than
a parameter-based interface provided by PyRAF and IRAF.

The primary requirement is that the Windows executables were built
against and, therefore, can run only under Python 2.5.


WHERE TO OBTAIN THIS SOFTWARE
=============================
STSDAS/TABLES v3.10 can be downloaded from the STSDAS web site

http://www.stsci.edu/resources/software_hardware/stsdas/download

Installation instructions are available on the web site. Precompiled
binaries also exist for some of the ports supported by NOAO, including
Sun Solaris, RedHat Linux and Mac OS X.

STSCI_PYTHON v 2.8 be downloaded from the PyRAF web site

http://www.stsci.edu/resources/software_hardware/pyraf/stsci_python/current/download

Installation instructions are available from the PyRAF web site, with
support for installations on Sun Solaris, RedHat Linux, Mac OS X,
and Windows.

Please contact us through e-mail (help at stsci.edu), by telephone at
(410) 338-1082.

_______________________________________________
AstroPy mailing list
AstroPy at scipy.org
http://mail.scipy.org/mailman/listinfo/astropy

-- 
Christopher Hanley
Senior Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338


From jsseabold at gmail.com  Tue May 12 15:53:52 2009
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 12 May 2009 15:53:52 -0400
Subject: [SciPy-dev] Question about editing docs
Message-ID: <c048da1c0905121253y1309c747mf3f5e754e4e2da5e@mail.gmail.com>

I've been going through scipy.stats.mstats and the tests to get a
better feel for masked arrays and how they are handled for statistical
functions.  I noticed what I believed to be an error in the examples
of the docstring for mstats_basic.count_tied_groups, which I corrected
here: <http://docs.scipy.org/scipy/docs/scipy.stats.mstats_basic.count_tied_groups/>.

My question is (as you can see) I had to also correct some formatting
(and there is still some to be done), but I don't think I should be
doing this kind of thing incrementally.  Or should I?  Is there
another way I can edit multiple docstrings to get them to conform to
the numpy standard and rst when I notice something or is this best
left to someone else?  I see that mstats_basic "Needs Editing."  Sorry
if I'm missing something obvious.  This wouldn't be the first
time...today.

Cheers,

Skipper


From josef.pktd at gmail.com  Tue May 12 16:47:41 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 12 May 2009 16:47:41 -0400
Subject: [SciPy-dev] Question about editing docs
In-Reply-To: <c048da1c0905121253y1309c747mf3f5e754e4e2da5e@mail.gmail.com>
References: <c048da1c0905121253y1309c747mf3f5e754e4e2da5e@mail.gmail.com>
Message-ID: <1cd32cbb0905121347u4f694796x7754f9cdd772d697@mail.gmail.com>

On Tue, May 12, 2009 at 3:53 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> I've been going through scipy.stats.mstats and the tests to get a
> better feel for masked arrays and how they are handled for statistical
> functions. ?I noticed what I believed to be an error in the examples
> of the docstring for mstats_basic.count_tied_groups, which I corrected
> here: <http://docs.scipy.org/scipy/docs/scipy.stats.mstats_basic.count_tied_groups/>.
>
> My question is (as you can see) I had to also correct some formatting
> (and there is still some to be done), but I don't think I should be
> doing this kind of thing incrementally. ?Or should I? ?Is there
> another way I can edit multiple docstrings to get them to conform to
> the numpy standard and rst when I notice something or is this best
> left to someone else? ?I see that mstats_basic "Needs Editing." ?Sorry
> if I'm missing something obvious. ?This wouldn't be the first
> time...today.
>
> Cheers,
>
> Skipper

I think any edit and improvement to the docs is good. Often when I
check the help and I find a mistake, I just correct it, without fully
editing, and reviewing the doc string. I usually just fix additionally
the main formatting errors.

If OK to apply: is "No", then the changes won't go into the source in
svn immediately, and incremental changes accumulate.

Note, in the example you could also remove the >>>  in the output
lines, and additionally any extra functions are supposed to be
imported in the doc string, eg. ma, and masked wasn't defined.

>>> count_tied_groups(z)
{2: 1, 3: 2}
>>> import numpy.ma as ma
>>> z = ma.array([0, 0, 1, 2, 2, 2, 3, 3, 4, 5, 6])
>>> z[[1,-1]] = ma.masked
>>> count_tied_groups(z, use_missing=True)
{2: 2, 3: 1}

You could correct any these basic formatting problems to get them
closer to the numpy standard.
If you find something suspicious, and you don't have time to figure it
out, you can also just add a comment.

Josef


From bsouthey at gmail.com  Tue May 12 16:57:22 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Tue, 12 May 2009 15:57:22 -0500
Subject: [SciPy-dev] Question about editing docs
In-Reply-To: <c048da1c0905121253y1309c747mf3f5e754e4e2da5e@mail.gmail.com>
References: <c048da1c0905121253y1309c747mf3f5e754e4e2da5e@mail.gmail.com>
Message-ID: <4A09E2B2.2090406@gmail.com>

Skipper Seabold wrote:
> I've been going through scipy.stats.mstats and the tests to get a
> better feel for masked arrays and how they are handled for statistical
> functions.  I noticed what I believed to be an error in the examples
> of the docstring for mstats_basic.count_tied_groups, which I corrected
> here: <http://docs.scipy.org/scipy/docs/scipy.stats.mstats_basic.count_tied_groups/>.
>
> My question is (as you can see) I had to also correct some formatting
> (and there is still some to be done), but I don't think I should be
> doing this kind of thing incrementally.  Or should I?  Is there
> another way I can edit multiple docstrings to get them to conform to
> the numpy standard and rst when I notice something or is this best
> left to someone else?  I see that mstats_basic "Needs Editing."  Sorry
> if I'm missing something obvious.  This wouldn't be the first
> time...today.
>
> Cheers,
>
> Skipper
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   
Hi,
My understanding is that virtually none of the stats functions have been 
edited as part of the SciPy documentation marathon. This means that 
nobody has edited the original docstring.  Any effort to make the 
docstrings conform to the standards is welcome.

Based on the Numpy documentation marathon, in my view you really need to 
do each function separately. You can cut and paste followed any 
necessary editing especially between scipy.stats and scipy.mstats 
because the former use 'standard' numpy arrays and the latter uses 
masked arrays. Ideally there should be a single version and the 
associated docstring where possible (so far I have only managed to 
supply patches for gmean and hmean).

Some of the functions are or should be depreciated but this information 
is not in the docstring. For example the mean and std in stats are 
depreciated because these duplicate the Numpy functions and I created 
some tickets for others that should also be depreciated. So I would 
recommend against documenting the depreciated functions.

Regards
Bruce






From d_l_goldsmith at yahoo.com  Tue May 12 17:45:02 2009
From: d_l_goldsmith at yahoo.com (David Goldsmith)
Date: Tue, 12 May 2009 14:45:02 -0700 (PDT)
Subject: [SciPy-dev] Question about editing docs
In-Reply-To: <4A09E2B2.2090406@gmail.com>
Message-ID: <652759.64379.qm@web52105.mail.re2.yahoo.com>


--- On Tue, 5/12/09, Bruce Southey <bsouthey at gmail.com> wrote:

> Based on the Numpy documentation marathon, in my view you
> really need to 
> do each function separately. You can cut and paste followed

Is there an existing team handling SciPy/NumPy Documentation Editor (the program) feature requests, bug fixes, etc. (i.e., program development needs)?  If so, is there a submission protocol/ticketing system in place?  I think this would be useful globally (i.e., not just within the stats functions).  FWIW,

DG



      


From d_l_goldsmith at yahoo.com  Wed May 13 02:31:26 2009
From: d_l_goldsmith at yahoo.com (David Goldsmith)
Date: Tue, 12 May 2009 23:31:26 -0700 (PDT)
Subject: [SciPy-dev] Note at beginning of "Other Math" Milestones
In-Reply-To: <4A09E2B2.2090406@gmail.com>
Message-ID: <640662.15712.qm@web52102.mail.re2.yahoo.com>


"the routines in scimath should probably just refer to the standard numpy routines" - i.e., do we want their docstrings to recommend that the numpy routines be used instead, or simply refer the user to those functions' docstrings for help?

DG


      


From pav at iki.fi  Wed May 13 02:33:56 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 13 May 2009 06:33:56 +0000 (UTC)
Subject: [SciPy-dev] Question about editing docs
References: <c048da1c0905121253y1309c747mf3f5e754e4e2da5e@mail.gmail.com>
	<1cd32cbb0905121347u4f694796x7754f9cdd772d697@mail.gmail.com>
Message-ID: <gudpkk$qp5$1@ger.gmane.org>

Tue, 12 May 2009 16:47:41 -0400, josef.pktd kirjoitti:
[clip]
> If OK to apply: is "No", then the changes won't go into the source in
> svn immediately, and incremental changes accumulate.

This is not actually so: all changes must still be applied manually. 
Nothing goes directly to SVN, because of security reasons.

"OK to apply" is a flag, used by reviewers to easily mark those changes 
that seem sensible improvements over the old docstring.

-- 
Pauli Virtanen



From pav at iki.fi  Wed May 13 02:38:00 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 13 May 2009 06:38:00 +0000 (UTC)
Subject: [SciPy-dev] Question about editing docs
References: <4A09E2B2.2090406@gmail.com>
	<652759.64379.qm@web52105.mail.re2.yahoo.com>
Message-ID: <gudps8$qp5$2@ger.gmane.org>

Tue, 12 May 2009 14:45:02 -0700, David Goldsmith kirjoitti:

> --- On Tue, 5/12/09, Bruce Southey <bsouthey at gmail.com> wrote:
> 
>> Based on the Numpy documentation marathon, in my view you really need
>> to
>> do each function separately. You can cut and paste followed
> 
> Is there an existing team handling SciPy/NumPy Documentation Editor (the
> program) feature requests, bug fixes, etc. (i.e., program development
> needs)?  If so, is there a submission protocol/ticketing system in
> place?  I think this would be useful globally (i.e., not just within the
> stats functions).  FWIW,

The editor bugs and issues are handled here:

	http://code.google.com/p/pydocweb/

I think issues specifically concerning Scipy and Numpy documentation are 
best either discussed on this ML, if they are very broad; in the comments 
system in the documentation editor; or filed in the Numpy/Scipy Trac.

-- 
Pauli Virtanen



From pav at iki.fi  Wed May 13 02:45:44 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 13 May 2009 06:45:44 +0000 (UTC)
Subject: [SciPy-dev] Question about editing docs
References: <c048da1c0905121253y1309c747mf3f5e754e4e2da5e@mail.gmail.com>
	<4A09E2B2.2090406@gmail.com>
Message-ID: <gudqao$qp5$3@ger.gmane.org>

Tue, 12 May 2009 15:57:22 -0500, Bruce Southey kirjoitti:
[clip]
> My understanding is that virtually none of the stats functions have been
> edited as part of the SciPy documentation marathon. This means that
> nobody has edited the original docstring.  Any effort to make the
> docstrings conform to the standards is welcome.

I don't think there has been an "official" push for a Scipy documentation 
marathon. The doc editor is there, but nobody has checked eg. for

- Which docstrings are automatically generated.

  For example, it may well be that scipy.stats.mstats copies its
  docstrings from scipy.stats. There's no reliable way to determine this
  automatically. So, I suggest checking this from the scipy.stats.mstats
  source code, before editing the docstring of each function.

  At least the functions gmean, hmean, and mode, do copy their docstrings.

- Which docstrings are not worth much concern.

-- 
Pauli Virtanen



From stefan at sun.ac.za  Wed May 13 03:11:31 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 13 May 2009 09:11:31 +0200
Subject: [SciPy-dev] Note at beginning of "Other Math" Milestones
In-Reply-To: <640662.15712.qm@web52102.mail.re2.yahoo.com>
References: <4A09E2B2.2090406@gmail.com>
	<640662.15712.qm@web52102.mail.re2.yahoo.com>
Message-ID: <9457e7c80905130011g15937ceveb436429f97cad5a@mail.gmail.com>

2009/5/13 David Goldsmith <d_l_goldsmith at yahoo.com>:
> "the routines in scimath should probably just refer to the standard numpy routines" - i.e., do we want their docstrings to recommend that the numpy routines be used instead, or simply refer the user to those functions' docstrings for help?

numpy.lib.scimath is convenient in that it provides output in the
complex domain, if required:

In [5]: np.sqrt(-1)
Out[5]: nan

In [6]: np.lib.scimath.sqrt(-1)
Out[6]: 1j

The documentation for these functions may refer to the standard numpy
routines, but should document the differences explicitly.

Regards
St?fan


From d_l_goldsmith at yahoo.com  Wed May 13 03:17:26 2009
From: d_l_goldsmith at yahoo.com (David Goldsmith)
Date: Wed, 13 May 2009 00:17:26 -0700 (PDT)
Subject: [SciPy-dev] Note at beginning of "Other Math" Milestones
In-Reply-To: <9457e7c80905130011g15937ceveb436429f97cad5a@mail.gmail.com>
Message-ID: <800220.55218.qm@web52106.mail.re2.yahoo.com>


Excellent, Stefan, thanks.

DG

--- On Wed, 5/13/09, St?fan van der Walt <stefan at sun.ac.za> wrote:

> numpy.lib.scimath is convenient in that it provides output
> in the
> complex domain, if required:
> 
> In [5]: np.sqrt(-1)
> Out[5]: nan
> 
> In [6]: np.lib.scimath.sqrt(-1)
> Out[6]: 1j
> 
> The documentation for these functions may refer to the
> standard numpy
> routines, but should document the differences explicitly.


> 
> Regards
> St?fan


      


From bsouthey at gmail.com  Wed May 13 09:26:00 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Wed, 13 May 2009 08:26:00 -0500
Subject: [SciPy-dev] Question about editing docs
In-Reply-To: <gudps8$qp5$2@ger.gmane.org>
References: <4A09E2B2.2090406@gmail.com>	<652759.64379.qm@web52105.mail.re2.yahoo.com>
	<gudps8$qp5$2@ger.gmane.org>
Message-ID: <4A0ACA68.9010203@gmail.com>

Pauli Virtanen wrote:
> Tue, 12 May 2009 14:45:02 -0700, David Goldsmith kirjoitti:
>
>   
>> --- On Tue, 5/12/09, Bruce Southey <bsouthey at gmail.com> wrote:
>>
>>     
>>> Based on the Numpy documentation marathon, in my view you really need
>>> to
>>> do each function separately. You can cut and paste followed
>>>       
>> Is there an existing team handling SciPy/NumPy Documentation Editor (the
>> program) feature requests, bug fixes, etc. (i.e., program development
>> needs)?  If so, is there a submission protocol/ticketing system in
>> place?  I think this would be useful globally (i.e., not just within the
>> stats functions).  FWIW,
>>     
For the most part this is handled by the Trac system  
(http://projects.scipy.org/numpy and http://projects.scipy.org/scipy) by 
creating tickets preferably with code.
> The editor bugs and issues are handled here:
>
> 	http://code.google.com/p/pydocweb/
>
> I think issues specifically concerning Scipy and Numpy documentation are 
> best either discussed on this ML, if they are very broad; in the comments 
> system in the documentation editor; or filed in the Numpy/Scipy Trac.
>
>   
Emails to the appropriate list are also effective for different reasons 
from understanding code to adding new code.

Bruce




From josef.pktd at gmail.com  Thu May 14 10:37:27 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 14 May 2009 10:37:27 -0400
Subject: [SciPy-dev] backporting changes in scipy.stats
Message-ID: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>

Most of the changes in scipy.stats could be backported to scipy 7.1

changesets: 5591, 5592, 5597, 5599, 5601, 5600, 5602, 5604, 5623,
5636, 5637, 5638, 5646, 5672(replay) are (mostly) bugfixes and
improvement in numerical precision.
changesets: 5598, 5645 are additional tests
changesets: 5562, 5564, 5565, 5566, 5567, 5587 are switching to
np.mean, np.var and can be backported or not

I never did any backporting and I'm currently not set up for it.
I would appreciate if someone could backport these changes, otherwise
I will try to do it next weekend.

Thanks,

Josef


From charlesr.harris at gmail.com  Thu May 14 14:58:03 2009
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 14 May 2009 12:58:03 -0600
Subject: [SciPy-dev] backporting changes in scipy.stats
In-Reply-To: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>
References: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>
Message-ID: <e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>

On Thu, May 14, 2009 at 8:37 AM, <josef.pktd at gmail.com> wrote:

> Most of the changes in scipy.stats could be backported to scipy 7.1
>
> changesets: 5591, 5592, 5597, 5599, 5601, 5600, 5602, 5604, 5623,
> 5636, 5637, 5638, 5646, 5672(replay) are (mostly) bugfixes and
> improvement in numerical precision.
> changesets: 5598, 5645 are additional tests
> changesets: 5562, 5564, 5565, 5566, 5567, 5587 are switching to
> np.mean, np.var and can be backported or not
>
> I never did any backporting and I'm currently not set up for it.
> I would appreciate if someone could backport these changes, otherwise
> I will try to do it next weekend.
>
> Thanks,
>

If the changes are all specific to a certain file/directory and there are no
changes you *don't* want to backport from that file/directory, the easiest
thing to do is check out the 7.1.x branch, copy over the files/directory
from the trunk, and check in the result. There are doubtless more
sophisticated ways if you happen to have the whole repository including
branches checked out, but I think that is the simplest and safest.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090514/d4a8cb2c/attachment.html>

From josef.pktd at gmail.com  Thu May 14 15:34:03 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 14 May 2009 15:34:03 -0400
Subject: [SciPy-dev] backporting changes in scipy.stats
In-Reply-To: <e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>
References: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>
	<e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>
Message-ID: <1cd32cbb0905141234k59294b1fl7961351627bb17c2@mail.gmail.com>

On Thu, May 14, 2009 at 2:58 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Thu, May 14, 2009 at 8:37 AM, <josef.pktd at gmail.com> wrote:
>>
>> Most of the changes in scipy.stats could be backported to scipy 7.1
>>
>> changesets: 5591, 5592, 5597, 5599, 5601, 5600, 5602, 5604, 5623,
>> 5636, 5637, 5638, 5646, 5672(replay) are (mostly) bugfixes and
>> improvement in numerical precision.
>> changesets: 5598, 5645 are additional tests
>> changesets: 5562, 5564, 5565, 5566, 5567, 5587 are switching to
>> np.mean, np.var and can be backported or not
>>
>> I never did any backporting and I'm currently not set up for it.
>> I would appreciate if someone could backport these changes, otherwise
>> I will try to do it next weekend.
>>
>> Thanks,
>
> If the changes are all specific to a certain file/directory and there are no
> changes you *don't* want to backport from that file/directory, the easiest
> thing to do is check out the 7.1.x branch, copy over the files/directory
> from the trunk, and check in the result. There are doubtless more
> sophisticated ways if you happen to have the whole repository including
> branches checked out, but I think that is the simplest and safest.
>



Thanks, that sounds easy enough. I have usually only the trunk checked out.
There is one set of changes, the removal of stats.mean, var, std and
median, that we don't want to backport, but this could be confined to
the four functions. I'm very used to winmerge, so I can do this by the
end of the week.

Josef


From bsouthey at gmail.com  Thu May 14 16:16:43 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Thu, 14 May 2009 15:16:43 -0500
Subject: [SciPy-dev] backporting changes in scipy.stats
In-Reply-To: <1cd32cbb0905141234k59294b1fl7961351627bb17c2@mail.gmail.com>
References: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>	<e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>
	<1cd32cbb0905141234k59294b1fl7961351627bb17c2@mail.gmail.com>
Message-ID: <4A0C7C2B.9040903@gmail.com>

josef.pktd at gmail.com wrote:
> On Thu, May 14, 2009 at 2:58 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>   
>> On Thu, May 14, 2009 at 8:37 AM, <josef.pktd at gmail.com> wrote:
>>     
>>> Most of the changes in scipy.stats could be backported to scipy 7.1
>>>
>>> changesets: 5591, 5592, 5597, 5599, 5601, 5600, 5602, 5604, 5623,
>>> 5636, 5637, 5638, 5646, 5672(replay) are (mostly) bugfixes and
>>> improvement in numerical precision.
>>> changesets: 5598, 5645 are additional tests
>>> changesets: 5562, 5564, 5565, 5566, 5567, 5587 are switching to
>>> np.mean, np.var and can be backported or not
>>>
>>> I never did any backporting and I'm currently not set up for it.
>>> I would appreciate if someone could backport these changes, otherwise
>>> I will try to do it next weekend.
>>>
>>> Thanks,
>>>       
>> If the changes are all specific to a certain file/directory and there are no
>> changes you *don't* want to backport from that file/directory, the easiest
>> thing to do is check out the 7.1.x branch, copy over the files/directory
>> from the trunk, and check in the result. There are doubtless more
>> sophisticated ways if you happen to have the whole repository including
>> branches checked out, but I think that is the simplest and safest.
>>
>>     
>
>
>
> Thanks, that sounds easy enough. I have usually only the trunk checked out.
> There is one set of changes, the removal of stats.mean, var, std and
> median, that we don't want to backport, but this could be confined to
> the four functions. I'm very used to winmerge, so I can do this by the
> end of the week.
>
> Josef
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   
Hi,
Can you also add ticket 877 (Depreciation of samplevar in scipy.stats) 
since it is the same reason as removing stats.std?

The other possibility is ticket 906 Depreciate erfc function.

Bruce



From josef.pktd at gmail.com  Thu May 14 16:31:06 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 14 May 2009 16:31:06 -0400
Subject: [SciPy-dev] backporting changes in scipy.stats
In-Reply-To: <4A0C7C2B.9040903@gmail.com>
References: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>
	<e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>
	<1cd32cbb0905141234k59294b1fl7961351627bb17c2@mail.gmail.com>
	<4A0C7C2B.9040903@gmail.com>
Message-ID: <1cd32cbb0905141331n12a0f032l3b14c38c1609fe06@mail.gmail.com>

On Thu, May 14, 2009 at 4:16 PM, Bruce Southey <bsouthey at gmail.com> wrote:
> josef.pktd at gmail.com wrote:
>> On Thu, May 14, 2009 at 2:58 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>
>>> On Thu, May 14, 2009 at 8:37 AM, <josef.pktd at gmail.com> wrote:
>>>
>>>> Most of the changes in scipy.stats could be backported to scipy 7.1
>>>>
>>>> changesets: 5591, 5592, 5597, 5599, 5601, 5600, 5602, 5604, 5623,
>>>> 5636, 5637, 5638, 5646, 5672(replay) are (mostly) bugfixes and
>>>> improvement in numerical precision.
>>>> changesets: 5598, 5645 are additional tests
>>>> changesets: 5562, 5564, 5565, 5566, 5567, 5587 are switching to
>>>> np.mean, np.var and can be backported or not
>>>>
>>>> I never did any backporting and I'm currently not set up for it.
>>>> I would appreciate if someone could backport these changes, otherwise
>>>> I will try to do it next weekend.
>>>>
>>>> Thanks,
>>>>
>>> If the changes are all specific to a certain file/directory and there are no
>>> changes you *don't* want to backport from that file/directory, the easiest
>>> thing to do is check out the 7.1.x branch, copy over the files/directory
>>> from the trunk, and check in the result. There are doubtless more
>>> sophisticated ways if you happen to have the whole repository including
>>> branches checked out, but I think that is the simplest and safest.
>>>
>>>
>>
>>
>>
>> Thanks, that sounds easy enough. I have usually only the trunk checked out.
>> There is one set of changes, the removal of stats.mean, var, std and
>> median, that we don't want to backport, but this could be confined to
>> the four functions. I'm very used to winmerge, so I can do this by the
>> end of the week.
>>
>> Josef
>> _______________________________________________
>> Scipy-dev mailing list
>> Scipy-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
> Hi,
> Can you also add ticket 877 (Depreciation of samplevar in scipy.stats)
> since it is the same reason as removing stats.std?
>
> The other possibility is ticket 906 Depreciate erfc function.
>

What's the policy for depreciation warnings, should they be
backported?  I can commit and backport them, but do the functions
remain available during the 0.8.x cycle.

erfc, samplevar and samplestd are redundant, but I'm starting to
change my mind a bit about the removal of stats.var and stats.mean.

see my comments to http://projects.scipy.org/scipy/ticket/604

Josef


From pav at iki.fi  Thu May 14 16:57:00 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 14 May 2009 20:57:00 +0000 (UTC)
Subject: [SciPy-dev] backporting changes in scipy.stats
References: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>
	<e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>
	<1cd32cbb0905141234k59294b1fl7961351627bb17c2@mail.gmail.com>
	<4A0C7C2B.9040903@gmail.com>
	<1cd32cbb0905141331n12a0f032l3b14c38c1609fe06@mail.gmail.com>
Message-ID: <gui0is$4s2$1@ger.gmane.org>

Thu, 14 May 2009 16:31:06 -0400, josef.pktd wrote:
[clip]
> What's the policy for depreciation warnings, should they be backported? 
> I can commit and backport them, but do the functions remain available
> during the 0.8.x cycle.

I don't think they should be ported, stuff should be deprecated only in 
major versions, I think.

-- 
Pauli Virtanen



From bsouthey at gmail.com  Thu May 14 17:01:46 2009
From: bsouthey at gmail.com (Bruce Southey)
Date: Thu, 14 May 2009 16:01:46 -0500
Subject: [SciPy-dev] backporting changes in scipy.stats
In-Reply-To: <1cd32cbb0905141331n12a0f032l3b14c38c1609fe06@mail.gmail.com>
References: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>	<e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>	<1cd32cbb0905141234k59294b1fl7961351627bb17c2@mail.gmail.com>	<4A0C7C2B.9040903@gmail.com>
	<1cd32cbb0905141331n12a0f032l3b14c38c1609fe06@mail.gmail.com>
Message-ID: <4A0C86BA.8020503@gmail.com>

josef.pktd at gmail.com wrote:
> On Thu, May 14, 2009 at 4:16 PM, Bruce Southey <bsouthey at gmail.com> wrote:
>   
>> josef.pktd at gmail.com wrote:
>>     
>>> On Thu, May 14, 2009 at 2:58 PM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>>
>>>       
>>>> On Thu, May 14, 2009 at 8:37 AM, <josef.pktd at gmail.com> wrote:
>>>>
>>>>         
>>>>> Most of the changes in scipy.stats could be backported to scipy 7.1
>>>>>
>>>>> changesets: 5591, 5592, 5597, 5599, 5601, 5600, 5602, 5604, 5623,
>>>>> 5636, 5637, 5638, 5646, 5672(replay) are (mostly) bugfixes and
>>>>> improvement in numerical precision.
>>>>> changesets: 5598, 5645 are additional tests
>>>>> changesets: 5562, 5564, 5565, 5566, 5567, 5587 are switching to
>>>>> np.mean, np.var and can be backported or not
>>>>>
>>>>> I never did any backporting and I'm currently not set up for it.
>>>>> I would appreciate if someone could backport these changes, otherwise
>>>>> I will try to do it next weekend.
>>>>>
>>>>> Thanks,
>>>>>
>>>>>           
>>>> If the changes are all specific to a certain file/directory and there are no
>>>> changes you *don't* want to backport from that file/directory, the easiest
>>>> thing to do is check out the 7.1.x branch, copy over the files/directory
>>>> from the trunk, and check in the result. There are doubtless more
>>>> sophisticated ways if you happen to have the whole repository including
>>>> branches checked out, but I think that is the simplest and safest.
>>>>
>>>>
>>>>         
>>>
>>> Thanks, that sounds easy enough. I have usually only the trunk checked out.
>>> There is one set of changes, the removal of stats.mean, var, std and
>>> median, that we don't want to backport, but this could be confined to
>>> the four functions. I'm very used to winmerge, so I can do this by the
>>> end of the week.
>>>
>>> Josef
>>> _______________________________________________
>>> Scipy-dev mailing list
>>> Scipy-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>>
>>>       
>> Hi,
>> Can you also add ticket 877 (Depreciation of samplevar in scipy.stats)
>> since it is the same reason as removing stats.std?
>>
>> The other possibility is ticket 906 Depreciate erfc function.
>>
>>     
>
> What's the policy for depreciation warnings, should they be
> backported?  I can commit and backport them, but do the functions
> remain available during the 0.8.x cycle.
>
> erfc, samplevar and samplestd are redundant, but I'm starting to
> change my mind a bit about the removal of stats.var and stats.mean.
>
> see my comments to http://projects.scipy.org/scipy/ticket/604
>
> Josef
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>   
Hi,
Okay, I misunderstood what you were suggesting.

I do not think that redundant functions should be present as per the Zen 
of Python:
'There should be one-- and preferably only one --obvious way to do it'.

The issue of using weights is a different aspect that should really be 
handled at a low level not the way done in that ticket (and I will add 
my comments to the ticket). I'll avoid the NaN issue as that has been 
raised in the past.

Bruce






From jsseabold at gmail.com  Thu May 14 20:26:32 2009
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 14 May 2009 20:26:32 -0400
Subject: [SciPy-dev] Broken Link in "Getting Started"
Message-ID: <c048da1c0905141726r7f7241m94f36061a58c5ad6@mail.gmail.com>

Sorry if this is the wrong place to report this, but I was curious if
I'm getting the most out of using ipython and saw a link to an article
that is broken on the "Getting Started" page here
<http://www.scipy.org/Getting_Started?action=show#head-9aed725bd569d40f625240b2b6ec710550ff14b9>
at the end of the second to last bullet point.  Was curious if #549
here <http://amath.colorado.edu/activities/preprints/> might be the
same article and a more stable link.

Cheers,

Skipper


From millman at berkeley.edu  Thu May 14 20:34:30 2009
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 14 May 2009 17:34:30 -0700
Subject: [SciPy-dev] Broken Link in "Getting Started"
In-Reply-To: <c048da1c0905141726r7f7241m94f36061a58c5ad6@mail.gmail.com>
References: <c048da1c0905141726r7f7241m94f36061a58c5ad6@mail.gmail.com>
Message-ID: <c7009a550905141734i1741d2c6l2e87855698e79dad@mail.gmail.com>

On Thu, May 14, 2009 at 5:26 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> Sorry if this is the wrong place to report this, but I was curious if
> I'm getting the most out of using ipython and saw a link to an article
> that is broken on the "Getting Started" page here
> <http://www.scipy.org/Getting_Started?action=show#head-9aed725bd569d40f625240b2b6ec710550ff14b9>
> at the end of the second to last bullet point. ?Was curious if #549
> here <http://amath.colorado.edu/activities/preprints/> might be the
> same article and a more stable link.

I would use this link:
https://cirl.berkeley.edu/fperez/papers/ipython-cise-final.pdf


From matthew.brett at gmail.com  Thu May 14 22:24:13 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 14 May 2009 19:24:13 -0700
Subject: [SciPy-dev] ReST tutorial - where should it go?
Message-ID: <1e2af89e0905141924r241d3b59j7717625181238f68@mail.gmail.com>

Hello,

I think this one is for the documentation gurus.

For a talk I did today, I made a quick introduction to matlab IO in
ReST, copy here:

https://cirl.berkeley.edu/mb312/nss/matlab_io.html

It seemed like a good idea to put it somewhere in the Scipy source
tree docs, but I wasn't quite sure where...

Thanks a lot,

Matthew


From david at ar.media.kyoto-u.ac.jp  Thu May 14 22:14:06 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 15 May 2009 11:14:06 +0900
Subject: [SciPy-dev] backporting changes in scipy.stats
In-Reply-To: <gui0is$4s2$1@ger.gmane.org>
References: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>	<e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>	<1cd32cbb0905141234k59294b1fl7961351627bb17c2@mail.gmail.com>	<4A0C7C2B.9040903@gmail.com>	<1cd32cbb0905141331n12a0f032l3b14c38c1609fe06@mail.gmail.com>
	<gui0is$4s2$1@ger.gmane.org>
Message-ID: <4A0CCFEE.4010508@ar.media.kyoto-u.ac.jp>

Pauli Virtanen wrote:
> Thu, 14 May 2009 16:31:06 -0400, josef.pktd wrote:
> [clip]
>   
>> What's the policy for depreciation warnings, should they be backported? 
>> I can commit and backport them, but do the functions remain available
>> during the 0.8.x cycle.
>>     
>
> I don't think they should be ported, stuff should be deprecated only in 
> major versions, I think.
>   

Yes, nothing related to deprecation should change. A good rule is: if it
does not close a ticket, think twice before committing a change to
0.7.x. Another important point is that 0.7.x should be compatible with 
numpy >= 1.2.0.

cheers,

David



From josef.pktd at gmail.com  Thu May 14 22:49:34 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 14 May 2009 22:49:34 -0400
Subject: [SciPy-dev] ReST tutorial - where should it go?
In-Reply-To: <1e2af89e0905141924r241d3b59j7717625181238f68@mail.gmail.com>
References: <1e2af89e0905141924r241d3b59j7717625181238f68@mail.gmail.com>
Message-ID: <1cd32cbb0905141949q70fb9525r5695a84d63c1f5b8@mail.gmail.com>

On Thu, May 14, 2009 at 10:24 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hello,
>
> I think this one is for the documentation gurus.
>
> For a talk I did today, I made a quick introduction to matlab IO in
> ReST, copy here:
>
> https://cirl.berkeley.edu/mb312/nss/matlab_io.html
>
> It seemed like a good idea to put it somewhere in the Scipy source
> tree docs, but I wasn't quite sure where...
>

I think we can add the missing subpackages to the tutorial, e.g.

http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/io.rst/

but I never remember how to create a new rst file in the doc editor.

Josef


From josef.pktd at gmail.com  Thu May 14 23:15:04 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 14 May 2009 23:15:04 -0400
Subject: [SciPy-dev] backporting changes in scipy.stats
In-Reply-To: <4A0CCFEE.4010508@ar.media.kyoto-u.ac.jp>
References: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>
	<e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>
	<1cd32cbb0905141234k59294b1fl7961351627bb17c2@mail.gmail.com>
	<4A0C7C2B.9040903@gmail.com>
	<1cd32cbb0905141331n12a0f032l3b14c38c1609fe06@mail.gmail.com>
	<gui0is$4s2$1@ger.gmane.org> <4A0CCFEE.4010508@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0905142015q2c3d3404wdecb015d652e0ef9@mail.gmail.com>

On Thu, May 14, 2009 at 10:14 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Pauli Virtanen wrote:
>> Thu, 14 May 2009 16:31:06 -0400, josef.pktd wrote:
>> [clip]
>>
>>> What's the policy for depreciation warnings, should they be backported?
>>> I can commit and backport them, but do the functions remain available
>>> during the 0.8.x cycle.
>>>
>>
>> I don't think they should be ported, stuff should be deprecated only in
>> major versions, I think.
>>
>
> Yes, nothing related to deprecation should change. A good rule is: if it
> does not close a ticket, think twice before committing a change to
> 0.7.x. Another important point is that 0.7.x should be compatible with
> numpy >= 1.2.0.

There shouldn't be any problems with the changes, I still have numpy
1.2.0, so I can run the testsuite before committing.

The main changes that are not bug fixes or numerical precision
improvements are the removal of the usage of  stats.mean, var, ... in
the other stats functions, which are not supposed to cause any
problems.

Josef


From matthew.brett at gmail.com  Fri May 15 00:43:48 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 14 May 2009 21:43:48 -0700
Subject: [SciPy-dev] ReST tutorial - where should it go?
In-Reply-To: <1cd32cbb0905141949q70fb9525r5695a84d63c1f5b8@mail.gmail.com>
References: <1e2af89e0905141924r241d3b59j7717625181238f68@mail.gmail.com>
	<1cd32cbb0905141949q70fb9525r5695a84d63c1f5b8@mail.gmail.com>
Message-ID: <1e2af89e0905142143s6f6e9bb5vcc3d12fb911b1d11@mail.gmail.com>

Hi,

>> I think this one is for the documentation gurus.
>>
>> For a talk I did today, I made a quick introduction to matlab IO in
>> ReST, copy here:
>>
>> https://cirl.berkeley.edu/mb312/nss/matlab_io.html
>>
>> It seemed like a good idea to put it somewhere in the Scipy source
>> tree docs, but I wasn't quite sure where...
>>
>
> I think we can add the missing subpackages to the tutorial, e.g.
>
> http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/io.rst/

Thanks - done...

Matthew


From gael.varoquaux at normalesup.org  Fri May 15 01:08:54 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 15 May 2009 07:08:54 +0200
Subject: [SciPy-dev] Broken Link in "Getting Started"
In-Reply-To: <c048da1c0905141726r7f7241m94f36061a58c5ad6@mail.gmail.com>
References: <c048da1c0905141726r7f7241m94f36061a58c5ad6@mail.gmail.com>
Message-ID: <20090515050854.GC27068@phare.normalesup.org>

On Thu, May 14, 2009 at 08:26:32PM -0400, Skipper Seabold wrote:
> Sorry if this is the wrong place to report this, but I was curious if
> I'm getting the most out of using ipython and saw a link to an article
> that is broken on the "Getting Started" page here
> <http://www.scipy.org/Getting_Started?action=show#head-9aed725bd569d40f625240b2b6ec710550ff14b9>
> at the end of the second to last bullet point.  

Fixed! Thanks for reporting.

Ga?l


From stefan at sun.ac.za  Fri May 15 01:35:55 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 15 May 2009 07:35:55 +0200
Subject: [SciPy-dev] ReST tutorial - where should it go?
In-Reply-To: <1e2af89e0905142143s6f6e9bb5vcc3d12fb911b1d11@mail.gmail.com>
References: <1e2af89e0905141924r241d3b59j7717625181238f68@mail.gmail.com> 
	<1cd32cbb0905141949q70fb9525r5695a84d63c1f5b8@mail.gmail.com> 
	<1e2af89e0905142143s6f6e9bb5vcc3d12fb911b1d11@mail.gmail.com>
Message-ID: <9457e7c80905142235v7571e4b7nb494eae712c947cf@mail.gmail.com>

2009/5/15 Matthew Brett <matthew.brett at gmail.com>:
>> I think we can add the missing subpackages to the tutorial, e.g.
>>
>> http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/io.rst/

Thanks, Matthew!  I never knew structs were actually arrays of structs.

Cheers
St?fan


From matthew.brett at gmail.com  Fri May 15 03:02:56 2009
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 15 May 2009 00:02:56 -0700
Subject: [SciPy-dev] ReST tutorial - where should it go?
In-Reply-To: <9457e7c80905142235v7571e4b7nb494eae712c947cf@mail.gmail.com>
References: <1e2af89e0905141924r241d3b59j7717625181238f68@mail.gmail.com>
	<1cd32cbb0905141949q70fb9525r5695a84d63c1f5b8@mail.gmail.com>
	<1e2af89e0905142143s6f6e9bb5vcc3d12fb911b1d11@mail.gmail.com>
	<9457e7c80905142235v7571e4b7nb494eae712c947cf@mail.gmail.com>
Message-ID: <1e2af89e0905150002j6be9b788h518d9f05dcb0647e@mail.gmail.com>

I sense some gentle South African teasing here, but I'm too tired to
understand, so I say:

Great - glad I taught you something new about structs!

See you,

Matthew

2009/5/14 St?fan van der Walt <stefan at sun.ac.za>:
> 2009/5/15 Matthew Brett <matthew.brett at gmail.com>:
>>> I think we can add the missing subpackages to the tutorial, e.g.
>>>
>>> http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/io.rst/
>
> Thanks, Matthew! ?I never knew structs were actually arrays of structs.
>
> Cheers
> St?fan
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>


From david at ar.media.kyoto-u.ac.jp  Fri May 15 04:58:49 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 15 May 2009 17:58:49 +0900
Subject: [SciPy-dev] backporting changes in scipy.stats
In-Reply-To: <1cd32cbb0905142015q2c3d3404wdecb015d652e0ef9@mail.gmail.com>
References: <1cd32cbb0905140737n7f96e85cjaee02e147b1cb641@mail.gmail.com>	<e06186140905141158m5c54125i161cef095be90f99@mail.gmail.com>	<1cd32cbb0905141234k59294b1fl7961351627bb17c2@mail.gmail.com>	<4A0C7C2B.9040903@gmail.com>	<1cd32cbb0905141331n12a0f032l3b14c38c1609fe06@mail.gmail.com>	<gui0is$4s2$1@ger.gmane.org>
	<4A0CCFEE.4010508@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905142015q2c3d3404wdecb015d652e0ef9@mail.gmail.com>
Message-ID: <4A0D2EC9.8000405@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
> On Thu, May 14, 2009 at 10:14 PM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>   
>> Pauli Virtanen wrote:
>>     
>>> Thu, 14 May 2009 16:31:06 -0400, josef.pktd wrote:
>>> [clip]
>>>
>>>       
>>>> What's the policy for depreciation warnings, should they be backported?
>>>> I can commit and backport them, but do the functions remain available
>>>> during the 0.8.x cycle.
>>>>
>>>>         
>>> I don't think they should be ported, stuff should be deprecated only in
>>> major versions, I think.
>>>
>>>       
>> Yes, nothing related to deprecation should change. A good rule is: if it
>> does not close a ticket, think twice before committing a change to
>> 0.7.x. Another important point is that 0.7.x should be compatible with
>> numpy >= 1.2.0.
>>     
>
> There shouldn't be any problems with the changes, I still have numpy
> 1.2.0, so I can run the testsuite before committing.
>   

ok, good.

> The main changes that are not bug fixes or numerical precision
> improvements are the removal of the usage of  stats.mean, var, ... in
> the other stats functions, which are not supposed to cause any
> problems.
>   

I myself try to backport things only if they fix something and don't
break something else. Changing code is always more risky than not
changing it, so the golden rule is to do as little as possible :)

Thank you very much for your work !

David


From stefan at sun.ac.za  Fri May 15 06:17:39 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 15 May 2009 12:17:39 +0200
Subject: [SciPy-dev] ReST tutorial - where should it go?
In-Reply-To: <1e2af89e0905150002j6be9b788h518d9f05dcb0647e@mail.gmail.com>
References: <1e2af89e0905141924r241d3b59j7717625181238f68@mail.gmail.com> 
	<1cd32cbb0905141949q70fb9525r5695a84d63c1f5b8@mail.gmail.com> 
	<1e2af89e0905142143s6f6e9bb5vcc3d12fb911b1d11@mail.gmail.com> 
	<9457e7c80905142235v7571e4b7nb494eae712c947cf@mail.gmail.com> 
	<1e2af89e0905150002j6be9b788h518d9f05dcb0647e@mail.gmail.com>
Message-ID: <9457e7c80905150317j1b7120acr4243f7b5d569171@mail.gmail.com>

2009/5/15 Matthew Brett <matthew.brett at gmail.com>:
> I sense some gentle South African teasing here, but I'm too tired to
> understand, so I say:
>
> Great - glad I taught you something new about structs!

Always so suspicious of the former colonies! :)

But no, I was being serious: thanks for all your work on scipy.io and
for writing the tutorial!

I was surprised, because I thought the Octave structure:

octave:2> s = struct('foo', 1, 'bar', 2)
s =
{
  foo =  1
  bar =  2
}

would be read in by SciPy to a variable x that could be accessed:

x.foo and x.bar

In the tutorial I saw that you had to use

x[0, 0].foo and x[0, 0].bar

instead, which didn't make sense to me, mainly because in the 3 years
since I last used Octave I forgot that one could do:

octave:4> s = struct('foo', {1, 2}, 'bar', {4, 5})
s =
{
  foo =

  (,
    [1] =  1
    [2] =  2
  ,)

  bar =

  (,
    [1] =  4
    [2] =  5
  ,)

}

In Octave you then see that

octave:6> size(s)
ans =

   1   2

octave:7> s(1, 1)
ans =
{
  bar =  4
  foo =  1
}

after which the way SciPy indexes the data makes a lot more sense!

Enjoy the weekend!
St?fan


From josef.pktd at gmail.com  Fri May 15 09:39:23 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 15 May 2009 09:39:23 -0400
Subject: [SciPy-dev] binary compatibility question
Message-ID: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>

I didn't follow the discussion of binary compatibility between
different versions very closely.

I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
against numpy 1.2.1 and I get an immediate segfault when trying to run
scipy.test(). I get the usual 4 failures when I run scipy with numpy
1.3.0

Is this expected?

(I didn't do it on purpose, the version mismatch was just a
consequence of how I wanted to setup my testing environment for scipy
0.7.1 and is easily fixed)

Josef


From josef.pktd at gmail.com  Fri May 15 10:19:08 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 15 May 2009 10:19:08 -0400
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
Message-ID: <1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>

On Fri, May 15, 2009 at 9:39 AM,  <josef.pktd at gmail.com> wrote:
> I didn't follow the discussion of binary compatibility between
> different versions very closely.
>
> I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
> against numpy 1.2.1 and I get an immediate segfault when trying to run
> scipy.test(). I get the usual 4 failures when I run scipy with numpy
> 1.3.0
>
> Is this expected?
>
> (I didn't do it on purpose, the version mismatch was just a
> consequence of how I wanted to setup my testing environment for scipy
> 0.7.1 and is easily fixed)
>

I build scipy 0.7.1 against numpy 1.2.1 (official installer) and I get
the same segfault. I get it already trying to `import scipy.stats`

Josef


From josef.pktd at gmail.com  Fri May 15 10:44:28 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 15 May 2009 10:44:28 -0400
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
Message-ID: <1cd32cbb0905150744v1dc9cf78x9ba2ab4935d2198c@mail.gmail.com>

On Fri, May 15, 2009 at 10:19 AM,  <josef.pktd at gmail.com> wrote:
> On Fri, May 15, 2009 at 9:39 AM, ?<josef.pktd at gmail.com> wrote:
>> I didn't follow the discussion of binary compatibility between
>> different versions very closely.
>>
>> I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
>> against numpy 1.2.1 and I get an immediate segfault when trying to run
>> scipy.test(). I get the usual 4 failures when I run scipy with numpy
>> 1.3.0
>>
>> Is this expected?
>>
>> (I didn't do it on purpose, the version mismatch was just a
>> consequence of how I wanted to setup my testing environment for scipy
>> 0.7.1 and is easily fixed)
>>
>
> I build scipy 0.7.1 against numpy 1.2.1 (official installer) and I get
> the same segfault. I get it already trying to `import scipy.stats`
>

Just to see if it works, I tried to build scipy trunk against numpy
1.2.1, but this time I get a compiler error

building 'scipy.signal.sigtools' extension
compiling C sources
C compiler: gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes

creating build\temp.win32-2.5\Release\scipy\signal
compile options: '-Ibuild\src.win32-2.5\scipy\signal -IC:\Programs\Python25\lib\
site-packages\numpy\core\include -IC:\Programs\Python25\include -IC:\Programs\Py
thon25\PC -c'
gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes -Ibuild\src.win32-2.5\scipy\signal
 -IC:\Programs\Python25\lib\site-packages\numpy\core\include -IC:\Programs\Pytho
n25\include -IC:\Programs\Python25\PC -c scipy\signal\sigtoolsmodule.c -o build\
temp.win32-2.5\Release\scipy\signal\sigtoolsmodule.o
scipy\signal\sigtoolsmodule.c:161: warning: parameter names (without types) in f
unction declaration
scipy\signal\sigtoolsmodule.c:161: warning: parameter names (without types) in f
unction declaration
scipy\signal\sigtoolsmodule.c:161: error: redefinition of parameter 'NPY_UNUSED'

scipy\signal\sigtoolsmodule.c:161: error: previous definition of 'NPY_UNUSED' wa
s here
scipy\signal\sigtoolsmodule.c:162: warning: parameter names (without types) in f
unction declaration
<....>

So it looks like, I cannot test backports to scipy 0.7.1 against numpy 1.2.1

Josef


From pav at iki.fi  Fri May 15 10:53:33 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 15 May 2009 14:53:33 +0000 (UTC)
Subject: [SciPy-dev] binary compatibility question
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
	<1cd32cbb0905150744v1dc9cf78x9ba2ab4935d2198c@mail.gmail.com>
Message-ID: <gujvld$t04$1@ger.gmane.org>

Fri, 15 May 2009 10:44:28 -0400, josef.pktd kirjoitti:

> On Fri, May 15, 2009 at 10:19 AM,  <josef.pktd at gmail.com> wrote:
>> On Fri, May 15, 2009 at 9:39 AM, ?<josef.pktd at gmail.com> wrote:
>>> I didn't follow the discussion of binary compatibility between
>>> different versions very closely.
>>>
>>> I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
>>> against numpy 1.2.1 and I get an immediate segfault when trying to run
>>> scipy.test(). I get the usual 4 failures when I run scipy with numpy
>>> 1.3.0

The segfaults are because we forgot to increment the ABI version.
There's a thread about this on the Numpy list.

[clip]
>> I build scipy 0.7.1 against numpy 1.2.1 (official installer) and I get
>> the same segfault. I get it already trying to `import scipy.stats`
>
> Just to see if it works, I tried to build scipy trunk against numpy
> 1.2.1, but this time I get a compiler error
[clip]
> So it looks like, I cannot test backports to scipy 0.7.1 against numpy
> 1.2.1

Scipy 0.8 will not be compatible with Numpy 1.2.x, and so the trunk will 
not build with Numpy 1.2.x.

If you want to test your backports to 0.7.x, I think you should anyway 
compile the 0.7.x branch, to which you are backporting the fixes, not the 
trunk from which you backport them.

-- 
Pauli Virtanen



From thouis at broad.mit.edu  Fri May 15 10:46:24 2009
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Fri, 15 May 2009 10:46:24 -0400
Subject: [SciPy-dev] Should ndimage.measurements.* should return lists
	if index is a list?
In-Reply-To: <6c17e6f50905081143t61917ed0ub9e0e9e5326eafd1@mail.gmail.com>
References: <6c17e6f50904171157i2bc822c1q3e666cce860b6992@mail.gmail.com> 
	<9457e7c80905011021t6ae12883ncce41bb945d24a6e@mail.gmail.com> 
	<6c17e6f50905011102n3bc43acds4fdd0dd470d3c0f1@mail.gmail.com> 
	<9457e7c80905011107i16e09749saec0d33879cc7393@mail.gmail.com> 
	<1cd32cbb0905011311x73c6eff1u9d0580a60e6d9746@mail.gmail.com> 
	<6c17e6f50905041300q25dca2c0m93ec9f6cd74f6391@mail.gmail.com> 
	<1cd32cbb0905041343u433fcf49o68ef89928a0a71f0@mail.gmail.com> 
	<6c17e6f50905041411wdc519f7o5812618beab518f3@mail.gmail.com> 
	<1cd32cbb0905051154t689ad8cdw898ce5e97b9872fc@mail.gmail.com> 
	<6c17e6f50905081143t61917ed0ub9e0e9e5326eafd1@mail.gmail.com>
Message-ID: <6c17e6f50905150746p49fc2e60j564848273efbaf44@mail.gmail.com>

I was going to rewrite ndimage.histogram() to be faster, but I think
it's worth leaving as is to track numpy.histogram() which has been in
flux recently.

I filed a scipy ticket: 945

Ray Jones


On Fri, May 8, 2009 at 14:43, Thouis (Ray) Jones <thouis at broad.mit.edu> wrote:
> I've just pushed a new version to
> http://broad.mit.edu/~thouis/scipy.git ndimage_measurements_rewrite
>
> This includes rewrites of: ?sum, mean, variance, standard_deviation,
> minimum, maximum, minimum_position, maximum_position, center_of_mass,
> and histogram in pure python. ?It also adds a labeled_comprehension()
> function.
>
> The histogram() code could be much improved, speed-wise. ?It uses
> labeled_comprehension and numpy.histogram, but could use similar
> bincount tricks as have been done for sum et al.
>
> Some timing comparisons below, with test script attached.
>
> Ray
>
>
> Timing information (1K x 1K image), secs per call.
> New:
> variance 0.0620911860466
> minimum 0.48847284317
> maximum_position 0.588409199715
> extrema 0.813310542107
> center_of_mass 0.220837605
> histogram 5.70109256506
>
> Old:
> variance 0.0640315508842
> minimum 0.0331222701073
> maximum_position 1.07506107569
> extrema 2.27282817364
> center_of_mass 0.0554035758972
> histogram 0.0865430688858
>
>
>
>
> On Tue, May 5, 2009 at 14:54, ?<josef.pktd at gmail.com> wrote:
>> On Mon, May 4, 2009 at 5:11 PM, Thouis (Ray) Jones <thouis at broad.mit.edu> wrote:
>>> On Mon, May 4, 2009 at 16:43, ?<josef.pktd at gmail.com> wrote:
>>>> The use case, I wrote my application for, also allowed for labels that
>>>> are strings. The return inverse of unique1d is very flexible since it
>>>> can handle many different types including structured arrays, but for
>>>> numeric labels, I just found out, that it is a large speed penalty. If
>>>> the labels are already integers or can be easily converted to
>>>> integers, then not using unique1d to create the integer labels should
>>>> be much faster.
>>>
>>> Using bincount probably requires calling unique1d (or some equivalent)
>>> for this usage, since otherwise the call to bincount could create a
>>> very large output (2^32 or even 2^64 bins).
>>>
>>> Ray Jones
>>
>> In my version, I started to use a conditional call to unique1d, if the
>> type of the labels is not int (to handle float and strings). This
>> could be extended to a very large max(labels). ?If the label type is
>> int then, I just use bincount without unique1d.
>>
>> I tried it with 50% unused numbers, and the speed of bincount is
>> comparable to ndimage, labels are generated with
>> factors = (100+2*np.random.randint(nfact,size=nobs))
>> #.astype('S6')#.astype(float)
>> I tried with up to 4 million observations and 10000 unique labels.
>>
>> I'm starting to get confused with the many different versions, that I
>> have now for this, the two typical examples are below. ndimage looks
>> very picky which arrays it accepts, every once in a while I get a not
>> implemented type exception or even a segfault.
>>
>> I also attach my full timing and test script.
>>
>> Josef
>>
>>
>> def labelmean_ndi(factors, values, index=None):
>> ? ?# works also for string labels in ys, but requires 1D
>> ? ?# from mailing list scipy-user 2009-02-11
>> ? ?unilinv = np.asanyarray(factors)
>>
>> ? ?if not np.issubdtype(unilinv.dtype, int):
>> ? ? ? ?unil, unilinv = np.unique1d(factors, return_inverse=1)
>> ? ? ? ?if index is None:
>> ? ? ? ? ? ?index = np.arange(len(unil)) #unil
>> ? ?elif index is None:
>> ? ? ? ?index = np.arange(np.max(unilinv)+ 1)
>>
>> ? ?labelmeans = np.array(ndimage.mean(values, labels=unilinv, index=index))
>> ? ?return labelmeans
>>
>> def groupstatsbin(factors, values, ddof=0, stat='mvnx', drop=True):
>> ? ?'''uses np.bincount, assumes factors/labels are integers,
>> ? ?create labels with unique1d return_inverse if string labels
>> ? ?'''
>> ? ?#n = len(factors)
>>
>> ? ?rind = np.asanyarray(factors)
>> ? ?#note: bincount uses complete list of integers, maybe range(max(factor)+1)
>> ? ?if not np.issubdtype(rind.dtype, int):
>> ? ? ? ?ix, rind = np.unique1d(factors, return_inverse=1)
>> ## ? ?else:
>> ## ? ? ? ?rind = factors
>> ? ?#print rind.shape
>> ? ?res = []
>> ? ?if 'c' in stat or 'm' in stat or 'v' in stat:
>> ? ? ? ?gcount = np.bincount(rind)
>> ? ? ? ?if drop:
>> ? ? ? ? ? ?keepidx = np.nonzero(gcount)
>> ? ? ? ?if 'c' in stat:
>> ? ? ? ? ? ?if drop:
>> ? ? ? ? ? ? ? ?res.append(gcount[keepidx])
>> ? ? ? ? ? ?else:
>> ? ? ? ? ? ? ? ?res.append(gcount)
>>
>> ? ?if 'm' in stat or 'v' in stat:
>> ? ? ? ?gmean = np.bincount(rind, weights=values)/ (1.0*gcount)
>> ? ? ? ?if np.max(np.abs(gmean)) > 1e6:
>> ? ? ? ? ? ?# improve numerical precision if means are large
>> ? ? ? ? ? ?# (to "cheat" on NIST anova test examples)
>> ? ? ? ? ? ?meanarr = gmean[rind]
>> ? ? ? ? ? ?gmean += np.bincount(rind, weights=(values-meanarr)) / (1.0*gcount)
>> ? ? ? ?if 'm' in stat:
>> ? ? ? ? ? ?if drop:
>> ? ? ? ? ? ? ? ?res.append(gmean[keepidx])
>> ? ? ? ? ? ?else:
>> ? ? ? ? ? ? ? ?res.append(gmean)
>>
>> ? ?if 'v' in stat:
>> ? ? ? ?meanarr = gmean[rind]
>> ? ? ? ?withinvar = np.bincount(rind, weights=(values-meanarr)**2) /
>> (1.0*gcount-ddof)
>> ? ? ? ?if drop:
>> ? ? ? ? ? ?res.append(withinvar[keepidx])
>> ? ? ? ?else:
>> ? ? ? ? ? ?res.append(withinvar)
>>
>> ? ?if 'n' in stat or 'x' in stat:
>> ? ? ? ?#calculate min, max per factor
>> ? ? ? ?sortind = np.lexsort((values, rind))
>> ? ? ? ?fs = rind[sortind]
>> ? ? ? ?vs = values[sortind]
>> ? ? ? ?fsd = np.hstack((np.inf,np.diff(fs),np.inf))
>> ? ?if 'n' in stat:
>> ? ? ? ?#minimum
>> ? ? ? ?res.append(vs[fsd[:-1] != 0])
>> ? ?if 'x' in stat:
>> ? ? ? ?#maximum
>> ? ? ? ?res.append(vs[fsd[1:] != 0])
>> ? ?return res
>>
>> _______________________________________________
>> Scipy-dev mailing list
>> Scipy-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>>
>


From josef.pktd at gmail.com  Fri May 15 11:45:45 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 15 May 2009 11:45:45 -0400
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <gujvld$t04$1@ger.gmane.org>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
	<1cd32cbb0905150744v1dc9cf78x9ba2ab4935d2198c@mail.gmail.com>
	<gujvld$t04$1@ger.gmane.org>
Message-ID: <1cd32cbb0905150845v23117c9bv9123adb248143724@mail.gmail.com>

On Fri, May 15, 2009 at 10:53 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Fri, 15 May 2009 10:44:28 -0400, josef.pktd kirjoitti:
>
>> On Fri, May 15, 2009 at 10:19 AM, ?<josef.pktd at gmail.com> wrote:
>>> On Fri, May 15, 2009 at 9:39 AM, ?<josef.pktd at gmail.com> wrote:
>>>> I didn't follow the discussion of binary compatibility between
>>>> different versions very closely.
>>>>
>>>> I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
>>>> against numpy 1.2.1 and I get an immediate segfault when trying to run
>>>> scipy.test(). I get the usual 4 failures when I run scipy with numpy
>>>> 1.3.0
>
> The segfaults are because we forgot to increment the ABI version.
> There's a thread about this on the Numpy list.
>
> [clip]
>>> I build scipy 0.7.1 against numpy 1.2.1 (official installer) and I get
>>> the same segfault. I get it already trying to `import scipy.stats`
>>
>> Just to see if it works, I tried to build scipy trunk against numpy
>> 1.2.1, but this time I get a compiler error
> [clip]
>> So it looks like, I cannot test backports to scipy 0.7.1 against numpy
>> 1.2.1
>
> Scipy 0.8 will not be compatible with Numpy 1.2.x, and so the trunk will
> not build with Numpy 1.2.x.

I wasn't sure about this.

>
> If you want to test your backports to 0.7.x, I think you should anyway
> compile the 0.7.x branch, to which you are backporting the fixes, not the
> trunk from which you backport them.

That's what I started with, but my built of scipy  0.7.x against numpy
1.2.1 segfaults.

So I would like to know whether this is working and my setup is wrong,
or if scipy 0.7.x cannot currently be built against numpy 1.2.1

Thanks,
Josef


From david at ar.media.kyoto-u.ac.jp  Fri May 15 23:03:53 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 16 May 2009 12:03:53 +0900
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
Message-ID: <4A0E2D19.8070201@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
> I didn't follow the discussion of binary compatibility between
> different versions very closely.
>
> I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
> against numpy 1.2.1 and I get an immediate segfault when trying to run
> scipy.test().

Yes, you can't do that. The term binary compatible is too vague, which
is why there is some confusion in that matter I believe. There is
forward compatibility and backward compatibility:
    - Backward compatibility means that you can build something against
numpy version M, later update numpy to version N >M, and it still works.
numpy 1.3.0 is backward compatible with 1.2.1
    - Forward compatibility means that you can build something against
numpy version N, later downgrade to numpy version M < N, and it still
works. That's almost impossible to achieve without a huge amount of
work; we do not attempt it.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Fri May 15 23:06:01 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 16 May 2009 12:06:01 +0900
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
Message-ID: <4A0E2D99.4060300@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
> On Fri, May 15, 2009 at 9:39 AM,  <josef.pktd at gmail.com> wrote:
>   
>> I didn't follow the discussion of binary compatibility between
>> different versions very closely.
>>
>> I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
>> against numpy 1.2.1 and I get an immediate segfault when trying to run
>> scipy.test(). I get the usual 4 failures when I run scipy with numpy
>> 1.3.0
>>
>> Is this expected?
>>
>> (I didn't do it on purpose, the version mismatch was just a
>> consequence of how I wanted to setup my testing environment for scipy
>> 0.7.1 and is easily fixed)
>>
>>     
>
> I build scipy 0.7.1 against numpy 1.2.1 (official installer) and I get
> the same segfault. I get it already trying to `import scipy.stats`
>   

This is strange. Did you remove the build directory and the previously
installed scipy before building scipy 0.7.1 against 1.2.1 ? This should
definitely work. I am testing it ATM,

David


From skorpio11 at gmail.com  Sat May 16 02:07:23 2009
From: skorpio11 at gmail.com (Leon Adams)
Date: Sat, 16 May 2009 02:07:23 -0400
Subject: [SciPy-dev] f2py vector valued functions and optimize.fmin_slsqp
Message-ID: <9e3090c0905152307m5a3470e7q784e9e7932d3f9ec@mail.gmail.com>

Hi...

I am experiencing some difficulties arriving at a soln for my optimization
problem.

Background:

My problem is cast as a minimizer problem but is very computer intense (lots
of loops).
My design soln was to wrap the looping parts in FORTRAN using f2py, use
python as a steering program
and optimize.fmin_slsqp as the optimizer.  Everything seems to work well for
scalar valued functions but not for vector valued functions involving
arrays.
Anytime an array is involved SLSQP returns with a success exit status [0]
without updating the input vector. That is no change is being made to the
input parameters.

My question is: is it possible to wrap a vector valued fortran function and
still use SLSQP?

Thanks in advance

-- 
Leon Adams
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090516/28ee2f3e/attachment.html>

From josef.pktd at gmail.com  Sat May 16 06:36:47 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 16 May 2009 06:36:47 -0400
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <4A0E2D99.4060300@ar.media.kyoto-u.ac.jp>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
	<4A0E2D99.4060300@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0905160336l443ef6dcl43badd778e3b9faf@mail.gmail.com>

On Fri, May 15, 2009 at 11:06 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> josef.pktd at gmail.com wrote:
>> On Fri, May 15, 2009 at 9:39 AM, ?<josef.pktd at gmail.com> wrote:
>>
>>> I didn't follow the discussion of binary compatibility between
>>> different versions very closely.
>>>
>>> I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
>>> against numpy 1.2.1 and I get an immediate segfault when trying to run
>>> scipy.test(). I get the usual 4 failures when I run scipy with numpy
>>> 1.3.0
>>>
>>> Is this expected?
>>>
>>> (I didn't do it on purpose, the version mismatch was just a
>>> consequence of how I wanted to setup my testing environment for scipy
>>> 0.7.1 and is easily fixed)
>>>
>>>
>>
>> I build scipy 0.7.1 against numpy 1.2.1 (official installer) and I get
>> the same segfault. I get it already trying to `import scipy.stats`
>>
>
> This is strange. Did you remove the build directory and the previously
> installed scipy before building scipy 0.7.1 against 1.2.1 ? This should
> definitely work. I am testing it ATM,
>

I removed the build directory between builts, but I had a recent scipy
trunk on the python path, since this usually didn't cause any problems
for the building. I followed my usual procedure for compiling scipy,
but I had never tried with a downgraded numpy before.

If the build of  scipy 0.7.1 against numpy 1.2.1 works for you, then I
will check everything to see why my usual setup didn't work for this
case.

Thanks,

Josef


From david at ar.media.kyoto-u.ac.jp  Sat May 16 06:25:28 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 16 May 2009 19:25:28 +0900
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <1cd32cbb0905160336l443ef6dcl43badd778e3b9faf@mail.gmail.com>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>	<4A0E2D99.4060300@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905160336l443ef6dcl43badd778e3b9faf@mail.gmail.com>
Message-ID: <4A0E9498.8070907@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
> On Fri, May 15, 2009 at 11:06 PM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>   
>> josef.pktd at gmail.com wrote:
>>     
>>> On Fri, May 15, 2009 at 9:39 AM,  <josef.pktd at gmail.com> wrote:
>>>
>>>       
>>>> I didn't follow the discussion of binary compatibility between
>>>> different versions very closely.
>>>>
>>>> I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
>>>> against numpy 1.2.1 and I get an immediate segfault when trying to run
>>>> scipy.test(). I get the usual 4 failures when I run scipy with numpy
>>>> 1.3.0
>>>>
>>>> Is this expected?
>>>>
>>>> (I didn't do it on purpose, the version mismatch was just a
>>>> consequence of how I wanted to setup my testing environment for scipy
>>>> 0.7.1 and is easily fixed)
>>>>
>>>>
>>>>         
>>> I build scipy 0.7.1 against numpy 1.2.1 (official installer) and I get
>>> the same segfault. I get it already trying to `import scipy.stats`
>>>
>>>       
>> This is strange. Did you remove the build directory and the previously
>> installed scipy before building scipy 0.7.1 against 1.2.1 ? This should
>> definitely work. I am testing it ATM,
>>
>>     
>
> I removed the build directory between builts, but I had a recent scipy
> trunk on the python path, since this usually didn't cause any problems
> for the building. I followed my usual procedure for compiling scipy,
> but I had never tried with a downgraded numpy before.
>   

You are using virtualenv, right ? I think that's the best solution on
windows. In that context, you keep one environment per released numpy
you are interested in (1.2.1 here), and one for development purpose.

> If the build of  scipy 0.7.1 against numpy 1.2.1 works for you, then I
> will check everything to see why my usual setup didn't work for this
> case.
>   

I can confirm it works. I fixed a couple of issues (none of which would
have caused the crash, though) on 0.7.x + numpy 1.2.1 combination on win
XP, with several full test runs, without any trouble.

cheers,

David


From josef.pktd at gmail.com  Sat May 16 07:10:01 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 16 May 2009 07:10:01 -0400
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <4A0E9498.8070907@ar.media.kyoto-u.ac.jp>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
	<4A0E2D99.4060300@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905160336l443ef6dcl43badd778e3b9faf@mail.gmail.com>
	<4A0E9498.8070907@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0905160410y62c4fe2co4842a647366f9331@mail.gmail.com>

On Sat, May 16, 2009 at 6:25 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> josef.pktd at gmail.com wrote:
>> On Fri, May 15, 2009 at 11:06 PM, David Cournapeau
>> <david at ar.media.kyoto-u.ac.jp> wrote:
>>
>>> josef.pktd at gmail.com wrote:
>>>
>>>> On Fri, May 15, 2009 at 9:39 AM, ?<josef.pktd at gmail.com> wrote:
>>>>
>>>>
>>>>> I didn't follow the discussion of binary compatibility between
>>>>> different versions very closely.
>>>>>
>>>>> I compiled scipy 0.7.1 against numpy 1.3.0 and ran it in a virtualenv
>>>>> against numpy 1.2.1 and I get an immediate segfault when trying to run
>>>>> scipy.test(). I get the usual 4 failures when I run scipy with numpy
>>>>> 1.3.0
>>>>>
>>>>> Is this expected?
>>>>>
>>>>> (I didn't do it on purpose, the version mismatch was just a
>>>>> consequence of how I wanted to setup my testing environment for scipy
>>>>> 0.7.1 and is easily fixed)
>>>>>
>>>>>
>>>>>
>>>> I build scipy 0.7.1 against numpy 1.2.1 (official installer) and I get
>>>> the same segfault. I get it already trying to `import scipy.stats`
>>>>
>>>>
>>> This is strange. Did you remove the build directory and the previously
>>> installed scipy before building scipy 0.7.1 against 1.2.1 ? This should
>>> definitely work. I am testing it ATM,
>>>
>>>
>>
>> I removed the build directory between builts, but I had a recent scipy
>> trunk on the python path, since this usually didn't cause any problems
>> for the building. I followed my usual procedure for compiling scipy,
>> but I had never tried with a downgraded numpy before.
>>
>
> You are using virtualenv, right ? I think that's the best solution on
> windows. In that context, you keep one environment per released numpy
> you are interested in (1.2.1 here), and one for development purpose.
>
>> If the build of ?scipy 0.7.1 against numpy 1.2.1 works for you, then I
>> will check everything to see why my usual setup didn't work for this
>> case.
>>
>
> I can confirm it works. I fixed a couple of issues (none of which would
> have caused the crash, though) on 0.7.x + numpy 1.2.1 combination on win
> XP, with several full test runs, without any trouble.
>

good, which means I need to start from the beginning.

I'm using the virtualenv to run different versions, but not to build.
With earlier versions of virtualenv it wasn't possible to build
extension in a virtualenv on Windows, but I haven't tried this in half
a year or so.

To build against numpy 1.2.1, I replaced numpy in pythons  sitepackages.

Josef


From david at ar.media.kyoto-u.ac.jp  Sat May 16 07:10:16 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 16 May 2009 20:10:16 +0900
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <1cd32cbb0905160410y62c4fe2co4842a647366f9331@mail.gmail.com>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>	<4A0E2D99.4060300@ar.media.kyoto-u.ac.jp>	<1cd32cbb0905160336l443ef6dcl43badd778e3b9faf@mail.gmail.com>	<4A0E9498.8070907@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905160410y62c4fe2co4842a647366f9331@mail.gmail.com>
Message-ID: <4A0E9F18.6080202@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
> good, which means I need to start from the beginning.
>   

:)

> I'm using the virtualenv to run different versions, but not to build.
> With earlier versions of virtualenv it wasn't possible to build
> extension in a virtualenv on Windows, but I haven't tried this in half
> a year or so.
>   

Ah, yes, sorry, I forgot. You're right, it does not work to build python
extensions - you have to install virtualenv from sources, where this is
fixed. To make things even more fun, the last svn version is broken. I
don't remember the exact number and my windows vm is shut down, but if
you look at the svn revision log, somewhere around beginning of april,
there is a mention on that fact - get this version, it works. Once you
get the right version, installing is a breeze, though, just python
setup.py bdist_wininst  + executing the installer.

> To build against numpy 1.2.1, I replaced numpy in pythons  sitepackages.
>   

Yes, but this breaks eventually, because you always forget to update it
at some point. That's why things like controlled environments are
better. On windows, virtualenv is the only tool I know of which can do
that painlessly for python.

David



From robfalck at gmail.com  Sat May 16 11:32:29 2009
From: robfalck at gmail.com (Rob Falck)
Date: Sat, 16 May 2009 11:32:29 -0400
Subject: [SciPy-dev] f2py vector valued functions and optimize.fmin_slsqp
In-Reply-To: <9e3090c0905152307m5a3470e7q784e9e7932d3f9ec@mail.gmail.com>
References: <9e3090c0905152307m5a3470e7q784e9e7932d3f9ec@mail.gmail.com>
Message-ID: <ccb1b5850905160832g3ea38c4cqb1f819013c0b67f6@mail.gmail.com>

SLSQP is intended to optimize scalar valued functions.  If your function is
vector valued you need to wrap it in a way such that a single scalar value
(your performance index) is returned.

On Sat, May 16, 2009 at 2:07 AM, Leon Adams <skorpio11 at gmail.com> wrote:

> Hi...
>
> I am experiencing some difficulties arriving at a soln for my optimization
> problem.
>
> Background:
>
> My problem is cast as a minimizer problem but is very computer intense
> (lots of loops).
> My design soln was to wrap the looping parts in FORTRAN using f2py, use
> python as a steering program
> and optimize.fmin_slsqp as the optimizer.  Everything seems to work well
> for scalar valued functions but not for vector valued functions involving
> arrays.
> Anytime an array is involved SLSQP returns with a success exit status [0]
> without updating the input vector. That is no change is being made to the
> input parameters.
>
> My question is: is it possible to wrap a vector valued fortran function and
> still use SLSQP?
>
> Thanks in advance
>
> --
> Leon Adams
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>


-- 
- Rob Falck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090516/27d8c8e1/attachment.html>

From skorpio11 at gmail.com  Sat May 16 12:27:47 2009
From: skorpio11 at gmail.com (Leon Adams)
Date: Sat, 16 May 2009 12:27:47 -0400
Subject: [SciPy-dev] f2py vector valued functions and optimize.fmin_slsqp
In-Reply-To: <ccb1b5850905160832g3ea38c4cqb1f819013c0b67f6@mail.gmail.com>
References: <9e3090c0905152307m5a3470e7q784e9e7932d3f9ec@mail.gmail.com>
	<ccb1b5850905160832g3ea38c4cqb1f819013c0b67f6@mail.gmail.com>
Message-ID: <4A0EE983.6060909@gmail.com>

Rob Falck wrote:
> SLSQP is intended to optimize scalar valued functions.  If your 
> function is vector valued you need to wrap it in a way such that a 
> single scalar value (your performance index) is returned.
>
> On Sat, May 16, 2009 at 2:07 AM, Leon Adams <skorpio11 at gmail.com 
> <mailto:skorpio11 at gmail.com>> wrote:
>
>     Hi...
>
>     I am experiencing some difficulties arriving at a soln for my
>     optimization problem.
>
>     Background:
>
>     My problem is cast as a minimizer problem but is very computer
>     intense (lots of loops).
>     My design soln was to wrap the looping parts in FORTRAN using
>     f2py, use python as a steering program
>     and optimize.fmin_slsqp as the optimizer.  Everything seems to
>     work well for scalar valued functions but not for vector valued
>     functions involving arrays.
>     Anytime an array is involved SLSQP returns with a success exit
>     status [0] without updating the input vector. That is no change is
>     being made to the input parameters.
>
>     My question is: is it possible to wrap a vector valued fortran
>     function and still use SLSQP?
>
>     Thanks in advance
>
>     -- 
>     Leon Adams
>
>     _______________________________________________
>     Scipy-dev mailing list
>     Scipy-dev at scipy.org <mailto:Scipy-dev at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
>
>
> -- 
> - Rob Falck
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>    
Thanks for the response.

I have wrapped the function to return the performance index. Don't know 
if this is another symptom or the problem itself, but the soln 
terminates with:

Optimization terminated successfully.    (Exit mode 0)
             Current function value: 6.45723021956
             Iterations: 1
             Function evaluations: 20
             Gradient evaluations: 1

If the generating routine for the performance index is coded in straight 
python I get:

Optimization terminated successfully.    (Exit mode 0)
             Current function value: 0.858946186765
             Iterations: 12
             Function evaluations: 245
             Gradient evaluations: 12

For some reason, the iterations stops at 1 when I try to use the f2py 
imported libraries. Is it possible that it is getting hung up on the 
Gradient evaluations??


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090516/8b66a70d/attachment.html>

From skorpio11 at gmail.com  Sat May 16 12:29:33 2009
From: skorpio11 at gmail.com (Leon Adams)
Date: Sat, 16 May 2009 12:29:33 -0400
Subject: [SciPy-dev] f2py vector valued functions and optimize.fmin_slsqp
In-Reply-To: <ccb1b5850905160832g3ea38c4cqb1f819013c0b67f6@mail.gmail.com>
References: <9e3090c0905152307m5a3470e7q784e9e7932d3f9ec@mail.gmail.com>
	<ccb1b5850905160832g3ea38c4cqb1f819013c0b67f6@mail.gmail.com>
Message-ID: <9e3090c0905160929i1c03654awcde3efdcede61d23@mail.gmail.com>

On Sat, May 16, 2009 at 11:32 AM, Rob Falck <robfalck at gmail.com> wrote:

> SLSQP is intended to optimize scalar valued functions.  If your function is
> vector valued you need to wrap it in a way such that a single scalar value
> (your performance index) is returned.
>
> On Sat, May 16, 2009 at 2:07 AM, Leon Adams <skorpio11 at gmail.com> wrote:
>
>> Hi...
>>
>> I am experiencing some difficulties arriving at a soln for my optimization
>> problem.
>>
>> Background:
>>
>> My problem is cast as a minimizer problem but is very computer intense
>> (lots of loops).
>> My design soln was to wrap the looping parts in FORTRAN using f2py, use
>> python as a steering program
>> and optimize.fmin_slsqp as the optimizer.  Everything seems to work well
>> for scalar valued functions but not for vector valued functions involving
>> arrays.
>> Anytime an array is involved SLSQP returns with a success exit status [0]
>> without updating the input vector. That is no change is being made to the
>> input parameters.
>>
>> My question is: is it possible to wrap a vector valued fortran function
>> and still use SLSQP?
>>
>> Thanks in advance
>>
>> --
>> Leon Adams
>>
>> _______________________________________________
>> Scipy-dev mailing list
>> Scipy-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-dev
>>
>>
>
>
> --
> - Rob Falck
>
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
Thanks for the response.

I have wrapped the function to return the performance index. Don't know if
this is another symptom or the problem itself, but the soln terminates with:

Optimization terminated successfully.    (Exit mode 0)
            Current function value: 6.45723021956
            Iterations: 1
            Function evaluations: 20
            Gradient evaluations: 1

If the generating routine for the performance index is coded in straight
python I get:

Optimization terminated successfully.    (Exit mode 0)
            Current function value: 0.858946186765
            Iterations: 12
            Function evaluations: 245
            Gradient evaluations: 12

For some reason, the iterations stops at 1 when I try to use the f2py
imported libraries. Is it possible that it is getting hung up on the
Gradient evaluations??

-- 
Leon Adams
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090516/27bcd72d/attachment.html>

From arokem at berkeley.edu  Sat May 16 13:13:15 2009
From: arokem at berkeley.edu (Ariel Rokem)
Date: Sat, 16 May 2009 10:13:15 -0700
Subject: [SciPy-dev] ReST tutorial - where should it go?
In-Reply-To: <9457e7c80905150317j1b7120acr4243f7b5d569171@mail.gmail.com>
References: <1e2af89e0905141924r241d3b59j7717625181238f68@mail.gmail.com> 
	<1cd32cbb0905141949q70fb9525r5695a84d63c1f5b8@mail.gmail.com> 
	<1e2af89e0905142143s6f6e9bb5vcc3d12fb911b1d11@mail.gmail.com> 
	<9457e7c80905142235v7571e4b7nb494eae712c947cf@mail.gmail.com> 
	<1e2af89e0905150002j6be9b788h518d9f05dcb0647e@mail.gmail.com> 
	<9457e7c80905150317j1b7120acr4243f7b5d569171@mail.gmail.com>
Message-ID: <43958ee60905161013s1af1e8fdv69b85216a6d3af33@mail.gmail.com>

Hi - you can also make scipy.io.loadmat behave just like you expected by
setting the optional parameter 'squeeze_me' to True.
I have found that to be a very useful thing to do when dealing with Matlab
variables with a lot of nested structs and cell arrays. Big fan of
'squeeze_me'

Cheers,

Ariel

2009/5/15 St?fan van der Walt <stefan at sun.ac.za>

> 2009/5/15 Matthew Brett <matthew.brett at gmail.com>:
> > I sense some gentle South African teasing here, but I'm too tired to
> > understand, so I say:
> >
> > Great - glad I taught you something new about structs!
>
> Always so suspicious of the former colonies! :)
>
> But no, I was being serious: thanks for all your work on scipy.io and
> for writing the tutorial!
>
> I was surprised, because I thought the Octave structure:
>
> octave:2> s = struct('foo', 1, 'bar', 2)
> s =
> {
>  foo =  1
>  bar =  2
> }
>
> would be read in by SciPy to a variable x that could be accessed:
>
> x.foo and x.bar
>
> In the tutorial I saw that you had to use
>
> x[0, 0].foo and x[0, 0].bar
>
> instead, which didn't make sense to me, mainly because in the 3 years
> since I last used Octave I forgot that one could do:
>
> octave:4> s = struct('foo', {1, 2}, 'bar', {4, 5})
> s =
> {
>  foo =
>
>  (,
>    [1] =  1
>    [2] =  2
>  ,)
>
>  bar =
>
>  (,
>    [1] =  4
>    [2] =  5
>  ,)
>
> }
>
> In Octave you then see that
>
> octave:6> size(s)
> ans =
>
>   1   2
>
> octave:7> s(1, 1)
> ans =
> {
>  bar =  4
>  foo =  1
> }
>
> after which the way SciPy indexes the data makes a lot more sense!
>
> Enjoy the weekend!
> St?fan
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>



-- 
Ariel Rokem
Helen Wills Neuroscience Institute
University of California, Berkeley
http://argentum.ucbso.berkeley.edu/ariel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090516/d627f35b/attachment.html>

From josef.pktd at gmail.com  Sat May 16 13:58:54 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 16 May 2009 13:58:54 -0400
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <4A0E9F18.6080202@ar.media.kyoto-u.ac.jp>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
	<4A0E2D99.4060300@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905160336l443ef6dcl43badd778e3b9faf@mail.gmail.com>
	<4A0E9498.8070907@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905160410y62c4fe2co4842a647366f9331@mail.gmail.com>
	<4A0E9F18.6080202@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0905161058k1bb8338eu945da03ce3e57aa3@mail.gmail.com>

On Sat, May 16, 2009 at 7:10 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> josef.pktd at gmail.com wrote:
>> good, which means I need to start from the beginning.
>>
>
> :)
>
>> I'm using the virtualenv to run different versions, but not to build.
>> With earlier versions of virtualenv it wasn't possible to build
>> extension in a virtualenv on Windows, but I haven't tried this in half
>> a year or so.
>>
>
> Ah, yes, sorry, I forgot. You're right, it does not work to build python
> extensions - you have to install virtualenv from sources, where this is
> fixed. To make things even more fun, the last svn version is broken. I
> don't remember the exact number and my windows vm is shut down, but if
> you look at the svn revision log, somewhere around beginning of april,
> there is a mention on that fact - get this version, it works. Once you
> get the right version, installing is a breeze, though, just python
> setup.py bdist_wininst ?+ executing the installer.
>
>> To build against numpy 1.2.1, I replaced numpy in pythons ?sitepackages.
>>
>
> Yes, but this breaks eventually, because you always forget to update it
> at some point. That's why things like controlled environments are
> better. On windows, virtualenv is the only tool I know of which can do
> that painlessly for python.
>

ok, I managed to compile scipy 0.7.x in a virtualenv, and get zero failures.

for the record:
I'm using the latest release virtualenv-1.3.3,
easy_installed numpy from exe installer
additional to the usual changes to numpy/distutils/site.cfg, I added
the [DEFAULT] link and include directories of my system-wide python
version
But then I still needed to copy ``python25/libs`` directory to the virtualenv.
After that, building scipy from inside the virtualenv works without errors

I haven't tried any other (non numpy) extensions yet.

>>> numpy.version.version
'1.2.1'
>>> scipy.version.version
'0.7.1.dev5744'

>>> scipy.test()
Ran 3456 tests in 66.657s
OK (KNOWNFAIL=2, SKIP=29)
<nose.result.TextTestResult run=3456 errors=0 failures=0>

Thanks,

Josef


From david at ar.media.kyoto-u.ac.jp  Sun May 17 02:57:37 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 17 May 2009 15:57:37 +0900
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <1cd32cbb0905161058k1bb8338eu945da03ce3e57aa3@mail.gmail.com>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>	<4A0E2D99.4060300@ar.media.kyoto-u.ac.jp>	<1cd32cbb0905160336l443ef6dcl43badd778e3b9faf@mail.gmail.com>	<4A0E9498.8070907@ar.media.kyoto-u.ac.jp>	<1cd32cbb0905160410y62c4fe2co4842a647366f9331@mail.gmail.com>	<4A0E9F18.6080202@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905161058k1bb8338eu945da03ce3e57aa3@mail.gmail.com>
Message-ID: <4A0FB561.5090201@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
> Ran 3456 tests in 66.657s
> OK (KNOWNFAIL=2, SKIP=29)
> <nose.result.TextTestResult run=3456 errors=0 failures=0>
>   

Great. I fixed yesterday the last win32-specific failures. Except for
weave, there is no windows-specific failure in scipy anymore.

cheers,

David


From josef.pktd at gmail.com  Sun May 17 03:24:24 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 17 May 2009 03:24:24 -0400
Subject: [SciPy-dev] binary compatibility question
In-Reply-To: <4A0FB561.5090201@ar.media.kyoto-u.ac.jp>
References: <1cd32cbb0905150639m50db5b36k1a7961a5cc73876e@mail.gmail.com>
	<1cd32cbb0905150719v4cf07051jd86d24bf9fbb6e93@mail.gmail.com>
	<4A0E2D99.4060300@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905160336l443ef6dcl43badd778e3b9faf@mail.gmail.com>
	<4A0E9498.8070907@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905160410y62c4fe2co4842a647366f9331@mail.gmail.com>
	<4A0E9F18.6080202@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905161058k1bb8338eu945da03ce3e57aa3@mail.gmail.com>
	<4A0FB561.5090201@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0905170024w34acfe48y27c8141a3120cb1e@mail.gmail.com>

On Sun, May 17, 2009 at 2:57 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> josef.pktd at gmail.com wrote:
>> Ran 3456 tests in 66.657s
>> OK (KNOWNFAIL=2, SKIP=29)
>> <nose.result.TextTestResult run=3456 errors=0 failures=0>
>>
>
> Great. I fixed yesterday the last win32-specific failures. Except for
> weave, there is no windows-specific failure in scipy anymore.
>

Yes, I saw, I rebuild also the trunk after your last changes and it is
very nice to be back to  errors=0 failures=0.

Thanks for taking care of us Windows users.

Josef


From robfalck at gmail.com  Sun May 17 19:53:00 2009
From: robfalck at gmail.com (Rob Falck)
Date: Sun, 17 May 2009 19:53:00 -0400
Subject: [SciPy-dev] f2py vector valued functions and optimize.fmin_slsqp
In-Reply-To: <9e3090c0905160929i1c03654awcde3efdcede61d23@mail.gmail.com>
References: <9e3090c0905152307m5a3470e7q784e9e7932d3f9ec@mail.gmail.com>
	<ccb1b5850905160832g3ea38c4cqb1f819013c0b67f6@mail.gmail.com>
	<9e3090c0905160929i1c03654awcde3efdcede61d23@mail.gmail.com>
Message-ID: <ccb1b5850905171653v38cbdb1u21abe3040035f1ae@mail.gmail.com>

>
>>
> Thanks for the response.
>
> I have wrapped the function to return the performance index. Don't know if
> this is another symptom or the problem itself, but the soln terminates with:
>
> Optimization terminated successfully.    (Exit mode 0)
>             Current function value: 6.45723021956
>             Iterations: 1
>             Function evaluations: 20
>             Gradient evaluations: 1
>
> If the generating routine for the performance index is coded in straight
> python I get:
>
> Optimization terminated successfully.    (Exit mode 0)
>             Current function value: 0.858946186765
>             Iterations: 12
>             Function evaluations: 245
>             Gradient evaluations: 12
>
> For some reason, the iterations stops at 1 when I try to use the f2py
> imported libraries. Is it possible that it is getting hung up on the
> Gradient evaluations??
>
> --
> Leon Adams


I wish I could help.  I havent tried to use SLSQP with F2PY.



-- 
- Rob Falck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090517/7f685d19/attachment.html>

From sebastian.walter at gmail.com  Mon May 18 04:09:06 2009
From: sebastian.walter at gmail.com (Sebastian Walter)
Date: Mon, 18 May 2009 10:09:06 +0200
Subject: [SciPy-dev] f2py vector valued functions and optimize.fmin_slsqp
In-Reply-To: <ccb1b5850905171653v38cbdb1u21abe3040035f1ae@mail.gmail.com>
References: <9e3090c0905152307m5a3470e7q784e9e7932d3f9ec@mail.gmail.com>
	<ccb1b5850905160832g3ea38c4cqb1f819013c0b67f6@mail.gmail.com>
	<9e3090c0905160929i1c03654awcde3efdcede61d23@mail.gmail.com>
	<ccb1b5850905171653v38cbdb1u21abe3040035f1ae@mail.gmail.com>
Message-ID: <ec9f80fa0905180109u471c24bey81fad787fd2826b6@mail.gmail.com>

In 90% of the cases of an unsuccessful optimization the provided
gradient is wrong.
There should be a routine in scipy to check if your provided gradient
is correct.



On Mon, May 18, 2009 at 1:53 AM, Rob Falck <robfalck at gmail.com> wrote:
>
>>>
>>
>> Thanks for the response.
>>
>> I have wrapped the function to return the performance index. Don't know if
>> this is another symptom or the problem itself, but the soln terminates with:
>>
>> Optimization terminated successfully.    (Exit mode 0)
>>             Current function value: 6.45723021956
>>             Iterations: 1
>>             Function evaluations: 20
>>             Gradient evaluations: 1
>>
>> If the generating routine for the performance index is coded in straight
>> python I get:
>>
>> Optimization terminated successfully.    (Exit mode 0)
>>             Current function value: 0.858946186765
>>             Iterations: 12
>>             Function evaluations: 245
>>             Gradient evaluations: 12
>>
>> For some reason, the iterations stops at 1 when I try to use the f2py
>> imported libraries. Is it possible that it is getting hung up on the
>> Gradient evaluations??
>>
>> --
>> Leon Adams
>
> I wish I could help.  I havent tried to use SLSQP with F2PY.
>
>
>
> --
> - Rob Falck
>
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>


From millman at berkeley.edu  Mon May 18 10:48:10 2009
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 18 May 2009 07:48:10 -0700
Subject: [SciPy-dev] SciPy 2009 Call for Papers
Message-ID: <c7009a550905180748g6d2d7e1cr1508db219164f85e@mail.gmail.com>

==========================
SciPy 2009 Call for Papers
==========================

SciPy 2009, the 8th Python in Science conference, will be held
from August 18-23, 2009 at Caltech in Pasadena, CA, USA.

Each year SciPy attracts leading figures in research and scientific
software development with Python from a wide range of scientific and
engineering disciplines. The focus of the conference is both on scientific
libraries and tools developed with Python and on scientific or engineering
achievements using Python.

We welcome contributions from the industry as well as the academic world.
Indeed, industrial research and development as well academic research
face the challenge of mastering IT tools for exploration, modeling and
analysis.

We look forward to hearing your recent breakthroughs using Python!

Submission of Papers
====================

The program features tutorials, contributed papers, lightning talks, and
bird-of-a-feather sessions. We are soliciting talks and accompanying
papers (either formal academic or magazine-style articles) that discuss
topics which center around scientific computing using Python. These
include applications, teaching, future development directions, and
research. A collection of peer-reviewed articles will be published as
part of the proceedings.

Proposals for talks are submitted as extended abstracts. There are two
categories of talks:

 Paper presentations

     These talks are 35 minutes in duration (including questions). A one page
     abstract of no less than 500 words (excluding figures and references)
     should give an outline of the final paper. Proceeding papers are due two
     weeks after the conference, and may be in a formal academic style, or in
     a more relaxed magazine-style format.

 Rapid presentations

     These talks are 10 minutes in duration. An abstract of between
     300 and 700 words should describe the topic and motivate its
     relevance to scientific computing.

In addition, there will be an open session for lightning talks during which
any attendee willing to do so is invited to do a couple-of-minutes-long
presentation.

If you wish to present a talk at the conference, please create an account
on the website (http://conference.scipy.org). You may then submit an abstract
by logging in, clicking on your profile and following the "Submit an
abstract" link.

Submission Guidelines
---------------------

* Submissions should be uploaded via the online form.
* Submissions whose main purpose is to promote a commercial product or
  service will be refused.
* All accepted proposals must be presented at the SciPy conference by
  at least one author.
* Authors of an accepted proposal can provide a final paper for
  publication in the conference proceedings. Final papers are limited
  to 7 pages, including diagrams, figures, references, and appendices.
  The papers will be reviewed to help ensure the high-quality of the
  proceedings.

For further information, please visit the conference homepage:
http://conference.scipy.org.

Important Dates
===============

* Friday, June 26: Abstracts Due
* Saturday, July 4: Announce accepted talks, post schedule
* Friday, July 10: Early Registration ends
* Tuesday-Wednesday, August 18-19: Tutorials
* Thursday-Friday, August 20-21: Conference
* Saturday-Sunday, August 22-23: Sprints
* Friday, September 4: Papers for proceedings due

Tutorials
=========

Two days of tutorials to the scientific Python tools will precede the
conference. There will be two tracks:  one for introduction of the basic
tools to beginners and one for more advanced tools. Tutorials will be
announced later.

Birds of a Feather Sessions
===========================

If you wish to organize a birds-of-a-feather session to discuss some
specific area of scientific development with Python, please contact the
organizing committee.

Executive Committee
===================

* Jarrod Millman, UC Berkeley, USA (Conference Chair)
* Ga?l Varoquaux, INRIA Saclay, France (Program Co-Chair)
* St?fan van der Walt, University of Stellenbosch, South Africa
(Program Co-Chair)
* Fernando P?rez, UC Berkeley, USA (Tutorial Chair)


From nwagner at iam.uni-stuttgart.de  Tue May 19 14:18:12 2009
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 19 May 2009 20:18:12 +0200
Subject: [SciPy-dev] test_lgmres.py
Message-ID: <web-120299235@uni-stuttgart.de>

Hi all,

An import nose is missing in test_lgmres.py

It works for me with:
  
if __name__ == "__main__":
     import nose
     nose.run(argv=['', __file__])


Cheers,
               Nils


From stefan at sun.ac.za  Wed May 20 19:34:37 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 21 May 2009 01:34:37 +0200
Subject: [SciPy-dev] RFR: Add io to ndimage
Message-ID: <9457e7c80905201634h2ecc6002l1f3ffb04fa8da1b5@mail.gmail.com>

Hi all,

It's bothered me for a long time that our image reading routines are
hidden in scipy.misc.  Here is a patch that implements image reading
via the PIL array interface.  Please let me know whether you like this
or not.

Regards
St?fan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Add-image-IO-to-ndimage.patch
Type: application/octet-stream
Size: 2481 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20090521/02435dd4/attachment.obj>

From stefan at sun.ac.za  Wed May 20 19:44:19 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 21 May 2009 01:44:19 +0200
Subject: [SciPy-dev] Hilbert-matrix
Message-ID: <9457e7c80905201644r54037014nf86d5b2cb0599d64@mail.gmail.com>

Hi all,

The Ill-conditioned Hilbert-matrix is often studied by students in
numerical linear algebra, and as such I thought it might deserve a
place in scipy.linalg.

def hilbert(N):
    x, y = np.ogrid[1.:N+1, 1.:N+1]
    return 1 / (x + y - 1)

The trickier issue, however, would be calculating the inverse -- does
anybody have suggestions on how to implement it accurately?  Ideally,
we would have made use of Python's very long integers, but that would
mean saving the array with dtype object.

Regards
St?fan


From robert.kern at gmail.com  Wed May 20 20:35:57 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 20 May 2009 19:35:57 -0500
Subject: [SciPy-dev] Hilbert-matrix
In-Reply-To: <9457e7c80905201644r54037014nf86d5b2cb0599d64@mail.gmail.com>
References: <9457e7c80905201644r54037014nf86d5b2cb0599d64@mail.gmail.com>
Message-ID: <3d375d730905201735u7f34a3b1l22bf766f106d8d89@mail.gmail.com>

2009/5/20 St?fan van der Walt <stefan at sun.ac.za>:
> Hi all,
>
> The Ill-conditioned Hilbert-matrix is often studied by students in
> numerical linear algebra, and as such I thought it might deserve a
> place in scipy.linalg.
>
> def hilbert(N):
> ? ?x, y = np.ogrid[1.:N+1, 1.:N+1]
> ? ?return 1 / (x + y - 1)
>
> The trickier issue, however, would be calculating the inverse -- does
> anybody have suggestions on how to implement it accurately?

There is an analytic formula involving combinations:

  http://mathworld.wolfram.com/HilbertMatrix.html

You could probably implement that straightforwardly in
double-precision with gammaln followed by exponentiation. That would
probably be about as good as any double-precision inversion routine
should be expected to get, best-case, since for largish matrices
double-precision won't even accurately represent the correct answer. A
fully-correct bigint expansion is neat, but not particularly necessary
to demonstrate instability.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Thu May 21 01:33:30 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 21 May 2009 14:33:30 +0900
Subject: [SciPy-dev] 0.7.1: new schedule
Message-ID: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>

Hi,

    I have changed the schedule for scipy 0.7.1. I set up the release
date for the first June. Ideally, I would like that all backports go
into the 0.7.x branch within the end of next WE. Is that ok with
everyone involved working on 0.7.x branch ? Some matlab IO problems have
been fixed by Matthew, Pauli and Josef have backported some fixes in
stats and special, and I have fixed everything I wanted to see fixed for
0.7.x.

cheers,

David


From pav at iki.fi  Thu May 21 09:46:36 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 21 May 2009 13:46:36 +0000 (UTC)
Subject: [SciPy-dev] 0.7.1: new schedule
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>
Message-ID: <gv3lvs$95u$1@ger.gmane.org>

Thu, 21 May 2009 14:33:30 +0900, David Cournapeau wrote:
>     I have changed the schedule for scipy 0.7.1. I set up the release
> date for the first June. Ideally, I would like that all backports go
> into the 0.7.x branch within the end of next WE. Is that ok with
> everyone involved working on 0.7.x branch ? Some matlab IO problems have
> been fixed by Matthew, Pauli and Josef have backported some fixes in
> stats and special, and I have fixed everything I wanted to see fixed for
> 0.7.x.

Probably OK. However, several of the bugfixes to scipy.special have not 
been backported.

I think at the following should be ported:

	-- bug fix for struve function
	5465
	5466

	-- bug fix for AMOS bessel functions, fix negative orders
	5516
	5521

Also the new Bessel Iv implementation should maybe be ported... I'll try 
to find time for doing all this, though doing it by the end of next 
weekend may be a bit tight.

-- 
Pauli Virtanen



From david at ar.media.kyoto-u.ac.jp  Thu May 21 10:17:10 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 21 May 2009 23:17:10 +0900
Subject: [SciPy-dev] 0.7.1: new schedule
In-Reply-To: <gv3lvs$95u$1@ger.gmane.org>
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>
	<gv3lvs$95u$1@ger.gmane.org>
Message-ID: <4A156266.9000600@ar.media.kyoto-u.ac.jp>

Pauli Virtanen wrote:
> Probably OK. However, several of the bugfixes to scipy.special have not 
> been backported.
>
> I think at the following should be ported:
>
> 	-- bug fix for struve function
> 	5465
> 	5466
>
> 	-- bug fix for AMOS bessel functions, fix negative orders
> 	5516
> 	5521

Those look easy enough, and they have tests. I can do the backport if
you want, it should only take me a few minutes,

David



From david at ar.media.kyoto-u.ac.jp  Fri May 22 01:51:36 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 22 May 2009 14:51:36 +0900
Subject: [SciPy-dev] 0.7.1: new schedule
In-Reply-To: <gv3lvs$95u$1@ger.gmane.org>
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>
	<gv3lvs$95u$1@ger.gmane.org>
Message-ID: <4A163D68.6080803@ar.media.kyoto-u.ac.jp>

Pauli Virtanen wrote:
> Thu, 21 May 2009 14:33:30 +0900, David Cournapeau wrote:
>   
>>     I have changed the schedule for scipy 0.7.1. I set up the release
>> date for the first June. Ideally, I would like that all backports go
>> into the 0.7.x branch within the end of next WE. Is that ok with
>> everyone involved working on 0.7.x branch ? Some matlab IO problems have
>> been fixed by Matthew, Pauli and Josef have backported some fixes in
>> stats and special, and I have fixed everything I wanted to see fixed for
>> 0.7.x.
>>     
>
> Probably OK. However, several of the bugfixes to scipy.special have not 
> been backported.
>
> I think at the following should be ported:
>
> 	-- bug fix for struve function
> 	5465
> 	5466
>   

I backported this one (r5764).

> 	-- bug fix for AMOS bessel functions, fix negative orders
> 	5516
> 	5521
>   

But not this one: the changes do not apply, there need to be other
commits as well - even by importing assert_to_tol code as well for the
tests to run, 5 tests are failing.

David


From pav at iki.fi  Fri May 22 19:41:40 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 22 May 2009 23:41:40 +0000 (UTC)
Subject: [SciPy-dev] 0.7.1: new schedule
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>
	<gv3lvs$95u$1@ger.gmane.org> <4A163D68.6080803@ar.media.kyoto-u.ac.jp>
Message-ID: <gv7d7k$fj7$2@ger.gmane.org>

Fri, 22 May 2009 14:51:36 +0900, David Cournapeau wrote:
[clip]
> But not this one: the changes do not apply, there need to be other
> commits as well - even by importing assert_to_tol code as well for the
> tests to run, 5 tests are failing.

I'd like to backport everything from trunk in scipy.special, except the 
following:

	9600bcf... Revert "use np.power in rdist (test commit with bzr)"
	ff046c6... Do not detect Endianess in Cephes.
	3a99ec1... Add test case for ndtr.

The bzr commit was broken, and the ndtr/endianness fixes break other 
things. All other commits are bugfixes of some sort; no feature additions 
here. What are fixed is:

	#924
	#803	- pbdv errors
	#503	- reimplementation of real Bessel I
	#854	- correct NaN returns for iv/jv/ etc when out-of-domain
	#853	- fix invalid return values from complex Bessel functions
	#623	- fix invalid return values from real Bessel J
	#852	- make Bessel zeros computable for orders > 100

Plus some minor documentation fixes.

And also the following:

- The special functions now raise warnings instead of printing to stderr,
  if the errprint toggle is enabled (it was and is disabled by default.

- The real-valued jve, ive, yve, kve, and kv functions return now NAN
  when the result would be complex-valued. (Previously they returned the
  real part.)

    ***

I have the stuff already backported here:

	http://github.com/pv/scipy-work/commits/work-0.7.x

The differences to trunk are minimal:

	git diff --stat ..svn/trunk -- scipy/special

	 scipy/special/cephes/mconf.h      |   12 +++++++++++-
	 scipy/special/tests/test_basic.py |    4 +++-
	 2 files changed, 14 insertions(+), 2 deletions(-)

I used this script to do the backporting:

	http://www.iki.fi/pav/tmp/git-cherry-tree

And it went like this:

	http://www.iki.fi/pav/tmp/special-rebase-trace.txt

The -o mode of git-cherry-tree seems to be useful for keeping track of 
stuff to backport.

BTW, do we want to do backports commit-by-commit, or in larger chunks? 
Currently, this is commit-by-commit.

-- 
Pauli Virtanen



From stefan at sun.ac.za  Fri May 22 20:01:05 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sat, 23 May 2009 02:01:05 +0200
Subject: [SciPy-dev] 0.7.1: new schedule
In-Reply-To: <gv7d7k$fj7$2@ger.gmane.org>
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>
	<gv3lvs$95u$1@ger.gmane.org> 
	<4A163D68.6080803@ar.media.kyoto-u.ac.jp> <gv7d7k$fj7$2@ger.gmane.org>
Message-ID: <9457e7c80905221701n5632d17by53a80c67a9c1578@mail.gmail.com>

2009/5/23 Pauli Virtanen <pav at iki.fi>:
> BTW, do we want to do backports commit-by-commit, or in larger chunks?
> Currently, this is commit-by-commit.

I'd prefer to see functional chunks -- it would make them much easier
to track.  Then again, if we combined patches that way in trunk it
would also have helped.

Regards
St?fan


From david at ar.media.kyoto-u.ac.jp  Sun May 24 06:16:05 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 24 May 2009 19:16:05 +0900
Subject: [SciPy-dev] Fixing correlate: handling api breakage ?
Message-ID: <4A191E65.1040607@ar.media.kyoto-u.ac.jp>

Hi,

    I have taken a look at the correlate function in scipy.signal. There
are several problems with it. First, it is wrong on several accounts:
       - It assumes that the correlation of complex numbers corresponds
to complex multiplication, but this is not the definition followed by
most textbooks, at least as far as signal processing is concerned.
       - More significantly, it is wrong with respect to the ordering:
it assumes that correlate(a, b) == correlate(b, a), which is not true in
general.

It may be argued that the first problem is a matter of definition, but I
don't think the 2nd one is. I would expect some people depending on the
functionality, though. What should we do ? Adding a new function which
implements the usual definition, replacing the current one or something
else ?

cheers,

David


From josef.pktd at gmail.com  Sun May 24 07:28:27 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 24 May 2009 07:28:27 -0400
Subject: [SciPy-dev] Fixing correlate: handling api breakage ?
In-Reply-To: <4A191E65.1040607@ar.media.kyoto-u.ac.jp>
References: <4A191E65.1040607@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0905240428y2985c4d5p6ba7c907238ef9ec@mail.gmail.com>

On Sun, May 24, 2009 at 6:16 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
> ? ?I have taken a look at the correlate function in scipy.signal. There
> are several problems with it. First, it is wrong on several accounts:
> ? ? ? - It assumes that the correlation of complex numbers corresponds
> to complex multiplication, but this is not the definition followed by
> most textbooks, at least as far as signal processing is concerned.
> ? ? ? - More significantly, it is wrong with respect to the ordering:
> it assumes that correlate(a, b) == correlate(b, a), which is not true in
> general.

I don't see this in the results. There was recently the report on the
mailing list that np.correlate
and signal.correlate switch arrays if the second array is longer.

>>> signal.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0])
array([0, 0, 1, 2, 0, 0, 0, 0, 0])
>>> signal.correlate([0, 0, 1, 0, 0],[1, 2, 0, 0, 0] )
array([0, 0, 0, 0, 0, 2, 1, 0, 0])

>
> It may be argued that the first problem is a matter of definition, but I
> don't think the 2nd one is. I would expect some people depending on the
> functionality, though. What should we do ? Adding a new function which
> implements the usual definition, replacing the current one or something
> else ?
>

I found it a bit confusing, that every package needs it's own correlate function
numpy, signal, ndimage, stsci.

Maybe, before adding additional versions of correlate, maybe it is
worth checking how much  overlap and differences there are. I never
tried them with complex numbers, but for floats there is a lot of
duplication (at least from the outside).

Josef


From david at ar.media.kyoto-u.ac.jp  Sun May 24 07:38:11 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 24 May 2009 20:38:11 +0900
Subject: [SciPy-dev] Fixing correlate: handling api breakage ?
In-Reply-To: <1cd32cbb0905240428y2985c4d5p6ba7c907238ef9ec@mail.gmail.com>
References: <4A191E65.1040607@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905240428y2985c4d5p6ba7c907238ef9ec@mail.gmail.com>
Message-ID: <4A1931A3.5060109@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
> On Sun, May 24, 2009 at 6:16 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>   
>> Hi,
>>
>>    I have taken a look at the correlate function in scipy.signal. There
>> are several problems with it. First, it is wrong on several accounts:
>>       - It assumes that the correlation of complex numbers corresponds
>> to complex multiplication, but this is not the definition followed by
>> most textbooks, at least as far as signal processing is concerned.
>>       - More significantly, it is wrong with respect to the ordering:
>> it assumes that correlate(a, b) == correlate(b, a), which is not true in
>> general.
>>     
>
> I don't see this in the results. There was recently the report on the
> mailing list that np.correlate
> and signal.correlate switch arrays if the second array is longer.
>
>   
>>>> signal.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0])
>>>>         
> array([0, 0, 1, 2, 0, 0, 0, 0, 0])
>   
>>>> signal.correlate([0, 0, 1, 0, 0],[1, 2, 0, 0, 0] )
>>>>         
> array([0, 0, 0, 0, 0, 2, 1, 0, 0])
>   

Well, you just happened to have very peculiar entries :)

signal.correlate([-1, -2, -3], [1, 2, 3])
-> array([ -3,  -8, -14,  -8,  -3])

signal.correlate([1, 2, 3], [-1, -2, -3])
-> array([ -3,  -8, -14,  -8,  -3])

> I found it a bit confusing, that every package needs it's own correlate function
> numpy, signal, ndimage, stsci.
>   

Yes, it is. Ndimage and stsci are geared toward particular usages,
signal and numpy correlate are the general functions (or maybe I am too
biased toward signal processing - let's say signal.correlate is the
matlab correlate).

numpy.correlate only handles 1d arrays (what matlab xcorr does), whereas
signal's one handle arbitrary dimension (what matlab image toolbox
imfilter does). It is already quite hard to handle arbitrary dimension,
but doing so in a fast way is very difficult, so this justifies at least
both implementations (i.e. replacing numpy.correlate with scipy.signal
one is not feasible IMHO, at least not with the current implementation).

ndimage's one is much more complete, by handling many boundaries
conditions (zero and constant padding, mirroring and repeating). This
makes things even slower without specialized codepaths.

I find it annoying that numpy.correlate and scipy.signal.correlate do
not use the same defaults. Ideally, I would like that for every
supported input x/y, np.correlate(x, y) == scipy.signal.correlate(x, y)
(precision problems aside), but I guess this won't change.

I should also note that my replacement for scipy correlate (not
committed yet) is based on a new "neighborhood iterator" I have created
for that purpose, so the code is much easier to follow I believe
(without being much slower than the current code). This should enables
implementation of faster codepaths for common cases in
scipy.signal.correlate much easier (scipy.signal.correlate is often
unusable for very large arrays because it is either too memory hungry or
too slow - we should care about the cases where fft-based convolution is
not usable, though, as fft is almost always the way to go for large
arrays of comparable size).

cheers,

David


From josef.pktd at gmail.com  Sun May 24 08:24:12 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 24 May 2009 08:24:12 -0400
Subject: [SciPy-dev] Fixing correlate: handling api breakage ?
In-Reply-To: <4A1931A3.5060109@ar.media.kyoto-u.ac.jp>
References: <4A191E65.1040607@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905240428y2985c4d5p6ba7c907238ef9ec@mail.gmail.com>
	<4A1931A3.5060109@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0905240524l4f822574qc56940d758574686@mail.gmail.com>

On Sun, May 24, 2009 at 7:38 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> josef.pktd at gmail.com wrote:
>> On Sun, May 24, 2009 at 6:16 AM, David Cournapeau
>> <david at ar.media.kyoto-u.ac.jp> wrote:
>>
>>> Hi,
>>>
>>> ? ?I have taken a look at the correlate function in scipy.signal. There
>>> are several problems with it. First, it is wrong on several accounts:
>>> ? ? ? - It assumes that the correlation of complex numbers corresponds
>>> to complex multiplication, but this is not the definition followed by
>>> most textbooks, at least as far as signal processing is concerned.
>>> ? ? ? - More significantly, it is wrong with respect to the ordering:
>>> it assumes that correlate(a, b) == correlate(b, a), which is not true in
>>> general.
>>>
>>
>> I don't see this in the results. There was recently the report on the
>> mailing list that np.correlate
>> and signal.correlate switch arrays if the second array is longer.
>>
>>
>>>>> signal.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0])
>>>>>
>> array([0, 0, 1, 2, 0, 0, 0, 0, 0])
>>
>>>>> signal.correlate([0, 0, 1, 0, 0],[1, 2, 0, 0, 0] )
>>>>>
>> array([0, 0, 0, 0, 0, 2, 1, 0, 0])
>>
>
> Well, you just happened to have very peculiar entries :)
>
> signal.correlate([-1, -2, -3], [1, 2, 3])
> -> array([ -3, ?-8, -14, ?-8, ?-3])
>
> signal.correlate([1, 2, 3], [-1, -2, -3])
> -> array([ -3, ?-8, -14, ?-8, ?-3])

One of your arrays is just the negative of the other, and correlate is
the same in this case. For other cases, the results differ:

>>> signal.correlate([-1, -2, -4], [1, 2, 3])
array([ -3,  -8, -17, -10,  -4])
>>> signal.correlate([1, 2, 3],[-1, -2, -4])
array([ -4, -10, -17,  -8,  -3])

>>> signal.correlate([-1, -2, 3], [1, 2, 3])
array([-3, -8,  4,  4,  3])
>>> signal.correlate([1, 2, 3], [-1, -2, 3])
array([ 3,  4,  4, -8, -3])

>
>> I found it a bit confusing, that every package needs it's own correlate function
>> numpy, signal, ndimage, stsci.
>>
>
> Yes, it is. Ndimage and stsci are geared toward particular usages,
> signal and numpy correlate are the general functions (or maybe I am too
> biased toward signal processing - let's say signal.correlate is the
> matlab correlate).
>
> numpy.correlate only handles 1d arrays (what matlab xcorr does), whereas
> signal's one handle arbitrary dimension (what matlab image toolbox
> imfilter does). It is already quite hard to handle arbitrary dimension,
> but doing so in a fast way is very difficult, so this justifies at least
> both implementations (i.e. replacing numpy.correlate with scipy.signal
> one is not feasible IMHO, at least not with the current implementation).
>
> ndimage's one is much more complete, by handling many boundaries
> conditions (zero and constant padding, mirroring and repeating). This
> makes things even slower without specialized codepaths.
>
> I find it annoying that numpy.correlate and scipy.signal.correlate do
> not use the same defaults. Ideally, I would like that for every
> supported input x/y, np.correlate(x, y) == scipy.signal.correlate(x, y)
> (precision problems aside), but I guess this won't change.
>
> I should also note that my replacement for scipy correlate (not
> committed yet) is based on a new "neighborhood iterator" I have created
> for that purpose, so the code is much easier to follow I believe
> (without being much slower than the current code). This should enables
> implementation of faster codepaths for common cases in
> scipy.signal.correlate much easier (scipy.signal.correlate is often
> unusable for very large arrays because it is either too memory hungry or
> too slow - we should care about the cases where fft-based convolution is
> not usable, though, as fft is almost always the way to go for large
> arrays of comparable size).
>

I looked at it only for examples to calculate auto-correlation and
cross-correlation in time series, and had to try out to see which
version works best.

Having a comparison, as you gave, is very helpful in choosing between
the versions, at least when not working directly with signals or
ndimages, i.e. the data the package is designed for.

Are the convolve in all cases compatible, identical (through
delegation) to correlate?

Thanks for the info
Josef


From david at ar.media.kyoto-u.ac.jp  Sun May 24 08:26:34 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 24 May 2009 21:26:34 +0900
Subject: [SciPy-dev] Fixing correlate: handling api breakage ?
In-Reply-To: <1cd32cbb0905240524l4f822574qc56940d758574686@mail.gmail.com>
References: <4A191E65.1040607@ar.media.kyoto-u.ac.jp>	<1cd32cbb0905240428y2985c4d5p6ba7c907238ef9ec@mail.gmail.com>	<4A1931A3.5060109@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905240524l4f822574qc56940d758574686@mail.gmail.com>
Message-ID: <4A193CFA.7070702@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
> On Sun, May 24, 2009 at 7:38 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
>   
>> josef.pktd at gmail.com wrote:
>>     
>>> On Sun, May 24, 2009 at 6:16 AM, David Cournapeau
>>> <david at ar.media.kyoto-u.ac.jp> wrote:
>>>
>>>       
>>>> Hi,
>>>>
>>>>    I have taken a look at the correlate function in scipy.signal. There
>>>> are several problems with it. First, it is wrong on several accounts:
>>>>       - It assumes that the correlation of complex numbers corresponds
>>>> to complex multiplication, but this is not the definition followed by
>>>> most textbooks, at least as far as signal processing is concerned.
>>>>       - More significantly, it is wrong with respect to the ordering:
>>>> it assumes that correlate(a, b) == correlate(b, a), which is not true in
>>>> general.
>>>>
>>>>         
>>> I don't see this in the results. There was recently the report on the
>>> mailing list that np.correlate
>>> and signal.correlate switch arrays if the second array is longer.
>>>
>>>
>>>       
>>>>>> signal.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0])
>>>>>>
>>>>>>             
>>> array([0, 0, 1, 2, 0, 0, 0, 0, 0])
>>>
>>>       
>>>>>> signal.correlate([0, 0, 1, 0, 0],[1, 2, 0, 0, 0] )
>>>>>>
>>>>>>             
>>> array([0, 0, 0, 0, 0, 2, 1, 0, 0])
>>>
>>>       
>> Well, you just happened to have very peculiar entries :)
>>
>> signal.correlate([-1, -2, -3], [1, 2, 3])
>> -> array([ -3,  -8, -14,  -8,  -3])
>>
>> signal.correlate([1, 2, 3], [-1, -2, -3])
>> -> array([ -3,  -8, -14,  -8,  -3])
>>     
>
> One of your arrays is just the negative of the other, and correlate is
> the same in this case. For other cases, the results differ
>   

Gr, you're right of course :) But it still fails for arrays where the
second argument has any dimension which is larger than the first one. I
don't know if the assumption is relied on in the C implementation (the
arrays are inverted in the C code in that case - even though they seem
to be already inverted in python).

> I looked at it only for examples to calculate auto-correlation and
> cross-correlation in time series, and had to try out to see which
> version works best.
>   

Yes, it depends. I know that correlate is way too slow for my own usage
in speech processing, for example for linear prediction coding. As only
a few lags are necessary, direct implementation is often faster than FFT
one - I have my own straightfoward autocorrelation in scikits.talkbox; I
believe matlab xcorr (1d correlation) always uses the FFT.

I know other people have problems with the scipy.signal correlate as
well for large arrays (the C code does a copy if the inputs are not
contiguous, for example - my own code using iterators should not need
any copy of inputs).

> Are the convolve in all cases compatible, identical (through
> delegation) to correlate?
>   

I don't think so - I don't think convolution uses the conjugate for
complex values. But I don't know any use of complex convolution,
although I am sure there is. Correlation is always defined with the
complex conjugate of the second argument AFAIK. For real cases,
convolutions should always be implementable as correlation, at least
when considering  0 padding for boundaries. There may be problem for
huge arrays, though - doing convolution from correlation without using
copies while staying fast may not always be easy, but I have never tried
to do so.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Sun May 24 09:09:13 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 24 May 2009 22:09:13 +0900
Subject: [SciPy-dev] 0.7.1: new schedule
In-Reply-To: <gv7d7k$fj7$2@ger.gmane.org>
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>	<gv3lvs$95u$1@ger.gmane.org>
	<4A163D68.6080803@ar.media.kyoto-u.ac.jp>
	<gv7d7k$fj7$2@ger.gmane.org>
Message-ID: <4A1946F9.4000908@ar.media.kyoto-u.ac.jp>

Pauli Virtanen wrote:
> Fri, 22 May 2009 14:51:36 +0900, David Cournapeau wrote:
> [clip]
>   
>> But not this one: the changes do not apply, there need to be other
>> commits as well - even by importing assert_to_tol code as well for the
>> tests to run, 5 tests are failing.
>>     
>
> I'd like to backport everything from trunk in scipy.special, except the 
> following:
>
> 	9600bcf... Revert "use np.power in rdist (test commit with bzr)"
> 	ff046c6... Do not detect Endianess in Cephes.
> 	3a99ec1... Add test case for ndtr.
>
> The bzr commit was broken, and the ndtr/endianness fixes break other 
> things. All other commits are bugfixes of some sort; no feature additions 
> here. What are fixed is:
>
> 	#924
> 	#803	- pbdv errors
> 	#503	- reimplementation of real Bessel I
> 	#854	- correct NaN returns for iv/jv/ etc when out-of-domain
> 	#853	- fix invalid return values from complex Bessel functions
> 	#623	- fix invalid return values from real Bessel J
> 	#852	- make Bessel zeros computable for orders > 100
>
> Plus some minor documentation fixes.
>
> And also the following:
>
> - The special functions now raise warnings instead of printing to stderr,
>   if the errprint toggle is enabled (it was and is disabled by default.
>
> - The real-valued jve, ive, yve, kve, and kv functions return now NAN
>   when the result would be complex-valued. (Previously they returned the
>   real part.)
>
>     ***
>
> I have the stuff already backported here:
>
> 	http://github.com/pv/scipy-work/commits/work-0.7.x
>
> The differences to trunk are minimal:
>
> 	git diff --stat ..svn/trunk -- scipy/special
>
> 	 scipy/special/cephes/mconf.h      |   12 +++++++++++-
> 	 scipy/special/tests/test_basic.py |    4 +++-
> 	 2 files changed, 14 insertions(+), 2 deletions(-)
>
> I used this script to do the backporting:
>
> 	http://www.iki.fi/pav/tmp/git-cherry-tree
>
> And it went like this:
>
> 	http://www.iki.fi/pav/tmp/special-rebase-trace.txt
>
> The -o mode of git-cherry-tree seems to be useful for keeping track of 
> stuff to backport.
>   

Great, thanks.

> BTW, do we want to do backports commit-by-commit, or in larger chunks?
>   

One commit per bug fix would be good, for example. I generally backport
things using git cherry-pick -n + inplace build + tests, and refer to
the svn commit in the git commit. There may be a better way.

cheers,

David


From josef.pktd at gmail.com  Sun May 24 09:46:08 2009
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 24 May 2009 09:46:08 -0400
Subject: [SciPy-dev] Fixing correlate: handling api breakage ?
In-Reply-To: <4A193CFA.7070702@ar.media.kyoto-u.ac.jp>
References: <4A191E65.1040607@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905240428y2985c4d5p6ba7c907238ef9ec@mail.gmail.com>
	<4A1931A3.5060109@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905240524l4f822574qc56940d758574686@mail.gmail.com>
	<4A193CFA.7070702@ar.media.kyoto-u.ac.jp>
Message-ID: <1cd32cbb0905240646w37ecf138s89a73a3b02f470cf@mail.gmail.com>

On Sun, May 24, 2009 at 8:26 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> josef.pktd at gmail.com wrote:
>> On Sun, May 24, 2009 at 7:38 AM, David Cournapeau
>> <david at ar.media.kyoto-u.ac.jp> wrote:
>>
>>> josef.pktd at gmail.com wrote:
>>>
>>>> On Sun, May 24, 2009 at 6:16 AM, David Cournapeau
>>>> <david at ar.media.kyoto-u.ac.jp> wrote:
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> ? ?I have taken a look at the correlate function in scipy.signal. There
>>>>> are several problems with it. First, it is wrong on several accounts:
>>>>> ? ? ? - It assumes that the correlation of complex numbers corresponds
>>>>> to complex multiplication, but this is not the definition followed by
>>>>> most textbooks, at least as far as signal processing is concerned.
>>>>> ? ? ? - More significantly, it is wrong with respect to the ordering:
>>>>> it assumes that correlate(a, b) == correlate(b, a), which is not true in
>>>>> general.
>>>>>
>>>>>
>>>> I don't see this in the results. There was recently the report on the
>>>> mailing list that np.correlate
>>>> and signal.correlate switch arrays if the second array is longer.
>>>>
>>>>
>>>>
>>>>>>> signal.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0])
>>>>>>>
>>>>>>>
>>>> array([0, 0, 1, 2, 0, 0, 0, 0, 0])
>>>>
>>>>
>>>>>>> signal.correlate([0, 0, 1, 0, 0],[1, 2, 0, 0, 0] )
>>>>>>>
>>>>>>>
>>>> array([0, 0, 0, 0, 0, 2, 1, 0, 0])
>>>>
>>>>
>>> Well, you just happened to have very peculiar entries :)
>>>
>>> signal.correlate([-1, -2, -3], [1, 2, 3])
>>> -> array([ -3, ?-8, -14, ?-8, ?-3])
>>>
>>> signal.correlate([1, 2, 3], [-1, -2, -3])
>>> -> array([ -3, ?-8, -14, ?-8, ?-3])
>>>
>>
>> One of your arrays is just the negative of the other, and correlate is
>> the same in this case. For other cases, the results differ
>>
>
> Gr, you're right of course :) But it still fails for arrays where the
> second argument has any dimension which is larger than the first one. I
> don't know if the assumption is relied on in the C implementation (the
> arrays are inverted in the C code in that case - even though they seem
> to be already inverted in python).

I read somewhere in the description for one of the correlate that the
code is faster if the first array is longer than the second. I thought
that maybe the python code forgot to switch it back, when the two
arrays are switched for the c-code.


>
>> I looked at it only for examples to calculate auto-correlation and
>> cross-correlation in time series, and had to try out to see which
>> version works best.
>>
>
> Yes, it depends. I know that correlate is way too slow for my own usage
> in speech processing, for example for linear prediction coding. As only
> a few lags are necessary, direct implementation is often faster than FFT
> one - I have my own straightfoward autocorrelation in scikits.talkbox; I
> believe matlab xcorr (1d correlation) always uses the FFT.
>
> I know other people have problems with the scipy.signal correlate as
> well for large arrays (the C code does a copy if the inputs are not
> contiguous, for example - my own code using iterators should not need
> any copy of inputs).

I'm new to this, but following the discussion on the cython list, I
thought that to have fast c code you always need to copy to a
contiguous array.

Are there performance differences  between calling it once with a huge
array versus calling it very often (eg. for sequential "online"
computation) for medium sized arrays.

What is huge in your talk box applications?

>
>> Are the convolve in all cases compatible, identical (through
>> delegation) to correlate?
>>
>
> I don't think so - I don't think convolution uses the conjugate for
> complex values. But I don't know any use of complex convolution,
> although I am sure there is. Correlation is always defined with the
> complex conjugate of the second argument AFAIK. For real cases,
> convolutions should always be implementable as correlation, at least
> when considering ?0 padding for boundaries. There may be problem for
> huge arrays, though - doing convolution from correlation without using
> copies while staying fast may not always be easy, but I have never tried
> to do so.
>

I was only looking at some implementation where convolve is just
correlate with second array reversed, as in the description of
numpy.correlate. So I thought that convolve and correlate should have
the same interface, whether they are implemented only once or
separately.

If we have different implementations based on performance for
different use cases, it would we very helpful to add your notes to the
docs.

Josef


From gael.varoquaux at normalesup.org  Mon May 25 03:05:20 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 25 May 2009 09:05:20 +0200
Subject: [SciPy-dev] RFR: Add io to ndimage
In-Reply-To: <9457e7c80905201634h2ecc6002l1f3ffb04fa8da1b5@mail.gmail.com>
References: <9457e7c80905201634h2ecc6002l1f3ffb04fa8da1b5@mail.gmail.com>
Message-ID: <20090525070520.GA23079@phare.normalesup.org>

On Thu, May 21, 2009 at 01:34:37AM +0200, St?fan van der Walt wrote:
> Hi all,

> It's bothered me for a long time that our image reading routines are
> hidden in scipy.misc.  Here is a patch that implements image reading
> via the PIL array interface.  Please let me know whether you like this
> or not.

Docstring of imread seems wrong: the fname input argument is not a PIL
image, it is the filename of the input image.

Also, I think you should fail gracefully if PIL is not installed. RIght
now this will give a traceback that might surprise the user
('ImportError: no module named PIL').

My 2 cents,

Ga?l


From stefan at sun.ac.za  Mon May 25 04:56:34 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 25 May 2009 10:56:34 +0200
Subject: [SciPy-dev] RFR: Add io to ndimage
In-Reply-To: <20090525070520.GA23079@phare.normalesup.org>
References: <9457e7c80905201634h2ecc6002l1f3ffb04fa8da1b5@mail.gmail.com> 
	<20090525070520.GA23079@phare.normalesup.org>
Message-ID: <9457e7c80905250156w768c2bcy445eb6c337bcfdc8@mail.gmail.com>

2009/5/25 Gael Varoquaux <gael.varoquaux at normalesup.org>:
> On Thu, May 21, 2009 at 01:34:37AM +0200, St?fan van der Walt wrote:
> Docstring of imread seems wrong: the fname input argument is not a PIL
> image, it is the filename of the input image.
>
> Also, I think you should fail gracefully if PIL is not installed. RIght
> now this will give a traceback that might surprise the user
> ('ImportError: no module named PIL').

Thank you for the feedback, Ga?l:

http://projects.scipy.org/scipy/changeset/5765

Regards
St?fan


From cournape at gmail.com  Mon May 25 22:32:11 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 26 May 2009 11:32:11 +0900
Subject: [SciPy-dev] 0.7.1: new schedule
In-Reply-To: <gv7d7k$fj7$2@ger.gmane.org>
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>
	<gv3lvs$95u$1@ger.gmane.org> <4A163D68.6080803@ar.media.kyoto-u.ac.jp>
	<gv7d7k$fj7$2@ger.gmane.org>
Message-ID: <5b8d13220905251932m71928791w9f990b294d726def@mail.gmail.com>

On Sat, May 23, 2009 at 8:41 AM, Pauli Virtanen <pav at iki.fi> wrote:
>
>
> BTW, do we want to do backports commit-by-commit, or in larger chunks?
> Currently, this is commit-by-commit.

I have tested your changes on linux and wine, and I pushed your
changes into the 0.7.x branch. I still need to do a quick test for Mac
OS X, and I will then release the RC for 0.7.x.

cheers,

David


From tpk at kraussfamily.org  Mon May 25 22:40:43 2009
From: tpk at kraussfamily.org (Tom K.)
Date: Tue, 26 May 2009 02:40:43 +0000 (UTC)
Subject: [SciPy-dev] RFR: #902 firwin upgrades
References: <loom.20090511T023202-872@post.gmane.org>
	<e06186140905102109o7e9fb26brb6d279422b5d3bc1@mail.gmail.com>
	<loom.20090512T011204-100@post.gmane.org>
	<e06186140905112034y4fb2401emc770315cc11dcfe5@mail.gmail.com>
Message-ID: <loom.20090526T021918-407@post.gmane.org>

Thanks for the earlier review and discussion Chuck and others.  
Sorry it has taken so long to get back to this issue.  For now I have 
moved the PassBand class inside firwin to take it out of the 
filter_design module's local variables; let me know if you'd prefer
a different option for this class.  I've submitted a new patch
to the track issue, 
  http://projects.scipy.org/scipy/ticket/902

I think this patch is worth including for the following reasons:
  - it adds functionality to firwin that has been requested by scipy users,
in a way that is backwards compatible with the previous lowpass-only
functionality, that functionality being highpass, bandpass, bandstop, and
general multiband filters; also an option to *not* scale the filter is added
 - it adds a *test* for firwin where none existed before, testing different
filter lengths and options and making sure the frequency response is sane;
the test of course exercises not only the old functionality but the new as well.

Other questions about organization of signal (e.g. creating new modules for
fir filtering, or equiripple and window fir filter submodules) and
harmonization of different filter design routines (remez and window)
are not precluded and are largely orthogonal to incorporation of the new
functionality to this one small routine.

I see scipy as being an incredible resource and I would like to add this
tick mark to its feature list to compete with other commercial tools.
I would love to have scipy developed into a world class tool in every way
so that potential adopters are not able to cite the lack of some particular
feature as an excuse for sticking with a different commercial tool.





From pav at iki.fi  Tue May 26 03:16:04 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 26 May 2009 07:16:04 +0000 (UTC)
Subject: [SciPy-dev] 0.7.1: new schedule
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>
	<gv3lvs$95u$1@ger.gmane.org> <4A163D68.6080803@ar.media.kyoto-u.ac.jp>
	<gv7d7k$fj7$2@ger.gmane.org>
	<5b8d13220905251932m71928791w9f990b294d726def@mail.gmail.com>
Message-ID: <gvg4vk$uq2$1@ger.gmane.org>

Hi,

Tue, 26 May 2009 11:32:11 +0900, David Cournapeau kirjoitti:
> On Sat, May 23, 2009 at 8:41 AM, Pauli Virtanen <pav at iki.fi> wrote:
>>
>>
>> BTW, do we want to do backports commit-by-commit, or in larger chunks?
>> Currently, this is commit-by-commit.
> 
> I have tested your changes on linux and wine, and I pushed your changes
> into the 0.7.x branch. I still need to do a quick test for Mac OS X, and
> I will then release the RC for 0.7.x.

Thanks!

	Pauli



From Norbert.Nemec.List at gmx.de  Wed May 27 05:59:39 2009
From: Norbert.Nemec.List at gmx.de (Norbert Nemec)
Date: Wed, 27 May 2009 11:59:39 +0200
Subject: [SciPy-dev] Updating blitz++ within weave?
Message-ID: <20090527095939.123620@gmx.net>

Hi there,

it seems that the blitz++ code within weave is branched off some ancient version and only minimally patched. Wouldn't it make sense to pull the latest version from the blitz CVS repo?

Greetings,
Norbert
-- 
Neu: GMX FreeDSL Komplettanschluss mit DSL 6.000 Flatrate + Telefonanschluss f?r nur 17,95 Euro/mtl.!* http://portal.gmx.net/de/go/dsl02


From cournape at gmail.com  Thu May 28 03:29:38 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 28 May 2009 16:29:38 +0900
Subject: [SciPy-dev] 0.7.1: new schedule
In-Reply-To: <gvg4vk$uq2$1@ger.gmane.org>
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>
	<gv3lvs$95u$1@ger.gmane.org> <4A163D68.6080803@ar.media.kyoto-u.ac.jp>
	<gv7d7k$fj7$2@ger.gmane.org>
	<5b8d13220905251932m71928791w9f990b294d726def@mail.gmail.com>
	<gvg4vk$uq2$1@ger.gmane.org>
Message-ID: <5b8d13220905280029s6a5a4426x10cc43fe8cd9cdc5@mail.gmail.com>

On Tue, May 26, 2009 at 4:16 PM, Pauli Virtanen <pav at iki.fi> wrote:
> Hi,
>
> Tue, 26 May 2009 11:32:11 +0900, David Cournapeau kirjoitti:
>> On Sat, May 23, 2009 at 8:41 AM, Pauli Virtanen <pav at iki.fi> wrote:
>>>
>>>
>>> BTW, do we want to do backports commit-by-commit, or in larger chunks?
>>> Currently, this is commit-by-commit.
>>
>> I have tested your changes on linux and wine, and I pushed your changes
>> into the 0.7.x branch. I still need to do a quick test for Mac OS X, and
>> I will then release the RC for 0.7.x.
>
> Thanks!

Ok, there are 2 failures on mac os x in scipy.special. test_yn_zeros
and test_ynp_zeros are (way) below the tolerance you set up, I put the
output of the test in #953.

If you don't have access to a mac os x machine, don't hesitate to ask
me some additional info,

David


From david at ar.media.kyoto-u.ac.jp  Thu May 28 07:19:35 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 28 May 2009 20:19:35 +0900
Subject: [SciPy-dev] Fixing correlate: handling api breakage ?
In-Reply-To: <1cd32cbb0905240646w37ecf138s89a73a3b02f470cf@mail.gmail.com>
References: <4A191E65.1040607@ar.media.kyoto-u.ac.jp>	<1cd32cbb0905240428y2985c4d5p6ba7c907238ef9ec@mail.gmail.com>	<4A1931A3.5060109@ar.media.kyoto-u.ac.jp>	<1cd32cbb0905240524l4f822574qc56940d758574686@mail.gmail.com>	<4A193CFA.7070702@ar.media.kyoto-u.ac.jp>
	<1cd32cbb0905240646w37ecf138s89a73a3b02f470cf@mail.gmail.com>
Message-ID: <4A1E7347.6020201@ar.media.kyoto-u.ac.jp>

josef.pktd at gmail.com wrote:
>
> I read somewhere in the description for one of the correlate that the
> code is faster if the first array is longer than the second.

Yes, but that's an implementation detail - it should not affect the
result. The speed difference may come from zero padding handling. But if
copies are acceptable, correlate(a, b) and correlate(b, a) can be
obtained from 'inverted index', i.e., ignoring boundaries effects
correlate(a, b) == correlate(b, a)[::-1, ::-1] (for rank 2 arrays). This
may be much faster.

FWIW, matlab is much slower at doing xcorr2(a, b) than xcorr2(b, a) if b
has significantly smaller number of items.

>
> I'm new to this, but following the discussion on the cython list, I
> thought that to have fast c code you always need to copy to a
> contiguous array.
>   

It depends on the computation. The idea is that generally, it is really
slow to do "memory jumps" on common CPU, especially when the data cannot
fit the cache anymore.  Doing operations one item after the other in
memory is much faster than doing it 'randomly' (for a real explanation,
you may consult http://people.redhat.com/drepper/cpumemory.pdf). For
things like item/item computation (cos, sin, or things which are
processed axis per axis), copy + contiguity is often the way to go,
because this way, the items you need for computation are near in term of
memory location.

Now, for correlation or convolution, it is more complex, because you
need to do the operation for a neighborhood. The whole problem is that
the neighborhood is defined in terms of the image (spacial locality, the
multi-dimensional indexes are close to each other), not in terms of
memory. For rank one array, it is the same for contiguous arrays. For 2d
arrays, it is already a bit difficult, but for rank 3 or rank 4, it is
much more complex: for any point in a 3d image, for example, the nearest
voxels can be pretty far in terms of memory. To match spatial locality
to memory locality, you would need a more complex memory model than what
numpy can offer out of the box, I think.

In my rewriting of the correlate function, I implemented things in term
of a 'neighborhood' iterator (it looks like this is nothing new - some
package like itk already had this for years). The resulting code is
simpler, because the complexity is hidden in the iterator, and speed is
approximately the same. We are quite slower than matlab, though (a
factor of 3-4 compared to imfilter).

The code is there:

http://github.com/cournape/scipy3/tree/clean_correlate

> Are there performance differences  between calling it once with a huge
> array versus calling it very often (eg. for sequential "online"
> computation) for medium sized arrays.
>
> What is huge in your talk box applications?
>   

In my own case, it not huge or even big. For many audio tasks, you need
to compute things on a frame-per-frame basis (for example, splitting a
big signal into overlapping segments of say 512 samples, and do 1d
autocorrelation of this). So the usual trick is to split the big signal
into a matrix where each row or column is one frame, and call a function
which processes in one axis. None of the correlation function in
numpy/scipy has this AFAIK. Some people sent me code for talkbox so that
it can handle correlation with very big arrays, but it is one or two
dimensions only.

cheers,

David


From matthieu.brucher at gmail.com  Fri May 29 05:14:22 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 29 May 2009 11:14:22 +0200
Subject: [SciPy-dev] Setup doesn't pickup Intel Fortran compiler
Message-ID: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>

Hi,

I have trouble getting Scipy (and numpy) using the correct Fortran compiler :

Found executable /appli/intel/10.1.018/intel64/cce/bin/icc
customize GnuFCompiler
Found executable /appli/intel/10.1.018/intel64/fce/bin/ifort
gnu: no Fortran 90 compiler found
Found executable /usr/bin/g77
gnu: no Fortran 90 compiler found
customize IntelFCompiler
customize LaheyFCompiler
Could not locate executable lf95
lahey: no Fortran 90 compiler found
lahey: no Fortran 90 compiler found
customize PGroupFCompiler
Could not locate executable pgf90
pg: no Fortran 90 compiler found
pg: no Fortran 90 compiler found
customize AbsoftFCompiler
Could not locate executable f90
absoft: no Fortran 90 compiler found
absoft: no Fortran 90 compiler found
absoft: no Fortran 90 compiler found
absoft: no Fortran 90 compiler found
absoft: no Fortran 90 compiler found
absoft: no Fortran 90 compiler found
customize NAGFCompiler
Found executable /usr/bin/f95
customize VastFCompiler
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize CompaqFCompiler
Could not locate executable fort
compaq: no Fortran 90 compiler found
compaq: no Fortran 90 compiler found
customize IntelItaniumFCompiler
customize IntelEM64TFCompiler
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize UnixCCompiler
customize UnixCCompiler using scons
Found executable /appli/intel/10.1.018/intel64/cce/bin/icpc

Why is it trying to search for another Fortran compiler when it
actually found the Intel one?
I used python setup.py install or python setup.py install
--fcompiler=intelem, but the output was the same :|

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From david at ar.media.kyoto-u.ac.jp  Fri May 29 05:14:17 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 29 May 2009 18:14:17 +0900
Subject: [SciPy-dev] Setup doesn't pickup Intel Fortran compiler
In-Reply-To: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>
References: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>
Message-ID: <4A1FA769.60102@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
> Hi,
>
> I have trouble getting Scipy (and numpy) using the correct Fortran compiler :
>
> Found executable /appli/intel/10.1.018/intel64/cce/bin/icc
> customize GnuFCompiler
> Found executable /appli/intel/10.1.018/intel64/fce/bin/ifort
> gnu: no Fortran 90 compiler found
> Found executable /usr/bin/g77
> gnu: no Fortran 90 compiler found
> customize IntelFCompiler
> customize LaheyFCompiler
> Could not locate executable lf95
> lahey: no Fortran 90 compiler found
> lahey: no Fortran 90 compiler found
> customize PGroupFCompiler
> Could not locate executable pgf90
> pg: no Fortran 90 compiler found
> pg: no Fortran 90 compiler found
> customize AbsoftFCompiler
> Could not locate executable f90
> absoft: no Fortran 90 compiler found
> absoft: no Fortran 90 compiler found
> absoft: no Fortran 90 compiler found
> absoft: no Fortran 90 compiler found
> absoft: no Fortran 90 compiler found
> absoft: no Fortran 90 compiler found
> customize NAGFCompiler
> Found executable /usr/bin/f95
> customize VastFCompiler
> customize GnuFCompiler
> gnu: no Fortran 90 compiler found
> gnu: no Fortran 90 compiler found
> customize CompaqFCompiler
> Could not locate executable fort
> compaq: no Fortran 90 compiler found
> compaq: no Fortran 90 compiler found
> customize IntelItaniumFCompiler
> customize IntelEM64TFCompiler
> customize Gnu95FCompiler
> Found executable /usr/bin/gfortran
> customize Gnu95FCompiler
> customize UnixCCompiler
> customize UnixCCompiler using scons
> Found executable /appli/intel/10.1.018/intel64/cce/bin/icpc
>
> Why is it trying to search for another Fortran compiler when it
> actually found the Intel one?
>   

Maybe nobody knows :)
> I used python setup.py install or python setup.py install
> --fcompiler=intelem, but the output was the same :|
>   

Yes, intelem is for ia architectures IIRC

I have just tried compiling numpy with both icc/ifort (11.0), it works
without trouble with numscons:

CFLAGS="-debug" FFLAGS="-debug" CC=icc F77=ifort python setupscons.py scons

I used the -debug flag to avoid waiting forever - the whole numpy test
suite passes.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Fri May 29 05:15:57 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 29 May 2009 18:15:57 +0900
Subject: [SciPy-dev] Setup doesn't pickup Intel Fortran compiler
In-Reply-To: <4A1FA769.60102@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>
	<4A1FA769.60102@ar.media.kyoto-u.ac.jp>
Message-ID: <4A1FA7CD.7070806@ar.media.kyoto-u.ac.jp>

David Cournapeau wrote:
>
> Yes, intelem is for ia architectures IIRC
>
> I have just tried compiling numpy with both icc/ifort (11.0), it works
> without trouble with numscons:
>
> CFLAGS="-debug" FFLAGS="-debug" CC=icc F77=ifort python setupscons.py scons
>
> I used the -debug flag to avoid waiting forever - the whole numpy test
> suite passes.
>   

Just after sending the email, I saw your bug-report :)

You may want to try the last numscons code:

http://github.com/cournape/numscons/tree/master

cheers,

David


From matthieu.brucher at gmail.com  Fri May 29 05:35:42 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 29 May 2009 11:35:42 +0200
Subject: [SciPy-dev] Setup doesn't pickup Intel Fortran compiler
In-Reply-To: <4A1FA7CD.7070806@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>
	<4A1FA769.60102@ar.media.kyoto-u.ac.jp>
	<4A1FA7CD.7070806@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0905290235t7b9fa3edh3a43419a7fb00002@mail.gmail.com>

2009/5/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> David Cournapeau wrote:
>>
>> Yes, intelem is for ia architectures IIRC
>>
>> I have just tried compiling numpy with both icc/ifort (11.0), it works
>> without trouble with numscons:
>>
>> CFLAGS="-debug" FFLAGS="-debug" CC=icc F77=ifort python setupscons.py scons
>>
>> I used the -debug flag to avoid waiting forever - the whole numpy test
>> suite passes.
>>
>
> Just after sending the email, I saw your bug-report :)
>
> You may want to try the last numscons code:
>
> http://github.com/cournape/numscons/tree/master
>
> cheers,

Thanks, I'll try this at once ;)
I didn't remember you switched to github :|

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From matthieu.brucher at gmail.com  Fri May 29 05:48:02 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 29 May 2009 11:48:02 +0200
Subject: [SciPy-dev] Setup doesn't pickup Intel Fortran compiler
In-Reply-To: <e76aa17f0905290235t7b9fa3edh3a43419a7fb00002@mail.gmail.com>
References: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>
	<4A1FA769.60102@ar.media.kyoto-u.ac.jp>
	<4A1FA7CD.7070806@ar.media.kyoto-u.ac.jp>
	<e76aa17f0905290235t7b9fa3edh3a43419a7fb00002@mail.gmail.com>
Message-ID: <e76aa17f0905290248r26be5621ybdb360103cedaba7@mail.gmail.com>

2009/5/29 Matthieu Brucher <matthieu.brucher at gmail.com>:
> 2009/5/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>> David Cournapeau wrote:
>>>
>>> Yes, intelem is for ia architectures IIRC
>>>
>>> I have just tried compiling numpy with both icc/ifort (11.0), it works
>>> without trouble with numscons:
>>>
>>> CFLAGS="-debug" FFLAGS="-debug" CC=icc F77=ifort python setupscons.py scons
>>>
>>> I used the -debug flag to avoid waiting forever - the whole numpy test
>>> suite passes.
>>>
>>
>> Just after sending the email, I saw your bug-report :)
>>
>> You may want to try the last numscons code:
>>
>> http://github.com/cournape/numscons/tree/master
>>
>> cheers,
>
> Thanks, I'll try this at once ;)
> I didn't remember you switched to github :|
>
> Matthieu

Now, I don't have an error, but the folders are still empty:

Executing scons command (pkg is numpy.core):
/users/brucher/pau112/local/x86_64_RH5/bin/python
"/users/brucher/pau112/local/x86_64_RH5/lib/python2.5/site-packages/numscons/scons-local/scons.py"
-f numpy/core/SConstruct -I. scons_tool_path="" src_dir="numpy/core"
pkg_name="numpy.core" log_level=50
distutils_libdir="../../../../build/lib.linux-x86_64-2.5" cc_opt=icc
cc_opt_path="/appli/intel/10.1.018/intel64/cce/bin" cxx_opt=icpc
cxx_opt_path="/appli/intel/10.1.018/intel64/cce/bin"
include_bootstrap=../../../../numpy/core/include silent=0
bootstrapping=1
scons: Reading SConscript files ...

scons: warning: Ignoring missing SConscript 'build/scons/numpy/core/SConscript'
File "/users/brucher/pau112/local/x86_64_RH5/lib/python2.5/site-packages/numscons/core/numpyenv.py",
line 108, in DistutilsSConscript
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.

gfortran is still detected in the wrong folder: f77_opt=gfortran
f77_opt_path="/appli/intel/10.1.018/intel64/fce/bin"
I deleted the numpy folder completely and started again from scratch,
but I keep getting the same errors :|
I deleted the environment variables, and the fortran compilers seem
OK. If I export F77=ifort, it gives me something even stranger:
f77_opt=fortran f77_opt_path="/appli/intel/10.1.018/intel64/fce/bin"

The strangest thing for me is that the scons subfolders are almost
empty (just have the .dblite file).

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From david at ar.media.kyoto-u.ac.jp  Fri May 29 05:45:31 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 29 May 2009 18:45:31 +0900
Subject: [SciPy-dev] Setup doesn't pickup Intel Fortran compiler
In-Reply-To: <e76aa17f0905290248r26be5621ybdb360103cedaba7@mail.gmail.com>
References: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>	<4A1FA769.60102@ar.media.kyoto-u.ac.jp>	<4A1FA7CD.7070806@ar.media.kyoto-u.ac.jp>	<e76aa17f0905290235t7b9fa3edh3a43419a7fb00002@mail.gmail.com>
	<e76aa17f0905290248r26be5621ybdb360103cedaba7@mail.gmail.com>
Message-ID: <4A1FAEBB.2020202@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
> 2009/5/29 Matthieu Brucher <matthieu.brucher at gmail.com>:
>   
>> 2009/5/29 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>>     
>>> David Cournapeau wrote:
>>>       
>>>> Yes, intelem is for ia architectures IIRC
>>>>
>>>> I have just tried compiling numpy with both icc/ifort (11.0), it works
>>>> without trouble with numscons:
>>>>
>>>> CFLAGS="-debug" FFLAGS="-debug" CC=icc F77=ifort python setupscons.py scons
>>>>
>>>> I used the -debug flag to avoid waiting forever - the whole numpy test
>>>> suite passes.
>>>>
>>>>         
>>> Just after sending the email, I saw your bug-report :)
>>>
>>> You may want to try the last numscons code:
>>>
>>> http://github.com/cournape/numscons/tree/master
>>>
>>> cheers,
>>>       
>> Thanks, I'll try this at once ;)
>> I didn't remember you switched to github :|
>>
>> Matthieu
>>     
>
> Now, I don't have an error, but the folders are still empty:
>   

Which version of numpy are you using ? Are you building from a svn
version or a source tarball ? The missing SConscript warning is strange
(it should not be a warning - missing scons files is pretty fatal :) ).

> gfortran is still detected in the wrong folder: f77_opt=gfortran
> f77_opt_path="/appli/intel/10.1.018/intel64/fce/bin"
> I deleted the numpy folder completely and started again from scratch,
> but I keep getting the same errors :|
> I deleted the environment variables, and the fortran compilers seem
> OK. If I export F77=ifort, it gives me something even stranger:
> f77_opt=fortran f77_opt_path="/appli/intel/10.1.018/intel64/fce/bin"
>   

Hm, the problem is that the scons command in numpy.distutils was
designed to make sure the same compiler was used as distutils. But here,
we would like to bypass distutils completely... I could add an option to
the scons command to tell numscons to ignore compilers given by
distutils - or maybe just bypassing it altogether. When I started
numscons, I tried to be as compatible as possible with distutils, but
that's a lost battle. Just using a default compiler, and customization
from CC would be better and easier to set up...

The code to detect fortran compiler in distutils is in
numpy/distutils/fcompiler/__init__.py (line 688 in last revision). As a
temporary workaround, you could remove every compiler of the list except
intel. The problem is that some compilers depend on other compilers, and
I have never really taken the time to understand how this was supposed
to work.

cheers,

David


From matthieu.brucher at gmail.com  Fri May 29 06:17:18 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 29 May 2009 12:17:18 +0200
Subject: [SciPy-dev] Setup doesn't pickup Intel Fortran compiler
In-Reply-To: <4A1FAEBB.2020202@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>
	<4A1FA769.60102@ar.media.kyoto-u.ac.jp>
	<4A1FA7CD.7070806@ar.media.kyoto-u.ac.jp>
	<e76aa17f0905290235t7b9fa3edh3a43419a7fb00002@mail.gmail.com>
	<e76aa17f0905290248r26be5621ybdb360103cedaba7@mail.gmail.com>
	<4A1FAEBB.2020202@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0905290317k633a2d50s5e78629e05c890b3@mail.gmail.com>

>> Now, I don't have an error, but the folders are still empty:
>>
>
> Which version of numpy are you using ? Are you building from a svn
> version or a source tarball ? The missing SConscript warning is strange
> (it should not be a warning - missing scons files is pretty fatal :) ).

The numpy 1.3.0 tarball from the sf webpage.
Missing SConscript is something I use regularly, it isn't a problem.
The really strange thing is that the folders are empty :|

>> gfortran is still detected in the wrong folder: f77_opt=gfortran
>> f77_opt_path="/appli/intel/10.1.018/intel64/fce/bin"
>> I deleted the numpy folder completely and started again from scratch,
>> but I keep getting the same errors :|
>> I deleted the environment variables, and the fortran compilers seem
>> OK. If I export F77=ifort, it gives me something even stranger:
>> f77_opt=fortran f77_opt_path="/appli/intel/10.1.018/intel64/fce/bin"
>>
>
> Hm, the problem is that the scons command in numpy.distutils was
> designed to make sure the same compiler was used as distutils. But here,
> we would like to bypass distutils completely... I could add an option to
> the scons command to tell numscons to ignore compilers given by
> distutils - or maybe just bypassing it altogether. When I started
> numscons, I tried to be as compatible as possible with distutils, but
> that's a lost battle. Just using a default compiler, and customization
> from CC would be better and easier to set up...
>
> The code to detect fortran compiler in distutils is in
> numpy/distutils/fcompiler/__init__.py (line 688 in last revision). As a
> temporary workaround, you could remove every compiler of the list except
> intel. The problem is that some compilers depend on other compilers, and
> I have never really taken the time to understand how this was supposed
> to work.

It seems indeed that the Fortran detection is tricky ;)

I've deleted some compilers in 'linux', and I had to add intel in
'posix'. I had to use intelem for 'linux' as I use a 64bits processor,
which is thus the EM64T compiler. For 'posix', it works as long as I
do not export LD_PRELOAD with something ld doesn't like.
So know, it works, safe for the fact that f77_opt is fortran and not ifort.

Matthieu, one step closer to using the correct Fortran compiler...
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From matthieu.brucher at gmail.com  Fri May 29 06:33:38 2009
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 29 May 2009 12:33:38 +0200
Subject: [SciPy-dev] Setup doesn't pickup Intel Fortran compiler
In-Reply-To: <e76aa17f0905290317k633a2d50s5e78629e05c890b3@mail.gmail.com>
References: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>
	<4A1FA769.60102@ar.media.kyoto-u.ac.jp>
	<4A1FA7CD.7070806@ar.media.kyoto-u.ac.jp>
	<e76aa17f0905290235t7b9fa3edh3a43419a7fb00002@mail.gmail.com>
	<e76aa17f0905290248r26be5621ybdb360103cedaba7@mail.gmail.com>
	<4A1FAEBB.2020202@ar.media.kyoto-u.ac.jp>
	<e76aa17f0905290317k633a2d50s5e78629e05c890b3@mail.gmail.com>
Message-ID: <e76aa17f0905290333j168d3692qb795fa945ce72a83@mail.gmail.com>

> It seems indeed that the Fortran detection is tricky ;)
>
> I've deleted some compilers in 'linux', and I had to add intel in
> 'posix'. I had to use intelem for 'linux' as I use a 64bits processor,
> which is thus the EM64T compiler. For 'posix', it works as long as I
> do not export LD_PRELOAD with something ld doesn't like.
> So know, it works, safe for the fact that f77_opt is fortran and not ifort.
>
> Matthieu, one step closer to using the correct Fortran compiler...

OK, I found what's missing. In distutils/scons.py, you only test for
intel, not intele or intelem, which means the name is then fortran
instead of ifort. (line 109).

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From pav at iki.fi  Fri May 29 14:36:51 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 29 May 2009 18:36:51 +0000 (UTC)
Subject: [SciPy-dev] 0.7.1: new schedule
References: <4A14E7AA.3060503@ar.media.kyoto-u.ac.jp>
	<gv3lvs$95u$1@ger.gmane.org> <4A163D68.6080803@ar.media.kyoto-u.ac.jp>
	<gv7d7k$fj7$2@ger.gmane.org>
	<5b8d13220905251932m71928791w9f990b294d726def@mail.gmail.com>
	<gvg4vk$uq2$1@ger.gmane.org>
	<5b8d13220905280029s6a5a4426x10cc43fe8cd9cdc5@mail.gmail.com>
Message-ID: <gvpa03$c2q$1@ger.gmane.org>

Thu, 28 May 2009 16:29:38 +0900, David Cournapeau wrote:
[clip]
> Ok, there are 2 failures on mac os x in scipy.special. test_yn_zeros and
> test_ynp_zeros are (way) below the tolerance you set up, I put the
> output of the test in #953.

I think I found a single-line fix:

--- a.f	2009-05-29 00:12:14.000000000 +0300
+++ b.f	2009-05-29 21:27:18.246736714 +0300
@@ -159,7 +159,7 @@
                  BS=BS+2.0D0*F
                  SU=SU+(-1)**(K/2)*F/K
               ELSE IF (K.GT.1) THEN
-                 SV=SV+(-1)**(K/2)*K/(K*K-1.0)*F
+                 SV=SV+(-1)**(K/2)*K*F/(K*K-1.0)
               ENDIF
               F2=F1
 15            F1=F

Fixes the issue at least with Gfortran ffloat-store.

Who said multiplication is commutative :)

-- 
Pauli Virtanen



From david at ar.media.kyoto-u.ac.jp  Sat May 30 01:56:21 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 30 May 2009 14:56:21 +0900
Subject: [SciPy-dev] Setup doesn't pickup Intel Fortran compiler
In-Reply-To: <e76aa17f0905290317k633a2d50s5e78629e05c890b3@mail.gmail.com>
References: <e76aa17f0905290214s7aeb9b9bja67dc03b28012cbd@mail.gmail.com>	<4A1FA769.60102@ar.media.kyoto-u.ac.jp>	<4A1FA7CD.7070806@ar.media.kyoto-u.ac.jp>	<e76aa17f0905290235t7b9fa3edh3a43419a7fb00002@mail.gmail.com>	<e76aa17f0905290248r26be5621ybdb360103cedaba7@mail.gmail.com>	<4A1FAEBB.2020202@ar.media.kyoto-u.ac.jp>
	<e76aa17f0905290317k633a2d50s5e78629e05c890b3@mail.gmail.com>
Message-ID: <4A20CA85.90505@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
> The numpy 1.3.0 tarball from the sf webpage.
> Missing SConscript is something I use regularly, it isn't a problem.
>   

Ok, the SConscript files were not added to the tarballs - I fixed
MANIFEST.in so that won't happen again. I never use tarballs, and only
test them with distutils, which is why I have never caught this problem.

>
> It seems indeed that the Fortran detection is tricky ;)
>   

Yes - and to make it even more 'interesting', tool detection is not
great with scons either. The current scheme in numscons is very crude.

cheers,

David