From alan.mcintyre at gmail.com  Wed Oct  1 00:37:38 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Tue, 30 Sep 2008 21:37:38 -0700
Subject: [SciPy-dev] Next scipy release 0.7
In-Reply-To: <c7009a550809290955i125cb6c4tb7350edf54e036be@mail.gmail.com>
References: <web-111958955@uni-stuttgart.de>
	<c7009a550809290955i125cb6c4tb7350edf54e036be@mail.gmail.com>
Message-ID: <1d36917a0809302137u78a9644fu9da38e3a79157a40@mail.gmail.com>

On Mon, Sep 29, 2008 at 9:55 AM, Jarrod Millman <millman at berkeley.edu> wrote:
>> Just curious. Is there a new deadline for scipy 0.7 ?
>
> If anyone has time and interest in helping, it would make a huge
> difference in getting this release out.  If someone could hunt down
> the and fix the test failures and clean up all the deprecation
> warnings, we would be much closer to getting this release out the
> door.

I'll try to fix some of the deprecation warnings and/or look at
failing tests this week or over the weekend.

Alan


From millman at berkeley.edu  Wed Oct  1 01:11:46 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 30 Sep 2008 22:11:46 -0700
Subject: [SciPy-dev] Next scipy release 0.7
In-Reply-To: <1d36917a0809302137u78a9644fu9da38e3a79157a40@mail.gmail.com>
References: <web-111958955@uni-stuttgart.de>
	<c7009a550809290955i125cb6c4tb7350edf54e036be@mail.gmail.com>
	<1d36917a0809302137u78a9644fu9da38e3a79157a40@mail.gmail.com>
Message-ID: <c7009a550809302211w5acb6216j1d0b8e03b61079c7@mail.gmail.com>

On Tue, Sep 30, 2008 at 9:37 PM, Alan McIntyre <alan.mcintyre at gmail.com> wrote:
> On Mon, Sep 29, 2008 at 9:55 AM, Jarrod Millman <millman at berkeley.edu> wrote:
>>> Just curious. Is there a new deadline for scipy 0.7 ?
>>
>> If anyone has time and interest in helping, it would make a huge
>> difference in getting this release out.  If someone could hunt down
>> the and fix the test failures and clean up all the deprecation
>> warnings, we would be much closer to getting this release out the
>> door.
>
> I'll try to fix some of the deprecation warnings and/or look at
> failing tests this week or over the weekend.

Thanks!  The sooner we can get some of these issues resolved the
sooner we can get a release out.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From prabhu at aero.iitb.ac.in  Wed Oct  1 02:09:08 2008
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Wed, 01 Oct 2008 11:39:08 +0530
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>	<48E1A6A2.4050406@noaa.gov>	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
Message-ID: <48E31404.7000009@aero.iitb.ac.in>

Anne Archibald wrote:
> 2008/9/30 Peter <numpy-discussion at maubp.freeserve.co.uk>:
> I think the profusion of different implementations is an argument for
> including this in scipy. I think it is also an argument for providing
> a standard interface with (at least potentially) several different
> implementations. At the moment, that proposed interface looks like:
[...]
> In the first two cases, missing neighbors are represented with an
> infinite distance and an invalid index. In the last case, distances
> and indices are both either lists (if there's only one query point) or
> object arrays of lists (if there are many query points). If only one
> neighbor is requested, the array does not have a dimension of length 1
> in the which-neighbor position. If (potentially) many neighbors are
> returned, they are sorted by distance, nearest first.
> 
> What do you think of this interface?

This is nice.  How about a keyword argument "sort=True" that lets people 
choose if they want the results sorted by distance or not?  I have seen 
cases where unsorted results are useful.

cheers,
prabhu


From peridot.faceted at gmail.com  Wed Oct  1 05:46:45 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 1 Oct 2008 05:46:45 -0400
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <48E31404.7000009@aero.iitb.ac.in>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<48E31404.7000009@aero.iitb.ac.in>
Message-ID: <ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>

2008/10/1 Prabhu Ramachandran <prabhu at aero.iitb.ac.in>:
> Anne Archibald wrote:
>> 2008/9/30 Peter <numpy-discussion at maubp.freeserve.co.uk>:
>> I think the profusion of different implementations is an argument for
>> including this in scipy. I think it is also an argument for providing
>> a standard interface with (at least potentially) several different
>> implementations. At the moment, that proposed interface looks like:
> [...]
>> In the first two cases, missing neighbors are represented with an
>> infinite distance and an invalid index. In the last case, distances
>> and indices are both either lists (if there's only one query point) or
>> object arrays of lists (if there are many query points). If only one
>> neighbor is requested, the array does not have a dimension of length 1
>> in the which-neighbor position. If (potentially) many neighbors are
>> returned, they are sorted by distance, nearest first.
>>
>> What do you think of this interface?
>
> This is nice.  How about a keyword argument "sort=True" that lets people
> choose if they want the results sorted by distance or not?  I have seen
> cases where unsorted results are useful.

That is a possibility. My current code produces results that are in a
heap, so sorting can in principle be an O(n) activity. Since the
number of results returned is rarely going to be large, I'm not sure
the gain in efficiency is worth the loss in determinism. And
additional API complexity does have a cost - in an ideal world it
should at the least double the number of unit tests. I think perhaps
this should wait on a C implementation.

Anne


From peridot.faceted at gmail.com  Wed Oct  1 05:54:29 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 1 Oct 2008 05:54:29 -0400
Subject: [SciPy-dev] Proposal: scipy.spatial
In-Reply-To: <11F5C399-FE46-4321-AC9E-8B0699D06A84@nist.gov>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<d05265cb0809292140n27592a76gea1786c90756365b@mail.gmail.com>
	<ce557a360809301745x786457d4v9956a2803620bd8d@mail.gmail.com>
	<11F5C399-FE46-4321-AC9E-8B0699D06A84@nist.gov>
Message-ID: <ce557a360810010254t3c3e2572lb8987d68e3eb9a19@mail.gmail.com>

2008/9/30 Jonathan Guyer <guyer at nist.gov>:
>
> On Sep 30, 2008, at 8:45 PM, Anne Archibald wrote:
>
>> (Also I have no experience with SVN and
>> merging, but reports suggest it's a horrible experience.)
>
> It is not a horrible experience. It involves a little bookkeeping, but
> it's entirely manageable. Our collected instructions for handling
> branches of FiPy (derived largely from the SVN book at red-bean) is at http://matforge.org/fipy/browser/trunk/documentation/ADMINISTRATA.txt
> . We branch all the time.

Thanks for the pointer! I've now split it off as a separate branch,
"spatial". With luck it won't be much work to reintegrate it once 0.7
is out.

Anne


From david at ar.media.kyoto-u.ac.jp  Wed Oct  1 05:42:53 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 01 Oct 2008 18:42:53 +0900
Subject: [SciPy-dev] Proposal: scipy.spatial
In-Reply-To: <ce557a360810010254t3c3e2572lb8987d68e3eb9a19@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>	<d05265cb0809292140n27592a76gea1786c90756365b@mail.gmail.com>	<ce557a360809301745x786457d4v9956a2803620bd8d@mail.gmail.com>	<11F5C399-FE46-4321-AC9E-8B0699D06A84@nist.gov>
	<ce557a360810010254t3c3e2572lb8987d68e3eb9a19@mail.gmail.com>
Message-ID: <48E3461D.6090808@ar.media.kyoto-u.ac.jp>

Anne Archibald wrote:
>
> Thanks for the pointer! I've now split it off as a separate branch,
> "spatial". With luck it won't be much work to reintegrate it once 0.7
> is out.

This is also documented on the numpy wiki:

http://projects.scipy.org/scipy/numpy/wiki/MakingBranches

As Robert said, the problem of svn with merging is unlikely to happen in
your case (Problems arise mostly when you use multiple branches and
merge between them; as long as you have one branch which keeps following
the trunk, svn poor merging capabilities can be worked around with
svnmerge).

cheers,

David


From Per.Brodtkorb at ffi.no  Wed Oct  1 07:14:34 2008
From: Per.Brodtkorb at ffi.no (Per.Brodtkorb at ffi.no)
Date: Wed, 1 Oct 2008 13:14:34 +0200
Subject: [SciPy-dev] Fwd: tests and partial patch
	forscipy.stats.distributions
In-Reply-To: <1cd32cbb0809290551g7c00a5ddh44369352ece506d2@mail.gmail.com>
References: <1cd32cbb0809282100x75eccfb9la109e92032f798ac@mail.gmail.com>
	<1cd32cbb0809290551g7c00a5ddh44369352ece506d2@mail.gmail.com>
Message-ID: <1ED225FF18AA8B48AC192F7E1D032C6E0C5C1D@hbu-posten.ffi.no>

I have looked into the bugs in the patch of the discrete distributions you sent. The error in _ppf method of dlaplace can be corrected by replacing it with:

class dlaplace_gen(rv_discrete):
.....
    def _ppf(self, q, a):
        const = 1.0/(1+exp(-a))
        cons2 = 1+exp(a)
        ind = q < const
        return ceil(where(ind, log(q*cons2)/a-1, -log((1-q)*cons2)/a))
......

The tests written for the generic moment, fails for those distributions that set the limits (self.a and/or self.b) for the support of the distribution in the self._argcheck method. I think the generic_moment method is meant to be private to the self._munp method, since it does only limited input checks.
Perhaps better to rename generic_moment to _generic_moment

Per A.

-----Opprinnelig melding-----
Fra: scipy-dev-bounces at scipy.org [mailto:scipy-dev-bounces at scipy.org] P? vegne av josef.pktd at gmail.com
Sendt: 29. september 2008 14:51
Til: scipy-dev at scipy.org
Emne: [SciPy-dev] Fwd: tests and partial patch forscipy.stats.distributions

removed patch because of file size, mail bounced

---------- Forwarded message ----------
From: josef.pktd at gmail.com
Date: Mon, 29 Sep 2008 00:00:25 -0400
Subject: tests and partial patch for scipy.stats.distributions
To: scipy-dev at scipy.org

To help in bug hunting and patching of the discrete distribution in
scipy.stats, I wrote several test scripts that could be incorporated
in scipy tests. The attachment contains the corrected
scipy.stats.distributions.py file and 2 test files.

The tests check all discrete distributions at predefined (not randomly
chosen) values. Some distributions have some wrong results for some
other range of parameters.

To try out the patch, it is possible just to backup the current
scipy.stats.distributions.py
and copy the patched version in its place.

Right now, I still left many comments (and some commented out failed
attempts for fixing some persistent bugs). In direct comparison with
the current trunk file this might help in understanding what the
underlying bug is.

Since there are a large number of bug fixes in this patch, if any of
them get incorporated in scipy, then they can be selectively copied.
Many bugs only show up once other bugs are fixed.

In terms of importance and sequence the bug fix categories are roughly:

* nin correction for vectorize with *args: without this, some methods
raise exceptions, e.g. _ppf (I think after many shaky attempts, I
found the correct solution)
* _ppf did not handle infinite support properly and requires a .tolist
in call to ``place`` (no idea why, but this works)
* _drv2_moment(self, n, *args) was completely broken, it didn't even
have a return and is the base for generic moment calculations

individual fixes:
* recipinvgauss_gen: rvs added, didn't work before
* def moment: for 2nd moment returned the mean (both discrete and
continuous distribution
* randint: -1 missing in _ppf, definitions differs from those in description pdf
* dlaplace: _ppf gives wrong results in tests, replace with generic calculation

(Note: the first vectorize nin correction in continuous distributions
is still at the wrong spot. I was only looking at discrete
distributions in the last few days, since I figured out how vectorize
nin works)

If you want a cleaner patch or more information, I can provide it next
week. Because, I never used numpy/scipy heavily, it took me a long
time to figure out what is going on, but it should be more obvious to
an expert. But overall my impression is, that using features in
numpy.stats.distribution, that are not commonly used, is pretty risky,
and until recently didn't have any stringent tests and does not have
any reasonably large test coverage. For serious work, I would want to
make sure that the results are actually correct.

There are still several things that are not covered, e.g. handling of
loc and scale, other methods, correctness over full range of
parameters. I haven't checked the continuous distributions since the
initial fuzz tests, but the test coverage for all methods looks pretty
low.

Josef

Test f Basic Properties
----------------------

The first test just tests for basic properties and their consistency,
e.g. cdf, pmf, ppf, stats and moments and the internal methods e.g
_ppf. The tests check quite a bit for internal consistency (private
methods) since I used the tests for debugging and trying out fixes.


>python C:\Programs\Python25\Scripts\nosetests-script.py  -s test_discrete_basic.py
...

with current trunk, I get for this test:

Ran 112 tests in 0.140s
FAILED (errors=41, failures=15)


after the bugfixes, I get:

Ran 112 tests in 8.672s
FAILED (failures=5)

some failures in binom, dlaplace, zipf are still remaining

Chisquare Test
--------------

the second test is a chisquare test for the random variables to be
close to the theoretical distribution as defined by .cdf

with current trunk I get for this test:

>python C:\Programs\Python25\Scripts\nosetests-script.py  -s test_ndiscrete.py

Ran 12 tests in 0.453s
FAILED (errors=5)

after patching scipy.stats.distribution the only test failure left is
in logser, where I think the numpy.random random numbers are wrong

>python C:\Programs\Python25\Scripts\nosetests-script.py  -s test_ndiscrete.py

bernoulli. binom. boltzmann. dlaplace. geom. hypergeom. logserF nbinom. planck.
poisson. randint. zipf.
======================================================================
FAIL: test_ndiscrete.test_discrete_rvscdf('logser', (0.59999999999999998,))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python25\lib\site-packages\nose-0.10.3-py2.5.egg\nose\case.p
y", line 182, in runTest
    self.test(*self.arg)
  File "C:\Josef\work-oth\sort\pypi\test_ndiscrete.py", line 76, in checkchisqua
re_discrete
    assert (pval > alpha), 'chisquare - test for %s at arg = %s' % (distname,str
(arg))
AssertionError: chisquare - test for logser at arg = (0.59999999999999998,)

----------------------------------------------------------------------
Ran 12 tests in 19.781s

FAILED (failures=1)


scipy.stats tests: no failures
-----------------------------------------

running the current trunk test for scipy.stats on the pre- and post
patch distributions.py gives no errors:

>>> from scipy import stats
>>> stats.test()
Running unit tests for scipy.stats
NumPy version 1.2.0rc2
NumPy is installed in C:\Programs\Python25\lib\site-packages\numpy
SciPy version 0.7.0.dev
SciPy is installed in C:\Josef\_progs\virtualpy25\envscipy\lib\site-packages\sci
py
Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Int
el)]
nose version 0.10.3
................................................................................
C:\Josef\_progs\virtualpy25\envscipy\lib\site-packages\scipy\stats\morestats.py:
618: UserWarning: Ties preclude use of exact statistic.
  warnings.warn("Ties preclude use of exact statistic.")
............................................................C:\Programs\Python25
\lib\site-packages\numpy\lib\function_base.py:343: Warning:
            The semantics of histogram has been modified in
            the current release to fix long-standing issues with
            outliers handling. The main changes concern
            1. the definition of the bin edges,
               now including the rightmost edge, and
            2. the handling of upper outliers, now ignored rather
               than tallied in the rightmost bin.
            The previous behaviour is still accessible using
            `new=False`, but is scheduled to be deprecated in the
            next release (1.3).

            *This warning will not printed in the 1.3 release.*

            Use `new=True` to bypass this warning.

            Please read the docstring for more information.

  """, Warning)
................................................................................
.............
----------------------------------------------------------------------
Ran 233 tests in 2.266s

OK
<nose.result.TextTestResult run=233 errors=0 failures=0>


From guyer at nist.gov  Wed Oct  1 09:54:12 2008
From: guyer at nist.gov (Jonathan Guyer)
Date: Wed, 1 Oct 2008 09:54:12 -0400
Subject: [SciPy-dev] Proposal: scipy.spatial
In-Reply-To: <48E3461D.6090808@ar.media.kyoto-u.ac.jp>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>	<d05265cb0809292140n27592a76gea1786c90756365b@mail.gmail.com>	<ce557a360809301745x786457d4v9956a2803620bd8d@mail.gmail.com>	<11F5C399-FE46-4321-AC9E-8B0699D06A84@nist.gov>
	<ce557a360810010254t3c3e2572lb8987d68e3eb9a19@mail.gmail.com>
	<48E3461D.6090808@ar.media.kyoto-u.ac.jp>
Message-ID: <68B2C042-C4FD-426F-8E1D-5E207465B377@nist.gov>


On Oct 1, 2008, at 5:42 AM, David Cournapeau wrote:

> As Robert said, the problem of svn with merging is unlikely to  
> happen in
> your case (Problems arise mostly when you use multiple branches and
> merge between them; as long as you have one branch which keeps  
> following
> the trunk,

We keep things simple. We release from the trunk and strive to do all  
development on branches from the trunk. I don't believe we've ever  
branched a branch and I don't think we'd ever want to. The keys to  
keeping things manageable for us are (1) disciplined commenting and  
(2) merging *from* the trunk to the branches on a pathologically  
regular basis. If you don't do the latter, then when you do try to  
merge your branch to the trunk, you can have an insurmountable amount  
of work facing you.

> svn poor merging capabilities can be worked around with
> svnmerge).

Thanks for the pointer to svnmerge. Although I'd read about it early  
on, I'd forgotten about it. I definitely see its value in keeping the  
history intact.



From david at ar.media.kyoto-u.ac.jp  Wed Oct  1 10:15:46 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 01 Oct 2008 23:15:46 +0900
Subject: [SciPy-dev] Proposal: scipy.spatial
In-Reply-To: <68B2C042-C4FD-426F-8E1D-5E207465B377@nist.gov>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>	<d05265cb0809292140n27592a76gea1786c90756365b@mail.gmail.com>	<ce557a360809301745x786457d4v9956a2803620bd8d@mail.gmail.com>	<11F5C399-FE46-4321-AC9E-8B0699D06A84@nist.gov>	<ce557a360810010254t3c3e2572lb8987d68e3eb9a19@mail.gmail.com>	<48E3461D.6090808@ar.media.kyoto-u.ac.jp>
	<68B2C042-C4FD-426F-8E1D-5E207465B377@nist.gov>
Message-ID: <48E38612.1010002@ar.media.kyoto-u.ac.jp>

Jonathan Guyer wrote:
>
> We keep things simple. We release from the trunk and strive to do all  
> development on branches from the trunk. I don't believe we've ever  
> branched a branch and I don't think we'd ever want to.

My remark was general: I am one of the most (too, I guess) vocal about
switching from svn to something which handles merge correctly on the ML,
and wanted to clarify why it is not really a problem in Anne's case, and
how people use svnmerge on numpy as documented (since it is quite easy
to screw up the first time you do it without any documentation).

>  The keys to  
> keeping things manageable for us are (1) disciplined commenting and  
> (2) merging *from* the trunk to the branches on a pathologically  
> regular basis. If you don't do the latter, then when you do try to  
> merge your branch to the trunk, you can have an insurmountable amount  
> of work facing you.

Whatever works best for you, of course.

cheers,

David


From thouis at broad.mit.edu  Wed Oct  1 11:52:21 2008
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Wed, 1 Oct 2008 11:52:21 -0400
Subject: [SciPy-dev] [patch] read/write v5 .mat files with structs,
	cell arrays, objects, or function handles
Message-ID: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>

I just submitted a patch (http://scipy.org/scipy/scipy/ticket/743)
extending the functionality of io.matlab for v5 files, adding writers
for structs, cell arrays, & objects, and readers+writers for function
handles and 64-bit ints.

The patch modifies the numpy types of strings, structs, and objects
loaded from matlab:
matlab strings are loaded as numpy arrays of unicode strings, rather
than numpy arrays of objects.
structs are loaded as numpy arrays with dtype [(field_name, object), ...]
matlab objects are wrapped in an mio5.MatlabObject  instance, which
contains a 1x1 struct & the object classname.
(function handles are also wrapped in a special object in mio5.py)

These changes were to make the scipy<->matlab mapping more explicit.
The changes to the string type probably don't break any existing code.
 The changes to the struct & matlab objects probably do.  It's
possible to make both of these backward compatible, if that's a
priority.  We're looking for feedback on that (or any other) issue.

Ray Jones


From prabhu at aero.iitb.ac.in  Wed Oct  1 12:53:01 2008
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Wed, 01 Oct 2008 22:23:01 +0530
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>	<48E1A6A2.4050406@noaa.gov>	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>	<48E31404.7000009@aero.iitb.ac.in>
	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>
Message-ID: <48E3AAED.6080601@aero.iitb.ac.in>

Anne Archibald wrote:
>> This is nice.  How about a keyword argument "sort=True" that lets people
>> choose if they want the results sorted by distance or not?  I have seen
>> cases where unsorted results are useful.
> 
> That is a possibility. My current code produces results that are in a
> heap, so sorting can in principle be an O(n) activity. Since the
> number of results returned is rarely going to be large, I'm not sure

I thought this was a discussion on a good API for such an algorithm 
(sorry if I misunderstood!).  Given that context, if this is a general 
purpose algorithm I'd rather not assume that the number of results is 
typically going to be small.

> the gain in efficiency is worth the loss in determinism. And

If there is no need for sorting and there are a large number of nearest 
neighbor queries (in my case there are O(N) nearest neighbor queries), 
this quickly adds up to an unnecessary O(N) inefficiency that I'd like 
to avoid.

> additional API complexity does have a cost - in an ideal world it
> should at the least double the number of unit tests. I think perhaps
> this should wait on a C implementation.

I don't see why the addition of a sort keyword will double your unit 
tests.  Another option would be to return unsorted results and have a 
separate sort method if it makes testing easier.

Thanks for listening.

cheers,
prabhu


From wnbell at gmail.com  Wed Oct  1 13:56:22 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Wed, 1 Oct 2008 13:56:22 -0400
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <48E3AAED.6080601@aero.iitb.ac.in>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<48E31404.7000009@aero.iitb.ac.in>
	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>
	<48E3AAED.6080601@aero.iitb.ac.in>
Message-ID: <d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>

On Wed, Oct 1, 2008 at 12:53 PM, Prabhu Ramachandran
<prabhu at aero.iitb.ac.in> wrote:
>
> I thought this was a discussion on a good API for such an algorithm
> (sorry if I misunderstood!).  Given that context, if this is a general
> purpose algorithm I'd rather not assume that the number of results is
> typically going to be small.

However, you *can* assume log(k) of a k-nearest neighbor query is
going to be small.

>
> If there is no need for sorting and there are a large number of nearest
> neighbor queries (in my case there are O(N) nearest neighbor queries),
> this quickly adds up to an unnecessary O(N) inefficiency that I'd like
> to avoid.
>

Although you might not sort a k-nearest neighbor query, but as Anne
has said, you end up needing a sorted structure (like a priority
queue) to perform the query itself anyway.

The only case where not sorting makes sense is an "in sphere" query
where there's no need to prioritize the results.  In this case, I'd
still argue that the overhead of sorting isn't likely to matter.

>
> I don't see why the addition of a sort keyword will double your unit
> tests.  Another option would be to return unsorted results and have a
> separate sort method if it makes testing easier.
>

It would be more profitable to offload the queries to native code.
That is a much larger speedup than the O(log k) sorting factor.
Interpreter overhead is going to dominate any pure-python hierarchy
traversal.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From zachary.pincus at yale.edu  Wed Oct  1 14:52:32 2008
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 1 Oct 2008 14:52:32 -0400
Subject: [SciPy-dev] [patch] read/write v5 .mat files with structs,
	cell arrays, objects, or function handles
In-Reply-To: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
Message-ID: <F8826C8B-81B6-44F1-B736-3C62A4C6F663@yale.edu>

Hi Ray,

Those changes look really useful!

St?fan van der Walt <stefan at sun.ac.za> (on this email list) and  
Matthew Brett <matthew.brett at gmail.com> (not on the list?) seem to be  
coordinating the matlab IO (at least, they were the ones who  
shepherded my recent patch to the same).

I'm not 100% sure what the backwards-compatibility guarantees for  
scipy are right now, but the matlab 5 IO is pretty new, so hopefully  
useful changes like these won't be a problem. (Any one can say for  
sure?)

One comment -- perhaps for objects, would a 0-d "array scalar" (i.e.  
shape=()) struct be better than a 1x1 struct? I'm not really sure  
either way...

Anyhow, thanks for the patch. This should be very helpful.

Best,
Zach


On Oct 1, 2008, at 11:52 AM, Thouis (Ray) Jones wrote:

> I just submitted a patch (http://scipy.org/scipy/scipy/ticket/743)
> extending the functionality of io.matlab for v5 files, adding writers
> for structs, cell arrays, & objects, and readers+writers for function
> handles and 64-bit ints.
>
> The patch modifies the numpy types of strings, structs, and objects
> loaded from matlab:
> matlab strings are loaded as numpy arrays of unicode strings, rather
> than numpy arrays of objects.
> structs are loaded as numpy arrays with dtype [(field_name,  
> object), ...]
> matlab objects are wrapped in an mio5.MatlabObject  instance, which
> contains a 1x1 struct & the object classname.
> (function handles are also wrapped in a special object in mio5.py)
>
> These changes were to make the scipy<->matlab mapping more explicit.
> The changes to the string type probably don't break any existing code.
> The changes to the struct & matlab objects probably do.  It's
> possible to make both of these backward compatible, if that's a
> priority.  We're looking for feedback on that (or any other) issue.
>
> Ray Jones
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev



From millman at berkeley.edu  Wed Oct  1 15:27:16 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 1 Oct 2008 12:27:16 -0700
Subject: [SciPy-dev] [patch] read/write v5 .mat files with structs,
	cell arrays, objects, or function handles
In-Reply-To: <F8826C8B-81B6-44F1-B736-3C62A4C6F663@yale.edu>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<F8826C8B-81B6-44F1-B736-3C62A4C6F663@yale.edu>
Message-ID: <c7009a550810011227v705336cej5a5d4f6054649b02@mail.gmail.com>

On Wed, Oct 1, 2008 at 11:52 AM, Zachary Pincus <zachary.pincus at yale.edu> wrote:
> I'm not 100% sure what the backwards-compatibility guarantees for
> scipy are right now, but the matlab 5 IO is pretty new, so hopefully
> useful changes like these won't be a problem. (Any one can say for
> sure?)

In general, backward compatibility for SciPy isn't as big a constraint
as it is for NumPy.  NumPy is an extremely mature codebase that is
widely used by many projects, so we have to be *extremely* careful
about ABI/API breaks.

SciPy on the other hand is still considered 'beta' software, so we
should be focusing on getting it to the point that we consider it more
'production' code.  Once we get to that point we will need to start
being much more careful about ABI/API breaks.  Of course, no one is
going to want to use SciPy if we arbitrarily break their code with
every release of SciPy.  So we just need to strike a reasonable
balance of fixing and improving SciPy, while not chasing off users.

In this particular case, the changes to the mat file io looks very
reasonable.  I will let St?fan van der Walt  or Matthew Brett make the
final call, but I would like to see this patch (or a revised version
of it) get in.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From stefan at sun.ac.za  Wed Oct  1 16:20:31 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 1 Oct 2008 22:20:31 +0200
Subject: [SciPy-dev] [patch] read/write v5 .mat files with structs,
	cell arrays, objects, or function handles
In-Reply-To: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
Message-ID: <9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>

Hi Ray

2008/10/1 Thouis (Ray) Jones <thouis at broad.mit.edu>:
> These changes were to make the scipy<->matlab mapping more explicit.
> The changes to the string type probably don't break any existing code.
>  The changes to the struct & matlab objects probably do.  It's
> possible to make both of these backward compatible, if that's a
> priority.  We're looking for feedback on that (or any other) issue.

Thank you for improving the SciPy IO capabilities!  Before we go
ahead, would you kindly do the following:

- Make sure that the existing test suite passes with your patch
applied (I can't do that from here right now)
- Write tests to verify the working of the new functionality

You'll see that we have a very thorough round-tripping test set for
the IO code already, and it would be good to keep this up to date with
your changes.

Requiring tests for contributions is certainly not intended to slow
the adoption of patches; we simply need to see what the code does, and
must ensure that it does so correctly.

Ideally, we'd also like documentation for all the classes in the IO
module, but it would hardly be fair to ask you to write them :)

Thanks again,
St?fan


From thouis at broad.mit.edu  Wed Oct  1 20:11:45 2008
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Wed, 1 Oct 2008 20:11:45 -0400
Subject: [SciPy-dev] [patch] read/write v5 .mat files with structs,
	cell arrays, objects, or function handles
In-Reply-To: <9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
Message-ID: <6c17e6f50810011711n34da69b8w58ecd42938cec547@mail.gmail.com>

On Wed, Oct 1, 2008 at 4:20 PM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Thank you for improving the SciPy IO capabilities!  Before we go
> ahead, would you kindly do the following:
>
> - Make sure that the existing test suite passes with your patch
> applied (I can't do that from here right now)
> - Write tests to verify the working of the new functionality

Is it acceptable for us to modify the tests to handle the new form of
some matlab objects.  Structures, objects, and function handles are
definitely different.  String arrays have a different dtype, as well.

Also, I noticed there is a test for reading matlab objects.  I think
this should be removed, as the structure that matlab uses to save
objects is not well documented (besides being a matlab 1x1 struct
array with an extra element for the classname).

We'll try to get this done soon.

Best,
Ray Jones


From cournape at gmail.com  Wed Oct  1 20:37:58 2008
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 2 Oct 2008 09:37:58 +0900
Subject: [SciPy-dev] [patch] read/write v5 .mat files with structs,
	cell arrays, objects, or function handles
In-Reply-To: <9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
Message-ID: <5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>

On Thu, Oct 2, 2008 at 5:20 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Ray
>
> 2008/10/1 Thouis (Ray) Jones <thouis at broad.mit.edu>:
>> These changes were to make the scipy<->matlab mapping more explicit.
>> The changes to the string type probably don't break any existing code.
>>  The changes to the struct & matlab objects probably do.  It's
>> possible to make both of these backward compatible, if that's a
>> priority.  We're looking for feedback on that (or any other) issue.
>
> Thank you for improving the SciPy IO capabilities!  Before we go
> ahead, would you kindly do the following:

Would it be possible for this to wait after 0.7 ? More exactly, I
would like to keep the current behavior of 0.7, with a deprecation
warning, and maybe the new code available through some mechanism to
adapt (and would replace in 0.8). Even if scipy is not considered
stable, I think we should allow at least one version for deprecation.

cheers,

David

From stefan at sun.ac.za  Thu Oct  2 03:41:27 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Thu, 2 Oct 2008 09:41:27 +0200
Subject: [SciPy-dev] [patch] read/write v5 .mat files with structs,
	cell arrays, objects, or function handles
In-Reply-To: <5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
	<5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
Message-ID: <9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>

2008/10/2 David Cournapeau <cournape at gmail.com>:
> Would it be possible for this to wait after 0.7 ? More exactly, I
> would like to keep the current behavior of 0.7, with a deprecation
> warning, and maybe the new code available through some mechanism to
> adapt (and would replace in 0.8). Even if scipy is not considered
> stable, I think we should allow at least one version for deprecation.

We should certainly look at applying the non-API-changing parts,
though.  I'm not sure what the best way is to represent these
structures on the Python side.

Thouis, you've thought about this a lot: could you tell us the pros
and cons of switching to the new representation?

I don't mind the tests changing, as long as the round-trip completes
successfully and all the data is still read.

Cheers
St?fan


From josef.pktd at gmail.com  Thu Oct  2 09:29:48 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 2 Oct 2008 09:29:48 -0400
Subject: [SciPy-dev] Fwd: tests and partial patch
	forscipy.stats.distributions
In-Reply-To: <1ED225FF18AA8B48AC192F7E1D032C6E0C5C1D@hbu-posten.ffi.no>
References: <1cd32cbb0809282100x75eccfb9la109e92032f798ac@mail.gmail.com>
	<1cd32cbb0809290551g7c00a5ddh44369352ece506d2@mail.gmail.com>
	<1ED225FF18AA8B48AC192F7E1D032C6E0C5C1D@hbu-posten.ffi.no>
Message-ID: <1cd32cbb0810020629w56cf8e89t58954256e87bb29e@mail.gmail.com>

I finally filed a ticket on this on scipy trac:
http://scipy.org/scipy/scipy/ticket/745

On Wed, Oct 1, 2008 at 7:14 AM,  <Per.Brodtkorb at ffi.no> wrote:
> I have looked into the bugs in the patch of the discrete distributions you sent. The error in _ppf method of dlaplace can be corrected by replacing it with:
>
> class dlaplace_gen(rv_discrete):
> .....
>    def _ppf(self, q, a):
>        const = 1.0/(1+exp(-a))
>        cons2 = 1+exp(a)
>        ind = q < const
>        return ceil(where(ind, log(q*cons2)/a-1, -log((1-q)*cons2)/a))
> ......
>

Thanks, I didn't see the misplaced bracket, I changed my corrections
to this. Do you have any idea or reference about the theoretical
second moment of dlaplace?


> The tests written for the generic moment, fails for those distributions that set the limits (self.a and/or self.b) for the support of the distribution in the self._argcheck method. I think the generic_moment method is meant to be private to the self._munp method, since it does only limited input checks.
> Perhaps better to rename generic_moment to _generic_moment

I aggree, I didn't check this systematically, but I guess
'vec_generic_moment', 'vecentropy' are also private methods.
>
> Per A.

I did not realize that some of my tests on private methods might fail
for some distributions by design.
To make it a bit clearer, I separated the tests into tests for public
methods, and tests for private methods. I still like the tests of
private methods, even if some are known failures, since they help to
narrow down what the source of an exception or test failure in the
public methods is.

Josef


From ljosa at broad.mit.edu  Thu Oct  2 10:14:14 2008
From: ljosa at broad.mit.edu (Vebjorn Ljosa)
Date: Thu, 02 Oct 2008 10:14:14 -0400
Subject: [SciPy-dev] SciPy-dev] [patch] read/write v5 .mat files with
	structs, cell arrays, objects, or function handles
In-Reply-To: <9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
	<5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
	<9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>
Message-ID: <nq5prmjf5jd.fsf_-_@broad.mit.edu>

St?fan van der Walt <stefan at sun.ac.za> writes:
>
> We should certainly look at applying the non-API-changing parts,
> though.  I'm not sure what the best way is to represent these
> structures on the Python side.
>
> Thouis, you've thought about this a lot: could you tell us the pros
> and cons of switching to the new representation?

The reason Ray and I changed some of the representations is that we
wanted the mapping from Matlab to Python to be symmetric: anything read
from a MAT-file should be represented in a way that allows the writer
code to write it back in its original form.  This requires that the
original Matlab type be deducible from the Python representation.

 * Struct arrays: Matlab struct arrays were previously represented as
   numpy arrays of dtype=object filled with instances of mat_struct.
   The problem is that Matlab cell arrays were also represented as numpy
   arrays of dtype=objects.  The writer code could in most cases have
   identified structs by looking at the contents (instances of
   mat_struct), but there was no way to distinguish a 0x0 cell array
   from a 0x0 struct array.  We therefore opted to represent struct
   arrays as numpy record arrays.

   In order not to break existing code, we could introduce a keyword
   argument to loadmat that selects the old or new representation,
   similar to numpy.histogram's "new" argument.  In 0.7, leaving the
   argument out would default to False (old behavior), but give a
   deprecation warning.  Later versions can first change the default to
   True and then remove the old behavior entirely.  The best name I can
   think of for this keyword argument is "struct_as_record".

 * Char arrays/strings: Same story.  At the lowest level, the code
   represented char arrays as numpy arrays of dtype='U1', which is
   fine.  A very useful "processor function" (in miobase) turns them
   into arrays of strings, however.  This processor function created
   an array of dtype=object.  We changed this to 'U...' so the array
   could be distinguished from a cell array.  I think this is unlikely
   to break any code, do you agree?

 * Objects: This change in representation was purely for our
   convenience, and we should be able to fix our patch to keep the old
   representation.

Vebjorn


From matthew.brett at gmail.com  Thu Oct  2 11:34:05 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 2 Oct 2008 08:34:05 -0700
Subject: [SciPy-dev] SciPy-dev] [patch] read/write v5 .mat files with
	structs, cell arrays, objects, or function handles
In-Reply-To: <nq5prmjf5jd.fsf_-_@broad.mit.edu>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
	<5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
	<9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>
	<nq5prmjf5jd.fsf_-_@broad.mit.edu>
Message-ID: <1e2af89e0810020834t64e104b2oaf5abad1b07d7333@mail.gmail.com>

Hi,

> The reason Ray and I changed some of the representations is that we
> wanted the mapping from Matlab to Python to be symmetric: anything read
> from a MAT-file should be represented in a way that allows the writer
> code to write it back in its original form.  This requires that the
> original Matlab type be deducible from the Python representation.

Thank you very much for doing this work - it was certainly a major
deficiency in the scipy matlab io that it could not do this
roundtripping for structs and c.

For the API change, the API for the io has already changed in the
trunk, from an earlier change (well-motivated) by Nathan Bell.   I
would love to see a version of this patch in 0.7 if at all possible.

Best,

Matthew


From rmay31 at gmail.com  Thu Oct  2 13:08:10 2008
From: rmay31 at gmail.com (Ryan May)
Date: Thu, 02 Oct 2008 12:08:10 -0500
Subject: [SciPy-dev] Add density of water to constants?
Message-ID: <48E4FFFA.2030007@gmail.com>

Hi,

I was just cleaning up my own hand-rolled constants file and noticed
that the only one not already in scipy's own constants file was the
density of liquid water.  I'm not sure how much we want to accumulate
more of these physical constants in there, but here's a patch nonetheless.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: constants_water_density.diff
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081002/5677dc1a/attachment.ksh>

From pav at iki.fi  Thu Oct  2 14:17:43 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 2 Oct 2008 18:17:43 +0000 (UTC)
Subject: [SciPy-dev] Add density of water to constants?
References: <48E4FFFA.2030007@gmail.com>
Message-ID: <gc3387$f6s$1@ger.gmane.org>

Thu, 02 Oct 2008 12:08:10 -0500, Ryan May wrote:
> I was just cleaning up my own hand-rolled constants file and noticed
> that the only one not already in scipy's own constants file was the
> density of liquid water.  I'm not sure how much we want to accumulate
> more of these physical constants in there, but here's a patch
> nonetheless.

I'm not sure if the density of liquid water is actually a good constant.

First, it varies between 999.972 and 992.2 kg/m^3 already at temperatures 
between 0 ?C and 40 ?C.

Second, the kilogram is not defined in terms of the density of water, but 
in terms of a metal prototype weight that was manufactured based on a 
single measurement of the density of water. The maximum density of water 
turns out to be, funnily enough, not 1000 kg/m^3 but slightly less. [1,2]

.. [1] "Water". Wikipedia, http://en.wikipedia.org/wiki/H2O
.. [2] "Kilogram. Wikipedia, http://en.wikipedia.org/wiki/Kilogram

-- 
Pauli Virtanen



From rmay31 at gmail.com  Thu Oct  2 14:30:46 2008
From: rmay31 at gmail.com (Ryan May)
Date: Thu, 02 Oct 2008 13:30:46 -0500
Subject: [SciPy-dev] Add density of water to constants?
In-Reply-To: <gc3387$f6s$1@ger.gmane.org>
References: <48E4FFFA.2030007@gmail.com> <gc3387$f6s$1@ger.gmane.org>
Message-ID: <48E51356.5000806@gmail.com>

Pauli Virtanen wrote:
> Thu, 02 Oct 2008 12:08:10 -0500, Ryan May wrote:
>> I was just cleaning up my own hand-rolled constants file and noticed
>> that the only one not already in scipy's own constants file was the
>> density of liquid water.  I'm not sure how much we want to accumulate
>> more of these physical constants in there, but here's a patch
>> nonetheless.
> 
> I'm not sure if the density of liquid water is actually a good constant.
> 
> First, it varies between 999.972 and 992.2 kg/m^3 already at temperatures 
> between 0 ?C and 40 ?C.
> 
> Second, the kilogram is not defined in terms of the density of water, but 
> in terms of a metal prototype weight that was manufactured based on a 
> single measurement of the density of water. The maximum density of water 
> turns out to be, funnily enough, not 1000 kg/m^3 but slightly less. [1,2]
> 
> .. [1] "Water". Wikipedia, http://en.wikipedia.org/wiki/H2O
> .. [2] "Kilogram. Wikipedia, http://en.wikipedia.org/wiki/Kilogram

Wow. Point taken.  I knew about the kilogram "definition", but never
thought about it's impact on defining density.  Granted, in meteorology,
999.972 and 992.2 are more or less the same. :)

Maybe I'll just have maintain a few meteorologically relevant constants
in a file myself then.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From cournape at gmail.com  Fri Oct  3 01:27:19 2008
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 3 Oct 2008 14:27:19 +0900
Subject: [SciPy-dev] SciPy-dev] [patch] read/write v5 .mat files with
	structs, cell arrays, objects, or function handles
In-Reply-To: <nq5prmjf5jd.fsf_-_@broad.mit.edu>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
	<5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
	<9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>
	<nq5prmjf5jd.fsf_-_@broad.mit.edu>
Message-ID: <5b8d13220810022227k2d70bea3y2d91f5783ce78b8@mail.gmail.com>

On Thu, Oct 2, 2008 at 11:14 PM, Vebjorn Ljosa <ljosa at broad.mit.edu> wrote:
>
> The reason Ray and I changed some of the representations is that we
> wanted the mapping from Matlab to Python to be symmetric: anything read
> from a MAT-file should be represented in a way that allows the writer
> code to write it back in its original form.  This requires that the
> original Matlab type be deducible from the Python representation.

FWIW, I agree completely that the current io for matlab would be
greatly improved by those changes; there is no arguing that those
changes should come in scipy. The problem is that some people have
code which depend on the current shortcoming (well, *I* have), and it
is better to have one full version at least which says that it will
change (with deprecation warning). The code could be in something like
io.newmaltlab, which would be changed to io.matlab in 0.8: this way,
if people want to use it now, they can.

cheers,

David


From cournape at gmail.com  Fri Oct  3 01:29:13 2008
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 3 Oct 2008 14:29:13 +0900
Subject: [SciPy-dev] SciPy-dev] [patch] read/write v5 .mat files with
	structs, cell arrays, objects, or function handles
In-Reply-To: <1e2af89e0810020834t64e104b2oaf5abad1b07d7333@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
	<5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
	<9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>
	<nq5prmjf5jd.fsf_-_@broad.mit.edu>
	<1e2af89e0810020834t64e104b2oaf5abad1b07d7333@mail.gmail.com>
Message-ID: <5b8d13220810022229s5523d3bex93355ed687c8d6ea@mail.gmail.com>

On Fri, Oct 3, 2008 at 12:34 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> For the API change, the API for the io has already changed in the
> trunk, from an earlier change (well-motivated) by Nathan Bell.   I
> would love to see a version of this patch in 0.7 if at all possible.

Yes, I know, it broke my packages :) I don't think it is good
practice. We allow for scipy API changes, but it would still be better
it we were saying that something is about to change. It really does
not cost much, I think.

cheers,

David


From matthew.brett at gmail.com  Fri Oct  3 01:54:45 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 2 Oct 2008 22:54:45 -0700
Subject: [SciPy-dev] SciPy-dev] [patch] read/write v5 .mat files with
	structs, cell arrays, objects, or function handles
In-Reply-To: <5b8d13220810022229s5523d3bex93355ed687c8d6ea@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
	<5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
	<9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>
	<nq5prmjf5jd.fsf_-_@broad.mit.edu>
	<1e2af89e0810020834t64e104b2oaf5abad1b07d7333@mail.gmail.com>
	<5b8d13220810022229s5523d3bex93355ed687c8d6ea@mail.gmail.com>
Message-ID: <1e2af89e0810022254k3ff22324k39d656d6d0b0aa38@mail.gmail.com>

Hi,

>> For the API change, the API for the io has already changed in the
>> trunk, from an earlier change (well-motivated) by Nathan Bell.   I
>> would love to see a version of this patch in 0.7 if at all possible.
>
> Yes, I know, it broke my packages :) I don't think it is good
> practice. We allow for scipy API changes, but it would still be better
> it we were saying that something is about to change. It really does
> not cost much, I think.

All your points well taken, and I voted rather weakly against the
earlier change, but, given the API is already changed compared to the
previous release, and that Scipy releases are infrequent, my weak vote
would still be for going the whole way here, but I'm happy to be
outvoted.

See you,

Matthew


From thouis at broad.mit.edu  Fri Oct  3 10:49:51 2008
From: thouis at broad.mit.edu (Thouis (Ray) Jones)
Date: Fri, 3 Oct 2008 10:49:51 -0400
Subject: [SciPy-dev] SciPy-dev] [patch] read/write v5 .mat files with
	structs, cell arrays, objects, or function handles
In-Reply-To: <1e2af89e0810022254k3ff22324k39d656d6d0b0aa38@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
	<5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
	<9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>
	<nq5prmjf5jd.fsf_-_@broad.mit.edu>
	<1e2af89e0810020834t64e104b2oaf5abad1b07d7333@mail.gmail.com>
	<5b8d13220810022229s5523d3bex93355ed687c8d6ea@mail.gmail.com>
	<1e2af89e0810022254k3ff22324k39d656d6d0b0aa38@mail.gmail.com>
Message-ID: <6c17e6f50810030749p1966266bs23701e77f8c05b3a@mail.gmail.com>

I've updated the patch to provide backwards compatibility for matlab
structures, controlled by a keyword argument to loadmat
(struct_as_record), which defaults to False (old style), and gives a
deprecation warning if False.

All tests in test_mio.py pass.  I updated some of the tests for the
new object structure.

I tested both options for loading structures with some of my own data,
but it would be useful if David could check his code against it (if
that's not too difficult), as we don't really have a relevant codebase
prior to these changes.

Ray Jones
(note that I goofed and didn't supersede the old patch in the Trac,
but it' obvious which is the right one.)


From josef.pktd at gmail.com  Fri Oct  3 10:54:53 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 3 Oct 2008 10:54:53 -0400
Subject: [SciPy-dev] Next scipy release 0.7
In-Reply-To: <c7009a550809302211w5acb6216j1d0b8e03b61079c7@mail.gmail.com>
References: <web-111958955@uni-stuttgart.de>
	<c7009a550809290955i125cb6c4tb7350edf54e036be@mail.gmail.com>
	<1d36917a0809302137u78a9644fu9da38e3a79157a40@mail.gmail.com>
	<c7009a550809302211w5acb6216j1d0b8e03b61079c7@mail.gmail.com>
Message-ID: <1cd32cbb0810030754x149929e9sae69626dbca8c50a@mail.gmail.com>

Hi,

I think that my fixes to scipy.stats.distributions in
http://scipy.org/scipy/scipy/ticket/745
are now good enough that they can go into 0.7, after some review by
numpy experts which I am not.

As far as I can see, non of my fixes changes anything in the existing
functionality, but fixes several methods that throw exceptions in the
current trunk, and it corrects incorrect calculations in some places.

I provided basic tests. The test results now look pretty good,
compared to before.
This is still not a complete test suite for scipy.stats.distributions,
but in my opinion, a big enhancement to the current tests.

Josef


From charlesr.harris at gmail.com  Sat Oct  4 04:40:53 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 4 Oct 2008 02:40:53 -0600
Subject: [SciPy-dev] Question about Kaiser Implementation in firwin
In-Reply-To: <44B9C5B0.1050201@ieee.org>
References: <B9D0673A645E224B8B6032DD8D049A5801F4707C@cossmgmbx06.EMAIL.CORP.TLD>
	<44B9C5B0.1050201@ieee.org>
Message-ID: <e06186140810040140g25aab5d6i308aad7f569c8fd6@mail.gmail.com>

On Sat, Jul 15, 2006 at 10:50 PM, Travis Oliphant
<oliphant.travis at ieee.org>wrote:

> Buehler, Eric (AGRE) wrote:
> > Hello,
> >
> > However, the last line of the firwin function is causing me some
> > heartburn.
> > filter_design.py
> >
> > 1538    win = get_window(window,N,fftbins=1)
> > 1539    alpha = N//2
> > 1540    m = numpy.arange(0,N)
> > 1541    h = win*special.sinc(cutoff*(m-alpha))
> > 1542    return h / sum(h)
> >
> > Line 1542 of filter_design.py, "return h / sum(h)", normalizes the
> > function where it doesn't seem necessary, at least in the kaiser window
> > case.  Without the normalization, the kaiser window already returns a
> > value of 1 at the zero frequency point.  This normalization scales all
> > of the data, making the window difficult to use in the frequency domain.
> >
> > Can someone point me to the rationale for this line?  Looking at the
> > code, this seems to be a pretty recent change (within the last year/year
> > and a half).
> >
> >
> I'm not aware of if and when the change was made.   Was there a time
> when firwin did not have this normalization?
>
> The normalization is done so that the resulting filter has a 0dB gain at
> DC (which is the center of the pass-band).
>
> In other-words fft(h)[0] is approximately equal to 1.  This is usually
> what is desired as firwin returns "time-domain" filter coefficients.
> The return value of the function is not designed for use in the
> "frequency"-domain.  I'm not even sure what you mean by that in this
> context.
>
> The intended usage of the result of firwin is in a convolution:
>
> convolve(h, <mysignal>)
>

I almost always use area one. For looking at spectra and such it is best to
be able to integrate over a band in the frequency domain without the results
changing much. In particular, it is best to preserve the shape of continua.
This is also the best way to treat power spectra.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081004/b7ddfcd5/attachment.html>

From prabhu at aero.iitb.ac.in  Sat Oct  4 08:37:37 2008
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Sat, 04 Oct 2008 18:07:37 +0530
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>	<48E1A6A2.4050406@noaa.gov>	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>	<48E31404.7000009@aero.iitb.ac.in>	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>	<48E3AAED.6080601@aero.iitb.ac.in>
	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>
Message-ID: <48E76391.5050508@aero.iitb.ac.in>

Hi,

Nathan Bell wrote:
> On Wed, Oct 1, 2008 at 12:53 PM, Prabhu Ramachandran
> <prabhu at aero.iitb.ac.in> wrote:
>> I thought this was a discussion on a good API for such an algorithm
>> (sorry if I misunderstood!).  Given that context, if this is a general
>> purpose algorithm I'd rather not assume that the number of results is
>> typically going to be small.
> 
> However, you *can* assume log(k) of a k-nearest neighbor query is
> going to be small.

True.

>> If there is no need for sorting and there are a large number of nearest
>> neighbor queries (in my case there are O(N) nearest neighbor queries),
>> this quickly adds up to an unnecessary O(N) inefficiency that I'd like
>> to avoid.
> 
> Although you might not sort a k-nearest neighbor query, but as Anne
> has said, you end up needing a sorted structure (like a priority
> queue) to perform the query itself anyway.

In the context of a kd-tree, yes, but not in other contexts like the "in 
sphere" query you mention below.

> The only case where not sorting makes sense is an "in sphere" query
> where there's no need to prioritize the results.  In this case, I'd
> still argue that the overhead of sorting isn't likely to matter.

Well, in the case of the kd-tree I understand that this is not likely to 
matter.  I was under the mistaken impression that this was to be a 
common API.  I re-read the thread and understand now that this is a 
first cut for just the KDTree.

Personally, I have a need for finding the exact nearest neighbors within 
an epsilon ball (in 1, 2, and 3D) of *all* the nodes.  Typically epislon 
is known (or at least bounded) a-priori.  In this context I typically 
build a binary/quad/oct tree and limit the size of the leaves to around 
epsilon. I avoid tree traversals by iterating over the leaves in order 
to get to the points they contain -- note that the target point is 
always contained in the tree which is why I can do this.   Anyway, in 
this context sorting the results ends up in O(N klog(k)) operations if 
you have k neighbors on the average.  This is expensive.

In any case, I am not sure there is a general need for this kind of data 
structure.  I need it a whole lot.  If there is a general need I'll be 
glad to try and extract any of my code and contribute it to scipy.spatial.

cheers,
prabhu


From peridot.faceted at gmail.com  Sat Oct  4 08:57:23 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 4 Oct 2008 08:57:23 -0400
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <48E76391.5050508@aero.iitb.ac.in>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E1A6A2.4050406@noaa.gov>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<48E31404.7000009@aero.iitb.ac.in>
	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>
	<48E3AAED.6080601@aero.iitb.ac.in>
	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>
	<48E76391.5050508@aero.iitb.ac.in>
Message-ID: <ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>

2008/10/4 Prabhu Ramachandran <prabhu at aero.iitb.ac.in>:
> Hi,
>
> Nathan Bell wrote:
>> On Wed, Oct 1, 2008 at 12:53 PM, Prabhu Ramachandran
>> <prabhu at aero.iitb.ac.in> wrote:
>>> I thought this was a discussion on a good API for such an algorithm
>>> (sorry if I misunderstood!).  Given that context, if this is a general
>>> purpose algorithm I'd rather not assume that the number of results is
>>> typically going to be small.
>>
>> However, you *can* assume log(k) of a k-nearest neighbor query is
>> going to be small.
>
> True.
>
>>> If there is no need for sorting and there are a large number of nearest
>>> neighbor queries (in my case there are O(N) nearest neighbor queries),
>>> this quickly adds up to an unnecessary O(N) inefficiency that I'd like
>>> to avoid.
>>
>> Although you might not sort a k-nearest neighbor query, but as Anne
>> has said, you end up needing a sorted structure (like a priority
>> queue) to perform the query itself anyway.
>
> In the context of a kd-tree, yes, but not in other contexts like the "in
> sphere" query you mention below.
>
>> The only case where not sorting makes sense is an "in sphere" query
>> where there's no need to prioritize the results.  In this case, I'd
>> still argue that the overhead of sorting isn't likely to matter.
>
> Well, in the case of the kd-tree I understand that this is not likely to
> matter.  I was under the mistaken impression that this was to be a
> common API.  I re-read the thread and understand now that this is a
> first cut for just the KDTree.
>
> Personally, I have a need for finding the exact nearest neighbors within
> an epsilon ball (in 1, 2, and 3D) of *all* the nodes.  Typically epislon
> is known (or at least bounded) a-priori.  In this context I typically
> build a binary/quad/oct tree and limit the size of the leaves to around
> epsilon. I avoid tree traversals by iterating over the leaves in order
> to get to the points they contain -- note that the target point is
> always contained in the tree which is why I can do this.   Anyway, in
> this context sorting the results ends up in O(N klog(k)) operations if
> you have k neighbors on the average.  This is expensive.
>
> In any case, I am not sure there is a general need for this kind of data
> structure.  I need it a whole lot.  If there is a general need I'll be
> glad to try and extract any of my code and contribute it to scipy.spatial.

There does seem to be a general need for a two-tree query that
extracts (or just counts) all items of one tree within a ball centered
at each item of another tree. You're right, too, that "in a ball"
queries often don't naturally produce sorted results, or even known
distances. But I think this has to be a different query. In fact, even
for single-point queries it may make sense to have a separate query
function for in-a-ball queries.

Anne


From prabhu at aero.iitb.ac.in  Sat Oct  4 12:19:44 2008
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Sat, 04 Oct 2008 21:49:44 +0530
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>	<48E1A6A2.4050406@noaa.gov>	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>	<48E31404.7000009@aero.iitb.ac.in>	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>	<48E3AAED.6080601@aero.iitb.ac.in>	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>	<48E76391.5050508@aero.iitb.ac.in>
	<ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>
Message-ID: <48E797A0.3070604@aero.iitb.ac.in>

Anne Archibald wrote:
> 2008/10/4 Prabhu Ramachandran <prabhu at aero.iitb.ac.in>:
>> In any case, I am not sure there is a general need for this kind of data
>> structure.  I need it a whole lot.  If there is a general need I'll be
>> glad to try and extract any of my code and contribute it to scipy.spatial.
> 
> There does seem to be a general need for a two-tree query that
> extracts (or just counts) all items of one tree within a ball centered
> at each item of another tree. You're right, too, that "in a ball"
> queries often don't naturally produce sorted results, or even known
> distances. But I think this has to be a different query. In fact, even
> for single-point queries it may make sense to have a separate query
> function for in-a-ball queries.

True and another reason for a different approach would be performance. 
According to Lee and Wong (1970, Acta Informatica) the worst case 
performance for partial region searches on k-d trees is O(k N^{1-1/k}). 
  So if the dimensionality is small and the region size is know we could 
do a lot better with other specialized algorithms for these cases.

cheers,
prabhu


From peridot.faceted at gmail.com  Sat Oct  4 18:07:15 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 4 Oct 2008 18:07:15 -0400
Subject: [SciPy-dev] Using nose
Message-ID: <ce557a360810041507o6aa00ff4oc2690056f10f3df6@mail.gmail.com>

Hi,

I have a particular pattern of tests I'd like to implement, and I
don't know a good way to make nose carry it out. The pattern is this:
I have a class with a setUp function that creates various variables
(an array of test data, etc.). I also have a handful of test_whatever
methods. This works fine. But now I'd like to apply exactly the same
test methods to a different set of data. My first attempt was to
simply create a derived class which overrode the setUp method, but
unfortunately neither unittest nor nose will run tests defined in the
parent class.

Any suggestions?

Thanks,
Anne


From alan.mcintyre at gmail.com  Sat Oct  4 19:22:42 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sat, 4 Oct 2008 16:22:42 -0700
Subject: [SciPy-dev] Using nose
In-Reply-To: <ce557a360810041507o6aa00ff4oc2690056f10f3df6@mail.gmail.com>
References: <ce557a360810041507o6aa00ff4oc2690056f10f3df6@mail.gmail.com>
Message-ID: <1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>

On Sat, Oct 4, 2008 at 3:07 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> I have a particular pattern of tests I'd like to implement, and I
> don't know a good way to make nose carry it out. The pattern is this:
> I have a class with a setUp function that creates various variables
> (an array of test data, etc.). I also have a handful of test_whatever
> methods. This works fine. But now I'd like to apply exactly the same
> test methods to a different set of data. My first attempt was to
> simply create a derived class which overrode the setUp method, but
> unfortunately neither unittest nor nose will run tests defined in the
> parent class.

Anne,

If I understand correctly, I think you might be able to use a
variation of this technique:

http://projects.scipy.org/scipy/numpy/wiki/TestingGuidelines#creating-many-similar-tests

Please let me know if that doesn't work (or if the doc isn't clear on
something).

Cheers,
Alan


From matthew.brett at gmail.com  Sat Oct  4 19:32:34 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 4 Oct 2008 16:32:34 -0700
Subject: [SciPy-dev] Using nose
In-Reply-To: <1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>
References: <ce557a360810041507o6aa00ff4oc2690056f10f3df6@mail.gmail.com>
	<1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>
Message-ID: <1e2af89e0810041632j609ee5dcv1c1fda5b69811712@mail.gmail.com>

Hi Anne,

> If I understand correctly, I think you might be able to use a
> variation of this technique:

I am confident I don't understand correctly, but how about:

data1 = [1]
data2 = [2]
datasets = [data1, data2]

def func1(data):
   pass

def func2(data):
   pass

funcs = [func1, func2]

# Nose test generator
def test_datas():
    for func in funcs:
        for dataset in datasets:
              yield func, data

?

Matthew


From peridot.faceted at gmail.com  Sat Oct  4 20:42:59 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 4 Oct 2008 20:42:59 -0400
Subject: [SciPy-dev] Using nose
In-Reply-To: <1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>
References: <ce557a360810041507o6aa00ff4oc2690056f10f3df6@mail.gmail.com>
	<1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>
Message-ID: <ce557a360810041742l59240da5l45b4be7528e6bdb3@mail.gmail.com>

2008/10/4 Alan McIntyre <alan.mcintyre at gmail.com>:
> On Sat, Oct 4, 2008 at 3:07 PM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
>> I have a particular pattern of tests I'd like to implement, and I
>> don't know a good way to make nose carry it out. The pattern is this:
>> I have a class with a setUp function that creates various variables
>> (an array of test data, etc.). I also have a handful of test_whatever
>> methods. This works fine. But now I'd like to apply exactly the same
>> test methods to a different set of data. My first attempt was to
>> simply create a derived class which overrode the setUp method, but
>> unfortunately neither unittest nor nose will run tests defined in the
>> parent class.
>
> Anne,
>
> If I understand correctly, I think you might be able to use a
> variation of this technique:
>
> http://projects.scipy.org/scipy/numpy/wiki/TestingGuidelines#creating-many-similar-tests
>
> Please let me know if that doesn't work (or if the doc isn't clear on
> something).

Aha! Thanks. This is pretty much what I tried to do, but there was a
bug in my code and so I assumed that nose behaved the same as unittest
in this.

Anne


From matthew.brett at gmail.com  Sat Oct  4 21:04:49 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 4 Oct 2008 18:04:49 -0700
Subject: [SciPy-dev] SciPy-dev] [patch] read/write v5 .mat files with
	structs, cell arrays, objects, or function handles
In-Reply-To: <6c17e6f50810030749p1966266bs23701e77f8c05b3a@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
	<5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
	<9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>
	<nq5prmjf5jd.fsf_-_@broad.mit.edu>
	<1e2af89e0810020834t64e104b2oaf5abad1b07d7333@mail.gmail.com>
	<5b8d13220810022229s5523d3bex93355ed687c8d6ea@mail.gmail.com>
	<1e2af89e0810022254k3ff22324k39d656d6d0b0aa38@mail.gmail.com>
	<6c17e6f50810030749p1966266bs23701e77f8c05b3a@mail.gmail.com>
Message-ID: <1e2af89e0810041804j411009o8024fd3b73bf4389@mail.gmail.com>

Hi,

On Fri, Oct 3, 2008 at 7:49 AM, Thouis (Ray) Jones <thouis at broad.mit.edu> wrote:
> I've updated the patch to provide backwards compatibility for matlab
> structures, controlled by a keyword argument to loadmat
> (struct_as_record), which defaults to False (old style), and gives a
> deprecation warning if False.

I've applied this patch, with thanks, all tests pass for me too.
David - any problems?

Matthew


From cournape at gmail.com  Sun Oct  5 01:57:43 2008
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 5 Oct 2008 14:57:43 +0900
Subject: [SciPy-dev] SciPy-dev] [patch] read/write v5 .mat files with
	structs, cell arrays, objects, or function handles
In-Reply-To: <1e2af89e0810041804j411009o8024fd3b73bf4389@mail.gmail.com>
References: <6c17e6f50810010852r24d2064qdffd92b5b9796dd9@mail.gmail.com>
	<9457e7c80810011320q41b8cf3fw8ec95308bc0a6e89@mail.gmail.com>
	<5b8d13220810011737x4b38a449ybe495dc7b961060a@mail.gmail.com>
	<9457e7c80810020041x485c1cb1w4581009630d53a6@mail.gmail.com>
	<nq5prmjf5jd.fsf_-_@broad.mit.edu>
	<1e2af89e0810020834t64e104b2oaf5abad1b07d7333@mail.gmail.com>
	<5b8d13220810022229s5523d3bex93355ed687c8d6ea@mail.gmail.com>
	<1e2af89e0810022254k3ff22324k39d656d6d0b0aa38@mail.gmail.com>
	<6c17e6f50810030749p1966266bs23701e77f8c05b3a@mail.gmail.com>
	<1e2af89e0810041804j411009o8024fd3b73bf4389@mail.gmail.com>
Message-ID: <5b8d13220810042257g12f57884td5abd4b18c1bd432@mail.gmail.com>

On Sun, Oct 5, 2008 at 10:04 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
>
> I've applied this patch, with thanks, all tests pass for me too.
> David - any problems?

Nope, works for me,

David


From peridot.faceted at gmail.com  Sun Oct  5 02:08:51 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 5 Oct 2008 02:08:51 -0400
Subject: [SciPy-dev] Using nose
In-Reply-To: <1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>
References: <ce557a360810041507o6aa00ff4oc2690056f10f3df6@mail.gmail.com>
	<1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>
Message-ID: <ce557a360810042308j1296bf11l972b11cbc751af4f@mail.gmail.com>

2008/10/4 Alan McIntyre <alan.mcintyre at gmail.com>:

> http://projects.scipy.org/scipy/numpy/wiki/TestingGuidelines#creating-many-similar-tests
>
> Please let me know if that doesn't work (or if the doc isn't clear on
> something).

While we're on the subject of testing, what's the best way to run a
(or some) tests and get dropped into the debugger when something goes
wrong? With my own code and unittest I there was a magic invocation (I
think it was MyTestClass().debug()) that I could run from inside
ipython with %pdb, so that when an exception was triggered I wound up
in the debugger. What's the best way to arrange this when I'm dealing
with debugging a module inside scipy?

Thanks,
Anne


From robert.kern at gmail.com  Sun Oct  5 02:12:58 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 5 Oct 2008 01:12:58 -0500
Subject: [SciPy-dev] Using nose
In-Reply-To: <ce557a360810042308j1296bf11l972b11cbc751af4f@mail.gmail.com>
References: <ce557a360810041507o6aa00ff4oc2690056f10f3df6@mail.gmail.com>
	<1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>
	<ce557a360810042308j1296bf11l972b11cbc751af4f@mail.gmail.com>
Message-ID: <3d375d730810042312w44a35986n3a7e59c3ad6123@mail.gmail.com>

On Sun, Oct 5, 2008 at 01:08, Anne Archibald <peridot.faceted at gmail.com> wrote:
> While we're on the subject of testing, what's the best way to run a
> (or some) tests and get dropped into the debugger when something goes
> wrong? With my own code and unittest I there was a magic invocation (I
> think it was MyTestClass().debug()) that I could run from inside
> ipython with %pdb, so that when an exception was triggered I wound up
> in the debugger. What's the best way to arrange this when I'm dealing
> with debugging a module inside scipy?

nosetests --pdb --pdb-failure

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From wnbell at gmail.com  Sun Oct  5 02:13:29 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Sun, 5 Oct 2008 02:13:29 -0400
Subject: [SciPy-dev] Using nose
In-Reply-To: <ce557a360810042308j1296bf11l972b11cbc751af4f@mail.gmail.com>
References: <ce557a360810041507o6aa00ff4oc2690056f10f3df6@mail.gmail.com>
	<1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>
	<ce557a360810042308j1296bf11l972b11cbc751af4f@mail.gmail.com>
Message-ID: <d05265cb0810042313x2ffed837h88b26fbf8423664e@mail.gmail.com>

On Sun, Oct 5, 2008 at 2:08 AM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
>
> While we're on the subject of testing, what's the best way to run a
> (or some) tests and get dropped into the debugger when something goes
> wrong? With my own code and unittest I there was a magic invocation (I
> think it was MyTestClass().debug()) that I could run from inside
> ipython with %pdb, so that when an exception was triggered I wound up
> in the debugger. What's the best way to arrange this when I'm dealing
> with debugging a module inside scipy?
>

Is this what you're looking for?

 $nosetests --pdb --pdb-failures scipy.something

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From peridot.faceted at gmail.com  Sun Oct  5 02:16:19 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 5 Oct 2008 02:16:19 -0400
Subject: [SciPy-dev] Using nose
In-Reply-To: <d05265cb0810042313x2ffed837h88b26fbf8423664e@mail.gmail.com>
References: <ce557a360810041507o6aa00ff4oc2690056f10f3df6@mail.gmail.com>
	<1d36917a0810041622r15db1d76h5669377960b30885@mail.gmail.com>
	<ce557a360810042308j1296bf11l972b11cbc751af4f@mail.gmail.com>
	<d05265cb0810042313x2ffed837h88b26fbf8423664e@mail.gmail.com>
Message-ID: <ce557a360810042316r6ad4cf52mc63d6b867ce36501@mail.gmail.com>

2008/10/5 Nathan Bell <wnbell at gmail.com>:
> On Sun, Oct 5, 2008 at 2:08 AM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
>>
>> While we're on the subject of testing, what's the best way to run a
>> (or some) tests and get dropped into the debugger when something goes
>> wrong? With my own code and unittest I there was a magic invocation (I
>> think it was MyTestClass().debug()) that I could run from inside
>> ipython with %pdb, so that when an exception was triggered I wound up
>> in the debugger. What's the best way to arrange this when I'm dealing
>> with debugging a module inside scipy?
>>
>
> Is this what you're looking for?
>
>  $nosetests --pdb --pdb-failures scipy.something

Perfect, thank you!

Anne


From peridot.faceted at gmail.com  Sun Oct  5 04:56:37 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 5 Oct 2008 04:56:37 -0400
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <48E797A0.3070604@aero.iitb.ac.in>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<48E31404.7000009@aero.iitb.ac.in>
	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>
	<48E3AAED.6080601@aero.iitb.ac.in>
	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>
	<48E76391.5050508@aero.iitb.ac.in>
	<ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>
	<48E797A0.3070604@aero.iitb.ac.in>
Message-ID: <ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>

2008/10/4 Prabhu Ramachandran <prabhu at aero.iitb.ac.in>:
> Anne Archibald wrote:
>> 2008/10/4 Prabhu Ramachandran <prabhu at aero.iitb.ac.in>:
>>> In any case, I am not sure there is a general need for this kind of data
>>> structure.  I need it a whole lot.  If there is a general need I'll be
>>> glad to try and extract any of my code and contribute it to scipy.spatial.
>>
>> There does seem to be a general need for a two-tree query that
>> extracts (or just counts) all items of one tree within a ball centered
>> at each item of another tree. You're right, too, that "in a ball"
>> queries often don't naturally produce sorted results, or even known
>> distances. But I think this has to be a different query. In fact, even
>> for single-point queries it may make sense to have a separate query
>> function for in-a-ball queries.
>
> True and another reason for a different approach would be performance.
> According to Lee and Wong (1970, Acta Informatica) the worst case
> performance for partial region searches on k-d trees is O(k N^{1-1/k}).
>  So if the dimensionality is small and the region size is know we could
> do a lot better with other specialized algorithms for these cases.

I have added several in-a-ball queries to the kd-tree implementation
in the spatial branch. Please let me know what you think. The API is a
bit ad-hoc, since I can see a number of different ways one might want
to use the code. In particular, some query functions do not currently
return the actual distance, since one of the shortcuts is to include
whole branches of the tree if their bounding box lies inside the ball
of interest. I suspect that for many algorithms of interest it will be
necessary to write a custom tree-walking algorithm. I'm not sure
whether it makes sense to try to provide functions to support this.

Anne


From nwagner at iam.uni-stuttgart.de  Sun Oct  5 05:51:39 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sun, 05 Oct 2008 11:51:39 +0200
Subject: [SciPy-dev] recent scipy.io
Message-ID: <web-112316743@uni-stuttgart.de>

======================================================================
ERROR: Failure: ImportError (No module named 
byteordercodes)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/loader.py", 
line 364, in loadTestsFromName
     addr.filename, addr.module)
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py", 
line 39, in importFromPath
     return self.importFromDir(dir_path, fqname)
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/importer.py", 
line 84, in importFromDir
     mod = load_module(part_fqname, fh, filename, desc)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_byteordercodes.py", 
line 9, in <module>
     import scipy.io.byteordercodes as sibc
ImportError: No module named byteordercodes

======================================================================
FAIL: test_mio.test_load('double', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testdouble_6.1_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testdouble_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testdouble_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testdouble_4.2c_SOL2.mat'], 
{'testdouble': array([[ 0.   ],
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 57, in _check_level
     assert_array_almost_equal(actual, expected, 
err_msg=label, decimal=5)
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", 
line 310, in assert_array_almost_equal
     header='Arrays are not almost equal')
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", 
line 260, in assert_array_compare
     raise AssertionError(msg)
AssertionError:
Arrays are not almost equal
test double; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testdouble_6.1_SOL2.mat, 
variable testdouble
(shapes (1, 9), (9, 1) mismatch)
  x: array([[ 0.        ,  0.78539816,  1.57079633, 
 2.35619449,  3.14159265,
          3.92699082,  4.71238898,  5.49778714, 
 6.28318531]])
  y: array([[ 0.        ],
        [ 0.78539816],
        [ 1.57079633],...

======================================================================
FAIL: test_mio.test_load('string', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststring_4.2c_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststring_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststring_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststring_6.1_SOL2.mat'], 
{'teststring': u'"Do nine men interpret?" "Nine men," I 
nod.'})
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 60, in _check_level
     "Types %s and %s do not match at %s" % (typex, typac, 
label)
AssertionError: Types <type 'unicode'> and <type 
'numpy.ndarray'> do not match at test string; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststring_4.2c_SOL2.mat, 
variable teststring

======================================================================
FAIL: test_mio.test_load('complex', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcomplex_4.2c_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcomplex_6.1_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcomplex_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcomplex_7.1_GLNX86.mat'], 
{'testcomplex': array([[  1.000e+00 +0.000e+00j],
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 57, in _check_level
     assert_array_almost_equal(actual, expected, 
err_msg=label, decimal=5)
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", 
line 310, in assert_array_almost_equal
     header='Arrays are not almost equal')
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", 
line 260, in assert_array_compare
     raise AssertionError(msg)
AssertionError:
Arrays are not almost equal
test complex; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcomplex_4.2c_SOL2.mat, 
variable testcomplex
(shapes (1, 9), (9, 1) mismatch)
  x: array([[  1.00000000e+00 +0.00000000e+00j,
           7.07106781e-01 +7.07106781e-01j,
           6.12323400e-17 +1.00000000e+00j,...
  y: array([[  1.00000000e+00 +0.00000000e+00j],
        [  7.07106781e-01 +7.07106781e-01j],
        [  6.12323400e-17 +1.00000000e+00j],...

======================================================================
FAIL: test_mio.test_load('multi', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testmulti_4.2c_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testmulti_7.1_GLNX86.mat'], 
{'a': array([[ 1.,  2.,  3.,  4.,  5.],
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 57, in _check_level
     assert_array_almost_equal(actual, expected, 
err_msg=label, decimal=5)
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", 
line 310, in assert_array_almost_equal
     header='Arrays are not almost equal')
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", 
line 260, in assert_array_compare
     raise AssertionError(msg)
AssertionError:
Arrays are not almost equal
test multi; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testmulti_4.2c_SOL2.mat, 
variable theta
(shapes (1, 9), (9, 1) mismatch)
  x: array([[ 0.        ,  0.78539816,  1.57079633, 
 2.35619449,  3.14159265,
          3.92699082,  4.71238898,  5.49778714, 
 6.28318531]])
  y: array([[ 0.        ],
        [ 0.78539816],
        [ 1.57079633],...

======================================================================
FAIL: test_mio.test_load('onechar', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testonechar_4.2c_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testonechar_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testonechar_6.1_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testonechar_6.5.1_GLNX86.mat'], 
{'testonechar': u'r'})
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 60, in _check_level
     "Types %s and %s do not match at %s" % (typex, typac, 
label)
AssertionError: Types <type 'unicode'> and <type 
'numpy.ndarray'> do not match at test onechar; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testonechar_4.2c_SOL2.mat, 
variable testonechar

======================================================================
FAIL: test_mio.test_load('cell', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcell_6.1_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcell_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcell_6.5.1_GLNX86.mat'], 
{'testcell': array([This cell contains this string and 3 
arrays of increasing length, 1, [1 2], [1 2 3]], 
dtype=object)})
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 27, in _check_level
     assert len(expected) == len(actual), "Different list 
lengths at %s" % label
AssertionError: Different list lengths at test cell; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcell_6.1_SOL2.mat, 
variable testcell

======================================================================
FAIL: test_mio.test_load('emptycell', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testemptycell_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testemptycell_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testemptycell_5.3_SOL2.mat'], 
{'testemptycell': array([1, 2, [], [], 3], dtype=object)})
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 27, in _check_level
     assert len(expected) == len(actual), "Different list 
lengths at %s" % label
AssertionError: Different list lengths at test emptycell; 
file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testemptycell_7.1_GLNX86.mat, 
variable testemptycell

======================================================================
FAIL: test_mio.test_load('stringarray', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststringarray_6.1_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststringarray_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststringarray_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststringarray_4.2c_SOL2.mat'], 
{'teststringarray': array([one  , two  , three], 
dtype=object)})
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 30, in _check_level
     _check_level(level_label, ev, actual[i])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 60, in _check_level
     "Types %s and %s do not match at %s" % (typex, typac, 
label)
AssertionError: Types <type 'unicode'> and <type 
'numpy.unicode_'> do not match at test stringarray; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststringarray_6.1_SOL2.mat, 
variable teststringarray, [0],

======================================================================
FAIL: test_mio.test_load('struct', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststruct_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststruct_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststruct_6.1_SOL2.mat'], 
{'teststruct': array([ (u'Rats live on no evil star.', 
[1.4142135623730951, 2.7182818284590451, 
3.1415926535897931], 
[(1.4142135623730951+1.4142135623730951j), 
(2.7182818284590451+2.7182818284590451j), 
(3.1415926535897931+3.1415926535897931j)])],
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 30, in _check_level
     _check_level(level_label, ev, actual[i])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 60, in _check_level
     "Types %s and %s do not match at %s" % (typex, typac, 
label)
AssertionError: Types <type 'numpy.void'> and <type 
'numpy.ndarray'> do not match at test struct; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststruct_6.5.1_GLNX86.mat, 
variable teststruct, [0],

======================================================================
FAIL: test_mio.test_load('cellnest', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcellnest_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcellnest_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcellnest_6.1_SOL2.mat'], 
{'testcellnest': array([1, [2 3 [4 5]]], dtype=object)})
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 27, in _check_level
     assert len(expected) == len(actual), "Different list 
lengths at %s" % label
AssertionError: Different list lengths at test cellnest; 
file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testcellnest_6.5.1_GLNX86.mat, 
variable testcellnest

======================================================================
FAIL: test_mio.test_load('structnest', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststructnest_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststructnest_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststructnest_6.1_SOL2.mat'], 
{'teststructnest': array([[(1, [[(u'number 3',)]])]],
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 30, in _check_level
     _check_level(level_label, ev, actual[i])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 30, in _check_level
     _check_level(level_label, ev, actual[i])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 60, in _check_level
     "Types %s and %s do not match at %s" % (typex, typac, 
label)
AssertionError: Types <type 'numpy.void'> and <class 
'scipy.io.matlab.mio5.mat_struct'> do not match at test 
structnest; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststructnest_7.1_GLNX86.mat, 
variable teststructnest, [0], , [0],

======================================================================
FAIL: test_mio.test_load('structarr', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststructarr_6.1_SOL2.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststructarr_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststructarr_7.1_GLNX86.mat'], 
{'teststructarr': array([[(1, 2)],
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 27, in _check_level
     assert len(expected) == len(actual), "Different list 
lengths at %s" % label
AssertionError: Different list lengths at test structarr; 
file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/teststructarr_6.1_SOL2.mat, 
variable teststructarr

======================================================================
FAIL: test_mio.test_load('object', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testobject_7.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testobject_6.5.1_GLNX86.mat', 
'/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testobject_6.1_SOL2.mat'], 
{'testobject': <scipy.io.matlab.mio5.MatlabObject instance 
at 0x33bbcf8>})
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 41, in _check_level
     assert k in ac_fields, "Missing property at %s" % 
label
AssertionError: Missing property at test object; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testobject_7.1_GLNX86.mat, 
variable testobject

======================================================================
FAIL: test_mio.test_load('unicode', 
['/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testunicode_7.1_GLNX86.mat'], 
{'testunicode': u'Japanese: 
\n\u3059\u3079\u3066\u306e\u4eba\u9593\u306f\u3001\u751f\u307e\u308c\u306a\u304c\u3089\u306b\u3057\u3066\u81ea\u7531\u3067\u3042\u308a\u3001\n\u304b\u3064\u3001\u5c0a\u53b3\u3068\u6a29\u5229\u3068 
\u306b\u3064\u3044\u3066\u5e73\u7b49\u3067\u3042\u308b\u3002\n\u4eba\u9593\u306f\u3001\u7406\u6027\u3068\u826f\u5fc3\u3068\u3092\u6388\u3051\u3089\u308c\u3066\u304a\u308a\u3001\n\u4e92\u3044\u306b\u540c\u80de\u306e\u7cbe\u795e\u3092\u3082\u3063\u3066\u884c\u52d5\u3057\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3044\u3002'})
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 60, in _check_level
     "Types %s and %s do not match at %s" % (typex, typac, 
label)
AssertionError: Types <type 'unicode'> and <type 
'numpy.ndarray'> do not match at test unicode; file 
/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/data/testunicode_7.1_GLNX86.mat, 
variable testunicode

======================================================================
FAIL: test_mio.test_round_trip('string_round_trip', 
{'teststring': u'"Do nine men interpret?" "Nine men," I 
nod.'}, '4')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 77, in _rt_check_case
     _check_case(name, [mat_stream], expected, 
struct_as_record=True)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 60, in _check_level
     "Types %s and %s do not match at %s" % (typex, typac, 
label)
AssertionError: Types <type 'unicode'> and <type 
'numpy.ndarray'> do not match at test string_round_trip; 
file <cStringIO.StringO object at 0x3751c00>, variable 
teststring

======================================================================
FAIL: test_mio.test_round_trip('onechar_round_trip', 
{'testonechar': u'r'}, '4')
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/nose-0.10.1-py2.5.egg/nose/case.py", 
line 203, in runTest
     self.test(*self.arg)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 77, in _rt_check_case
     _check_case(name, [mat_stream], expected, 
struct_as_record=True)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 70, in _check_case
     _check_level(k_label, expected, matdict[k])
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/io/matlab/tests/test_mio.py", 
line 60, in _check_level
     "Types %s and %s do not match at %s" % (typex, typac, 
label)
AssertionError: Types <type 'unicode'> and <type 
'numpy.ndarray'> do not match at test onechar_round_trip; 
file <cStringIO.StringO object at 0x3751810>, variable 
testonechar

======================================================================
FAIL: test_imresize (test_pilutil.TestPILUtil)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/testing/decorators.py", 
line 82, in skipper
     return f(*args, **kwargs)
   File 
"/usr/local/lib64/python2.5/site-packages/scipy/misc/tests/test_pilutil.py", 
line 24, in test_imresize
     assert_equal(im1.shape,(11,22))
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", 
line 174, in assert_equal
     assert_equal(len(actual),len(desired),err_msg,verbose)
   File 
"/usr/local/lib64/python2.5/site-packages/numpy/testing/utils.py", 
line 183, in assert_equal
     raise AssertionError(msg)
AssertionError:
Items are not equal:
  ACTUAL: 0
  DESIRED: 2

----------------------------------------------------------------------
Ran 2342 tests in 36.958s

FAILED (failures=17, errors=1)

  


From matthew.brett at gmail.com  Sun Oct  5 14:01:40 2008
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sun, 5 Oct 2008 11:01:40 -0700
Subject: [SciPy-dev] recent scipy.io
In-Reply-To: <web-112316743@uni-stuttgart.de>
References: <web-112316743@uni-stuttgart.de>
Message-ID: <1e2af89e0810051101g6c61b0d2i358cd9e13d8db16f@mail.gmail.com>

Hi,

> ======================================================================
> ERROR: Failure: ImportError (No module named
> byteordercodes)
> ----------------------------------------------------------------------

etc - my fault - refactoring leak, I got too used to bzr.

Thanks for the report,

Matthew


From alan.mcintyre at gmail.com  Mon Oct  6 04:54:03 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Mon, 6 Oct 2008 01:54:03 -0700
Subject: [SciPy-dev] PyArray_FromDims and friends
In-Reply-To: <2CC40A34-8316-4F21-B73E-E3A7E1148AB7@yale.edu>
References: <F4219518-E35B-47E0-B7C9-EF1CB7123994@yale.edu>
	<48A43348.2030006@netvision.net.il>
	<5C95B98C-5922-4D34-B3D3-B23C49E29300@yale.edu>
	<48A45D30.8040508@llnl.gov>
	<c7009a550808141322v59d81e4fgc7fc639f6c0390c6@mail.gmail.com>
	<3FA4ADBE-DD0B-4143-9D32-AA2971263FE2@yale.edu>
	<e06186140808150855v51ecdfedu6cb9068aaed28261@mail.gmail.com>
	<2CC40A34-8316-4F21-B73E-E3A7E1148AB7@yale.edu>
Message-ID: <1d36917a0810060154p4c55853cu9067fc22c6b352d1@mail.gmail.com>

On Fri, Aug 15, 2008 at 10:26 AM, Zachary Pincus
<zachary.pincus at yale.edu> wrote:
>> No, no, you can't just cast the pointer to a different type, it
>> needs to point to npy_intp instead of int to start with, i.e., n has
>> to be an array of npy_intp.
>
> Thanks for looking that over. This is of course what happens when I
> try to deal with C too late at night.
>
> For cases where the dimension was 1, I had originally had:
>
> [whatever] = (PyArrayObject
> *)PyArray_SimpleNew(1,&(npy_intp)n,PyArray_[whatever]);
>
> But I got compiler warnings to the tune of "The argument to & is not
> strictly an lvalue. This will be a hard error in the future." Which
> makes sense, (npy_intp)n isn't a proper lvalue... So I changed it to:
>
> [whatever] = (PyArrayObject *)PyArray_SimpleNew(1,
> (npy_intp*)&n,PyArray_[whatever]);
>
> which, as you point out, is pretty daft.
>
> Is there a correct, portable, and warning-free way to do this in one
> line? I'm guessing not. Anyhow, I'll fix the patch with a multi-line
> solution:
> npy_int intp_n;
> ...
> intp_n = (npy_intp) n;
> [whatever] = (PyArrayObject
> *)PyArray_SimpleNew(1,&intp_n,PyArray_[whatever]);
>
> Sorry for the noise, and thanks Chuck for the catch.

I added a patch to #723 (http://scipy.org/scipy/scipy/ticket/723) that
(I think) switches to use PyArray_SimpleNew instead of
PyArray_FromDims.  If nobody sees any problems with it I'll look it
over again tomorrow or Tuesday and apply it.  I'll try to get to
switching out PyArray_FromDimsAndDataAndDescr later this week if
nobody else gets around to it.


From Per.Brodtkorb at ffi.no  Mon Oct  6 10:47:28 2008
From: Per.Brodtkorb at ffi.no (Per.Brodtkorb at ffi.no)
Date: Mon, 6 Oct 2008 16:47:28 +0200
Subject: [SciPy-dev] Fwd: tests and partial
	patchforscipy.stats.distributions
In-Reply-To: <1cd32cbb0810020629w56cf8e89t58954256e87bb29e@mail.gmail.com>
References: <1cd32cbb0809282100x75eccfb9la109e92032f798ac@mail.gmail.com><1cd32cbb0809290551g7c00a5ddh44369352ece506d2@mail.gmail.com><1ED225FF18AA8B48AC192F7E1D032C6E0C5C1D@hbu-posten.ffi.no>
	<1cd32cbb0810020629w56cf8e89t58954256e87bb29e@mail.gmail.com>
Message-ID: <1ED225FF18AA8B48AC192F7E1D032C6E0C5C26@hbu-posten.ffi.no>

I have some suggestions for further improvement on distributions.py and is listed in the following:

When I import your scipy.stats.distributions into ipython, the doc-string for the distributions are not updated correctly.

To fix it I would also suggest to replace this part in .__init__ method of rv_discrete and rv_continous class:

if self.__doc__ is None:
	self.__doc__ = rv_discrete.__doc__
else:
	self.__doc__ = self.__doc__.replace("A Generic",longname)

with

if self.__doc__ is None:
	self.__doc__ = rv_discrete.__doc__
if self.__doc__ is None:
	self.__doc__ = self.__doc__.replace("A Generic",longname)


Replace
.... 
goodargs = argsreduce(cond, *((q,)+args+(loc,)))
loc, goodargs = goodargs[-1], goodargs[:-1]

if [1 for i in goodargs if any(i)]:
	place(output,cond,(self._ppf(*goodargs) + loc).tolist())  

with 

if any(cond):
	goodargs = argsreduce(cond, *((q,)+args+(loc,)))
	loc, goodargs = goodargs[-1], goodargs[:-1]
	place(output,cond,self._ppf(*goodargs) + loc)  


(I also think this fix could be applied to cdf, ppf......)

To fix the stats method of rv_frozen class, replace it with:
def stats(self,moments='mv'):
	kwds = self.kwds
      kwds.update({'moments':moments})
      return self.dist.stats(*self.args,**kwds)


In order to improve accuracy in calculation for small x values, I would also suggest to replace all calls to
log(1-x) and 1-exp(-x)
with 
log1p(-x) and -expm1(-x), respectively,


The _isf method of the rv_discrete class is never called.
In the isf method of the rv_discrete class replace

place(output,cond,self._ppf(*goodargs) + loc)
with
place(output,cond,self._isf(*goodargs) + loc)


Per A.

-----Opprinnelig melding-----
Fra: scipy-dev-bounces at scipy.org [mailto:scipy-dev-bounces at scipy.org] P? vegne av josef.pktd at gmail.com
Sendt: 2. oktober 2008 15:30
Til: SciPy Developers List
Emne: Re: [SciPy-dev] Fwd: tests and partial patchforscipy.stats.distributions

I finally filed a ticket on this on scipy trac:
http://scipy.org/scipy/scipy/ticket/745

On Wed, Oct 1, 2008 at 7:14 AM,  <Per.Brodtkorb at ffi.no> wrote:
> I have looked into the bugs in the patch of the discrete distributions you sent. The error in _ppf method of dlaplace can be corrected by replacing it with:
>
> class dlaplace_gen(rv_discrete):
> .....
>    def _ppf(self, q, a):
>        const = 1.0/(1+exp(-a))
>        cons2 = 1+exp(a)
>        ind = q < const
>        return ceil(where(ind, log(q*cons2)/a-1, -log((1-q)*cons2)/a))
> ......
>

Thanks, I didn't see the misplaced bracket, I changed my corrections
to this. Do you have any idea or reference about the theoretical
second moment of dlaplace?


> The tests written for the generic moment, fails for those distributions that set the limits (self.a and/or self.b) for the support of the distribution in the self._argcheck method. I think the generic_moment method is meant to be private to the self._munp method, since it does only limited input checks.
> Perhaps better to rename generic_moment to _generic_moment

I aggree, I didn't check this systematically, but I guess
'vec_generic_moment', 'vecentropy' are also private methods.
>
> Per A.

I did not realize that some of my tests on private methods might fail
for some distributions by design.
To make it a bit clearer, I separated the tests into tests for public
methods, and tests for private methods. I still like the tests of
private methods, even if some are known failures, since they help to
narrow down what the source of an exception or test failure in the
public methods is.

Josef
_______________________________________________
Scipy-dev mailing list
Scipy-dev at scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-dev


From prabhu at aero.iitb.ac.in  Mon Oct  6 16:05:30 2008
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Tue, 07 Oct 2008 01:35:30 +0530
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>	<320fb6e00809300129j1b45ee21k6756b4c2b1945d84@mail.gmail.com>	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>	<48E31404.7000009@aero.iitb.ac.in>	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>	<48E3AAED.6080601@aero.iitb.ac.in>	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>	<48E76391.5050508@aero.iitb.ac.in>	<ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>	<48E797A0.3070604@aero.iitb.ac.in>
	<ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>
Message-ID: <48EA6F8A.2020104@aero.iitb.ac.in>

Anne Archibald wrote:
> 2008/10/4 Prabhu Ramachandran <prabhu at aero.iitb.ac.in>:
> I have added several in-a-ball queries to the kd-tree implementation
> in the spatial branch. Please let me know what you think. The API is a
> bit ad-hoc, since I can see a number of different ways one might want
> to use the code. In particular, some query functions do not currently
> return the actual distance, since one of the shortcuts is to include
> whole branches of the tree if their bounding box lies inside the ball

I have only taken a cursory look and it looks very good to me.  I like 
the general API and love the fact that you have tests for everything. 
I'm too swamped to try out the branch at the moment though.  Thanks very 
much for doing this!

 > of interest. I suspect that for many algorithms of interest it will be
 > necessary to write a custom tree-walking algorithm. I'm not sure
 > whether it makes sense to try to provide functions to support this.

I am also not sure about this.  Once spatial is available in scipy 
proper perhaps we'll get more feedback on other things people are 
looking for.

regards,
prabhu


From wnbell at gmail.com  Mon Oct  6 16:17:50 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Mon, 6 Oct 2008 16:17:50 -0400
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<48E31404.7000009@aero.iitb.ac.in>
	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>
	<48E3AAED.6080601@aero.iitb.ac.in>
	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>
	<48E76391.5050508@aero.iitb.ac.in>
	<ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>
	<48E797A0.3070604@aero.iitb.ac.in>
	<ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>
Message-ID: <d05265cb0810061317l2fc86e00we1a4514693d553ed@mail.gmail.com>

On Sun, Oct 5, 2008 at 4:56 AM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
>
> to use the code. In particular, some query functions do not currently
> return the actual distance, since one of the shortcuts is to include
> whole branches of the tree if their bounding box lies inside the ball

That's a reasonable choice, we'd want to leave the door open for such
optimizations.

> of interest. I suspect that for many algorithms of interest it will be
> necessary to write a custom tree-walking algorithm. I'm not sure
> whether it makes sense to try to provide functions to support this.
>

I agree with Prabhu, start simple and elicit feedback.

IMO speed for the common query types is more important than support
for sophisticated traversals.

Thanks again for your work Anne.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From charlesr.harris at gmail.com  Mon Oct  6 17:03:56 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 6 Oct 2008 15:03:56 -0600
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<ce557a360809301431g58fd7a7dsca15e27828522d95@mail.gmail.com>
	<48E31404.7000009@aero.iitb.ac.in>
	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>
	<48E3AAED.6080601@aero.iitb.ac.in>
	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>
	<48E76391.5050508@aero.iitb.ac.in>
	<ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>
	<48E797A0.3070604@aero.iitb.ac.in>
	<ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>
Message-ID: <e06186140810061403g7622b35dn6f9e75e9a04b8069@mail.gmail.com>

On Sun, Oct 5, 2008 at 2:56 AM, Anne Archibald <peridot.faceted at gmail.com>wrote:

> 2008/10/4 Prabhu Ramachandran <prabhu at aero.iitb.ac.in>:
> > Anne Archibald wrote:
> >> 2008/10/4 Prabhu Ramachandran <prabhu at aero.iitb.ac.in>:
> >>> In any case, I am not sure there is a general need for this kind of
> data
> >>> structure.  I need it a whole lot.  If there is a general need I'll be
> >>> glad to try and extract any of my code and contribute it to
> scipy.spatial.
> >>
> >> There does seem to be a general need for a two-tree query that
> >> extracts (or just counts) all items of one tree within a ball centered
> >> at each item of another tree. You're right, too, that "in a ball"
> >> queries often don't naturally produce sorted results, or even known
> >> distances. But I think this has to be a different query. In fact, even
> >> for single-point queries it may make sense to have a separate query
> >> function for in-a-ball queries.
> >
> > True and another reason for a different approach would be performance.
> > According to Lee and Wong (1970, Acta Informatica) the worst case
> > performance for partial region searches on k-d trees is O(k N^{1-1/k}).
> >  So if the dimensionality is small and the region size is know we could
> > do a lot better with other specialized algorithms for these cases.
>
> I have added several in-a-ball queries to the kd-tree implementation
> in the spatial branch. Please let me know what you think. The API is a
> bit ad-hoc, since I can see a number of different ways one might want
> to use the code. In particular, some query functions do not currently
> return the actual distance, since one of the shortcuts is to include
> whole branches of the tree if their bounding box lies inside the ball
> of interest. I suspect that for many algorithms of interest it will be
> necessary to write a custom tree-walking algorithm. I'm not sure
> whether it makes sense to try to provide functions to support this.
>

I'm wondering if this might be a good place to add an equivalence relation
also. An equivalence relation is a good way to do clustering based on
distances, i.e., a.join(i,j) where i and j are objects creates a cluster
that is the union of those containing i and j respectively. All the
clustered elements can then be recovered using something like a.elements(i).

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081006/a41148ff/attachment.html>

From turian at gmail.com  Wed Oct  8 17:35:08 2008
From: turian at gmail.com (Joseph Turian)
Date: Wed, 8 Oct 2008 17:35:08 -0400
Subject: [SciPy-dev] Bug in scipy.weave.catalog.default_dir()
Message-ID: <4dacb2560810081435v3a161811wa79a1b97fbd41edd@mail.gmail.com>

I am using scipy.weave in an environment in which my home directory is
not accessable.
However, scipy.weave attempts to write to $HOME/.python25_compiled,
even though I set PYTHONCOMPILED.

Looking a little deeper, I see that scipy.weave.catalog.default_dir()
is implemented incorrectly.
The documentation says:
"If for some reason it isn't possible to build a default directory
in the user's home, /tmp/<uid>_pythonXX_compiled is used."

What actually happens is that if $HOME is defined,
$HOME/.python25_compiled is used and no exception is caught if writing
fails.
I would prefer that, as per the documentation, if
$HOME/.python25_compiled is not writeable then the exception is caught
and /tmp/<uid>_pythonXX_compiled is used instead.

Could someone please implement this?
It's a simple fix.

Thanks,
  Joseph

-- 
Academic: http://www-etud.iro.umontreal.ca/~turian/
Business: http://www.metaoptimize.com/


From peridot.faceted at gmail.com  Thu Oct  9 00:38:34 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 9 Oct 2008 00:38:34 -0400
Subject: [SciPy-dev] cython in scipy?
Message-ID: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>

Hi,

I've just been trying to clean up scipy.stats.vonmises.cdf: it's
currently badly broken (tickets 747, 748, and 749). I can fix the
outright bugs, but it also has distressingly large roundoff error
under some circumstances. I'd like to simply implement the standard
algorithm (Hill 1977) for evaluating it. Unfortunately, it involves
evaluating a series with a number of terms that depends on the
parameter, so vectorizing a python version (which is already written)
is going to be a nuisance. On the other hand, it would be very easy to
translate this into cython and get C performance out of it. So: is it
okay to include cython code in scipy? What is the best way to go about
this? Include the cython source in setup.py somehow? Check the
generated C into svn and run cython by hand when I remember to?

Thanks,
Anne


From cournape at gmail.com  Thu Oct  9 00:43:08 2008
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 9 Oct 2008 13:43:08 +0900
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
Message-ID: <5b8d13220810082143x5484eb09nad978d4008ad7228@mail.gmail.com>

Hi Anne,

On Thu, Oct 9, 2008 at 1:38 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
>On the other hand, it would be very easy to
> translate this into cython and get C performance out of it. So: is it
> okay to include cython code in scipy?

If there is a python version already (to serve as a reference), I
don't think it would be a big problem to include cython source.

> Check the
> generated C into svn and run cython by hand when I remember to?

Yes, that's already how it is done for numpy.random.mtrand. Cython is
not yet packaged by so many distributions, so depending on it for
building scipy is a bit premature IMHO.

cheers,

David


From millman at berkeley.edu  Thu Oct  9 00:51:37 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 8 Oct 2008 21:51:37 -0700
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
Message-ID: <c7009a550810082151v3e2ab375k82f384bfb4f4a166@mail.gmail.com>

On Wed, Oct 8, 2008 at 9:38 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> I've just been trying to clean up scipy.stats.vonmises.cdf: it's
> currently badly broken (tickets 747, 748, and 749). I can fix the
> outright bugs, but it also has distressingly large roundoff error
> under some circumstances. I'd like to simply implement the standard
> algorithm (Hill 1977) for evaluating it. Unfortunately, it involves
> evaluating a series with a number of terms that depends on the
> parameter, so vectorizing a python version (which is already written)
> is going to be a nuisance. On the other hand, it would be very easy to
> translate this into cython and get C performance out of it. So: is it
> okay to include cython code in scipy? What is the best way to go about
> this? Include the cython source in setup.py somehow? Check the
> generated C into svn and run cython by hand when I remember to?

Writing Cython code is fine with me.  I think you should check in the
generated C, though.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From peridot.faceted at gmail.com  Thu Oct  9 01:30:37 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 9 Oct 2008 01:30:37 -0400
Subject: [SciPy-dev] tickets 747, 748, 749 can be closed
Message-ID: <ce557a360810082230w3df1a6cep73433b2de71a158c@mail.gmail.com>

Tickets 747, 748, 749 can be closed, but unfortunately I (peridot)
don't seem to have permission to close them, in spite of having been
the one who submitted them. I did put a "closes tickets ..." in the
svn commit message, but I don't think there's an automatic
ticket-closing gizmo.

Anne


From robert.kern at gmail.com  Thu Oct  9 01:38:38 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 9 Oct 2008 00:38:38 -0500
Subject: [SciPy-dev] tickets 747, 748, 749 can be closed
In-Reply-To: <ce557a360810082230w3df1a6cep73433b2de71a158c@mail.gmail.com>
References: <ce557a360810082230w3df1a6cep73433b2de71a158c@mail.gmail.com>
Message-ID: <3d375d730810082238w483032a6wf302b67356366429@mail.gmail.com>

On Thu, Oct 9, 2008 at 00:30, Anne Archibald <peridot.faceted at gmail.com> wrote:
> Tickets 747, 748, 749 can be closed, but unfortunately I (peridot)
> don't seem to have permission to close them, in spite of having been
> the one who submitted them.

You now have such permissions.

> I did put a "closes tickets ..." in the
> svn commit message, but I don't think there's an automatic
> ticket-closing gizmo.

Correct.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From fperez.net at gmail.com  Thu Oct  9 02:09:22 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 8 Oct 2008 23:09:22 -0700
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
Message-ID: <db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>

On Wed, Oct 8, 2008 at 9:38 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> Hi,
>
> I've just been trying to clean up scipy.stats.vonmises.cdf: it's
> currently badly broken (tickets 747, 748, and 749). I can fix the
> outright bugs, but it also has distressingly large roundoff error
> under some circumstances. I'd like to simply implement the standard
> algorithm (Hill 1977) for evaluating it. Unfortunately, it involves
> evaluating a series with a number of terms that depends on the
> parameter, so vectorizing a python version (which is already written)
> is going to be a nuisance. On the other hand, it would be very easy to
> translate this into cython and get C performance out of it. So: is it
> okay to include cython code in scipy? What is the best way to go about
> this? Include the cython source in setup.py somehow? Check the
> generated C into svn and run cython by hand when I remember to?

This is the kind of very simple code I've used in the past (back in
the pyrex days, same thing for cython):

# Make this usable by people who don't have pyrex installed (I've committed
# the generated C sources to SVN).
try:
    from Pyrex.Distutils import build_ext
    has_pyrex = True
    cmdclass  = {'build_ext': build_ext}
except ImportError:
    has_pyrex = False
    cmdclass  = {}

# Set up key extension
key_sources = ['mwadap/key.pyx']
if not has_pyrex:
    key_sources = [s.replace('.pyx','.c') for s in key_sources]
keyx = Extension('mwadap.key',key_sources,
                 include_dirs = [numpy_include_dir])

###

Instead of triggering the use of the included C on the absence of
Cython, you could just trigger on the absence of the C sources to fall
back on the .pyx files (or even timestamp differences if you want).
This way, you can just nuke the C files with a little utility script
whenever you want and Cython will regenerate them for you at build
time.  Since they're checked into SVN, next  time you commit the
updated version will go back up.

The choice of whether to trigger the rebuild simply on timestamp
differences (a la .py/.pyc) or by manual removal of the C is up to
you, but the resulting workflow is pretty straightforward in any case.

Cheers,

f


From peridot.faceted at gmail.com  Thu Oct  9 03:40:04 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 9 Oct 2008 03:40:04 -0400
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>
Message-ID: <ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>

Okay, the cython code for the von Mises CDF is committed. It's
actually a mongrel, since I needed scipy.stats.norm.cdf and
scipy.special.i0 for one branch, and the only way I knew of get them
was through python, so I left that branch vectorized. Otherwise I
think the cython version would have been shorter than the python
version.

The new CDF code also makes a minor change in the API: for values
outside the domain -pi..pi, instead of returning cdf( (x+pi)%(2*pi)-pi
) it returns an increasing function. In particular, this means that
the probability of getting a value between x and y is cdf(y)-cdf(x)
whenever x and y differ by less than 2*pi. (Previously if x=3 and y=4
it gave you the probability minus 1.) This is valuable since the
"location" parameter shifts the domain: in particular, the CDF is zero
at "location" minus pi.

2008/10/9 Fernando Perez <fperez.net at gmail.com>:

> This is the kind of very simple code I've used in the past (back in
> the pyrex days, same thing for cython):
>
> # Make this usable by people who don't have pyrex installed (I've committed
> # the generated C sources to SVN).
> try:
>    from Pyrex.Distutils import build_ext
>    has_pyrex = True
>    cmdclass  = {'build_ext': build_ext}
> except ImportError:
>    has_pyrex = False
>    cmdclass  = {}
>
> # Set up key extension
> key_sources = ['mwadap/key.pyx']
> if not has_pyrex:
>    key_sources = [s.replace('.pyx','.c') for s in key_sources]
> keyx = Extension('mwadap.key',key_sources,
>                 include_dirs = [numpy_include_dir])
>
> ###

I didn't get this working, I'm afraid; I wasn't willing to delve into
the guts of numpy's distutils to figure out what was wrong, but it
complained that I'd given it a ".pyx" file but I didn't have Pyrex
installed.

Anne


From robince at gmail.com  Thu Oct  9 03:58:15 2008
From: robince at gmail.com (Robin)
Date: Thu, 9 Oct 2008 08:58:15 +0100
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>
	<ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>
Message-ID: <b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>

Hi

On Thu, Oct 9, 2008 at 8:40 AM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> I didn't get this working, I'm afraid; I wasn't willing to delve into
> the guts of numpy's distutils to figure out what was wrong, but it
> complained that I'd given it a ".pyx" file but I didn't have Pyrex
> installed.

There is a working cython example here:
http://projects.scipy.org/scipy/numpy/browser/trunk/doc/cython/setup.py

It might be as simple as changing Pyrex to Cython in the import.

Cheers

Robin


From fperez.net at gmail.com  Thu Oct  9 04:01:26 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 9 Oct 2008 01:01:26 -0700
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>
	<ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>
	<b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>
Message-ID: <db6b5ecc0810090101m27d94890l88c8eb35bc5b4cdb@mail.gmail.com>

On Thu, Oct 9, 2008 at 12:58 AM, Robin <robince at gmail.com> wrote:
> Hi
>
> On Thu, Oct 9, 2008 at 8:40 AM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
>> I didn't get this working, I'm afraid; I wasn't willing to delve into
>> the guts of numpy's distutils to figure out what was wrong, but it
>> complained that I'd given it a ".pyx" file but I didn't have Pyrex
>> installed.
>
> There is a working cython example here:
> http://projects.scipy.org/scipy/numpy/browser/trunk/doc/cython/setup.py
>
> It might be as simple as changing Pyrex to Cython in the import.

Yes, sorry about that.  I should have pointed to that example instead,
since it's been updated to use Cython and would have avoided confusing
Anne.  Given how I wrote that code, no valid excuse to hide behind...
Thanks for pointing it out and rescuing the idea :)

Cheers,

f


From robert.kern at gmail.com  Thu Oct  9 04:02:39 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 9 Oct 2008 03:02:39 -0500
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>
	<ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>
	<b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>
Message-ID: <3d375d730810090102x67b51428tdd7d662bdd3ca277@mail.gmail.com>

On Thu, Oct 9, 2008 at 02:58, Robin <robince at gmail.com> wrote:
> Hi
>
> On Thu, Oct 9, 2008 at 8:40 AM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
>> I didn't get this working, I'm afraid; I wasn't willing to delve into
>> the guts of numpy's distutils to figure out what was wrong, but it
>> complained that I'd given it a ".pyx" file but I didn't have Pyrex
>> installed.
>
> There is a working cython example here:
> http://projects.scipy.org/scipy/numpy/browser/trunk/doc/cython/setup.py
>
> It might be as simple as changing Pyrex to Cython in the import.

Not really. This isn't a standalone package where we can do things like that.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robince at gmail.com  Thu Oct  9 04:06:02 2008
From: robince at gmail.com (Robin)
Date: Thu, 9 Oct 2008 09:06:02 +0100
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <3d375d730810090102x67b51428tdd7d662bdd3ca277@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>
	<ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>
	<b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>
	<3d375d730810090102x67b51428tdd7d662bdd3ca277@mail.gmail.com>
Message-ID: <b14b627d0810090106u4995943ewa9145eab8dfd3141@mail.gmail.com>

On Thu, Oct 9, 2008 at 9:02 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Oct 9, 2008 at 02:58, Robin <robince at gmail.com> wrote:
>> It might be as simple as changing Pyrex to Cython in the import.
>
> Not really. This isn't a standalone package where we can do things like that.

Sorry - I meant in Anne's non-working example that Fernando originally
gave, the fix to get it working might be as simple as changing "from
Pyrex.Distutils" to "from Cython.Distutils"...

Not sure what you think I meant... :)

Cheers

Robin


From robert.kern at gmail.com  Thu Oct  9 04:12:27 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 9 Oct 2008 03:12:27 -0500
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <b14b627d0810090106u4995943ewa9145eab8dfd3141@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>
	<ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>
	<b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>
	<3d375d730810090102x67b51428tdd7d662bdd3ca277@mail.gmail.com>
	<b14b627d0810090106u4995943ewa9145eab8dfd3141@mail.gmail.com>
Message-ID: <3d375d730810090112y7e722687jcc3ab7885eaf715d@mail.gmail.com>

On Thu, Oct 9, 2008 at 03:06, Robin <robince at gmail.com> wrote:
> On Thu, Oct 9, 2008 at 9:02 AM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Thu, Oct 9, 2008 at 02:58, Robin <robince at gmail.com> wrote:
>>> It might be as simple as changing Pyrex to Cython in the import.
>>
>> Not really. This isn't a standalone package where we can do things like that.
>
> Sorry - I meant in Anne's non-working example that Fernando originally
> gave, the fix to get it working might be as simple as changing "from
> Pyrex.Distutils" to "from Cython.Distutils"...
>
> Not sure what you think I meant... :)

I thought you were talking about Anne's actual problem, which is using
Cython inside scipy.stats. The errors referring to Pyrex come from
inside numpy.distutils, which already has code to handle .pyx files as
Pyrex, not Cython files. Even if you fix the example, the actual
problem isn't solvable like that.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From peridot.faceted at gmail.com  Thu Oct  9 04:26:31 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 9 Oct 2008 04:26:31 -0400
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <3d375d730810090112y7e722687jcc3ab7885eaf715d@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>
	<ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>
	<b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>
	<3d375d730810090102x67b51428tdd7d662bdd3ca277@mail.gmail.com>
	<b14b627d0810090106u4995943ewa9145eab8dfd3141@mail.gmail.com>
	<3d375d730810090112y7e722687jcc3ab7885eaf715d@mail.gmail.com>
Message-ID: <ce557a360810090126t38a2b767q7f270db3c8075879@mail.gmail.com>

2008/10/9 Robert Kern <robert.kern at gmail.com>:
> On Thu, Oct 9, 2008 at 03:06, Robin <robince at gmail.com> wrote:
>> On Thu, Oct 9, 2008 at 9:02 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Thu, Oct 9, 2008 at 02:58, Robin <robince at gmail.com> wrote:
>>>> It might be as simple as changing Pyrex to Cython in the import.
>>>
>>> Not really. This isn't a standalone package where we can do things like that.
>>
>> Sorry - I meant in Anne's non-working example that Fernando originally
>> gave, the fix to get it working might be as simple as changing "from
>> Pyrex.Distutils" to "from Cython.Distutils"...
>>
>> Not sure what you think I meant... :)
>
> I thought you were talking about Anne's actual problem, which is using
> Cython inside scipy.stats. The errors referring to Pyrex come from
> inside numpy.distutils, which already has code to handle .pyx files as
> Pyrex, not Cython files. Even if you fix the example, the actual
> problem isn't solvable like that.

This was the conclusion I drew (having tried substituting Cython in
the import), and why I gave up. What would be involved in fixing
numpy.distutils to resolve this problem? I realize that it will be
awkward because cython and pyrex use the same extension but are not
compatible.

Anne


From robert.kern at gmail.com  Thu Oct  9 04:36:55 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 9 Oct 2008 03:36:55 -0500
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <ce557a360810090126t38a2b767q7f270db3c8075879@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>
	<ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>
	<b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>
	<3d375d730810090102x67b51428tdd7d662bdd3ca277@mail.gmail.com>
	<b14b627d0810090106u4995943ewa9145eab8dfd3141@mail.gmail.com>
	<3d375d730810090112y7e722687jcc3ab7885eaf715d@mail.gmail.com>
	<ce557a360810090126t38a2b767q7f270db3c8075879@mail.gmail.com>
Message-ID: <3d375d730810090136rcf7b98dsd05c6e36e4d4b6c3@mail.gmail.com>

On Thu, Oct 9, 2008 at 03:26, Anne Archibald <peridot.faceted at gmail.com> wrote:
> 2008/10/9 Robert Kern <robert.kern at gmail.com>:
>> On Thu, Oct 9, 2008 at 03:06, Robin <robince at gmail.com> wrote:
>>> On Thu, Oct 9, 2008 at 9:02 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>>> On Thu, Oct 9, 2008 at 02:58, Robin <robince at gmail.com> wrote:
>>>>> It might be as simple as changing Pyrex to Cython in the import.
>>>>
>>>> Not really. This isn't a standalone package where we can do things like that.
>>>
>>> Sorry - I meant in Anne's non-working example that Fernando originally
>>> gave, the fix to get it working might be as simple as changing "from
>>> Pyrex.Distutils" to "from Cython.Distutils"...
>>>
>>> Not sure what you think I meant... :)
>>
>> I thought you were talking about Anne's actual problem, which is using
>> Cython inside scipy.stats. The errors referring to Pyrex come from
>> inside numpy.distutils, which already has code to handle .pyx files as
>> Pyrex, not Cython files. Even if you fix the example, the actual
>> problem isn't solvable like that.
>
> This was the conclusion I drew (having tried substituting Cython in
> the import), and why I gave up. What would be involved in fixing
> numpy.distutils to resolve this problem? I realize that it will be
> awkward because cython and pyrex use the same extension but are not
> compatible.

We could extend the API somehow. Configuration.add_cython_extension()
or something like that.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Fri Oct 10 02:35:01 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 10 Oct 2008 15:35:01 +0900
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <ce557a360810090126t38a2b767q7f270db3c8075879@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>	<db6b5ecc0810082309n64ab62k57d885ec2702b11@mail.gmail.com>	<ce557a360810090040o6e4362bbr8a0130aaa4ab17cc@mail.gmail.com>	<b14b627d0810090058s7dc6071fra9249d05026a9bda@mail.gmail.com>	<3d375d730810090102x67b51428tdd7d662bdd3ca277@mail.gmail.com>	<b14b627d0810090106u4995943ewa9145eab8dfd3141@mail.gmail.com>	<3d375d730810090112y7e722687jcc3ab7885eaf715d@mail.gmail.com>
	<ce557a360810090126t38a2b767q7f270db3c8075879@mail.gmail.com>
Message-ID: <48EEF795.3010901@ar.media.kyoto-u.ac.jp>

Anne Archibald wrote:
>
> This was the conclusion I drew (having tried substituting Cython in
> the import), and why I gave up. What would be involved in fixing
> numpy.distutils to resolve this problem? I realize that it will be
> awkward because cython and pyrex use the same extension but are not
> compatible.

Should be doable if you don't want to mix cython and pyrex in the same
setup.py. FWIW, numscons has cython support, so this won't be a problem
either.

cheers,

David


From charlesr.harris at gmail.com  Sat Oct 11 13:49:36 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Sat, 11 Oct 2008 11:49:36 -0600
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <c7009a550810082151v3e2ab375k82f384bfb4f4a166@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<c7009a550810082151v3e2ab375k82f384bfb4f4a166@mail.gmail.com>
Message-ID: <e06186140810111049s699b239laac0ac0ce09e324b@mail.gmail.com>

On Wed, Oct 8, 2008 at 10:51 PM, Jarrod Millman <millman at berkeley.edu>wrote:

> On Wed, Oct 8, 2008 at 9:38 PM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
> > I've just been trying to clean up scipy.stats.vonmises.cdf: it's
> > currently badly broken (tickets 747, 748, and 749). I can fix the
> > outright bugs, but it also has distressingly large roundoff error
> > under some circumstances. I'd like to simply implement the standard
> > algorithm (Hill 1977) for evaluating it. Unfortunately, it involves
> > evaluating a series with a number of terms that depends on the
> > parameter, so vectorizing a python version (which is already written)
> > is going to be a nuisance. On the other hand, it would be very easy to
> > translate this into cython and get C performance out of it. So: is it
> > okay to include cython code in scipy? What is the best way to go about
> > this? Include the cython source in setup.py somehow? Check the
> > generated C into svn and run cython by hand when I remember to?
>
> Writing Cython code is fine with me.  I think you should check in the
> generated C, though.
>

It would probably be better to check in both so that the .pyx file could be
used for fixes or a template for other functions.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081011/14e2de82/attachment.html>

From millman at berkeley.edu  Sat Oct 11 14:36:06 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sat, 11 Oct 2008 11:36:06 -0700
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <e06186140810111049s699b239laac0ac0ce09e324b@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<c7009a550810082151v3e2ab375k82f384bfb4f4a166@mail.gmail.com>
	<e06186140810111049s699b239laac0ac0ce09e324b@mail.gmail.com>
Message-ID: <c7009a550810111136j631f4a45ifbb706c56d5cd90a@mail.gmail.com>

On Sat, Oct 11, 2008 at 10:49 AM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>> Writing Cython code is fine with me.  I think you should check in the
>> generated C, though.
>
> It would probably be better to check in both so that the .pyx file could be
> used for fixes or a template for other functions.

+1, I meant check in the generated C in addition to the cython code.


From peridot.faceted at gmail.com  Sat Oct 11 19:37:37 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 11 Oct 2008 19:37:37 -0400
Subject: [SciPy-dev] cython in scipy?
In-Reply-To: <c7009a550810111136j631f4a45ifbb706c56d5cd90a@mail.gmail.com>
References: <ce557a360810082138v7e703b9fge815f37babbf3c7e@mail.gmail.com>
	<c7009a550810082151v3e2ab375k82f384bfb4f4a166@mail.gmail.com>
	<e06186140810111049s699b239laac0ac0ce09e324b@mail.gmail.com>
	<c7009a550810111136j631f4a45ifbb706c56d5cd90a@mail.gmail.com>
Message-ID: <ce557a360810111637w6dac1f8dvd8aca0d1e16d16a0@mail.gmail.com>

2008/10/11 Jarrod Millman <millman at berkeley.edu>:
> On Sat, Oct 11, 2008 at 10:49 AM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>> Writing Cython code is fine with me.  I think you should check in the
>>> generated C, though.
>>
>> It would probably be better to check in both so that the .pyx file could be
>> used for fixes or a template for other functions.
>
> +1, I meant check in the generated C in addition to the cython code.

In the event, I checked in the C, the pyx, and the python it was based
on. I may end up moving the code around, since the von Mises CDF is
actually some sort of incomplete Bessel function and since if my code
were in scipy.special it might be easier to get at compiled versions
of i0 and erf (and therefore provide an entirely compiled function
without the gruesome vectorization).

Anne


From peridot.faceted at gmail.com  Sun Oct 12 09:43:13 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 12 Oct 2008 09:43:13 -0400
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <d05265cb0810061317l2fc86e00we1a4514693d553ed@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E31404.7000009@aero.iitb.ac.in>
	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>
	<48E3AAED.6080601@aero.iitb.ac.in>
	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>
	<48E76391.5050508@aero.iitb.ac.in>
	<ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>
	<48E797A0.3070604@aero.iitb.ac.in>
	<ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>
	<d05265cb0810061317l2fc86e00we1a4514693d553ed@mail.gmail.com>
Message-ID: <ce557a360810120643k4ac8afedgd43cfb0a0ee3dad0@mail.gmail.com>

2008/10/6 Nathan Bell <wnbell at gmail.com>:

> IMO speed for the common query types is more important than support
> for sophisticated traversals.

There is now a compiled kd-tree implementation in scipy.spatial. It is
written in cython and based on the python implementation. It supports
only optionally-bounded, optionally-approximate, k-nearest neighbor
queries but runs without any per-point python code. It includes all
the algorithmic optimizations described by the ANN authors (sliding
midpoint subdivision, multiple-entry leaves, updating minimum-distance
calculation, priority search, and short-circuit distance
calculations). I think it's pretty good. The major feature it is
missing, from what people have asked for, is an all-neighbors query.

It's more or less impossible to traverse these trees from within
python. I am considering a tree-walking API to make it easy to write
algorithms that work on kd-trees efficiently, but it may make more
sense to just let people use the python implementation for this.

Anne


From wnbell at gmail.com  Sun Oct 12 14:44:42 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Sun, 12 Oct 2008 14:44:42 -0400
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <ce557a360810120643k4ac8afedgd43cfb0a0ee3dad0@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<ce557a360810010246n74d1357eh3c48956615612450@mail.gmail.com>
	<48E3AAED.6080601@aero.iitb.ac.in>
	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>
	<48E76391.5050508@aero.iitb.ac.in>
	<ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>
	<48E797A0.3070604@aero.iitb.ac.in>
	<ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>
	<d05265cb0810061317l2fc86e00we1a4514693d553ed@mail.gmail.com>
	<ce557a360810120643k4ac8afedgd43cfb0a0ee3dad0@mail.gmail.com>
Message-ID: <d05265cb0810121144p78a8120ax430f85f37d7c5ef4@mail.gmail.com>

On Sun, Oct 12, 2008 at 9:43 AM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
>
> There is now a compiled kd-tree implementation in scipy.spatial. It is
> written in cython and based on the python implementation. It supports
> only optionally-bounded, optionally-approximate, k-nearest neighbor
> queries but runs without any per-point python code. It includes all
> the algorithmic optimizations described by the ANN authors (sliding
> midpoint subdivision, multiple-entry leaves, updating minimum-distance
> calculation, priority search, and short-circuit distance
> calculations). I think it's pretty good. The major feature it is
> missing, from what people have asked for, is an all-neighbors query.

Very nice!  Do you know offhand how much faster the cython
implementation is for a typical case?

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From peridot.faceted at gmail.com  Sun Oct 12 18:41:47 2008
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 12 Oct 2008 18:41:47 -0400
Subject: [SciPy-dev] [Numpy-discussion] Proposal: scipy.spatial
In-Reply-To: <d05265cb0810121144p78a8120ax430f85f37d7c5ef4@mail.gmail.com>
References: <ce557a360809292024o121be824nfc3d015b54c48ed1@mail.gmail.com>
	<48E3AAED.6080601@aero.iitb.ac.in>
	<d05265cb0810011056t34b3b0b7u171b7f73e3805819@mail.gmail.com>
	<48E76391.5050508@aero.iitb.ac.in>
	<ce557a360810040557g2f11f1f3rd3114dbae6d91a83@mail.gmail.com>
	<48E797A0.3070604@aero.iitb.ac.in>
	<ce557a360810050156g23429727v5435346cb6c3ea71@mail.gmail.com>
	<d05265cb0810061317l2fc86e00we1a4514693d553ed@mail.gmail.com>
	<ce557a360810120643k4ac8afedgd43cfb0a0ee3dad0@mail.gmail.com>
	<d05265cb0810121144p78a8120ax430f85f37d7c5ef4@mail.gmail.com>
Message-ID: <ce557a360810121541j49570320wa4cfe0fc27ffc0b9@mail.gmail.com>

2008/10/12 Nathan Bell <wnbell at gmail.com>:
> On Sun, Oct 12, 2008 at 9:43 AM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
>>
>> There is now a compiled kd-tree implementation in scipy.spatial. It is
>> written in cython and based on the python implementation. It supports
>> only optionally-bounded, optionally-approximate, k-nearest neighbor
>> queries but runs without any per-point python code. It includes all
>> the algorithmic optimizations described by the ANN authors (sliding
>> midpoint subdivision, multiple-entry leaves, updating minimum-distance
>> calculation, priority search, and short-circuit distance
>> calculations). I think it's pretty good. The major feature it is
>> missing, from what people have asked for, is an all-neighbors query.
>
> Very nice!  Do you know offhand how much faster the cython
> implementation is for a typical case?

I'm not sure what's "typical" - dimensionality certainly affects the
performance of kd-trees - but it seems to be a speefup by a factor of
20-50:

dimension 3, 10000 points
KDTree constructed:	0.208447
cKDTree constructed:	0.00362301
KDTree 1000 lookups:	0.953508
cKDTree 1000 lookups:	0.0197051
flat cKDTree 1000 lookups:	1.46467

dimension 8, 10000 points
KDTree constructed:	0.394341
cKDTree constructed:	0.00567818
KDTree 1000 lookups:	17.0404
cKDTree 1000 lookups:	0.374597
flat cKDTree 1000 lookups:	2.96095

dimension 16, 10000 points
KDTree constructed:	0.29892
cKDTree constructed:	0.00891495
KDTree 100 lookups:	10.3783
cKDTree 100 lookups:	0.472985
flat cKDTree 100 lookups:	0.667156

These numbers are in seconds; the distribution is a mixture of normal
distributions some moderate distance apart; and the "flat cKDTree"
serves as a somewhat-optimized compiled brute-force search.

Anne
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kdtree_speed.py
Type: text/x-python
Size: 828 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081012/a500a78a/attachment.py>

From dave.hirschfeld at gmail.com  Tue Oct 14 13:52:02 2008
From: dave.hirschfeld at gmail.com (Dave)
Date: Tue, 14 Oct 2008 17:52:02 +0000 (UTC)
Subject: [SciPy-dev] Timeseries Unusual Behaviour
Message-ID: <loom.20081014T170518-386@post.gmane.org>

Firstly, thanks for a great package I find it very useful for my timeseries 
analysis at work. I have however just stumbled upon some unusual behaviour...

It appears that the `date_array` function sorts the dates when constructing a 
DateArray instance. This bit me when I tried to compare the timestamps against 
another `values` array so perhaps this behaviour should be mentioned in the 
docstring if it is indeed desirable.

The `time_series` function magically appears to be able to handle the sorted 
`DateArray` when first called, but when called again with the same arguments it 
gets the order wrong. Is this reproducible? Is it a bug? Or am I just doing 
something stupid?!

Thanks,
Dave


Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)]
IPython 0.9.1 -- An enhanced Interactive Python.

In [2]: import datetime

In [3]: import scikits.timeseries as ts

In [4]: ts.__version__
Out[4]: '0.67.dev'

In [5]: numpy.__version__
Out[5]: '1.3.0.dev5859'

In [6]: dates = (datetime.datetime(2011, 5, 1, 0, 0),
   ...: datetime.datetime(2011, 8, 1, 0, 0),
   ...: datetime.datetime(2011, 6, 1, 0, 0),
   ...: datetime.datetime(2011, 9, 1, 0, 0),
   ...: datetime.datetime(2011, 7, 1, 0, 0))

In [7]: values = rand(len(dates))

In [8]: zip(dates,values)
Out[8]:
[(datetime.datetime(2011, 5, 1, 0, 0), 0.68325143668413102),
 (datetime.datetime(2011, 8, 1, 0, 0), 0.5341201248082369),
 (datetime.datetime(2011, 6, 1, 0, 0), 0.419866941013046),
 (datetime.datetime(2011, 9, 1, 0, 0), 0.73489099683479775),
 (datetime.datetime(2011, 7, 1, 0, 0), 0.7040093508897648)]

In [9]: date_array = ts.date_array(dlist=dates)

In [10]: # INCORRECT !

In [11]: zip(date_array,values)
Out[11]:
[(<U : 01-May-2011>, 0.68325143668413102),
 (<U : 01-Jun-2011>, 0.5341201248082369),
 (<U : 01-Jul-2011>, 0.419866941013046),
 (<U : 01-Aug-2011>, 0.73489099683479775),
 (<U : 01-Sep-2011>, 0.7040093508897648)]

In [12]: # CORRECT

In [13]: timeseries = ts.time_series(values,dates=date_array)

In [14]: zip(timeseries.dates,timeseries.data)
Out[14]:
[(<U : 01-May-2011>, 0.68325143668413102),
 (<U : 01-Jun-2011>, 0.419866941013046),
 (<U : 01-Jul-2011>, 0.7040093508897648),
 (<U : 01-Aug-2011>, 0.5341201248082369),
 (<U : 01-Sep-2011>, 0.73489099683479775)]

In [15]: # INCORRECT !?!

In [16]: timeseries = ts.time_series(values,dates=date_array)

In [17]: zip(timeseries.dates,timeseries.data)
Out[17]:
[(<U : 01-May-2011>, 0.68325143668413102),
 (<U : 01-Jun-2011>, 0.5341201248082369),
 (<U : 01-Jul-2011>, 0.419866941013046),
 (<U : 01-Aug-2011>, 0.73489099683479775),
 (<U : 01-Sep-2011>, 0.7040093508897648)]

In [18]





From dpeterson at enthought.com  Fri Oct 17 19:01:41 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Fri, 17 Oct 2008 18:01:41 -0500
Subject: [SciPy-dev] ANNOUNCE: EPD with Py2.5 version 4.0.30002 RC2
	available for testing
Message-ID: <48F91955.70400@enthought.com>

Hello,

We've recently posted the RC2 build of EPD (the Enthought Python
Distribution) with Python 2.5 version 4.0.30002 to the EPD website.  You
may download the RC from here:
	http://www.enthought.com/products/epdbeta.php

You can check out the release notes here:
	https://svn.enthought.com/epd/wiki/Python2.5.2/4.0.300/RC2

Please help us test it out and provide feedback on the EPD Trac
instance: https://svn.enthought.com/epd or via e-mail to
epd-support at enthought.com.   If everything goes well, we are planning a
final release for this coming Tuesday, October 21st.


About EPD
---------
The Enthought Python Distribution (EPD) is a "kitchen-sink-included"
distribution of the Python? Programming Language, including over 60
additional tools and libraries. The EPD bundle includes NumPy, SciPy,
IPython, 2D and 3D visualization, database adapters, and a lot of
other tools right out of the box.

	http://www.enthought.com/products/epd.php

It is currently available as a single-click installer for Windows XP
(x86), Mac OS X (a universal binary for OS X 10.4 and above), and
RedHat 3 and 4 (x86 and amd64).

EPD is free for academic use.  An annual subscription and installation
support are available for individual commercial use.  An enterprise
subscription with support for particular deployment environments is also
available for commercial purchase.


-- Dave




From philbinj at gmail.com  Sat Oct 18 11:56:52 2008
From: philbinj at gmail.com (James Philbin)
Date: Sat, 18 Oct 2008 16:56:52 +0100
Subject: [SciPy-dev] scipy.sparse.dok_matrix changes
Message-ID: <2b1c8c4f0810180856q2ef0d97cqf5defe1cd1aae173@mail.gmail.com>

Hi,

I've recently been using dok_matrix a fair bit and have made a few
improvements, mainly:
i. Improved the docstring.
ii. Fixed the constructor (this fixes #755).
iii. Reduced the codepath for the common case of setting an individual
element (~2x speedup).

There's more work to do on dok_matrix, but this patch might be useful
for some people:
---
Index: scipy/sparse/tests/test_base.py
===================================================================
--- scipy/sparse/tests/test_base.py	(revision 4801)
+++ scipy/sparse/tests/test_base.py	(working copy)
@@ -1162,8 +1162,27 @@
         except:
             caught += 1
         assert_equal(caught,5)
+
+    def test_ctor(self):
+        caught = 0
+        # Empty ctor
+        try:
+            A = dok_matrix()
+        except TypeError, e:
+            caught+=1
+        assert_equal(caught, 1)
+
+        # Dense ctor
+        b = matrix([[1,0,0,0],[0,0,1,0],[0,2,0,3]],'d')
+        A = dok_matrix(b)
+        assert_equal(A.todense(), b)

+        # Sparse ctor
+        c = csr_matrix(b)
+        assert_equal(A.todense(), c.todense())

+
+
 class TestLIL( _TestCommon, _TestHorizSlicing, _TestVertSlicing,
         _TestBothSlicing, _TestGetSet, _TestSolve,
         _TestArithmetic, _TestInplaceArithmetic,
Index: scipy/sparse/dok.py
===================================================================
--- scipy/sparse/dok.py	(revision 4801)
+++ scipy/sparse/dok.py	(working copy)
@@ -14,45 +14,65 @@

 class dok_matrix(spmatrix, dict):
     """Dictionary Of Keys based matrix.  This is an efficient
-    structure for constructing sparse matrices
+    structure for constructing sparse matrices incrementally.
+
+    This can be instatiated in several ways:
+        dok_matrix(D)
+            with a dense matrix, D
+
+        dok_matrix(S)
+            with a sparse matrix, S
+
+        dok_matrix((M,N), [dtype])
+            create the matrix with initial shape (M,N)
+            dtype is optional, defaulting to dtype='d'
+
+    Notes
+    -----
+    Allows for efficient O(1) access of individual elements.
+    Duplicates are not allowed.
+    Can be efficiently converted to a coo_matrix once constructed.
+
+    Examples
+    --------
+    >>> from scipy.sparse import *
+    >>> from scipy import *
+    >>> S = dok_matrix((5,5), dtype=float32)
+    >>> for i in range(5):
+    >>>     for j in range(5):
+    >>>         S[i,j] = i+j # Update element
+
     """
-    def __init__(self, A=None, shape=None, dtype=None, copy=False):
-        """ Create a new dictionary-of-keys sparse matrix.  An optional
-        argument A is accepted, which initializes the dok_matrix with it.
-        This can be a tuple of dimensions (M, N) or a (dense) array
-        to copy.
-        """
-        #TODO deprecate argument A in favor of arg1 style
-
+
+    def __init__(self, arg1, shape=None, dtype=None, copy=False):
         dict.__init__(self)
         spmatrix.__init__(self)
-        self.dtype = getdtype(dtype, A, default=float)
-        if A is not None:
-            if isinstance(A, tuple):
-                # Interpret as dimensions
-                if not isshape(A):
-                    raise TypeError, "dimensions must be a 2-tuple of
positive"\
-                            " integers"
-                self.shape = A
-            elif isspmatrix(A):
-                if isspmatrix_dok(A) and copy:
-                    A = A.copy()
-                else:
-                    A = A.todok()
-                self.update( A )
-                self.shape = A.shape
-                self.dtype = A.dtype
+
+        self.dtype = getdtype(dtype, default=float)
+        if isinstance(arg1, tuple) and isshape(arg1): # (M,N)
+            M, N = arg1
+            self.shape = (M, N)
+        elif isspmatrix(arg1): # Sparse ctor
+            if isspmatrix_dok(arg1) and copy:
+                arg1 = arg1.copy()
             else:
-                #must be dense, convert to COO first, then to DOK
-                try:
-                    A = asarray(A)
-                except:
-                    raise ValueError, "unrecognized form for" \
-                            " %s_matrix constructor" % self.format
-                from coo import coo_matrix
-                self.update( coo_matrix(A).todok() )
-                self.shape = A.shape
-                self.dtype = A.dtype
+                arg1 = arg1.todok()
+            self.update(arg1)
+            self.shape = arg1.shape
+            self.dtype = arg1.dtype
+        else: # Dense ctor
+            try:
+                arg1 = asarray(arg1)
+            except:
+                raise TypeError('invalid input format')
+
+            if len(arg1.shape)!=2:
+                raise TypeError('expected rank <=2 dense array or matrix')
+
+            from coo import coo_matrix
+            self.update( coo_matrix(arg1).todok() )
+            self.shape = arg1.shape
+            self.dtype = arg1.dtype

     def getnnz(self):
         return dict.__len__(self)
@@ -90,7 +110,7 @@

         # Bounds checking
         if isintlike(i):
-            if i < 0:
+            if i < 0:
                 i += self.shape[0]
             if i < 0 or i >= self.shape[0]:
                 raise IndexError, "index out of bounds"
@@ -177,13 +197,10 @@

     def __setitem__(self, key, value):
         try:
-            assert len(key) == 2
-        except (AssertionError, TypeError):
-            raise TypeError, "index must be a pair of integers, slices, or" \
-                    " sequences"
-        i, j = key
+            i, j = key
+        except (ValueError, TypeError):
+            raise TypeError, "index must be a pair of integers or slices"

-
         # First deal with the case where both i and j are integers
         if isintlike(i) and isintlike(j):
             if i < 0:
@@ -193,20 +210,14 @@

             if i < 0 or i >= self.shape[0] or j < 0 or j >= self.shape[1]:
                 raise IndexError, "index out of bounds"
-            if isintlike(value) and value == 0:
-                if key in self.keys():  # get rid of it something already there
-                    del self[key]
+
+            if isscalar(value):
+                if value==0:
+                    del self[(i,j)]
+                else:
+                    dict.__setitem__(self, (i,j), self.dtype.type(value))
             else:
-                # Ensure value is a single element, not a sequence
-                if isinstance(value, float) or isintlike(value) or \
-                        isinstance(value, complex):
-                    dict.__setitem__(self, (i,j), self.dtype.type(value))
-                    newrows = max(self.shape[0], int(key[0])+1)
-                    newcols = max(self.shape[1], int(key[1])+1)
-                    self.shape = (newrows, newcols)
-                else:
-                    raise TypeError, "cannot set matrix element to non-scalar"
-            return                 # done
+                raise TypeError, "cannot set matrix element to a non-scalar"
         else:
             # Either i or j is a slice, sequence, or invalid.  If i is a slice
             # or sequence, unfold it first and call __setitem__ recursively.
---

Thanks,
James


From mxhf at gmx.net  Sun Oct 19 05:37:29 2008
From: mxhf at gmx.net (Maximilian Fabricius)
Date: Sun, 19 Oct 2008 11:37:29 +0200
Subject: [SciPy-dev] parameter control in scipy.optimize.leastsq
Message-ID: <88b277e60810190237k15f27343xeb27793ea13d856b@mail.gmail.com>

Hello,

I have been using scipy for about one and a half years now and have
been extremely pleased.

In this time I have made extensive use of scipy.optimize.leastsq.
While I am generally fairly happy
with its performance, I think it does lacks one important feature
which I am used to from similar
routines in other languages. (Please excuse if I have just missed that
feature so far.)

I would like to be able to control which parameters actually are
fitted. For example I would like to be able to
do

leastsq( redsiduals, p0, args=(y), fit=([True,True,False,True]) )

where the parameter "fit" would tell leastsq to leave the parameter
p0[2] untouched while fitting the others.

Right now I have to modify my source code to change the parameters
which are actually fit.

I'd be happy to participate and start getting my hands dirty if
someone would point out where to start.

Regards,

Maximilian


From robince at gmail.com  Sun Oct 19 06:06:32 2008
From: robince at gmail.com (Robin)
Date: Sun, 19 Oct 2008 11:06:32 +0100
Subject: [SciPy-dev] parameter control in scipy.optimize.leastsq
In-Reply-To: <88b277e60810190237k15f27343xeb27793ea13d856b@mail.gmail.com>
References: <88b277e60810190237k15f27343xeb27793ea13d856b@mail.gmail.com>
Message-ID: <b14b627d0810190306r3e0d2dd8r3c3b04fde5684b1b@mail.gmail.com>

On Sun, Oct 19, 2008 at 10:37 AM, Maximilian Fabricius <mxhf at gmx.net> wrote:
> Hello,
>
> I have been using scipy for about one and a half years now and have
> been extremely pleased.
>
> In this time I have made extensive use of scipy.optimize.leastsq.
> While I am generally fairly happy
> with its performance, I think it does lacks one important feature
> which I am used to from similar
> routines in other languages. (Please excuse if I have just missed that
> feature so far.)
>
> I would like to be able to control which parameters actually are
> fitted. For example I would like to be able to
> do
>
> leastsq( redsiduals, p0, args=(y), fit=([True,True,False,True]) )
>
> where the parameter "fit" would tell leastsq to leave the parameter
> p0[2] untouched while fitting the others.

I think you can do this by 'wrapping' your existing function:

Something like:

fit_function = lambda x, fixed:  residuals(x[0],x[1],fixed[0],x[2])

Then you can call least squared with the fixed argument specified
leastsq( fit_function, p0, args=(y,fixed))

You can either have the fixed parameters passed as a argument to the
lambda, or have them defined where the lambda is run (but I find it a
bit confusiong becuase the scoping gets funny). I'm sure you could do
something to get the [True, False] type notation you want - but since
you will have to specify the fixed values as well.

Robin


From mxhf at gmx.net  Sun Oct 19 06:28:54 2008
From: mxhf at gmx.net (Maximilian Fabricius)
Date: Sun, 19 Oct 2008 12:28:54 +0200
Subject: [SciPy-dev] parameter control in scipy.optimize.leastsq
In-Reply-To: <b14b627d0810190306r3e0d2dd8r3c3b04fde5684b1b@mail.gmail.com>
References: <88b277e60810190237k15f27343xeb27793ea13d856b@mail.gmail.com>
	<b14b627d0810190306r3e0d2dd8r3c3b04fde5684b1b@mail.gmail.com>
Message-ID: <88b277e60810190328l3731c71ds91b8030ca9547ba1@mail.gmail.com>

On Sun, Oct 19, 2008 at 12:06 PM, Robin <robince at gmail.com> wrote:
> On Sun, Oct 19, 2008 at 10:37 AM, Maximilian Fabricius <mxhf at gmx.net> wrote:
>> Hello,
>>
>> I have been using scipy for about one and a half years now and have
>> been extremely pleased.
>>
>> In this time I have made extensive use of scipy.optimize.leastsq.
>> While I am generally fairly happy
>> with its performance, I think it does lacks one important feature
>> which I am used to from similar
>> routines in other languages. (Please excuse if I have just missed that
>> feature so far.)
>>
>> I would like to be able to control which parameters actually are
>> fitted. For example I would like to be able to
>> do
>>
>> leastsq( redsiduals, p0, args=(y), fit=([True,True,False,True]) )
>>
>> where the parameter "fit" would tell leastsq to leave the parameter
>> p0[2] untouched while fitting the others.
>
> I think you can do this by 'wrapping' your existing function:
>
> Something like:
>
> fit_function = lambda x, fixed:  residuals(x[0],x[1],fixed[0],x[2])
>
> Then you can call least squared with the fixed argument specified
> leastsq( fit_function, p0, args=(y,fixed))
>
> You can either have the fixed parameters passed as a argument to the
> lambda, or have them defined where the lambda is run (but I find it a
> bit confusiong becuase the scoping gets funny). I'm sure you could do
> something to get the [True, False] type notation you want - but since
> you will have to specify the fixed values as well.
>
> Robin
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>


Robin,

thank you for the quick reply.
Indeed, but this would still involve to swap parameters around between
fixed and non-fixed and
changing the lambda function manually rather then just switching the
fissint on and off
with a single variable.

But no doubt, a wrapper can be done.

Thanks!

Maximilian


From robince at gmail.com  Sun Oct 19 06:52:24 2008
From: robince at gmail.com (Robin)
Date: Sun, 19 Oct 2008 11:52:24 +0100
Subject: [SciPy-dev] parameter control in scipy.optimize.leastsq
In-Reply-To: <88b277e60810190328l3731c71ds91b8030ca9547ba1@mail.gmail.com>
References: <88b277e60810190237k15f27343xeb27793ea13d856b@mail.gmail.com>
	<b14b627d0810190306r3e0d2dd8r3c3b04fde5684b1b@mail.gmail.com>
	<88b277e60810190328l3731c71ds91b8030ca9547ba1@mail.gmail.com>
Message-ID: <b14b627d0810190352j2700458fl61f7633a25db2d6f@mail.gmail.com>

On Sun, Oct 19, 2008 at 11:28 AM, Maximilian Fabricius <mxhf at gmx.net> wrote:
> Robin,
>
> thank you for the quick reply.
> Indeed, but this would still involve to swap parameters around between
> fixed and non-fixed and
> changing the lambda function manually rather then just switching the
> fissint on and off
> with a single variable.
>
> But no doubt, a wrapper can be done.

I couldn't think of a way to squeeze it into a lamdba single
statement, but something like this would do:

def fit_function(x, fit, y):
    p0 = zeros(fit.size)
    p0[fit] = x
    p0[~fit] = y

where fit is a boolean array, x is the ones to optimised and y are the
fixed values.
You have the extra overhead of the p0 array allocation for every
iteration of the function - but perhaps that is worth it for the
convenience.

Robin


From mxhf at gmx.net  Sun Oct 19 11:28:14 2008
From: mxhf at gmx.net (Maximilian Fabricius)
Date: Sun, 19 Oct 2008 17:28:14 +0200
Subject: [SciPy-dev] parameter control in scipy.optimize.leastsq
In-Reply-To: <b14b627d0810190352j2700458fl61f7633a25db2d6f@mail.gmail.com>
References: <88b277e60810190237k15f27343xeb27793ea13d856b@mail.gmail.com>
	<b14b627d0810190306r3e0d2dd8r3c3b04fde5684b1b@mail.gmail.com>
	<88b277e60810190328l3731c71ds91b8030ca9547ba1@mail.gmail.com>
	<b14b627d0810190352j2700458fl61f7633a25db2d6f@mail.gmail.com>
Message-ID: <88b277e60810190828w43418c02i8969c31b597f4ef8@mail.gmail.com>

On Sun, Oct 19, 2008 at 12:52 PM, Robin <robince at gmail.com> wrote:
> On Sun, Oct 19, 2008 at 11:28 AM, Maximilian Fabricius <mxhf at gmx.net> wrote:
>> Robin,
>>
>> thank you for the quick reply.
>> Indeed, but this would still involve to swap parameters around between
>> fixed and non-fixed and
>> changing the lambda function manually rather then just switching the
>> fissint on and off
>> with a single variable.
>>
>> But no doubt, a wrapper can be done.
>
> I couldn't think of a way to squeeze it into a lamdba single
> statement, but something like this would do:
>
> def fit_function(x, fit, y):
>    p0 = zeros(fit.size)
>    p0[fit] = x
>    p0[~fit] = y
>
> where fit is a boolean array, x is the ones to optimised and y are the
> fixed values.
> You have the extra overhead of the p0 array allocation for every
> iteration of the function - but perhaps that is worth it for the
> convenience.
>
> Robin
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>


Robin,

great, nice idea! As this may be of use to others, my solution now is:


def fit_function(p_fit, fit, p_fixed, ... ):
        # assemble p_fit and p_fixed into a single array
        # while restoring the correct order of the original p0 array.
        p = zeros(fit.size)
        p[ where(fit == True) ] = p_fit
        p[ where(fit == False) ] = p_fixed
        # calls the original residuals routine
        return residuals(p, ... )

main():
        # best guesses
        p0 = zeros(10)
        p0[0] = vrot_d
        p0[1] = vsig_d
        ....

        # choose which parameters to fit
        fit = zeros(len(p0))
        fit[0:7] = True
        fit[8] = False
        fit[9] = False

        # store parameters which should be fit in p_fit
        #  all the others in p_fixed
        p_fit = p0[ where(fit == True) ]
        p_fixed = p0[ where(fit == False) ]
        plsq = leastsq(fit_function, p_fit, args=(fit, p_fixed,  ...) )

Thanks!

Cheers,

Maximilian


From robince at gmail.com  Sun Oct 19 11:40:17 2008
From: robince at gmail.com (Robin)
Date: Sun, 19 Oct 2008 16:40:17 +0100
Subject: [SciPy-dev] parameter control in scipy.optimize.leastsq
In-Reply-To: <88b277e60810190828w43418c02i8969c31b597f4ef8@mail.gmail.com>
References: <88b277e60810190237k15f27343xeb27793ea13d856b@mail.gmail.com>
	<b14b627d0810190306r3e0d2dd8r3c3b04fde5684b1b@mail.gmail.com>
	<88b277e60810190328l3731c71ds91b8030ca9547ba1@mail.gmail.com>
	<b14b627d0810190352j2700458fl61f7633a25db2d6f@mail.gmail.com>
	<88b277e60810190828w43418c02i8969c31b597f4ef8@mail.gmail.com>
Message-ID: <b14b627d0810190840k1778aca3w16d7f7ceaed7abcf@mail.gmail.com>

On Sun, Oct 19, 2008 at 4:28 PM, Maximilian Fabricius <mxhf at gmx.net> wrote:
> Robin,
>
> great, nice idea! As this may be of use to others, my solution now is:
>
>
> def fit_function(p_fit, fit, p_fixed, ... ):
>        # assemble p_fit and p_fixed into a single array
>        # while restoring the correct order of the original p0 array.
>        p = zeros(fit.size)
>        p[ where(fit == True) ] = p_fit
>        p[ where(fit == False) ] = p_fixed
>        # calls the original residuals routine
>        return residuals(p, ... )
>
> main():
>        # best guesses
>        p0 = zeros(10)
>        p0[0] = vrot_d
>        p0[1] = vsig_d
>        ....
>
>        # choose which parameters to fit
>        fit = zeros(len(p0))
>        fit[0:7] = True
>        fit[8] = False
>        fit[9] = False
>
>        # store parameters which should be fit in p_fit
>        #  all the others in p_fixed
>        p_fit = p0[ where(fit == True) ]
>        p_fixed = p0[ where(fit == False) ]
>        plsq = leastsq(fit_function, p_fit, args=(fit, p_fixed,  ...) )
>
> Thanks!

Just want to point out that if fit is a boolean array (ie fit =
array([True,False,False,True]) etc. then you can index with it
directly p0[fit] instead of p0[where(fit==True)] and p0[~fit] instead
of p0[where(fit==False)] which is likely to be a bit quicker. Doesn't
matter outside, but in the fit_function you probably want to save any
time you can since it will be run many times.

Robin


From benny.malengier at gmail.com  Sun Oct 19 13:44:04 2008
From: benny.malengier at gmail.com (Benny Malengier)
Date: Sun, 19 Oct 2008 19:44:04 +0200
Subject: [SciPy-dev] parameter control in scipy.optimize.leastsq
In-Reply-To: <88b277e60810190828w43418c02i8969c31b597f4ef8@mail.gmail.com>
References: <88b277e60810190237k15f27343xeb27793ea13d856b@mail.gmail.com>
	<b14b627d0810190306r3e0d2dd8r3c3b04fde5684b1b@mail.gmail.com>
	<88b277e60810190328l3731c71ds91b8030ca9547ba1@mail.gmail.com>
	<b14b627d0810190352j2700458fl61f7633a25db2d6f@mail.gmail.com>
	<88b277e60810190828w43418c02i8969c31b597f4ef8@mail.gmail.com>
Message-ID: <bcafe8290810191044y6f05d94cgf3802225be30ea22@mail.gmail.com>

To avoid array creation overhead, I use objects encapsulating everything, so


def fit_function(self, x, ... ):
       # x is a subset of all the parameters, self.tofit is bool array with
those arg to fit
       self.p[self.tofit ][:] = x[:]
       self.p[~self.tofit][:] = self.all_arg [~self.tofit][:]
       # calls the original residuals routine
       return residuals(self.p, ... )

Then in your loop, you can change the self.tofit array to change what you
fit.
Obviously, self.all_arg default to something in the __init__() method and
self.p is created once there, and things can be changed.
If things are slow, the line
     self.p[~self.tofit][:] = self.all_arg [~self.tofit][:]
can be removed and put into it's own method, only to be called if self.tofit
changes.

Benny

2008/10/19 Maximilian Fabricius <mxhf at gmx.net>

> On Sun, Oct 19, 2008 at 12:52 PM, Robin <robince at gmail.com> wrote:
> > On Sun, Oct 19, 2008 at 11:28 AM, Maximilian Fabricius <mxhf at gmx.net>
> wrote:
> >> Robin,
> >>
> >> thank you for the quick reply.
> >> Indeed, but this would still involve to swap parameters around between
> >> fixed and non-fixed and
> >> changing the lambda function manually rather then just switching the
> >> fissint on and off
> >> with a single variable.
> >>
> >> But no doubt, a wrapper can be done.
> >
> > I couldn't think of a way to squeeze it into a lamdba single
> > statement, but something like this would do:
> >
> > def fit_function(x, fit, y):
> >    p0 = zeros(fit.size)
> >    p0[fit] = x
> >    p0[~fit] = y
> >
> > where fit is a boolean array, x is the ones to optimised and y are the
> > fixed values.
> > You have the extra overhead of the p0 array allocation for every
> > iteration of the function - but perhaps that is worth it for the
> > convenience.
> >
> > Robin
> > _______________________________________________
> > Scipy-dev mailing list
> > Scipy-dev at scipy.org
> > http://projects.scipy.org/mailman/listinfo/scipy-dev
> >
>
>
> Robin,
>
> great, nice idea! As this may be of use to others, my solution now is:
>
>
> def fit_function(p_fit, fit, p_fixed, ... ):
>        # assemble p_fit and p_fixed into a single array
>        # while restoring the correct order of the original p0 array.
>        p = zeros(fit.size)
>        p[ where(fit == True) ] = p_fit
>        p[ where(fit == False) ] = p_fixed
>        # calls the original residuals routine
>        return residuals(p, ... )
>
> main():
>        # best guesses
>        p0 = zeros(10)
>        p0[0] = vrot_d
>        p0[1] = vsig_d
>        ....
>
>        # choose which parameters to fit
>        fit = zeros(len(p0))
>        fit[0:7] = True
>        fit[8] = False
>        fit[9] = False
>
>        # store parameters which should be fit in p_fit
>        #  all the others in p_fixed
>        p_fit = p0[ where(fit == True) ]
>        p_fixed = p0[ where(fit == False) ]
>        plsq = leastsq(fit_function, p_fit, args=(fit, p_fixed,  ...) )
>
> Thanks!
>
> Cheers,
>
> Maximilian
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081019/0dcb9872/attachment.html>

From wnbell at gmail.com  Sun Oct 19 23:18:45 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Sun, 19 Oct 2008 23:18:45 -0400
Subject: [SciPy-dev] scipy.sparse.dok_matrix changes
In-Reply-To: <2b1c8c4f0810180856q2ef0d97cqf5defe1cd1aae173@mail.gmail.com>
References: <2b1c8c4f0810180856q2ef0d97cqf5defe1cd1aae173@mail.gmail.com>
Message-ID: <d05265cb0810192018m25a5eb6du59a13788f4cb8b58@mail.gmail.com>

On Sat, Oct 18, 2008 at 11:56 AM, James Philbin <philbinj at gmail.com> wrote:
>
> I've recently been using dok_matrix a fair bit and have made a few
> improvements, mainly:
> i. Improved the docstring.
> ii. Fixed the constructor (this fixes #755).
> iii. Reduced the codepath for the common case of setting an individual
> element (~2x speedup).
>

Thanks James.  I've merged your patch in r4802:
http://projects.scipy.org/scipy/scipy/changeset/4802

The dok_matrix class is a neglected part of scipy.sparse, so your
contribution is most appreciated.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From pgmdevlist at gmail.com  Mon Oct 20 01:56:04 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 20 Oct 2008 01:56:04 -0400
Subject: [SciPy-dev] Timeseries Unusual Behaviour
In-Reply-To: <loom.20081014T170518-386@post.gmane.org>
References: <loom.20081014T170518-386@post.gmane.org>
Message-ID: <200810200156.04924.pgmdevlist@gmail.com>

David,
Sorry for the delayed answer.

* When you create a DateArray from a series of dates using `date_array` (the 
recommended way), the dates will automatically be sorted in chronological 
order, but the original order stored temporarily into a private attribute.

* When you create a time series using `time_series`, the same thing happens:
1. a DateArray is created from the list of dates with `date_array`
2. it is sorted chronologically and the initial order is stored temporarily
3. the values are then sorted chronologically,
4. the attribute storing the initial order of the dates reset to None.

The reason behind the automatic sorting is that in a huge majority of cases, 
one works with chronological series. 

Now, let's go through your example
>>> dates = (datetime.datetime(2011, 5, 1, 0, 0),
... datetime.datetime(2011, 8, 1, 0, 0),
... datetime.datetime(2011, 6, 1, 0, 0),
... datetime.datetime(2011, 9, 1, 0, 0),
... datetime.datetime(2011, 7, 1, 0, 0))
>>> values = [1, 4, 2, 5, 3]
>>> d1 = ts.date_array(dates)

d1 is sorted chronologically by default.

>>> zip(d1, values)
[(<U : 01-May-2011>, 1),
 (<U : 01-Jun-2011>, 4),
 (<U : 01-Jul-2011>, 2),
 (<U : 01-Aug-2011>, 5),
 (<U : 01-Sep-2011>, 3)]

Here, we're mixing d1 (sorted) and values (unsorted).

>>> series = ts.time_series(values,dates=d1)
timeseries([1 2 3 4 5],
           dates = [01-May-2011 01-Jun-2011 01-Jul-2011 01-Aug-2011 
01-Sep-2011],
           freq  = U)

d1 is sorted chronologically, values is sorted according to the initial order 
of d1, the private flag of d1 is reset to show it has been sorted.

>>> series = ts.time_series(values,dates=d1)
timeseries([1 4 2 5 3],
           dates = [01-May-2011 01-Jun-2011 01-Jul-2011 01-Aug-2011 
01-Sep-2011],
           freq  = U)

d1 had already been sorted and its flag reset, the values are *not* sorted.

I agree that the last line is so confusing that it should be considered a bug. 
I'll try to find some workaround.
Meanwhile, a couple of advices:

* As time_series calls date_array anyway, you don't gain a lot by creating a 
DateArray from your dates in the first place. On the contrary, it can lead to 
the surprises we just discuss: just use the dates instead.
* Take the habit of always specifying a frequency, it will save you some 
headaches in the long run when converting between frequencies


From mxhf at gmx.net  Mon Oct 20 03:09:40 2008
From: mxhf at gmx.net (Maximilian Fabricius)
Date: Mon, 20 Oct 2008 09:09:40 +0200
Subject: [SciPy-dev] parameter control in scipy.optimize.leastsq
In-Reply-To: <b14b627d0810190840k1778aca3w16d7f7ceaed7abcf@mail.gmail.com>
References: <88b277e60810190237k15f27343xeb27793ea13d856b@mail.gmail.com>
	<b14b627d0810190306r3e0d2dd8r3c3b04fde5684b1b@mail.gmail.com>
	<88b277e60810190328l3731c71ds91b8030ca9547ba1@mail.gmail.com>
	<b14b627d0810190352j2700458fl61f7633a25db2d6f@mail.gmail.com>
	<88b277e60810190828w43418c02i8969c31b597f4ef8@mail.gmail.com>
	<b14b627d0810190840k1778aca3w16d7f7ceaed7abcf@mail.gmail.com>
Message-ID: <88b277e60810200009w4d3efb25t7ca4f13fdeb5b18e@mail.gmail.com>

On Sun, Oct 19, 2008 at 5:40 PM, Robin <robince at gmail.com> wrote:
> On Sun, Oct 19, 2008 at 4:28 PM, Maximilian Fabricius <mxhf at gmx.net> wrote:
>> Robin,
>>
>> great, nice idea! As this may be of use to others, my solution now is:
>>
>>
>> def fit_function(p_fit, fit, p_fixed, ... ):
>>        # assemble p_fit and p_fixed into a single array
>>        # while restoring the correct order of the original p0 array.
>>        p = zeros(fit.size)
>>        p[ where(fit == True) ] = p_fit
>>        p[ where(fit == False) ] = p_fixed
>>        # calls the original residuals routine
>>        return residuals(p, ... )
>>
>> main():
>>        # best guesses
>>        p0 = zeros(10)
>>        p0[0] = vrot_d
>>        p0[1] = vsig_d
>>        ....
>>
>>        # choose which parameters to fit
>>        fit = zeros(len(p0))
>>        fit[0:7] = True
>>        fit[8] = False
>>        fit[9] = False
>>
>>        # store parameters which should be fit in p_fit
>>        #  all the others in p_fixed
>>        p_fit = p0[ where(fit == True) ]
>>        p_fixed = p0[ where(fit == False) ]
>>        plsq = leastsq(fit_function, p_fit, args=(fit, p_fixed,  ...) )
>>
>> Thanks!
>
> Just want to point out that if fit is a boolean array (ie fit =
> array([True,False,False,True]) etc. then you can index with it
> directly p0[fit] instead of p0[where(fit==True)] and p0[~fit] instead
> of p0[where(fit==False)] which is likely to be a bit quicker. Doesn't
> matter outside, but in the fit_function you probably want to save any
> time you can since it will be run many times.
>
> Robin

Robin,

thank you. Indeed, I should know that. Python sometime is
overwhelmingly convenient.

Maximilian


From simon.caschili at gmail.com  Mon Oct 20 10:44:38 2008
From: simon.caschili at gmail.com (Simone Caschili)
Date: Mon, 20 Oct 2008 16:44:38 +0200
Subject: [SciPy-dev] Genetic Algorithm documentation
Message-ID: <256b49550810200744n40cfd26eq8631228f0c1b8a64@mail.gmail.com>

Hello,

I am thinking to use scipy to implement a genetic algorithm project,
but I have found any documentation about it.

Could you suggest me where I can find it?

Thank you very much.

Simone


From philbinj at gmail.com  Mon Oct 20 11:07:01 2008
From: philbinj at gmail.com (James Philbin)
Date: Mon, 20 Oct 2008 16:07:01 +0100
Subject: [SciPy-dev] scipy.sparse.dok_matrix changes
In-Reply-To: <d05265cb0810192018m25a5eb6du59a13788f4cb8b58@mail.gmail.com>
References: <2b1c8c4f0810180856q2ef0d97cqf5defe1cd1aae173@mail.gmail.com>
	<d05265cb0810192018m25a5eb6du59a13788f4cb8b58@mail.gmail.com>
Message-ID: <2b1c8c4f0810200807s73bc5e3cm1a122a1879401c2c@mail.gmail.com>

Hi,

> Thanks James.  I've merged your patch in r4802:
> http://projects.scipy.org/scipy/scipy/changeset/4802
Good stuff. I'd like to make some more changes/cleanups, but there's a
few things i'm not sure about.

Firstly, I'd like to remove the functions 'conjtranspose', 'take' +
'split'. All of them seem non-standard and can be expressed in other
ways. If functions are removed, should they be deprecated first? I'm
not sure what the policy is on a pre 1.0 release.

Secondly, using __getitem__ with slices currently requires an O(nnz)
crawl over all the values. Should we support this usage? My preference
would be to only allow elementwise access, though for __setitem__
using slices is fine.

James


From wnbell at gmail.com  Mon Oct 20 12:58:48 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Mon, 20 Oct 2008 12:58:48 -0400
Subject: [SciPy-dev] scipy.sparse.dok_matrix changes
In-Reply-To: <2b1c8c4f0810200807s73bc5e3cm1a122a1879401c2c@mail.gmail.com>
References: <2b1c8c4f0810180856q2ef0d97cqf5defe1cd1aae173@mail.gmail.com>
	<d05265cb0810192018m25a5eb6du59a13788f4cb8b58@mail.gmail.com>
	<2b1c8c4f0810200807s73bc5e3cm1a122a1879401c2c@mail.gmail.com>
Message-ID: <d05265cb0810200958x5d245c6ahf979d47454b035ba@mail.gmail.com>

On Mon, Oct 20, 2008 at 11:07 AM, James Philbin <philbinj at gmail.com> wrote:
> Good stuff. I'd like to make some more changes/cleanups, but there's a
> few things i'm not sure about.
>
> Firstly, I'd like to remove the functions 'conjtranspose', 'take' +
> 'split'. All of them seem non-standard and can be expressed in other
> ways. If functions are removed, should they be deprecated first? I'm
> not sure what the policy is on a pre 1.0 release.

There's no fixed policy, but I have tended to deprecate for one minor
release before removing functionality.  For instance:
http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/base.py#L579

> Secondly, using __getitem__ with slices currently requires an O(nnz)
> crawl over all the values. Should we support this usage? My preference
> would be to only allow elementwise access, though for __setitem__
> using slices is fine.

Again, no fixed policy.  For costly operations on CSR/CSC matrices,
we've used the SparseEfficiencyWarning:
http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/compressed.py#L621

Ideally all get/set operations would be symmetric.  However, we
haven't maintained this property elsewhere in scipy.sparse, so users
will not expect it to be so.

Since you're the defacto dok_matrix maintainer, I'd say it's
ultimately up to you :)

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From nwagner at iam.uni-stuttgart.de  Mon Oct 20 12:59:44 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 20 Oct 2008 18:59:44 +0200
Subject: [SciPy-dev] Genetic Algorithm documentation
In-Reply-To: <256b49550810200744n40cfd26eq8631228f0c1b8a64@mail.gmail.com>
References: <256b49550810200744n40cfd26eq8631228f0c1b8a64@mail.gmail.com>
Message-ID: <web-113104098@uni-stuttgart.de>

On Mon, 20 Oct 2008 16:44:38 +0200
  "Simone Caschili" <simon.caschili at gmail.com> wrote:
> Hello,
> 
> I am thinking to use scipy to implement a genetic 
>algorithm project,
> but I have found any documentation about it.
> 
> Could you suggest me where I can find it?
> 

http://scipy.org/scipy/scikits/browser/trunk/learn/scikits/learn/machine/ga

  
Nils


From simon.caschili at gmail.com  Mon Oct 20 13:14:20 2008
From: simon.caschili at gmail.com (Simone Caschili)
Date: Mon, 20 Oct 2008 19:14:20 +0200
Subject: [SciPy-dev] Genetic Algorithm documentation
Message-ID: <256b49550810201014j707d498ct36d54bdbc67f3be8@mail.gmail.com>

Thanks Nil for your answer.
I had already found that documentation but I would like to get a
tutorial or an official documentation about scipy Genetic Algorithm
implementation.

Could you recommend any official documentation/tutorial ?

Simone


>
> On Mon, 20 Oct 2008 16:44:38 +0200
>  "Simone Caschili" <simon.caschili at gmail.com> wrote:
>> Hello,
>>
>> I am thinking to use scipy to implement a genetic
>>algorithm project,
>> but I have found any documentation about it.
>>
>> Could you suggest me where I can find it?
>>
>
> http://scipy.org/scipy/scikits/browser/trunk/learn/scikits/learn/machine/ga
>
>
> Nils


From travis at enthought.com  Tue Oct 21 17:53:51 2008
From: travis at enthought.com (Travis Vaught)
Date: Tue, 21 Oct 2008 16:53:51 -0500
Subject: [SciPy-dev] ANN: Enthought Python Distribution - New Release
Message-ID: <02DFC911-B86B-4F92-A83F-1FB7AADAB98B@enthought.com>

Greetings,

Enthought, Inc. is very pleased to announce the newest release of the  
Enthought Python Distribution (EPD) Py2.5 v4.0.30002:

http://www.enthought.com/epd

This release contains updates to many of EPD's packages, including  
NumPy, IPython, matplotlib, VTK, etc.  This is also the first release  
to include a 3.x version of the Enthought Tool Suite (http://code.enthought.com/ 
).

The release notes for this release, including the list of included  
packages, may be found here:
https://svn.enthought.com/epd/wiki/Python2.5.2/4.0.300/GA

Many thanks to the EPD team for putting this release together, and to  
the community of folks who have provided all of the valuable tools  
bundled here.

Best Regards,

Travis


---------
About EPD
---------
The Enthought Python Distribution (EPD) is a "kitchen-sink-included"  
distribution of the Python? Programming Language, including over 80  
additional tools and libraries. The EPD bundle includes NumPy, SciPy,  
IPython, 2D and 3D visualization, database adapters, and a lot of  
other tools right out of the box.

http://www.enthought.com/products/epd.php

It is currently available as an easy, single-click installer for  
Windows XP (x86), Mac OS X (a universal binary for Intel 10.4 and  
above) and RedHat EL3 (x86 and amd64).

EPD is free for 30-day trial use and for use in degree-granting  
academic institutions.  An annual Subscription and installation  
support are available for commercial use (http://www.enthought.com/products/epddownload.php 
) including an Enterprise Subscription with support for particular  
deployment environments (http://www.enthought.com/products/enterprise.php 
).



From david at ar.media.kyoto-u.ac.jp  Wed Oct 22 02:28:52 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 22 Oct 2008 15:28:52 +0900
Subject: [SciPy-dev] Should scipy 0.7 be compatible with 1.1 serie at all ?
Message-ID: <48FEC824.5060204@ar.media.kyoto-u.ac.jp>

Hi,

    When building scipy against numpy 1.2.0, there are quite a few
deprecation warnings related to recent changes in the C API. Should we
update the code for 1.2.0, or shall we keep backward compatibility with
1.1.x serie ?

cheers,

David


From alan.mcintyre at gmail.com  Wed Oct 22 03:01:31 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 22 Oct 2008 00:01:31 -0700
Subject: [SciPy-dev] Should scipy 0.7 be compatible with 1.1 serie at
	all ?
In-Reply-To: <48FEC824.5060204@ar.media.kyoto-u.ac.jp>
References: <48FEC824.5060204@ar.media.kyoto-u.ac.jp>
Message-ID: <1d36917a0810220001g720535e3t9a79ce2668562f24@mail.gmail.com>

On Tue, Oct 21, 2008 at 11:28 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
>    When building scipy against numpy 1.2.0, there are quite a few
> deprecation warnings related to recent changes in the C API. Should we
> update the code for 1.2.0, or shall we keep backward compatibility with
> 1.1.x serie ?

There's a patch at http://scipy.org/scipy/scipy/ticket/723 to fix some
of the PyArray_* warnings.  I don't have the time or knowledge of the
Fortran side of Sci/NumPy to figure out how to avoid using the
temporary variables that Charles doesn't like, though.  ;)


From robert.kern at gmail.com  Wed Oct 22 09:12:42 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 22 Oct 2008 08:12:42 -0500
Subject: [SciPy-dev] Should scipy 0.7 be compatible with 1.1 serie at
	all ?
In-Reply-To: <48FEC824.5060204@ar.media.kyoto-u.ac.jp>
References: <48FEC824.5060204@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730810220612x52953cf4q2575c5cf496d0f8a@mail.gmail.com>

On Wed, Oct 22, 2008 at 01:28, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
>    When building scipy against numpy 1.2.0, there are quite a few
> deprecation warnings related to recent changes in the C API. Should we
> update the code for 1.2.0, or shall we keep backward compatibility with
> 1.1.x serie ?

There is no backwards compatibility with 1.1.x to keep. scipy 0.7-svn
has required numpy 1.2-svn for some time.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Wed Oct 22 09:20:44 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 22 Oct 2008 22:20:44 +0900
Subject: [SciPy-dev] Should scipy 0.7 be compatible with 1.1 serie at
 all ?
In-Reply-To: <3d375d730810220612x52953cf4q2575c5cf496d0f8a@mail.gmail.com>
References: <48FEC824.5060204@ar.media.kyoto-u.ac.jp>
	<3d375d730810220612x52953cf4q2575c5cf496d0f8a@mail.gmail.com>
Message-ID: <48FF28AC.6040808@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
> There is no backwards compatibility with 1.1.x to keep. scipy 0.7-svn
> has required numpy 1.2-svn for some time.
>   

Great, thanks for the information. Would it be ok to add a guard against
using numpy < 1.2  when importing scipy in the trunk ?

David



From robert.kern at gmail.com  Wed Oct 22 09:37:32 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 22 Oct 2008 08:37:32 -0500
Subject: [SciPy-dev] Should scipy 0.7 be compatible with 1.1 serie at
	all ?
In-Reply-To: <48FF28AC.6040808@ar.media.kyoto-u.ac.jp>
References: <48FEC824.5060204@ar.media.kyoto-u.ac.jp>
	<3d375d730810220612x52953cf4q2575c5cf496d0f8a@mail.gmail.com>
	<48FF28AC.6040808@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730810220637j3428750ala594520b0eb30717@mail.gmail.com>

On Wed, Oct 22, 2008 at 08:20, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Robert Kern wrote:
>>
>> There is no backwards compatibility with 1.1.x to keep. scipy 0.7-svn
>> has required numpy 1.2-svn for some time.
>
> Great, thanks for the information. Would it be ok to add a guard against
> using numpy < 1.2  when importing scipy in the trunk ?

You can issue a warning, but I would prefer that we not absolutely prevent it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Wed Oct 22 14:52:20 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 22 Oct 2008 14:52:20 -0400
Subject: [SciPy-dev] Timeseries Unusual Behaviour
In-Reply-To: <loom.20081014T170518-386@post.gmane.org>
References: <loom.20081014T170518-386@post.gmane.org>
Message-ID: <200810221452.20605.pgmdevlist@gmail.com>

Dave,
The unusal behavior you documented (trying to create 2 series with the same 
values and same DateArray as dates giving different answers depending on the 
order) should now be fixed on SVN.
Thanks again for reporting.


From charlesr.harris at gmail.com  Thu Oct 23 01:12:55 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 22 Oct 2008 23:12:55 -0600
Subject: [SciPy-dev] Should scipy 0.7 be compatible with 1.1 serie at
	all ?
In-Reply-To: <1d36917a0810220001g720535e3t9a79ce2668562f24@mail.gmail.com>
References: <48FEC824.5060204@ar.media.kyoto-u.ac.jp>
	<1d36917a0810220001g720535e3t9a79ce2668562f24@mail.gmail.com>
Message-ID: <e06186140810222212ye7d241dq37d9948672b749d3@mail.gmail.com>

On Wed, Oct 22, 2008 at 1:01 AM, Alan McIntyre <alan.mcintyre at gmail.com>wrote:

> On Tue, Oct 21, 2008 at 11:28 PM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
> >    When building scipy against numpy 1.2.0, there are quite a few
> > deprecation warnings related to recent changes in the C API. Should we
> > update the code for 1.2.0, or shall we keep backward compatibility with
> > 1.1.x serie ?
>
> There's a patch at http://scipy.org/scipy/scipy/ticket/723 to fix some
> of the PyArray_* warnings.  I don't have the time or knowledge of the
> Fortran side of Sci/NumPy to figure out how to avoid using the
> temporary variables that Charles doesn't like, though.  ;)


You could put a fixme comment on them. Mostly I don't want the workarounds
to hide the need for later cleanups.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081022/dfc84191/attachment.html>

From alan.mcintyre at gmail.com  Thu Oct 23 02:05:03 2008
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Wed, 22 Oct 2008 23:05:03 -0700
Subject: [SciPy-dev] Should scipy 0.7 be compatible with 1.1 serie at
	all ?
In-Reply-To: <e06186140810222212ye7d241dq37d9948672b749d3@mail.gmail.com>
References: <48FEC824.5060204@ar.media.kyoto-u.ac.jp>
	<1d36917a0810220001g720535e3t9a79ce2668562f24@mail.gmail.com>
	<e06186140810222212ye7d241dq37d9948672b749d3@mail.gmail.com>
Message-ID: <1d36917a0810222305h6d4c0c18w928cfd4caf5f4408@mail.gmail.com>

On Wed, Oct 22, 2008 at 10:12 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>> There's a patch at http://scipy.org/scipy/scipy/ticket/723 to fix some
>> of the PyArray_* warnings.  I don't have the time or knowledge of the
>> Fortran side of Sci/NumPy to figure out how to avoid using the
>> temporary variables that Charles doesn't like, though.  ;)
>
> You could put a fixme comment on them. Mostly I don't want the workarounds
> to hide the need for later cleanups.

Sure thing; is there a standard fixme comment for SciPy/NumPy?  I will
probably get around to doing this over the weekend.


From dave.hirschfeld at gmail.com  Thu Oct 23 09:12:39 2008
From: dave.hirschfeld at gmail.com (Dave)
Date: Thu, 23 Oct 2008 13:12:39 +0000 (UTC)
Subject: [SciPy-dev] Timeseries Unusual Behaviour
References: <loom.20081014T170518-386@post.gmane.org>
	<200810221452.20605.pgmdevlist@gmail.com>
Message-ID: <loom.20081023T130519-312@post.gmane.org>

Pierre GM <pgmdevlist <at> gmail.com> writes:

> 
> Dave,
> The unusal behavior you documented (trying to create 2 series with the same 
> values and same DateArray as dates giving different answers depending on the 
> order) should now be fixed on SVN.
> Thanks again for reporting.
> 

Thanks for the explanation and the fix. I've updated from svn and noticed the 
autosort parameter which does make its behaviour more explicit :)

Cheers,
Dave




From opossumnano at gmail.com  Thu Oct 23 12:04:02 2008
From: opossumnano at gmail.com (Tiziano Zito)
Date: Thu, 23 Oct 2008 18:04:02 +0200
Subject: [SciPy-dev] integration of symeig in SciPy
Message-ID: <20081023160402.GA3443@localhost>

Dear SciPy devs, 

there has been some discussion about integrating symeig in SciPy.
symeig <http://mdp-toolkit.sourceforge.net/symeig.html> is a Python
wrapper for the LAPACK functions to solve the standard and
generalized eigenvalue problems for symmetric (hermitian) positive
definite matrices. It is superior to the scipy "linalg.eigh" wrapper
because it wraps *all* relevant LAPACK routines, enabling for
example to just return a subset of all egenvectors/eigenvalues,
which is a killer feature if you are dealing with huge-nonsparse
matrices. Some of these LAPACK routines are available in
scipy.lib.lapack.flapack and can be accessed through
scipy.lib.lapack.get_lapack_funcs . Some of them are still missing
in scipy, while symeig offers a unified interface to all relevant
LAPACK functions.  symeig has been dowloaded  more than 1300 times
since its first public appearance on sourceforge (under the
mdp-toolkit package) in 2004. It features a complete unit test and a
telling doc-string. I am one of the authors of symeig and have been
contacted by some scipy devs to discuss the integration of symeig in
scipy. I am more than willing to help doing this. The most difficult
part (the LGPL license) has been addressed already: I've re-issued
symeig under a BSD license. Next step would be to adapt symeig code
and tests to the scipy framework. Some questions to you:

- where would you like to have symeig? (I would suggest scipy.linalg)

- should we change the signature to match or resamble that of other
  eigenproblem-related functions in scipy?

- is there a responsible scipy dev for the linalg package I can bother
  with any questions I may have in the process?

- do I get svn commit rights or should I work on a local copy and
  send a patch?

let me know, 

tiziano




From robert.kern at gmail.com  Thu Oct 23 12:33:57 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 23 Oct 2008 11:33:57 -0500
Subject: [SciPy-dev] integration of symeig in SciPy
In-Reply-To: <20081023160402.GA3443@localhost>
References: <20081023160402.GA3443@localhost>
Message-ID: <3d375d730810230933w511046dfrba829f2434824211@mail.gmail.com>

On Thu, Oct 23, 2008 at 11:04, Tiziano Zito <opossumnano at gmail.com> wrote:
> Dear SciPy devs,
>
> there has been some discussion about integrating symeig in SciPy.
> symeig <http://mdp-toolkit.sourceforge.net/symeig.html> is a Python
> wrapper for the LAPACK functions to solve the standard and
> generalized eigenvalue problems for symmetric (hermitian) positive
> definite matrices. It is superior to the scipy "linalg.eigh" wrapper
> because it wraps *all* relevant LAPACK routines, enabling for
> example to just return a subset of all egenvectors/eigenvalues,
> which is a killer feature if you are dealing with huge-nonsparse
> matrices. Some of these LAPACK routines are available in
> scipy.lib.lapack.flapack and can be accessed through
> scipy.lib.lapack.get_lapack_funcs . Some of them are still missing
> in scipy, while symeig offers a unified interface to all relevant
> LAPACK functions.  symeig has been dowloaded  more than 1300 times
> since its first public appearance on sourceforge (under the
> mdp-toolkit package) in 2004. It features a complete unit test and a
> telling doc-string. I am one of the authors of symeig and have been
> contacted by some scipy devs to discuss the integration of symeig in
> scipy. I am more than willing to help doing this. The most difficult
> part (the LGPL license) has been addressed already: I've re-issued
> symeig under a BSD license. Next step would be to adapt symeig code
> and tests to the scipy framework. Some questions to you:
>
> - where would you like to have symeig? (I would suggest scipy.linalg)

Yup.

> - should we change the signature to match or resamble that of other
>  eigenproblem-related functions in scipy?

Probably.

> - is there a responsible scipy dev for the linalg package I can bother
>  with any questions I may have in the process?

You can bother the list.

> - do I get svn commit rights or should I work on a local copy and
>  send a patch?

We'll get you commit rights.

Thanks! I'm looking forward to seeing this in scipy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Fri Oct 24 03:54:59 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 24 Oct 2008 09:54:59 +0200
Subject: [SciPy-dev] [SciPy-user] SciPy Sprint Weekend: 1 and 2 November
In-Reply-To: <4C4FE880-8830-4153-83CF-7565E5EA4646@cs.toronto.edu>
References: <9457e7c80810231422m6cab2e75wbcb343472449312@mail.gmail.com>
	<4C4FE880-8830-4153-83CF-7565E5EA4646@cs.toronto.edu>
Message-ID: <9457e7c80810240054o3f00cc5fp7528a65af10cc2c7@mail.gmail.com>

2008/10/24 David Warde-Farley <dwf at cs.toronto.edu>:
> I think you mean 2008? ;)

Yes, of course!

Event: SciPy Sprint
Date: 1, 2 November 2008
Find us on: irc.freenode.net in channel #scipy

I hope that all of you can join us.  Nathan has put a lot of work into
sparse matrices, David has been working on build systems, other David
improved special functions, we have a brand new spatial module by
Anne, other other David is documenting ndimage, Damian implemented
hierarchical clustering and Tiziano is contributing generalised
eigenproblems.  This is going to be a very useful release -- we must
just get it out the door!

Cheers
St?fan


From cimrman3 at ntc.zcu.cz  Fri Oct 24 04:31:19 2008
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Fri, 24 Oct 2008 10:31:19 +0200
Subject: [SciPy-dev] integration of symeig in SciPy
In-Reply-To: <20081023160402.GA3443@localhost>
References: <20081023160402.GA3443@localhost>
Message-ID: <490187D7.5060805@ntc.zcu.cz>

Tiziano Zito wrote:
> Dear SciPy devs, 
> 
> there has been some discussion about integrating symeig in SciPy.
> symeig <http://mdp-toolkit.sourceforge.net/symeig.html> is a Python
> wrapper for the LAPACK functions to solve the standard and
> generalized eigenvalue problems for symmetric (hermitian) positive
> definite matrices. It is superior to the scipy "linalg.eigh" wrapper
> because it wraps *all* relevant LAPACK routines, enabling for
> example to just return a subset of all egenvectors/eigenvalues,
> which is a killer feature if you are dealing with huge-nonsparse
> matrices. Some of these LAPACK routines are available in
> scipy.lib.lapack.flapack and can be accessed through
> scipy.lib.lapack.get_lapack_funcs . Some of them are still missing
> in scipy, while symeig offers a unified interface to all relevant
> LAPACK functions.  symeig has been dowloaded  more than 1300 times
> since its first public appearance on sourceforge (under the
> mdp-toolkit package) in 2004. It features a complete unit test and a
> telling doc-string. I am one of the authors of symeig and have been
> contacted by some scipy devs to discuss the integration of symeig in
> scipy. I am more than willing to help doing this. The most difficult
> part (the LGPL license) has been addressed already: I've re-issued
> symeig under a BSD license. Next step would be to adapt symeig code
> and tests to the scipy framework. Some questions to you:

This is really good news. I had to use the missing functions directly 
via scipy.lib.lapack.get_lapack_funcs which was not very convenient. For 
example the lobpcg sparse eigensolver requires the symmetric routines.

Thank you!
r.



From argriffi at ncsu.edu  Fri Oct 24 08:04:46 2008
From: argriffi at ncsu.edu (alex griffing)
Date: Fri, 24 Oct 2008 08:04:46 -0400
Subject: [SciPy-dev] integration of symeig in SciPy
In-Reply-To: <20081023160402.GA3443@localhost>
References: <20081023160402.GA3443@localhost>
Message-ID: <4901B9DE.2050809@ncsu.edu>

Tiziano Zito wrote:
> Dear SciPy devs, 
>
> there has been some discussion about integrating symeig in SciPy.
> symeig <http://mdp-toolkit.sourceforge.net/symeig.html> is a Python
> wrapper for the LAPACK functions to solve the standard and
> generalized eigenvalue problems for symmetric (hermitian) positive
> definite matrices. It is superior to the scipy "linalg.eigh" wrapper
> because it wraps *all* relevant LAPACK routines, enabling for
> example to just return a subset of all egenvectors/eigenvalues,
> which is a killer feature if you are dealing with huge-nonsparse
> matrices. Some of these LAPACK routines are available in
> scipy.lib.lapack.flapack and can be accessed through
> scipy.lib.lapack.get_lapack_funcs . Some of them are still missing
> in scipy, while symeig offers a unified interface to all relevant
> LAPACK functions.  symeig has been dowloaded  more than 1300 times
> since its first public appearance on sourceforge (under the
> mdp-toolkit package) in 2004. It features a complete unit test and a
> telling doc-string. I am one of the authors of symeig and have been
> contacted by some scipy devs to discuss the integration of symeig in
> scipy. I am more than willing to help doing this. The most difficult
> part (the LGPL license) has been addressed already: I've re-issued
> symeig under a BSD license. Next step would be to adapt symeig code
> and tests to the scipy framework. Some questions to you:
>
> - where would you like to have symeig? (I would suggest scipy.linalg)
>
> - should we change the signature to match or resamble that of other
>   eigenproblem-related functions in scipy?
>
> - is there a responsible scipy dev for the linalg package I can bother
>   with any questions I may have in the process?
>
> - do I get svn commit rights or should I work on a local copy and
>   send a patch?
>
> let me know, 
>
> tiziano
>   

This sounds like it would be very useful for scipy although I am just a 
user not a dev.  Could I use symeig to extract only the eigenvector 
corresponding to the largest eigenvalue, given a non-sparse symmetric 
real matrix whose eigenvalues are all positive except for exactly one 
that is zero?  The documentation for symeig says that it works for 
"symmetric (hermitian) definite positive matrices" but I guess mine 
would be a "symmetric (hermitian) semi-definite positive matrix".  If 
symeig wouldn't work, is there another way I could do this without using 
eigh to get the full eigendecomposition?

Thanks,
Alex


From nwagner at iam.uni-stuttgart.de  Fri Oct 24 09:52:28 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 24 Oct 2008 15:52:28 +0200
Subject: [SciPy-dev] integration of symeig in SciPy
In-Reply-To: <4901B9DE.2050809@ncsu.edu>
References: <20081023160402.GA3443@localhost>
 <4901B9DE.2050809@ncsu.edu>
Message-ID: <web-113283074@uni-stuttgart.de>

On Fri, 24 Oct 2008 08:04:46 -0400
  alex griffing <argriffi at ncsu.edu> wrote:
> Tiziano Zito wrote:
>> Dear SciPy devs, 
>>
>> there has been some discussion about integrating symeig 
>>in SciPy.
>> symeig <http://mdp-toolkit.sourceforge.net/symeig.html> 
>>is a Python
>> wrapper for the LAPACK functions to solve the standard 
>>and
>> generalized eigenvalue problems for symmetric 
>>(hermitian) positive
>> definite matrices. It is superior to the scipy 
>>"linalg.eigh" wrapper
>> because it wraps *all* relevant LAPACK routines, 
>>enabling for
>> example to just return a subset of all 
>>egenvectors/eigenvalues,
>> which is a killer feature if you are dealing with 
>>huge-nonsparse
>> matrices. Some of these LAPACK routines are available in
>> scipy.lib.lapack.flapack and can be accessed through
>> scipy.lib.lapack.get_lapack_funcs . Some of them are 
>>still missing
>> in scipy, while symeig offers a unified interface to all 
>>relevant
>> LAPACK functions.  symeig has been dowloaded  more than 
>>1300 times
>> since its first public appearance on sourceforge (under 
>>the
>> mdp-toolkit package) in 2004. It features a complete 
>>unit test and a
>> telling doc-string. I am one of the authors of symeig 
>>and have been
>> contacted by some scipy devs to discuss the integration 
>>of symeig in
>> scipy. I am more than willing to help doing this. The 
>>most difficult
>> part (the LGPL license) has been addressed already: I've 
>>re-issued
>> symeig under a BSD license. Next step would be to adapt 
>>symeig code
>> and tests to the scipy framework. Some questions to you:
>>
>> - where would you like to have symeig? (I would suggest 
>>scipy.linalg)
>>
>> - should we change the signature to match or resamble 
>>that of other
>>   eigenproblem-related functions in scipy?
>>
>> - is there a responsible scipy dev for the linalg 
>>package I can bother
>>   with any questions I may have in the process?
>>
>> - do I get svn commit rights or should I work on a local 
>>copy and
>>   send a patch?
>>
>> let me know, 
>>
>> tiziano
>>   
> 
> This sounds like it would be very useful for scipy 
>although I am just a 
> user not a dev.  Could I use symeig to extract only the 
>eigenvector 
> corresponding to the largest eigenvalue, given a 
>non-sparse symmetric 
> real matrix whose eigenvalues are all positive except 
>for exactly one 
> that is zero?  The documentation for symeig says that it 
>works for 
> "symmetric (hermitian) definite positive matrices" but I 
>guess mine 
> would be a "symmetric (hermitian) semi-definite positive 
>matrix".  If 
> symeig wouldn't work, is there another way I could do 
>this without using 
> eigh to get the full eigendecomposition?
> 
> Thanks,
> Alex
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev

from symeig import symeig
from numpy import diag, ones

n = 10
M = 2.0*diag(ones(n))
K = 2*diag(ones(n))-diag(ones(n-1),1)-diag(ones(n-1),-1)

print M
print K

lo = n
hi = n
w,Z = symeig(K,M,range=(lo,hi))
print
print 'Largest eigenvalue',w
print
print 'Corresponding eigenvector'
print
print Z


Nils
  


From nwagner at iam.uni-stuttgart.de  Fri Oct 24 11:29:29 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Fri, 24 Oct 2008 17:29:29 +0200
Subject: [SciPy-dev] integration of symeig in SciPy
In-Reply-To: <4901B9DE.2050809@ncsu.edu>
References: <20081023160402.GA3443@localhost>
 <4901B9DE.2050809@ncsu.edu>
Message-ID: <web-113286781@uni-stuttgart.de>

On Fri, 24 Oct 2008 08:04:46 -0400
  alex griffing <argriffi at ncsu.edu> wrote:
> Tiziano Zito wrote:
>> Dear SciPy devs, 
>>
>> there has been some discussion about integrating symeig 
>>in SciPy.
>> symeig <http://mdp-toolkit.sourceforge.net/symeig.html> 
>>is a Python
>> wrapper for the LAPACK functions to solve the standard 
>>and
>> generalized eigenvalue problems for symmetric 
>>(hermitian) positive
>> definite matrices. It is superior to the scipy 
>>"linalg.eigh" wrapper
>> because it wraps *all* relevant LAPACK routines, 
>>enabling for
>> example to just return a subset of all 
>>egenvectors/eigenvalues,
>> which is a killer feature if you are dealing with 
>>huge-nonsparse
>> matrices. Some of these LAPACK routines are available in
>> scipy.lib.lapack.flapack and can be accessed through
>> scipy.lib.lapack.get_lapack_funcs . Some of them are 
>>still missing
>> in scipy, while symeig offers a unified interface to all 
>>relevant
>> LAPACK functions.  symeig has been dowloaded  more than 
>>1300 times
>> since its first public appearance on sourceforge (under 
>>the
>> mdp-toolkit package) in 2004. It features a complete 
>>unit test and a
>> telling doc-string. I am one of the authors of symeig 
>>and have been
>> contacted by some scipy devs to discuss the integration 
>>of symeig in
>> scipy. I am more than willing to help doing this. The 
>>most difficult
>> part (the LGPL license) has been addressed already: I've 
>>re-issued
>> symeig under a BSD license. Next step would be to adapt 
>>symeig code
>> and tests to the scipy framework. Some questions to you:
>>
>> - where would you like to have symeig? (I would suggest 
>>scipy.linalg)
>>
>> - should we change the signature to match or resamble 
>>that of other
>>   eigenproblem-related functions in scipy?
>>
>> - is there a responsible scipy dev for the linalg 
>>package I can bother
>>   with any questions I may have in the process?
>>
>> - do I get svn commit rights or should I work on a local 
>>copy and
>>   send a patch?
>>
>> let me know, 
>>
>> tiziano
>>   
> 
> This sounds like it would be very useful for scipy 
>although I am just a 
> user not a dev.  Could I use symeig to extract only the 
>eigenvector 
> corresponding to the largest eigenvalue, given a 
>non-sparse symmetric 
> real matrix whose eigenvalues are all positive except 
>for exactly one 
> that is zero?  The documentation for symeig says that it 
>works for 
> "symmetric (hermitian) definite positive matrices" but I 
>guess mine 
> would be a "symmetric (hermitian) semi-definite positive 
>matrix".  If 
> symeig wouldn't work, is there another way I could do 
>this without using 
> eigh to get the full eigendecomposition?
> 
> Thanks,
> Alex
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
  
Here is an example where K is positive semidefinite.

from symeig import symeig
from numpy import diag, ones, r_

n = 10
M = 2.0*diag(ones(n))
#
# Symmetric positive-semidefinite
#
K = 
diag(r_[1.0,2*ones(n-2),1])-diag(ones(n-1),1)-diag(ones(n-1),-1)
K[0,0]   = 1.0
K[-1,-1] = 1.0
print M
print K

lo = 1
hi = 1
w,Z = symeig(K,M,range=(lo,hi))
print
print 'Smallest eigenvalue',w
print
print 'Rigid body mode'
print
print Z

lo = n
hi = n
w,Z = symeig(K,M,range=(lo,hi))
print
print 'Largest eigenvalue',w
print
print 'Corresponding eigenvector'
print
print Z

Nils



From millman at berkeley.edu  Sun Oct 26 05:15:55 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 26 Oct 2008 02:15:55 -0700
Subject: [SciPy-dev] Unifying lapack/blas in lib and linalg
Message-ID: <c7009a550810260215sde43fcfheadf4d008f2e4024@mail.gmail.com>

Hello,

Before releasing 0.7, I would like to see atlas/blas wrappers in just
one place.  Currently the wrappers are in both lib and linalg.  The
code duplication has existed for way too long.  Given that Tiziano
Zito has offered to contribute symeig, I think that this might be a
good time to clean up our existing code.

Is anyone interested in working on this?  Maybe a few people will be
willing to step forward and focus on this during the SciPy 0.7 sprint
that we will be holding Nov. 1-2.  Please let me know ASAP if you are
interested or have suggestions.  Thoughts?

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From pav at iki.fi  Sun Oct 26 10:13:37 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 26 Oct 2008 16:13:37 +0200
Subject: [SciPy-dev] docs.scipy.org -- new site for the documentation
	marathon
Message-ID: <1225030418.8147.86.camel@idol>

Hi all,

Ga?l and I finished moving the Scipy documentation editor web app to a
new site (at Enthought, thanks!). The Numpy documentation marathon can
now be found at

	http://docs.scipy.org/numpy/

instead of the old location. Everything should work as previously, or
even better, but if something is broken, please notify me.

    ***

The domain name was, however, very tempting for hosting also the
Sphinx-generated documentation, so we put those there too. You can
browse to

	http://docs.scipy.org/

to see how they currently look like. Suggestions how to improve this
are welcome!

Currently, the sources for the Sphinx stuff can be found here:

	https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide
	http://www.iki.fi/pav/tmp/scipy-refguide.tar.gz
	http://www.iki.fi/pav/tmp/docsscipy-frontpage.tar.gz
	(The latter two are tarballs because of problems with
	launchpad.net and apparently Bzr 1.6.1.)

But I think I will move all of them to Scipy's SVN -- I believe the
advantages of using the same revision control system as numpy and scipy
themselves will outweigh those of using Bzr.

Eventually, we can also make all this documentation editable on the web
via the Documentation marathon app. (Actually, we already could, but
creating new files is not possible using the app yet, and there are some
performance issues remaining in rendering long documentation files.)

   ***

Now, the role of docs.scipy.org warrants discussion, because on the one
hand, the domain "docs.scipy.org" looks very official, and on the other
hand, "scipy.org/Documentation" claims to be the place for official
documentation. What do you think: should we use the current front page
of docs.scipy.org, shifting the focus and entry point of documentation
to the Sphinx-generated pages, or still keep the focus on the Moin wiki
at scipy.org?

I'd like to see a move towards a full dedicated documentation site -- I
believe using Sphinx is the way to go in the future, and documentation
generated by it does have a reassuring and clear visual appearance.
Also, making the Sphinx documentation more prominent could help people
to focus documentation efforts, and write more of it :)

Cheers,

-- 
Pauli Virtanen




From millman at berkeley.edu  Sun Oct 26 13:25:07 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 26 Oct 2008 10:25:07 -0700
Subject: [SciPy-dev] [Numpy-discussion] docs.scipy.org -- new site for
	the documentation marathon
In-Reply-To: <1225030418.8147.86.camel@idol>
References: <1225030418.8147.86.camel@idol>
Message-ID: <c7009a550810261025g5d9415b7w1fef2d9059f5c644@mail.gmail.com>

Thanks so much for doing this.  It looks great.

On Sun, Oct 26, 2008 at 7:13 AM, Pauli Virtanen <pav at iki.fi> wrote:
> Now, the role of docs.scipy.org warrants discussion, because on the one
> hand, the domain "docs.scipy.org" looks very official, and on the other
> hand, "scipy.org/Documentation" claims to be the place for official
> documentation. What do you think: should we use the current front page
> of docs.scipy.org, shifting the focus and entry point of documentation
> to the Sphinx-generated pages, or still keep the focus on the Moin wiki
> at scipy.org?

docs.scipy.org should be the official documentation.  The Moin wiki
documentation served its purpose, but now there is something much
better.

I propose that the 'Documentation' sidebar on http://www.scipy.org/
point to http://docs.scipy.org.  Eventually, I would like to see all
the content from http://www.scipy.org/Documentation move into the
Sphinx-based system (and then we can just delete the Moin page).
Until that happens we can just leave the link on http://docs.scipy.org
to http://www.scipy.org/.

Also I have cross-posted my response to the numpy list, but let's have
this discussion on the scipy developer's list going forward.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Sun Oct 26 13:37:00 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 26 Oct 2008 10:37:00 -0700
Subject: [SciPy-dev] [Numpy-discussion] docs.scipy.org -- new site for
	the documentation marathon
In-Reply-To: <d05265cb0810260941l2857709fgf76d27494473eb10@mail.gmail.com>
References: <1225030418.8147.86.camel@idol>
	<d05265cb0810260941l2857709fgf76d27494473eb10@mail.gmail.com>
Message-ID: <c7009a550810261037x7dc44769k23da3c41d6dc8f2f@mail.gmail.com>

On Sun, Oct 26, 2008 at 9:41 AM, Nathan Bell <wnbell at gmail.com> wrote:
> As a concrete example of your question, I've been working on some
> scipy.sparse documentation [1], and it seems like it will be fairly
> lengthy when completed.  Would it be appropriate to merge it into the
> Sphinx documentation for scipy.sparse [2], or should the Sphinx docs
> be more concise?
>
> [1] http://www.scipy.org/SciPyPackages/Sparse
> [2] http://docs.scipy.org/doc/scipy/reference/sparse.html

Thanks Nathan for working on the sparse documentation!  I would like
to see all your documentation in the Sphinx docs:
http://docs.scipy.org/doc/scipy/reference/sparse.htm

All the documentation should be in the Sphinx system.  One of the main
advantages of using the Sphinx system is that it is much easier to
make living documentation if it is more closely integrated with the
code.  First, when changes are made to the code it is much easier to
just use a tool like grep or grin to find all the places where you
need to update the documentation.  Since we can use matplotlib to
generate figures when the documentation is created, we can better
ensure that the figures in our documentation can be generated by the
code in our documentation.  We can also use our testing infrastructure
to run any code in the documentation.  This has the advantages of
providing us with more tests and helps keep code in our documentation
runnable.

I would like to see http://www.scipy.org/SciPyPackages/Sparse removed
and all the content merged into
http://docs.scipy.org/doc/scipy/reference/sparse.html

I have cross-posted my response to the numpy list, but let's have this
discussion on the scipy developer's list going forward.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From pav at iki.fi  Sun Oct 26 13:41:07 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 26 Oct 2008 17:41:07 +0000 (UTC)
Subject: [SciPy-dev] docs.scipy.org -- new site for the	documentation
	marathon
References: <1225030418.8147.86.camel@idol>
	<d05265cb0810260941l2857709fgf76d27494473eb10@mail.gmail.com>
Message-ID: <ge2a3j$u5a$1@ger.gmane.org>

Sun, 26 Oct 2008 12:41:28 -0400, Nathan Bell wrote:

> On Sun, Oct 26, 2008 at 10:13 AM, Pauli Virtanen <pav at iki.fi> wrote:
>>
>> Now, the role of docs.scipy.org warrants discussion, because on the one
>> hand, the domain "docs.scipy.org" looks very official, and on the other
>> hand, "scipy.org/Documentation" claims to be the place for official
>> documentation. What do you think: should we use the current front page
>> of docs.scipy.org, shifting the focus and entry point of documentation
>> to the Sphinx-generated pages, or still keep the focus on the Moin wiki
>> at scipy.org?
>>
>> I'd like to see a move towards a full dedicated documentation site -- I
>> believe using Sphinx is the way to go in the future, and documentation
>> generated by it does have a reassuring and clear visual appearance.
>> Also, making the Sphinx documentation more prominent could help people
>> to focus documentation efforts, and write more of it :)
>
> Great work, I definitely like the look of the Sphinx documentation.
> 
> As a concrete example of your question, I've been working on some
> scipy.sparse documentation [1], and it seems like it will be fairly
> lengthy when completed.  Would it be appropriate to merge it into the
> Sphinx documentation for scipy.sparse [2], or should the Sphinx docs be
> more concise?

I think we will in the end need both brief API reference documentation 
(docstrings etc., explanations of common semantics such as dtype 
specifications) and then more detailed usage documentation. The current 
Sphinx docs are brief because they almost exclusively contain only the 
former, and for Scipy, mostly only the docstrings are there.

To me, it would make sense to keep all the documentation as closely 
integrated as possible -- then you can use Sphinx's cross-referencing, 
indexing, glossary, math, etc. facilities, and the documentation will be 
easy to find, searchable, and have consistent appearance.

So, I'd like to see your work merged to the Sphinx docs. If I was writing 
the scipy.sparse documentation, I would probably do something like the 
following: one page, say scipy.sparse-reference.rst for a terse listing 
of the API of the package (something like what's currently there), and 
then dedicate as many pages as necessary for more narrative usage 
documentation (what you are writing). Or, I'd put everything on one page, 
as done eg. in Python's optparse documentation [1].

.. [1] http://docs.python.org/library/optparse.html

-- 
Pauli Virtanen



From millman at berkeley.edu  Sun Oct 26 13:44:36 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Sun, 26 Oct 2008 10:44:36 -0700
Subject: [SciPy-dev] docs.scipy.org -- new site for the documentation
	marathon
In-Reply-To: <ge2a3j$u5a$1@ger.gmane.org>
References: <1225030418.8147.86.camel@idol>
	<d05265cb0810260941l2857709fgf76d27494473eb10@mail.gmail.com>
	<ge2a3j$u5a$1@ger.gmane.org>
Message-ID: <c7009a550810261044l6868ba23p4202850f88aa584e@mail.gmail.com>

On Sun, Oct 26, 2008 at 10:41 AM, Pauli Virtanen <pav at iki.fi> wrote:
> I think we will in the end need both brief API reference documentation
> (docstrings etc., explanations of common semantics such as dtype
> specifications) and then more detailed usage documentation. The current
> Sphinx docs are brief because they almost exclusively contain only the
> former, and for Scipy, mostly only the docstrings are there.
>
> To me, it would make sense to keep all the documentation as closely
> integrated as possible -- then you can use Sphinx's cross-referencing,
> indexing, glossary, math, etc. facilities, and the documentation will be
> easy to find, searchable, and have consistent appearance.

+1
If my opinion isn't obvious all ready ;)

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From pav at iki.fi  Sun Oct 26 14:22:21 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 26 Oct 2008 18:22:21 +0000 (UTC)
Subject: [SciPy-dev] docs.scipy.org -- new site for the
	documentation	marathon
References: <1225030418.8147.86.camel@idol>
Message-ID: <ge2cgt$u5a$2@ger.gmane.org>

Hi,

Sun, 26 Oct 2008 16:13:37 +0200, Pauli Virtanen wrote:
[clip]
> Currently, the sources for the Sphinx stuff can be found here:
> 
> 	https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide
> 	http://www.iki.fi/pav/tmp/scipy-refguide.tar.gz
> 	http://www.iki.fi/pav/tmp/docsscipy-frontpage.tar.gz (The latter 
two
> 	are tarballs because of problems with launchpad.net and 
apparently Bzr
> 	1.6.1.)
> 
> But I think I will move all of them to Scipy's SVN -- I believe the
> advantages of using the same revision control system as numpy and scipy
> themselves will outweigh those of using Bzr.

Sources for the docs live from now on at

	http://svn.scipy.org/svn/numpy/numpy-docs/trunk
	http://svn.scipy.org/svn/scipy/scipy-docs/trunk

-- 
Pauli Virtanen



From stefan at sun.ac.za  Mon Oct 27 04:44:42 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 27 Oct 2008 10:44:42 +0200
Subject: [SciPy-dev] Tasks for SciPy Sprint, 1 and 2 November
Message-ID: <9457e7c80810270144w6cb45746hb8e291854b8d2f50@mail.gmail.com>

Hi all,

The wiki page for the sprints,

http://scipy.org/scipy/scipy/wiki/SciPySprintOhSeven

now lists the main topics for the sprint.  Please feel free to add your own.

Regards
St?fan


From josef.pktd at gmail.com  Mon Oct 27 09:18:06 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 27 Oct 2008 09:18:06 -0400
Subject: [SciPy-dev] Tasks for SciPy Sprint, 1 and 2 November
In-Reply-To: <9457e7c80810270144w6cb45746hb8e291854b8d2f50@mail.gmail.com>
References: <9457e7c80810270144w6cb45746hb8e291854b8d2f50@mail.gmail.com>
Message-ID: <1cd32cbb0810270618l25fc1249s6d544b63a0c11c9b@mail.gmail.com>

Is there any interest at all in fixing scipy.stats.distribution?

Josef


From stefan at sun.ac.za  Mon Oct 27 09:46:02 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 27 Oct 2008 15:46:02 +0200
Subject: [SciPy-dev] Tasks for SciPy Sprint, 1 and 2 November
In-Reply-To: <1cd32cbb0810270618l25fc1249s6d544b63a0c11c9b@mail.gmail.com>
References: <9457e7c80810270144w6cb45746hb8e291854b8d2f50@mail.gmail.com>
	<1cd32cbb0810270618l25fc1249s6d544b63a0c11c9b@mail.gmail.com>
Message-ID: <9457e7c80810270646k78796c2ei4bdd0d8b374744ae@mail.gmail.com>

2008/10/27  <josef.pktd at gmail.com>:
> Is there any interest at all in fixing scipy.stats.distribution?

Certainly!  I saw you filed tickets, so those will be taken care of.

Cheers
St?fan


From jh at physics.ucf.edu  Mon Oct 27 15:16:14 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Mon, 27 Oct 2008 15:16:14 -0400
Subject: [SciPy-dev] docs.scipy.org -- new site for
	the	documentation	marathon
In-Reply-To: <mailman.15.1225126804.32664.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.15.1225126804.32664.scipy-dev@scipy.org>
Message-ID: <wl8prllzw5d.fsf@glup.physics.ucf.edu>

I like the Sphinx site!

But, I like the scipy.org/Documentation URL, since docs are part of
the package and not a separate project (like, say, ipython.scipy.org).
We should have a consistent layout of URLs, so that things that are
part of numpy/scipy are under http://scipy.org/ and things that are
not might still be hosted on a scipy.org domain but are not directly
under http://scipy.org/.  Is it technically possible/feasible for a
Moin site to (appear to) include another site underneath it?  If not,
I'd choose function over form, grudgingly.

For the top page layout, I propose (rationale below):

(No left sidebar)
------------------------------------------------------------
DOCUMENTATION EDITOR
Write, review, or proof the docs!

MATURE DOCUMENTS

Guide to Numpy
PDF book by Travis Oliphant

DOCUMENT-IN-PROGRESS SNAPSHOTS

Numpy Reference Guide (as of yyyy-mm-dd)
PDF   zipped HTML
  refguide glossary shortcut

Numpy User Guide (as of yyyy-mm-dd)
PDF   zipped HTML

Scipy Reference Guide (as of yyyy-mm-dd)
PDF   zipped HTML

SEE ALSO

SciPy.org
all things NumPy/SciPy (bug reports, downloads, conferences, etc.)

SciPy.org/Documentation
more documentation not (yet) in the right format for this site

SciPy.org/Cookbook
live, user-contributed examples and recipes for common tasks
------------------------------------------------------------

Rationale: We have had a real problem getting a critical mass on doc
writing.  We rallied in the summer, but for the past two months almost
no contributions have been made (click on stats to see).  Originally,
the only purpose of the site was editing the docs, and someone who
wanted to read what was current did that through our editing
infrastructure.  This made it very encouraging to edit.  Now, when
they click on "improved docstrings" on scipy.org/Documentation, they
instead go to a passive presentation of the current content, no
editor.  They have to go to the last link on the page to get into the
editor, and the text there doesn't scream "Editor!".  Until we have
*at least* version 1 of each of our docs, we should keep the edit link
first.  I'm almost ready to say the docs in progress should *only* be
accessible through the editor (which does present HTML, zipped HTML,
and PDF) until we have version 1 done.

We should also remove irrelevant links and text.  The Developer Zone
and its instructions for reporting bugs have their own section of the
main web site and are not docs, so they don't belong here.  The
content of the left sidebar is redundant and wastes space.  The
download links should go under the live-doc links, as above, since
this format is easier to read than that on the download page and there
really isn't any other content on this page.  Scipy.org is either a
see-also or nonexistent, depending on whether we can get this page to
appear under it.  The glossary shortcut is also accomodated above.

--jh--


From stefan at sun.ac.za  Mon Oct 27 15:45:24 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 27 Oct 2008 21:45:24 +0200
Subject: [SciPy-dev] docs.scipy.org -- new site for the documentation
	marathon
In-Reply-To: <wl8prllzw5d.fsf@glup.physics.ucf.edu>
References: <mailman.15.1225126804.32664.scipy-dev@scipy.org>
	<wl8prllzw5d.fsf@glup.physics.ucf.edu>
Message-ID: <9457e7c80810271245yf7309e5pc779c3e4a8c48d94@mail.gmail.com>

2008/10/27  <jh at physics.ucf.edu>:
> But, I like the scipy.org/Documentation URL, since docs are part of

There is one big advantage to docs.scipy.org -- it is short and easy
to remember.  In other words, from the user's point of view: "Where
would I expect the docs to be?" "Probably at doc(s).scipy.org or
scipy.org/doc(s)".

St?fan


From robert.kern at gmail.com  Mon Oct 27 15:47:59 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 27 Oct 2008 14:47:59 -0500
Subject: [SciPy-dev] docs.scipy.org -- new site for the documentation
	marathon
In-Reply-To: <9457e7c80810271245yf7309e5pc779c3e4a8c48d94@mail.gmail.com>
References: <mailman.15.1225126804.32664.scipy-dev@scipy.org>
	<wl8prllzw5d.fsf@glup.physics.ucf.edu>
	<9457e7c80810271245yf7309e5pc779c3e4a8c48d94@mail.gmail.com>
Message-ID: <3d375d730810271247j46a6ad5bh8eee79886808086b@mail.gmail.com>

On Mon, Oct 27, 2008 at 14:45, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/10/27  <jh at physics.ucf.edu>:
>> But, I like the scipy.org/Documentation URL, since docs are part of
>
> There is one big advantage to docs.scipy.org -- it is short and easy
> to remember.  In other words, from the user's point of view: "Where
> would I expect the docs to be?" "Probably at doc(s).scipy.org or
> scipy.org/doc(s)".

It's also easy to Google with site:docs.scipy.org

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From pav at iki.fi  Mon Oct 27 19:31:20 2008
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 27 Oct 2008 23:31:20 +0000 (UTC)
Subject: [SciPy-dev] docs.scipy.org -- new site
	for	the	documentation	marathon
References: <mailman.15.1225126804.32664.scipy-dev@scipy.org>
	<wl8prllzw5d.fsf@glup.physics.ucf.edu>
Message-ID: <ge5j08$mps$1@ger.gmane.org>

Hi,

Mon, 27 Oct 2008 15:16:14 -0400, jh wrote:
> I like the Sphinx site!
> 
> But, I like the scipy.org/Documentation URL, since docs are part of the
> package and not a separate project (like, say, ipython.scipy.org). We
> should have a consistent layout of URLs, so that things that are part of
> numpy/scipy are under http://scipy.org/ and things that are not might
> still be hosted on a scipy.org domain but are not directly under
> http://scipy.org/.  Is it technically possible/feasible for a Moin site
> to (appear to) include another site underneath it?  If not, I'd choose
> function over form, grudgingly.

I don't see a problem having a separate domain docs.XXX -- I think it's 
relatively clear to the user what the site contains, and emphasizes that 
this is the "official" documentation. Also, several projects, eg. Python, 
have a similar layout, so I don't think we are in a bad company with 
this). But technically, it would be possible to have scipy.org display 
documents residing on docs.scipy.org (eg. using mod_rewrite [P] in apache 
-- involving Moin in the process is wasteful).

> For the top page layout, I propose (rationale below):
> 
> (No left sidebar)

The branding on the docs site should probabably be made a bit more 
similar to that on scipy.org, and this probably involves deciding what to 
do with the sidebar. There should also be an obvious way back to 
scipy.org. The muse is silent, so I don't know how the page should look 
like now.

> ------------------------------------------------------------
> DOCUMENTATION EDITOR
> Write, review, or proof the docs!
> 
> MATURE DOCUMENTS
> 
> Guide to Numpy
> PDF book by Travis Oliphant
> 
> DOCUMENT-IN-PROGRESS SNAPSHOTS
> 
> Numpy Reference Guide (as of yyyy-mm-dd) PDF   zipped HTML
>   refguide glossary shortcut
> 
> Numpy User Guide (as of yyyy-mm-dd)
> PDF   zipped HTML
> 
> Scipy Reference Guide (as of yyyy-mm-dd) PDF   zipped HTML
[clip]

Looks good to me. How about combining the "DOCUMENTATION EDITOR" and 
"SNAPSHOTS" sections?

[clip]
> Rationale: We have had a real problem getting a critical mass on doc
> writing.  We rallied in the summer, but for the past two months almost
> no contributions have been made (click on stats to see).  Originally,
> the only purpose of the site was editing the docs, and someone who
> wanted to read what was current did that through our editing
> infrastructure.  This made it very encouraging to edit.  Now, when they
> click on "improved docstrings" on scipy.org/Documentation, they instead
> go to a passive presentation of the current content, no editor.  They
> have to go to the last link on the page to get into the editor, and the
> text there doesn't scream "Editor!".  Until we have *at least* version 1
> of each of our docs, we should keep the edit link first.

It's possible (and easy) to add an edit link on each editable page in the 
static documentation. I can add these.

The changes won't, however, be immediately visible in the static 
documentation, so maybe clicking the link the first time should show a 
popup (or something else easy to implement in static javascript) that 
explains this.

> I'm almost ready to say the docs in progress should *only* be
> accessible through the editor (which does present HTML, zipped HTML,
> and PDF) until we have version 1 done.

The problem is that presently not all pages are editable via the 
documentation wiki, only the docstrings, so for motivation (including 
mine), I think it's important to have a snapshot of the "final" versions 
available somewhere. Maybe not linked to with big letters, but still 
available.

Issue [1] is a blocker for this, and one needs to think also about [2].

.. [1] http://code.google.com/p/pydocweb/issues/detail?id=26
.. [2] http://code.google.com/p/pydocweb/issues/detail?id=27

> We should also remove irrelevant links and text. 
[clip: downloads to main page, remove unnecessary links]

Sure, these seem like good improvements.

-- 
Pauli Virtanen



From gael.varoquaux at normalesup.org  Tue Oct 28 04:18:08 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 28 Oct 2008 09:18:08 +0100
Subject: [SciPy-dev] Of use and abuse of sphinx's math directive
Message-ID: <20081028081807.GB11820@phare.normalesup.org>

Howdy,

Sphinx is great, will all love Sphinx. But I do wonder if sometimes we do
not use the math directive a bit too much, eg the "n" and "n-1" on:
http://scipy.org/scipy/scipy/browser/trunk/scipy/cluster/hierarchy.py#L1090
(this is just an example that I picked randomly).

When the formula written is simple-enought that its readibility is not
too much affected by having it in text in the Sphinxified document, I
think it is a pitty to render the plain text docstrings hard-to-read.

What do people think?

Ga?l


From david at ar.media.kyoto-u.ac.jp  Tue Oct 28 04:49:21 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 28 Oct 2008 17:49:21 +0900
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
Message-ID: <4906D211.6070505@ar.media.kyoto-u.ac.jp>

Hi,

    There are some things I would like to do for scipy.fftpack during
the sprint, and I would like to know if anybody would be strongly
against any one of those:

renaming scipy.fftpack -> scipy.fft
===================================

Rationale: fftpack is the name of an implementation (fftpack being the
name of the Fortran library we use when no other fft library is
available), and does not fit the convention of any other scipy top
subpackage.

How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
placeholder (like scipy.linsolve), and deprecating scipy.fftpack.

Removing all backends but fftpack
=================================

Rationale: we have 5 backends (fftpack, djbfft, fftw2 and 3, mkl), and
none of them is complete. I would much prefer focus on one (fftpack),
make it complete (for example supporting float32 type), than having 5
half implemented ones. Although, more backends mean more reliability
problems for installation, distribution, support, etc...

People may argue that fftw3 or MKL is faster: it is true in theory, but
in practice, not so much. In particular, with the current
implementation, both fftw2 and fftw3 are as fast or even slower than
fftpack (because the wrappers are relatively naive, and do not use fftw
effectively to get max speed). MKL is faster, but a lot of
functionalities are missing for this backend.

How: putting all the code not related to fftpack to a scikits or
something, so people who really need the speed can use this. Since you
need to compile scipy by yourself anyway for MKL, I don't see this as a
huge problem.

If I don't get answer within the sprint, I will proceed with the above plan,

cheers,

David


From stefan at sun.ac.za  Tue Oct 28 05:33:37 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 28 Oct 2008 11:33:37 +0200
Subject: [SciPy-dev] Of use and abuse of sphinx's math directive
In-Reply-To: <20081028081807.GB11820@phare.normalesup.org>
References: <20081028081807.GB11820@phare.normalesup.org>
Message-ID: <9457e7c80810280233j618c95fcn997505098be63d23@mail.gmail.com>

2008/10/28 Gael Varoquaux <gael.varoquaux at normalesup.org>:
> Sphinx is great, will all love Sphinx. But I do wonder if sometimes we do
> not use the math directive a bit too much, eg the "n" and "n-1" on:
> http://scipy.org/scipy/scipy/browser/trunk/scipy/cluster/hierarchy.py#L1090
> (this is just an example that I picked randomly).
>
> When the formula written is simple-enought that its readibility is not
> too much affected by having it in text in the Sphinxified document, I
> think it is a pitty to render the plain text docstrings hard-to-read.

Certainly -- the idea is to use the :math: tag sparingly.

Cheers
St?fan


From millman at berkeley.edu  Tue Oct 28 07:31:22 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 28 Oct 2008 04:31:22 -0700
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
Message-ID: <c7009a550810280431s3e048954qaaf0961d0cd02c8f@mail.gmail.com>

On Tue, Oct 28, 2008 at 1:49 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> renaming scipy.fftpack -> scipy.fft
> ===================================
>
> Rationale: fftpack is the name of an implementation (fftpack being the
> name of the Fortran library we use when no other fft library is
> available), and does not fit the convention of any other scipy top
> subpackage.
>
> How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
> placeholder (like scipy.linsolve), and deprecating scipy.fftpack.

+1

> Removing all backends but fftpack
> =================================
>
> Rationale: we have 5 backends (fftpack, djbfft, fftw2 and 3, mkl), and
> none of them is complete. I would much prefer focus on one (fftpack),
> make it complete (for example supporting float32 type), than having 5
> half implemented ones. Although, more backends mean more reliability
> problems for installation, distribution, support, etc...
>
> People may argue that fftw3 or MKL is faster: it is true in theory, but
> in practice, not so much. In particular, with the current
> implementation, both fftw2 and fftw3 are as fast or even slower than
> fftpack (because the wrappers are relatively naive, and do not use fftw
> effectively to get max speed). MKL is faster, but a lot of
> functionalities are missing for this backend.
>
> How: putting all the code not related to fftpack to a scikits or
> something, so people who really need the speed can use this. Since you
> need to compile scipy by yourself anyway for MKL, I don't see this as a
> huge problem.

+1

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From matthieu.brucher at gmail.com  Tue Oct 28 09:52:20 2008
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 28 Oct 2008 14:52:20 +0100
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0810280652xfab465an2d80f6b46e2bc4d@mail.gmail.com>

> People may argue that fftw3 or MKL is faster: it is true in theory, but
> in practice, not so much. In particular, with the current
> implementation, both fftw2 and fftw3 are as fast or even slower than
> fftpack (because the wrappers are relatively naive, and do not use fftw
> effectively to get max speed). MKL is faster, but a lot of
> functionalities are missing for this backend.

Besides, the latest MKL versions may crash with Python at the moment
(and the Intel engineers do not seem not care about the issue).

Matthieu
-- 
Information System Engineer, Ph.D.
Website: http://matthieu-brucher.developpez.com/
Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92
LinkedIn: http://www.linkedin.com/in/matthieubrucher


From oliphant at enthought.com  Tue Oct 28 10:28:19 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Tue, 28 Oct 2008 09:28:19 -0500
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
Message-ID: <49072183.8000200@enthought.com>

David Cournapeau wrote:
> Hi,
>
>     There are some things I would like to do for scipy.fftpack during
> the sprint, and I would like to know if anybody would be strongly
> against any one of those:
>
> renaming scipy.fftpack -> scipy.fft
> ===================================
>
> Rationale: fftpack is the name of an implementation (fftpack being the
> name of the Fortran library we use when no other fft library is
> available), and does not fit the convention of any other scipy top
> subpackage.
>
> How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
> placeholder (like scipy.linsolve), and deprecating scipy.fftpack.
>   
+1
> Removing all backends but fftpack
> =================================
>
> Rationale: we have 5 backends (fftpack, djbfft, fftw2 and 3, mkl), and
> none of them is complete. I would much prefer focus on one (fftpack),
> make it complete (for example supporting float32 type), than having 5
> half implemented ones. Although, more backends mean more reliability
> problems for installation, distribution, support, etc...
>
> People may argue that fftw3 or MKL is faster: it is true in theory, but
> in practice, not so much. In particular, with the current
> implementation, both fftw2 and fftw3 are as fast or even slower than
> fftpack (because the wrappers are relatively naive, and do not use fftw
> effectively to get max speed). MKL is faster, but a lot of
> functionalities are missing for this backend.
>
> How: putting all the code not related to fftpack to a scikits or
> something, so people who really need the speed can use this. Since you
> need to compile scipy by yourself anyway for MKL, I don't see this as a
> huge problem.
>   
+1



From ellisonbg.net at gmail.com  Tue Oct 28 11:21:21 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 28 Oct 2008 08:21:21 -0700
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
Message-ID: <6ce0ac130810280821r5c01ec34kda20ced46f6849cd@mail.gmail.com>

+1

On Tue, Oct 28, 2008 at 1:49 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
>    There are some things I would like to do for scipy.fftpack during
> the sprint, and I would like to know if anybody would be strongly
> against any one of those:
>
> renaming scipy.fftpack -> scipy.fft
> ===================================
>
> Rationale: fftpack is the name of an implementation (fftpack being the
> name of the Fortran library we use when no other fft library is
> available), and does not fit the convention of any other scipy top
> subpackage.
>
> How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
> placeholder (like scipy.linsolve), and deprecating scipy.fftpack.
>
> Removing all backends but fftpack
> =================================
>
> Rationale: we have 5 backends (fftpack, djbfft, fftw2 and 3, mkl), and
> none of them is complete. I would much prefer focus on one (fftpack),
> make it complete (for example supporting float32 type), than having 5
> half implemented ones. Although, more backends mean more reliability
> problems for installation, distribution, support, etc...
>
> People may argue that fftw3 or MKL is faster: it is true in theory, but
> in practice, not so much. In particular, with the current
> implementation, both fftw2 and fftw3 are as fast or even slower than
> fftpack (because the wrappers are relatively naive, and do not use fftw
> effectively to get max speed). MKL is faster, but a lot of
> functionalities are missing for this backend.
>
> How: putting all the code not related to fftpack to a scikits or
> something, so people who really need the speed can use this. Since you
> need to compile scipy by yourself anyway for MKL, I don't see this as a
> huge problem.
>
> If I don't get answer within the sprint, I will proceed with the above plan,
>
> cheers,
>
> David
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>


From charlesr.harris at gmail.com  Tue Oct 28 13:04:15 2008
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 28 Oct 2008 11:04:15 -0600
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
Message-ID: <e06186140810281004p3d8c5133k3c063481a306093e@mail.gmail.com>

On Tue, Oct 28, 2008 at 2:49 AM, David Cournapeau <
david at ar.media.kyoto-u.ac.jp> wrote:

> Hi,
>
>    There are some things I would like to do for scipy.fftpack during
> the sprint, and I would like to know if anybody would be strongly
> against any one of those:
>
> renaming scipy.fftpack -> scipy.fft
> ===================================
>
> Rationale: fftpack is the name of an implementation (fftpack being the
> name of the Fortran library we use when no other fft library is
> available), and does not fit the convention of any other scipy top
> subpackage.
>
> How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
> placeholder (like scipy.linsolve), and deprecating scipy.fftpack.
>
> Removing all backends but fftpack
> =================================
>
> Rationale: we have 5 backends (fftpack, djbfft, fftw2 and 3, mkl), and
> none of them is complete. I would much prefer focus on one (fftpack),
> make it complete (for example supporting float32 type), than having 5
> half implemented ones. Although, more backends mean more reliability
> problems for installation, distribution, support, etc...
>
> People may argue that fftw3 or MKL is faster: it is true in theory, but
> in practice, not so much. In particular, with the current
> implementation, both fftw2 and fftw3 are as fast or even slower than
> fftpack (because the wrappers are relatively naive, and do not use fftw
> effectively to get max speed). MKL is faster, but a lot of
> functionalities are missing for this backend.
>

IIRC correctly, there are also differences in the way the results of real
transforms are stored, so these backends aren't all that compatible.

>
> How: putting all the code not related to fftpack to a scikits or
> something, so people who really need the speed can use this. Since you
> need to compile scipy by yourself anyway for MKL, I don't see this as a
> huge problem.
>

+1. The fftpack is a pretty good (if old) implementation whose main failing
is not taking maximum advantage of cache. It does need some supplementation,
a Bluestein implementation for instance, but that is easier to do with a
single target.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081028/013b502c/attachment.html>

From jh at physics.ucf.edu  Tue Oct 28 13:21:43 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Tue, 28 Oct 2008 13:21:43 -0400
Subject: [SciPy-dev] docs.scipy.org -- new
	site	for	the	documentation	marathon
In-Reply-To: <mailman.1201.1225201956.2878.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.1201.1225201956.2878.scipy-dev@scipy.org>
Message-ID: <wl8fxmgzlco.fsf@glup.physics.ucf.edu>

>> 2008/10/27  <jh at physics.ucf.edu>:
>>> But, I like the scipy.org/Documentation URL, since docs are part of
>
>> There is one big advantage to docs.scipy.org -- it is short and easy
>> to remember.  In other words, from the user's point of view: "Where
>> would I expect the docs to be?" "Probably at doc(s).scipy.org or
>> scipy.org/doc(s)".

How about, go to scipy.org and click on the giant doc image at the top
or the Documentation item on the left, just as for every other major
topic?

> It's also easy to Google with site:docs.scipy.org

site:scipy.org/Documentation works just as well.  If you don't like
the length of the word "Documentation", I'm all for shortening it to
docs.

I actually don't have a strong preference between docs.scipy.org and
scipy.org/docs PROVIDED that it's consistent with the other URLs on
the site.  Consistency makes it easiest for new users to understand
the site as a whole, gives an impression of professionalism, and tends
to push out the date when you have to scrap everything and start over.

Right now, each major project has an xxx.scipy.org domain, with all
its stuff (install, docs, download, etc.) below that.  As the flagship
products, the numpy and scipy projects are on the eponymous portal
domain.  If we promote some of the portal's top-level links to project
status, it prevents doing the same for, say, ipython's site sections
of the same name, and also confuses users, who can't figure out why
it's docs.scipy.org but scipy.org/Cookbook.  It also sends the message
that docs are a different project from numpy, which I think is a poor
message to send.

On the other hand, we could think of an arrangement where all projects
can use the doc-wiki infrastructure, share the same cookbook, have a
common set of bug-reporting instructions and download pages, etc.
Then we can make cookbook.scipy.org, download.scipy.org,
bugs.scipy.org, install.scipy.org (that sounds nice! maybe we should
say "document.scipy.org" as a subliminal suggestion...ok, maybe not),
and so on, and see stuff for numpy, scipy, ipython, etc. in each area.
I just don't understand special-casing docs in the scipy.org site
layout.  If this reorg makes sense for docs, let's do it for the
others.  If not, then not.

> Looks good to me. How about combining the "DOCUMENTATION EDITOR" and 
> "SNAPSHOTS" sections?

I want the editor to be the first link.  That was the purpose of the
wiki and we have a real problem getting enough people to help (thanks
to all who have, and keep it up!).  I think that mature docs should be
ahead of immature ones.  So that's why to split them.  Once the numpy
and scipy reference manuals and the user guide are complete, we can
move the editor link down to the top of the snapshots.  My $0.02.

> The problem is that presently not all pages are editable via the 
> documentation wiki, only the docstrings,

Ok, good point.  Let's keep the download links.  I don't really want
to separate people from the info they seek, I just want to get them
writing.

Hey everyone!  Write!  Write!  We're over 60% of the way to a good
first draft!  Write!  ;-)

--jh--


From robert.kern at gmail.com  Tue Oct 28 13:31:02 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 28 Oct 2008 12:31:02 -0500
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
Message-ID: <3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>

On Tue, Oct 28, 2008 at 03:49, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Hi,
>
>    There are some things I would like to do for scipy.fftpack during
> the sprint, and I would like to know if anybody would be strongly
> against any one of those:
>
> renaming scipy.fftpack -> scipy.fft
> ===================================
>
> Rationale: fftpack is the name of an implementation (fftpack being the
> name of the Fortran library we use when no other fft library is
> available), and does not fit the convention of any other scipy top
> subpackage.
>
> How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
> placeholder (like scipy.linsolve), and deprecating scipy.fftpack.

This needs to be handled carefully. scipy.fft() already exists. It
also breaks long-standing code (after the deprecation) for no reason
that I find compelling. The name has been there for many years. The
only complaint about it that I have ever seen on the lists was that
people thought that only FFTPACK was supported. Given the following
proposal, that reason goes away. Furthermore, I sometimes have trouble
remembering whether the 'fft' symbol I have imported was numpy.fft or
numpy.fft.fft. This change compounds that confusion.

Overall, I don't think that mere renaming is ever a good reason to
break code. It just feels gratuitous. Yes, there might be some benefit
to having clearer names than what we have lived with so far (although
in this case, I think this is dubious for the reasons given above),
but there is always an enormous downside to breaking code. The
deprecation policy does nothing to reduce that downside. It was never
intended to reduce that downside. Right now, I'm almost sorry for
having introduced it because it appears to be giving the impression
that it is okay to make these kinds of changes.

-1

> Removing all backends but fftpack
> =================================
>
> Rationale: we have 5 backends (fftpack, djbfft, fftw2 and 3, mkl), and
> none of them is complete. I would much prefer focus on one (fftpack),
> make it complete (for example supporting float32 type), than having 5
> half implemented ones. Although, more backends mean more reliability
> problems for installation, distribution, support, etc...

+1

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From opossumnano at gmail.com  Tue Oct 28 13:30:57 2008
From: opossumnano at gmail.com (Tiziano Zito)
Date: Tue, 28 Oct 2008 18:30:57 +0100
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
Message-ID: <20081028173055.GA15367@localhost>

dear list, 

I have started to work on the integration of symeig in scipy. I am
unfortunately unavailable for the sprint on november 1st and 2nd,
but I hope to make up for lost time during next week. This mail is
mainly directed to robert, who was assigned the task of helping me
with this issues, but other devs may have something to say
nonetheless. This is a looong message, be aware! 

I've tried to understand the organization of the symmetric 
eigenproblem routines in numpy and scipy, and came up with this:

- numpy.linalg.eigh:
    - only standard problems ( Ax = lx)
    - wraps (or uses its own lapack_lite version) EVD routines

- scipy.linalg.eigh:
    - only standard problems (Ax = lx)
    - wraps EV routines
    - signature differs from numpy.linalg.eigh

- symeig:
    - for standard problems (Ax = lx) wraps EVR routines
    - for generalized problems (Ax = lBx) wraps GV, GVX, and GVD routines
    - the routine is chosen depending on some flags and depending on
      the type of problem

Those, who don't know the corresponding LAPACK routines, can find a
description here:
http://www.netlib.org/lapack/lug/node30.html
and here:
http://www.netlib.org/lapack/lug/node34.html

Basically, for the standard problems 
- the EV routines are the oldest one (less optimized and less
  precise)
- the EVD are faster than the EV but require more memory
- the EVX offer the possibility to extract just a subset of the
  eigenvectors/eigenvalues
- the EVR are the fastest and use less memory than all others

for the generalized problems:
- the GV routines are the oldest one
- the GVX offer the possibility to extract just a subset of the
  eigenvectors/eigenvalues
- the GVD are faster then GV but require more memory

This is the status quo, now:
- why are numpy and scipy wrapping different routines?
  I think numpy should keep on wrapping the EVD routines (unless
  someone manages to add the EVR routines to lapack_lite.c) and I
  see no reason for scipy to wrap the old EV routines instead of the
  EVD or EVR ones.

- In my opinion, scipy.linalg.eigh should take over the
  functionality now offered by symeig. Note that this would be
  consistent with what is already done with linalg.eig: the scipy
  version adds generalized problem functionality to the standard
  problem functionality already available in numpy (as a plus eigh
  would also grow the possibility to request just 
  a subset of the eigenvs).

- While developing symeig with tested the EVR routines: they are
  really better, so I see no reason not to use them by default. On
  top of that, they fail when a matrix is nearly singular, while EV
  and EVD tend to give wrong results without failing. When an EVR
  routine fails the user can be advised that linalg.svd may be
  better for his case.

- The pyf description files are (if I am not mistaken) automatically
  generated. Should I try to let the generator create pyf files for
  our routines, or do you want to have the little (?) inconsistency
  of having some hand-made pyf in the distribution?

- In the symeig test suite we use two very useful routines:
  random_rot and symrand. Their docstrings:
  
  random_rot:
    """Return a random rotation matrix, drawn from the Haar distribution
    (the only uniform distribution on SO(n)).
    The algorithm is described in the paper
    Stewart, G.W., "The efficient generation of random orthogonal
    matrices with an application to condition estimators", SIAM Journal
    on Numerical Analysis, 17(3), pp. 403-409, 1980.
    For more information see
    http://en.wikipedia.org/wiki/Orthogonal_matrix#Randomization"""

  symrand:
    """Return a random symmetric (Hermitian) matrix.
    
    If 'dim_or_eigv' is an integer N, return a NxN matrix, with eigenvalues
        uniformly distributed on (0,1].
        
    If 'dim_or_eigv' is  1-D real array 'a', return a matrix whose
                      eigenvalues are sort(a)."""

  Do you think these routines may be a useful addition to scipy, or
  should I keep them hidden in the tests?

let me know what do you think!

ciao,
tiziano









From robert.kern at gmail.com  Tue Oct 28 13:32:41 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 28 Oct 2008 12:32:41 -0500
Subject: [SciPy-dev] docs.scipy.org -- new site for the documentation
	marathon
In-Reply-To: <wl8fxmgzlco.fsf@glup.physics.ucf.edu>
References: <mailman.1201.1225201956.2878.scipy-dev@scipy.org>
	<wl8fxmgzlco.fsf@glup.physics.ucf.edu>
Message-ID: <3d375d730810281032j62713ccfx2f209823ba8678e5@mail.gmail.com>

On Tue, Oct 28, 2008 at 12:21,  <jh at physics.ucf.edu> wrote:

[I wrote:]
>> It's also easy to Google with site:docs.scipy.org
>
> site:scipy.org/Documentation works just as well.  If you don't like
> the length of the word "Documentation", I'm all for shortening it to
> docs.

No, it doesn't. Almost none of the actual documentation is a child
under scipy.org/Documentation/.

And please keep attributions.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From wnbell at gmail.com  Tue Oct 28 13:40:05 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Tue, 28 Oct 2008 13:40:05 -0400
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
In-Reply-To: <20081028173055.GA15367@localhost>
References: <20081028173055.GA15367@localhost>
Message-ID: <d05265cb0810281040o362a74fdu3d066ad230a8d7b6@mail.gmail.com>

On Tue, Oct 28, 2008 at 1:30 PM, Tiziano Zito <opossumnano at gmail.com> wrote:
>
> - In the symeig test suite we use two very useful routines:
>  random_rot and symrand. Their docstrings:
>
>  random_rot:
>    """Return a random rotation matrix, drawn from the Haar distribution
>    (the only uniform distribution on SO(n)).
>    The algorithm is described in the paper
>    Stewart, G.W., "The efficient generation of random orthogonal
>    matrices with an application to condition estimators", SIAM Journal
>    on Numerical Analysis, 17(3), pp. 403-409, 1980.
>    For more information see
>    http://en.wikipedia.org/wiki/Orthogonal_matrix#Randomization"""
>
>  symrand:
>    """Return a random symmetric (Hermitian) matrix.
>
>    If 'dim_or_eigv' is an integer N, return a NxN matrix, with eigenvalues
>        uniformly distributed on (0,1].
>
>    If 'dim_or_eigv' is  1-D real array 'a', return a matrix whose
>                      eigenvalues are sort(a)."""
>
>  Do you think these routines may be a useful addition to scipy, or
>  should I keep them hidden in the tests?
>

I'd like to see these and many others in a hypothetical scipy.gallery
module.  That's unlikely to happen before 0.7, so you'll probably want
to hide them in the unittest files for now.

FWIW we have a small set of sparse matrix generators that we use pyamg here:
http://code.google.com/p/pyamg/source/browse/#svn/trunk/pyamg/gallery

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From robert.kern at gmail.com  Tue Oct 28 13:43:34 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 28 Oct 2008 12:43:34 -0500
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
In-Reply-To: <20081028173055.GA15367@localhost>
References: <20081028173055.GA15367@localhost>
Message-ID: <3d375d730810281043q4b3ce5cft5f0c2b4337054850@mail.gmail.com>

On Tue, Oct 28, 2008 at 12:30, Tiziano Zito <opossumnano at gmail.com> wrote:
> dear list,
>
> I have started to work on the integration of symeig in scipy. I am
> unfortunately unavailable for the sprint on november 1st and 2nd,
> but I hope to make up for lost time during next week. This mail is
> mainly directed to robert, who was assigned the task of helping me
> with this issues, but other devs may have something to say
> nonetheless. This is a looong message, be aware!
>
> I've tried to understand the organization of the symmetric
> eigenproblem routines in numpy and scipy, and came up with this:
>
> - numpy.linalg.eigh:
>    - only standard problems ( Ax = lx)
>    - wraps (or uses its own lapack_lite version) EVD routines
>
> - scipy.linalg.eigh:
>    - only standard problems (Ax = lx)
>    - wraps EV routines
>    - signature differs from numpy.linalg.eigh
>
> - symeig:
>    - for standard problems (Ax = lx) wraps EVR routines
>    - for generalized problems (Ax = lBx) wraps GV, GVX, and GVD routines
>    - the routine is chosen depending on some flags and depending on
>      the type of problem
>
> Those, who don't know the corresponding LAPACK routines, can find a
> description here:
> http://www.netlib.org/lapack/lug/node30.html
> and here:
> http://www.netlib.org/lapack/lug/node34.html
>
> Basically, for the standard problems
> - the EV routines are the oldest one (less optimized and less
>  precise)
> - the EVD are faster than the EV but require more memory
> - the EVX offer the possibility to extract just a subset of the
>  eigenvectors/eigenvalues
> - the EVR are the fastest and use less memory than all others
>
> for the generalized problems:
> - the GV routines are the oldest one
> - the GVX offer the possibility to extract just a subset of the
>  eigenvectors/eigenvalues
> - the GVD are faster then GV but require more memory
>
> This is the status quo, now:
> - why are numpy and scipy wrapping different routines?

The usual reasons. History. Limited available time. Possibly limited
understanding of the capabilities of LAPACK. Possibly limited
capabilities of the common version of LAPACK at the time the wrappers
were done.

>  I think numpy should keep on wrapping the EVD routines (unless
>  someone manages to add the EVR routines to lapack_lite.c) and I
>  see no reason for scipy to wrap the old EV routines instead of the
>  EVD or EVR ones.

Agreed.

> - In my opinion, scipy.linalg.eigh should take over the
>  functionality now offered by symeig. Note that this would be
>  consistent with what is already done with linalg.eig: the scipy
>  version adds generalized problem functionality to the standard
>  problem functionality already available in numpy (as a plus eigh
>  would also grow the possibility to request just
>  a subset of the eigenvs).

Agreed.

> - While developing symeig with tested the EVR routines: they are
>  really better, so I see no reason not to use them by default. On
>  top of that, they fail when a matrix is nearly singular, while EV
>  and EVD tend to give wrong results without failing. When an EVR
>  routine fails the user can be advised that linalg.svd may be
>  better for his case.

When were they introduced to LAPACK? We may need to fall back on the
EVD routines if the EVR routines are not present.

> - The pyf description files are (if I am not mistaken) automatically
>  generated. Should I try to let the generator create pyf files for
>  our routines, or do you want to have the little (?) inconsistency
>  of having some hand-made pyf in the distribution?

They are f2py-generated, then edited. You should use f2py to create
descriptions the new subroutines, then add them to the appropriate
.pyf files, then edit them by hand to provide the missing information.

> - In the symeig test suite we use two very useful routines:
>  random_rot and symrand. Their docstrings:
>
>  random_rot:
>    """Return a random rotation matrix, drawn from the Haar distribution
>    (the only uniform distribution on SO(n)).
>    The algorithm is described in the paper
>    Stewart, G.W., "The efficient generation of random orthogonal
>    matrices with an application to condition estimators", SIAM Journal
>    on Numerical Analysis, 17(3), pp. 403-409, 1980.
>    For more information see
>    http://en.wikipedia.org/wiki/Orthogonal_matrix#Randomization"""
>
>  symrand:
>    """Return a random symmetric (Hermitian) matrix.
>
>    If 'dim_or_eigv' is an integer N, return a NxN matrix, with eigenvalues
>        uniformly distributed on (0,1].
>
>    If 'dim_or_eigv' is  1-D real array 'a', return a matrix whose
>                      eigenvalues are sort(a)."""
>
>  Do you think these routines may be a useful addition to scipy, or
>  should I keep them hidden in the tests?

Add them, please. I have needed at least the first one on a few occasions.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From wnbell at gmail.com  Tue Oct 28 13:57:20 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Tue, 28 Oct 2008 13:57:20 -0400
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
In-Reply-To: <3d375d730810281043q4b3ce5cft5f0c2b4337054850@mail.gmail.com>
References: <20081028173055.GA15367@localhost>
	<3d375d730810281043q4b3ce5cft5f0c2b4337054850@mail.gmail.com>
Message-ID: <d05265cb0810281057k2a465a9tbdd8fc05c30fdce@mail.gmail.com>

On Tue, Oct 28, 2008 at 1:43 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>  Do you think these routines may be a useful addition to scipy, or
>>  should I keep them hidden in the tests?
>
> Add them, please. I have needed at least the first one on a few occasions.
>

Where should they live?  Should we create scipy.gallery or something equivalent?

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From robert.kern at gmail.com  Tue Oct 28 14:03:46 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 28 Oct 2008 13:03:46 -0500
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
In-Reply-To: <d05265cb0810281057k2a465a9tbdd8fc05c30fdce@mail.gmail.com>
References: <20081028173055.GA15367@localhost>
	<3d375d730810281043q4b3ce5cft5f0c2b4337054850@mail.gmail.com>
	<d05265cb0810281057k2a465a9tbdd8fc05c30fdce@mail.gmail.com>
Message-ID: <3d375d730810281103u15f5feceuc05b5b37b066ee04@mail.gmail.com>

On Tue, Oct 28, 2008 at 12:57, Nathan Bell <wnbell at gmail.com> wrote:
> On Tue, Oct 28, 2008 at 1:43 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>>  Do you think these routines may be a useful addition to scipy, or
>>>  should I keep them hidden in the tests?
>>
>> Add them, please. I have needed at least the first one on a few occasions.
>
> Where should they live?  Should we create scipy.gallery or something equivalent?

Not sure. My uses for them were statistical in nature, not just for
generating arbitrary-within-constraints test cases. I guess I'd prefer
them to live "primarily" in linalg over gallery. A gallery package
would be useful, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From wnbell at gmail.com  Tue Oct 28 14:19:45 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Tue, 28 Oct 2008 14:19:45 -0400
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
In-Reply-To: <3d375d730810281103u15f5feceuc05b5b37b066ee04@mail.gmail.com>
References: <20081028173055.GA15367@localhost>
	<3d375d730810281043q4b3ce5cft5f0c2b4337054850@mail.gmail.com>
	<d05265cb0810281057k2a465a9tbdd8fc05c30fdce@mail.gmail.com>
	<3d375d730810281103u15f5feceuc05b5b37b066ee04@mail.gmail.com>
Message-ID: <d05265cb0810281119s43e8700ev65a97028a90c47fc@mail.gmail.com>

On Tue, Oct 28, 2008 at 2:03 PM, Robert Kern <robert.kern at gmail.com> wrote:
>
> Not sure. My uses for them were statistical in nature, not just for
> generating arbitrary-within-constraints test cases. I guess I'd prefer
> them to live "primarily" in linalg over gallery. A gallery package
> would be useful, though.
>

Fair enough.  We should return to this question after 0.7 is released.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From nwagner at iam.uni-stuttgart.de  Tue Oct 28 15:12:53 2008
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 28 Oct 2008 20:12:53 +0100
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
In-Reply-To: <d05265cb0810281119s43e8700ev65a97028a90c47fc@mail.gmail.com>
References: <20081028173055.GA15367@localhost>
	<3d375d730810281043q4b3ce5cft5f0c2b4337054850@mail.gmail.com>
	<d05265cb0810281057k2a465a9tbdd8fc05c30fdce@mail.gmail.com>
	<3d375d730810281103u15f5feceuc05b5b37b066ee04@mail.gmail.com>
	<d05265cb0810281119s43e8700ev65a97028a90c47fc@mail.gmail.com>
Message-ID: <web-113421606@uni-stuttgart.de>

On Tue, 28 Oct 2008 14:19:45 -0400
  "Nathan Bell" <wnbell at gmail.com> wrote:
> On Tue, Oct 28, 2008 at 2:03 PM, Robert Kern 
><robert.kern at gmail.com> wrote:
>>
>> Not sure. My uses for them were statistical in nature, 
>>not just for
>> generating arbitrary-within-constraints test cases. I 
>>guess I'd prefer
>> them to live "primarily" in linalg over gallery. A 
>>gallery package
>> would be useful, though.
>>
> 
>Fair enough.  We should return to this question after 0.7 
>is released.
> 
I look forward to seeing the improvements in the trunk.

The following tickets are inherently connected

http://projects.scipy.org/scipy/scipy/ticket/632
http://projects.scipy.org/scipy/scipy/ticket/630
http://projects.scipy.org/scipy/scipy/ticket/456

Cheers,
            Nils


From dominique.orban at gmail.com  Tue Oct 28 16:46:53 2008
From: dominique.orban at gmail.com (Dominique Orban)
Date: Tue, 28 Oct 2008 16:46:53 -0400
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
In-Reply-To: <web-113421606@uni-stuttgart.de>
References: <20081028173055.GA15367@localhost>
	<3d375d730810281043q4b3ce5cft5f0c2b4337054850@mail.gmail.com>
	<d05265cb0810281057k2a465a9tbdd8fc05c30fdce@mail.gmail.com>
	<3d375d730810281103u15f5feceuc05b5b37b066ee04@mail.gmail.com>
	<d05265cb0810281119s43e8700ev65a97028a90c47fc@mail.gmail.com>
	<web-113421606@uni-stuttgart.de>
Message-ID: <8793ae6e0810281346w590a5c5dqd619ea205f9baad5@mail.gmail.com>

On Tue, Oct 28, 2008 at 3:12 PM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> On Tue, 28 Oct 2008 14:19:45 -0400
>  "Nathan Bell" <wnbell at gmail.com> wrote:
>> On Tue, Oct 28, 2008 at 2:03 PM, Robert Kern
>><robert.kern at gmail.com> wrote:
>>>
>>> Not sure. My uses for them were statistical in nature,
>>>not just for
>>> generating arbitrary-within-constraints test cases. I
>>>guess I'd prefer
>>> them to live "primarily" in linalg over gallery. A
>>>gallery package
>>> would be useful, though.
>>>
>>
>>Fair enough.  We should return to this question after 0.7
>>is released.
>>
> I look forward to seeing the improvements in the trunk.
>
> The following tickets are inherently connected
>
> http://projects.scipy.org/scipy/scipy/ticket/632
> http://projects.scipy.org/scipy/scipy/ticket/630
> http://projects.scipy.org/scipy/scipy/ticket/456
>
> Cheers,
>            Nils
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>

Sorry if I'm catching this conversation a bit late and sorry if this
has been mentioned before. I'm wondering whether dense linear algebra
users would not be better served if all LAPACK and BLAS-related
interfaces were in Numpy instead of Scipy. After all, it is Numpy that
defines the array and dense matrix types. Why not make sure that Numpy
provides a complete and consistent interface to all BLAS 1, 2, 3 and
LAPACK functions?

I mention this for several reasons:

1. (as mentioned earlier), there are inconsistencies between the Numpy
and Scipy interfaces to LAPACK. It seems to me that a Numpy user is
also a good candidate to be a dense linear algebra user.

2. In my experience, Scipy is substantially more complex to install
than Numpy. As a user, if I can have access to all my dense linear
algebra in Numpy, it can save me time and trouble, especially if I
don't need the more advanced packages that Scipy offers.

3. As a wild guess, this might also avoid work duplication. I see that
other packages define new dense matrix types and BLAS and LAPACK
interfaces (such as cvxopt). Might it be because they don't want to
require from their users that they install Scipy?

Also, is there a profound reason to have a trimmed down version of
LAPACK only in Numpy?

Thanks for any comments that might help me understand all this.

-- 
Dominique


From robert.kern at gmail.com  Tue Oct 28 17:16:26 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 28 Oct 2008 16:16:26 -0500
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
In-Reply-To: <8793ae6e0810281346w590a5c5dqd619ea205f9baad5@mail.gmail.com>
References: <20081028173055.GA15367@localhost>
	<3d375d730810281043q4b3ce5cft5f0c2b4337054850@mail.gmail.com>
	<d05265cb0810281057k2a465a9tbdd8fc05c30fdce@mail.gmail.com>
	<3d375d730810281103u15f5feceuc05b5b37b066ee04@mail.gmail.com>
	<d05265cb0810281119s43e8700ev65a97028a90c47fc@mail.gmail.com>
	<web-113421606@uni-stuttgart.de>
	<8793ae6e0810281346w590a5c5dqd619ea205f9baad5@mail.gmail.com>
Message-ID: <3d375d730810281416x65d57fuda4a987c77ea56fd@mail.gmail.com>

On Tue, Oct 28, 2008 at 15:46, Dominique Orban
<dominique.orban at gmail.com> wrote:

> 2. In my experience, Scipy is substantially more complex to install
> than Numpy. As a user, if I can have access to all my dense linear
> algebra in Numpy, it can save me time and trouble, especially if I
> don't need the more advanced packages that Scipy offers.

Needing a complete BLAS/LAPACK is one of the reasons scipy is more
difficult to install. Moving it all to numpy doesn't solve this
problem.

> 3. As a wild guess, this might also avoid work duplication. I see that
> other packages define new dense matrix types and BLAS and LAPACK
> interfaces (such as cvxopt). Might it be because they don't want to
> require from their users that they install Scipy?

cvxopt doesn't even use *numpy*.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dominique.orban at gmail.com  Tue Oct 28 21:53:38 2008
From: dominique.orban at gmail.com (Dominique Orban)
Date: Tue, 28 Oct 2008 21:53:38 -0400
Subject: [SciPy-dev] eig, eigh, and symeig in scipy
In-Reply-To: <3d375d730810281416x65d57fuda4a987c77ea56fd@mail.gmail.com>
References: <20081028173055.GA15367@localhost>
	<3d375d730810281043q4b3ce5cft5f0c2b4337054850@mail.gmail.com>
	<d05265cb0810281057k2a465a9tbdd8fc05c30fdce@mail.gmail.com>
	<3d375d730810281103u15f5feceuc05b5b37b066ee04@mail.gmail.com>
	<d05265cb0810281119s43e8700ev65a97028a90c47fc@mail.gmail.com>
	<web-113421606@uni-stuttgart.de>
	<8793ae6e0810281346w590a5c5dqd619ea205f9baad5@mail.gmail.com>
	<3d375d730810281416x65d57fuda4a987c77ea56fd@mail.gmail.com>
Message-ID: <8793ae6e0810281853h1a8c0e15qd24d851c48e07164@mail.gmail.com>

On Tue, Oct 28, 2008 at 5:16 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Oct 28, 2008 at 15:46, Dominique Orban
> <dominique.orban at gmail.com> wrote:
>
>> 2. In my experience, Scipy is substantially more complex to install
>> than Numpy. As a user, if I can have access to all my dense linear
>> algebra in Numpy, it can save me time and trouble, especially if I
>> don't need the more advanced packages that Scipy offers.
>
> Needing a complete BLAS/LAPACK is one of the reasons scipy is more
> difficult to install. Moving it all to numpy doesn't solve this
> problem.

>> 3. As a wild guess, this might also avoid work duplication. I see that
>> other packages define new dense matrix types and BLAS and LAPACK
>> interfaces (such as cvxopt). Might it be because they don't want to
>> require from their users that they install Scipy?
>
> cvxopt doesn't even use *numpy*.

I understand that. cvxopt redefines everything and then ensures
compatibility with numpy arrays. I'm not entirely sure how complete
their interface to BLAS/LAPACK is, but it is a breeze to install.

Dominique


From josef.pktd at gmail.com  Wed Oct 29 00:49:03 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 29 Oct 2008 00:49:03 -0400
Subject: [SciPy-dev] Recipe: distribution of non-linear transformation of
	continuous variables
Message-ID: <1cd32cbb0810282149y2893d90asca5dac38057c417b@mail.gmail.com>

I wanted to check out the subclassing of scipy.stats.distributions,
and wrote a class that
provides the distribution of a non-linear monotonic transformation  of
a continuous random variable.
The transformation can be increasing or decreasing.

Code and examples should be easily readable.  At the bottom is (most
of) the printout when I run this script.
It works pretty well for "well-behaved" transformations.

I hope Google does not screw up the formatting too much, and I hope
somebody finds this useful.
Josef

----------------------------- begin file nonlinear_transform_gen.py
----------------------------------

''' A class for the distribution of a non-linear monotonic
transformation of a continuous random variable

simplest usage:
example: create log-gamma distribution, i.e. y = log(x), where x is
gamma distributed (also available in scipy.stats)
    loggammaexpg = Transf_gen(stats.gamma, np.log, np.exp)

example: what is the distribution of the discount factor y=1/(1+x)
                      where interest rate x is normally distributed
with N(mux,stdx**2)')?
                      (just to come up with a story that implies a
nice transformation)
    invnormalg = Transf_gen(stats.norm, inversew, inversew_inv,
decr=True, a=-np.inf)

This class does not work well for distributions with difficult shapes,
    e.g. 1/x where x is standard normal, because of the singularity
and jump at zero.

Note: I'm working from my version of scipy.stats.distribution.
         But this script runs under scipy 0.6.0 (checked with numpy:
1.2.0rc2 and python 2.4)
'''

from scipy import integrate # for scipy 0.6.0

from scipy import stats, info
from scipy.stats import distributions
import numpy as np



class Transf_gen(distributions.rv_continuous):
    #a class for non-linear monotonic transformation of a continuous
random variable
    def __init__(self, kls, func, funcinv, *args, **kwargs):
        #print args
        #print kwargs
        self.kls = kls
        self.func = func
        self.funcinv = funcinv
        #explicit for self.__dict__.update(kwargs)
        if 'numargs' in kwargs:
            self.numargs = kwargs['numargs']
        else:
            self.numargs = 1
        if 'name' in kwargs:
            name = kwargs['name']
        else:
            name = 'transfdist'
        if 'longname' in kwargs:
            longname = kwargs['longname']
        else:
            longname = 'Non-linear transformed distribution'
        if 'extradoc' in kwargs:
            extradoc = kwargs['extradoc']
        else:
            extradoc = None
        if 'a' in kwargs:
            a = kwargs['a']
        else:
            a = 0
        if 'decr' in kwargs:
            #defines whether it is a decreasing (True)
            #       or increasing (False) monotone transformation
            self.decr = kwargs['decr']
        else:
            self.decr = False

        super(Transf_gen,self).__init__(a=a, name = name,
                                longname = longname, extradoc = extradoc)

    def _cdf(self,x,*args, **kwargs):
        #print args
        if not self.decr:
            return self.kls._cdf(self.funcinv(x),*args, **kwargs)
        else:
            return 1-self.kls.cdf(self.funcinv(x),*args, **kwargs)
    def _ppf(self, q, *args, **kwargs):
        if not self.decr:
            return self.func(self.kls._ppf(q,*args, **kwargs))
        else:
            return self.func(self.kls._ppf(1-q,*args, **kwargs))


def inverse(x):
    return np.divide(1.0,x)

mux, stdx = 0.05, 0.1
def inversew(x):
    return 1.0/(1+mux+x*stdx)
def inversew_inv(x):
    return (1.0/x - 1.0 - mux)/stdx #.np.divide(1.0,x)-10

def identit(x):
    return x





print '\nResults for invnormal with regularization y=1/(1+x), x is
N(0.05, 0.1*0.1)'
print   '-----------------------------------------------------------------------'
invnormalg = Transf_gen(stats.norm, inversew, inversew_inv, decr=True,
a=-np.inf,
                name = 'discf', longname = 'normal-based discount factor',
                extradoc = '\ndistribution of discount factor
y=1/(1+x)) with x N(0.05, 0.1**2)')

l,s = 0,1
print
print invnormalg.__doc__
print
print 'cdf for [0.95,1.0,1.1]:', invnormalg.cdf([0.95,1.0,1.1],loc=l, scale=s)
print 'pdf for [0.95,1.0,1.1]:', invnormalg.pdf([0.95,1.0,1.1],loc=l, scale=s)

print 'rvs:', invnormalg.rvs(loc=l, scale=s,size=5)
print 'stats: ', invnormalg.stats(loc=l, scale=s)
print 'stats kurtosis, skew: ', invnormalg.stats(moments='ks')
print 'median:', invnormalg.ppf(0.5)
rvs = invnormalg.rvs(loc=l, scale=s,size=10000)
print 'sample stats:              ', rvs.mean(), rvs.var()
rvs = inversew(stats.norm.rvs(loc=l, scale=s,size=10000))
print 'std norm sample stats:    ', rvs.mean(), rvs.var()
rvs = inversew(stats.norm.rvs(size=100000))*s + l
print 'transf. norm sample stats: ', rvs.mean(), rvs.var()


print
print 'Results for lognormal'
print '---------------------'

lognormalg = Transf_gen(stats.norm, np.exp, np.log,
                a=0, name = 'lnnorm', longname = 'Exp transformed normal',
                extradoc = '\ndistribution of y = exp(x), with x
standard normal')
print 'cdf for [2.0,2.5,3.0,3.5]:      ',lognormalg.cdf([2.0,2.5,3.0,3.5])
print 'scipy cdf for [2.0,2.5,3.0,3.5]:', stats.lognorm.cdf([2.0,2.5,3.0,3.5],1)
print 'pdf for [2.0,2.5,3.0,3.5]:     ', lognormalg.pdf([2.0,2.5,3.0,3.5])
print 'scipy pdf for [2.0,2.5,3.0,3.5]:', stats.lognorm.pdf([2.0,2.5,3.0,3.5],1)
print 'stats:      ', lognormalg.stats()
print 'scipy stats:', stats.lognorm.stats(1)
print 'rvs:', lognormalg.rvs(size=5)
print info(lognormalg)


print '\nResults for idnormal'
print   '--------------------'
idnormalg = Transf_gen(stats.norm, identit, identit, a=-np.inf, name = 'normal')
print idnormalg.cdf(1,loc=100,scale=10)
print stats.norm.cdf(1,loc=100,scale=10)
print idnormalg.stats(loc=100,scale=10)
print stats.norm.stats(loc=100,scale=10)
print idnormalg.rvs(loc=100,scale=10,size=5)
rvs = idnormalg.rvs(loc=100,scale=10,size=10000)
print rvs.mean(), rvs.var()



print '\nResults for expgamma'
print   '--------------------'
loggammaexpg = Transf_gen(stats.gamma, np.log, np.exp)
print loggammaexpg._cdf(1,10)
print stats.loggamma.cdf(1,10)
print loggammaexpg._cdf(2,15)
print stats.loggamma.cdf(2,15)

print 'cdf for [2.0,2.5,3.0,3.5]:      ',
loggammaexpg._cdf([2.0,2.5,3.0,3.5],10)
print 'scipy cdf for [2.0,2.5,3.0,3.5]:',
stats.loggamma.cdf([2.0,2.5,3.0,3.5],10)
#print 'pdf for [2.0,2.5,3.0,3.5]:     ',
loggammaexpg.pdf([2.0,2.5,3.0,3.5],10) # not in scipy 0.6.0
print loggammaexpg._pdf(2.0,10) # ok in scipy 0.6.0
print 'scipy pdf for [2.0,2.5,3.0,3.5]:',
stats.loggamma.pdf([2.0,2.5,3.0,3.5],10)



print '\n\n the rest is not so good\n\n'


print '\nResults for invnormal'
print   '--------------------'
invnormalg2 = Transf_gen(stats.norm, inverse, inverse, decr=True, a=-np.inf,
                        longname = 'inverse normal')
print invnormalg2.cdf(1,loc=10)
print stats.invnorm.cdf(1,1,loc=10)
print invnormalg2.stats(loc=10)
print stats.invnorm.stats(1,loc=10)
print invnormalg2.rvs(loc=10,size=5)



print '\nResults for invexpon'
print   '--------------------'
invexpong = Transf_gen(stats.expon, inverse, inverse, a=0.0, name =
'Log transformed normal general')
print invexpong.cdf(1)
#print stats.invnorm.cdf(1,1)
print invexpong.stats()
#print stats.invnorm.stats(1)
print invexpong.rvs(size=5)

print '\nResults for inv gamma'
print   '---------------------'
invgammag = Transf_gen(stats.gamma, inverse, inverse, a=0.0, name =
'Log transformed normal general')


print invgammag._cdf(1,5) # .cdf  #not in scipy 0.6.0
print stats.invgamma.cdf(1,1)
#print invgammag.stats(1) #not in scipy 0.6.0
print stats.invgamma.stats(1)
#print invgammag.rvs(1,size=5) #not in scipy 0.6.0
print invgammag._rvs(1)




print '\nResults for loglaplace'
print   '----------------------'
#no idea what the relationship is
loglaplaceg = Transf_gen(stats.laplace, np.log, np.exp)
print loglaplaceg._cdf(2)
print stats.loglaplace.cdf(2,10)
loglaplaceexpg = Transf_gen(stats.laplace, np.exp, np.log)
print loglaplaceexpg._cdf(2)


#this are the results, that I get:
results = \
'''
Results for invnormal with regularization y=1/(1+x), x is N(0, 0.1*0.1)
-----------------------------------------------------------------------

normal-based discount factor continuous random variable.

    Continuous random variables are defined from a standard form chosen
    for simplicity of representation.  The standard form may require
    some shape parameters to complete its specification.  The distributions
    also take optional location and scale parameters using loc= and scale=
    keywords (defaults: loc=0, scale=1)

    These shape, scale, and location parameters can be passed to any of the
    methods of the RV object such as the following:

    discf.rvs(loc=0,scale=1)
        - random variates

    discf.pdf(x,loc=0,scale=1)
        - probability density function

    discf.cdf(x,loc=0,scale=1)
        - cumulative density function

    discf.sf(x,loc=0,scale=1)
        - survival function (1-cdf --- sometimes more accurate)

    discf.ppf(q,loc=0,scale=1)
        - percent point function (inverse of cdf --- percentiles)

    discf.isf(q,loc=0,scale=1)
        - inverse survival function (inverse of sf)

    discf.stats(loc=0,scale=1,moments='mv')
        - mean('m',axis=0), variance('v'), skew('s'), and/or kurtosis('k')

    discf.entropy(loc=0,scale=1)
        - (differential) entropy of the RV.

    Alternatively, the object may be called (as a function) to fix
       the shape, location, and scale parameters returning a
       "frozen" continuous RV object:

    myrv = discf(loc=0,scale=1)
        - frozen RV object with the same methods but holding the
            given shape, location, and scale fixed

distribution of discount factor y=1/(1+x)) with x N(0,0.1**2)

cdf for [0.95,1.0,1.1]: [ 0.48950273  0.69146246  0.92059586]
pdf for [0.95,1.0,1.1]: [ 4.41888273  3.52065327  1.22171744]
rvs: [ 0.83915166  1.05916973  0.92895054  0.97653308  0.88997135]
stats: (array(0.9612657841613822), array(0.0088754849141799985))
stats kurtosis, skew:  (array(0.61482268025485831), array(0.78380821248995103))
median: 0.952380952381
sample stats:               0.962079869848 0.00884562090518
std norm sample stats:     0.961089776974 0.00874620747261
transf. norm sample stats:  0.961224369048 0.00883332533467

Results for lognormal
---------------------
cdf for [2.0,2.5,3.0,3.5]:       [ 0.7558914   0.82024279  0.86403139
0.89485401]
scipy cdf for [2.0,2.5,3.0,3.5]: [ 0.7558914   0.82024279  0.86403139
0.89485401]
pdf for [2.0,2.5,3.0,3.5]:      [ 0.15687402  0.10487107  0.07272826
0.05200533]
scipy pdf for [2.0,2.5,3.0,3.5]: [ 0.15687402  0.10487107  0.07272826
0.05200533]
stats:       (array(1.6487212707089971), array(4.6707742108633177))
scipy stats: (array(1.6487212707001282), array(4.670774270471604))
rvs: [ 0.45054343  0.17302502  0.15131355  0.74181241  0.30077315]
Exp transformed normal continuous random variable.

    Continuous random variables are defined from a standard form chosen
    for simplicity of representation.  The standard form may require
    some shape parameters to complete its specification.  The distributions
    also take optional location and scale parameters using loc= and scale=
    keywords (defaults: loc=0, scale=1)

    These shape, scale, and location parameters can be passed to any of the
    methods of the RV object such as the following:

    lnnorm.rvs(loc=0,scale=1)
        - random variates

    lnnorm.pdf(x,loc=0,scale=1)
        - probability density function

    lnnorm.cdf(x,loc=0,scale=1)
        - cumulative density function

    lnnorm.sf(x,loc=0,scale=1)
        - survival function (1-cdf --- sometimes more accurate)

    lnnorm.ppf(q,loc=0,scale=1)
        - percent point function (inverse of cdf --- percentiles)

    lnnorm.isf(q,loc=0,scale=1)
        - inverse survival function (inverse of sf)

    lnnorm.stats(loc=0,scale=1,moments='mv')
        - mean('m',axis=0), variance('v'), skew('s'), and/or kurtosis('k')

    lnnorm.entropy(loc=0,scale=1)
        - (differential) entropy of the RV.

    Alternatively, the object may be called (as a function) to fix
       the shape, location, and scale parameters returning a
       "frozen" continuous RV object:

    myrv = lnnorm(loc=0,scale=1)
        - frozen RV object with the same methods but holding the
            given shape, location, and scale fixed

distribution of y = exp(x), with x standard normal
None

Results for idnormal
--------------------
2.08137521949e-023
2.08137521949e-023
(array(100.0), array(99.999999999476046))
(array(100.0), array(100.0))
[ 108.95135206  105.97192526   96.56950463  103.55316046   95.69133148]
99.9529496989 102.460238876

Results for expgamma
--------------------
0.00052773949978
0.00052773949978
0.00906529986325
0.00906529986325
cdf for [2.0,2.5,3.0,3.5]:       [ 0.21103986  0.77318778  0.99524757
0.99999926]
scipy cdf for [2.0,2.5,3.0,3.5]: [ 0.21103986  0.77318778  0.99524757
0.99999926]
pdf for [2.0,2.5,3.0,3.5]:      [  8.26228773e-01   1.01580211e+00
5.57228823e-02   1.81420198e-05]
scipy pdf for [2.0,2.5,3.0,3.5]: [  8.26228773e-01   1.01580211e+00
5.57228823e-02   1.81420259e-05]


 the rest is not so good
 '''


From david at ar.media.kyoto-u.ac.jp  Wed Oct 29 02:55:43 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 29 Oct 2008 15:55:43 +0900
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
Message-ID: <490808EF.1030305@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
>
> This needs to be handled carefully. scipy.fft() already exists. 

This scipy.fft is numpy one, right ? I thought those numpy import were
deprecated. If not, then it becomes much less clear to me to do the
change as suggested, indeed.

> It
> also breaks long-standing code (after the deprecation) for no reason
> that I find compelling. The name has been there for many years. The
> only complaint about it that I have ever seen on the lists was that
> people thought that only FFTPACK was supported.

This is only anecdotal, but I thought fftpack was confusing when I
started using scipy. I did not even know what fftpack was.

>  Given the following
> proposal, that reason goes away. Furthermore, I sometimes have trouble
> remembering whether the 'fft' symbol I have imported was numpy.fft or
> numpy.fft.fft. This change compounds that confusion.

Hm, I have the same problem, and that's why my goal for the renaming was
to be the same as numpy: numpy.fft and scipy.fft both are modules, and
numpy.fft.fft and scipy.fft.fft are fft functions.  So this intended,
although I can see why this can be seen as making it worse instead of
better, specially with scipy.fft already existing (which I did not know
existed, or at least forgot about).

>
> Overall, I don't think that mere renaming is ever a good reason to
> break code. It just feels gratuitous. Yes, there might be some benefit
> to having clearer names than what we have lived with so far (although
> in this case, I think this is dubious for the reasons given above),
> but there is always an enormous downside to breaking code. The
> deprecation policy does nothing to reduce that downside. It was never
> intended to reduce that downside. Right now, I'm almost sorry for
> having introduced it because it appears to be giving the impression
> that it is okay to make these kinds of changes.

I don't agree that deprecation policy does nothing to alleviate the
breaking code problem: if something breaks, it is much better to know it
with a deprecation warning than not knowing it and getting silent
breakages. I wish there were a deprecation warning when axis default
changed in numpy two years ago, for example. I understand the point that
introducing policies somewhat makes it easier to break API, though.

I don't want to start this discussion here, because that's a bit
off-topic, but I think we need to discuss those policies (for scipy - I
agree we should be really conservative for numpy) to reach an agreement
at some point.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Wed Oct 29 02:59:47 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 29 Oct 2008 15:59:47 +0900
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <e06186140810281004p3d8c5133k3c063481a306093e@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<e06186140810281004p3d8c5133k3c063481a306093e@mail.gmail.com>
Message-ID: <490809E3.2090302@ar.media.kyoto-u.ac.jp>

Charles R Harris wrote:
>
> +1. The fftpack is a pretty good (if old) implementation whose main
> failing is not taking maximum advantage of cache. It does need some
> supplementation, a Bluestein implementation for instance, but that is
> easier to do with a single target.

Bluestein would be useful even for numpy. The cache thing is handled
better by fftw2 and fftw3, but the advantage is largely outweight by the
fact we need to copy the input in a temporary buffer to guarantee SIMD
alignement. In a lot of signal processing tasks, float32 FFT is enough,
and that would give pretty good  speedup, too.

cheers,

David


From stefan at sun.ac.za  Wed Oct 29 03:43:17 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 29 Oct 2008 09:43:17 +0200
Subject: [SciPy-dev] Recipe: distribution of non-linear transformation
	of continuous variables
In-Reply-To: <1cd32cbb0810282149y2893d90asca5dac38057c417b@mail.gmail.com>
References: <1cd32cbb0810282149y2893d90asca5dac38057c417b@mail.gmail.com>
Message-ID: <9457e7c80810290043hbbead4bm4389457b2d0cfd9c@mail.gmail.com>

Hi Josef

2008/10/29  <josef.pktd at gmail.com>:
> I hope Google does not screw up the formatting too much, and I hope
> somebody finds this useful.

Would you please attach the code to a message, instead of including it
inline?  That makes it a lot easier to save it to file.

Thanks, this looks very interesting!

St?fan


From josef.pktd at gmail.com  Wed Oct 29 08:42:59 2008
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 29 Oct 2008 08:42:59 -0400
Subject: [SciPy-dev] Recipe: distribution of non-linear transformation
	of continuous variables
In-Reply-To: <9457e7c80810290043hbbead4bm4389457b2d0cfd9c@mail.gmail.com>
References: <1cd32cbb0810282149y2893d90asca5dac38057c417b@mail.gmail.com>
	<9457e7c80810290043hbbead4bm4389457b2d0cfd9c@mail.gmail.com>
Message-ID: <1cd32cbb0810290542y46652eb0xa96293d3d3f2b81@mail.gmail.com>

Here is the script file as an attachment.

Josef

On Wed, Oct 29, 2008 at 3:43 AM, St?fan van der Walt <stefan at sun.ac.za> wrote:
> Hi Josef
>
> 2008/10/29  <josef.pktd at gmail.com>:
>> I hope Google does not screw up the formatting too much, and I hope
>> somebody finds this useful.
>
> Would you please attach the code to a message, instead of including it
> inline?  That makes it a lot easier to save it to file.
>
> Thanks, this looks very interesting!
>
> St?fan
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: nonlinear_transform_gen.py
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081029/c9432fa6/attachment.ksh>

From vanforeest at gmail.com  Wed Oct 29 09:53:51 2008
From: vanforeest at gmail.com (nicky van foreest)
Date: Wed, 29 Oct 2008 14:53:51 +0100
Subject: [SciPy-dev] Of use and abuse of sphinx's math directive
In-Reply-To: <9457e7c80810280233j618c95fcn997505098be63d23@mail.gmail.com>
References: <20081028081807.GB11820@phare.normalesup.org>
	<9457e7c80810280233j618c95fcn997505098be63d23@mail.gmail.com>
Message-ID: <fa510ff80810290653p720393fav742e44032ec7d05a@mail.gmail.com>

Hi,

Suppose for later uses you want to export the documentation to some
format (e.g. latex, mediawiki (?) ) in which you like to make the
difference explicit between n and $n$. This would entail a lot of
searching. As an anecdote, Knuth used to write 1 rather than $1$. When
he developped the Euler font with Zapf, it turned out that 1 had a
different appearance than $1$. Knuth is Knuth, so he changed in all 1,
2, \ldots in one of his books that used the Euler font to $1$, $2$,
etc.

bye

Nicky

2008/10/28 St?fan van der Walt <stefan at sun.ac.za>:
> 2008/10/28 Gael Varoquaux <gael.varoquaux at normalesup.org>:
>> Sphinx is great, will all love Sphinx. But I do wonder if sometimes we do
>> not use the math directive a bit too much, eg the "n" and "n-1" on:
>> http://scipy.org/scipy/scipy/browser/trunk/scipy/cluster/hierarchy.py#L1090
>> (this is just an example that I picked randomly).
>>
>> When the formula written is simple-enought that its readibility is not
>> too much affected by having it in text in the Sphinxified document, I
>> think it is a pitty to render the plain text docstrings hard-to-read.
>
> Certainly -- the idea is to use the :math: tag sparingly.
>
> Cheers
> St?fan
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>


From cournape at gmail.com  Wed Oct 29 10:10:48 2008
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 29 Oct 2008 23:10:48 +0900
Subject: [SciPy-dev] Of use and abuse of sphinx's math directive
In-Reply-To: <fa510ff80810290653p720393fav742e44032ec7d05a@mail.gmail.com>
References: <20081028081807.GB11820@phare.normalesup.org>
	<9457e7c80810280233j618c95fcn997505098be63d23@mail.gmail.com>
	<fa510ff80810290653p720393fav742e44032ec7d05a@mail.gmail.com>
Message-ID: <5b8d13220810290710j69a68df7qedb598b97b9e0ee3@mail.gmail.com>

On Wed, Oct 29, 2008 at 10:53 PM, nicky van foreest
<vanforeest at gmail.com> wrote:
> Hi,
>
> Suppose for later uses you want to export the documentation to some
> format (e.g. latex, mediawiki (?) ) in which you like to make the
> difference explicit between n and $n$. This would entail a lot of
> searching. As an anecdote, Knuth used to write 1 rather than $1$. When
> he developped the Euler font with Zapf, it turned out that 1 had a
> different appearance than $1$. Knuth is Knuth, so he changed in all 1,
> 2, \ldots in one of his books that used the Euler font to $1$, $2$,
> etc.

I am not sure the comparison is accurate: docstrings are not meant to
be edition-ready. They are meant to be read from the interpreter
mainly; for help meant to be put in a general documentation, I think
separate doc is better, and there you can go crazy about  latex and
co.

David


From millman at berkeley.edu  Wed Oct 29 10:52:22 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 29 Oct 2008 07:52:22 -0700
Subject: [SciPy-dev] ANN: NumPy 1.2.1
Message-ID: <c7009a550810290752p7bb1a282qe682db2efd7e262c@mail.gmail.com>

I'm pleased to announce the release of NumPy 1.2.1.

NumPy is the fundamental package needed for scientific computing with
Python.  It contains:

 * a powerful N-dimensional array object
 * sophisticated (broadcasting) functions
 * basic linear algebra functions
 * basic Fourier transforms
 * sophisticated random number capabilities
 * tools for integrating Fortran code.

Besides it's obvious scientific uses, NumPy can also be used as an
efficient multi-dimensional container of generic data. Arbitrary
data-types can be defined. This allows NumPy to seamlessly and
speedily integrate with a wide-variety of databases.

This bugfix release comes almost one month after the 1.2.0 release.
Please note that NumPy 1.2.1 requires Python 2.4 or greater.

For information, please see the release notes:
https://sourceforge.net/project/shownotes.php?release_id=636728&group_id=1369

You can download the release from here:
https://sourceforge.net/project/showfiles.php?group_id=1369

Thank you to everybody who contributed to this release.

Enjoy,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From david.huard at gmail.com  Wed Oct 29 12:41:08 2008
From: david.huard at gmail.com (David Huard)
Date: Wed, 29 Oct 2008 12:41:08 -0400
Subject: [SciPy-dev] Timeseries Unusual Behaviour
In-Reply-To: <loom.20081023T130519-312@post.gmane.org>
References: <loom.20081014T170518-386@post.gmane.org>
	<200810221452.20605.pgmdevlist@gmail.com>
	<loom.20081023T130519-312@post.gmane.org>
Message-ID: <91cf711d0810290941i5d15ca75wfb3b21de584cbdad@mail.gmail.com>

Pierre,

A similar problem occurs with fromrecords. Dates are ordered but not
the data that is passed.


Regards,

David

On Thu, Oct 23, 2008 at 9:12 AM, Dave <dave.hirschfeld at gmail.com> wrote:
> Pierre GM <pgmdevlist <at> gmail.com> writes:
>
>>
>> Dave,
>> The unusal behavior you documented (trying to create 2 series with the same
>> values and same DateArray as dates giving different answers depending on the
>> order) should now be fixed on SVN.
>> Thanks again for reporting.
>>
>
> Thanks for the explanation and the fix. I've updated from svn and noticed the
> autosort parameter which does make its behaviour more explicit :)
>
> Cheers,
> Dave
>
>
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>


From pgmdevlist at gmail.com  Wed Oct 29 12:47:31 2008
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 29 Oct 2008 12:47:31 -0400
Subject: [SciPy-dev] Timeseries Unusual Behaviour
In-Reply-To: <91cf711d0810290941i5d15ca75wfb3b21de584cbdad@mail.gmail.com>
References: <loom.20081014T170518-386@post.gmane.org>
	<loom.20081023T130519-312@post.gmane.org>
	<91cf711d0810290941i5d15ca75wfb3b21de584cbdad@mail.gmail.com>
Message-ID: <200810291247.32068.pgmdevlist@gmail.com>

David,
Thanks for reporting. Would you mind sending me a self-contained example ? 
That would really make things easier for debugging/testing.
Cheers.
P.


On Wednesday 29 October 2008 12:41:08 David Huard wrote:
> Pierre,
>
> A similar problem occurs with fromrecords. Dates are ordered but not
> the data that is passed.
>
>
> Regards,
>
> David


From jh at physics.ucf.edu  Wed Oct 29 12:58:38 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Wed, 29 Oct 2008 12:58:38 -0400
Subject: [SciPy-dev] Of use and abuse of sphinx's math directive
In-Reply-To: <mailman.1230.1225288437.2878.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.1230.1225288437.2878.scipy-dev@scipy.org>
Message-ID: <wl83aifz6bl.fsf@glup.physics.ucf.edu>

2008/10/28 St?fan van der Walt <stefan at sun.ac.za>:
> 2008/10/28 Gael Varoquaux <gael.varoquaux at normalesup.org>:
>> Sphinx is great, will all love Sphinx. But I do wonder if sometimes we do
>> not use the math directive a bit too much, eg the "n" and "n-1" on:
>> http://scipy.org/scipy/scipy/browser/trunk/scipy/cluster/hierarchy.py#L1090
>> (this is just an example that I picked randomly).
>>
>> When the formula written is simple-enought that its readibility is not
>> too much affected by having it in text in the Sphinxified document, I
>> think it is a pitty to render the plain text docstrings hard-to-read.
>
> Certainly -- the idea is to use the :math: tag sparingly.

This is a classic case of too many constraints.  Not only is it right
to do one thing in plain text and another in formatted print, but I
think the preferred solution will change from leave it alone (today)
to flag it (future).  The use of the plain text doc interface is
diminishing.  It will always have some uses, such as on a text-only
terminal or while editing a function in an ap like emacs (i.e.,
looking at the function's own docstring while editing the code).  But,
people are going to get used to doc interfaces more useful than
help(), and those will soon be able (optionally) to tell your browser
to pull up a given page rather than feeding you the help in the
terminal.

So my suggestion is that we keep the text clean for now, but we flag
such pages in a comment so that they're easy to find when we do want
to go through the pages and switch to tagging everything.  How about a
checkbox for "This page contains unflagged, inline math" on the doc
wiki.  That could generate a comment in the source code but not in the
docstring.  It would then be easy to identify the pages to fix if/when
we decide to do so.  Finding the locations in the page to do that
isn't hard and we're not talking the majority of pages.

--jh--


From jh at physics.ucf.edu  Wed Oct 29 13:23:51 2008
From: jh at physics.ucf.edu (jh at physics.ucf.edu)
Date: Wed, 29 Oct 2008 13:23:51 -0400
Subject: [SciPy-dev] Of use and abuse of sphinx's math directive
In-Reply-To: <mailman.9.1225299604.21432.scipy-dev@scipy.org>
	(scipy-dev-request@scipy.org)
References: <mailman.9.1225299604.21432.scipy-dev@scipy.org>
Message-ID: <wl81vxzz55k.fsf@glup.physics.ucf.edu>

"David Cournapeau" <cournape at gmail.com>:

> I am not sure the comparison is accurate: docstrings are not meant to
> be edition-ready. They are meant to be read from the interpreter
> mainly; for help meant to be put in a general documentation, I think
> separate doc is better, and there you can go crazy about  latex and
> co.

At the start of the doc marathon, we considered having an ueberdoc
that produced the text docstrings and the PDF and HTML manuals, but
decided that developers would be most likely to update docs if they
were the docstrings and not some separate set of files.  We knew this
would result in some formatting imperfections, but decided to favor
content over form.  We have one doc source, the docstrings.  Even the
general-interest docs (slicing, glossary, etc.) are docstrings.

E unum pluribus, I guess.

--jh--


From stefan at sun.ac.za  Wed Oct 29 13:35:00 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 29 Oct 2008 19:35:00 +0200
Subject: [SciPy-dev] Of use and abuse of sphinx's math directive
In-Reply-To: <wl81vxzz55k.fsf@glup.physics.ucf.edu>
References: <mailman.9.1225299604.21432.scipy-dev@scipy.org>
	<wl81vxzz55k.fsf@glup.physics.ucf.edu>
Message-ID: <9457e7c80810291035v7126217cq6b11ffba5f875ca4@mail.gmail.com>

2008/10/29  <jh at physics.ucf.edu>:

> E unum pluribus, I guess.

With its corollary "falsus in uno, falsus in omnibus"!

:)

St?fan


From ndbecker2 at gmail.com  Thu Oct 30 19:46:27 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 30 Oct 2008 19:46:27 -0400
Subject: [SciPy-dev] scipy.org is broken
Message-ID: <gedh0j$vta$1@ger.gmane.org>

Attempting to visit scipy.org gives me errors from moinmoin.  Interestingly, I had just finished fixing my own moinmoin site that was IIRC giving the same errors.



From robert.kern at gmail.com  Thu Oct 30 20:04:07 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 30 Oct 2008 19:04:07 -0500
Subject: [SciPy-dev] scipy.org is broken
In-Reply-To: <gedh0j$vta$1@ger.gmane.org>
References: <gedh0j$vta$1@ger.gmane.org>
Message-ID: <3d375d730810301704i1af61490j50ed78a09406675d@mail.gmail.com>

On Thu, Oct 30, 2008 at 18:46, Neal Becker <ndbecker2 at gmail.com> wrote:
> Attempting to visit scipy.org gives me errors from moinmoin.  Interestingly, I had just finished fixing my own moinmoin site that was IIRC giving the same errors.

Thanks. Working on it. No ETA, yet.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Oct 30 20:31:39 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 30 Oct 2008 19:31:39 -0500
Subject: [SciPy-dev] scipy.org is broken
In-Reply-To: <gedh0j$vta$1@ger.gmane.org>
References: <gedh0j$vta$1@ger.gmane.org>
Message-ID: <3d375d730810301731u757ff41bt251f5f0e76d4e2ba@mail.gmail.com>

On Thu, Oct 30, 2008 at 18:46, Neal Becker <ndbecker2 at gmail.com> wrote:
> Attempting to visit scipy.org gives me errors from moinmoin.  Interestingly, I had just finished fixing my own moinmoin site that was IIRC giving the same errors.

Fixed. Thank you for spotting this.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david.huard at gmail.com  Fri Oct 31 09:53:50 2008
From: david.huard at gmail.com (David Huard)
Date: Fri, 31 Oct 2008 09:53:50 -0400
Subject: [SciPy-dev] Timeseries Unusual Behaviour
In-Reply-To: <200810291247.32068.pgmdevlist@gmail.com>
References: <loom.20081014T170518-386@post.gmane.org>
	<loom.20081023T130519-312@post.gmane.org>
	<91cf711d0810290941i5d15ca75wfb3b21de584cbdad@mail.gmail.com>
	<200810291247.32068.pgmdevlist@gmail.com>
Message-ID: <91cf711d0810310653s62924955w1f101640f02d1c06@mail.gmail.com>

On Wed, Oct 29, 2008 at 12:47 PM, Pierre GM <pgmdevlist at gmail.com> wrote:

> David,
> Thanks for reporting. Would you mind sending me a self-contained example ?
> That would really make things easier for debugging/testing.
> Cheers.
> P.
>

Pierre,

here you are:

"""Test for recordarrays ordering at instantiation."""

import numpy as np
import scikits.timeseries as TS
from numpy.testing import *

# Arrays a and b are ordered in descending order, and the dates are in the
# inverse chronological order.
# Once the dates are ordered in chronological order, we expect the arrays a
and
# b to be in ascending order.

a = np.array([4,3,2,1])
b = np.array([14,13,12,11])
date_str = ['2000-01-01 04:00', '2000-01-01 03:00', '2000-01-01 02:00',
'2000-01-01 01:00']
dates = [TS.Date('S', date) for date in date_str]

trec = TS.fromrecords(zip(a,b), dates, names=['a', 'b'])

assert_array_equal(np.diff(trec.dates), 3600)   # dates are in chronological
order
assert_array_equal(np.diff(trec['a']), 1)  # This fails
assert_array_equal(np.diff(trec['b']), 1)  # This fails.




>
>
> On Wednesday 29 October 2008 12:41:08 David Huard wrote:
> > Pierre,
> >
> > A similar problem occurs with fromrecords. Dates are ordered but not
> > the data that is passed.
> >
> >
> > Regards,
> >
> > David
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20081031/0bed2ef4/attachment.html>

From stefan at sun.ac.za  Fri Oct 31 12:23:00 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 31 Oct 2008 18:23:00 +0200
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
Message-ID: <9457e7c80810310923m60aa9008pfef3f6a0ed70c45@mail.gmail.com>

2008/10/28 Robert Kern <robert.kern at gmail.com>:
>> How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
>> placeholder (like scipy.linsolve), and deprecating scipy.fftpack.
>
> This needs to be handled carefully. scipy.fft() already exists. It

scipy.fft() just contributes to SciPy root pollution.  Unlike with
NumPy, we still have the opportunity to organise SciPy well, and I
don't think fft() belongs at the top level.

> also breaks long-standing code (after the deprecation) for no reason
> that I find compelling. The name has been there for many years. The

Improving the package structure is a good reason, in my opinion.

> proposal, that reason goes away. Furthermore, I sometimes have trouble
> remembering whether the 'fft' symbol I have imported was numpy.fft or
> numpy.fft.fft. This change compounds that confusion.

Maybe, but I still find it better than having

numpy.almostanything -> scipy.almostanything

but

numpy.fft -> package while scipy.fft -> function

It seems more natural to have

module.fft -> package

module.fft.fft -> function

> Overall, I don't think that mere renaming is ever a good reason to
> break code. It just feels gratuitous. Yes, there might be some benefit
> to having clearer names than what we have lived with so far (although
> in this case, I think this is dubious for the reasons given above),
> but there is always an enormous downside to breaking code. The
> deprecation policy does nothing to reduce that downside. It was never
> intended to reduce that downside. Right now, I'm almost sorry for
> having introduced it because it appears to be giving the impression
> that it is okay to make these kinds of changes.

We're not at 1.0 yet, so we need to allow for some leeway in adjusting
the package to have the best possible layout.  The sooner we sort
these things out, the sooner we'll have an API worth freezing.

Regards
St?fan


From robert.kern at gmail.com  Fri Oct 31 15:25:10 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 31 Oct 2008 14:25:10 -0500
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <9457e7c80810310923m60aa9008pfef3f6a0ed70c45@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
	<9457e7c80810310923m60aa9008pfef3f6a0ed70c45@mail.gmail.com>
Message-ID: <3d375d730810311225w79ec9deey67d7e980eff798e6@mail.gmail.com>

On Fri, Oct 31, 2008 at 11:23, St?fan van der Walt <stefan at sun.ac.za> wrote:
> 2008/10/28 Robert Kern <robert.kern at gmail.com>:
>>> How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
>>> placeholder (like scipy.linsolve), and deprecating scipy.fftpack.
>>
>> This needs to be handled carefully. scipy.fft() already exists. It
>
> scipy.fft() just contributes to SciPy root pollution.  Unlike with
> NumPy, we still have the opportunity to organise SciPy well, and I
> don't think fft() belongs at the top level.

I agree that it doesn't belong there. Personally, I would like to
remove everything from the top-level. However, I don't particularly
agree that we have no constraints on reorganizing scipy. With numpy we
made a clear statement that the API was going to change up until 1.0,
at which point we made some promises of stability. Consequently, we
felt comfortable changing the API before 1.0 without concern for
breaking people's code. With scipy, that's not the case. 1.0 is not
and has never been the arbitrary dividing line between instability and
stability for this project. scipy is installed, people are using it,
and we have an obligation not to break their code for minor
tidying-up.

>> also breaks long-standing code (after the deprecation) for no reason
>> that I find compelling. The name has been there for many years. The
>
> Improving the package structure is a good reason, in my opinion.

There's a difference between "structure" and "naming".

>> proposal, that reason goes away. Furthermore, I sometimes have trouble
>> remembering whether the 'fft' symbol I have imported was numpy.fft or
>> numpy.fft.fft. This change compounds that confusion.
>
> Maybe, but I still find it better than having
>
> numpy.almostanything -> scipy.almostanything
>
> but
>
> numpy.fft -> package while scipy.fft -> function
>
> It seems more natural to have
>
> module.fft -> package
>
> module.fft.fft -> function
>
>> Overall, I don't think that mere renaming is ever a good reason to
>> break code. It just feels gratuitous. Yes, there might be some benefit
>> to having clearer names than what we have lived with so far (although
>> in this case, I think this is dubious for the reasons given above),
>> but there is always an enormous downside to breaking code. The
>> deprecation policy does nothing to reduce that downside. It was never
>> intended to reduce that downside. Right now, I'm almost sorry for
>> having introduced it because it appears to be giving the impression
>> that it is okay to make these kinds of changes.
>
> We're not at 1.0 yet, so we need to allow for some leeway in adjusting
> the package to have the best possible layout.  The sooner we sort
> these things out, the sooner we'll have an API worth freezing.

That has never been scipy's policy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From millman at berkeley.edu  Fri Oct 31 15:50:30 2008
From: millman at berkeley.edu (Jarrod Millman)
Date: Fri, 31 Oct 2008 12:50:30 -0700
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <3d375d730810311225w79ec9deey67d7e980eff798e6@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
	<9457e7c80810310923m60aa9008pfef3f6a0ed70c45@mail.gmail.com>
	<3d375d730810311225w79ec9deey67d7e980eff798e6@mail.gmail.com>
Message-ID: <c7009a550810311250u460d35a1wd04f3b146ac52733@mail.gmail.com>

On Fri, Oct 31, 2008 at 12:25 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> We're not at 1.0 yet, so we need to allow for some leeway in adjusting
>> the package to have the best possible layout.  The sooner we sort
>> these things out, the sooner we'll have an API worth freezing.
>
> That has never been scipy's policy.

It may not have been the 'official' policy, but I think that it is
what a fair number of people including (it seems) Stefan, David, and
myself believed was the policy.  I think a large number of people view
the 1.0 release of any project as a dividing line (arbitrary or not).
Furthermore, the development status of scipy has been advertised as
"Beta" for as long as I can remember:
http://pypi.python.org/pypi/scipy

Most importantly, scipy has the feel of an unfinished project.
Documentation, naming, calling conventions, and many other aspects of
it highlight its nature as a work in progress.  We need to come to
some agreement, I think, as how to proceed with the API changes that
we need to make and what the official policy for moving toward a 1.0
release should be.  Maybe we need to change the release numbers to
include some indication that the software is still rapidly evolving.
Personally, I am fine with using the pre-1.0 numbering to signify
this; but I am open to other ideas.  At the very least, we need to
decide whether we believe that we need to make significant changes to
the API and how to best convey that this is the case to our users.
Clearly, none of us want to ignore the needs of our users.  But we
need to balance the needs of current users to have relatively stable
dependencies with the needs of future users to have a highly polished,
cleanly designed, and easy to use packages for scientific computing in
Python.

Without thinking to hard, I can quickly think of a number of changes
that I would like to see prior to what I would consider a "1.0"
release.  For example, these top-level packages are good candidates, I
believe, for renaming, removing, or reintroducing:
 - scipy.fftpack
 - scipy.linsolv
 - scipy.stsci
 - scipy.misc
 - scipy.lib
 - scipy.ga

There are also a number of other API cleanups that I would like to see
before a 1.0 release.

Thoughts? Ideas?

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From wnbell at gmail.com  Fri Oct 31 16:14:06 2008
From: wnbell at gmail.com (Nathan Bell)
Date: Fri, 31 Oct 2008 16:14:06 -0400
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <c7009a550810311250u460d35a1wd04f3b146ac52733@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
	<9457e7c80810310923m60aa9008pfef3f6a0ed70c45@mail.gmail.com>
	<3d375d730810311225w79ec9deey67d7e980eff798e6@mail.gmail.com>
	<c7009a550810311250u460d35a1wd04f3b146ac52733@mail.gmail.com>
Message-ID: <d05265cb0810311314k12a92775w340f39b186ac3191@mail.gmail.com>

On Fri, Oct 31, 2008 at 3:50 PM, Jarrod Millman <millman at berkeley.edu> wrote:
> release.  For example, these top-level packages are good candidates, I
> believe, for renaming, removing, or reintroducing:
>  - scipy.linsolv[e]
>

FWIW scipy.linsolve is going to disappear after 0.7.  That
functionality now lives in scipy.sparse.linalg.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


From robert.kern at gmail.com  Fri Oct 31 16:40:06 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 31 Oct 2008 15:40:06 -0500
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <c7009a550810311250u460d35a1wd04f3b146ac52733@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
	<9457e7c80810310923m60aa9008pfef3f6a0ed70c45@mail.gmail.com>
	<3d375d730810311225w79ec9deey67d7e980eff798e6@mail.gmail.com>
	<c7009a550810311250u460d35a1wd04f3b146ac52733@mail.gmail.com>
Message-ID: <3d375d730810311340s592a729aj1eb587e72d3b4f7@mail.gmail.com>

On Fri, Oct 31, 2008 at 14:50, Jarrod Millman <millman at berkeley.edu> wrote:
> On Fri, Oct 31, 2008 at 12:25 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>> We're not at 1.0 yet, so we need to allow for some leeway in adjusting
>>> the package to have the best possible layout.  The sooner we sort
>>> these things out, the sooner we'll have an API worth freezing.
>>
>> That has never been scipy's policy.
>
> It may not have been the 'official' policy, but I think that it is
> what a fair number of people including (it seems) Stefan, David, and
> myself believed was the policy.  I think a large number of people view
> the 1.0 release of any project as a dividing line (arbitrary or not).
> Furthermore, the development status of scipy has been advertised as
> "Beta" for as long as I can remember:
> http://pypi.python.org/pypi/scipy
>
> Most importantly, scipy has the feel of an unfinished project.
> Documentation, naming, calling conventions, and many other aspects of
> it highlight its nature as a work in progress.  We need to come to
> some agreement, I think, as how to proceed with the API changes that
> we need to make and what the official policy for moving toward a 1.0
> release should be.  Maybe we need to change the release numbers to
> include some indication that the software is still rapidly evolving.
> Personally, I am fine with using the pre-1.0 numbering to signify
> this; but I am open to other ideas.

If you want to do that, fine. But make it public and official soon.
And when we get to 1.0, we need to keep our promises.

> At the very least, we need to
> decide whether we believe that we need to make significant changes to
> the API and how to best convey that this is the case to our users.
> Clearly, none of us want to ignore the needs of our users.  But we
> need to balance the needs of current users to have relatively stable
> dependencies with the needs of future users to have a highly polished,
> cleanly designed, and easy to use packages for scientific computing in
> Python.
>
> Without thinking to hard, I can quickly think of a number of changes
> that I would like to see prior to what I would consider a "1.0"
> release.  For example, these top-level packages are good candidates, I
> believe, for renaming, removing, or reintroducing:
>  - scipy.fftpack
>  - scipy.linsolv
>  - scipy.stsci
>  - scipy.misc
>  - scipy.lib
>  - scipy.ga
>
> There are also a number of other API cleanups that I would like to see
> before a 1.0 release.
>
> Thoughts? Ideas?

I am less opposed to substantive restructurings. They usually provide
a concrete, technical benefit to counter the concrete, technical costs
of having to change code. Simple renamings don't. Their benefits are
typically hypothetical or anecdotal. In the case of fftpack->fft, I
have given reasons why the new name would have problems, too (when
possible, never have a package/module named the same thing as a
function or class it provides), so I think any benefit from renaming
ultimately comes out as a wash. That just leaves us with the
substantial costs.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From stefan at sun.ac.za  Fri Oct 31 16:42:28 2008
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 31 Oct 2008 22:42:28 +0200
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <c7009a550810311250u460d35a1wd04f3b146ac52733@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
	<9457e7c80810310923m60aa9008pfef3f6a0ed70c45@mail.gmail.com>
	<3d375d730810311225w79ec9deey67d7e980eff798e6@mail.gmail.com>
	<c7009a550810311250u460d35a1wd04f3b146ac52733@mail.gmail.com>
Message-ID: <9457e7c80810311342i5d4d59f5g6314d736f4d2ed18@mail.gmail.com>

2008/10/31 Jarrod Millman <millman at berkeley.edu>:
> There are also a number of other API cleanups that I would like to see
> before a 1.0 release.
>
> Thoughts? Ideas?

I agree with the sentiments of your post.  As for the SciPy API, I
would prefer if we took a different route than we did with NumPy, and
made an explicit effort to review each part of the API before freezing
it, rather than simply waiting for development to "stabilise".  I was
very pleased with the recent discussions around scipy.spatial -- that
was a good example of how an API should be determined!

Regards
St?fan


From damian.eads.lists at gmail.com  Fri Oct 31 17:23:24 2008
From: damian.eads.lists at gmail.com (Damian Eads)
Date: Fri, 31 Oct 2008 14:23:24 -0700
Subject: [SciPy-dev] Reminder: SciPy Sprint is TOMORROW (Saturday) and Sunday
Message-ID: <91b4b1ab0810311423t1cc32c68x8501076730c15fa4@mail.gmail.com>

Hi there,

As upcoming events can slip people's minds, this is a reminder that
the sprint for SciPy is November 1-2, 2008, which starts tomorrow,
Saturday. There will be sprint teams working in Stellenbosch, South
Africa (led by Stefan van der Walt); Austin, TX; and Berkeley, CA (led
by Jarrod Millman).

Let's get our hands dirty...

Damian


-----------------------------------------------------
Damian Eads                             Ph.D. Student
Jack Baskin School of Engineering, UCSC        E2-489
1156 High Street                 Machine Learning Lab
Santa Cruz, CA 95064    http://www.soe.ucsc.edu/~eads


From oliphant at enthought.com  Fri Oct 31 18:32:16 2008
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 31 Oct 2008 17:32:16 -0500
Subject: [SciPy-dev] Reminder: SciPy Sprint is TOMORROW (Saturday) and
 Sunday
In-Reply-To: <91b4b1ab0810311423t1cc32c68x8501076730c15fa4@mail.gmail.com>
References: <91b4b1ab0810311423t1cc32c68x8501076730c15fa4@mail.gmail.com>
Message-ID: <490B8770.6040202@enthought.com>

Damian Eads wrote:
> Hi there,
>
> As upcoming events can slip people's minds, this is a reminder that
> the sprint for SciPy is November 1-2, 2008, which starts tomorrow,
> Saturday. There will be sprint teams working in Stellenbosch, South
> Africa (led by Stefan van der Walt); Austin, TX; and Berkeley, CA (led
> by Jarrod Millman).
>
> Let's get our hands dirty...
>   
This sounds great.    I'm excited to see what will come out of the 
effort.  Many thanks to those that participate.

-Travis





From cournape at gmail.com  Fri Oct 31 23:15:35 2008
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 1 Nov 2008 12:15:35 +0900
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <3d375d730810311225w79ec9deey67d7e980eff798e6@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
	<9457e7c80810310923m60aa9008pfef3f6a0ed70c45@mail.gmail.com>
	<3d375d730810311225w79ec9deey67d7e980eff798e6@mail.gmail.com>
Message-ID: <5b8d13220810312015g1366e5c0m821ad3ed893f6275@mail.gmail.com>

On Sat, Nov 1, 2008 at 4:25 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Fri, Oct 31, 2008 at 11:23, St?fan van der Walt <stefan at sun.ac.za> wrote:
>> 2008/10/28 Robert Kern <robert.kern at gmail.com>:
>>>> How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
>>>> placeholder (like scipy.linsolve), and deprecating scipy.fftpack.
>>>
>>> This needs to be handled carefully. scipy.fft() already exists. It
>>
>> scipy.fft() just contributes to SciPy root pollution.  Unlike with
>> NumPy, we still have the opportunity to organise SciPy well, and I
>> don't think fft() belongs at the top level.
>
> I agree that it doesn't belong there. Personally, I would like to
> remove everything from the top-level. However, I don't particularly
> agree that we have no constraints on reorganizing scipy. With numpy we
> made a clear statement that the API was going to change up until 1.0,
> at which point we made some promises of stability. Consequently, we
> felt comfortable changing the API before 1.0 without concern for
> breaking people's code. With scipy, that's not the case. 1.0 is not
> and has never been the arbitrary dividing line between instability and
> stability for this project. scipy is installed, people are using it,
> and we have an obligation not to break their code for minor
> tidying-up.

If we have this policy, it only makes sense to follow it strictly. API
compatibility is mostly a binary thing: either we allow breakage, or
we don't. To break "a bit" is the same as breaking a lot: it means
people can't rely on it as a stable dependency. That's the part of
your argument that I don't follow, I guess.

Again, I don't mind so much which decision is made, but it would be nice that:
  - the decision is made clearly, so that people are aware of it. At
least Stefan and me were not aware of it. How are we supposed to know
it ?
  - if we go toward API stability, the decision should be enforced. I
don't see how middle ground has any value: it just means people will
break it silently (as it has happened constantly in the past, at least
since I am following it) instead of discussing it.

cheers,

David

From robert.kern at gmail.com  Fri Oct 31 23:50:55 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 31 Oct 2008 22:50:55 -0500
Subject: [SciPy-dev] Renaming fftpack to fft, removing backends
In-Reply-To: <5b8d13220810312015g1366e5c0m821ad3ed893f6275@mail.gmail.com>
References: <4906D211.6070505@ar.media.kyoto-u.ac.jp>
	<3d375d730810281031s77008de4q1e956b28dd93e117@mail.gmail.com>
	<9457e7c80810310923m60aa9008pfef3f6a0ed70c45@mail.gmail.com>
	<3d375d730810311225w79ec9deey67d7e980eff798e6@mail.gmail.com>
	<5b8d13220810312015g1366e5c0m821ad3ed893f6275@mail.gmail.com>
Message-ID: <3d375d730810312050k65b1661et1078983efda255af@mail.gmail.com>

On Fri, Oct 31, 2008 at 22:15, David Cournapeau <cournape at gmail.com> wrote:
> On Sat, Nov 1, 2008 at 4:25 AM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Fri, Oct 31, 2008 at 11:23, St?fan van der Walt <stefan at sun.ac.za> wrote:
>>> 2008/10/28 Robert Kern <robert.kern at gmail.com>:
>>>>> How: For 0.7, we could rename it while keeping scipy.fftpack as an empty
>>>>> placeholder (like scipy.linsolve), and deprecating scipy.fftpack.
>>>>
>>>> This needs to be handled carefully. scipy.fft() already exists. It
>>>
>>> scipy.fft() just contributes to SciPy root pollution.  Unlike with
>>> NumPy, we still have the opportunity to organise SciPy well, and I
>>> don't think fft() belongs at the top level.
>>
>> I agree that it doesn't belong there. Personally, I would like to
>> remove everything from the top-level. However, I don't particularly
>> agree that we have no constraints on reorganizing scipy. With numpy we
>> made a clear statement that the API was going to change up until 1.0,
>> at which point we made some promises of stability. Consequently, we
>> felt comfortable changing the API before 1.0 without concern for
>> breaking people's code. With scipy, that's not the case. 1.0 is not
>> and has never been the arbitrary dividing line between instability and
>> stability for this project. scipy is installed, people are using it,
>> and we have an obligation not to break their code for minor
>> tidying-up.
>
> If we have this policy, it only makes sense to follow it strictly. API
> compatibility is mostly a binary thing: either we allow breakage, or
> we don't. To break "a bit" is the same as breaking a lot: it means
> people can't rely on it as a stable dependency. That's the part of
> your argument that I don't follow, I guess.

It's not true that it's entirely binary. Some changes are more severe
than others, and some changes have more benefits than others. Not all
parts of a library are widely used. Specifications of behavior are
sometimes vague or nonexistent, so changes to them might be
acceptable. APIs are sometimes buggy (not just their implementation)
and need to be fixed. Absolute, strict compatibility is impossible to
achieve while still  improving the software. But sensible compromises
can be made.

For example, Python itself is a pretty good project for maintaining
backwards compatibility. But behavior changes do happen, and
deeply-diving codebases like Twisted do break on just about every new
2.x release. Because the changes are usually beneficial to a wide
audience while the breakage is limited to small parts of libraries
that are depending on mostly unspecified behavior, these changes are
usually considered acceptable. To avoid this level of breakage would
mean stopping developing Python altogether (this is not my opinion,
but Martin van L?wis's, and he knows more about it than I).

> Again, I don't mind so much which decision is made, but it would be nice that:
>  - the decision is made clearly, so that people are aware of it. At
> least Stefan and me were not aware of it. How are we supposed to know
> it ?
>  - if we go toward API stability, the decision should be enforced. I
> don't see how middle ground has any value: it just means people will
> break it silently (as it has happened constantly in the past, at least
> since I am following it) instead of discussing it.

I don't see how it follows that taking my "middle ground" stance means
that people won't discuss possibly-code-breaking changes they are
making. It seems to me that it would encourage more discussion because
the tradeoffs need to be justified and agreed upon. Whereas taking the
"any code-breaking changes are acceptable before 1.0" seems like it
might let these changes get lost in the shuffle because no
justification needs to happen.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco