From dlrt2 at ast.cam.ac.uk  Mon Mar  1 06:46:42 2010
From: dlrt2 at ast.cam.ac.uk (David Trethewey)
Date: Mon, 01 Mar 2010 11:46:42 +0000
Subject: [SciPy-User] Error on importing scikits.learn
In-Reply-To: <4B8BA8B7.9040305@cam.ac.uk>
References: <4B8BA8B7.9040305@cam.ac.uk>
Message-ID: <4B8BA922.6010900@ast.cam.ac.uk>

I get the following error message when attempting to import the 
expectation maximisation (em) module from scikits.learn :
>
> >>> import scipy
> >>> import numpy
> >>> from scikits.learn.machine import em
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>    from online_em import OnGMM as _OnGMM
>  File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>    import _rawden
> ImportError: No module named _rawden
>
> Does anyone know what this _rawden module is and why might it not be 
> found?
>
> David Trethewey
> Institute of Astronomy, Cambridge, UK
>
>



From dlrt2 at ast.cam.ac.uk  Mon Mar  1 06:51:14 2010
From: dlrt2 at ast.cam.ac.uk (David Trethewey)
Date: Mon, 01 Mar 2010 11:51:14 +0000
Subject: [SciPy-User] Error on importing scikits.learn
In-Reply-To: <4B8BA922.6010900@ast.cam.ac.uk>
References: <4B8BA8B7.9040305@cam.ac.uk> <4B8BA922.6010900@ast.cam.ac.uk>
Message-ID: <4B8BAA32.3000004@ast.cam.ac.uk>

I have traced it back to an error in compiling scikits.learn itself:

building 'scikits.learn.machine.em._rawden' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 
-Wall -Wstrict-prototypes -fPIC

compile options: '-I/usr/lib/python2.6/dist-packages/numpy/core/include 
-I/usr/include/python2.6 -c'
gcc: scikits/learn/machine/em/src/pure_den.c
scikits/learn/machine/em/src/pure_den.c:7:20: error: Python.h: No such 
file or directory

David

David Trethewey wrote:
> I get the following error message when attempting to import the 
> expectation maximisation (em) module from scikits.learn :
>>
>> >>> import scipy
>> >>> import numpy
>> >>> from scikits.learn.machine import em
>> Traceback (most recent call last):
>>  File "<stdin>", line 1, in <module>
>>  File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>>    from online_em import OnGMM as _OnGMM
>>  File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>>    import _rawden
>> ImportError: No module named _rawden
>>
>> Does anyone know what this _rawden module is and why might it not be 
>> found?
>>
>> David Trethewey
>> Institute of Astronomy, Cambridge, UK
>>
>>
>
>



From helge.reikeras at gmail.com  Mon Mar  1 06:55:01 2010
From: helge.reikeras at gmail.com (Helge Reikeras)
Date: Mon, 01 Mar 2010 13:55:01 +0200
Subject: [SciPy-User] Error on importing scikits.learn
In-Reply-To: <4B8BAA32.3000004@ast.cam.ac.uk>
References: <4B8BA8B7.9040305@cam.ac.uk> <4B8BA922.6010900@ast.cam.ac.uk>
	<4B8BAA32.3000004@ast.cam.ac.uk>
Message-ID: <op.u8v0pqws9y0l3i@helge-desktop>

On Mon, 01 Mar 2010 13:51:14 +0200, David Trethewey <dlrt2 at ast.cam.ac.uk>  
wrote:

>
> compile options: '-I/usr/lib/python2.6/dist-packages/numpy/core/include
> -I/usr/include/python2.6 -c'
> gcc: scikits/learn/machine/em/src/pure_den.c
> scikits/learn/machine/em/src/pure_den.c:7:20: error: Python.h: No such
> file or directory

This is because you don't have the Python header files installed. You need  
to install these somehow. E.g. if you are running Ubuntu you do "sudo  
apt-get install python-dev".

-- 
Helge Reikeras
http://www.google.com/profiles/helge.reikeras


From cimrman3 at ntc.zcu.cz  Mon Mar  1 12:20:36 2010
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Mon, 01 Mar 2010 18:20:36 +0100
Subject: [SciPy-User] ANN: SfePy 2010.1
Message-ID: <4B8BF764.5080305@ntc.zcu.cz>

I am pleased to announce release 2010.1 of SfePy.

Description
-----------

SfePy (simple finite elements in Python) is a software for solving systems of 
coupled partial differential equations by the finite element method. The code 
is based on NumPy and SciPy packages. It is distributed under the new BSD license.

Mailing lists, issue tracking, git repository: http://sfepy.org
Home page: http://sfepy.kme.zcu.cz

Documentation: http://docs.sfepy.org/doc

Contributors to this release: Vladim?r Luke?, Logan Sorenson.

Highlights of this release
--------------------------
- new sphinx-based documentation
- refactoring of base functions (polynomial spaces) and element geometry 
description
- interpolation between different meshes
- terms for describing perfusion and active fibres in the total Lagrangian 
formulation (applicable, for example, to active muscle tissue models)

Major improvements
------------------
Apart from many bug-fixes, let us mention:
- data probing:
   - automatic refinement of probe points, speed-up
- postprocessing and visualization:
   - VTK source construction for any format supported by MeshIO classes:
     - this means displaying meshes in formats Mayavi knows nothing about
- graphical logging:
   - support logging to a text file, vertical line plot, allow several Log 
instances
- new examples:
   - application of the theory of homogenization to elasticity
   - perfusion, active fibres
- new tests and many new terms

For more information on this release, see
http://sfepy.googlecode.com/svn/web/releases/2010.1_RELEASE_NOTES.txt
(full release notes, rather long).

Best regards,
Robert Cimrman


From nwagner at iam.uni-stuttgart.de  Mon Mar  1 15:52:43 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Mon, 01 Mar 2010 21:52:43 +0100
Subject: [SciPy-User] scipy.io.wavfile versus audiolab
Message-ID: <web-130881092@uni-stuttgart.de>

Hi all,

what are the differences between audiolab and 
scipy.io.wavfile ?

Nils


From Chris.Barker at noaa.gov  Mon Mar  1 17:15:29 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 01 Mar 2010 14:15:29 -0800
Subject: [SciPy-User] Google Summer of Code...
Message-ID: <4B8C3C81.7000509@noaa.gov>

Hi folks,

Are we going to try to get some sciPy project in GSoC under the Python 
umbrella this year? Or might it be worth applying to be a mentoring 
organization?

Has anyone started thinking about it?

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From josef.pktd at gmail.com  Mon Mar  1 17:35:32 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 1 Mar 2010 17:35:32 -0500
Subject: [SciPy-User] Google Summer of Code...
In-Reply-To: <4B8C3C81.7000509@noaa.gov>
References: <4B8C3C81.7000509@noaa.gov>
Message-ID: <1cd32cbb1003011435u725d37e5ne3c98475f3cfa8a@mail.gmail.com>

On Mon, Mar 1, 2010 at 5:15 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> Hi folks,
>
> Are we going to try to get some sciPy project in GSoC under the Python
> umbrella this year? Or might it be worth applying to be a mentoring
> organization?
>
> Has anyone started thinking about it?

I sure hope we do, and one project is stats.

Who is organizing gsoc for scipy/numpy?

Josef

>
> -Chris
>
>
>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice
> 7600 Sand Point Way NE ? (206) 526-6329 ? fax
> Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From robert.kern at gmail.com  Mon Mar  1 17:43:34 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 1 Mar 2010 16:43:34 -0600
Subject: [SciPy-User] Google Summer of Code...
In-Reply-To: <1cd32cbb1003011435u725d37e5ne3c98475f3cfa8a@mail.gmail.com>
References: <4B8C3C81.7000509@noaa.gov>
	<1cd32cbb1003011435u725d37e5ne3c98475f3cfa8a@mail.gmail.com>
Message-ID: <3d375d731003011443h1a5c0b02h837f5af3f5348037@mail.gmail.com>

On Mon, Mar 1, 2010 at 16:35,  <josef.pktd at gmail.com> wrote:
> On Mon, Mar 1, 2010 at 5:15 PM, Christopher Barker
> <Chris.Barker at noaa.gov> wrote:
>> Hi folks,
>>
>> Are we going to try to get some sciPy project in GSoC under the Python
>> umbrella this year? Or might it be worth applying to be a mentoring
>> organization?
>>
>> Has anyone started thinking about it?
>
> I sure hope we do, and one project is stats.
>
> Who is organizing gsoc for scipy/numpy?

You, apparently. Congratulations!

More seriously, I don't think there is going to be any organization on
our side, per se. Like every other year, scipy-relevant projects will
most likely be taken care of by the PSF acting as an umbrella
organization. People willing to mentor should contact the PSF
coordinator. By all means, use our mailing lists and Wiki to talk
about possible projects and connect students to mentors, but it's
mostly going to be up to the volunteer mentors to work with the PSF
and your students to get the project proposals submitted and selected.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From jsseabold at gmail.com  Mon Mar  1 17:59:46 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Mon, 1 Mar 2010 17:59:46 -0500
Subject: [SciPy-User] Google Summer of Code...
In-Reply-To: <3d375d731003011443h1a5c0b02h837f5af3f5348037@mail.gmail.com>
References: <4B8C3C81.7000509@noaa.gov>
	<1cd32cbb1003011435u725d37e5ne3c98475f3cfa8a@mail.gmail.com> 
	<3d375d731003011443h1a5c0b02h837f5af3f5348037@mail.gmail.com>
Message-ID: <c048da1c1003011459g6d8faa6eg38751db3f88b6417@mail.gmail.com>

On Mon, Mar 1, 2010 at 5:43 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Mon, Mar 1, 2010 at 16:35, ?<josef.pktd at gmail.com> wrote:
>> On Mon, Mar 1, 2010 at 5:15 PM, Christopher Barker
>> <Chris.Barker at noaa.gov> wrote:
>>> Hi folks,
>>>
>>> Are we going to try to get some sciPy project in GSoC under the Python
>>> umbrella this year? Or might it be worth applying to be a mentoring
>>> organization?
>>>
>>> Has anyone started thinking about it?
>>
>> I sure hope we do, and one project is stats.
>>
>> Who is organizing gsoc for scipy/numpy?
>
> You, apparently. Congratulations!
>
> More seriously, I don't think there is going to be any organization on
> our side, per se. Like every other year, scipy-relevant projects will
> most likely be taken care of by the PSF acting as an umbrella
> organization. People willing to mentor should contact the PSF
> coordinator. By all means, use our mailing lists and Wiki to talk
> about possible projects and connect students to mentors, but it's
> mostly going to be up to the volunteer mentors to work with the PSF
> and your students to get the project proposals submitted and selected.
>

One thing I would suggest (I'll be able to pitch in in the next few
days) is cleaning up this with the progress made last summer (anything
from 2008?): http://projects.scipy.org/scipy/wiki/SummerOfCode

I would also encourage anyone who has ideas to edit the wiki here to
reflect the current project wishlist (especially if you're willing to
mentor): http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas  IMHO,
the more ideas there are, the more chance of grabbing interested and
diverse sets of students provided people are willing to mentor.

Also, should we put a more prominent link to the first page somewhere?
 Maybe on the developer page?  I'm not even sure how to get there
except through googling.  Once this is cleaned up a bit perhaps we
should advertise a little?  Just a thought.

Skipper


From millman at berkeley.edu  Mon Mar  1 18:01:18 2010
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 1 Mar 2010 15:01:18 -0800
Subject: [SciPy-User] Google Summer of Code...
In-Reply-To: <4B8C3C81.7000509@noaa.gov>
References: <4B8C3C81.7000509@noaa.gov>
Message-ID: <c7009a551003011501l15057321scfd847e5a13dcf61@mail.gmail.com>

On Mon, Mar 1, 2010 at 2:15 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> Are we going to try to get some sciPy project in GSoC under the Python
> umbrella this year? Or might it be worth applying to be a mentoring
> organization?

We will definitely be going under the PSF umbrella.  In the past, I've
discussed whether we should apply as a mentoring organization with
Chris DiBona and he has repeatedly strongly encouraged me not to do
that.  He much prefers us remaining under the PSF umbrella.

Moreover, even if Google was interested in us being our own mentoring
organization, I don't see any reason we should pursue it.  It is a
fair bit of additional administrative work for potentially very little
gain.  If we were set up as our own mentoring organization, I am
fairly certain there is no chance we would get any additional reserved
slots.  As far as I remember, the PSF has always given us as many (or
more) scipy-related slots as we request based on the applications that
come in and our pool of potential mentors.

If you are interested in mentoring this year, please register for these lists:
  http://mail.python.org/mailman/listinfo/soc2010-general
  http://mail.python.org/mailman/listinfo/soc2010-mentors

Feel free to add ideas or edit the existing ones here:
http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas

Thanks,

-- 
Jarrod Millman
Helen Wills Neuroscience Institute
10 Giannini Hall, UC Berkeley
http://cirl.berkeley.edu/


From dwf at cs.toronto.edu  Mon Mar  1 18:32:06 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Mon, 1 Mar 2010 18:32:06 -0500
Subject: [SciPy-User] Google Summer of Code...
In-Reply-To: <c048da1c1003011459g6d8faa6eg38751db3f88b6417@mail.gmail.com>
References: <4B8C3C81.7000509@noaa.gov>
	<1cd32cbb1003011435u725d37e5ne3c98475f3cfa8a@mail.gmail.com>
	<3d375d731003011443h1a5c0b02h837f5af3f5348037@mail.gmail.com>
	<c048da1c1003011459g6d8faa6eg38751db3f88b6417@mail.gmail.com>
Message-ID: <EEDF0DA5-C8FD-4AA1-9CF5-B0708B079E6B@cs.toronto.edu>

On 1-Mar-10, at 5:59 PM, Skipper Seabold wrote:

> I would also encourage anyone who has ideas to edit the wiki here to
> reflect the current project wishlist (especially if you're willing to
> mentor): http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas  IMHO,
> the more ideas there are, the more chance of grabbing interested and
> diverse sets of students provided people are willing to mentor.

I just reorganized the page a bit, to separate out the stuff that is  
already going on, like statsmodels, fwrap and the Py3k/datetime stuff,  
from things that have been listed for a while but never really picked  
up on.

David


From david at silveregg.co.jp  Mon Mar  1 19:52:08 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Tue, 02 Mar 2010 09:52:08 +0900
Subject: [SciPy-User] scipy.io.wavfile versus audiolab
In-Reply-To: <web-130881092@uni-stuttgart.de>
References: <web-130881092@uni-stuttgart.de>
Message-ID: <4B8C6138.8090308@silveregg.co.jp>

Nils Wagner wrote:
> Hi all,
> 
> what are the differences between audiolab and 
> scipy.io.wavfile ?

For audiolab:
  - more features
  - is likely to be faster, since it uses libsndfile which is coded in C
  - handles more file formats (libsndfile is by far the best library for 
audio file IO and handles a lot of software-specific details).

For scipy.io.wavfile:
  - pure python and no need for 3rd party library

License-wise, audiolab is LGPL (because libsndfile is) whereas 
scipy.io.wavfile is BSD as the rest of scipy.

cheers,

David



From massimodisasha at yahoo.it  Mon Mar  1 20:17:16 2010
From: massimodisasha at yahoo.it (Massimo Di Stefano)
Date: Tue, 2 Mar 2010 02:17:16 +0100
Subject: [SciPy-User] helmert implementation (7 parameters - geometric
	transformation)
Message-ID: <B8F98F3B-3A4E-47B6-935D-302A85A7B395@yahoo.it>

Hi All,

i'm tring to implement a function to perform a geometric trasformation 
between a set of points in 2 different reference systems

it is a 7 parameters trasformation (helmert) , based on last square method
it use as input 2 set of x,y,z coordinates in 2 different reference systems
and give as output 7 parameters needed to performs transformation 
from one reference system to the other one.

http://en.wikipedia.org/wiki/Helmert_transformation

this function is used in geodesy to "reproject" coordinates between 2 different Datum

i wrote it as :

import numpy as np
from scipy import linalg

def helmert(p1, p2):
    L = np.loadtxt(p1)
    A = np.zeros((3*L.shape[0],7),float)
    A[ ::3, 0] = 1.0
    A[1::3, 1] = 1.0
    A[2::3, 2] = 1.0
    A[ ::3, 3] = L[:,0]
    A[1::3, 3] = L[:,1]
    A[2::3, 3] = L[:,2]
    A[1::3, 4] = L[:,2]
    A[2::3, 4] = -L[:,1]
    A[ ::3, 5] = -L[:,2]
    A[2::3, 5] = L[:,0]
    A[ ::3, 6] = L[:,1]
    A[1::3, 6] = -L[:,0]
    G = np.loadtxt(p2)
    Y = np.zeros((3*G.shape[0],1),float)
    Y[ ::3, 0] = G[:,0] - L[:,0]
    Y[1::3, 0] = G[:,1] - L[:,1]
    Y[2::3, 0] = G[:,2] - L[:,2]
    N = np.dot(A.T.conj(), A)
    T = np.dot(A.T.conj(), Y)
    C = np.dot(linalg.inv(N), T)
    print C


from a pdf i find online
it has numerical examples
and i get different results :'(


p1 :

4402553.334 727053.937 4542823.474 
4399375.347 703845.876 4549215.105 
4412911.150 701094.435 4536518.139


p2 :

4402553.569 727053.737 4542823.332 
4399375.518 703845.639 4549214.880 
4412911.336 701094.214 4536517.955


pdf results :

x0    -9.256 m 
y0    -23.701 m 
z0    16.792 m
Rx    -0.0001990982 rad 
Ry    0.0001778762 rad
Rz    0.00015 rad 
?      0.00000046


my results :

[[ -9.91564629e+00]
 [ -2.36172028e+01]
 [  1.57283853e+01]
 [ -2.68063331e-07]
 [  3.04330048e-06]
 [ -2.76148185e-06]
 [ -2.31598150e-06]]



anyone here can give it a try ?
the results seems wrong but
i haveb't yet find a solution.

thanks to ALL!

regards,

Massimo.







From charlesr.harris at gmail.com  Mon Mar  1 23:08:20 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 1 Mar 2010 21:08:20 -0700
Subject: [SciPy-User] helmert implementation (7 parameters - geometric
	transformation)
In-Reply-To: <B8F98F3B-3A4E-47B6-935D-302A85A7B395@yahoo.it>
References: <B8F98F3B-3A4E-47B6-935D-302A85A7B395@yahoo.it>
Message-ID: <e06186141003012008k677ecce9l4317d07d1a4044bd@mail.gmail.com>

On Mon, Mar 1, 2010 at 6:17 PM, Massimo Di Stefano
<massimodisasha at yahoo.it>wrote:

> Hi All,
>
> i'm tring to implement a function to perform a geometric trasformation
> between a set of points in 2 different reference systems
>
> it is a 7 parameters trasformation (helmert) , based on last square method
> it use as input 2 set of x,y,z coordinates in 2 different reference systems
> and give as output 7 parameters needed to performs transformation
> from one reference system to the other one.
>
> http://en.wikipedia.org/wiki/Helmert_transformation
>
> this function is used in geodesy to "reproject" coordinates between 2
> different Datum
>
> i wrote it as :
>
> import numpy as np
> from scipy import linalg
>
> def helmert(p1, p2):
>    L = np.loadtxt(p1)
>    A = np.zeros((3*L.shape[0],7),float)
>    A[ ::3, 0] = 1.0
>    A[1::3, 1] = 1.0
>    A[2::3, 2] = 1.0
>    A[ ::3, 3] = L[:,0]
>    A[1::3, 3] = L[:,1]
>    A[2::3, 3] = L[:,2]
>    A[1::3, 4] = L[:,2]
>    A[2::3, 4] = -L[:,1]
>    A[ ::3, 5] = -L[:,2]
>    A[2::3, 5] = L[:,0]
>    A[ ::3, 6] = L[:,1]
>    A[1::3, 6] = -L[:,0]
>    G = np.loadtxt(p2)
>    Y = np.zeros((3*G.shape[0],1),float)
>    Y[ ::3, 0] = G[:,0] - L[:,0]
>    Y[1::3, 0] = G[:,1] - L[:,1]
>    Y[2::3, 0] = G[:,2] - L[:,2]
>    N = np.dot(A.T.conj(), A)
>    T = np.dot(A.T.conj(), Y)
>    C = np.dot(linalg.inv(N), T)
>    print C
>
>
> from a pdf i find online
> it has numerical examples
> and i get different results :'(
>
>
> p1 :
>
> 4402553.334 727053.937 4542823.474
> 4399375.347 703845.876 4549215.105
> 4412911.150 701094.435 4536518.139
>
>
> p2 :
>
> 4402553.569 727053.737 4542823.332
> 4399375.518 703845.639 4549214.880
> 4412911.336 701094.214 4536517.955
>
>
> pdf results :
>
> x0    -9.256 m
> y0    -23.701 m
> z0    16.792 m
> Rx    -0.0001990982 rad
> Ry    0.0001778762 rad
> Rz    0.00015 rad
> ?      0.00000046
>
>
> my results :
>
> [[ -9.91564629e+00]
>  [ -2.36172028e+01]
>  [  1.57283853e+01]
>  [ -2.68063331e-07]
>  [  3.04330048e-06]
>  [ -2.76148185e-06]
>  [ -2.31598150e-06]]
>
>
>
> anyone here can give it a try ?
> the results seems wrong but
> i haveb't yet find a solution.
>
> thanks to ALL!
>
> regards,
>
> Massimo.
>
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100301/4ef0d873/attachment.html>

From charlesr.harris at gmail.com  Tue Mar  2 01:05:05 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 1 Mar 2010 23:05:05 -0700
Subject: [SciPy-User] helmert implementation (7 parameters - geometric
	transformation)
In-Reply-To: <B8F98F3B-3A4E-47B6-935D-302A85A7B395@yahoo.it>
References: <B8F98F3B-3A4E-47B6-935D-302A85A7B395@yahoo.it>
Message-ID: <e06186141003012205r3ebf7529uba99beb9b5237dbe@mail.gmail.com>

On Mon, Mar 1, 2010 at 6:17 PM, Massimo Di Stefano
<massimodisasha at yahoo.it>wrote:

> Hi All,
>
> i'm tring to implement a function to perform a geometric trasformation
> between a set of points in 2 different reference systems
>
> it is a 7 parameters trasformation (helmert) , based on last square method
> it use as input 2 set of x,y,z coordinates in 2 different reference systems
> and give as output 7 parameters needed to performs transformation
> from one reference system to the other one.
>
> http://en.wikipedia.org/wiki/Helmert_transformation
>
> this function is used in geodesy to "reproject" coordinates between 2
> different Datum
>
> i wrote it as :
>
> import numpy as np
> from scipy import linalg
>
> def helmert(p1, p2):
>    L = np.loadtxt(p1)
>    A = np.zeros((3*L.shape[0],7),float)
>    A[ ::3, 0] = 1.0
>    A[1::3, 1] = 1.0
>    A[2::3, 2] = 1.0
>    A[ ::3, 3] = L[:,0]
>    A[1::3, 3] = L[:,1]
>    A[2::3, 3] = L[:,2]
>    A[1::3, 4] = L[:,2]
>    A[2::3, 4] = -L[:,1]
>    A[ ::3, 5] = -L[:,2]
>    A[2::3, 5] = L[:,0]
>    A[ ::3, 6] = L[:,1]
>    A[1::3, 6] = -L[:,0]
>    G = np.loadtxt(p2)
>    Y = np.zeros((3*G.shape[0],1),float)
>    Y[ ::3, 0] = G[:,0] - L[:,0]
>    Y[1::3, 0] = G[:,1] - L[:,1]
>    Y[2::3, 0] = G[:,2] - L[:,2]
>    N = np.dot(A.T.conj(), A)
>    T = np.dot(A.T.conj(), Y)
>    C = np.dot(linalg.inv(N), T)
>    print C
>
>
> from a pdf i find online
> it has numerical examples
> and i get different results :'(
>
>
> p1 :
>
> 4402553.334 727053.937 4542823.474
> 4399375.347 703845.876 4549215.105
> 4412911.150 701094.435 4536518.139
>
>
> p2 :
>
> 4402553.569 727053.737 4542823.332
> 4399375.518 703845.639 4549214.880
> 4412911.336 701094.214 4536517.955
>
>
> pdf results :
>
> x0    -9.256 m
> y0    -23.701 m
> z0    16.792 m
> Rx    -0.0001990982 rad
> Ry    0.0001778762 rad
> Rz    0.00015 rad
> ?      0.00000046
>
>
The angles here look like degrees, not radians. Also, the last factor is
pretty definitely wrong. It is given directly as the square root of the
ratio of the variance of the two sets of vectors around their means minus 1.
For instance

sqrt(pt2.var(0).sum()/pt1.var(0).sum()) - 1


>
> my results :
>
> [[ -9.91564629e+00]
>  [ -2.36172028e+01]
>  [  1.57283853e+01]
>  [ -2.68063331e-07]
>  [  3.04330048e-06]
>  [ -2.76148185e-06]
>  [ -2.31598150e-06]]
>
>
>
And my result agrees with neither ;) BTW, are you going from pt2 -> pt1 or
vice versa?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100301/5ca0d197/attachment.html>

From charlesr.harris at gmail.com  Tue Mar  2 01:18:34 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 1 Mar 2010 23:18:34 -0700
Subject: [SciPy-User] helmert implementation (7 parameters - geometric
	transformation)
In-Reply-To: <e06186141003012205r3ebf7529uba99beb9b5237dbe@mail.gmail.com>
References: <B8F98F3B-3A4E-47B6-935D-302A85A7B395@yahoo.it>
	<e06186141003012205r3ebf7529uba99beb9b5237dbe@mail.gmail.com>
Message-ID: <e06186141003012218i3b1439bu40f38ae8f5a05f69@mail.gmail.com>

2010/3/1 Charles R Harris <charlesr.harris at gmail.com>

>
>
> On Mon, Mar 1, 2010 at 6:17 PM, Massimo Di Stefano <
> massimodisasha at yahoo.it> wrote:
>
>> Hi All,
>>
>> i'm tring to implement a function to perform a geometric trasformation
>> between a set of points in 2 different reference systems
>>
>> it is a 7 parameters trasformation (helmert) , based on last square method
>> it use as input 2 set of x,y,z coordinates in 2 different reference
>> systems
>> and give as output 7 parameters needed to performs transformation
>> from one reference system to the other one.
>>
>> http://en.wikipedia.org/wiki/Helmert_transformation
>>
>> this function is used in geodesy to "reproject" coordinates between 2
>> different Datum
>>
>> i wrote it as :
>>
>> import numpy as np
>> from scipy import linalg
>>
>> def helmert(p1, p2):
>>    L = np.loadtxt(p1)
>>    A = np.zeros((3*L.shape[0],7),float)
>>    A[ ::3, 0] = 1.0
>>    A[1::3, 1] = 1.0
>>    A[2::3, 2] = 1.0
>>    A[ ::3, 3] = L[:,0]
>>    A[1::3, 3] = L[:,1]
>>    A[2::3, 3] = L[:,2]
>>    A[1::3, 4] = L[:,2]
>>    A[2::3, 4] = -L[:,1]
>>    A[ ::3, 5] = -L[:,2]
>>    A[2::3, 5] = L[:,0]
>>    A[ ::3, 6] = L[:,1]
>>    A[1::3, 6] = -L[:,0]
>>    G = np.loadtxt(p2)
>>    Y = np.zeros((3*G.shape[0],1),float)
>>    Y[ ::3, 0] = G[:,0] - L[:,0]
>>    Y[1::3, 0] = G[:,1] - L[:,1]
>>    Y[2::3, 0] = G[:,2] - L[:,2]
>>    N = np.dot(A.T.conj(), A)
>>    T = np.dot(A.T.conj(), Y)
>>    C = np.dot(linalg.inv(N), T)
>>    print C
>>
>>
>> from a pdf i find online
>> it has numerical examples
>> and i get different results :'(
>>
>>
>> p1 :
>>
>> 4402553.334 727053.937 4542823.474
>> 4399375.347 703845.876 4549215.105
>> 4412911.150 701094.435 4536518.139
>>
>>
>> p2 :
>>
>> 4402553.569 727053.737 4542823.332
>> 4399375.518 703845.639 4549214.880
>> 4412911.336 701094.214 4536517.955
>>
>>
>> pdf results :
>>
>> x0    -9.256 m
>> y0    -23.701 m
>> z0    16.792 m
>> Rx    -0.0001990982 rad
>> Ry    0.0001778762 rad
>> Rz    0.00015 rad
>> ?      0.00000046
>>
>>
> The angles here look like degrees, not radians. Also, the last factor is
> pretty definitely wrong. It is given directly as the square root of the
> ratio of the variance of the two sets of vectors around their means minus 1.
> For instance
>
> sqrt(pt2.var(0).sum()/pt1.var(0).sum()) - 1
>
>
>>
>> my results :
>>
>> [[ -9.91564629e+00]
>>  [ -2.36172028e+01]
>>  [  1.57283853e+01]
>>  [ -2.68063331e-07]
>>  [  3.04330048e-06]
>>  [ -2.76148185e-06]
>>  [ -2.31598150e-06]]
>>
>>
>>
> And my result agrees with neither ;) BTW, are you going from pt2 -> pt1 or
> vice versa?
>
>
Are you sure the data you gave is correct? Lengths don't seem to be
preserved under rotation.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100301/b09e2283/attachment.html>

From dlrt2 at ast.cam.ac.uk  Tue Mar  2 06:02:09 2010
From: dlrt2 at ast.cam.ac.uk (David Trethewey)
Date: Tue, 02 Mar 2010 11:02:09 +0000
Subject: [SciPy-User] scikits.learn installation
In-Reply-To: <mailman.13.1267466403.14696.scipy-user@scipy.org>
References: <mailman.13.1267466403.14696.scipy-user@scipy.org>
Message-ID: <4B8CF031.6030207@ast.cam.ac.uk>

I installed the python headers and libboost. It now compiles without 
error but still when I actually try to import the em module I get the 
same error message;
>>>> import scipy
>>>> import numpy
>>>> from scikits.learn.machine import em
>>>>         
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>    from online_em import OnGMM as _OnGMM
>  File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>    import _rawden
> ImportError: No module named _rawden
>
>
> David Trethewey
> Institute of Astronomy, Cambridge, UK
>
>
>   



scipy-user-request at scipy.org wrote:
> Send SciPy-User mailing list submissions to
> 	scipy-user at scipy.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://mail.scipy.org/mailman/listinfo/scipy-user
> or, via email, send a message with subject or body 'help' to
> 	scipy-user-request at scipy.org
>
> You can reach the person managing the list at
> 	scipy-user-owner at scipy.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of SciPy-User digest..."
>
>
> Today's Topics:
>
>    1. Re: Error on importing scikits.learn (David Trethewey)
>    2. Re: Error on importing scikits.learn (David Trethewey)
>    3. Re: Error on importing scikits.learn (Helge Reikeras)
>    4. ANN: SfePy 2010.1 (Robert Cimrman)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 01 Mar 2010 11:46:42 +0000
> From: David Trethewey <dlrt2 at ast.cam.ac.uk>
> Subject: Re: [SciPy-User] Error on importing scikits.learn
> To: scipy-user at scipy.org
> Message-ID: <4B8BA922.6010900 at ast.cam.ac.uk>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> I get the following error message when attempting to import the 
> expectation maximisation (em) module from scikits.learn :
>   
>>>>> import scipy
>>>>> import numpy
>>>>> from scikits.learn.machine import em
>>>>>           
>> Traceback (most recent call last):
>>  File "<stdin>", line 1, in <module>
>>  File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>>    from online_em import OnGMM as _OnGMM
>>  File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>>    import _rawden
>> ImportError: No module named _rawden
>>
>> Does anyone know what this _rawden module is and why might it not be 
>> found?
>>
>> David Trethewey
>> Institute of Astronomy, Cambridge, UK
>>
>>
>>     
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 01 Mar 2010 11:51:14 +0000
> From: David Trethewey <dlrt2 at ast.cam.ac.uk>
> Subject: Re: [SciPy-User] Error on importing scikits.learn
> To: scipy-user at scipy.org
> Message-ID: <4B8BAA32.3000004 at ast.cam.ac.uk>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> I have traced it back to an error in compiling scikits.learn itself:
>
> building 'scikits.learn.machine.em._rawden' extension
> compiling C sources
> C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 
> -Wall -Wstrict-prototypes -fPIC
>
> compile options: '-I/usr/lib/python2.6/dist-packages/numpy/core/include 
> -I/usr/include/python2.6 -c'
> gcc: scikits/learn/machine/em/src/pure_den.c
> scikits/learn/machine/em/src/pure_den.c:7:20: error: Python.h: No such 
> file or directory
>
> David
>
> David Trethewey wrote:
>   
>> I get the following error message when attempting to import the 
>> expectation maximisation (em) module from scikits.learn :
>>     
>>>>>> import scipy
>>>>>> import numpy
>>>>>> from scikits.learn.machine import em
>>>>>>             
>>> Traceback (most recent call last):
>>>  File "<stdin>", line 1, in <module>
>>>  File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>>>    from online_em import OnGMM as _OnGMM
>>>  File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>>>    import _rawden
>>> ImportError: No module named _rawden
>>>
>>> Does anyone know what this _rawden module is and why might it not be 
>>> found?
>>>
>>> David Trethewey
>>> Institute of Astronomy, Cambridge, UK
>>>
>>>
>>>       
>>     
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 01 Mar 2010 13:55:01 +0200
> From: "Helge Reikeras" <helge.reikeras at gmail.com>
> Subject: Re: [SciPy-User] Error on importing scikits.learn
> To: "SciPy Users List" <scipy-user at scipy.org>
> Message-ID: <op.u8v0pqws9y0l3i at helge-desktop>
> Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
>
> On Mon, 01 Mar 2010 13:51:14 +0200, David Trethewey <dlrt2 at ast.cam.ac.uk>  
> wrote:
>
>   
>> compile options: '-I/usr/lib/python2.6/dist-packages/numpy/core/include
>> -I/usr/include/python2.6 -c'
>> gcc: scikits/learn/machine/em/src/pure_den.c
>> scikits/learn/machine/em/src/pure_den.c:7:20: error: Python.h: No such
>> file or directory
>>     
>
> This is because you don't have the Python header files installed. You need  
> to install these somehow. E.g. if you are running Ubuntu you do "sudo  
> apt-get install python-dev".
>
>   



From pitam at deepspace.ucsb.edu  Tue Mar  2 06:21:25 2010
From: pitam at deepspace.ucsb.edu (Pitam Mitra)
Date: Tue, 02 Mar 2010 03:21:25 -0800
Subject: [SciPy-User] segfaults with weave.inline
Message-ID: <web-4550506@deepspace.ucsb.edu>

Hi,

I am creating a new median filter for an astronomy
application. Due to speed issues, I decided to code it with
weave.inline. However, I have run into segfaults and I have
no clue what to do next.

I am sorry if this is the wrong section to post this - I
searched the site, I could not find any forum. :(


Any help is appreciated!!

Here is the output:

*** glibc detected *** python: free(): invalid next size
(fast): 0x09746358 ***
======= Backtrace: =========
/lib/libc.so.6[0x428751]
/usr/lib/python2.6/site-packages/numpy/core/multiarray.so(+0x3643c)[0x8db43c]
/usr/lib/libpython2.6.so.1.0[0x3669924]
/usr/lib/libpython2.6.so.1.0[0x3656559]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5359)[0x36c8fc9]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x7aa)[0x36c9e4a]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalCode+0x64)[0x36c9fb4]
/usr/lib/libpython2.6.so.1.0[0x36e525c]
/usr/lib/libpython2.6.so.1.0(PyRun_FileExFlags+0x93)[0x36e5323]
/usr/lib/libpython2.6.so.1.0(PyRun_SimpleFileExFlags+0xe1)[0x36e68c1]
/usr/lib/libpython2.6.so.1.0(PyRun_AnyFileExFlags+0x82)[0x36e7162]
/usr/lib/libpython2.6.so.1.0(Py_Main+0xb95)[0x36f3bd5]
python(main+0x28)[0x80485c8]
/lib/libc.so.6(__libc_start_main+0xe6)[0x3d0bb6]
python[0x8048501]



Then there is a huge memory map. The full output can be
seen here:http://www.pastebay.com/88049

and the code here: http://www.pastebay.com/88050

Regards

Pitam



From josef.pktd at gmail.com  Tue Mar  2 06:45:08 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 2 Mar 2010 06:45:08 -0500
Subject: [SciPy-User] scikits.learn installation
In-Reply-To: <4B8CF031.6030207@ast.cam.ac.uk>
References: <mailman.13.1267466403.14696.scipy-user@scipy.org>
	<4B8CF031.6030207@ast.cam.ac.uk>
Message-ID: <1cd32cbb1003020345k7d51d85ah5fd703139a20afbc@mail.gmail.com>

On Tue, Mar 2, 2010 at 6:02 AM, David Trethewey <dlrt2 at ast.cam.ac.uk> wrote:
> I installed the python headers and libboost. It now compiles without
> error but still when I actually try to import the em module I get the
> same error message;
>>>>> import scipy
>>>>> import numpy
>>>>> from scikits.learn.machine import em
>>>>>
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> ?File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>> ? ?from online_em import OnGMM as _OnGMM
>> ?File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>> ? ?import _rawden
>> ImportError: No module named _rawden

You could search the source and build directories for *.pyd .
It happened to me in the pre-release version that the pyd files didn't
get copied correctly to the install directories.

Josef

>>
>>
>> David Trethewey
>> Institute of Astronomy, Cambridge, UK
>>
>>
>>
>
>
>
> scipy-user-request at scipy.org wrote:
>> Send SciPy-User mailing list submissions to
>> ? ? ? scipy-user at scipy.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> ? ? ? http://mail.scipy.org/mailman/listinfo/scipy-user
>> or, via email, send a message with subject or body 'help' to
>> ? ? ? scipy-user-request at scipy.org
>>
>> You can reach the person managing the list at
>> ? ? ? scipy-user-owner at scipy.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of SciPy-User digest..."
>>
>>
>> Today's Topics:
>>
>> ? ?1. Re: Error on importing scikits.learn (David Trethewey)
>> ? ?2. Re: Error on importing scikits.learn (David Trethewey)
>> ? ?3. Re: Error on importing scikits.learn (Helge Reikeras)
>> ? ?4. ANN: SfePy 2010.1 (Robert Cimrman)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 01 Mar 2010 11:46:42 +0000
>> From: David Trethewey <dlrt2 at ast.cam.ac.uk>
>> Subject: Re: [SciPy-User] Error on importing scikits.learn
>> To: scipy-user at scipy.org
>> Message-ID: <4B8BA922.6010900 at ast.cam.ac.uk>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> I get the following error message when attempting to import the
>> expectation maximisation (em) module from scikits.learn :
>>
>>>>>> import scipy
>>>>>> import numpy
>>>>>> from scikits.learn.machine import em
>>>>>>
>>> Traceback (most recent call last):
>>> ?File "<stdin>", line 1, in <module>
>>> ?File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>>> ? ?from online_em import OnGMM as _OnGMM
>>> ?File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>>> ? ?import _rawden
>>> ImportError: No module named _rawden
>>>
>>> Does anyone know what this _rawden module is and why might it not be
>>> found?
>>>
>>> David Trethewey
>>> Institute of Astronomy, Cambridge, UK
>>>
>>>
>>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Mon, 01 Mar 2010 11:51:14 +0000
>> From: David Trethewey <dlrt2 at ast.cam.ac.uk>
>> Subject: Re: [SciPy-User] Error on importing scikits.learn
>> To: scipy-user at scipy.org
>> Message-ID: <4B8BAA32.3000004 at ast.cam.ac.uk>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> I have traced it back to an error in compiling scikits.learn itself:
>>
>> building 'scikits.learn.machine.em._rawden' extension
>> compiling C sources
>> C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
>> -Wall -Wstrict-prototypes -fPIC
>>
>> compile options: '-I/usr/lib/python2.6/dist-packages/numpy/core/include
>> -I/usr/include/python2.6 -c'
>> gcc: scikits/learn/machine/em/src/pure_den.c
>> scikits/learn/machine/em/src/pure_den.c:7:20: error: Python.h: No such
>> file or directory
>>
>> David
>>
>> David Trethewey wrote:
>>
>>> I get the following error message when attempting to import the
>>> expectation maximisation (em) module from scikits.learn :
>>>
>>>>>>> import scipy
>>>>>>> import numpy
>>>>>>> from scikits.learn.machine import em
>>>>>>>
>>>> Traceback (most recent call last):
>>>> ?File "<stdin>", line 1, in <module>
>>>> ?File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>>>> ? ?from online_em import OnGMM as _OnGMM
>>>> ?File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>>>> ? ?import _rawden
>>>> ImportError: No module named _rawden
>>>>
>>>> Does anyone know what this _rawden module is and why might it not be
>>>> found?
>>>>
>>>> David Trethewey
>>>> Institute of Astronomy, Cambridge, UK
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Mon, 01 Mar 2010 13:55:01 +0200
>> From: "Helge Reikeras" <helge.reikeras at gmail.com>
>> Subject: Re: [SciPy-User] Error on importing scikits.learn
>> To: "SciPy Users List" <scipy-user at scipy.org>
>> Message-ID: <op.u8v0pqws9y0l3i at helge-desktop>
>> Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
>>
>> On Mon, 01 Mar 2010 13:51:14 +0200, David Trethewey <dlrt2 at ast.cam.ac.uk>
>> wrote:
>>
>>
>>> compile options: '-I/usr/lib/python2.6/dist-packages/numpy/core/include
>>> -I/usr/include/python2.6 -c'
>>> gcc: scikits/learn/machine/em/src/pure_den.c
>>> scikits/learn/machine/em/src/pure_den.c:7:20: error: Python.h: No such
>>> file or directory
>>>
>>
>> This is because you don't have the Python header files installed. You need
>> to install these somehow. E.g. if you are running Ubuntu you do "sudo
>> apt-get install python-dev".
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From pav+sp at iki.fi  Tue Mar  2 06:47:31 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Tue, 2 Mar 2010 11:47:31 +0000 (UTC)
Subject: [SciPy-User] segfaults with weave.inline
References: <web-4550506@deepspace.ucsb.edu>
Message-ID: <hmitsj$b36$1@dough.gmane.org>

Tue, 02 Mar 2010 03:21:25 -0800, Pitam Mitra wrote:
> I am creating a new median filter for an astronomy application. Due to
> speed issues, I decided to code it with weave.inline. However, I have
> run into segfaults and I have no clue what to do next.

You initialize

	zone_vector = array(())

but then the weave.inline code proceeds to write to positions

	zone_vector(0) ... zone_vector(n_pixels+1)

which are out of bounds. weave.inline probably does not do bounds 
checking, so you get a segfault instead of bounds error. The fix is to 
make zone_vector large enough.

-- 
Pauli Virtanen



From dlrt2 at ast.cam.ac.uk  Tue Mar  2 07:02:44 2010
From: dlrt2 at ast.cam.ac.uk (David Trethewey)
Date: Tue, 02 Mar 2010 12:02:44 +0000
Subject: [SciPy-User] scikits.learn installation
In-Reply-To: <mailman.2676.1267530315.2398.scipy-user@scipy.org>
References: <mailman.2676.1267530315.2398.scipy-user@scipy.org>
Message-ID: <4B8CFE64.6070706@ast.cam.ac.uk>

Problem is sorted now, thanks for your help.

David
>
> Message: 5
> Date: Tue, 2 Mar 2010 06:45:08 -0500
> From: josef.pktd at gmail.com
> Subject: Re: [SciPy-User] scikits.learn installation
> To: SciPy Users List <scipy-user at scipy.org>
> Message-ID:
> 	<1cd32cbb1003020345k7d51d85ah5fd703139a20afbc at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Tue, Mar 2, 2010 at 6:02 AM, David Trethewey <dlrt2 at ast.cam.ac.uk> wrote:
>   
>> I installed the python headers and libboost. It now compiles without
>> error but still when I actually try to import the em module I get the
>> same error message;
>>     
>>>>>> import scipy
>>>>>> import numpy
>>>>>> from scikits.learn.machine import em
>>>>>>
>>>>>>             
>>> Traceback (most recent call last):
>>> ?File "<stdin>", line 1, in <module>
>>> ?File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>>> ? ?from online_em import OnGMM as _OnGMM
>>> ?File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>>> ? ?import _rawden
>>> ImportError: No module named _rawden
>>>       
>
> You could search the source and build directories for *.pyd .
> It happened to me in the pre-release version that the pyd files didn't
> get copied correctly to the install directories.
>
> Josef
>
>   
>>> David Trethewey
>>> Institute of Astronomy, Cambridge, UK
>>>
>>>
>>>
>>>       
>>
>> scipy-user-request at scipy.org wrote:
>>     
>>> Send SciPy-User mailing list submissions to
>>> ? ? ? scipy-user at scipy.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> ? ? ? http://mail.scipy.org/mailman/listinfo/scipy-user
>>> or, via email, send a message with subject or body 'help' to
>>> ? ? ? scipy-user-request at scipy.org
>>>
>>> You can reach the person managing the list at
>>> ? ? ? scipy-user-owner at scipy.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of SciPy-User digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> ? ?1. Re: Error on importing scikits.learn (David Trethewey)
>>> ? ?2. Re: Error on importing scikits.learn (David Trethewey)
>>> ? ?3. Re: Error on importing scikits.learn (Helge Reikeras)
>>> ? ?4. ANN: SfePy 2010.1 (Robert Cimrman)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Mon, 01 Mar 2010 11:46:42 +0000
>>> From: David Trethewey <dlrt2 at ast.cam.ac.uk>
>>> Subject: Re: [SciPy-User] Error on importing scikits.learn
>>> To: scipy-user at scipy.org
>>> Message-ID: <4B8BA922.6010900 at ast.cam.ac.uk>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> I get the following error message when attempting to import the
>>> expectation maximisation (em) module from scikits.learn :
>>>
>>>       
>>>>>>> import scipy
>>>>>>> import numpy
>>>>>>> from scikits.learn.machine import em
>>>>>>>
>>>>>>>               
>>>> Traceback (most recent call last):
>>>> ?File "<stdin>", line 1, in <module>
>>>> ?File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>>>> ? ?from online_em import OnGMM as _OnGMM
>>>> ?File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>>>> ? ?import _rawden
>>>> ImportError: No module named _rawden
>>>>
>>>> Does anyone know what this _rawden module is and why might it not be
>>>> found?
>>>>
>>>> David Trethewey
>>>> Institute of Astronomy, Cambridge, UK
>>>>
>>>>
>>>>
>>>>         
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Mon, 01 Mar 2010 11:51:14 +0000
>>> From: David Trethewey <dlrt2 at ast.cam.ac.uk>
>>> Subject: Re: [SciPy-User] Error on importing scikits.learn
>>> To: scipy-user at scipy.org
>>> Message-ID: <4B8BAA32.3000004 at ast.cam.ac.uk>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> I have traced it back to an error in compiling scikits.learn itself:
>>>
>>> building 'scikits.learn.machine.em._rawden' extension
>>> compiling C sources
>>> C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
>>> -Wall -Wstrict-prototypes -fPIC
>>>
>>> compile options: '-I/usr/lib/python2.6/dist-packages/numpy/core/include
>>> -I/usr/include/python2.6 -c'
>>> gcc: scikits/learn/machine/em/src/pure_den.c
>>> scikits/learn/machine/em/src/pure_den.c:7:20: error: Python.h: No such
>>> file or directory
>>>
>>> David
>>>
>>> David Trethewey wrote:
>>>
>>>       
>>>> I get the following error message when attempting to import the
>>>> expectation maximisation (em) module from scikits.learn :
>>>>
>>>>         
>>>>>>>> import scipy
>>>>>>>> import numpy
>>>>>>>> from scikits.learn.machine import em
>>>>>>>>
>>>>>>>>                 
>>>>> Traceback (most recent call last):
>>>>> ?File "<stdin>", line 1, in <module>
>>>>> ?File "scikits/learn/machine/em/__init__.py", line 8, in <module>
>>>>> ? ?from online_em import OnGMM as _OnGMM
>>>>> ?File "scikits/learn/machine/em/online_em.py", line 164, in <module>
>>>>> ? ?import _rawden
>>>>> ImportError: No module named _rawden
>>>>>
>>>>> Does anyone know what this _rawden module is and why might it not be
>>>>> found?
>>>>>
>>>>> David Trethewey
>>>>> Institute of Astronomy, Cambridge, UK
>>>>>
>>>>>
>>>>>
>>>>>           
>>>
>>> ------------------------------
>>>
>>> Message: 3
>>> Date: Mon, 01 Mar 2010 13:55:01 +0200
>>> From: "Helge Reikeras" <helge.reikeras at gmail.com>
>>> Subject: Re: [SciPy-User] Error on importing scikits.learn
>>> To: "SciPy Users List" <scipy-user at scipy.org>
>>> Message-ID: <op.u8v0pqws9y0l3i at helge-desktop>
>>> Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
>>>
>>> On Mon, 01 Mar 2010 13:51:14 +0200, David Trethewey <dlrt2 at ast.cam.ac.uk>
>>> wrote:
>>>
>>>
>>>       
>>>> compile options: '-I/usr/lib/python2.6/dist-packages/numpy/core/include
>>>> -I/usr/include/python2.6 -c'
>>>> gcc: scikits/learn/machine/em/src/pure_den.c
>>>> scikits/learn/machine/em/src/pure_den.c:7:20: error: Python.h: No such
>>>> file or directory
>>>>
>>>>         
>>> This is because you don't have the Python header files installed. You need
>>> to install these somehow. E.g. if you are running Ubuntu you do "sudo
>>> apt-get install python-dev".
>>>
>>>
>>>       
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>     
>
>
> ------------------------------
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> End of SciPy-User Digest, Vol 79, Issue 3
> *****************************************
>   



From ewald.zietsman at gmail.com  Tue Mar  2 07:03:12 2010
From: ewald.zietsman at gmail.com (Ewald Zietsman)
Date: Tue, 2 Mar 2010 14:03:12 +0200
Subject: [SciPy-User] helmert implementation (7 parameters - geometric
	transformation)
Message-ID: <be152e7d1003020403p56f14926v10bb5bb6f1953f9f@mail.gmail.com>

Hi,

I see your coordinates are very big numbers. These can lead to an
ill-conditioned (AtA) matrix which can lead to a wrong inverse. Try
using cholesky decomposition instead of using linalg.inv. The method
is outlined here http://en.wikipedia.org/wiki/Cholesky_decomposition
and numpy has routines for doing just this. The other option is to
subtract the centre-of-mass from your coordinates i.e. P1 -> P1x -
P1xave, P1y - P1yave etc. Do the same for P2. Calculate the
transformation again. The shifts should now be zero but your rotations
and scale should be statistically independent and correct. The
difference between the two coordinate systems' centres of mass gives
the shifts. Transformations far away from the origin rarely behaves
well.

Good luck.

>>
>>> Hi All,
>>>
>>> i'm tring to implement a function to perform a geometric trasformation
>>> between a set of points in 2 different reference systems
>>>
>>> it is a 7 parameters trasformation (helmert) , based on last square method
>>> it use as input 2 set of x,y,z coordinates in 2 different reference
>>> systems
>>> and give as output 7 parameters needed to performs transformation
>>> from one reference system to the other one.
>>>
>>> http://en.wikipedia.org/wiki/Helmert_transformation
>>>
>>> this function is used in geodesy to "reproject" coordinates between 2
>>> different Datum
>>>
>>> i wrote it as :
>>>
>>> import numpy as np
>>> from scipy import linalg
>>>
>>> def helmert(p1, p2):
>>> ? ?L = np.loadtxt(p1)
>>> ? ?A = np.zeros((3*L.shape[0],7),float)
>>> ? ?A[ ::3, 0] = 1.0
>>> ? ?A[1::3, 1] = 1.0
>>> ? ?A[2::3, 2] = 1.0
>>> ? ?A[ ::3, 3] = L[:,0]
>>> ? ?A[1::3, 3] = L[:,1]
>>> ? ?A[2::3, 3] = L[:,2]
>>> ? ?A[1::3, 4] = L[:,2]
>>> ? ?A[2::3, 4] = -L[:,1]
>>> ? ?A[ ::3, 5] = -L[:,2]
>>> ? ?A[2::3, 5] = L[:,0]
>>> ? ?A[ ::3, 6] = L[:,1]
>>> ? ?A[1::3, 6] = -L[:,0]
>>> ? ?G = np.loadtxt(p2)
>>> ? ?Y = np.zeros((3*G.shape[0],1),float)
>>> ? ?Y[ ::3, 0] = G[:,0] - L[:,0]
>>> ? ?Y[1::3, 0] = G[:,1] - L[:,1]
>>> ? ?Y[2::3, 0] = G[:,2] - L[:,2]
>>> ? ?N = np.dot(A.T.conj(), A)
>>> ? ?T = np.dot(A.T.conj(), Y)
>>> ? ?C = np.dot(linalg.inv(N), T)
>>> ? ?print C
>>>
>>>
>>> from a pdf i find online
>>> it has numerical examples
>>> and i get different results :'(
>>>
>>>
>>> p1 :
>>>
>>> 4402553.334 727053.937 4542823.474
>>> 4399375.347 703845.876 4549215.105
>>> 4412911.150 701094.435 4536518.139
>>>
>>>
>>> p2 :
>>>
>>> 4402553.569 727053.737 4542823.332
>>> 4399375.518 703845.639 4549214.880
>>> 4412911.336 701094.214 4536517.955
>>>
>>>
>>> pdf results :
>>>
>>> x0 ? ?-9.256 m
>>> y0 ? ?-23.701 m
>>> z0 ? ?16.792 m
>>> Rx ? ?-0.0001990982 rad
>>> Ry ? ?0.0001778762 rad
>>> Rz ? ?0.00015 rad
>>> ? ? ? ?0.00000046
>>>
>>>
>> The angles here look like degrees, not radians. Also, the last factor is
>> pretty definitely wrong. It is given directly as the square root of the
>> ratio of the variance of the two sets of vectors around their means minus 1.
>> For instance
>>
>> sqrt(pt2.var(0).sum()/pt1.var(0).sum()) - 1
>>
>>
>>>
>>> my results :
>>>
>>> [[ -9.91564629e+00]
>>> ?[ -2.36172028e+01]
>>> ?[ ?1.57283853e+01]
>>> ?[ -2.68063331e-07]
>>> ?[ ?3.04330048e-06]
>>> ?[ -2.76148185e-06]
>>> ?[ -2.31598150e-06]]
>>>
>>>
>>>
>> And my result agrees with neither ;) BTW, are you going from pt2 -> pt1 or
>> vice versa?
>>


From dlrt2 at ast.cam.ac.uk  Tue Mar  2 07:04:00 2010
From: dlrt2 at ast.cam.ac.uk (David Trethewey)
Date: Tue, 02 Mar 2010 12:04:00 +0000
Subject: [SciPy-User] scikits.learn installation
In-Reply-To: <4B8CFE99.20600@cam.ac.uk>
References: <mailman.2676.1267530315.2398.scipy-user@scipy.org>
	<4B8CFE99.20600@cam.ac.uk>
Message-ID: <4B8CFEB0.3030905@ast.cam.ac.uk>

Problem is sorted now, thanks for your help.



From spons at utm.csic.es  Tue Mar  2 11:07:11 2010
From: spons at utm.csic.es (Sergi Pons Freixes)
Date: Tue, 2 Mar 2010 17:07:11 +0100
Subject: [SciPy-User] Crash with Report from scikits.timeseries
Message-ID: <2d88eb101003020807j62100bach77a413262f8c911b@mail.gmail.com>

Hi All,

When creating a simple report, I get an error which I'm not able to
find out the cause:

#Load the modules:
In [1]: import scikits.timeseries as ts
In [2]: import scikits.timeseries.lib.reportlib as rl

#Load data:
In [3]: labels = ["Temp", "S", "O2(%)", "dinophyceae", "diatoms",
"chlorophyceae", "resta", "pseudonitzschia",
"prymnesiophyceae(cocolitoforal)", "dinophysis",
"alexandrium_minutum", "alexandrium_catenella", "dinophysis_caudata",
"dinophysis_rotundata", "dinophysis_sacculus", "event"]
In [4]: cols = [0] + range(3,19)
In [5]: len(labels)
Out[5]: 16
In [6]: len(cols)
Out[6]: 17
In [7]: tmp = ts.tsfromtxt("AS.txt", freq='D', delimiter='\t',
skip_header=1, usecols=cols, datecols=0, missing_values='nan' )
In [8]: tmp.data.shape
Out[8]: (711,)

#Create Report
In [9]: report = rl.Report(tmp)
In [10]: report()
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)

/home/sergi/Dropbox/doctorat/dades_alfacs/<ipython console> in <module>()

/usr/local/lib/python2.6/dist-packages/scikits.timeseries-0.91.3-py2.6-linux-i686.egg/scikits/timeseries/lib/reportlib.pyc
in __call__(self, *tseries, **kwargs)
   344                 [datefmt_func(d)] + \
   345                 [fmt_func[i](ser.series[d - _sd]) \
--> 346                  for i, ser in enumerate(tseries)]
   347             )
   348

/usr/lib/python2.6/dist-packages/numpy/ma/core.pyc in __getitem__(self, indx)
  2576 #            msg = "Masked arrays must be filled before they
can be used as indices!"

  2577 #            raise IndexError, msg

-> 2578         dout = ndarray.__getitem__(ndarray.view(self, ndarray), indx)
  2579         # We could directly use ndarray.__getitem__ on self...

  2580         # But then we would have to modify __array_finalize__
to prevent the


IndexError: index out of bounds

Is there something wrong with my data (even it is loaded with no errors).

Regards,
Sergi

-- 
Sergi Pons Freixes
Ph.D. Student
Marine Technology Unit
Centre Mediterrani d'Investigacions Marines i Ambientals (CMIMA-CSIC)
Pg. Mar?tim Barceloneta, 37-49
E-08003 Barcelona (Spain)
Ph.  +34 93 230 95 00 (ext. 1510)
Fax. +34 93 230 95 55
spons at utm.csic.es
http://www.utm.csic.es


From guilherme at gpfreitas.com  Tue Mar  2 14:14:43 2010
From: guilherme at gpfreitas.com (Guilherme P. de Freitas)
Date: Tue, 2 Mar 2010 11:14:43 -0800
Subject: [SciPy-User] Lagrange Multipliers in optimize.slsqp
In-Reply-To: <c048da1c1002260735p44263cc3gc394b403f8f2100@mail.gmail.com>
References: <a193bbb51002251717j3ab63e73heebcf4f2e229a73e@mail.gmail.com>
	<c048da1c1002252008i124597dbl476c6e6dc95e96b0@mail.gmail.com>
	<1cd32cbb1002260555k3b72a50bsc2e606c9f0e68265@mail.gmail.com>
	<c048da1c1002260735p44263cc3gc394b403f8f2100@mail.gmail.com>
Message-ID: <a193bbb51003021114l709bbbc9oca3d4941409acd72@mail.gmail.com>

Hi everyone,

> Ah, this should be enough then to return the multipliers.  See if
> it gives what you'd expect?  Then you can file an enhancement ticket
> if you want the multipliers back.
>
> Index: slsqp.py
> ===================================================================
> --- slsqp.py    (revision 6242)
> +++ slsqp.py    (working copy)
> @@ -371,5 +371,6 @@
>         return [list(x),
>                 float(fx),
>                 int(majiter),
> -                int(mode),
> +                list(w[:m]),
> +                int(mode),
>                 exit_modes[int(mode)] ]
>
> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>

Thanks! That seems to give the right multipliers back. I'll file the ticket.

Best,

Guilherme


-- 
Guilherme P. de Freitas
http://www.gpfreitas.com


From nwagner at iam.uni-stuttgart.de  Tue Mar  2 14:31:04 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Tue, 02 Mar 2010 20:31:04 +0100
Subject: [SciPy-User] scipy.io.wavfile.read
Message-ID: <web-130931732@uni-stuttgart.de>

Hi all,

I tried to read a *.wav file using

scipy.io.wavfile.read

from scipy.io import wavfile
from pylab import plot, show
samplerate, data = wavfile.read('BagPipes.wav')

It failed with

  /home/nwagner/local/lib64/python2.6/site-packages/pytz/__init__.py:29: 
UserWarning: Module pkg_resources was already imported 
from 
/home/nwagner/local/lib64/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/pkg_resources.py, 
but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
   from pkg_resources import resource_stream
/home/nwagner/local/lib64/python2.6/site-packages/pytz/__init__.py:29: 
UserWarning: Module site was already imported from 
/usr/lib64/python2.6/site.pyc, but 
/home/nwagner/local/lib64/python2.6/site-packages/distribute-0.6.4-py2.6.egg 
is being added to sys.path
   from pkg_resources import resource_stream
/home/nwagner/local/lib64/python2.6/site-packages/scipy/io/wavfile.py:20: 
WavFileWarning: Unfamiliar format bytes
   warnings.warn("Unfamiliar format bytes", 
WavFileWarning)
/home/nwagner/local/lib64/python2.6/site-packages/scipy/io/wavfile.py:92: 
WavFileWarning: chunk not understood
   warnings.warn("chunk not understood", WavFileWarning)
Traceback (most recent call last):
   File "read_wav.py", line 9, in <module>
     samplerate, data = wavfile.read('BagPipes.wav')
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/io/wavfile.py", 
line 90, in read
     data = _read_data_chunk(fid, noc, bits)
   File 
"/home/nwagner/local/lib64/python2.6/site-packages/scipy/io/wavfile.py", 
line 43, in _read_data_chunk
     data = numpy.fromfile(fid, dtype=dtype, 
count=size//bytes)
ZeroDivisionError: integer division or modulo by zero

Any idea ?


The example file is available at 
http://www.ilovewavs.com/Effects/Music/Music.htm

Cheers,

                    Nils


From jason-sage at creativetrax.com  Tue Mar  2 14:45:30 2010
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 02 Mar 2010 13:45:30 -0600
Subject: [SciPy-User] splmake gone from documentation
Message-ID: <4B8D6ADA.9020508@creativetrax.com>

I'm teaching a numerical analysis class and we are talking about 
different kinds of splines.  I notice in some old scipy documentation 
that splmake is a nice function for making different kinds of splines 
(i.e., clamped, free, not_a_knot, etc.).  However, I can't find splmake 
in the reference documentation at http://docs.scipy.org/doc/ in either 
the 0.7 or 0.8dev docs.  I see splmake in my 0.7.0 installation, though 
(in the interpolate/interpolate.py file).

Is the function splmake deprecated?  I also notice that it is called (in 
the code) in the interp1d class init function, but it's impossible to 
specify the type (e.g., a clamped spline).  Is there a reason why the 
kind argument is not passed to splmake?

Thanks,

Jason

--
Jason Grout



From guilherme at gpfreitas.com  Tue Mar  2 14:57:17 2010
From: guilherme at gpfreitas.com (Guilherme P. de Freitas)
Date: Tue, 2 Mar 2010 11:57:17 -0800
Subject: [SciPy-User] Lagrange Multipliers in optimize.slsqp
In-Reply-To: <a193bbb51003021114l709bbbc9oca3d4941409acd72@mail.gmail.com>
References: <a193bbb51002251717j3ab63e73heebcf4f2e229a73e@mail.gmail.com>
	<c048da1c1002252008i124597dbl476c6e6dc95e96b0@mail.gmail.com>
	<1cd32cbb1002260555k3b72a50bsc2e606c9f0e68265@mail.gmail.com>
	<c048da1c1002260735p44263cc3gc394b403f8f2100@mail.gmail.com>
	<a193bbb51003021114l709bbbc9oca3d4941409acd72@mail.gmail.com>
Message-ID: <a193bbb51003021157h2fa8765dn8d8304b93349ba7a@mail.gmail.com>

Just one note on this: the multipliers don't seem very accurate. On a
*very* simple problem, the multiplier was correct only up to the 2nd
decimal digit. Here is the problem:


minimize     - [ log(x) + log(y) ]

subject to      x + y = 8


The solution is at (4, 4) and the multiplier of the single equality
constraint should be 1/4. The routine returns 0.2526. I thought this
could be because I was not providing the derivatives, but that is not
the case: when I provided the derivatives of both the objective
function ('fprime') and the constraints ('fprime_eqcons'), I also got
multipliers that were correct only up to the second decimal digit.

Here is the full code if you want to verify it by yourself (first
apply the modifications to slsqp.py as suggested by Skipper Seabold):



import numpy as np
from scipy.optimize import fmin_slsqp

def u(x):
    return -np.sum(np.log(x))

def du(x):
    return -1.0/x

def c(x):
    return np.sum(x) - 8

def dc(x):
    return np.ones((1, np.size(x)))

initguess = np.array([1, 8])

sol = fmin_slsqp(func=u, x0=initguess, fprime=du, eqcons=[c], fprime_eqcons=dc,
        full_output=1, iprint=2, acc=1.0e-14)

print(sol)
keys = ('xopt', 'fopt', 'its', 'multipliers', 'exitmode', 'exitmsg')
answer = dict(zip(keys, sol))
for key in answer:
    print(key, answer[key])



On Tue, Mar 2, 2010 at 11:14 AM, Guilherme P. de Freitas
<guilherme at gpfreitas.com> wrote:
> Hi everyone,
>
>> Ah, this should be enough then to return the multipliers. ?See if
>> it gives what you'd expect? ?Then you can file an enhancement ticket
>> if you want the multipliers back.
>>
>> Index: slsqp.py
>> ===================================================================
>> --- slsqp.py ? ?(revision 6242)
>> +++ slsqp.py ? ?(working copy)
>> @@ -371,5 +371,6 @@
>> ? ? ? ? return [list(x),
>> ? ? ? ? ? ? ? ? float(fx),
>> ? ? ? ? ? ? ? ? int(majiter),
>> - ? ? ? ? ? ? ? ?int(mode),
>> + ? ? ? ? ? ? ? ?list(w[:m]),
>> + ? ? ? ? ? ? ? ?int(mode),
>> ? ? ? ? ? ? ? ? exit_modes[int(mode)] ]
>>
>> Skipper
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
> Thanks! That seems to give the right multipliers back. I'll file the ticket.
>
> Best,
>
> Guilherme
>
>
> --
> Guilherme P. de Freitas
> http://www.gpfreitas.com
>



-- 
Guilherme P. de Freitas
http://www.gpfreitas.com


From d.l.goldsmith at gmail.com  Tue Mar  2 15:42:20 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Tue, 2 Mar 2010 12:42:20 -0800
Subject: [SciPy-User] splmake gone from documentation
In-Reply-To: <4B8D6ADA.9020508@creativetrax.com>
References: <4B8D6ADA.9020508@creativetrax.com>
Message-ID: <45d1ab481003021242t523bd7b1ya4492d6947b1fa48@mail.gmail.com>

On Tue, Mar 2, 2010 at 11:45 AM, Jason Grout <jason-sage at creativetrax.com>wrote:

> I'm teaching a numerical analysis class and we are talking about
> different kinds of splines.  I notice in some old scipy documentation
> that splmake is a nice function for making different kinds of splines
> (i.e., clamped, free, not_a_knot, etc.).  However, I can't find splmake
> in the reference documentation at http://docs.scipy.org/doc/ in either
> the 0.7 or 0.8dev docs.  I see splmake in my 0.7.0 installation, though
> (in the interpolate/interpolate.py file).
>

It also appears in the scipy doc editor:

    http://docs.scipy.org/scipy/docs/scipy.interpolate.interpolate.splmake/

though with a pretty deficient docstring (perhaps you'd be willing to fix
that?) ;-)

I confirmed that it doesn't appear online at:

    http://docs.scipy.org/doc/scipy/reference/interpolate.html

but I can't say why; perhaps because it's "buried" in module
interpolate.interpolate?  (The latter also being absent from the above
online reference page.)


> Is the function splmake deprecated?  I also notice that it is called (in
> the code) in the interp1d class init function,


This suggests to me that the sub-package designer(s) considered splmake to
be a "pseudo-private" method, primarily to be used internally within the
sub-package and too low level for general use, but that would certainly beg
your next question.


> but it's impossible to
> specify the type (e.g., a clamped spline).  Is there a reason why the
> kind argument is not passed to splmake?
>

Definitely someone else will have to answer that, sorry. :-(  (Not that my
answers above are all that useful...)

DG

>
> Thanks,
>
> Jason
>
> --
> Jason Grout
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100302/640a6cd2/attachment.html>

From timmichelsen at gmx-topmail.de  Tue Mar  2 16:56:35 2010
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Tue, 02 Mar 2010 22:56:35 +0100
Subject: [SciPy-User] calculating intersection points in probability curves
Message-ID: <hmk1ik$n2s$1@dough.gmane.org>

Hello,
I am performing a curve fit and also showing a distribution histogram
envelope in my plot.

In order to help the reader to evaluate the result I would like to draw
certain
boundaries (vertical and horizontal lines).
While I am aware on how to draw such lines with matplotlib, I would like
to know whether there
are some functions in scipy or matplotlib which help me to retrieve the
coordinates

a) at which two curves intersect
b) at which a distribution reaches a certain value?

Example:
How do I get the y-axis value which is reached by the green curve in
http://matplotlib.sourceforge.net/_images/histogram_demo_extended_021.png
at a x-axis value of in 175?

I could proably use a solver from numpy like
http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.solve.html#numpy.linalg.solve
but if I plot a distribution, the equation of the envelove is unknown at the
first place.

I'd appreciate your help or pointers to examples.

Thanks a lot in advance,
Timmie



From jason-sage at creativetrax.com  Tue Mar  2 17:01:52 2010
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 02 Mar 2010 16:01:52 -0600
Subject: [SciPy-User] splmake gone from documentation
In-Reply-To: <4B8D6ADA.9020508@creativetrax.com>
References: <4B8D6ADA.9020508@creativetrax.com>
Message-ID: <4B8D8AD0.2030202@creativetrax.com>

On 03/02/2010 01:45 PM, Jason Grout wrote:
> I'm teaching a numerical analysis class and we are talking about
> different kinds of splines.  I notice in some old scipy documentation
> that splmake is a nice function for making different kinds of splines
> (i.e., clamped, free, not_a_knot, etc.).  However, I can't find splmake
> in the reference documentation at http://docs.scipy.org/doc/ in either
> the 0.7 or 0.8dev docs.  I see splmake in my 0.7.0 installation, though
> (in the interpolate/interpolate.py file).
>
> Is the function splmake deprecated?  I also notice that it is called (in
> the code) in the interp1d class init function, but it's impossible to
> specify the type (e.g., a clamped spline).  Is there a reason why the
> kind argument is not passed to splmake?
>    


Digging a bit more, it seems that the splmake function is somewhat 
inoperable; that would explain why it's not in the docs.  Apparently it 
looks for a '_find_%s'%kind function (e.g., _find_clamped function), 
which should specify the right conditions for a clamped spline.  
However, this and similar functions look like:

def _find_clamped(xk, yk, order, conds, B):
     raise NotImplementedError
     return _find_user(xk, yk, order, conds, B)

So a user just gets a NotImplementedError whenever trying to create 
pretty much any kind of spline using the kind argument of splmake.

On the other hand, there is a function _get_spline3_Bb that appears to 
have some of the conditions specified (i.e., there is a switch 
statement, and there are some conditions for 'clamped', for example, but 
I haven't checked if these conditions are correct).  However, 
_get_spline3_Bb is not called from anything.  I don't know what to make 
of that!

Thanks for your help and time,

Jason

--
Jason Grout



From d.l.goldsmith at gmail.com  Tue Mar  2 18:04:01 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Tue, 2 Mar 2010 15:04:01 -0800
Subject: [SciPy-User] splmake gone from documentation
In-Reply-To: <4B8D8AD0.2030202@creativetrax.com>
References: <4B8D6ADA.9020508@creativetrax.com>
	<4B8D8AD0.2030202@creativetrax.com>
Message-ID: <45d1ab481003021504w3cac3379sed371816be0e4ded@mail.gmail.com>

On Tue, Mar 2, 2010 at 2:01 PM, Jason Grout <jason-sage at creativetrax.com>wrote:

> On 03/02/2010 01:45 PM, Jason Grout wrote:
> > I'm teaching a numerical analysis class and we are talking about
> > different kinds of splines.  I notice in some old scipy documentation
> > that splmake is a nice function for making different kinds of splines
> > (i.e., clamped, free, not_a_knot, etc.).  However, I can't find splmake
> > in the reference documentation at http://docs.scipy.org/doc/ in either
> > the 0.7 or 0.8dev docs.  I see splmake in my 0.7.0 installation, though
> > (in the interpolate/interpolate.py file).
> >
> > Is the function splmake deprecated?  I also notice that it is called (in
> > the code) in the interp1d class init function, but it's impossible to
> > specify the type (e.g., a clamped spline).  Is there a reason why the
> > kind argument is not passed to splmake?
> >
>
>
> Digging a bit more, it seems that the splmake function is somewhat
> inoperable; that would explain why it's not in the docs.  Apparently it
> looks for a '_find_%s'%kind function (e.g., _find_clamped function),
> which should specify the right conditions for a clamped spline.
> However, this and similar functions look like:
>
> def _find_clamped(xk, yk, order, conds, B):
>     raise NotImplementedError
>     return _find_user(xk, yk, order, conds, B)
>
> So a user just gets a NotImplementedError whenever trying to create
> pretty much any kind of spline using the kind argument of splmake.
>
> On the other hand, there is a function _get_spline3_Bb that appears to
> have some of the conditions specified (i.e., there is a switch
> statement, and there are some conditions for 'clamped', for example, but
> I haven't checked if these conditions are correct).  However,
> _get_spline3_Bb is not called from anything.  I don't know what to make
> of that!
>
> Thanks for your help and time,
>
> Jason
>

This is pretty embarrassing: it should either be implemented or cleared out,
IMO. I searched the scipy tickets for interpolate NotImplementedError and
got no hits (plenty of hits for interpolate and NotImplementedError, though)
so I'm turning this into a ticket; if I'm wrong about it being a "bug,"
someone will just mark it "Will not fix" (and presumably explain why.)

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100302/662dc359/attachment.html>

From d.l.goldsmith at gmail.com  Tue Mar  2 18:16:43 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Tue, 2 Mar 2010 15:16:43 -0800
Subject: [SciPy-User] splmake gone from documentation
In-Reply-To: <45d1ab481003021504w3cac3379sed371816be0e4ded@mail.gmail.com>
References: <4B8D6ADA.9020508@creativetrax.com>
	<4B8D8AD0.2030202@creativetrax.com>
	<45d1ab481003021504w3cac3379sed371816be0e4ded@mail.gmail.com>
Message-ID: <45d1ab481003021516t79aa4fe7j21608653f04797f8@mail.gmail.com>

On Tue, Mar 2, 2010 at 3:04 PM, David Goldsmith <d.l.goldsmith at gmail.com>wrote:

> On Tue, Mar 2, 2010 at 2:01 PM, Jason Grout <jason-sage at creativetrax.com>wrote:
>
>> On 03/02/2010 01:45 PM, Jason Grout wrote:
>> > I'm teaching a numerical analysis class and we are talking about
>> > different kinds of splines.  I notice in some old scipy documentation
>> > that splmake is a nice function for making different kinds of splines
>> > (i.e., clamped, free, not_a_knot, etc.).  However, I can't find splmake
>> > in the reference documentation at http://docs.scipy.org/doc/ in either
>> > the 0.7 or 0.8dev docs.  I see splmake in my 0.7.0 installation, though
>> > (in the interpolate/interpolate.py file).
>> >
>> > Is the function splmake deprecated?  I also notice that it is called (in
>> > the code) in the interp1d class init function, but it's impossible to
>> > specify the type (e.g., a clamped spline).  Is there a reason why the
>> > kind argument is not passed to splmake?
>> >
>>
>> Digging a bit more, it seems that the splmake function is somewhat
>> inoperable; that would explain why it's not in the docs.  Apparently it
>> looks for a '_find_%s'%kind function (e.g., _find_clamped function),
>> which should specify the right conditions for a clamped spline.
>> However, this and similar functions look like:
>>
>> def _find_clamped(xk, yk, order, conds, B):
>>     raise NotImplementedError
>>     return _find_user(xk, yk, order, conds, B)
>>
>> So a user just gets a NotImplementedError whenever trying to create
>> pretty much any kind of spline using the kind argument of splmake.
>>
>> On the other hand, there is a function _get_spline3_Bb that appears to
>> have some of the conditions specified (i.e., there is a switch
>> statement, and there are some conditions for 'clamped', for example, but
>> I haven't checked if these conditions are correct).  However,
>> _get_spline3_Bb is not called from anything.  I don't know what to make
>> of that!
>>
>> Thanks for your help and time,
>>
>> Jason
>>
>
> This is pretty embarrassing: it should either be implemented or cleared
> out, IMO. I searched the scipy tickets for interpolate NotImplementedError
> and got no hits (plenty of hits for interpolate and NotImplementedError,
> though) so I'm turning this into a ticket; if I'm wrong about it being a
> "bug," someone will just mark it "Will not fix" (and presumably explain
> why.)
>
> DG
>

Ticket filed.

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100302/a5569e05/attachment.html>

From massimodisasha at yahoo.it  Tue Mar  2 18:29:42 2010
From: massimodisasha at yahoo.it (Massimo Di Stefano)
Date: Wed, 3 Mar 2010 00:29:42 +0100
Subject: [SciPy-User] helmert implementation (7 parameters - geometric
	transformation)
In-Reply-To: <mailman.11.1267552806.496.scipy-user@scipy.org>
References: <mailman.11.1267552806.496.scipy-user@scipy.org>
Message-ID: <9C573176-8F0F-485E-A9EA-0F04F9E8ECFE@yahoo.it>

Hi,

from here :

http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.cholesky.html

seems it is applicable only on a square matrix.
is that true ?

reading this pdf :

http://geomatica.como.polimi.it/corsi/misure_geodetiche/MG_03SistRif2.pdf

scroll to the page : 15

i can see it isn't a common system :  Y = AX  
but it is :  Y = AX + B + V


the points are :



p1:
4402553.569 727053.737 4542823.332 
4399375.518 703845.639 4549214.880 
4412911.336 701094.214 4536517.955


p2 :
4402553.334 727053.937 4542823.474 
4399375.347 703845.876 4549215.105 
4412911.150 701094.435 4536518.139




tring to rewrite the code i did :

L = np.loadtxt(p1)

A = np.zeros((3*L.shape[0],7),float)
A[ ::3, 0] = 1.0
A[1::3, 1] = 1.0
A[2::3, 2] = 1.0
A[1::3, 3] = L[:,2]
A[2::3, 3] = -L[:,1]
A[ ::3, 4] = -L[:,2]
A[2::3, 4] = L[:,0]
A[ ::3, 5] = L[:,1]
A[1::3, 5] = -L[:,0]
A[ ::3, 6] = L[:,0]
A[1::3, 6] = L[:,1]
A[2::3, 6] = L[:,2]

G = np.loadtxt(p2)

V = np.zeros((3*G.shape[0],1),float)
V[ ::3, 0] = G[:,0] - L[:,0]
V[1::3, 0] = G[:,1] - L[:,1]
V[2::3, 0] = G[:,2] - L[:,2]


i used the code in the previouse mail :

N = np.dot(A.T.conj(), A)
T = np.dot(A.T.conj(), Y)
C = np.dot(linalg.inv(N), T)

to solve a system like : 

 Y = AX  

how can i change the code to solve : 

 Y = AX + B + V

instead ?

the results in the dpf are :

x0     -9.256 m 
y0     -23.701 m 
z0     16.792 m
Rx     -0.0001990982 rad 
Ry     0.0001778762 rad 
Rz     0.00015 rad 
?    0.00000046 

any suggestion, also on how to apply a decomposition can give me a great help!



Il giorno 02/mar/2010, alle ore 19.00, scipy-user-request at scipy.org ha scritto:

> Hi,
> 
> I see your coordinates are very big numbers. These can lead to an
> ill-conditioned (AtA) matrix which can lead to a wrong inverse. Try
> using cholesky decomposition instead of using linalg.inv. The method
> is outlined here http://en.wikipedia.org/wiki/Cholesky_decomposition
> and numpy has routines for doing just this. The other option is to
> subtract the centre-of-mass from your coordinates i.e. P1 -> P1x -
> P1xave, P1y - P1yave etc. Do the same for P2. Calculate the
> transformation again. The shifts should now be zero but your rotations
> and scale should be statistically independent and correct. The
> difference between the two coordinate systems' centres of mass gives
> the shifts. Transformations far away from the origin rarely behaves
> well.
> 
> Good luck.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100303/548a759d/attachment.html>

From charlesr.harris at gmail.com  Tue Mar  2 20:02:31 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 2 Mar 2010 18:02:31 -0700
Subject: [SciPy-User] helmert implementation (7 parameters - geometric
	transformation)
In-Reply-To: <9C573176-8F0F-485E-A9EA-0F04F9E8ECFE@yahoo.it>
References: <mailman.11.1267552806.496.scipy-user@scipy.org>
	<9C573176-8F0F-485E-A9EA-0F04F9E8ECFE@yahoo.it>
Message-ID: <e06186141003021702m20a363bau19546eba0eecc594@mail.gmail.com>

On Tue, Mar 2, 2010 at 4:29 PM, Massimo Di Stefano
<massimodisasha at yahoo.it>wrote:

> Hi,
>
> from here :
>
>
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.cholesky.html
>
> seems it is applicable only on a square matrix.
> is that true ?
>
> reading this pdf :
>
> http://geomatica.como.polimi.it/corsi/misure_geodetiche/MG_03SistRif2.pdf
>
> scroll to the page : 15
>
> i can see it isn't a common system :  Y = AX
> but it is :  Y = AX + B + V
>
>
> the points are :
>
>
>
> p1:
> 4402553.569 727053.737 4542823.332
> 4399375.518 703845.639 4549214.880
> 4412911.336 701094.214 4536517.955
>
>
> p2 :
> 4402553.334 727053.937 4542823.474
> 4399375.347 703845.876 4549215.105
> 4412911.150 701094.435 4536518.139
>
>
>
>
> tring to rewrite the code i did :
>
> L = np.loadtxt(p1)
>
> A = np.zeros((3*L.shape[0],7),float)
> A[ ::3, 0] = 1.0
> A[1::3, 1] = 1.0
> A[2::3, 2] = 1.0
> A[1::3, 3] = L[:,2]
> A[2::3, 3] = -L[:,1]
> A[ ::3, 4] = -L[:,2]
> A[2::3, 4] = L[:,0]
> A[ ::3, 5] = L[:,1]
> A[1::3, 5] = -L[:,0]
> A[ ::3, 6] = L[:,0]
> A[1::3, 6] = L[:,1]
> A[2::3, 6] = L[:,2]
>
> G = np.loadtxt(p2)
>
> V = np.zeros((3*G.shape[0],1),float)
> V[ ::3, 0] = G[:,0] - L[:,0]
> V[1::3, 0] = G[:,1] - L[:,1]
> V[2::3, 0] = G[:,2] - L[:,2]
>
>
> i used the code in the previouse mail :
>
> N = np.dot(A.T.conj(), A)
> T = np.dot(A.T.conj(), Y)
> C = np.dot(linalg.inv(N), T)
>
> to solve a system like :
>
>  Y = AX
>
> how can i change the code to solve :
>
>  Y = AX + B + V
>
> instead ?
>
> the results in the dpf are :
>
> *x**0     -9.256 m *
> ***y**0     -23.701 m *
> ***z**0     16.792 m*
> ***R**x     -0.0001990982 rad *
> ***R**y     0.0001778762 rad *
> ***R**z     0.00015 rad *
> **?    0.00000046
>
> any suggestion, also on how to apply a decomposition can give me a great
> help!
>
>
I've attached a script for the computation. Two things of note: the angles
in the pdf results are in degrees, the value of ? is pretty much in the
noise (the condition number of the matrix is 10^10), and the fit you get
with my version differs from the data by about 7mm. Note that the Helmert
parameters result from a linearization of the rotation matrix, see
Gauss-Newton for more about linearization and least squares, but in this
case I think the limiting factor is the accuracy of the coordinates, not the
linearization.
<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100302/113cb4b3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: helmert.py
Type: text/x-python
Size: 876 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100302/113cb4b3/attachment.py>

From david at silveregg.co.jp  Tue Mar  2 20:24:38 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Wed, 03 Mar 2010 10:24:38 +0900
Subject: [SciPy-User] scipy.io.wavfile.read
In-Reply-To: <web-130931732@uni-stuttgart.de>
References: <web-130931732@uni-stuttgart.de>
Message-ID: <4B8DBA56.5090207@silveregg.co.jp>

Nils Wagner wrote:
> Hi all,
> 
> I tried to read a *.wav file using
> 
> scipy.io.wavfile.read
> 
> from scipy.io import wavfile
> from pylab import plot, show
> samplerate, data = wavfile.read('BagPipes.wav')
> 

This file uses ADPCM encoding (a form of compression based on non linear 
encoding of each sample), which is most likely not supported by 
scipy.io.wavfile. Your two options are to convert this to a "normal" wav 
file with linear encoding, or to use audiolab which supports those files 
(because libsndile does).

cheeers,

David


From jason-sage at creativetrax.com  Tue Mar  2 20:42:58 2010
From: jason-sage at creativetrax.com (Jason Grout)
Date: Tue, 02 Mar 2010 19:42:58 -0600
Subject: [SciPy-User] splmake gone from documentation
In-Reply-To: <45d1ab481003021516t79aa4fe7j21608653f04797f8@mail.gmail.com>
References: <4B8D6ADA.9020508@creativetrax.com>	<4B8D8AD0.2030202@creativetrax.com>	<45d1ab481003021504w3cac3379sed371816be0e4ded@mail.gmail.com>
	<45d1ab481003021516t79aa4fe7j21608653f04797f8@mail.gmail.com>
Message-ID: <4B8DBEA2.2020003@creativetrax.com>

On 03/02/2010 05:16 PM, David Goldsmith wrote:
> On Tue, Mar 2, 2010 at 3:04 PM, David Goldsmith 
> <d.l.goldsmith at gmail.com <mailto:d.l.goldsmith at gmail.com>> wrote:
>
>     On Tue, Mar 2, 2010 at 2:01 PM, Jason Grout
>     <jason-sage at creativetrax.com <mailto:jason-sage at creativetrax.com>>
>     wrote:
>
>         On 03/02/2010 01:45 PM, Jason Grout wrote:
>         > I'm teaching a numerical analysis class and we are talking about
>         > different kinds of splines.  I notice in some old scipy
>         documentation
>         > that splmake is a nice function for making different kinds
>         of splines
>         > (i.e., clamped, free, not_a_knot, etc.).  However, I can't
>         find splmake
>         > in the reference documentation at http://docs.scipy.org/doc/
>         in either
>         > the 0.7 or 0.8dev docs.  I see splmake in my 0.7.0
>         installation, though
>         > (in the interpolate/interpolate.py file).
>         >
>         > Is the function splmake deprecated?  I also notice that it
>         is called (in
>         > the code) in the interp1d class init function, but it's
>         impossible to
>         > specify the type (e.g., a clamped spline).  Is there a
>         reason why the
>         > kind argument is not passed to splmake?
>         >
>
>         Digging a bit more, it seems that the splmake function is somewhat
>         inoperable; that would explain why it's not in the docs.
>          Apparently it
>         looks for a '_find_%s'%kind function (e.g., _find_clamped
>         function),
>         which should specify the right conditions for a clamped spline.
>         However, this and similar functions look like:
>
>         def _find_clamped(xk, yk, order, conds, B):
>             raise NotImplementedError
>             return _find_user(xk, yk, order, conds, B)
>
>         So a user just gets a NotImplementedError whenever trying to
>         create
>         pretty much any kind of spline using the kind argument of splmake.
>
>         On the other hand, there is a function _get_spline3_Bb that
>         appears to
>         have some of the conditions specified (i.e., there is a switch
>         statement, and there are some conditions for 'clamped', for
>         example, but
>         I haven't checked if these conditions are correct).  However,
>         _get_spline3_Bb is not called from anything.  I don't know
>         what to make
>         of that!
>
>         Thanks for your help and time,
>
>         Jason
>
>
>     This is pretty embarrassing: it should either be implemented or
>     cleared out, IMO. I searched the scipy tickets for interpolate
>     NotImplementedError and got no hits (plenty of hits for
>     interpolate and NotImplementedError, though) so I'm turning this
>     into a ticket; if I'm wrong about it being a "bug," someone will
>     just mark it "Will not fix" (and presumably explain why.)
>
>     DG
>
>
> Ticket filed.

Tracking this down a bit more through the revision history (using 
someone's git repository, since it's much easier for me to deal with git 
than svn...)

This commit commented out the call to the spline3_Bb function (the call 
was then lines 597-600 in the splmake function)

oliphant, 5/27/07 5:58 AM, Fix up recently added interpolation so that 
it is more general and scales to higher orders by using the B-spline 
representation of a piece-wise polynomial curve. Implemented the deBoor 
algorithm multiple times.



Then the next commit to interpolate.py:

oliphant, 5/30/07 4:49 AM, Set-up interpolate structure to support more 
than just smoothest condition.

deleted the commented call to spline3_Bb and replaced it with a call to 
the find_* functions, and introduced a lot of find_* functions that were 
the stubs that we see today (basically, none of the options are 
implemented).  The spline3_Bb function was left in, but nothing calls it.

Travis, I'm curious about the rationale behind these changes.  Were you 
planning on copying the code from spline3_Bb into the various stub 
_find_* functions, or was there something that made it difficult to do that?

Thanks,

Jason

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100302/ed3a7a27/attachment.html>

From pitam at deepspace.ucsb.edu  Tue Mar  2 21:00:22 2010
From: pitam at deepspace.ucsb.edu (pitam)
Date: Tue, 02 Mar 2010 18:00:22 -0800
Subject: [SciPy-User]  segfaults with weave.inline
Message-ID: <1267581622.9204.8.camel@zephyr.deepspace.ucsb.edu>

Hi,

I did some more testing on this. The zone_vector() had a wrong comment
on it. Its really small, it is 8*(number of elements in the star shaped
pattern - that is zone_outer_radius - zone_inner_radius)

I added some extra lines to see where the thing breaks - it does the
median filter to the full image. After the last pixel, it breaks. This
is about the same time where the C code should end.

Hence, my guess is there is a memory leak. IS there any way to check
this?

Pitam Mitra
Deepspace Group,
UCSB Physics.









From pitam at deepspace.ucsb.edu  Tue Mar  2 23:08:08 2010
From: pitam at deepspace.ucsb.edu (pitam)
Date: Tue, 02 Mar 2010 20:08:08 -0800
Subject: [SciPy-User]  segfaults with weave.inline
Message-ID: <1267589288.9204.11.camel@zephyr.deepspace.ucsb.edu>

I did some more research on this.

My code uses blitz array, where the original array was a numeric array
of numpy. This is where the fault is at. I need to convert the blitz
array back to a numeric numpy array. How do I do this, any Ideas??


Original post:
http://mail.scipy.org/pipermail/scipy-user/2010-March/024478.html

Follow-ups:
http://mail.scipy.org/pipermail/scipy-user/2010-March/024498.html


Pitam Mitra
Deepspace Group
UCSB Physics




From sebastian.walter at gmail.com  Wed Mar  3 04:14:39 2010
From: sebastian.walter at gmail.com (Sebastian Walter)
Date: Wed, 3 Mar 2010 10:14:39 +0100
Subject: [SciPy-User] Google Summer of Code...
In-Reply-To: <EEDF0DA5-C8FD-4AA1-9CF5-B0708B079E6B@cs.toronto.edu>
References: <4B8C3C81.7000509@noaa.gov>
	<1cd32cbb1003011435u725d37e5ne3c98475f3cfa8a@mail.gmail.com>
	<3d375d731003011443h1a5c0b02h837f5af3f5348037@mail.gmail.com>
	<c048da1c1003011459g6d8faa6eg38751db3f88b6417@mail.gmail.com>
	<EEDF0DA5-C8FD-4AA1-9CF5-B0708B079E6B@cs.toronto.edu>
Message-ID: <ec9f80fa1003030114i14018e00n6153a9dc6926a860@mail.gmail.com>

I could mentor an algorithmic differentiation (AD) project, but I'm
still not quite sure how to proceed.
Should I write up a proposal on the wiki and after that propose the
project on the
http://mail.python.org/mailman/listinfo/soc2010-mentors mailing list?

I have already created a package that I believe would be a very good
addition to scipy/numpy (preferably numpy) when it is mature enough.
It is hosted on http://github.com/b45ch1/taylorpoly

The package is not an AD tool but makes it very easy to create AD
tools using those algorithms.
An illustrative example how to do (forward mode) AD is given in:
http://github.com/b45ch1/taylorpoly/blob/master/examples/utps_for_algorithmic_differentiation.py

What is missing up to now are differentiated algorithms for tan, atan,
asin, sinh, etc. (in Taylor arithmetic)
as well as the core algorithms for reverse mode AD.

Implementing those algorithms should be doable in 4 months. Also,
since the task is to implement many small algorithms,
this project is unlikely to fail completely. The worst that can happen
is that not as many algorithms get implemented as one could wish for.

Could anyone give feedback if that project is ok?

best regards,
Sebastian





On Tue, Mar 2, 2010 at 12:32 AM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> On 1-Mar-10, at 5:59 PM, Skipper Seabold wrote:
>
>> I would also encourage anyone who has ideas to edit the wiki here to
>> reflect the current project wishlist (especially if you're willing to
>> mentor): http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas ?IMHO,
>> the more ideas there are, the more chance of grabbing interested and
>> diverse sets of students provided people are willing to mentor.
>
> I just reorganized the page a bit, to separate out the stuff that is
> already going on, like statsmodels, fwrap and the Py3k/datetime stuff,
> from things that have been listed for a while but never really picked
> up on.
>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From nwagner at iam.uni-stuttgart.de  Wed Mar  3 04:47:28 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Wed, 03 Mar 2010 10:47:28 +0100
Subject: [SciPy-User] scipy.io.wavfile.read
In-Reply-To: <4B8DBA56.5090207@silveregg.co.jp>
References: <web-130931732@uni-stuttgart.de> <4B8DBA56.5090207@silveregg.co.jp>
Message-ID: <web-130951090@uni-stuttgart.de>

On Wed, 03 Mar 2010 10:24:38 +0900
  David Cournapeau <david at silveregg.co.jp> wrote:
> Nils Wagner wrote:
>> Hi all,
>> 
>> I tried to read a *.wav file using
>> 
>> scipy.io.wavfile.read
>> 
>> from scipy.io import wavfile
>> from pylab import plot, show
>> samplerate, data = wavfile.read('BagPipes.wav')
>> 
> 
> This file uses ADPCM encoding (a form of compression 
>based on non linear 
> encoding of each sample), which is most likely not 
>supported by 
> scipy.io.wavfile. Your two options are to convert this 
>to a "normal" wav 
> file with linear encoding, or to use audiolab which 
>supports those files 
> (because libsndile does).
> 

Is it possible to improve the message returned by 
wavfile.read in case of special encodings ?
  
Nils


From pav+sp at iki.fi  Wed Mar  3 10:22:07 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Wed, 3 Mar 2010 15:22:07 +0000 (UTC)
Subject: [SciPy-User] helmert implementation (7 parameters -
	geometric	transformation)
References: <mailman.11.1267552806.496.scipy-user@scipy.org>
	<9C573176-8F0F-485E-A9EA-0F04F9E8ECFE@yahoo.it>
Message-ID: <hmluqv$hvn$1@dough.gmane.org>

Wed, 03 Mar 2010 00:29:42 +0100, Massimo Di Stefano wrote:
> from here :
> 
> http://docs.scipy.org/doc/numpy/reference/generated/
> numpy.linalg.cholesky.html
> 
> seems it is applicable only on a square matrix. is that true ?

Cholesky decomposition works only on Hermitian matrices -- so being 
square is one requirement.

Also, if you are going to do least-squares fitting, use linalg.lstsq; it 
does the same thing as (A^T A)^-1 A^T b, but should be numerically more 
stable.

-- 
Pauli Virtanen



From pav+sp at iki.fi  Wed Mar  3 10:36:59 2010
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Wed, 3 Mar 2010 15:36:59 +0000 (UTC)
Subject: [SciPy-User] segfaults with weave.inline
References: <1267589288.9204.11.camel@zephyr.deepspace.ucsb.edu>
Message-ID: <hmlvmr$hvn$2@dough.gmane.org>

Tue, 02 Mar 2010 20:08:08 -0800, pitam wrote:
> I did some more research on this.
> 
> My code uses blitz array, where the original array was a numeric array
> of numpy. This is where the fault is at. I need to convert the blitz
> array back to a numeric numpy array. How do I do this, any Ideas??

As said, to my understanding,

1. zone_vector is modified *in-place*, you don't need to convert it back.

2. Instead, you have to allocate the numpy array big enough to hold *all*
   of the data

3. Blitz does not check array bounds

4. Writing out of array bounds does not necessarily result to a
   segmentation fault immediately, it can manifest later.

So, you should replace

	zone_vector = array(())

with

	zone_vector = zeros((size_of_zone_vector,))

Did you try this? If yes, does it fix the problem?

Best regards,
-- 
Pauli Virtanen



From nwagner at iam.uni-stuttgart.de  Wed Mar  3 12:06:14 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Wed, 03 Mar 2010 18:06:14 +0100
Subject: [SciPy-User] scikits.audiolab NameError
Message-ID: <web-130973881@uni-stuttgart.de>

Hi all,

I installed audiolab via

git clone git://github.com/cournape/audiolab.git
python setup.py install --prefix=$HOME/local

How can I fix the backend problem wrt. alsa ?
  

python -i test_sndfile.py
/data/home/nwagner/local/lib/python2.5/site-packages/scikits.audiolab-0.11.0.dev-py2.5-linux-x86_64.egg/scikits/audiolab/soundio/play.py:48: 
UserWarning: Could not import alsa backend; most probably, 
you did not have alsa headers when building audiolab
   warnings.warn("Could not import alsa backend; most 
probably, "
Traceback (most recent call last):
   File "test_sndfile.py", line 332, in <module>
     NumpyTest().run()
NameError: name 'NumpyTest' is not defined
  

AFAIK, NumpyTest is obsolete. Am I missing something ?

Nils


From agile.aspect at gmail.com  Wed Mar  3 13:01:22 2010
From: agile.aspect at gmail.com (Agile Aspect)
Date: Wed, 3 Mar 2010 10:01:22 -0800
Subject: [SciPy-User] scikits.audiolab NameError
In-Reply-To: <web-130973881@uni-stuttgart.de>
References: <web-130973881@uni-stuttgart.de>
Message-ID: <c6a6f63b1003031001q1abef478w881b9d4a0260c3dd@mail.gmail.com>

On Wed, Mar 3, 2010 at 9:06 AM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> Hi all,
>
> I installed audiolab via
>
> git clone git://github.com/cournape/audiolab.git
> python setup.py install --prefix=$HOME/local
>
> How can I fix the backend problem wrt. alsa ?
>
>
> python -i test_sndfile.py
> /data/home/nwagner/local/lib/python2.5/site-packages/scikits.audiolab-0.11.0.dev-py2.5-linux-x86_64.egg/scikits/audiolab/soundio/play.py:48:
> UserWarning: Could not import alsa backend; most probably,
> you did not have alsa headers when building audiolab
> ? warnings.warn("Could not import alsa backend; most
> probably, "
> Traceback (most recent call last):
> ? File "test_sndfile.py", line 332, in <module>
> ? ? NumpyTest().run()
> NameError: name 'NumpyTest' is not defined
>
>
> AFAIK, NumpyTest is obsolete. Am I missing something ?
>
> Nils
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>

On CentOS/Fedora

       yum install alsa\*

-- 
      Enjoy global warming while it lasts.


From nwagner at iam.uni-stuttgart.de  Wed Mar  3 14:01:55 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Wed, 03 Mar 2010 20:01:55 +0100
Subject: [SciPy-User] scikits.audiolab NameError
In-Reply-To: <c6a6f63b1003031001q1abef478w881b9d4a0260c3dd@mail.gmail.com>
References: <web-130973881@uni-stuttgart.de>
	<c6a6f63b1003031001q1abef478w881b9d4a0260c3dd@mail.gmail.com>
Message-ID: <web-130976895@uni-stuttgart.de>

On Wed, 3 Mar 2010 10:01:22 -0800
  Agile Aspect <agile.aspect at gmail.com> wrote:
> On Wed, Mar 3, 2010 at 9:06 AM, Nils Wagner
> <nwagner at iam.uni-stuttgart.de> wrote:
>> Hi all,
>>
>> I installed audiolab via
>>
>> git clone git://github.com/cournape/audiolab.git
>> python setup.py install --prefix=$HOME/local
>>
>> How can I fix the backend problem wrt. alsa ?
>>
>>
>> python -i test_sndfile.py
>> /data/home/nwagner/local/lib/python2.5/site-packages/scikits.audiolab-0.11.0.dev-py2.5-linux-x86_64.egg/scikits/audiolab/soundio/play.py:48:
>> UserWarning: Could not import alsa backend; most 
>>probably,
>> you did not have alsa headers when building audiolab
>> ? warnings.warn("Could not import alsa backend; most
>> probably, "
>> Traceback (most recent call last):
>> ? File "test_sndfile.py", line 332, in <module>
>> ? ? NumpyTest().run()
>> NameError: name 'NumpyTest' is not defined
>>
>>
>> AFAIK, NumpyTest is obsolete. Am I missing something ?
>>
>> Nils
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> 
> On CentOS/Fedora
> 
>       yum install alsa\*
> 
  
alsa is already installed...

header file are available in /usr/include/alsa
the library libasound.a is in /usr/lib64

I haven't a clue what is going on here.

Nils


From PHobson at Geosyntec.com  Wed Mar  3 14:09:37 2010
From: PHobson at Geosyntec.com (PHobson at Geosyntec.com)
Date: Wed, 3 Mar 2010 14:09:37 -0500
Subject: [SciPy-User] Scipy's probplot compared to R's qqplot
Message-ID: <C97487CD346EEA4E80423F51F6D1EB2FA976939437@EXCHANGE-00.GeoSyntec.net>

Hey folks,

I've taken more of an interest in statistics and Scipy lately and decided to compare the scipy.stats.probplot() function to R's qqplot(). For a given dataset, the results are slightly different.

Here's a link to the script I wrote to do the comparison. 
http://dpaste.com/167464/

Basically, it does the following:
-Uses numpy to generate some fake, noramlly distributed data
-Uses both R and Scipy to compute the values needed for quantile/probability plot
-Computes linear regressions on the quantile data with both R and Scipy.
-prints some output to compare the two

My initial conclusions:
1) R's lm(y~x) and scipy.stats.linregress(x,y) yield the same slope and intercept of a linear model. (good)
2) R and Scipy compute the quantiles of a dataset in slightly different manners (??)

Any clue as to why the discrepancy in #2 occurs? Would you consider it a big deal? I'm using:
Python v2.6.2 (XP) and v2.6.4 (Karmic and Snow Leopard)
Scipy v0.7.1
Numpy v1.4.0
R v2.10.0
Rpy2 v2.0.8

Thanks,
-Paul H.


From josef.pktd at gmail.com  Wed Mar  3 14:36:39 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 3 Mar 2010 14:36:39 -0500
Subject: [SciPy-User] Scipy's probplot compared to R's qqplot
In-Reply-To: <C97487CD346EEA4E80423F51F6D1EB2FA976939437@EXCHANGE-00.GeoSyntec.net>
References: <C97487CD346EEA4E80423F51F6D1EB2FA976939437@EXCHANGE-00.GeoSyntec.net>
Message-ID: <1cd32cbb1003031136w3f7dfcc8qdba53be55fef3cd5@mail.gmail.com>

On Wed, Mar 3, 2010 at 2:09 PM,  <PHobson at geosyntec.com> wrote:
> Hey folks,
>
> I've taken more of an interest in statistics and Scipy lately and decided to compare the scipy.stats.probplot() function to R's qqplot(). For a given dataset, the results are slightly different.
>
> Here's a link to the script I wrote to do the comparison.
> http://dpaste.com/167464/
>
> Basically, it does the following:
> -Uses numpy to generate some fake, noramlly distributed data
> -Uses both R and Scipy to compute the values needed for quantile/probability plot
> -Computes linear regressions on the quantile data with both R and Scipy.
> -prints some output to compare the two
>
> My initial conclusions:
> 1) R's lm(y~x) and scipy.stats.linregress(x,y) yield the same slope and intercept of a linear model. (good)
> 2) R and Scipy compute the quantiles of a dataset in slightly different manners (??)
>
> Any clue as to why the discrepancy in #2 occurs? Would you consider it a big deal?

I would consider any significant deviation a big deal, unless we know
that there are differences in the definitions or underlying
assumptions.

I'm not sure what's going on since I never looked at the details of
probplot. However, when I plot the quantiles
>>> plt.plot(np.sort(qR))
>>> plt.plot(qS[0])
>>> plt.show()

then the graph looks almost the same except for the first and last point.

qS[0]-np.sort(qR)

differs in the second decimal, except for first and last observation.
My guess would be that there are some differences for example in the
continuity correction, or similar.

The boundary points, however, look suspicious.

Thanks for checking this,

Josef




 I'm using:
> Python v2.6.2 (XP) and v2.6.4 (Karmic and Snow Leopard)
> Scipy v0.7.1
> Numpy v1.4.0
> R v2.10.0
> Rpy2 v2.0.8
>
> Thanks,
> -Paul H.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From robert.kern at gmail.com  Wed Mar  3 14:38:21 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 3 Mar 2010 13:38:21 -0600
Subject: [SciPy-User] Scipy's probplot compared to R's qqplot
In-Reply-To: <C97487CD346EEA4E80423F51F6D1EB2FA976939437@EXCHANGE-00.GeoSyntec.net>
References: <C97487CD346EEA4E80423F51F6D1EB2FA976939437@EXCHANGE-00.GeoSyntec.net>
Message-ID: <3d375d731003031138p42db6a4dg3ad5285ee34436c4@mail.gmail.com>

On Wed, Mar 3, 2010 at 13:09,  <PHobson at geosyntec.com> wrote:
> Hey folks,
>
> I've taken more of an interest in statistics and Scipy lately and decided to compare the scipy.stats.probplot() function to R's qqplot(). For a given dataset, the results are slightly different.
>
> Here's a link to the script I wrote to do the comparison.
> http://dpaste.com/167464/
>
> Basically, it does the following:
> -Uses numpy to generate some fake, noramlly distributed data
> -Uses both R and Scipy to compute the values needed for quantile/probability plot
> -Computes linear regressions on the quantile data with both R and Scipy.
> -prints some output to compare the two
>
> My initial conclusions:
> 1) R's lm(y~x) and scipy.stats.linregress(x,y) yield the same slope and intercept of a linear model. (good)
> 2) R and Scipy compute the quantiles of a dataset in slightly different manners (??)
>
> Any clue as to why the discrepancy in #2 occurs?

There are several, slightly different but mostly reasonable ways of
computing quantiles.

> Would you consider it a big deal?

Probably not, but I'm happy to entertain arguments to the contrary if
you would care to explain how R is computing the quantiles.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From PHobson at Geosyntec.com  Wed Mar  3 14:49:53 2010
From: PHobson at Geosyntec.com (PHobson at Geosyntec.com)
Date: Wed, 3 Mar 2010 14:49:53 -0500
Subject: [SciPy-User] Scipy's probplot compared to R's qqplot
In-Reply-To: <1cd32cbb1003031136w3f7dfcc8qdba53be55fef3cd5@mail.gmail.com>
References: <C97487CD346EEA4E80423F51F6D1EB2FA976939437@EXCHANGE-00.GeoSyntec.net>
	<1cd32cbb1003031136w3f7dfcc8qdba53be55fef3cd5@mail.gmail.com>
Message-ID: <C97487CD346EEA4E80423F51F6D1EB2FA97693943F@EXCHANGE-00.GeoSyntec.net>

> On Wed, Mar 3, 2010 at 2:09 PM,  <PHobson at geosyntec.com> wrote:
> > Hey folks,
> >
> > I've taken more of an interest in statistics and Scipy lately and
> decided to compare the scipy.stats.probplot() function to R's qqplot().
> For a given dataset, the results are slightly different.
> >
> > Here's a link to the script I wrote to do the comparison.
> > http://dpaste.com/167464/
> >
> > Basically, it does the following:
> > -Uses numpy to generate some fake, noramlly distributed data
> > -Uses both R and Scipy to compute the values needed for
> quantile/probability plot
> > -Computes linear regressions on the quantile data with both R and
> Scipy.
> > -prints some output to compare the two
> >
> > My initial conclusions:
> > 1) R's lm(y~x) and scipy.stats.linregress(x,y) yield the same slope and
> intercept of a linear model. (good)
> > 2) R and Scipy compute the quantiles of a dataset in slightly different
> manners (??)
> >
> > Any clue as to why the discrepancy in #2 occurs? Would you consider it
> a big deal?


> From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org]
> On Behalf Of josef.pktd at gmail.com
> I would consider any significant deviation a big deal, unless we know
> that there are differences in the definitions or underlying
> assumptions.
> 
> I'm not sure what's going on since I never looked at the details of
> probplot. However, when I plot the quantiles
> >>> plt.plot(np.sort(qR))
> >>> plt.plot(qS[0])
> >>> plt.show()
> 
> then the graph looks almost the same except for the first and last point.

Yes. When I plotted them, I could not visually distinguish them (see attached). I forgot to mention that.

> qS[0]-np.sort(qR)
> 
> differs in the second decimal, except for first and last observation.
> My guess would be that there are some differences for example in the
> continuity correction, or similar.
> 
> The boundary points, however, look suspicious.

Thanks for looking  further into this. When I saw that the slopes and intercepts were different, I immediately inspected just the max and min values (laziness, sorry). If I find some time next week, I'll dig around in the source and see if I can't figure out what's happening at those points.
-Paul H.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: prob_plot_test.png
Type: image/png
Size: 23775 bytes
Desc: prob_plot_test.png
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100303/753beaee/attachment.png>

From josef.pktd at gmail.com  Wed Mar  3 15:08:00 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 3 Mar 2010 15:08:00 -0500
Subject: [SciPy-User] Scipy's probplot compared to R's qqplot
In-Reply-To: <C97487CD346EEA4E80423F51F6D1EB2FA97693943F@EXCHANGE-00.GeoSyntec.net>
References: <C97487CD346EEA4E80423F51F6D1EB2FA976939437@EXCHANGE-00.GeoSyntec.net>
	<1cd32cbb1003031136w3f7dfcc8qdba53be55fef3cd5@mail.gmail.com>
	<C97487CD346EEA4E80423F51F6D1EB2FA97693943F@EXCHANGE-00.GeoSyntec.net>
Message-ID: <1cd32cbb1003031208j7dc059a3y47288da1c2d39feb@mail.gmail.com>

On Wed, Mar 3, 2010 at 2:49 PM,  <PHobson at geosyntec.com> wrote:
>> On Wed, Mar 3, 2010 at 2:09 PM, ?<PHobson at geosyntec.com> wrote:
>> > Hey folks,
>> >
>> > I've taken more of an interest in statistics and Scipy lately and
>> decided to compare the scipy.stats.probplot() function to R's qqplot().
>> For a given dataset, the results are slightly different.
>> >
>> > Here's a link to the script I wrote to do the comparison.
>> > http://dpaste.com/167464/
>> >
>> > Basically, it does the following:
>> > -Uses numpy to generate some fake, noramlly distributed data
>> > -Uses both R and Scipy to compute the values needed for
>> quantile/probability plot
>> > -Computes linear regressions on the quantile data with both R and
>> Scipy.
>> > -prints some output to compare the two
>> >
>> > My initial conclusions:
>> > 1) R's lm(y~x) and scipy.stats.linregress(x,y) yield the same slope and
>> intercept of a linear model. (good)
>> > 2) R and Scipy compute the quantiles of a dataset in slightly different
>> manners (??)
>> >
>> > Any clue as to why the discrepancy in #2 occurs? Would you consider it
>> a big deal?
>
>
>> From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org]
>> On Behalf Of josef.pktd at gmail.com
>> I would consider any significant deviation a big deal, unless we know
>> that there are differences in the definitions or underlying
>> assumptions.
>>
>> I'm not sure what's going on since I never looked at the details of
>> probplot. However, when I plot the quantiles
>> >>> plt.plot(np.sort(qR))
>> >>> plt.plot(qS[0])
>> >>> plt.show()
>>
>> then the graph looks almost the same except for the first and last point.
>
> Yes. When I plotted them, I could not visually distinguish them (see attached). I forgot to mention that.
>
>> qS[0]-np.sort(qR)
>>
>> differs in the second decimal, except for first and last observation.
>> My guess would be that there are some differences for example in the
>> continuity correction, or similar.
>>
>> The boundary points, however, look suspicious.
>
> Thanks for looking ?further into this. When I saw that the slopes and intercepts were different, I immediately inspected just the max and min values (laziness, sorry). If I find some time next week, I'll dig around in the source and see if I can't figure out what's happening at those points.

my prime candidate for the 2nd decimal differences, are differences in
the correction

Ui[1:-1] = (i-0.3175)/(N+0.365)

There are several conventions, David Huard posted a list of them
attached to a ticket (?), for empirical cdf.

There might be another correction for boundary points that is different.
    Ui[-1] = 0.5**(1.0/N)
    Ui[0] = 1-Ui[-1]

But for graphical inspection, R and scipy look close enough.

Josef





> -Paul H.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From PHobson at Geosyntec.com  Wed Mar  3 15:15:05 2010
From: PHobson at Geosyntec.com (PHobson at Geosyntec.com)
Date: Wed, 3 Mar 2010 15:15:05 -0500
Subject: [SciPy-User] Scipy's probplot compared to R's qqplot
In-Reply-To: <3d375d731003031138p42db6a4dg3ad5285ee34436c4@mail.gmail.com>
References: <C97487CD346EEA4E80423F51F6D1EB2FA976939437@EXCHANGE-00.GeoSyntec.net>
	<3d375d731003031138p42db6a4dg3ad5285ee34436c4@mail.gmail.com>
Message-ID: <C97487CD346EEA4E80423F51F6D1EB2FA976939448@EXCHANGE-00.GeoSyntec.net>

> On Wed, Mar 3, 2010 at 13:09,  <PHobson at geosyntec.com> wrote:
> > 2) R and Scipy compute the quantiles of a dataset in slightly different
> > manners (??)
> >
> > Any clue as to why the discrepancy in #2 occurs?

> From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org]
> On Behalf Of Robert Kern
> There are several, slightly different but mostly reasonable ways of
> computing quantiles.

Good to know. Thanks.

> > Would you consider it a big deal?

> Probably not, but I'm happy to entertain arguments to the contrary if
> you would care to explain how R is computing the quantiles.

After reading Josef's response, I'm only concerned with the min/max values. R, like you, mentions that several methods are available to compute quantiles. By default, it uses what it calls Type 7:
""" 
     All sample quantiles are defined as weighted averages of
     consecutive order statistics. Sample quantiles of type i are
     defined by:

                 Q[i](p) = (1 - gamma) x[j] + gamma x[j+1],             
     
     where 1 <= i <= 9, (j-m)/n <= p < (j-m+1)/n, x[j] is the jth order
     statistic, n is the sample size, the value of gamma is a function
     of j = floor(np + m) and g = np + m - j, and m is a constant
     determined by the sample quantile type.
[snip]
     *Continuous sample quantile types 4 through 9*

     For types 4 through 9, Q[i](p) is a continuous function of p, with
     gamma = g and m given below. The sample quantiles can be obtained
     equivalently by linear interpolation between the points
     (p[k],x[k]) where x[k] is the kth order statistic.  Specific
     expressions for p[k] are given below.
[snip]
     Type 7 m = 1-p.  p[k] = (k - 1) / (n - 1).  In this case, p[k] =
          mode[F(x[k])].  This is used by S.
"""

I haven't really had time to look into Scipy's method yet. I'll dig a little deeper as soon as I get the chance.

Thanks for the insight,
-Paul H.


From robert.kern at gmail.com  Wed Mar  3 16:15:14 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 3 Mar 2010 15:15:14 -0600
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <1cd32cbb1002230959u46a4da49ie6e7220d5964a381@mail.gmail.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<b9424bc2-9955-4bbd-8ce1-91ccc2f7f0e8@q16g2000yqq.googlegroups.com> 
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com> 
	<43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com> 
	<1cd32cbb1002220845o3ad94ba8g599f25ec170c6106@mail.gmail.com> 
	<1cd32cbb1002220927h229e578bhe18737d803fddc76@mail.gmail.com> 
	<1c841a64-786d-4069-9f88-fcaae5a0d063@t42g2000vbt.googlegroups.com> 
	<3d375d731002230946m75c7cacbk4a45e8e3db518d54@mail.gmail.com> 
	<1cd32cbb1002230957w35b7119ay7270bbbc1892ce25@mail.gmail.com> 
	<1cd32cbb1002230959u46a4da49ie6e7220d5964a381@mail.gmail.com>
Message-ID: <3d375d731003031315p7bc93005ibd7fed09c2ccbd41@mail.gmail.com>

On Tue, Feb 23, 2010 at 11:59,  <josef.pktd at gmail.com> wrote:
> On Tue, Feb 23, 2010 at 12:57 PM, ?<josef.pktd at gmail.com> wrote:
>> On Tue, Feb 23, 2010 at 12:46 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Tue, Feb 23, 2010 at 11:43, denis <denis-bz-gg at t-online.de> wrote:
>>>> Robert, Josef,
>>>> ?thanks much for taking the time to look at RBF some more.
>>>> Summary, correct me:
>>>> ? ?A - smooth*I in rbf.py is a sign error (ticket ?)
>>>
>>> Not necessarily. It seems to work well in at least some cases. Find a
>>> reference that says otherwise if you want it changed.
>>
>> chapter 2 page 16, for gaussian process. As I said I don't know about
>> the other methods
>>
>
> http://docs.google.com/viewer?a=v&q=cache:qs8AaAxO6nkJ:www.gaussianprocess.org/gpml/chapters/RW2.pdf+gaussian+process+noise+Ridge&hl=en&gl=ca&pid=bl&srcid=ADGEESj4j8osT6cOIc65r3OaeAtQO_dzgZD4YxSAEkFTeRZajBcROJpJJ9zTlMSrD2OaK1iOJYgy8QqH_Nr0rNxf41faNihCdIzWyVOYxtCFIR7H8mdQZAKFoeaRkFamQlCKhp_s1FOI&sig=AHIEtbQK35MLfnZAySw3lF-dR_mNcSaP3w
>
> google links are very short, missed a part

I've found a couple of things on RBFs specifically that agree.
However, the original source on which our implementation is based does
subtract. He may have a source that uses a negative sign.

http://www.mathworks.co.uk/matlabcentral/fileexchange/10056-scattered-data-interpolation-and-approximation-using-radial-base-functions

Playing around, it seems to me that some of the radial functions
create smoother approximations with large positive values (with the
current implementation) while others create smoother approximations
with large negative values. It's possible that it's just a convention
as to which sign you use.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Wed Mar  3 16:31:26 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 3 Mar 2010 16:31:26 -0500
Subject: [SciPy-User] scipy.interpolate.rbf sensitive to input noise ?
In-Reply-To: <3d375d731003031315p7bc93005ibd7fed09c2ccbd41@mail.gmail.com>
References: <b3a51763-8674-48d1-b1c2-72938849c6f2@v20g2000yqv.googlegroups.com>
	<1cd32cbb1002220608l3da7f92aue8d415d945d9e33b@mail.gmail.com>
	<43bfdeb3-9d1a-4307-a0ce-96b6639c217e@o3g2000yqb.googlegroups.com>
	<1cd32cbb1002220845o3ad94ba8g599f25ec170c6106@mail.gmail.com>
	<1cd32cbb1002220927h229e578bhe18737d803fddc76@mail.gmail.com>
	<1c841a64-786d-4069-9f88-fcaae5a0d063@t42g2000vbt.googlegroups.com>
	<3d375d731002230946m75c7cacbk4a45e8e3db518d54@mail.gmail.com>
	<1cd32cbb1002230957w35b7119ay7270bbbc1892ce25@mail.gmail.com>
	<1cd32cbb1002230959u46a4da49ie6e7220d5964a381@mail.gmail.com>
	<3d375d731003031315p7bc93005ibd7fed09c2ccbd41@mail.gmail.com>
Message-ID: <1cd32cbb1003031331m7d8142eje2e352a00f1bdfb1@mail.gmail.com>

On Wed, Mar 3, 2010 at 4:15 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Feb 23, 2010 at 11:59, ?<josef.pktd at gmail.com> wrote:
>> On Tue, Feb 23, 2010 at 12:57 PM, ?<josef.pktd at gmail.com> wrote:
>>> On Tue, Feb 23, 2010 at 12:46 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>>> On Tue, Feb 23, 2010 at 11:43, denis <denis-bz-gg at t-online.de> wrote:
>>>>> Robert, Josef,
>>>>> ?thanks much for taking the time to look at RBF some more.
>>>>> Summary, correct me:
>>>>> ? ?A - smooth*I in rbf.py is a sign error (ticket ?)
>>>>
>>>> Not necessarily. It seems to work well in at least some cases. Find a
>>>> reference that says otherwise if you want it changed.
>>>
>>> chapter 2 page 16, for gaussian process. As I said I don't know about
>>> the other methods
>>>
>>
>> http://docs.google.com/viewer?a=v&q=cache:qs8AaAxO6nkJ:www.gaussianprocess.org/gpml/chapters/RW2.pdf+gaussian+process+noise+Ridge&hl=en&gl=ca&pid=bl&srcid=ADGEESj4j8osT6cOIc65r3OaeAtQO_dzgZD4YxSAEkFTeRZajBcROJpJJ9zTlMSrD2OaK1iOJYgy8QqH_Nr0rNxf41faNihCdIzWyVOYxtCFIR7H8mdQZAKFoeaRkFamQlCKhp_s1FOI&sig=AHIEtbQK35MLfnZAySw3lF-dR_mNcSaP3w
>>
>> google links are very short, missed a part
>
> I've found a couple of things on RBFs specifically that agree.
> However, the original source on which our implementation is based does
> subtract. He may have a source that uses a negative sign.
>
> http://www.mathworks.co.uk/matlabcentral/fileexchange/10056-scattered-data-interpolation-and-approximation-using-radial-base-functions
>
> Playing around, it seems to me that some of the radial functions
> create smoother approximations with large positive values (with the
> current implementation) while others create smoother approximations
> with large negative values. It's possible that it's just a convention
> as to which sign you use.

>From the examples I checked, it looks like all other than gaussian
have large positive and negative eigenvalues, so adding or subtracting
doesn't change whether the matrix is definite.

For the gaussian case it is different, with smooth=0 the smallest
eigenvalues are zero and all others are positive. Only by *adding* a
penalization, the matrix becomes positive definite. (I've never seen a
case where eigenvalues are made negative to be able to invert a matrix
in the Ridge regression type of penalization.)

So it might be a convention if the kernel is increasing in distance
but not for the gaussian kernel that is decreasing.

If it's a convention than we could flip the sign and make it
correspond  to the gaussian case, and the other cases would not be
strongly affected.

I will look at the reference. I haven't found much in terms of
references directly for general rbf.

Josef

>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From agile.aspect at gmail.com  Wed Mar  3 17:13:49 2010
From: agile.aspect at gmail.com (Agile Aspect)
Date: Wed, 3 Mar 2010 14:13:49 -0800
Subject: [SciPy-User] scikits.audiolab NameError
In-Reply-To: <web-130976895@uni-stuttgart.de>
References: <web-130973881@uni-stuttgart.de>
	<c6a6f63b1003031001q1abef478w881b9d4a0260c3dd@mail.gmail.com>
	<web-130976895@uni-stuttgart.de>
Message-ID: <c6a6f63b1003031413w3d394c00r54c6d5441e6a1119@mail.gmail.com>

On Wed, Mar 3, 2010 at 11:01 AM, Nils Wagner
<nwagner at iam.uni-stuttgart.de> wrote:
> On Wed, 3 Mar 2010 10:01:22 -0800
> ?Agile Aspect <agile.aspect at gmail.com> wrote:
>> On Wed, Mar 3, 2010 at 9:06 AM, Nils Wagner
>> <nwagner at iam.uni-stuttgart.de> wrote:
>>> Hi all,
>>>
>>> I installed audiolab via
>>>
>>> git clone git://github.com/cournape/audiolab.git
>>> python setup.py install --prefix=$HOME/local
>>>
>>> How can I fix the backend problem wrt. alsa ?
>>>
>>>
>>> python -i test_sndfile.py
>>> /data/home/nwagner/local/lib/python2.5/site-packages/scikits.audiolab-0.11.0.dev-py2.5-linux-x86_64.egg/scikits/audiolab/soundio/play.py:48:
>>> UserWarning: Could not import alsa backend; most
>>>probably,
>>> you did not have alsa headers when building audiolab
>>> ? warnings.warn("Could not import alsa backend; most
>>> probably, "
>>> Traceback (most recent call last):
>>> ? File "test_sndfile.py", line 332, in <module>
>>> ? ? NumpyTest().run()
>>> NameError: name 'NumpyTest' is not defined
>>>
>>>
>>> AFAIK, NumpyTest is obsolete. Am I missing something ?
>>>
>>> Nils
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>
>> On CentOS/Fedora
>>
>> ? ? ? yum install alsa\*
>>
>
> alsa is already installed...
>
> header file are available in /usr/include/alsa
> the library libasound.a is in /usr/lib64
>
> I haven't a clue what is going on here.

If this is a runtime error then the ALSA backend was never built.

Despite the fact that no sound system was found during the build, it
still installs the files and creates the impression everything is
honky dory.

Rebuild it and direct the output to a file - the output will probably
indicate the ALSA backend was never found.

-- 
      Enjoy global warming while it lasts.


From timmichelsen at gmx-topmail.de  Wed Mar  3 17:13:36 2010
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Wed, 03 Mar 2010 23:13:36 +0100
Subject: [SciPy-User] calculating intersection points in probability
	curves
In-Reply-To: <hmk1ik$n2s$1@dough.gmane.org>
References: <hmk1ik$n2s$1@dough.gmane.org>
Message-ID: <hmmmuh$id2$1@dough.gmane.org>

Hello,
my mail seems to have been too theoretical.

Below is some example code.

How can I calculate the intersection point (coordinates) between
'distribution_curve' and 'limit_line'?

Which functions could help me here?

Thanks in advance for any help / pointers.

Best regards,
Timmie

#####  EXAMPLE CODE  #######
#!/usr/bin/env python
import pylab as P
import numpy as np

# first create a single histogram
#
mu, sigma = 200, 25
x = mu + sigma*P.randn(10000)

n, bins = np.histogram(x)

# add a line showing the expected distribution
y = P.normpdf( bins, mu, sigma).cumsum()
y /= y[-1]

# the curve for the plot shwoing the distribution
distribution_curve = P.plot(bins, y, 'k--', linewidth=1.5)
# the limit of which the intersection
# point (x / y) with distribution_curve is desired
limit_line = P.axvline(x.mean())

P.show()



From robert.kern at gmail.com  Wed Mar  3 17:41:37 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 3 Mar 2010 16:41:37 -0600
Subject: [SciPy-User] calculating intersection points in probability
	curves
In-Reply-To: <hmmmuh$id2$1@dough.gmane.org>
References: <hmk1ik$n2s$1@dough.gmane.org> <hmmmuh$id2$1@dough.gmane.org>
Message-ID: <3d375d731003031441t2cf09d99u97235f83f1106463@mail.gmail.com>

On Wed, Mar 3, 2010 at 16:13, Tim Michelsen <timmichelsen at gmx-topmail.de> wrote:
> Hello,
> my mail seems to have been too theoretical.
>
> Below is some example code.
>
> How can I calculate the intersection point (coordinates) between
> 'distribution_curve' and 'limit_line'?
>
> Which functions could help me here?

For finding where a monotonically increasing sequence crosses a
particular value, use np.searchsorted() with the Y coordinate array
and the value. This gives you the index into the Y array. Use that
index on the X values to find the X value where it crosses. Depending
on how you are interpreting those points, you may want that value, or
the next one, or some weighted average of the two.

For two sequences that cross, again it depends a little on the
sequences themselves and on how you interpret the sequences. Are they
step functions, or would you draw line segments from point to point?
Are they monotonic? Do they have the same X values? Do they have the
same set of Y values? etc. You could cobble something together where
you evaluate "h(x) = f(x) - g(x)" (where f(x) and g(x) use interp1d on
the original datasets) on the set of X points that is the union of the
two sequences of X points from the two datasets. Then search for zero
crossings (there may be more than one).

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From marcotuckner at public-files.de  Wed Mar  3 17:47:11 2010
From: marcotuckner at public-files.de (Marco Tuckner)
Date: Wed, 03 Mar 2010 23:47:11 +0100
Subject: [SciPy-User] ValueError with ts.convert of scikits.timeseries
Message-ID: <hmmotg$opd$1@dough.gmane.org>

Hello,
I encountered an error working with a recarray read from a ASCII file
with ts.tsfromtxt [1].

The steps were conducted:

1) read in with ts.tsfromtxt ==> recarray with data with minutely frequency
2) missing dates were filed with ts.fill_missing_dates ==> complete series
3) convert into hourly frequency with ts.convert
4) an error occurs, see below.

What could be the cause for this?

My recarray have the following dtypes:

 dtype = [('f2', '<i4'), ('f3', '<f8'), ('f4', '<f8'), ('f5', '<i4'),
('f6', '<i4'), ('f7', '<f8'), ('f8', '<f8'), ('f9', '<f8'), ('f10',
'<f8'), ('f11', '<f8'), ('f12', '<f8')],

Another question related to this:
How do I convert the recarrays returned by ts.tsfromtxt into a ndarray?

I would like to transform the recarray in a form similar to what is
returned data is loaded with np.loadtxt (and then pushed in a timeseries)?

I would appreciate any help.

Thanks in advance,
Marco

##### Traceback #####

[C:scripts/eval]|103> run dataeval.py
------------------------------------------------------------------------
---
ValueError                                Traceback (most recent call
last)

C:\scripts\eval\dataeval.py in <module>()
     42 myseries_filled_min = ts.fill_missing_dates(series)
     43
---> 44 myseries_hourly_mean =
ts.convert(myseries_filled_min,'H',func=np.mean)

C:\Python26\lib\site-packages\scikits\timeseries\tseries.pyc in
convert(series, freq, func, position, *args, **kwargs)
   2003
   2004     if series.ndim == 1:
-> 2005         obj = _convert1d(series, freq, func, position, *args,
**kwargs)
   2006     elif series.ndim == 2:
   2007         base = _convert1d(series[:,0], freq, func, position,
*args, **kwargs)

C:\Python26\lib\site-packages\scikits\timeseries\tseries.pyc in
_convert1d(series, freq, func, position, *args, **kwargs)
   1913
   1914     cdictresult = cseries.TS_convert(data_, from_freq, to_freq,
position,
-> 1915                                      int(start_date), mask_)
   1916     start_date = Date(freq=to_freq,
value=cdictresult['startindex'])
   1917     data_ = masked_array(cdictresult['values'],
mask=cdictresult['mask'])

ValueError: data type must provide an itemsize
WARNING: Failure executing file: <dataeval.py>


#####

[1]: http://pytseries.sourceforge.net/core.timeseries.io.html



From pitam at deepspace.ucsb.edu  Wed Mar  3 20:17:40 2010
From: pitam at deepspace.ucsb.edu (pitam)
Date: Wed, 03 Mar 2010 17:17:40 -0800
Subject: [SciPy-User]  segfaults with weave.inline
Message-ID: <1267665460.31585.3.camel@zephyr.deepspace.ucsb.edu>

Hi Pauli,

Thanks, that resolved the issue. 


I also want to contribute to make a good documentation for weave. Hence,
I will submit my code and explanations with the difficulties I had to
the present wiki. However, the wiki right now lacks a section for arrays
in weave.inline. Shall we create one?


Pitam




From pgmdevlist at gmail.com  Wed Mar  3 20:23:47 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 3 Mar 2010 20:23:47 -0500
Subject: [SciPy-User] ValueError with ts.convert of scikits.timeseries
In-Reply-To: <hmmotg$opd$1@dough.gmane.org>
References: <hmmotg$opd$1@dough.gmane.org>
Message-ID: <67573205-1CFA-4A09-884D-B3DABF3ABE8A@gmail.com>

On Mar 3, 2010, at 5:47 PM, Marco Tuckner wrote:
> Hello,
> I encountered an error working with a recarray read from a ASCII file
> with ts.tsfromtxt [1].
> 
> The steps were conducted:
> 
> 1) read in with ts.tsfromtxt ==> recarray with data with minutely frequency
> 2) missing dates were filed with ts.fill_missing_dates ==> complete series
> 3) convert into hourly frequency with ts.convert
> 4) an error occurs, see below.
> 
> What could be the cause for this?

The fact that you're using a structured array (that is, an array with named fields. a recarray is a special kind of structured array). The conversion functions don't work in that case, you have to convert field by field.
In your case, I'd do something like this:
* convert the first field to hours using .convert and save the reslut in a temporary array
* create an empty timeseries with the same dtype as your input and the same dates as the temporary array you just created.
* Loop on the fields of the input, convert them to hours and store them in the corresponding field of your new array

> 
> My recarray have the following dtypes:
> 
> dtype = [('f2', '<i4'), ('f3', '<f8'), ('f4', '<f8'), ('f5', '<i4'),
> ('f6', '<i4'), ('f7', '<f8'), ('f8', '<f8'), ('f9', '<f8'), ('f10',
> '<f8'), ('f11', '<f8'), ('f12', '<f8')],
> 
> Another question related to this:
> How do I convert the recarrays returned by ts.tsfromtxt into a ndarray?

Use .view(np.ndarray). Note that on top of dropping the dates, you also lose the mask. If you need the mask, use the .series attribute.



From arokem at berkeley.edu  Wed Mar  3 21:20:32 2010
From: arokem at berkeley.edu (Ariel Rokem)
Date: Wed, 3 Mar 2010 18:20:32 -0800
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <4B854AF8.3060503@enthought.com>
References: <web-130689370@uni-stuttgart.de> <4B854AF8.3060503@enthought.com>
Message-ID: <43958ee61003031820y2c63d1dew56d39bb3f9249fd4@mail.gmail.com>

Hi Warren,

thanks for this example!

I am getting the following error:

Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/signal/filter_design.py:224:
BadCoefficients: Badly conditionned filter coefficients (numerator): the
results may be meaningless
  "results may be meaningless", BadCoefficients)

For any attempt to filter a low-pass below 12 Hz in this example (so - I
don't get the plot you got - instead I get a flat line on the bottom
subplot). Do you (or anyone?) have any idea why that is?

Cheers,

Ariel

On Wed, Feb 24, 2010 at 7:51 AM, Warren Weckesser <
warren.weckesser at enthought.com> wrote:

> Nils Wagner wrote:
> > Hi all,
> >
> > I have two questions concerning signal processing
> >
> > I have used scipy.stats.signaltonoise to compute the
> > signal-to-noise ratio.
> > The value is 0.0447.
> > How can I judge it ?
> >
> > How can I filter out high frequencies using scipy ?
> >
>
> I posted an example low-pass filtering using 'butter' and 'lfilter' from
> scipy.signal here:
>
>    http://mail.scipy.org/pipermail/scipy-user/2010-January/024032.html
>
> Warren
>
> > How can I eliminate noise from the signal ?
> >
> > Nils
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Ariel Rokem
Helen Wills Neuroscience Institute
University of California, Berkeley
http://argentum.ucbso.berkeley.edu/ariel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100303/6e031a00/attachment.html>

From ivo.maljevic at gmail.com  Wed Mar  3 22:17:46 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Wed, 3 Mar 2010 22:17:46 -0500
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <43958ee61003031820y2c63d1dew56d39bb3f9249fd4@mail.gmail.com>
References: <web-130689370@uni-stuttgart.de> <4B854AF8.3060503@enthought.com>
	<43958ee61003031820y2c63d1dew56d39bb3f9249fd4@mail.gmail.com>
Message-ID: <826c64da1003031917j57ae8111j3b2303099087325c@mail.gmail.com>

I just tried Warren's example and it works on Ubuntu.

Ivo

On 3 March 2010 21:20, Ariel Rokem <arokem at berkeley.edu> wrote:

> Hi Warren,
>
> thanks for this example!
>
> I am getting the following error:
>
> Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/signal/filter_design.py:224:
> BadCoefficients: Badly conditionned filter coefficients (numerator): the
> results may be meaningless
>   "results may be meaningless", BadCoefficients)
>
> For any attempt to filter a low-pass below 12 Hz in this example (so - I
> don't get the plot you got - instead I get a flat line on the bottom
> subplot). Do you (or anyone?) have any idea why that is?
>
> Cheers,
>
> Ariel
>
> On Wed, Feb 24, 2010 at 7:51 AM, Warren Weckesser <
> warren.weckesser at enthought.com> wrote:
>
>> Nils Wagner wrote:
>> > Hi all,
>> >
>> > I have two questions concerning signal processing
>> >
>> > I have used scipy.stats.signaltonoise to compute the
>> > signal-to-noise ratio.
>> > The value is 0.0447.
>> > How can I judge it ?
>> >
>> > How can I filter out high frequencies using scipy ?
>> >
>>
>> I posted an example low-pass filtering using 'butter' and 'lfilter' from
>> scipy.signal here:
>>
>>    http://mail.scipy.org/pipermail/scipy-user/2010-January/024032.html
>>
>> Warren
>>
>> > How can I eliminate noise from the signal ?
>> >
>> > Nils
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
>
> --
> Ariel Rokem
> Helen Wills Neuroscience Institute
> University of California, Berkeley
> http://argentum.ucbso.berkeley.edu/ariel
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100303/92f0d6f8/attachment.html>

From D.J.Baker at soton.ac.uk  Thu Mar  4 06:17:54 2010
From: D.J.Baker at soton.ac.uk (Baker D.J.)
Date: Thu, 4 Mar 2010 11:17:54 +0000
Subject: [SciPy-User] Scipy failing -- undefined symbol:
	_ZNSt8ios_base4InitD1Ev
Message-ID: <E117AD245282BA48ABE1E16E9712070C956D5126B8@UOS-CL-EX7-L3.soton.ac.uk>

Hello,

I've just installed scipy on our Linux x86_64 machine using the Intel compilers and MKL. All is not well, however, since when I try to use scipy, for example...

from scipy import linsolve, sparse

I find that there is an unresolved symbol. That is, _ZNSt8ios_base4InitD1Ev. Using c++filt would appear to suggest that I need to add "-lstdc++" to my loader command. 

c++filt _ZNSt8ios_base4InitD1Ev
std::ios_base::Init::~Init()

I'm new to numpy and scipy, and so I'm working in the dark to some degree. I've tried modifying intelccompiler.py in the following way, and recompiling numpy and scipy (the idea being that some of the programs in the scipy build are cxx files)

    compiler_type = 'intel'
    cc_exe = 'icc -O3 -fPIC'
    cxx_exe = 'icpc -O3 -fPIC'
    def __init__ (self, verbose=0, dry_run=0, force=0):
        UnixCCompiler.__init__ (self, verbose,dry_run, force)
        compiler = self.cc_exe
        compiler_cxx = self.cxx_exe
        self.set_executables(compiler=compiler,
                             compiler_so=compiler,
                             compiler_cxx=compiler_cxx,
                             linker_exe=compiler,
                             linker_so=compiler_cxx + ' -shared -lstdc++')

I rebuild scipy using the following command, however the above hasn't helped:

python setup.py config --compiler=intel config_fc --fcompiler=intelem  install --prefix=/local/software/rh53/scipy/0.7.1/intel

Could someone please advise me how to ensure that the "-lstdc++" is successfully passed to the linker as and when I build scipy. This would really be appreciated.

Best -- David.

From warren.weckesser at enthought.com  Thu Mar  4 12:24:19 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Thu, 04 Mar 2010 11:24:19 -0600
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <43958ee61003031820y2c63d1dew56d39bb3f9249fd4@mail.gmail.com>
References: <web-130689370@uni-stuttgart.de> <4B854AF8.3060503@enthought.com>
	<43958ee61003031820y2c63d1dew56d39bb3f9249fd4@mail.gmail.com>
Message-ID: <4B8FECC3.8030607@enthought.com>

Ariel Rokem wrote:
> Hi Warren,
>
> thanks for this example!
>
> I am getting the following error:
>
> Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/signal/filter_design.py:224: 
> BadCoefficients: Badly conditionned filter coefficients (numerator): 
> the results may be meaningless
>   "results may be meaningless", BadCoefficients)
>
> For any attempt to filter a low-pass below 12 Hz in this example (so - 
> I don't get the plot you got - instead I get a flat line on the bottom 
> subplot). Do you (or anyone?) have any idea why that is?
>

Nils Wagner reported the same behavior a week or so ago, and I see the 
same behavior now ("BadCoefficients" and a flat line in the last plot).  
Try lowering the order of the Butterworth filter to order=6.

Warren


> Cheers,
>
> Ariel 
>
> On Wed, Feb 24, 2010 at 7:51 AM, Warren Weckesser 
> <warren.weckesser at enthought.com 
> <mailto:warren.weckesser at enthought.com>> wrote:
>
>     Nils Wagner wrote:
>     > Hi all,
>     >
>     > I have two questions concerning signal processing
>     >
>     > I have used scipy.stats.signaltonoise to compute the
>     > signal-to-noise ratio.
>     > The value is 0.0447.
>     > How can I judge it ?
>     >
>     > How can I filter out high frequencies using scipy ?
>     >
>
>     I posted an example low-pass filtering using 'butter' and
>     'lfilter' from
>     scipy.signal here:
>
>        http://mail.scipy.org/pipermail/scipy-user/2010-January/024032.html
>
>     Warren
>
>     > How can I eliminate noise from the signal ?
>     >
>     > Nils
>     > _______________________________________________
>     > SciPy-User mailing list
>     > SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>     > http://mail.scipy.org/mailman/listinfo/scipy-user
>     >
>
>     _______________________________________________
>     SciPy-User mailing list
>     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
>
> -- 
> Ariel Rokem
> Helen Wills Neuroscience Institute
> University of California, Berkeley
> http://argentum.ucbso.berkeley.edu/ariel
> ------------------------------------------------------------------------
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From ivo.maljevic at gmail.com  Thu Mar  4 12:48:50 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Thu, 4 Mar 2010 12:48:50 -0500
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <4B8FECC3.8030607@enthought.com>
References: <web-130689370@uni-stuttgart.de> <4B854AF8.3060503@enthought.com>
	<43958ee61003031820y2c63d1dew56d39bb3f9249fd4@mail.gmail.com>
	<4B8FECC3.8030607@enthought.com>
Message-ID: <826c64da1003040948x45e36a6l682157ca206ad61a@mail.gmail.com>

If your signal is sampled at 200 Hz (like in this example), and you want to
filter out everything above 4 Hz or 12 Hz, you might want to consider using
a wider LPF first, downsapling the signal, and then doing the filtering.
Usually, sampling rate is related to the frequency content of the signal
(Nyquist), and since your signal of interest is much narrower, you have very
high oversampling (maybe due to very wide noise?).

In any case, even Matlab wouldn't help you much if you wanted to implement
such a narrow LPF:

Python:

>>> b, a = butter(14, ws3, 'low')
C:\Python26\lib\site-packages\scipy\signal\filter_design.py:221:
BadCoefficients: Badly conditionned filter coefficients (numerator): the
results may be meaningless
  "results may be meaningless", BadCoefficients)
C:\Python26\lib\site-packages\scipy\signal\filter_design.py:221:
BadCoefficients: Badly conditionned filter coefficients (numerator): the
results may be meaningless
  "results may be meaningless", BadCoefficients)
>>> ws3
0.040000000000000001
>>> b
array([  1.21542041e-16,   7.90023266e-16,   3.16009306e-15,
         8.69025592e-15,   1.73805118e-14,   2.60707678e-14,
         2.97951632e-14,   2.60707678e-14,   1.73805118e-14,
         8.69025592e-15,   3.16009306e-15,   7.90023266e-16,
         1.21542041e-16,   8.68157435e-18])
>>> a
array([  1.00000000e+00,  -1.28776717e+01,   7.70365408e+01,
        -2.83749831e+02,   7.18912579e+02,  -1.32537586e+03,
         1.83351674e+03,  -1.93352359e+03,   1.56186993e+03,
        -9.61728148e+02,   4.44354147e+02,  -1.49385526e+02,
         3.45431991e+01,  -4.91770336e+00,   3.25194854e-01])

Matlab:

>> [b,a]=butter(14, 0.04, 'low')
b =
  Columns 1 through 12
     0     0     0     0     0     0     0     0     0     0     0     0
  Columns 13 through 15
     0     0     0

a =
  1.0e+003 *
  Columns 1 through 7
    0.0010   -0.0129    0.0770   -0.2837    0.7189   -1.3254    1.8335
  Columns 8 through 14
   -1.9335    1.5619   -0.9617    0.4444   -0.1494    0.0345   -0.0049
  Column 15
    0.0003

On 4 March 2010 12:24, Warren Weckesser <warren.weckesser at enthought.com>wrote:

> Ariel Rokem wrote:
> > Hi Warren,
> >
> > thanks for this example!
> >
> > I am getting the following error:
> >
> >
> Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/signal/filter_design.py:224:
> > BadCoefficients: Badly conditionned filter coefficients (numerator):
> > the results may be meaningless
> >   "results may be meaningless", BadCoefficients)
> >
> > For any attempt to filter a low-pass below 12 Hz in this example (so -
> > I don't get the plot you got - instead I get a flat line on the bottom
> > subplot). Do you (or anyone?) have any idea why that is?
> >
>
> Nils Wagner reported the same behavior a week or so ago, and I see the
> same behavior now ("BadCoefficients" and a flat line in the last plot).
> Try lowering the order of the Butterworth filter to order=6.
>
> Warren
>
>
> > Cheers,
> >
> > Ariel
> >
> > On Wed, Feb 24, 2010 at 7:51 AM, Warren Weckesser
> > <warren.weckesser at enthought.com
> > <mailto:warren.weckesser at enthought.com>> wrote:
> >
> >     Nils Wagner wrote:
> >     > Hi all,
> >     >
> >     > I have two questions concerning signal processing
> >     >
> >     > I have used scipy.stats.signaltonoise to compute the
> >     > signal-to-noise ratio.
> >     > The value is 0.0447.
> >     > How can I judge it ?
> >     >
> >     > How can I filter out high frequencies using scipy ?
> >     >
> >
> >     I posted an example low-pass filtering using 'butter' and
> >     'lfilter' from
> >     scipy.signal here:
> >
> >
> http://mail.scipy.org/pipermail/scipy-user/2010-January/024032.html
> >
> >     Warren
> >
> >     > How can I eliminate noise from the signal ?
> >     >
> >     > Nils
> >     > _______________________________________________
> >     > SciPy-User mailing list
> >     > SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
> >     > http://mail.scipy.org/mailman/listinfo/scipy-user
> >     >
> >
> >     _______________________________________________
> >     SciPy-User mailing list
> >     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
> >     http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> >
> >
> > --
> > Ariel Rokem
> > Helen Wills Neuroscience Institute
> > University of California, Berkeley
> > http://argentum.ucbso.berkeley.edu/ariel
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100304/bdbe2033/attachment.html>

From arokem at berkeley.edu  Thu Mar  4 13:47:33 2010
From: arokem at berkeley.edu (Ariel Rokem)
Date: Thu, 4 Mar 2010 10:47:33 -0800
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <4B8FECC3.8030607@enthought.com>
References: <web-130689370@uni-stuttgart.de> <4B854AF8.3060503@enthought.com> 
	<43958ee61003031820y2c63d1dew56d39bb3f9249fd4@mail.gmail.com> 
	<4B8FECC3.8030607@enthought.com>
Message-ID: <43958ee61003041047s2b4d27cahaed871674e8d4aab@mail.gmail.com>

Interesting. It does work with lower filter order. In fact, there is a range
of values of filter order for which I get the warning, but also some result.
This results looks slightly flatter than the result with filter order set to
6, but is not completely flat. What does the filter order do, intuitively?

Thanks - Ariel

On Thu, Mar 4, 2010 at 9:24 AM, Warren Weckesser <
warren.weckesser at enthought.com> wrote:

> Ariel Rokem wrote:
> > Hi Warren,
> >
> > thanks for this example!
> >
> > I am getting the following error:
> >
> >
> Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/signal/filter_design.py:224:
> > BadCoefficients: Badly conditionned filter coefficients (numerator):
> > the results may be meaningless
> >   "results may be meaningless", BadCoefficients)
> >
> > For any attempt to filter a low-pass below 12 Hz in this example (so -
> > I don't get the plot you got - instead I get a flat line on the bottom
> > subplot). Do you (or anyone?) have any idea why that is?
> >
>
> Nils Wagner reported the same behavior a week or so ago, and I see the
> same behavior now ("BadCoefficients" and a flat line in the last plot).
> Try lowering the order of the Butterworth filter to order=6.
>
> Warren
>
>
> > Cheers,
> >
> > Ariel
> >
> > On Wed, Feb 24, 2010 at 7:51 AM, Warren Weckesser
> > <warren.weckesser at enthought.com
> > <mailto:warren.weckesser at enthought.com>> wrote:
> >
> >     Nils Wagner wrote:
> >     > Hi all,
> >     >
> >     > I have two questions concerning signal processing
> >     >
> >     > I have used scipy.stats.signaltonoise to compute the
> >     > signal-to-noise ratio.
> >     > The value is 0.0447.
> >     > How can I judge it ?
> >     >
> >     > How can I filter out high frequencies using scipy ?
> >     >
> >
> >     I posted an example low-pass filtering using 'butter' and
> >     'lfilter' from
> >     scipy.signal here:
> >
> >
> http://mail.scipy.org/pipermail/scipy-user/2010-January/024032.html
> >
> >     Warren
> >
> >     > How can I eliminate noise from the signal ?
> >     >
> >     > Nils
> >     > _______________________________________________
> >     > SciPy-User mailing list
> >     > SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
> >     > http://mail.scipy.org/mailman/listinfo/scipy-user
> >     >
> >
> >     _______________________________________________
> >     SciPy-User mailing list
> >     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
> >     http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> >
> >
> > --
> > Ariel Rokem
> > Helen Wills Neuroscience Institute
> > University of California, Berkeley
> > http://argentum.ucbso.berkeley.edu/ariel
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Ariel Rokem
Helen Wills Neuroscience Institute
University of California, Berkeley
http://argentum.ucbso.berkeley.edu/ariel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100304/be68ead3/attachment.html>

From ivo.maljevic at gmail.com  Thu Mar  4 14:20:20 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Thu, 4 Mar 2010 14:20:20 -0500
Subject: [SciPy-User] Signal to noise ratio
In-Reply-To: <43958ee61003041047s2b4d27cahaed871674e8d4aab@mail.gmail.com>
References: <web-130689370@uni-stuttgart.de> <4B854AF8.3060503@enthought.com>
	<43958ee61003031820y2c63d1dew56d39bb3f9249fd4@mail.gmail.com>
	<4B8FECC3.8030607@enthought.com>
	<43958ee61003041047s2b4d27cahaed871674e8d4aab@mail.gmail.com>
Message-ID: <826c64da1003041120i7813e2afrc6ea11148aaa83f@mail.gmail.com>

Arieal,
I just did a quick search and found the same thing I told you on Matlab site
(i.e., narrowband filter):

http://www.mathworks.com/access/helpdesk/help/toolbox/signal/f4-1046.html

While it may be interesting to play with cuttoff frequencies and filter
order, it is quite useless from a practical point of view. To use a tool,
you need to understand what it does and what are its limitations. What you
are trying to do with the filter is almost the same as if you are trying to
find the mean value of the signal (hence the almost flat line). You
definitely do not need butterworth filter for that, just find the mean value
of the signal for such type of operation.

As for your order question, it basically determines how many past samples
are you using in the filtering operation. Maybe you can read this:

http://en.wikipedia.org/wiki/Digital_filter#Difference_equation

Cheers,
Ivo



On 4 March 2010 13:47, Ariel Rokem <arokem at berkeley.edu> wrote:

> Interesting. It does work with lower filter order. In fact, there is a
> range of values of filter order for which I get the warning, but also some
> result. This results looks slightly flatter than the result with filter
> order set to 6, but is not completely flat. What does the filter order do,
> intuitively?
>
> Thanks - Ariel
>
> On Thu, Mar 4, 2010 at 9:24 AM, Warren Weckesser <
> warren.weckesser at enthought.com> wrote:
>
>> Ariel Rokem wrote:
>> > Hi Warren,
>> >
>> > thanks for this example!
>> >
>> > I am getting the following error:
>> >
>> >
>> Library/Frameworks/Python.framework/Versions/6.0.0/lib/python2.6/site-packages/scipy/signal/filter_design.py:224:
>> > BadCoefficients: Badly conditionned filter coefficients (numerator):
>> > the results may be meaningless
>> >   "results may be meaningless", BadCoefficients)
>> >
>> > For any attempt to filter a low-pass below 12 Hz in this example (so -
>> > I don't get the plot you got - instead I get a flat line on the bottom
>> > subplot). Do you (or anyone?) have any idea why that is?
>> >
>>
>> Nils Wagner reported the same behavior a week or so ago, and I see the
>> same behavior now ("BadCoefficients" and a flat line in the last plot).
>> Try lowering the order of the Butterworth filter to order=6.
>>
>> Warren
>>
>>
>> > Cheers,
>> >
>> > Ariel
>> >
>> > On Wed, Feb 24, 2010 at 7:51 AM, Warren Weckesser
>> > <warren.weckesser at enthought.com
>> > <mailto:warren.weckesser at enthought.com>> wrote:
>> >
>> >     Nils Wagner wrote:
>> >     > Hi all,
>> >     >
>> >     > I have two questions concerning signal processing
>> >     >
>> >     > I have used scipy.stats.signaltonoise to compute the
>> >     > signal-to-noise ratio.
>> >     > The value is 0.0447.
>> >     > How can I judge it ?
>> >     >
>> >     > How can I filter out high frequencies using scipy ?
>> >     >
>> >
>> >     I posted an example low-pass filtering using 'butter' and
>> >     'lfilter' from
>> >     scipy.signal here:
>> >
>> >
>> http://mail.scipy.org/pipermail/scipy-user/2010-January/024032.html
>> >
>> >     Warren
>> >
>> >     > How can I eliminate noise from the signal ?
>> >     >
>> >     > Nils
>> >     > _______________________________________________
>> >     > SciPy-User mailing list
>> >     > SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>> >     > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >     >
>> >
>> >     _______________________________________________
>> >     SciPy-User mailing list
>> >     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>> >     http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> >
>> >
>> >
>> > --
>> > Ariel Rokem
>> > Helen Wills Neuroscience Institute
>> > University of California, Berkeley
>> > http://argentum.ucbso.berkeley.edu/ariel
>> > ------------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
>
> --
> Ariel Rokem
> Helen Wills Neuroscience Institute
> University of California, Berkeley
> http://argentum.ucbso.berkeley.edu/ariel
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100304/e6ad6aca/attachment.html>

From nwagner at iam.uni-stuttgart.de  Thu Mar  4 14:29:10 2010
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Thu, 04 Mar 2010 20:29:10 +0100
Subject: [SciPy-User] scikits.audiolab NameError
In-Reply-To: <c6a6f63b1003031413w3d394c00r54c6d5441e6a1119@mail.gmail.com>
References: <web-130973881@uni-stuttgart.de>
	<c6a6f63b1003031001q1abef478w881b9d4a0260c3dd@mail.gmail.com>
	<web-130976895@uni-stuttgart.de>
	<c6a6f63b1003031413w3d394c00r54c6d5441e6a1119@mail.gmail.com>
Message-ID: <web-131023353@uni-stuttgart.de>

On Wed, 3 Mar 2010 14:13:49 -0800
  Agile Aspect <agile.aspect at gmail.com> wrote:
> On Wed, Mar 3, 2010 at 11:01 AM, Nils Wagner
> <nwagner at iam.uni-stuttgart.de> wrote:
>> On Wed, 3 Mar 2010 10:01:22 -0800
>> ?Agile Aspect <agile.aspect at gmail.com> wrote:
>>> On Wed, Mar 3, 2010 at 9:06 AM, Nils Wagner
>>> <nwagner at iam.uni-stuttgart.de> wrote:
>>>> Hi all,
>>>>
>>>> I installed audiolab via
>>>>
>>>> git clone git://github.com/cournape/audiolab.git
>>>> python setup.py install --prefix=$HOME/local
>>>>
>>>> How can I fix the backend problem wrt. alsa ?
>>>>
>>>>
>>>> python -i test_sndfile.py
>>>> /data/home/nwagner/local/lib/python2.5/site-packages/scikits.audiolab-0.11.0.dev-py2.5-linux-x86_64.egg/scikits/audiolab/soundio/play.py:48:
>>>> UserWarning: Could not import alsa backend; most
>>>>probably,
>>>> you did not have alsa headers when building audiolab
>>>> ? warnings.warn("Could not import alsa backend; most
>>>> probably, "
>>>> Traceback (most recent call last):
>>>> ? File "test_sndfile.py", line 332, in <module>
>>>> ? ? NumpyTest().run()
>>>> NameError: name 'NumpyTest' is not defined
>>>>
>>>>
>>>> AFAIK, NumpyTest is obsolete. Am I missing something ?
>>>>
>>>> Nils
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>
>>> On CentOS/Fedora
>>>
>>> ? ? ? yum install alsa\*
>>>
>>
>> alsa is already installed...
>>
>> header file are available in /usr/include/alsa
>> the library libasound.a is in /usr/lib64
>>
>> I haven't a clue what is going on here.
> 
> If this is a runtime error then the ALSA backend was 
>never built.
> 
> Despite the fact that no sound system was found during 
>the build, it
> still installs the files and creates the impression 
>everything is
> honky dory.
> 
> Rebuild it and direct the output to a file - the output 
>will probably
> indicate the ALSA backend was never found.
> 
> -- 
>      Enjoy global warming while it lasts.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

Hi all,
  
I just managed the installation of audiolab on 
opensuse11.2

I have installed libsndfile from source. It doesn't work 
with the rpm's provided by opensuse.
Furthermore I have used the following site.cfg

[sndfile]
library_dirs = /home/nwagner/local/lib
include_dirs = /home/nwagner/local/include
sndfile_libs = sndfile
[alsa]
library_dirs = /usr/lib64
include_dirs = /usr/include

The [alsa] section is necessary to detect libasound.a

Now it works for me.

Cheers

                          Nils


From craig.howard at earthlink.net  Thu Mar  4 16:17:34 2010
From: craig.howard at earthlink.net (Craig)
Date: Thu, 4 Mar 2010 13:17:34 -0800 (PST)
Subject: [SciPy-User] Using Pyinstaller or cx_freeze with scipy
Message-ID: <e9f1f1b4-0842-45d9-9374-ff09ec7abea5@a18g2000yqc.googlegroups.com>

Hello:

I was attempting to use cx_freeze to create a stand-alone application
that uses scipy, but ran into a problem. When I attempt to run the
frozen application, I get the error:
scipy\interpolate\polyint.py
Cannot import name factorial

I looked at the scipy package and the factorial function is in
scipy.misc.common.py but the file scipy.interpolate.polyint.py has the
line:
from scipy import factorial

So the polyint script is looking for the factorial function in some
other place than its actual physical location. I guess scipy somehow
inserts the factorial function in the correct namespace so that it
runs correctly. Is there a way to get scipy and cx_freeze to get
along? Thanks for your help in advance.

Regards,
Craig Howard


From kwmsmith at gmail.com  Thu Mar  4 16:47:03 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 4 Mar 2010 15:47:03 -0600
Subject: [SciPy-User] interpolate.bispl{rep, ev} need some love,
	or I'm using them 	incorrectly...
Message-ID: <c266a7931003041347h7324ad81lfd7ec232e0c9b26@mail.gmail.com>

I've been trying to get interpolation working on a not-too-large array
(512 X 512) using interpolate.interp2d, which is a nice wrapper for
interpolate.bispl{rep,ev}.  Apparently these routines are meant to
work on smallish arrays (~ 50 X 50) and give me a segfault when I hand
them anything about (200 X 200) or so.  When they do work, they're
prohibitively slow.

My question is:  what is the usecase for these bivariate spline
routines?  I've discovered ndimage.spline_filter and
ndimage.map_coordinates -- these fit my purposes much better; fast and
nice interface.  What is the difference between the two approaches?

Kurt


From josef.pktd at gmail.com  Thu Mar  4 17:10:53 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 4 Mar 2010 17:10:53 -0500
Subject: [SciPy-User] interpolate.bispl{rep, ev} need some love,
	or I'm 	using them incorrectly...
In-Reply-To: <c266a7931003041347h7324ad81lfd7ec232e0c9b26@mail.gmail.com>
References: <c266a7931003041347h7324ad81lfd7ec232e0c9b26@mail.gmail.com>
Message-ID: <1cd32cbb1003041410y55f8dca3p7522eb9fddfae4cc@mail.gmail.com>

On Thu, Mar 4, 2010 at 4:47 PM, Kurt Smith <kwmsmith at gmail.com> wrote:
> I've been trying to get interpolation working on a not-too-large array
> (512 X 512) using interpolate.interp2d, which is a nice wrapper for
> interpolate.bispl{rep,ev}. ?Apparently these routines are meant to
> work on smallish arrays (~ 50 X 50) and give me a segfault when I hand
> them anything about (200 X 200) or so. ?When they do work, they're
> prohibitively slow.
>
> My question is: ?what is the usecase for these bivariate spline
> routines? ?I've discovered ndimage.spline_filter and
> ndimage.map_coordinates -- these fit my purposes much better; fast and
> nice interface. ?What is the difference between the two approaches?

>From the docs, I think, bispl is a true 2dim spline, while
ndimage.spline_filter is a sequence of 1d filter applied along each
axis.

For function approximation  200*200=40000  is a large number of points
For  z = f(x,y) I wouldn't usually get as many observations as in an
image, and the number of knot points might be required to be smallish.

I don't know anything about the c or fortran internals of either of them.

Josef

>
> Kurt
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From massimodisasha at yahoo.it  Thu Mar  4 17:18:00 2010
From: massimodisasha at yahoo.it (Massimo Di Stefano)
Date: Thu, 4 Mar 2010 23:18:00 +0100
Subject: [SciPy-User] helmert implementation (7 parameters - geometric
	transformation)
In-Reply-To: <mailman.2912.1267578161.2398.scipy-user@scipy.org>
References: <mailman.2912.1267578161.2398.scipy-user@scipy.org>
Message-ID: <C32AB4B1-ABCC-4757-8256-64F1E540E70A@yahoo.it>

Hi Charles,

apologize me, i had only now a chance to answer your mail.

i'm really gratefull for your help,
now the code works perfectly and i'm really really happy for this !

i tried it with some points and all works fine :-)

reading it i also learned some functions like : 

np.newaxis
linalg.lstsq

you gived me an awesome help, thanks!!!

i've a question about the code,
what does :

#
res = rhs - np.dot(A, sol)
#



in the code i added the equation to apply the transformation to an input file with coordinates to be converted :

XYZ = np.loadtxt(str(inputf))
XYZsize = XYZ.shape[0]
ENZglob = np.zeros((XYZsize,3),float)
for i in np.arange(XYZsize):
   X = sol[0] + (1 + sol[6]) * ( XYZ[i,0] - sol[5] * XYZ[i,1]  + sol[4] * XYZ[i,2] ) 
   Y = sol[1] + (1 + sol[6]) * ( sol[5] * XYZ[i,0] + XYZ[i,1] - sol[3] * XYZ[i,2] ) 
   Z = sol[2] + (1 + sol[6]) * ( -sol[4] * XYZ[i,0] + sol[3] * XYZ[i,1] + XYZ[i,2] ) 
   ENZglob[i,:] = np.hstack((X,Y,Z))
np.savetxt(output,ENZglob)



thanks again!!!

regards,

Massimo.

Il giorno 03/mar/2010, alle ore 02.02, scipy-user-request at scipy.org ha scritto:

> I've attached a script for the computation. Two things of note: the angles
> in the pdf results are in degrees, the value of ? is pretty much in the
> noise (the condition number of the matrix is 10^10), and the fit you get
> with my version differs from the data by about 7mm. Note that the Helmert
> parameters result from a linearization of the rotation matrix, see
> Gauss-Newton for more about linearization and least squares, but in this
> case I think the limiting factor is the accuracy of the coordinates, not the
> linearization.



From kwmsmith at gmail.com  Thu Mar  4 17:30:11 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 4 Mar 2010 16:30:11 -0600
Subject: [SciPy-User] interpolate.bispl{rep, ev} need some love,
	or I'm 	using them incorrectly...
In-Reply-To: <1cd32cbb1003041410y55f8dca3p7522eb9fddfae4cc@mail.gmail.com>
References: <c266a7931003041347h7324ad81lfd7ec232e0c9b26@mail.gmail.com>
	<1cd32cbb1003041410y55f8dca3p7522eb9fddfae4cc@mail.gmail.com>
Message-ID: <c266a7931003041430l3ebcd8f8rb2a12166b319c869@mail.gmail.com>

On Thu, Mar 4, 2010 at 4:10 PM,  <josef.pktd at gmail.com> wrote:
> On Thu, Mar 4, 2010 at 4:47 PM, Kurt Smith <kwmsmith at gmail.com> wrote:
>> I've been trying to get interpolation working on a not-too-large array
>> (512 X 512) using interpolate.interp2d, which is a nice wrapper for
>> interpolate.bispl{rep,ev}. ?Apparently these routines are meant to
>> work on smallish arrays (~ 50 X 50) and give me a segfault when I hand
>> them anything about (200 X 200) or so. ?When they do work, they're
>> prohibitively slow.
>>
>> My question is: ?what is the usecase for these bivariate spline
>> routines? ?I've discovered ndimage.spline_filter and
>> ndimage.map_coordinates -- these fit my purposes much better; fast and
>> nice interface. ?What is the difference between the two approaches?
>
> >From the docs, I think, bispl is a true 2dim spline, while
> ndimage.spline_filter is a sequence of 1d filter applied along each
> axis.
>
> For function approximation ?200*200=40000 ?is a large number of points
> For ?z = f(x,y) I wouldn't usually get as many observations as in an
> image, and the number of knot points might be required to be smallish.
>
> I don't know anything about the c or fortran internals of either of them.

Thanks.

It looks like interpolate.bisplrep is suitable for irregularly spaced
data, whereas ndimage.spline_filter & map_coordinates require gridded
data (naturally, since they're used on image-like data).


>
> Josef
>


From robfalck at gmail.com  Thu Mar  4 20:55:56 2010
From: robfalck at gmail.com (Rob Falck)
Date: Thu, 4 Mar 2010 20:55:56 -0500
Subject: [SciPy-User] Lagrange Multipliers in optimize.slsqp
In-Reply-To: <a193bbb51003021114l709bbbc9oca3d4941409acd72@mail.gmail.com>
References: <a193bbb51002251717j3ab63e73heebcf4f2e229a73e@mail.gmail.com>
	<c048da1c1002252008i124597dbl476c6e6dc95e96b0@mail.gmail.com>
	<1cd32cbb1002260555k3b72a50bsc2e606c9f0e68265@mail.gmail.com>
	<c048da1c1002260735p44263cc3gc394b403f8f2100@mail.gmail.com>
	<a193bbb51003021114l709bbbc9oca3d4941409acd72@mail.gmail.com>
Message-ID: <ccb1b5851003041755p5c7d1a29qe56081b3bd7ccd74@mail.gmail.com>

Any opinions on how this change should be implemented?  I'd prefer to
keep the code clean, and the solution below is simple, but I'd rather
not break current implementations of SLSQP that people have employed,
as inserting a new element into the output likely would.  I'm leaning
towards allowing both boolean or integer types for the full_output
argument.  0 or 1 would work as True and False do now, and 2 would
return the Lagrange multipliers as well as everything else.

On Tue, Mar 2, 2010 at 2:14 PM, Guilherme P. de Freitas
<guilherme at gpfreitas.com> wrote:
> Hi everyone,
>
>> Ah, this should be enough then to return the multipliers. ?See if
>> it gives what you'd expect? ?Then you can file an enhancement ticket
>> if you want the multipliers back.
>>
>> Index: slsqp.py
>> ===================================================================
>> --- slsqp.py ? ?(revision 6242)
>> +++ slsqp.py ? ?(working copy)
>> @@ -371,5 +371,6 @@
>> ? ? ? ? return [list(x),
>> ? ? ? ? ? ? ? ? float(fx),
>> ? ? ? ? ? ? ? ? int(majiter),
>> - ? ? ? ? ? ? ? ?int(mode),
>> + ? ? ? ? ? ? ? ?list(w[:m]),
>> + ? ? ? ? ? ? ? ?int(mode),
>> ? ? ? ? ? ? ? ? exit_modes[int(mode)] ]
>>
>> Skipper
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
> Thanks! That seems to give the right multipliers back. I'll file the ticket.
>
> Best,
>
> Guilherme
>
>
> --
> Guilherme P. de Freitas
> http://www.gpfreitas.com
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
- Rob Falck


From charlesr.harris at gmail.com  Thu Mar  4 21:54:10 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 4 Mar 2010 19:54:10 -0700
Subject: [SciPy-User] helmert implementation (7 parameters - geometric
	transformation)
In-Reply-To: <C32AB4B1-ABCC-4757-8256-64F1E540E70A@yahoo.it>
References: <mailman.2912.1267578161.2398.scipy-user@scipy.org>
	<C32AB4B1-ABCC-4757-8256-64F1E540E70A@yahoo.it>
Message-ID: <e06186141003041854t1a3035cbg45cb08ca004ccd0e@mail.gmail.com>

On Thu, Mar 4, 2010 at 3:18 PM, Massimo Di Stefano
<massimodisasha at yahoo.it>wrote:

> Hi Charles,
>
> apologize me, i had only now a chance to answer your mail.
>
> i'm really gratefull for your help,
> now the code works perfectly and i'm really really happy for this !
>
> i tried it with some points and all works fine :-)
>
> reading it i also learned some functions like :
>
> np.newaxis
> linalg.lstsq
>
> you gived me an awesome help, thanks!!!
>
>
You're welcome ;) Note that it is easy to extend to more than three vectors.


> i've a question about the code,
> what does :
>
> #
> res = rhs - np.dot(A, sol)
> #
>
>
It's the residuals, i.e., the error remaining after the fit. I find the
residuals generally more informative than their squared norm that is
returned by the lstsq function.


>
> in the code i added the equation to apply the transformation to an input
> file with coordinates to be converted :
>
> XYZ = np.loadtxt(str(inputf))
> XYZsize = XYZ.shape[0]
> ENZglob = np.zeros((XYZsize,3),float)
> for i in np.arange(XYZsize):
>   X = sol[0] + (1 + sol[6]) * ( XYZ[i,0] - sol[5] * XYZ[i,1]  + sol[4] *
> XYZ[i,2] )
>   Y = sol[1] + (1 + sol[6]) * ( sol[5] * XYZ[i,0] + XYZ[i,1] - sol[3] *
> XYZ[i,2] )
>   Z = sol[2] + (1 + sol[6]) * ( -sol[4] * XYZ[i,0] + sol[3] * XYZ[i,1] +
> XYZ[i,2] )
>   ENZglob[i,:] = np.hstack((X,Y,Z))
> np.savetxt(output,ENZglob)
>
>
>
> thanks again!!!
>
> regards,
>
> Massimo.
>
>
<snip>

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100304/5b21437e/attachment.html>

From steveredmond at sympatico.ca  Fri Mar  5 08:31:26 2010
From: steveredmond at sympatico.ca (SteveRedmond)
Date: Fri, 5 Mar 2010 05:31:26 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Audiolab on Py2.6
Message-ID: <27788613.post@talk.nabble.com>



ninjasmith wrote:
> 
> 
>> I made some tests and I am almost sure the problem is with this file:
>> "C:\Python26\Lib\site-packages\scikits\audiolab\pysndfile\_sndfile.pyd".
>>
>> But I don?t know how to see its contents or fix the problem.
>>
>> Any more tips? (Please!)
>> --
> 
> I have this problem as well, as far as I am aware its not fixed and is a
> problem linking to the sndfile.dll.
> 


I just want to echo that I have this problem too.  

I am using WinXP Home SP3 and a new full installation of Python(x,y)-2.6.2. 
The simplest attempt to import from _sndfile.pyd is causing audiolab to
fail, e.g., a one line script, "from scikits.audiolab import Sndfile",
results in the following error:

Traceback (most recent call last):
  File
"C:\Python26\lib\site-packages\scikits\audiolab\pysndfile\__init__.py", line
1, in <module>
    from _sndfile import Sndfile, Format, available_file_formats,
available_encodings
ImportError: DLL load failed: The specified procedure could not be found.

I hope this is only a small and familiar glitch, and that David Cournapeau
can get a round to it soon.  Audiolab looked promising until this issue
stopped me.

SteveR
-- 
View this message in context: http://old.nabble.com/Audiolab-on-Py2.6-tp26064218p27788613.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From josef.pktd at gmail.com  Fri Mar  5 09:40:22 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 5 Mar 2010 09:40:22 -0500
Subject: [SciPy-User] changing signs
Message-ID: <1cd32cbb1003050640s2b104542qaefb2c014d07dd28@mail.gmail.com>

http://boscoh.com/protein/a-sign-a-flipped-structure-and-a-scientific-flameout-of-epic-proportions

(link found at http://python.genedrift.org/2010/03/03/python-testing-beginner%E2%80%99s-guide-review/
)

Just a reminder about test coverage

Josef


From jsalvati at u.washington.edu  Fri Mar  5 14:19:51 2010
From: jsalvati at u.washington.edu (John Salvatier)
Date: Fri, 5 Mar 2010 11:19:51 -0800
Subject: [SciPy-User] Easy to make ufuncs from scalar Fortran function?
Message-ID: <113e17f21003051119s40e29ae3l8694cc42f0481a41@mail.gmail.com>

Hello,

Can anyone tell me if it is very difficult to make a numpy ufunc given a
scalar function written in Fortran? I have read part of
http://docs.scipy.org/doc/numpy/user/c-info.beyond-basics.html on creating
ufuncs, is there anything else I should read?

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100305/c237988d/attachment.html>

From johan.gronqvist at gmail.com  Fri Mar  5 16:02:28 2010
From: johan.gronqvist at gmail.com (=?ISO-8859-1?Q?Johan_Gr=F6nqvist?=)
Date: Fri, 05 Mar 2010 22:02:28 +0100
Subject: [SciPy-User] Easy to make ufuncs from scalar Fortran function?
In-Reply-To: <113e17f21003051119s40e29ae3l8694cc42f0481a41@mail.gmail.com>
References: <113e17f21003051119s40e29ae3l8694cc42f0481a41@mail.gmail.com>
Message-ID: <hmrrh0$ilt$1@dough.gmane.org>

John Salvatier skrev:
> Can anyone tell me if it is very difficult to make a numpy ufunc given a 
> scalar function written in Fortran? I have read part of 
> http://docs.scipy.org/doc/numpy/user/c-info.beyond-basics.html on 
> creating ufuncs, is there anything else I should read?

I have never created ufuncs, but as the docs you point to seem very 
c-oriented, I just wanted to make sure you know that calling fortran 
from c is not too complicated (at least with gnu compilers).

If I would try, I would first wrap the fortran code in c, and then look 
at how to create ufuncs from c-code, but I have no idea how to do the 
second part.

A page that seems rather informative and that was suggested by a google 
search on mixing c and fortran is 
<http://www.cae.tntech.edu/help/programming/mixed_languages>.


I hope it helps a bit, although it does not answer your question.

/ johan



From vincent at vincentdavis.net  Sat Mar  6 11:44:38 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 6 Mar 2010 09:44:38 -0700
Subject: [SciPy-User] structured array question? (slow learner :)
Message-ID: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com>

Ok not sure what I am doing wrong here, In fact I am sure this was working
and now I updated python and it doesn't now. I am skeptical that that is
true. My question is what is happening to the strings?

>>> import numpy as np
>>> a = [(1,1),(1,2),(1,3),(2,4),(2,5),(2,6)]
>>> b = [(1,1,'apple', 'pie'),(2,5,'boys','play')]
>>> aa = np.array(a, [('x',int),('y',int)])
>>> bb = np.array(b, [('x',int),('y',int), ('xb',str), ('yb',str)])
>>> bb
array([(1, 1, '', ''), (2, 5, '', '')],
      dtype=[('x', '<i8'), ('y', '<i8'), ('xb', '|S0'), ('yb', '|S0')])
>>>




  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100306/e185e8d1/attachment.html>

From Chris.Barker at noaa.gov  Sat Mar  6 12:25:14 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Sat, 06 Mar 2010 09:25:14 -0800
Subject: [SciPy-User] structured array question? (slow learner :)
In-Reply-To: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com>
References: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com>
Message-ID: <4B928FFA.8040605@noaa.gov>

Vincent Davis wrote:
> Ok not sure what I am doing wrong here, In fact I am sure this was 
> working and now I updated python and it doesn't now.

I'm not sure if anything has changed, but:

> I am skeptical that 
> that is true. My question is what is happening to the strings?

strings in a structured array need to have a length specified. They 
default to 0:
>       dtype=[('x', '<i8'), ('y', '<i8'), ('xb', '|S0'), ('yb', '|S0')])

this works:
In [9]: bb = np.array(b, dtype=[('x', np.int),('y', np.int), 
('xb','S10'), ('yb', 'S10')])

In [10]: bb
Out[10]:
array([(1, 1, 'apple', 'pie'), (2, 5, 'boys', 'play')],
       dtype=[('x', '<i4'), ('y', '<i4'), ('xb', '|S10'), ('yb', '|S10')])

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From rmay31 at gmail.com  Sat Mar  6 12:26:20 2010
From: rmay31 at gmail.com (Ryan May)
Date: Sat, 6 Mar 2010 11:26:20 -0600
Subject: [SciPy-User] structured array question? (slow learner :)
In-Reply-To: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com>
References: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com>
Message-ID: <c7b95a021003060926l105fc892v63141afae92a06b9@mail.gmail.com>

On Sat, Mar 6, 2010 at 10:44 AM, Vincent Davis <vincent at vincentdavis.net>wrote:

> Ok not sure what I am doing wrong here, In fact I am sure this was working
> and now I updated python and it doesn't now. I am skeptical that that is
> true. My question is what is happening to the strings?
>
> >>> import numpy as np
> >>> a = [(1,1),(1,2),(1,3),(2,4),(2,5),(2,6)]
> >>> b = [(1,1,'apple', 'pie'),(2,5,'boys','play')]
> >>> aa = np.array(a, [('x',int),('y',int)])
> >>> bb = np.array(b, [('x',int),('y',int), ('xb',str), ('yb',str)])
> >>> bb
> array([(1, 1, '', ''), (2, 5, '', '')],
>       dtype=[('x', '<i8'), ('y', '<i8'), ('xb', '|S0'), ('yb', '|S0')])
>

You need to give it a size for the strings (I've never really understood why
you need to with structured arrays, but not regular ones). Try this:

bb = np.array(b, [('x',int),('y',int), ('xb','S5'), ('yb','S5')])

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100306/52de355f/attachment.html>

From robert.kern at gmail.com  Sat Mar  6 13:46:42 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 6 Mar 2010 12:46:42 -0600
Subject: [SciPy-User] structured array question? (slow learner :)
In-Reply-To: <c7b95a021003060926l105fc892v63141afae92a06b9@mail.gmail.com>
References: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com> 
	<c7b95a021003060926l105fc892v63141afae92a06b9@mail.gmail.com>
Message-ID: <3d375d731003061046p1fe215bcv663c0ac6b9b4b2fa@mail.gmail.com>

On Sat, Mar 6, 2010 at 11:26, Ryan May <rmay31 at gmail.com> wrote:
> On Sat, Mar 6, 2010 at 10:44 AM, Vincent Davis <vincent at vincentdavis.net>
> wrote:
>>
>> Ok not sure what I am doing wrong here, In fact I am sure this was working
>> and now I updated python and it doesn't now. I am skeptical that that is
>> true. My question is what is happening to the strings?
>> >>> import numpy as np
>> >>> a = [(1,1),(1,2),(1,3),(2,4),(2,5),(2,6)]
>> >>> b = [(1,1,'apple', 'pie'),(2,5,'boys','play')]
>> >>> aa = np.array(a, [('x',int),('y',int)])
>> >>> bb = np.array(b, [('x',int),('y',int), ('xb',str), ('yb',str)])
>> >>> bb
>> array([(1, 1, '', ''), (2, 5, '', '')],
>> ?? ? ?dtype=[('x', '<i8'), ('y', '<i8'), ('xb', '|S0'), ('yb', '|S0')])
>
> You need to give it a size for the strings (I've never really understood why
> you need to with structured arrays, but not regular ones).

np.array() is a complicated function with many codepaths. It can do
some autodetection for simple dtypes, but it would be difficult to
write one for structured dtypes, I think.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From vincent at vincentdavis.net  Sat Mar  6 16:08:02 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 6 Mar 2010 14:08:02 -0700
Subject: [SciPy-User] structured array question? (slow learner :)
In-Reply-To: <3d375d731003061046p1fe215bcv663c0ac6b9b4b2fa@mail.gmail.com>
References: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com>
	<c7b95a021003060926l105fc892v63141afae92a06b9@mail.gmail.com>
	<3d375d731003061046p1fe215bcv663c0ac6b9b4b2fa@mail.gmail.com>
Message-ID: <77e831101003061308y7f4e5f31g549791d707e5b21@mail.gmail.com>

Again I am new to this but I spent 2hr on this and looked at the
documentation an tutorials. To be fare there are not a lot of examples using
strings.
I would suggest that there are two many ways to specify dtype. At least that
is my impression.
I would find a nice table that list all with the different way they are
used.

for example
@ Ryan used ('xb','S5')
@ Christopher used  ('xb', '|S10')
I tried ('xb', |S10)
there is also a10 or somthing
numpy.float64() can be used

Is it possible for me to help contribute something like this. As I learn
more about python and numpy I often come across topics that are not very
clear to me, or could benefit from more/different examples. I assume this is
most applicable to the tutorials.

Thanks again

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sat, Mar 6, 2010 at 11:46 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Sat, Mar 6, 2010 at 11:26, Ryan May <rmay31 at gmail.com> wrote:
> > On Sat, Mar 6, 2010 at 10:44 AM, Vincent Davis <vincent at vincentdavis.net
> >
> > wrote:
> >>
> >> Ok not sure what I am doing wrong here, In fact I am sure this was
> working
> >> and now I updated python and it doesn't now. I am skeptical that that is
> >> true. My question is what is happening to the strings?
> >> >>> import numpy as np
> >> >>> a = [(1,1),(1,2),(1,3),(2,4),(2,5),(2,6)]
> >> >>> b = [(1,1,'apple', 'pie'),(2,5,'boys','play')]
> >> >>> aa = np.array(a, [('x',int),('y',int)])
> >> >>> bb = np.array(b, [('x',int),('y',int), ('xb',str), ('yb',str)])
> >> >>> bb
> >> array([(1, 1, '', ''), (2, 5, '', '')],
> >>       dtype=[('x', '<i8'), ('y', '<i8'), ('xb', '|S0'), ('yb', '|S0')])
> >
> > You need to give it a size for the strings (I've never really understood
> why
> > you need to with structured arrays, but not regular ones).
>
> np.array() is a complicated function with many codepaths. It can do
> some autodetection for simple dtypes, but it would be difficult to
> write one for structured dtypes, I think.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100306/8b5ca997/attachment.html>

From vincent at vincentdavis.net  Sat Mar  6 16:46:25 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 6 Mar 2010 14:46:25 -0700
Subject: [SciPy-User] Combining arrays based on (x,y) index values
Message-ID: <77e831101003061346u1cca3d90n7231a0dfec35c5d5@mail.gmail.com>

I have 3 arrays, each have an x and y column and I need to extend each row
with the additional rows from the other arrays. Only one of the arrays has
all the x,y values. I am also not sure how to deal with the missing values.
I was thinking about making an array full of nan. So for example if I have
>>> bb
array([['1', '1', 'apple', 'pie'],
       ['2', '5', 'boys', 'play']],
      dtype='|S8')
>>> aa
array([[1, 1,free],
       [1, 2,upgrade],
       [1, 3, down],
       [2, 4, left],
       [2, 5, right],
       [2, 6, now]])
and I what to end up with

magicfunc(aa,bb)
array([[1, 1, free,  'apple', 'pie'],
       [1, 2, upgrade, '', ''],
       [1, 3, down,  '', ''],
       [2, 4, left, '', ''],
       [2, 5, right, 'boys', 'play'],
       [2, 6, now, '', '']])

Whats the "right" way to do this, I have several ideas but I can't be the
first to do this. I know the dimensions of the final array. but don't really
have a plan of where to go from there.

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100306/0e348ca0/attachment.html>

From robert.kern at gmail.com  Sat Mar  6 17:30:20 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 6 Mar 2010 16:30:20 -0600
Subject: [SciPy-User] Combining arrays based on (x,y) index values
In-Reply-To: <77e831101003061346u1cca3d90n7231a0dfec35c5d5@mail.gmail.com>
References: <77e831101003061346u1cca3d90n7231a0dfec35c5d5@mail.gmail.com>
Message-ID: <3d375d731003061430m67307c2dnb1b3c64bd9926482@mail.gmail.com>

On Sat, Mar 6, 2010 at 15:46, Vincent Davis <vincent at vincentdavis.net> wrote:
>
> I have 3 arrays, each have an x and y column and I need to extend each row with the additional rows from the other arrays. Only one of the arrays has all the x,y values. I am also not sure how to deal with the missing values. I was thinking about making an array full of nan.

You want what is called an outer join. You have to use structured arrays:

In [2]: import numpy as np

In [3]: b_dtype = np.dtype([('i', int), ('j', int), ('b_word1', 'S8'),
('b_word2', 'S8')])

In [4]: bb = np.array([(1, 1, 'apple', 'pie'), (2, 5, 'boys', 'play')])

In [5]: bb
Out[5]:
array([['1', '1', 'apple', 'pie'],
       ['2', '5', 'boys', 'play']],
      dtype='|S5')

In [6]: bb = np.array([(1, 1, 'apple', 'pie'), (2, 5, 'boys',
'play')], dtype=b_dtype)

In [7]: bb
Out[7]:
array([(1, 1, 'apple', 'pie'), (2, 5, 'boys', 'play')],
      dtype=[('i', '<i4'), ('j', '<i4'), ('b_word1', '|S8'),
('b_word2', '|S8')])

In [8]: a_dtype = np.dtype([('i', int), ('j', int), ('a_word', 'S8')])

In [9]: aa = np.array([(1, 1, 'free'), (1, 2, 'upgrade'), (1, 3,
'down'), (2, 5, 'right')], dtype=a_dtype)

In [10]: aa
Out[10]:
array([(1, 1, 'free'), (1, 2, 'upgrade'), (1, 3, 'down'), (2, 5, 'right')],
      dtype=[('i', '<i4'), ('j', '<i4'), ('a_word', '|S8')])

In [11]: from numpy.lib.recfunctions import join_by

In [12]: join_by(['i', 'j'], aa, bb, jointype='outer')
Out[12]:
masked_array(data = [(1, 1, 'free', 'apple', 'pie') (1, 2, 'upgrade', --, --)
 (1, 3, 'down', --, --) (2, 5, 'right', 'boys', 'play')],
             mask = [(False, False, False, False, False) (False,
False, False, True, True)
 (False, False, False, True, True) (False, False, False, False, False)],
       fill_value = (999999, 999999, 'N/A', 'N/A', 'N/A'),
            dtype = [('i', '<i4'), ('j', '<i4'), ('a_word', '|S8'),
('b_word1', '|S8'), ('b_word2', '|S8')])

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
?-- Umberto Eco


From vincent at vincentdavis.net  Sat Mar  6 18:32:22 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 6 Mar 2010 16:32:22 -0700
Subject: [SciPy-User] Combining arrays based on (x,y) index values
In-Reply-To: <3d375d731003061430m67307c2dnb1b3c64bd9926482@mail.gmail.com>
References: <77e831101003061346u1cca3d90n7231a0dfec35c5d5@mail.gmail.com>
	<3d375d731003061430m67307c2dnb1b3c64bd9926482@mail.gmail.com>
Message-ID: <77e831101003061532x7326b15fi6c3492ce0068172f@mail.gmail.com>

Thank you, that is exactly what I needed, and I was already using structured
arrays.

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sat, Mar 6, 2010 at 3:30 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Sat, Mar 6, 2010 at 15:46, Vincent Davis <vincent at vincentdavis.net>
> wrote:
> >
> > I have 3 arrays, each have an x and y column and I need to extend each
> row with the additional rows from the other arrays. Only one of the arrays
> has all the x,y values. I am also not sure how to deal with the missing
> values. I was thinking about making an array full of nan.
>
> You want what is called an outer join. You have to use structured arrays:
>
> In [2]: import numpy as np
>
> In [3]: b_dtype = np.dtype([('i', int), ('j', int), ('b_word1', 'S8'),
> ('b_word2', 'S8')])
>
> In [4]: bb = np.array([(1, 1, 'apple', 'pie'), (2, 5, 'boys', 'play')])
>
> In [5]: bb
> Out[5]:
> array([['1', '1', 'apple', 'pie'],
>       ['2', '5', 'boys', 'play']],
>       dtype='|S5')
>
> In [6]: bb = np.array([(1, 1, 'apple', 'pie'), (2, 5, 'boys',
> 'play')], dtype=b_dtype)
>
> In [7]: bb
> Out[7]:
> array([(1, 1, 'apple', 'pie'), (2, 5, 'boys', 'play')],
>       dtype=[('i', '<i4'), ('j', '<i4'), ('b_word1', '|S8'),
> ('b_word2', '|S8')])
>
> In [8]: a_dtype = np.dtype([('i', int), ('j', int), ('a_word', 'S8')])
>
> In [9]: aa = np.array([(1, 1, 'free'), (1, 2, 'upgrade'), (1, 3,
> 'down'), (2, 5, 'right')], dtype=a_dtype)
>
> In [10]: aa
> Out[10]:
> array([(1, 1, 'free'), (1, 2, 'upgrade'), (1, 3, 'down'), (2, 5, 'right')],
>      dtype=[('i', '<i4'), ('j', '<i4'), ('a_word', '|S8')])
>
> In [11]: from numpy.lib.recfunctions import join_by
>
> In [12]: join_by(['i', 'j'], aa, bb, jointype='outer')
> Out[12]:
> masked_array(data = [(1, 1, 'free', 'apple', 'pie') (1, 2, 'upgrade', --,
> --)
>  (1, 3, 'down', --, --) (2, 5, 'right', 'boys', 'play')],
>             mask = [(False, False, False, False, False) (False,
> False, False, True, True)
>  (False, False, False, True, True) (False, False, False, False, False)],
>       fill_value = (999999, 999999, 'N/A', 'N/A', 'N/A'),
>            dtype = [('i', '<i4'), ('j', '<i4'), ('a_word', '|S8'),
> ('b_word1', '|S8'), ('b_word2', '|S8')])
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100306/8bae2a69/attachment.html>

From vincent at vincentdavis.net  Sun Mar  7 00:22:56 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 6 Mar 2010 22:22:56 -0700
Subject: [SciPy-User] operations on large arrays
Message-ID: <77e831101003062122w2890311eh8a4fd856cbdc13b@mail.gmail.com>

I have arrays of 8-20 rows and 230,000 column, all the data is float64
I what to be able to find the difference in the correlation matrix between
arrays
let A and B be of size (10, 230000)
np.corrcoef(a)-np.corrcoef(b)

I can't seem to do this with more than 10000 columns at a time because of
memory limitations. (about 9GB usable to python)
Is there a better way?

I also have problem finding the column means which is surprising to me, I
was not able to get the column means for 10000 columns, but I can computer
 the corrcoeff ?
np.mean(a, axis=0)

Do I just need to divide up the job or is there a better approach?

Thanks

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100306/9a550d56/attachment.html>

From vincent at vincentdavis.net  Sun Mar  7 01:05:03 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 6 Mar 2010 23:05:03 -0700
Subject: [SciPy-User] operations on large arrays
In-Reply-To: <77e831101003062122w2890311eh8a4fd856cbdc13b@mail.gmail.com>
References: <77e831101003062122w2890311eh8a4fd856cbdc13b@mail.gmail.com>
Message-ID: <77e831101003062205n22790adw6804401c40e3ee60@mail.gmail.com>

I just figured out that I had a few arrays that where taking up a bunch of
the memory. That said I still wonder if there is a better way.

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sat, Mar 6, 2010 at 10:22 PM, Vincent Davis <vincent at vincentdavis.net>wrote:

> I have arrays of 8-20 rows and 230,000 column, all the data is float64
> I what to be able to find the difference in the correlation matrix between
> arrays
> let A and B be of size (10, 230000)
> np.corrcoef(a)-np.corrcoef(b)
>
> I can't seem to do this with more than 10000 columns at a time because of
> memory limitations. (about 9GB usable to python)
> Is there a better way?
>
> I also have problem finding the column means which is surprising to me, I
> was not able to get the column means for 10000 columns, but I can computer
>  the corrcoeff ?
> np.mean(a, axis=0)
>
> Do I just need to divide up the job or is there a better approach?
>
> Thanks
>
>   *Vincent Davis
> 720-301-3003 *
> vincent at vincentdavis.net
>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100306/748ef99d/attachment.html>

From rmay31 at gmail.com  Sun Mar  7 09:58:24 2010
From: rmay31 at gmail.com (Ryan May)
Date: Sun, 7 Mar 2010 08:58:24 -0600
Subject: [SciPy-User] structured array question? (slow learner :)
In-Reply-To: <77e831101003061308y7f4e5f31g549791d707e5b21@mail.gmail.com>
References: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com> 
	<c7b95a021003060926l105fc892v63141afae92a06b9@mail.gmail.com> 
	<3d375d731003061046p1fe215bcv663c0ac6b9b4b2fa@mail.gmail.com> 
	<77e831101003061308y7f4e5f31g549791d707e5b21@mail.gmail.com>
Message-ID: <c7b95a021003070658l57c03898ydb604a4b037c2624@mail.gmail.com>

On Sat, Mar 6, 2010 at 3:08 PM, Vincent Davis <vincent at vincentdavis.net>wrote:

> Again I am new to this but I spent 2hr on this and looked at the
> documentation an tutorials. To be fare there are not a lot of examples using
> strings.
> I would suggest that there are two many ways to specify dtype. At least
> that is my impression.
> I would find a nice table that list all with the different way they are
> used.
>
> for example
> @ Ryan used ('xb','S5')
> @ Christopher used  ('xb', '|S10')
>

These are both specifying strings.  'S5' is just specifying a string length
of 5 while '|S10' is specifying a string length of 10.  The '|' is optional
and specifies that the system-native endianness should be used to control
the byte ordering. So these are both the exact same way.


> I tried ('xb', |S10)
>

There's no way this actually ran without quotes.


> there is also a10 or somthing
>

'a' is the same as using 'S' above.


> numpy.float64() can be used
>

That's for floating point types, not strings. A lot of the complexity from
strings is that, unlike numeric types, there's no minium/typical sizes to
assume for strings. This, combined with the fact that all items within a
numpy array have to have the same size, makes string handling complex.
(This implies that in memory, (1,1,'apple', 'pie') and (2,5,'boys','play')
will occupy the same amount of memory.)

Did you look at this:

http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100307/df4e5940/attachment.html>

From eadrogue at gmx.net  Sun Mar  7 12:56:26 2010
From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=)
Date: Sun, 7 Mar 2010 18:56:26 +0100
Subject: [SciPy-User] structured array question? (slow learner :)
In-Reply-To: <77e831101003061308y7f4e5f31g549791d707e5b21@mail.gmail.com>
References: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com>
	<c7b95a021003060926l105fc892v63141afae92a06b9@mail.gmail.com>
	<3d375d731003061046p1fe215bcv663c0ac6b9b4b2fa@mail.gmail.com>
	<77e831101003061308y7f4e5f31g549791d707e5b21@mail.gmail.com>
Message-ID: <20100307175626.GA3210@doriath.local>

 6/03/10 @ 14:08 (-0700), thus spake Vincent Davis:
> Again I am new to this but I spent 2hr on this and looked at the
> documentation an tutorials. To be fare there are not a lot of examples using
> strings.

Notice that you can also use Python objects instead of strings,
as there's little or no performance penalty.

In [5]: np.array([('foo', 4.5), ('bar', 5.1)], 'O, d')

This way you need not specify the length of the string in the dtype. 

Ernest



From vincent at vincentdavis.net  Sun Mar  7 16:02:13 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sun, 7 Mar 2010 14:02:13 -0700
Subject: [SciPy-User] structured array question? (slow learner :)
In-Reply-To: <c7b95a021003070658l57c03898ydb604a4b037c2624@mail.gmail.com>
References: <77e831101003060844y1b475c39scac32e32219d3600@mail.gmail.com>
	<c7b95a021003060926l105fc892v63141afae92a06b9@mail.gmail.com>
	<3d375d731003061046p1fe215bcv663c0ac6b9b4b2fa@mail.gmail.com>
	<77e831101003061308y7f4e5f31g549791d707e5b21@mail.gmail.com>
	<c7b95a021003070658l57c03898ydb604a4b037c2624@mail.gmail.com>
Message-ID: <77e831101003071302v75456d2fr1e62c5e3129f2a65@mail.gmail.com>

Thanks again for the help, My point in mentioning that I was not able to
find the answer was to point out that the information is spread out and not
obvious the first time you look for the information. The fact that a10 |S10,
S10 are the same thing and numpy.str does not get rejected but does not work
is confusing the first time you look at it, I was suggesting that it could
be presented a little better. Probably the only contribution I can make is
to point out that there is a lack of documentation from the perspective of a
beginner.

Thanks Again

*Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sun, Mar 7, 2010 at 7:58 AM, Ryan May <rmay31 at gmail.com> wrote:

> On Sat, Mar 6, 2010 at 3:08 PM, Vincent Davis <vincent at vincentdavis.net>wrote:
>
>> Again I am new to this but I spent 2hr on this and looked at the
>> documentation an tutorials. To be fare there are not a lot of examples using
>> strings.
>> I would suggest that there are two many ways to specify dtype. At least
>> that is my impression.
>> I would find a nice table that list all with the different way they are
>> used.
>>
>> for example
>> @ Ryan used ('xb','S5')
>> @ Christopher used  ('xb', '|S10')
>>
>
> These are both specifying strings.  'S5' is just specifying a string length
> of 5 while '|S10' is specifying a string length of 10.  The '|' is optional
> and specifies that the system-native endianness should be used to control
> the byte ordering. So these are both the exact same way.
>
>
>> I tried ('xb', |S10)
>>
>
> There's no way this actually ran without quotes.
>
>
>> there is also a10 or somthing
>>
>
> 'a' is the same as using 'S' above.
>
>
>> numpy.float64() can be used
>>
>
> That's for floating point types, not strings. A lot of the complexity from
> strings is that, unlike numeric types, there's no minium/typical sizes to
> assume for strings. This, combined with the fact that all items within a
> numpy array have to have the same size, makes string handling complex.
> (This implies that in memory, (1,1,'apple', 'pie') and (2,5,'boys','play')
> will occupy the same amount of memory.)
>
> Did you look at this:
>
> http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
>
>
> Ryan
>
> --
> Ryan May
> Graduate Research Assistant
> School of Meteorology
> University of Oklahoma
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100307/ad8315d1/attachment.html>

From d.l.goldsmith at gmail.com  Mon Mar  8 02:42:58 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Sun, 7 Mar 2010 23:42:58 -0800
Subject: [SciPy-User] [OT] Anyone here know Audacity and WAV?
Message-ID: <45d1ab481003072342y5350324an434e0ac35fa71cbc@mail.gmail.com>

If so, please reply off list.

Thanks!

DG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100307/4f5b6dc3/attachment.html>

From millman at berkeley.edu  Mon Mar  8 03:44:52 2010
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 8 Mar 2010 00:44:52 -0800
Subject: [SciPy-User] PSF GSoC 2010 (Py3K focus)
Message-ID: <c7009a551003080044w656f4fc2o66fe9a417fc42247@mail.gmail.com>

Hello,

Given the interest in participating in the GSoC this summer, I am
forwarding a very interesting email from Titus Brown.  If you are
interested in doing a GSoC or mentoring, please read his email
carefully.

Basically, the PSF will be focuing on Py3K-related projects.  Given
Pauli's work on Py3K support for NumPy, I think we might be in a good
position to move forward on porting the rest of our stack to Py3K.  So
we should focus on projects to:

1. finish porting and testing NumPy with Py3K
2. port and test SciPy with Py3K
3. port and test matplotlib with Py3K
4. port and test ipython with Py3K
5. etc.

Given the PSF's stated emphasis this year, it probably doesn't make
sense to pursue any non-Py3K projects.

Jarrod

---------- Forwarded message ----------
From: C. Titus Brown <ctb at msu.edu>
Date: Tue, Mar 2, 2010 at 6:12 AM
Subject: [SoC2009-mentors] [ctb at msu.edu: GSoC 2010 - it's on!]
To: soc2009-mentors at python.org


----- Forwarded message from "C. Titus Brown" <ctb at msu.edu> -----

Date: Wed, 24 Feb 2010 12:54:52 -0800
From: "C. Titus Brown" <ctb at msu.edu>
To: psf-members at python.org
Cc: gsoc2010-mentors at python.org
Subject: GSoC 2010 - it's on!

Hi all,

it's that time of year again, and Google has decided to run the Google
Summer of Code again!

?http://groups.google.com/group/google-summer-of-code-discuss/browse_thread/thread/d839c0b02ac15b3f

?http://socghop.appspot.com/

Arc Riley has stepped up to run it for the PSF again this year, and I'm
backstopping him. ?If you are interested in mentoring or kibbitzing on those
who are, please sign up for the soc2010-mentors mailing list here,

?http://mail.python.org/mailman/listinfo/soc2010-mentors

This year we're proposing to solicit and prioritize applications for
Python 3.x -- 3K tools, porting old projects, etc. ?Python 2.x projects
will be a distinct second. ?There will be no "core" category this year,
although obviously if someone on one of the core teams wants to push a
project it'll help!

If you have an idea for a project, please send it to the -mentors list and add
it to the wiki at

? http://wiki.python.org/moin/SummerOfCode/2010

We're also going to change a few things up to make it more useful to the PSF.
Specifically,

?- the foundation is going to *require* 1 blog post/wk from each student.

?- we're going to hire an administrative assistant to monitor the students.

?- the student application process will be a bit more rigorous and job-app
? like; the Django SF has been doing this for at least one round and they
? claim that it results in much better and more serious students.

?- we'll be focusing on student quality more than on project egalitarianism.
? If project X can recruit three fantastic students to one fantastic and one
? mediocre student for project Y, then project X gets three and project Y
? gets one.

The hope is that this will make the GSoC much more useful for Python than
it has been in the past.

Arc will be posting something to the www.python.org site and python-announce
soon, too.

Followups to soc2010-mentors.

cheers,
--titus
--
C. Titus Brown, ctb at msu.edu

----- End forwarded message -----


From millman at berkeley.edu  Mon Mar  8 03:52:25 2010
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 8 Mar 2010 00:52:25 -0800
Subject: [SciPy-User] Google Summer of Code...
In-Reply-To: <ec9f80fa1003030114i14018e00n6153a9dc6926a860@mail.gmail.com>
References: <4B8C3C81.7000509@noaa.gov>
	<1cd32cbb1003011435u725d37e5ne3c98475f3cfa8a@mail.gmail.com>
	<3d375d731003011443h1a5c0b02h837f5af3f5348037@mail.gmail.com>
	<c048da1c1003011459g6d8faa6eg38751db3f88b6417@mail.gmail.com>
	<EEDF0DA5-C8FD-4AA1-9CF5-B0708B079E6B@cs.toronto.edu>
	<ec9f80fa1003030114i14018e00n6153a9dc6926a860@mail.gmail.com>
Message-ID: <c7009a551003080052q4bc1c438nb5ee6796604f05c0@mail.gmail.com>

Hello Sebastian,

Thanks for taking the time to outline a potential GSoC project.  It is
definitely worth taking the time to add it to the wiki:
http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas

Unfortunately, though, it isn't likely that a project like this will
be accepted this year.  But it will be nice to have the proposal
written down because someone might decide to work on it without a GSoC
or someone might do the project next year.

I forwarded an email from Titus Brown, which explains that the PSF
will be looking for Py3K-related projects this year.  So if we get any
GSoC projects funded this year, they will probably be focused on
porting numpy, scipy, matplotlib, ipython, etc. to Py3K.

Thanks,
Jarrod

On Wed, Mar 3, 2010 at 1:14 AM, Sebastian Walter
<sebastian.walter at gmail.com> wrote:
> I could mentor an algorithmic differentiation (AD) project, but I'm
> still not quite sure how to proceed.
> Should I write up a proposal on the wiki and after that propose the
> project on the
> http://mail.python.org/mailman/listinfo/soc2010-mentors mailing list?
>
> I have already created a package that I believe would be a very good
> addition to scipy/numpy (preferably numpy) when it is mature enough.
> It is hosted on http://github.com/b45ch1/taylorpoly
>
> The package is not an AD tool but makes it very easy to create AD
> tools using those algorithms.
> An illustrative example how to do (forward mode) AD is given in:
> http://github.com/b45ch1/taylorpoly/blob/master/examples/utps_for_algorithmic_differentiation.py
>
> What is missing up to now are differentiated algorithms for tan, atan,
> asin, sinh, etc. (in Taylor arithmetic)
> as well as the core algorithms for reverse mode AD.
>
> Implementing those algorithms should be doable in 4 months. Also,
> since the task is to implement many small algorithms,
> this project is unlikely to fail completely. The worst that can happen
> is that not as many algorithms get implemented as one could wish for.
>
> Could anyone give feedback if that project is ok?
>
> best regards,
> Sebastian


From sebastian.walter at gmail.com  Mon Mar  8 04:09:10 2010
From: sebastian.walter at gmail.com (Sebastian Walter)
Date: Mon, 8 Mar 2010 10:09:10 +0100
Subject: [SciPy-User] Google Summer of Code...
In-Reply-To: <c7009a551003080052q4bc1c438nb5ee6796604f05c0@mail.gmail.com>
References: <4B8C3C81.7000509@noaa.gov>
	<1cd32cbb1003011435u725d37e5ne3c98475f3cfa8a@mail.gmail.com>
	<3d375d731003011443h1a5c0b02h837f5af3f5348037@mail.gmail.com>
	<c048da1c1003011459g6d8faa6eg38751db3f88b6417@mail.gmail.com>
	<EEDF0DA5-C8FD-4AA1-9CF5-B0708B079E6B@cs.toronto.edu>
	<ec9f80fa1003030114i14018e00n6153a9dc6926a860@mail.gmail.com>
	<c7009a551003080052q4bc1c438nb5ee6796604f05c0@mail.gmail.com>
Message-ID: <ec9f80fa1003080109k4e8ab0fpff4da51c33c17e57@mail.gmail.com>

On Mon, Mar 8, 2010 at 9:52 AM, Jarrod Millman <millman at berkeley.edu> wrote:
> Hello Sebastian,
>
> Thanks for taking the time to outline a potential GSoC project. ?It is
> definitely worth taking the time to add it to the wiki:
> http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas
>
> Unfortunately, though, it isn't likely that a project like this will
> be accepted this year. ?But it will be nice to have the proposal
> written down because someone might decide to work on it without a GSoC
> or someone might do the project next year.

will do that

>
> I forwarded an email from Titus Brown, which explains that the PSF
> will be looking for Py3K-related projects this year. ?So if we get any
> GSoC projects funded this year, they will probably be focused on
> porting numpy, scipy, matplotlib, ipython, etc. to Py3K.

I've just read that email about the focus on Py3K.
It's a pity that the focus is solely on porting code and not on adding
new features at all.
But of course I perfectly understand the priorities.



>
> Thanks,
> Jarrod
>
> On Wed, Mar 3, 2010 at 1:14 AM, Sebastian Walter
> <sebastian.walter at gmail.com> wrote:
>> I could mentor an algorithmic differentiation (AD) project, but I'm
>> still not quite sure how to proceed.
>> Should I write up a proposal on the wiki and after that propose the
>> project on the
>> http://mail.python.org/mailman/listinfo/soc2010-mentors mailing list?
>>
>> I have already created a package that I believe would be a very good
>> addition to scipy/numpy (preferably numpy) when it is mature enough.
>> It is hosted on http://github.com/b45ch1/taylorpoly
>>
>> The package is not an AD tool but makes it very easy to create AD
>> tools using those algorithms.
>> An illustrative example how to do (forward mode) AD is given in:
>> http://github.com/b45ch1/taylorpoly/blob/master/examples/utps_for_algorithmic_differentiation.py
>>
>> What is missing up to now are differentiated algorithms for tan, atan,
>> asin, sinh, etc. (in Taylor arithmetic)
>> as well as the core algorithms for reverse mode AD.
>>
>> Implementing those algorithms should be doable in 4 months. Also,
>> since the task is to implement many small algorithms,
>> this project is unlikely to fail completely. The worst that can happen
>> is that not as many algorithms get implemented as one could wish for.
>>
>> Could anyone give feedback if that project is ok?
>>
>> best regards,
>> Sebastian
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From dlenski at gmail.com  Mon Mar  8 10:09:37 2010
From: dlenski at gmail.com (Daniel Lenski)
Date: Mon, 8 Mar 2010 15:09:37 +0000 (UTC)
Subject: [SciPy-User] fromimage/imread segfaults on my images
Message-ID: <hn33vh$bgo$1@dough.gmane.org>

Hi all,
I'm using SciPy to process some sparse binary images with a resolution of 
about 5000x5000 pixels.  I try to load images like this:

  im = Image.open('/tmp/foo.pbm')
  print "Image loaded, size is %s and mode is %s." % (im.size, im.mode)
  arr = fromimage(im)

or

  arr = imread('/tmp/foo.pbm')

In either case, I get a segfault.  The segfault is definitely in the 
SciPy code, rather than the PIL code, since PIL's Image.open runs fine. 

I ran strace, and there's an attempt to mmap() a region as large as the 
image, one byte per pixel, immediately before the segfault:

  mmap(NULL, 21463040, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
       -1, 0) = 0x7f278b5be000

The funny thing is... the imread() implementation in pylab *does* work, 
but it converts all images to RGB mode and reverses them vertically for 
no apparent reason.  So I'd rather not use it.  Is this a known bug in 
SciPy?  Any workarounds/fixes?  I don't know any other good, reliable way 
to get a big image into an array :-(

Dan



From dlenski at gmail.com  Mon Mar  8 10:31:18 2010
From: dlenski at gmail.com (Daniel Lenski)
Date: Mon, 8 Mar 2010 15:31:18 +0000 (UTC)
Subject: [SciPy-User] fromimage/imread segfaults on my images
References: <hn33vh$bgo$1@dough.gmane.org>
Message-ID: <hn3585$bgo$2@dough.gmane.org>

On Mon, 08 Mar 2010 15:09:37 +0000, Daniel Lenski wrote:

> I ran strace, and there's an attempt to mmap() a region as large as the
> image, one byte per pixel, immediately before the segfault:
> 
>   mmap(NULL, 21463040, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
>        -1, 0) = 0x7f278b5be000

Just to clarify... this system is running Linux 2.6, amd64 build, and has 
4gb of RAM.  So there's no reason why a 21 MB mmap would fail, normally.

Dan



From paul.anton.letnes at gmail.com  Mon Mar  8 12:47:51 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Mon, 8 Mar 2010 09:47:51 -0800
Subject: [SciPy-User] scipy+lapack array copy
Message-ID: <A0FC46B8-3D80-4681-A117-C77C16425CF1@gmail.com>

Hi everyone.

I have a numerical project where I link a fortran module to python using f2py. The fortran module basically sets up the left hand side coefficient matrix of a linear equation system. The right hand side is small, so I set that up in python. So far, everything works great.

Now, when I call scipy-lapack, I am using:
from scipy.linalg.lapack import flapack as lapack
lu, piv, x, info = lapack.zgesv(A, b)
A is created in the fortran part of the program. b is created from numpy.zeros(). Both report as F_CONTIGUOUS by printing A.flags and b.flags.

The problem is that I'd like A to be large (>1GB), but the lapack wrapper (or lapack itself?) copies the array, doubling the memory use. Aside from being unnecessary, this also reduces the practical problem size I can solve by a factor 2, which of course is very annoying.

How can I get around this problem?

Regards,
Paul.

From jsseabold at gmail.com  Mon Mar  8 13:10:32 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Mon, 8 Mar 2010 13:10:32 -0500
Subject: [SciPy-User] scipy+lapack array copy
In-Reply-To: <A0FC46B8-3D80-4681-A117-C77C16425CF1@gmail.com>
References: <A0FC46B8-3D80-4681-A117-C77C16425CF1@gmail.com>
Message-ID: <c048da1c1003081010t51998d71y6933f88e17bcad45@mail.gmail.com>

On Mon, Mar 8, 2010 at 12:47 PM, Paul Anton Letnes
<paul.anton.letnes at gmail.com> wrote:
> Hi everyone.
>
> I have a numerical project where I link a fortran module to python using f2py. The fortran module basically sets up the left hand side coefficient matrix of a linear equation system. The right hand side is small, so I set that up in python. So far, everything works great.
>
> Now, when I call scipy-lapack, I am using:
> from scipy.linalg.lapack import flapack as lapack
> lu, piv, x, info = lapack.zgesv(A, b)
> A is created in the fortran part of the program. b is created from numpy.zeros(). Both report as F_CONTIGUOUS by printing A.flags and b.flags.
>
> The problem is that I'd like A to be large (>1GB), but the lapack wrapper (or lapack itself?) copies the array, doubling the memory use. Aside from being unnecessary, this also reduces the practical problem size I can solve by a factor 2, which of course is very annoying.
>
> How can I get around this problem?
>

It looks like zgesv allows you to set overwrite_a = 1.  If this isn't
what you want, there might be another way to proceed.

In [5]: from scipy.linalg.lapack import flapack as lapack

In [6]: lapack.zgesv?
Type:           fortran
String Form:    <fortran object at 0x7fccc4f6af08>
Namespace:      Interactive
Docstring:
    zgesv - Function signature:
      lu,piv,x,info = zgesv(a,b,[overwrite_a,overwrite_b])
    Required arguments:
      a : input rank-2 array('D') with bounds (n,n)
      b : input rank-2 array('D') with bounds (n,nrhs)
    Optional arguments:
      overwrite_a := 0 input int
      overwrite_b := 0 input int
    Return objects:
      lu : rank-2 array('D') with bounds (n,n) and a storage
      piv : rank-1 array('i') with bounds (n)
      x : rank-2 array('D') with bounds (n,nrhs) and b storage
      info : int


Skipper


From Deil.Christoph at googlemail.com  Mon Mar  8 14:02:49 2010
From: Deil.Christoph at googlemail.com (Christoph Deil)
Date: Mon, 8 Mar 2010 20:02:49 +0100
Subject: [SciPy-User] flat / nonflat array index conversion
Message-ID: <7523C74A-2E1C-4AF1-8B6D-23CD51D39930@googlemail.com>

Is there a numpy function to convert corresponding array indices in flattened / nonflat multidimensional arrays for a given shape?

E.g. for a = array([0,1,2,3,4,5]).reshape(2,3) I want some function that converts e.g. 1 to [0,1] and 5 to [1,2] if I tell it a.shape. For 2D it's of course easy to do it by hand, but I need something that is fast and works for arrays of any dimension.



From josef.pktd at gmail.com  Mon Mar  8 14:11:36 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 8 Mar 2010 14:11:36 -0500
Subject: [SciPy-User] flat / nonflat array index conversion
In-Reply-To: <7523C74A-2E1C-4AF1-8B6D-23CD51D39930@googlemail.com>
References: <7523C74A-2E1C-4AF1-8B6D-23CD51D39930@googlemail.com>
Message-ID: <1cd32cbb1003081111y7ae45c96n1771c35b29ea2bf3@mail.gmail.com>

On Mon, Mar 8, 2010 at 2:02 PM, Christoph Deil
<Deil.Christoph at googlemail.com> wrote:
> Is there a numpy function to convert corresponding array indices in flattened / nonflat multidimensional arrays for a given shape?
>
> E.g. for a = array([0,1,2,3,4,5]).reshape(2,3) I want some function that converts e.g. 1 to [0,1] and 5 to [1,2] if I tell it a.shape. For 2D it's of course easy to do it by hand, but I need something that is fast and works for arrays of any dimension.

in numpy docs under Indexing Routines

>>> list(np.ndindex(3,2,4))
[(0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 0, 3), (0, 1, 0), (0, 1, 1), (0,
1, 2), (0, 1, 3), (1, 0, 0), (1, 0, 1), (1, 0, 2), (1, 0, 3), (1, 1,
0), (1, 1, 1), (1, 1, 2), (1, 1, 3), (2, 0, 0), (2, 0, 1), (2, 0, 2),
(2, 0, 3), (2, 1, 0), (2, 1, 1), (2, 1, 2), (2, 1, 3)]

there are also other iterators and functions

Josef

>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From kwgoodman at gmail.com  Mon Mar  8 14:11:42 2010
From: kwgoodman at gmail.com (Keith Goodman)
Date: Mon, 8 Mar 2010 11:11:42 -0800
Subject: [SciPy-User] flat / nonflat array index conversion
In-Reply-To: <7523C74A-2E1C-4AF1-8B6D-23CD51D39930@googlemail.com>
References: <7523C74A-2E1C-4AF1-8B6D-23CD51D39930@googlemail.com>
Message-ID: <f4f93d421003081111o3d693360s22c38fbd0245b868@mail.gmail.com>

On Mon, Mar 8, 2010 at 11:02 AM, Christoph Deil
<Deil.Christoph at googlemail.com> wrote:
> Is there a numpy function to convert corresponding array indices in flattened / nonflat multidimensional arrays for a given shape?
>
> E.g. for a = array([0,1,2,3,4,5]).reshape(2,3) I want some function that converts e.g. 1 to [0,1] and 5 to [1,2] if I tell it a.shape. For 2D it's of course easy to do it by hand, but I need something that is fast and works for arrays of any dimension.
>

You could create a dictionary using numpy.ndenumerate. Or you could
use a third party package such as the labeled array package, la:

>> a = np.array([0,1,2,3,4,5]).reshape(2,3)
>> from la import larry
>> y = larry(a)
>> y.todict()
   {(0, 0): 0, (0, 1): 1, (0, 2): 2, (1, 0): 3, (1, 1): 4, (1, 2): 5}


From paul.anton.letnes at gmail.com  Mon Mar  8 14:15:19 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Mon, 8 Mar 2010 11:15:19 -0800
Subject: [SciPy-User] scipy+lapack array copy
In-Reply-To: <c048da1c1003081010t51998d71y6933f88e17bcad45@mail.gmail.com>
References: <A0FC46B8-3D80-4681-A117-C77C16425CF1@gmail.com>
	<c048da1c1003081010t51998d71y6933f88e17bcad45@mail.gmail.com>
Message-ID: <B13A038A-1592-4B0F-835D-377CFD8EAC0F@gmail.com>


On 8. mars 2010, at 10.10, Skipper Seabold wrote:

> On Mon, Mar 8, 2010 at 12:47 PM, Paul Anton Letnes
> <paul.anton.letnes at gmail.com> wrote:
>> Hi everyone.
>> 
>> I have a numerical project where I link a fortran module to python using f2py. The fortran module basically sets up the left hand side coefficient matrix of a linear equation system. The right hand side is small, so I set that up in python. So far, everything works great.
>> 
>> Now, when I call scipy-lapack, I am using:
>> from scipy.linalg.lapack import flapack as lapack
>> lu, piv, x, info = lapack.zgesv(A, b)
>> A is created in the fortran part of the program. b is created from numpy.zeros(). Both report as F_CONTIGUOUS by printing A.flags and b.flags.
>> 
>> The problem is that I'd like A to be large (>1GB), but the lapack wrapper (or lapack itself?) copies the array, doubling the memory use. Aside from being unnecessary, this also reduces the practical problem size I can solve by a factor 2, which of course is very annoying.
>> 
>> How can I get around this problem?
>> 
> 
> It looks like zgesv allows you to set overwrite_a = 1.  If this isn't
> what you want, there might be another way to proceed.
> 
> In [5]: from scipy.linalg.lapack import flapack as lapack
> 
> In [6]: lapack.zgesv?
> Type:           fortran
> String Form:    <fortran object at 0x7fccc4f6af08>
> Namespace:      Interactive
> Docstring:
>    zgesv - Function signature:
>      lu,piv,x,info = zgesv(a,b,[overwrite_a,overwrite_b])
>    Required arguments:
>      a : input rank-2 array('D') with bounds (n,n)
>      b : input rank-2 array('D') with bounds (n,nrhs)
>    Optional arguments:
>      overwrite_a := 0 input int
>      overwrite_b := 0 input int
>    Return objects:
>      lu : rank-2 array('D') with bounds (n,n) and a storage
>      piv : rank-1 array('i') with bounds (n)
>      x : rank-2 array('D') with bounds (n,nrhs) and b storage
>      info : int
> 
> 
> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

Skipper,

thanks a lot. This looks like everything I need. I have to overwrite A I guess, otherwise it will have to be copied. I must have overlooked this in the documentation.

Regards,
Paul.



From rmay31 at gmail.com  Mon Mar  8 14:51:40 2010
From: rmay31 at gmail.com (Ryan May)
Date: Mon, 8 Mar 2010 13:51:40 -0600
Subject: [SciPy-User] flat / nonflat array index conversion
In-Reply-To: <7523C74A-2E1C-4AF1-8B6D-23CD51D39930@googlemail.com>
References: <7523C74A-2E1C-4AF1-8B6D-23CD51D39930@googlemail.com>
Message-ID: <c7b95a021003081151u31a8f20dp7378e4c8f65b58b4@mail.gmail.com>

On Mon, Mar 8, 2010 at 1:02 PM, Christoph Deil
<Deil.Christoph at googlemail.com> wrote:
> Is there a numpy function to convert corresponding array indices in flattened / nonflat multidimensional arrays for a given shape?
>
> E.g. for a = array([0,1,2,3,4,5]).reshape(2,3) I want some function that converts e.g. 1 to [0,1] and 5 to [1,2] if I tell it a.shape. For 2D it's of course easy to do it by hand, but I need something that is fast and works for arrays of any dimension.

Look at numpy.unravel_index:

    Convert a flat index to an index tuple for an array of given shape.

    Parameters
    ----------
    x : int
        Flattened index.
    dims : tuple of ints
        Input shape, the shape of an array into which indexing is
        required.

    Returns
    -------
    idx : tuple of ints
        Tuple of the same shape as `dims`, containing the unraveled index.

    Notes
    -----
    In the Examples section, since ``arr.flat[x] == arr.max()`` it may be
    easier to use flattened indexing than to re-map the index to a tuple.

    Examples
    --------
    >>> arr = np.arange(20).reshape(5, 4)
    >>> arr
    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11],
           [12, 13, 14, 15],
           [16, 17, 18, 19]])
    >>> x = arr.argmax()
    >>> x
    19
    >>> dims = arr.shape
    >>> idx = np.unravel_index(x, dims)
    >>> idx
    (4, 3)
    >>> arr[idx] == arr.max()
    True

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From Chris.Barker at noaa.gov  Mon Mar  8 14:58:08 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 08 Mar 2010 11:58:08 -0800
Subject: [SciPy-User] fromimage/imread segfaults on my images
In-Reply-To: <hn33vh$bgo$1@dough.gmane.org>
References: <hn33vh$bgo$1@dough.gmane.org>
Message-ID: <4B9556D0.9090409@noaa.gov>

Daniel Lenski wrote:
> I'm using SciPy to process some sparse binary images with a resolution of 
> about 5000x5000 pixels.  I try to load images like this:
> 
>   im = Image.open('/tmp/foo.pbm')
>   print "Image loaded, size is %s and mode is %s." % (im.size, im.mode)
>   arr = fromimage(im)
> 
> or
> 
>   arr = imread('/tmp/foo.pbm')

where do fromimage and imread come from? "namespaces are one honking 
great idea".

> In either case, I get a segfault.  The segfault is definitely in the 
> SciPy code, rather than the PIL code, since PIL's Image.open runs fine. 

don't be so sure -- I think PIL used lazy loading, so it doesn't 
actually read all the data with the open call, but rather when you try 
to do something with it.

I'd try making a few calls on the PIl image, and make sure it is what 
you expect. If it is, the easiest way to get it into a numpy array is:

np.asarray(pil_image)

do you have a smaller image in the same format you can experiment with? 
That might make it easier to figure out.

>  I don't know any other good, reliable way 
> to get a big image into an array :-(

if you really can't get PIL to work, pbm looks really simple:

http://netpbm.sourceforge.net/doc/pbm.html

read the header, then read the data with np.fromfile(), then convert to 
a uint8 array with np.unpackbits().

You could also take a look at MPL's imread() and see what it does -- I 
don't think MPL requires PIL, though I could be wrong.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From mudit_19a at yahoo.com  Mon Mar  8 15:05:59 2010
From: mudit_19a at yahoo.com (mudit sharma)
Date: Tue, 9 Mar 2010 01:35:59 +0530 (IST)
Subject: [SciPy-User] Finding coefficient
Message-ID: <803819.25205.qm@web94912.mail.in2.yahoo.com>


I have 2d array of observations  x1, x2, x3, x4 which I am looking to solve in such a way that z have a minimum value.

     ax1 + bx2 + cx3 + dx4 = s
     z = np.where(s < 0).shape[0]


I am currently using a brute force method by putting a constraint 0 > (a,b,c,d) < 20. I am looking for ideas to acheive this more efficiently.

Cheers,
M


      


From deil.christoph at googlemail.com  Mon Mar  8 16:47:51 2010
From: deil.christoph at googlemail.com (Christoph Deil)
Date: Mon, 8 Mar 2010 22:47:51 +0100
Subject: [SciPy-User] flat / nonflat array index conversion continued
Message-ID: <4D84E2BB-592B-4CAA-AC12-C87BB94CC95B@googlemail.com>

Sorry that I don't reply to the thread I started today (http://mail.scipy.org/pipermail/scipy-user/2010-March/024565.html), but I had the list in digest mode.
Anyways, Ryan suggested I use numpy.unravel_index, and indeed this does exactly what I wanted.

My question now is how I can vectorize numpy.unravel_index?
Here is what I tried:

idx = array([1,5]) # really I have 1e6 indices I want to unravel
unravel_index(idx,(2, 3)) # Doesn't work. Raises a ValueError
v_u_i = vectorize(unravel_index) # Try to vectorize it
v_u_i(idx,(2, 3)) # Doesn't work. Raises a TypeError

Thanks for you help!
--Christoph







From bsouthey at gmail.com  Mon Mar  8 17:29:18 2010
From: bsouthey at gmail.com (Bruce Southey)
Date: Mon, 08 Mar 2010 16:29:18 -0600
Subject: [SciPy-User] operations on large arrays
In-Reply-To: <77e831101003062205n22790adw6804401c40e3ee60@mail.gmail.com>
References: <77e831101003062122w2890311eh8a4fd856cbdc13b@mail.gmail.com>
	<77e831101003062205n22790adw6804401c40e3ee60@mail.gmail.com>
Message-ID: <4B957A3E.8030709@gmail.com>

On 03/07/2010 12:05 AM, Vincent Davis wrote:
> I just figured out that I had a few arrays that where taking up a 
> bunch of the memory. That said I still wonder if there is a better way.
>
> 	
> 	
>
> *Vincent Davis
> 720-301-3003 *
> vincent at vincentdavis.net <mailto:vincent at vincentdavis.net>
>
> my blog <http://vincentdavis.net> | LinkedIn 
> <http://www.linkedin.com/in/vincentdavis>
>
>
>
> On Sat, Mar 6, 2010 at 10:22 PM, Vincent Davis 
> <vincent at vincentdavis.net <mailto:vincent at vincentdavis.net>> wrote:
>
>     I have arrays of 8-20 rows and 230,000 column, all the data is float64
>     I what to be able to find the difference in the correlation matrix
>     between arrays
>     let A and B be of size (10, 230000)
>     np.corrcoef(a)-np.corrcoef(b)
>
>     I can't seem to do this with more than 10000 columns at a time
>     because of memory limitations. (about 9GB usable to python)
>     Is there a better way?
>
>     I also have problem finding the column means which is surprising
>     to me, I was not able to get the column means for 10000 columns,
>     but I can computer  the corrcoeff ?
>     np.mean(a, axis=0)
>
>     Do I just need to divide up the job or is there a better approach?
>
>     Thanks
>
>     	
>     	
>
>     *Vincent Davis
>     720-301-3003 *
>     vincent at vincentdavis.net <mailto:vincent at vincentdavis.net>
>
>     my blog <http://vincentdavis.net> | LinkedIn
>     <http://www.linkedin.com/in/vincentdavis>
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>    
Is there a better way to do what?
A problem with 'np.corrcoef(a)-np.corrcoef(b)' is that it is unclear 
what you want as if a and b have more than 1d then you get an array 
back. If the array is near zero then what does that mean? One 
interpretation that perhaps you should be seeing if these are the same 
array. If the array is not zero then what does that mean? Do you need to 
know which parts of a and b lead to different correlations?

You can always do np.corrcoef(a,b) such that the diagonal relating each 
column of a to each column of b is one.

Bruce





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100308/ecd375d8/attachment.html>

From dvanliere at gmail.com  Mon Mar  8 22:10:10 2010
From: dvanliere at gmail.com (Diederik van Liere)
Date: Mon, 8 Mar 2010 22:10:10 -0500
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools extension throws
	g++ internal error
Message-ID: <f99dcccd1003081910kc20c3b8ma6651b167b97ca5d@mail.gmail.com>

Dear scipy-user members,


I am trying to build Scipy for my 32bit Redhat server and I am running into
serious trouble, both with scipy 0.7.1 and the latest SVN version.

I get the following error:

building 'scipy.sparse.sparsetools._csr' extension
compiling C++ sources
C compiler: g++ -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-fPIC

creating build/temp.linux-i686-2.6/scipy/sparse/sparsetools
compile options:
'-I/usr/local/lib/python2.6/site-packages/numpy/core/include
-I/usr/local/include/python2.6 -c'
g++: scipy/sparse/sparsetools/csr_wrap.cxx
g++: Internal error: Terminated (program cc1plus)
Please submit a full bug report.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
g++: Internal error: Terminated (program cc1plus)
Please submit a full bug report.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
error: Command "g++ -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3
-Wall -fPIC -I/usr/local/lib/python2.6/site-packages/numpy/core/include
-I/usr/local/include/python2.6 -c scipy/sparse/sparsetools/csr_wrap.cxx -o
build/temp.linux-i686-2.6/scipy/sparse/sparsetools/csr_wrap.o" failed with
exit status 1

and I really don't know how to fix this. I have tried both Numpy 1.3.0 and
the latest Numpy SVN version but to no avail. I hope someone can point me in
the right direction.

Thanks so much,

Best regards,


Diederik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100308/a79401a2/attachment.html>

From david at silveregg.co.jp  Mon Mar  8 22:21:11 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Tue, 09 Mar 2010 12:21:11 +0900
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools extension
	throws g++ internal error
In-Reply-To: <f99dcccd1003081910kc20c3b8ma6651b167b97ca5d@mail.gmail.com>
References: <f99dcccd1003081910kc20c3b8ma6651b167b97ca5d@mail.gmail.com>
Message-ID: <4B95BEA7.7000603@silveregg.co.jp>

Diederik van Liere wrote:

> error: Command "g++ -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3
> -Wall -fPIC -I/usr/local/lib/python2.6/site-packages/numpy/core/include
> -I/usr/local/include/python2.6 -c scipy/sparse/sparsetools/csr_wrap.cxx -o
> build/temp.linux-i686-2.6/scipy/sparse/sparsetools/csr_wrap.o" failed with
> exit status 1

What's the version of g++ ?

> and I really don't know how to fix this. I have tried both Numpy 1.3.0 and
> the latest Numpy SVN version but to no avail. I hope someone can point me in
> the right direction.

Changing numpy version is unlikely to fix anything here, as the error is 
a crash from the C++ compiler.

One solution would be to update gcc (by compiling your own version), or 
more simply, if you don't need scipy.sparse to disable it in 
scipy/setup.py by commenting the sparse line.

David


From dvanliere at gmail.com  Mon Mar  8 22:29:42 2010
From: dvanliere at gmail.com (Diederik van Liere)
Date: Mon, 8 Mar 2010 22:29:42 -0500
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools extension throws
	g++ internal error
Message-ID: <f99dcccd1003081929v4923008ft4c21f3368ca507d5@mail.gmail.com>

> Diederik van Liere wrote:

>>* error: Command "g++ -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3*>>* -Wall -fPIC -I/usr/local/lib/python2.6/site-packages/numpy/core/include*>>* -I/usr/local/include/python2.6 -c scipy/sparse/sparsetools/csr_wrap.cxx -o*>>* build/temp.linux-i686-2.6/scipy/sparse/sparsetools/csr_wrap.o" failed with*>>* exit status 1*
> What's the version of g++ ?
I am using GCC 4.1.2 (32bit)

>>* and I really don't know how to fix this. I have tried both Numpy 1.3.0 and*>>* the latest Numpy SVN version but to no avail. I hope someone can point me in*>>* the right direction.*
> Changing numpy version is unlikely to fix anything here, as the error is
> a crash from the C++ compiler.
Okay, good to know.

> One solution would be to update gcc (by compiling your own version), or
> more simply, if you don't need scipy.sparse to disable it in
> scipy/setup.py by commenting the sparse line.
I don't think I will need this, this is for sparse matrix manipulation, right?

I need scipy.linalg although I prefer a complete working version of
Scipy because it might the case the sparse module is required by one
of other libraries I am suing.

But thanks David for giving me some workarounds.

> David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100308/6534368d/attachment.html>

From david at silveregg.co.jp  Mon Mar  8 22:38:25 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Tue, 09 Mar 2010 12:38:25 +0900
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools extension
	throws g++ internal error
In-Reply-To: <f99dcccd1003081929v4923008ft4c21f3368ca507d5@mail.gmail.com>
References: <f99dcccd1003081929v4923008ft4c21f3368ca507d5@mail.gmail.com>
Message-ID: <4B95C2B1.3050800@silveregg.co.jp>

Diederik van Liere wrote:

> I don't think I will need this, this is for sparse matrix manipulation, right?

Yes.


> I need scipy.linalg although I prefer a complete working version of
> Scipy because it might the case the sparse module is required by one
> of other libraries I am suing.

Indeed - not building scipy.sparse is clearly a temporary workaround, I 
would not advise to do this as a long term solution. Another obvious 
thing to try is building the scipy's trunk, or re-generating the c++ 
sources from swig. The latter, by generating a different file, may be 
able to exercise another codepath in g++, and avoid the crash depending 
on where the crash happens in g++.

Ideally, if you could find-out what part of the code causes the crash 
and report the bug, we may be able to implement a workaround in scipy.

cheers,

David


From dvanliere at gmail.com  Mon Mar  8 23:27:50 2010
From: dvanliere at gmail.com (Diederik van Liere)
Date: Mon, 8 Mar 2010 23:27:50 -0500
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools extension
	throws g++ internal error
Message-ID: <f99dcccd1003082027g247316a1s6682e3317eb5f24e@mail.gmail.com>

> Diederik van Liere wrote:

>>* I need scipy.linalg although I prefer a complete working version of*>>* Scipy because it might the case the sparse module is required by one*>>* of other libraries I am suing.*
> Indeed - not building scipy.sparse is clearly a temporary workaround, I
> would not advise to do this as a long term solution. Another obvious
> thing to try is building the scipy's trunk, or re-generating the c++
> sources from swig. The latter, by generating a different file, may be
> able to exercise another codepath in g++, and avoid the crash depending
> on where the crash happens in g++.

Thanks David, I disable the sparse package and scipy builds fine (I am
already using the latest SVN version of Scipy).

However,when I import scipy.linalg I receive the following error in Python:

ImportError: /usr/local/lib/python2.6/site-packages/scipy/linalg/clapack.so:
undefined symbol: clapack_sgesv

Which looks very similar to this bug report:
http://projects.scipy.org/scipy/ticket/1031

I would like to give you more detailed information but I am not sure
what you need / or how to provide that.

> Ideally, if you could find-out what part of the code causes the crash
> and report the bug, we may be able to implement a workaround in scipy.

> cheers,

> David

PS: I hope this reply does not start a new thread, don't know how to
prevent that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100308/cb22beec/attachment.html>

From david at silveregg.co.jp  Mon Mar  8 23:38:20 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Tue, 09 Mar 2010 13:38:20 +0900
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools
	extension	throws g++ internal error
In-Reply-To: <f99dcccd1003082027g247316a1s6682e3317eb5f24e@mail.gmail.com>
References: <f99dcccd1003082027g247316a1s6682e3317eb5f24e@mail.gmail.com>
Message-ID: <4B95D0BC.1060101@silveregg.co.jp>

Diederik van Liere wrote:

> 
> ImportError: /usr/local/lib/python2.6/site-packages/scipy/linalg/clapack.so:
> undefined symbol: clapack_sgesv

what is the output of python setup.py config inside scipy build ?

Also, the output of ldd on clapack.so may be useful.

David


From dvanliere at gmail.com  Tue Mar  9 00:06:49 2010
From: dvanliere at gmail.com (Diederik van Liere)
Date: Tue, 9 Mar 2010 00:06:49 -0500
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools extension
	throws g++ internal error
Message-ID: <f99dcccd1003082106k6798be49p29328850d6df1aa6@mail.gmail.com>

> Diederik van Liere wrote:

>>* *>>* ImportError: /usr/local/lib/python2.6/site-packages/scipy/linalg/clapack.so:*>>* undefined symbol: clapack_sgesv*
> what is the output of python setup.py config inside scipy build ?

I am not quite sure which output you mean, do you mean the console
output or a particular log file? I didn't see any log file in
particular.

The console output does not give any errors as far as I can tell.

Import scipy in python does work.

> Also, the output of ldd on clapack.so may be useful.
The output of ldd looks suspect (I compared it to other posts):

ldd /usr/local/lib/python2.6/site-packages/scipy/lib/lapack/clapack.so
libgfortran.so.1 => /usr/lib/libgfortran.so.1 (0x00b96000)
libm.so.6 => /lib/libm.so.6 (0x00f41000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00552000)
libc.so.6 => /lib/libc.so.6 (0x00110000)
/lib/ld-linux.so.2 (0x00e8f000)

However, Scipy does find the lapack, blas etc as this snippet from the log
shows:

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /usr/local/include
numpy.distutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/local/include']
    language = f77


How can I prevent starting a new thread every time?

Best,

Diederik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100309/2024b731/attachment.html>

From david at silveregg.co.jp  Tue Mar  9 00:15:46 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Tue, 09 Mar 2010 14:15:46 +0900
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools
	extension	throws g++ internal error
In-Reply-To: <f99dcccd1003082106k6798be49p29328850d6df1aa6@mail.gmail.com>
References: <f99dcccd1003082106k6798be49p29328850d6df1aa6@mail.gmail.com>
Message-ID: <4B95D982.90300@silveregg.co.jp>

Diederik van Liere wrote:
>> Diederik van Liere wrote:
> 
>>> * *>>* ImportError: /usr/local/lib/python2.6/site-packages/scipy/linalg/clapack.so:*>>* undefined symbol: clapack_sgesv*
>> what is the output of python setup.py config inside scipy build ?
> 
> I am not quite sure which output you mean, do you mean the console
> output or a particular log file? I didn't see any log file in
> particular.

I meant the console output, yes, to see which BLAS/LAPACK is detected.

> ldd /usr/local/lib/python2.6/site-packages/scipy/lib/lapack/clapack.so
> libgfortran.so.1 => /usr/lib/libgfortran.so.1 (0x00b96000)
> libm.so.6 => /lib/libm.so.6 (0x00f41000)
> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00552000)
> libc.so.6 => /lib/libc.so.6 (0x00110000)
> /lib/ld-linux.so.2 (0x00e8f000)

This is indeed suspicious. How did you build your Atlas ? It would be 
useful to have the full build output when building scipy.linalg.

> 
> However, Scipy does find the lapack, blas etc as this snippet from the log
> shows:
> 
> atlas_threads_info:
> Setting PTATLAS=ATLAS
>   libraries lapack_atlas not found in /usr/local/include
> numpy.distutils.system_info.atlas_threads_info
> Setting PTATLAS=ATLAS
> Setting PTATLAS=ATLAS
>   FOUND:
>     libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas']
>     library_dirs = ['/usr/local/include']
>     language = f77

I don't know if this linked to our issue, but this PTATLAS=ATLAS seems a 
bit strange, I don't remember having seen it.

> How can I prevent starting a new thread every time?

It is a setting in your email client - are you using email or something 
else to post ?

David


From millman at berkeley.edu  Tue Mar  9 00:29:28 2010
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 8 Mar 2010 21:29:28 -0800
Subject: [SciPy-User] PSF GSoC 2010 (Py3K focus)
In-Reply-To: <c7009a551003080044w656f4fc2o66fe9a417fc42247@mail.gmail.com>
References: <c7009a551003080044w656f4fc2o66fe9a417fc42247@mail.gmail.com>
Message-ID: <c7009a551003082129g69d89a31na4eb595743ec5156@mail.gmail.com>

I added Titus' email regarding the PSF's focus on Py3K-related
projects to our SoC ideas wiki page:
http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas

Given Titus' email, this is the most likely list of projects we will
get accepted this year:

- finish porting NumPy to Py3K
- port SciPy to Py3K
- port matplotlib to Py3K
- port ipython to Py3K

Given that we know what projects we will likely have accepted, it is
worth starting to flesh these proposals out in detail.  Also, we
should start discussing how we will choose which student's we want to
work on these ports.  In particular, we should list what skills and
background will be necessary to successfully complete these ports.

Thoughts? Ideas?

Best,
Jarrod


From bsouthey at gmail.com  Tue Mar  9 09:38:10 2010
From: bsouthey at gmail.com (Bruce Southey)
Date: Tue, 09 Mar 2010 08:38:10 -0600
Subject: [SciPy-User] scipy+lapack array copy
Message-ID: <4B965D52.9040708@gmail.com>

On 8. mars 2010, at 10.10, Skipper Seabold wrote:

 > On Mon, Mar 8, 2010 at 12:47 PM, Paul Anton Letnes
 > <paul.anton.letnes <at> gmail.com> wrote:
 >> Hi everyone.
 >>
 >> I have a numerical project where I link a fortran module to python 
using f2py. The fortran module
basically sets up the left hand side coefficient matrix of a linear 
equation system. The right hand side is
small, so I set that up in python. So far, everything works great.
 >>
 >> Now, when I call scipy-lapack, I am using:
 >> from scipy.linalg.lapack import flapack as lapack
 >> lu, piv, x, info = lapack.zgesv(A, b)
 >> A is created in the fortran part of the program. b is created from 
numpy.zeros(). Both report as
F_CONTIGUOUS by printing A.flags and b.flags.
 >>
 >> The problem is that I'd like A to be large (>1GB), but the lapack 
wrapper (or lapack itself?) copies the
array, doubling the memory use. Aside from being unnecessary, this also 
reduces the practical problem
size I can solve by a factor 2, which of course is very annoying.
 >>
 >> How can I get around this problem?
 >>

If memory usage is a concern then you can directly form the X.T*X matrix 
rather than doing the transpose and multiplication. The advantage is 
that X.T*X has a fixed dimensions regardless of the number of 
observations. Further, you can exploit symmetry and use half-storage 
(there are lapack routines). The disadvantage is that is more complex 
and probably not as fast as doing the multiplication.  If numerical 
stability is a concern then you probably need to use a better solving 
than LU with partial pivoting anyhow.

Bruce


From dlenski at gmail.com  Tue Mar  9 01:35:09 2010
From: dlenski at gmail.com (Daniel Lenski)
Date: Tue, 9 Mar 2010 06:35:09 +0000 (UTC)
Subject: [SciPy-User] fromimage/imread segfaults on my images
References: <hn33vh$bgo$1@dough.gmane.org> <4B9556D0.9090409@noaa.gov>
Message-ID: <hn4q6t$ib0$1@dough.gmane.org>

On Mon, 08 Mar 2010 11:58:08 -0800, Christopher Barker wrote:

> Daniel Lenski wrote:
>> I'm using SciPy to process some sparse binary images with a resolution
>> of about 5000x5000 pixels.  I try to load images like this:
>> 
>>   im = Image.open('/tmp/foo.pbm')
>>   print "Image loaded, size is %s and mode is %s." % (im.size, im.mode)
>>   arr = fromimage(im)
>> 
>> or
>> 
>>   arr = imread('/tmp/foo.pbm')
> 
> where do fromimage and imread come from? "namespaces are one honking
> great idea".

Whoops, sorry.  Those are from scipy.misc.

>> In either case, I get a segfault.  The segfault is definitely in the
>> SciPy code, rather than the PIL code, since PIL's Image.open runs fine.
> 
> don't be so sure -- I think PIL used lazy loading, so it doesn't
> actually read all the data with the open call, but rather when you try
> to do something with it.
> 
> I'd try making a few calls on the PIl image, and make sure it is what
> you expect. If it is, the easiest way to get it into a numpy array is:

Everything works fine as long as I stick with PIL calls only... I can 
even dump the entire contents of the image to a string with 
PIL.Image.fromstring().  No problems there.

> np.asarray(pil_image)

I looked at the code for scipy.misc.fromimage, and this is basically all 
it does.  This is where the segfault occurs.

> do you have a smaller image in the same format you can experiment with?
> That might make it easier to figure out.

Smaller images work fine.  I haven't figured out exactly where the cutoff 
is, but 500x500 works fine, for instance.

>>  I don't know any other good, reliable way
>> to get a big image into an array :-(
> 
> if you really can't get PIL to work, pbm looks really simple:
> 
> http://netpbm.sourceforge.net/doc/pbm.html
> 
> read the header, then read the data with np.fromfile(), then convert to
> a uint8 array with np.unpackbits().

That's what I've ended up doing... rolled my own loadpbm() function.

> You could also take a look at MPL's imread() and see what it does -- I
> don't think MPL requires PIL, though I could be wrong.

MPL requires PIL to load and save images in formats /other than/ PNG.

Dan




From dlenski at gmail.com  Mon Mar  8 22:26:14 2010
From: dlenski at gmail.com (Daniel Lenski)
Date: Tue, 9 Mar 2010 03:26:14 +0000 (UTC)
Subject: [SciPy-User] fromimage/imread segfaults on my images
References: <hn33vh$bgo$1@dough.gmane.org> <4B9556D0.9090409@noaa.gov>
Message-ID: <hn4f4l$st9$1@dough.gmane.org>

On Mon, 08 Mar 2010 11:58:08 -0800, Christopher Barker wrote:

> Daniel Lenski wrote:
>> I'm using SciPy to process some sparse binary images with a resolution
>> of about 5000x5000 pixels.  I try to load images like this:
>> 
>>   im = Image.open('/tmp/foo.pbm')
>>   print "Image loaded, size is %s and mode is %s." % (im.size, im.mode)
>>   arr = fromimage(im)
>> 
>> or
>> 
>>   arr = imread('/tmp/foo.pbm')
> 
> where do fromimage and imread come from? "namespaces are one honking
> great idea".

Sorry! That'd be

  from scipy.misc import fromimage, imread

> don't be so sure -- I think PIL used lazy loading, so it doesn't
> actually read all the data with the open call, but rather when you try
> to do something with it.
> 
> I'd try making a few calls on the PIl image, and make sure it is what
> you expect. If it is, the easiest way to get it into a numpy array is:

True, PIL lazy-loads the image.  However, I've checked this, and I'm able 
to manipulate the image via PIL with no problems.

> np.asarray(pil_image)

Yeah, I looked at the scipy.misc.fromimage code and this is all it does, 
basically.

> do you have a smaller image in the same format you can experiment with?
> That might make it easier to figure out.

Yes, smaller images in the same format work fine!

>>  I don't know any other good, reliable way
>> to get a big image into an array :-(
> 
> if you really can't get PIL to work, pbm looks really simple:
> 
> http://netpbm.sourceforge.net/doc/pbm.html
> 
> read the header, then read the data with np.fromfile(), then convert to
> a uint8 array with np.unpackbits().

That's what I've ended up doing...
 
> You could also take a look at MPL's imread() and see what it does -- I
> don't think MPL requires PIL, though I could be wrong.

MPL does require PIL for formats other than PNG.

Dan




From e.antero.tammi at gmail.com  Tue Mar  9 14:51:38 2010
From: e.antero.tammi at gmail.com (eat)
Date: Tue, 9 Mar 2010 19:51:38 +0000 (UTC)
Subject: [SciPy-User] <transferred from Scipy-Dev>Re: Seeking help/ advice
	for applying functions
Message-ID: <loom.20100309T204105-840@post.gmane.org>

> 
> On 9 March 2010 09:59, eat <e.antero.tammi <at> gmail.com> wrote:
> > Robert Kern <robert.kern <at> gmail.com> writes:
> >
> >>
> >> Your example is not very clear. Can you write a less cryptic one with
> >> informative variable names and perhaps some comments about what each
> >> part is doing?
> >>
> >
Anne Archibald <peridot.faceted <at> gmail.com> writes:

Hi,

I moved this thread from scipy-dev to scipy-user.

First of all, Thanks Anne you clarified a lot.


> > """
> > Hi,
> >
> > I have tried to clarify my code. First part is the relevant one, the rest 
is
> > there just to provide some context to run some tests.
> 
> The short answer is that no, there's no way to optimize what you're doing.
> 
> The long answer is: when numpy and scipy are fast, they are fast
> because they avoid running python code: if you add two arrays, there's
> only one line of python code, and all the work is done by loops
> written in C. If your code is calling many different python functions,
> well, since they're python functions, to apply them at all you must
> necessarily execute python code. There goes any potential speed
> advantage. (There may be a convenience advantage; if so, you can look
> into using np.vectorize, which is just a wrapper around a python loop,
> but is convenient.)

Not only that I'm new to Scipy/ Numpy but I'm new to python aswell.
I think I assumed that once python functions get compiled they'll would be
(almost) as efficient than builtins.

> 
> That said, I assume you are considering numpy/scipy because you have
> arrays of thousands or more. It also seems unlikely that you actually
> have thousands of different functions (that's an awful lot of source
> code!). So if your "different" functions are actually just a handful
> (or fewer) pieces of actual code, and you are getting your thousands
> of functions by wrapping them up with parameters and local variables,
> well, now there are possibilities. Exactly what possibilities depend
> on what your functions look like - which is one reason Robert Kern
> asked you to clarify your code - but they all boil down to rearranging
> the problem so that it goes back to "few functions, much data", then
> writing the functions in such a way that you can use numpy to apply
> them to thousands or millions of data points at once.

Yes, indeed I don't have so many different 'base' functions, but just
configured (parametrized) different ways. However there are situations
when it's advantage to be able to treat your fuctions as 'black boxes'.
For example when parametrization is based on optimization.

Yes, I'll also recognize that my original question was too generic. I'll
try to be more specific next time.

> 
> > Also originally I should have posted this to Scipy-User list. Would it be
> > more appropriate to continue the discussion there?
> 
> Probably.
> 
> Anne

Thanks a lot,
eat

> 
> > Regards,
> > eat
> > """
> >
> > import numpy as np
> >
> > ## relevant part
are in scipy-dev



From robert.kern at gmail.com  Tue Mar  9 14:59:15 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 9 Mar 2010 13:59:15 -0600
Subject: [SciPy-User] <transferred from Scipy-Dev>Re: Seeking help/
	advice for applying functions
In-Reply-To: <loom.20100309T204105-840@post.gmane.org>
References: <loom.20100309T204105-840@post.gmane.org>
Message-ID: <3d375d731003091159u4151c737j65f4b2f2a7ca945@mail.gmail.com>

On Tue, Mar 9, 2010 at 13:51, eat <e.antero.tammi at gmail.com> wrote:
> Not only that I'm new to Scipy/ Numpy but I'm new to python aswell.
> I think I assumed that once python functions get compiled they'll would be
> (almost) as efficient than builtins.

No, not at all. Python functions do not get compiled to machine code.
They get compiled to Python bytecode, but that Python bytecode is
interpreted. Python functions have tons of overhead compared to
builtins.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From peridot.faceted at gmail.com  Tue Mar  9 15:11:55 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 9 Mar 2010 15:11:55 -0500
Subject: [SciPy-User] <transferred from Scipy-Dev>Re: Seeking help/
	advice for applying functions
In-Reply-To: <loom.20100309T204105-840@post.gmane.org>
References: <loom.20100309T204105-840@post.gmane.org>
Message-ID: <ce557a361003091211q7166d9afke2d633b59e518dec@mail.gmail.com>

On 9 March 2010 14:51, eat <e.antero.tammi at gmail.com> wrote:
>>
>> On 9 March 2010 09:59, eat <e.antero.tammi <at> gmail.com> wrote:
>> > Robert Kern <robert.kern <at> gmail.com> writes:
>> >
>> >>
>> >> Your example is not very clear. Can you write a less cryptic one with
>> >> informative variable names and perhaps some comments about what each
>> >> part is doing?
>> >>
>> >
> Anne Archibald <peridot.faceted <at> gmail.com> writes:
>
> Hi,
>
> I moved this thread from scipy-dev to scipy-user.
>
> First of all, Thanks Anne you clarified a lot.
>
>
>> > """
>> > Hi,
>> >
>> > I have tried to clarify my code. First part is the relevant one, the rest
> is
>> > there just to provide some context to run some tests.
>>
>> The short answer is that no, there's no way to optimize what you're doing.
>>
>> The long answer is: when numpy and scipy are fast, they are fast
>> because they avoid running python code: if you add two arrays, there's
>> only one line of python code, and all the work is done by loops
>> written in C. If your code is calling many different python functions,
>> well, since they're python functions, to apply them at all you must
>> necessarily execute python code. There goes any potential speed
>> advantage. (There may be a convenience advantage; if so, you can look
>> into using np.vectorize, which is just a wrapper around a python loop,
>> but is convenient.)
>
> Not only that I'm new to Scipy/ Numpy but I'm new to python aswell.
> I think I assumed that once python functions get compiled they'll would be
> (almost) as efficient than builtins.

Not especially. They are compiled to bytecode, whose execution is not
particularly fast. But the big problem is all the baggage of python's
nature as a dynamic language: for example each value is allocated with
malloc() and contains type information; for another example, each
access to a list involves identifying that the object really is a
list, dispatching to the list-lookup function, determining the type of
the argument (integer, slice object, long integer, other), and bounds
checking before finally returning the list element. Thus even tools
like cython that let you write, effectively, python code that gets
compiled to machine code are not much faster unless you can turn off
the dynamic features of python (which cython lets you do, selectively;
it's great).

>>
>> That said, I assume you are considering numpy/scipy because you have
>> arrays of thousands or more. It also seems unlikely that you actually
>> have thousands of different functions (that's an awful lot of source
>> code!). So if your "different" functions are actually just a handful
>> (or fewer) pieces of actual code, and you are getting your thousands
>> of functions by wrapping them up with parameters and local variables,
>> well, now there are possibilities. Exactly what possibilities depend
>> on what your functions look like - which is one reason Robert Kern
>> asked you to clarify your code - but they all boil down to rearranging
>> the problem so that it goes back to "few functions, much data", then
>> writing the functions in such a way that you can use numpy to apply
>> them to thousands or millions of data points at once.
>
> Yes, indeed I don't have so many different 'base' functions, but just
> configured (parametrized) different ways. However there are situations
> when it's advantage to be able to treat your fuctions as 'black boxes'.
> For example when parametrization is based on optimization.

Think about whether you can write each 'base' function to take arrays
as arguments, for example:

def F(x, mu, sigma):
    return np.exp(-((x-mu)/sigma)**2)

If your current code does something like

fis = [lambda x: F(x, mui, sigmai) for (mui, sigmai) in zip(muis, sigmais)]

r = [f(7) for f in fis]

you can rewrite it as the single line

r = F(7, muis, sigmais)

(if muis and sigmais are numpy arrays). Now you have just a couple of
lines of python, and the heavy lifting all happens inside numpy loops.

If you have several different functions, look into separating your
input arrays based on which function needs to be applied to them;
remember numpy lets you select out all the elements of an array
meeting a criterion.

I realize this kind of rewriting will mess up a nice clean functional
design, but as is often the case, that is the price you pay for
performance. If your code works and is fast enough, I recommend
leaving it as is.

Anne


From josef.pktd at gmail.com  Tue Mar  9 16:01:31 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 9 Mar 2010 16:01:31 -0500
Subject: [SciPy-User] return "full_output" or how to stop throwing away
	already calculated results
Message-ID: <1cd32cbb1003091301m7fdd82e5je2cfcd6cf292d0ba@mail.gmail.com>

scipy functions throw away a lot of intermediate results

examples

fmin_slsqp ticket:1129
scipy.linalg.lstsq  throws away residuals :

    if n<m:
        x1 = x[:n]
        if rank==n: resids = sum(x[n:]**2,axis=0)
        x = x1
    return x,resids,rank,s

in statsmodels, I am switching more to the pattern when
full_output=True (or similar keyword) for a function, then all
intermediate results are returned, attached to a generic class
instance ( or could be a struct in matlab or a bunch for Gael,...)

Is it possible to start the same pattern in scipy?

E.g. for optimizers like slsqp, if we want to change the returns then
each time it breaks the API or we have to increase full_output= 1,2,3
( like nargout in matlab)
(ugly)

I saw linalg.lstsq by chance this morning, but I assume there is
additional useful information hidden in other functions.

Any opinion ?

Josef


From agile.aspect at gmail.com  Tue Mar  9 16:34:39 2010
From: agile.aspect at gmail.com (Agile Aspect)
Date: Tue, 9 Mar 2010 13:34:39 -0800
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools extension
	throws g++ internal error
In-Reply-To: <4B95D982.90300@silveregg.co.jp>
References: <f99dcccd1003082106k6798be49p29328850d6df1aa6@mail.gmail.com>
	<4B95D982.90300@silveregg.co.jp>
Message-ID: <c6a6f63b1003091334w159a8172j2e4438b882ca1348@mail.gmail.com>

On Mon, Mar 8, 2010 at 9:15 PM, David Cournapeau <david at silveregg.co.jp> wrote:
> Diederik van Liere wrote:
>>> Diederik van Liere wrote:
>>
>>>> * *>>* ImportError: /usr/local/lib/python2.6/site-packages/scipy/linalg/clapack.so:*>>* undefined symbol: clapack_sgesv*
>>> what is the output of python setup.py config inside scipy build ?
>>
>> I am not quite sure which output you mean, do you mean the console
>> output or a particular log file? I didn't see any log file in
>> particular.
>
> I meant the console output, yes, to see which BLAS/LAPACK is detected.
>
>> ldd /usr/local/lib/python2.6/site-packages/scipy/lib/lapack/clapack.so
>> libgfortran.so.1 => /usr/lib/libgfortran.so.1 (0x00b96000)
>> libm.so.6 => /lib/libm.so.6 (0x00f41000)
>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00552000)
>> libc.so.6 => /lib/libc.so.6 (0x00110000)
>> /lib/ld-linux.so.2 (0x00e8f000)
>
> This is indeed suspicious. How did you build your Atlas ? It would be
> useful to have the full build output when building scipy.linalg.
>
>>
>> However, Scipy does find the lapack, blas etc as this snippet from the log
>> shows:
>>
>> atlas_threads_info:
>> Setting PTATLAS=ATLAS
>> ? libraries lapack_atlas not found in /usr/local/include
>> numpy.distutils.system_info.atlas_threads_info
>> Setting PTATLAS=ATLAS
>> Setting PTATLAS=ATLAS
>> ? FOUND:
>> ? ? libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas']
>> ? ? library_dirs = ['/usr/local/include']
>> ? ? language = f77
>

You should post your site.cfg which is under

  /usr/local/lib/python2.6/site-packages/numpy/distutils/site.cfg


-- 
      Enjoy global warming while it lasts.


From e.antero.tammi at gmail.com  Tue Mar  9 16:50:43 2010
From: e.antero.tammi at gmail.com (eat)
Date: Tue, 9 Mar 2010 21:50:43 +0000 (UTC)
Subject: [SciPy-User]
	=?utf-8?q?=26lt=3Btransferred_from_Scipy-Dev=26gt=3B?=
	=?utf-8?q?Re=3A_Seeking_help/=09advice_for_applying_functions?=
References: <loom.20100309T204105-840@post.gmane.org>
	<3d375d731003091159u4151c737j65f4b2f2a7ca945@mail.gmail.com>
Message-ID: <loom.20100309T224823-787@post.gmane.org>

Robert Kern <robert.kern <at> gmail.com> writes:

> 
> On Tue, Mar 9, 2010 at 13:51, eat <e.antero.tammi <at> gmail.com> wrote:
> > Not only that I'm new to Scipy/ Numpy but I'm new to python aswell.
> > I think I assumed that once python functions get compiled they'll would be
> > (almost) as efficient than builtins.
> 
> No, not at all. Python functions do not get compiled to machine code.
> They get compiled to Python bytecode, but that Python bytecode is
> interpreted. Python functions have tons of overhead compared to
> builtins.

Aha, so python does not have a mechanism to optimize bytecode (nor higher 
level code) inorder to get rid off the dynamic nature of code when needed.

Well, I'll be better to go back to the drawing board then ;-)


Thanks,
eat







From e.antero.tammi at gmail.com  Tue Mar  9 16:53:35 2010
From: e.antero.tammi at gmail.com (eat)
Date: Tue, 9 Mar 2010 21:53:35 +0000 (UTC)
Subject: [SciPy-User]
	=?utf-8?q?=26lt=3Btransferred_from_Scipy-Dev=26gt=3B?=
	=?utf-8?q?Re=3A_Seeking_help/=09advice_for_applying_functions?=
References: <loom.20100309T204105-840@post.gmane.org>
	<ce557a361003091211q7166d9afke2d633b59e518dec@mail.gmail.com>
Message-ID: <loom.20100309T221226-404@post.gmane.org>

Anne Archibald <peridot.faceted <at> gmail.com> writes:

> Not especially. They are compiled to bytecode, whose execution is not
> particularly fast. But the big problem is all the baggage of python's
> nature as a dynamic language: for example each value is allocated with
> malloc() and contains type information; for another example, each
> access to a list involves identifying that the object really is a
> list, dispatching to the list-lookup function, determining the type of
> the argument (integer, slice object, long integer, other), and bounds
> checking before finally returning the list element. Thus even tools
> like cython that let you write, effectively, python code that gets
> compiled to machine code are not much faster unless you can turn off
> the dynamic features of python (which cython lets you do, selectively;
> it's great).

Good to know, I'll better postpone studying cython while I'm still
learning 'the very basics' of Scipy/ Numpy.

> Think about whether you can write each 'base' function to take arrays
> as arguments, for example:
> 
> def F(x, mu, sigma):
>     return np.exp(-((x-mu)/sigma)**2)
> 
> If your current code does something like
> 
> fis = [lambda x: F(x, mui, sigmai) for (mui, sigmai) in zip(muis, sigmais)]
> 
> r = [f(7) for f in fis]
> 
> you can rewrite it as the single line
> 
> r = F(7, muis, sigmais)
> 
> (if muis and sigmais are numpy arrays). Now you have just a couple of
> lines of python, and the heavy lifting all happens inside numpy loops.

Good quidelines in general to drive ones design!

> If you have several different functions, look into separating your
> input arrays based on which function needs to be applied to them;
                                  
Hmm, but I think that's what not possible in my particular case! That's
depends on what happens externally. I don't have any control of it.

> remember numpy lets you select out all the elements of an array
> meeting a criterion.
                                  
As far as I understod, select and where gives you only the first one.
In my case I need all of them.

> I realize this kind of rewriting will mess up a nice clean functional
> design, but as is often the case, that is the price you pay for
> performance. If your code works and is fast enough, I recommend
> leaving it as is.
                                  
True, I'll better figure out my toughts more clearly in higher level and
worry the performance later (if ever).
                                  
 
> Anne

Thanks a lot,
eat







From peridot.faceted at gmail.com  Tue Mar  9 16:56:33 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 9 Mar 2010 16:56:33 -0500
Subject: [SciPy-User] &lt; transferred from Scipy-Dev&gt;
	Re: Seeking help/ 	advice for applying functions
In-Reply-To: <loom.20100309T224823-787@post.gmane.org>
References: <loom.20100309T204105-840@post.gmane.org>
	<3d375d731003091159u4151c737j65f4b2f2a7ca945@mail.gmail.com>
	<loom.20100309T224823-787@post.gmane.org>
Message-ID: <ce557a361003091356j4720d919o230c6de32925f763@mail.gmail.com>

On 9 March 2010 16:50, eat <e.antero.tammi at gmail.com> wrote:
> Robert Kern <robert.kern <at> gmail.com> writes:
>
>>
>> On Tue, Mar 9, 2010 at 13:51, eat <e.antero.tammi <at> gmail.com> wrote:
>> > Not only that I'm new to Scipy/ Numpy but I'm new to python aswell.
>> > I think I assumed that once python functions get compiled they'll would be
>> > (almost) as efficient than builtins.
>>
>> No, not at all. Python functions do not get compiled to machine code.
>> They get compiled to Python bytecode, but that Python bytecode is
>> interpreted. Python functions have tons of overhead compared to
>> builtins.
>
> Aha, so python does not have a mechanism to optimize bytecode (nor higher
> level code) inorder to get rid off the dynamic nature of code when needed.

This kind of optimization is extremely difficult; it involves, for
example, inferring that a particular variable will always point to an
integer value, so that the type information attached to the integer
can be discarded. There are languages/programming environments that
are capable of this, but it is a difficult and expensive operation.
It's also unlikely to be productive for python, which is very dynamic,
as languages go: the value "None" is often used to signify "no answer
yet" in variables that would otherwise point to only one type of
value, and it's perfectly acceptable to change what function
"mymodule.myfunction" points to at any point.

Anne


From peridot.faceted at gmail.com  Tue Mar  9 16:59:41 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 9 Mar 2010 16:59:41 -0500
Subject: [SciPy-User] &lt; transferred from Scipy-Dev&gt;
	Re: Seeking help/ 	advice for applying functions
In-Reply-To: <loom.20100309T221226-404@post.gmane.org>
References: <loom.20100309T204105-840@post.gmane.org>
	<ce557a361003091211q7166d9afke2d633b59e518dec@mail.gmail.com>
	<loom.20100309T221226-404@post.gmane.org>
Message-ID: <ce557a361003091359r176adb42h2dee7bd1a5facf66@mail.gmail.com>

On 9 March 2010 16:53, eat <e.antero.tammi at gmail.com> wrote:
> Anne Archibald <peridot.faceted <at> gmail.com> writes:
>
>> Not especially. They are compiled to bytecode, whose execution is not
>> particularly fast. But the big problem is all the baggage of python's
>> nature as a dynamic language: for example each value is allocated with
>> malloc() and contains type information; for another example, each
>> access to a list involves identifying that the object really is a
>> list, dispatching to the list-lookup function, determining the type of
>> the argument (integer, slice object, long integer, other), and bounds
>> checking before finally returning the list element. Thus even tools
>> like cython that let you write, effectively, python code that gets
>> compiled to machine code are not much faster unless you can turn off
>> the dynamic features of python (which cython lets you do, selectively;
>> it's great).
>
> Good to know, I'll better postpone studying cython while I'm still
> learning 'the very basics' of Scipy/ Numpy.
>
>> Think about whether you can write each 'base' function to take arrays
>> as arguments, for example:
>>
>> def F(x, mu, sigma):
>>     return np.exp(-((x-mu)/sigma)**2)
>>
>> If your current code does something like
>>
>> fis = [lambda x: F(x, mui, sigmai) for (mui, sigmai) in zip(muis, sigmais)]
>>
>> r = [f(7) for f in fis]
>>
>> you can rewrite it as the single line
>>
>> r = F(7, muis, sigmais)
>>
>> (if muis and sigmais are numpy arrays). Now you have just a couple of
>> lines of python, and the heavy lifting all happens inside numpy loops.
>
> Good quidelines in general to drive ones design!
>
>> If you have several different functions, look into separating your
>> input arrays based on which function needs to be applied to them;
>
> Hmm, but I think that's what not possible in my particular case! That's
> depends on what happens externally. I don't have any control of it.

What I mean is something like:

A[A>3] = F(A[A>3])
A[A<=3] = G(A[A<=3])

Not that this is the most efficient way to write this sort of thing,
but it is a way to apply the appropriate function to each element in
the array.

>> remember numpy lets you select out all the elements of an array
>> meeting a criterion.
>
> As far as I understod, select and where gives you only the first one.
> In my case I need all of them.

There are many different functions for picking just some elements out
of an array, and for putting results back in the correct places. This
kind of operation - pull out certain elements, do some operation to
them, put the new values back in place of the old - happens all the
time, and numpy has all sorts of tools to support it.

Anne


From e.antero.tammi at gmail.com  Tue Mar  9 18:06:57 2010
From: e.antero.tammi at gmail.com (eat)
Date: Tue, 9 Mar 2010 23:06:57 +0000 (UTC)
Subject: [SciPy-User]
	=?utf-8?q?=26amp=3Blt=3B_transferred_from_Scipy-Dev?=
	=?utf-8?q?=26amp=3Bgt=3B=09Re=3A_Seeking_help/_=09advice_for_apply?=
	=?utf-8?q?ing_functions?=
References: <loom.20100309T204105-840@post.gmane.org>
	<3d375d731003091159u4151c737j65f4b2f2a7ca945@mail.gmail.com>
	<loom.20100309T224823-787@post.gmane.org>
	<ce557a361003091356j4720d919o230c6de32925f763@mail.gmail.com>
Message-ID: <loom.20100310T000620-731@post.gmane.org>

Anne Archibald <peridot.faceted <at> gmail.com> writes:

> This kind of optimization is extremely difficult; it involves, for
> example, inferring that a particular variable will always point to an
> integer value, so that the type information attached to the integer
> can be discarded. There are languages/programming environments that
> are capable of this, but it is a difficult and expensive operation.
> It's also unlikely to be productive for python, which is very dynamic,
> as languages go: the value "None" is often used to signify "no answer
> yet" in variables that would otherwise point to only one type of
> value, and it's perfectly acceptable to change what function
> "mymodule.myfunction" points to at any point.
> 
OK. I kind of understand, but wouldn't it possible to somehow sidetrack this 
whole dynamic nature of python (when really needed)? Perhaps giving some kind 
of exclusive gateway to the underlying libraries. Yes, I'm considering here 
only the mature libraries, where no API changes are expected.

Anyway this is out of my competence, but I'll be back at the other thread ;-)

> Anne
> 
Regards,
eat



From e.antero.tammi at gmail.com  Tue Mar  9 18:09:32 2010
From: e.antero.tammi at gmail.com (eat)
Date: Tue, 9 Mar 2010 23:09:32 +0000 (UTC)
Subject: [SciPy-User]
	=?utf-8?q?=26amp=3Blt=3B_transferred_from_Scipy-Dev?=
	=?utf-8?q?=26amp=3Bgt=3B=09Re=3A_Seeking_help/_=09advice_for_apply?=
	=?utf-8?q?ing_functions?=
References: <loom.20100309T204105-840@post.gmane.org>
	<ce557a361003091211q7166d9afke2d633b59e518dec@mail.gmail.com>
	<loom.20100309T221226-404@post.gmane.org>
	<ce557a361003091359r176adb42h2dee7bd1a5facf66@mail.gmail.com>
Message-ID: <loom.20100310T000847-886@post.gmane.org>

Anne Archibald <peridot.faceted <at> gmail.com> writes:

> > Hmm, but I think that's what not possible in my particular case! That's
> > depends on what happens externally. I don't have any control of it.
> 
> What I mean is something like:
> 
> A[A>3] = F(A[A>3])
> A[A<=3] = G(A[A<=3])

But ah, this looks so much what I'm looking for. Let me still digest this for 
a while (please be patient, while I'm groking this. The A[A>3] part is very 
clear, but I still have to confirm my self that F(A[A>3]) is really 
meaningfull. In my particular case A[A>3]= is not relevant).

> 
> Not that this is the most efficient way to write this sort of thing,
> but it is a way to apply the appropriate function to each element in
> the array.
> 
> >> remember numpy lets you select out all the elements of an array
> >> meeting a criterion.
> >
> > As far as I understod, select and where gives you only the first one.
> > In my case I need all of them.
> 
> There are many different functions for picking just some elements out
> of an array, and for putting results back in the correct places. This
> kind of operation - pull out certain elements, do some operation to
> them, put the new values back in place of the old - happens all the
> time, and numpy has all sorts of tools to support it.

I do understand, it's not the language per se, but the fluid browsing of libs 
and the ability to utilize them. You have allready guided me a lot.

> 
> Anne
> 
Thanks,
eat




From PHobson at Geosyntec.com  Tue Mar  9 19:08:05 2010
From: PHobson at Geosyntec.com (PHobson at Geosyntec.com)
Date: Tue, 9 Mar 2010 19:08:05 -0500
Subject: [SciPy-User] First attempt to import scipy.stats fails
Message-ID: <C97487CD346EEA4E80423F51F6D1EB2FA9769397ED@EXCHANGE-00.GeoSyntec.net>

But the second one works fine -- so it's not really an issue that bothers me, but I thought it might interest some folks. Here's an IPython session that demonstrates the issue:

Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)]
...
IPython 0.10 -- An enhanced Interactive Python.
...
In [1]: import numpy as np

In [2]: import scipy as sp

In [3]: import scipy.stats as st
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

c:\stuff\utils\py\<ipython console> in <module>()

C:\Python26\lib\site-packages\scipy\stats\__init__.py in <module>()
      5 from info import __doc__
      6
----> 7 from stats import *
      8 from distributions import *
      9 from rv import *

C:\Python26\lib\site-packages\scipy\stats\stats.py in <module>()
    201
    202 #import scipy.stats  #is this a circular import ?

--> 203 from morestats import find_repeats #is only reference to scipy.stats
    204 import distributions
    205

C:\Python26\lib\site-packages\scipy\stats\morestats.py in <module>()
      5 import statlib
      6 import stats
----> 7 import distributions
      8 from numpy import isscalar, r_, log, sum, around, unique, asarray
      9 from numpy import zeros, arange, sort, amin, amax, any, where, \

C:\Python26\lib\site-packages\scipy\stats\distributions.py in <module>()
     25 from scipy.special import gammaln as gamln
     26 from copy import copy
---> 27 import vonmises_cython
     28 import textwrap
     29

c:\stuff\utils\py\numpy.pxd in scipy.stats.vonmises_cython (scipy\stats\vonmises_cython.c:2939)()

ValueError: numpy.dtype does not appear to be the correct type object

In [4]: import scipy.stats as st

In [5]: np.__version__
Out[5]: '1.4.0'

In [6]: sp.__version__
Out[6]: '0.7.1'

Paul M. Hobson                  
Senior Staff Engineer
-- 
Geosyntec Consultants 
55 SW Yamhill St, Ste 200
Portland, OR 97204
Phone: 503.222.9518
www.geosyntec.com




From josef.pktd at gmail.com  Tue Mar  9 20:03:14 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 9 Mar 2010 20:03:14 -0500
Subject: [SciPy-User] First attempt to import scipy.stats fails
In-Reply-To: <C97487CD346EEA4E80423F51F6D1EB2FA9769397ED@EXCHANGE-00.GeoSyntec.net>
References: <C97487CD346EEA4E80423F51F6D1EB2FA9769397ED@EXCHANGE-00.GeoSyntec.net>
Message-ID: <1cd32cbb1003091703m6304694ei510d8db0e61fdfc2@mail.gmail.com>

On Tue, Mar 9, 2010 at 7:08 PM,  <PHobson at geosyntec.com> wrote:
> But the second one works fine -- so it's not really an issue that bothers me, but I thought it might interest some folks. Here's an IPython session that demonstrates the issue:
>
> Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)]
> ...
> IPython 0.10 -- An enhanced Interactive Python.
> ...
> In [1]: import numpy as np
>
> In [2]: import scipy as sp
>
> In [3]: import scipy.stats as st
> ---------------------------------------------------------------------------
> ValueError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call last)
>
> c:\stuff\utils\py\<ipython console> in <module>()
>
> C:\Python26\lib\site-packages\scipy\stats\__init__.py in <module>()
> ? ? ?5 from info import __doc__
> ? ? ?6
> ----> 7 from stats import *
> ? ? ?8 from distributions import *
> ? ? ?9 from rv import *
>
> C:\Python26\lib\site-packages\scipy\stats\stats.py in <module>()
> ? ?201
> ? ?202 #import scipy.stats ?#is this a circular import ?
>
> --> 203 from morestats import find_repeats #is only reference to scipy.stats
> ? ?204 import distributions
> ? ?205
>
> C:\Python26\lib\site-packages\scipy\stats\morestats.py in <module>()
> ? ? ?5 import statlib
> ? ? ?6 import stats
> ----> 7 import distributions
> ? ? ?8 from numpy import isscalar, r_, log, sum, around, unique, asarray
> ? ? ?9 from numpy import zeros, arange, sort, amin, amax, any, where, \
>
> C:\Python26\lib\site-packages\scipy\stats\distributions.py in <module>()
> ? ? 25 from scipy.special import gammaln as gamln
> ? ? 26 from copy import copy
> ---> 27 import vonmises_cython
> ? ? 28 import textwrap
> ? ? 29
>
> c:\stuff\utils\py\numpy.pxd in scipy.stats.vonmises_cython (scipy\stats\vonmises_cython.c:2939)()
>
> ValueError: numpy.dtype does not appear to be the correct type object
>
> In [4]: import scipy.stats as st
>
> In [5]: np.__version__
> Out[5]: '1.4.0'
>
> In [6]: sp.__version__
> Out[6]: '0.7.1'

scipy 0.7.1 is not compatible with numpy 1.4.0

new compatible versions are hopefully out soon. the released scipy
binary is only compatible with numpy 1.3..  scipy needs to be
recompiled to run with numpy 1.4.0,

Josef

>
> Paul M. Hobson
> Senior Staff Engineer
> --
> Geosyntec Consultants
> 55 SW Yamhill St, Ste 200
> Portland, OR 97204
> Phone: 503.222.9518
> www.geosyntec.com
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From dvanliere at gmail.com  Tue Mar  9 20:04:18 2010
From: dvanliere at gmail.com (Diederik van Liere)
Date: Tue, 9 Mar 2010 20:04:18 -0500
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools extension
	throws g++ internal error
Message-ID: <f99dcccd1003091704q58e1e450taaa58538c339fe16@mail.gmail.com>

As a follow up to the problem I had yesterday with the GCC compiler, I have
circumvented that by temporarily disabling the sparse module but now I am
running into the following import error:

ImportError: /usr/local/lib/python2.6/site-packages/scipy/linalg/clapack.so:
undefined symbol: clapack_sgesv

This error has been reported before and I have read those threads and
attempted their suggestions to fix it by I have not been lucky so far.
Below is the output of numpy/dist-utils/system_info.py
Hopefully this will give some pointers why I am having issues:

lapack_info:
  FOUND:
    libraries = ['lapack']
    library_dirs = ['/usr/local/atlas/lib/']
    language = f77

lapack_opt_info:
lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in /usr/local/atlas/lib/
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /usr/local/atlas/lib/
__main__.atlas_threads_info
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib/']
    language = f77
    include_dirs = ['/usr/local/atlas/include']

customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize IntelFCompiler
Could not locate executable ifort
Could not locate executable ifc
customize LaheyFCompiler
Could not locate executable lf95
customize PGroupFCompiler
Could not locate executable pgf90
Could not locate executable pgf77
customize AbsoftFCompiler
Could not locate executable f90
customize NAGFCompiler
Could not locate executable f95
customize VastFCompiler
customize CompaqFCompiler
Could not locate executable fort
customize IntelItaniumFCompiler
Could not locate executable efort
Could not locate executable efc
customize IntelEM64TFCompiler
customize Gnu95FCompiler
Found executable /usr/local/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
gcc -pthread _configtest.o -L/usr/local/atlas/lib/ -llapack -lptf77blas
-lptcblas -latlas -o _configtest
ATLAS version 3.8.3 built by root on Tue Mar  9 13:52:29 CST 2010:
   UNAME    : Linux vps.network-labs.org 2.6.18-028stab064.7-ent #1 SMP Thu
Aug 20 22:25:02 MSD 2009 i686 i686 i386 GNU/Linux
   INSTFLG  : -1 0 -a 1
   ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=2133 -DATL_SSE3
-DATL_SSE2 -DATL_SSE1 -DATL_GAS_x8632
   F2CDEFS  : -DAdd_ -DF77_INTEGER=int -DStringSunStyle
   CACHEEDGE: 786432
   F77      : gfortran, version GNU Fortran 95 (GCC) 4.1.2
   F77FLAGS : -O -fPIC -m32
   SMC      : gcc, version gcc (GCC) 4.1.2
   SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2
-fno-schedule-insns2 -fPIC -m32
   SKC      : gcc, version gcc (GCC) 4.1.2
   SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2
-fno-schedule-insns2 -fPIC -m32
success!
removing: _configtest.c _configtest.o _configtest
  FOUND:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib/']
    language = f77
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    include_dirs = ['/usr/local/atlas/include']

wx_info:
Could not locate executable wx-config
File not found: None. Cannot determine wx info.
  NOT AVAILABLE

lapack_atlas_info:
  libraries lapack_atlas,f77blas,cblas,atlas not found in
/usr/local/atlas/lib/
  libraries lapack_atlas not found in /usr/local/atlas/lib/
__main__.lapack_atlas_info
  NOT AVAILABLE

umfpack_info:
  libraries umfpack not found in /usr/local/atlas/lib/
  NOT AVAILABLE

_pkg_config_info:
Found executable /usr/bin/pkg-config
  NOT AVAILABLE

lapack_atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas,ptf77blas,ptcblas,atlas not found in
/usr/local/atlas/lib/
  libraries lapack_atlas not found in /usr/local/atlas/lib/
__main__.lapack_atlas_threads_info
  NOT AVAILABLE

x11_info:
  libraries X11 not found in /usr/local/atlas/lib/
  NOT AVAILABLE

blas_info:
  libraries blas not found in /usr/local/atlas/lib/
  NOT AVAILABLE

fftw_info:
  libraries fftw3 not found in /usr/local/atlas/lib/
  fftw3 not found
  libraries rfftw,fftw not found in /usr/local/atlas/lib/
  fftw2 not found
  NOT AVAILABLE

f2py_info:
  FOUND:
    sources =
['/usr/local/lib/python2.6/site-packages/numpy/f2py/src/fortranobject.c']
    include_dirs = ['/usr/local/lib/python2.6/site-packages/numpy/f2py/src']

gdk_pixbuf_xlib_2_info:
  NOT AVAILABLE

dfftw_threads_info:
  libraries drfftw_threads,dfftw_threads not found in /usr/local/atlas/lib/
  dfftw threads not found
  NOT AVAILABLE

atlas_blas_info:
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib/']
    language = c
    include_dirs = ['/usr/local/atlas/include']

fftw3_info:
  libraries fftw3 not found in /usr/local/atlas/lib/
  fftw3 not found
  NOT AVAILABLE

blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /usr/local/atlas/lib/
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib/']
    language = c
    include_dirs = ['/usr/local/atlas/include']

customize GnuFCompiler
customize IntelFCompiler
customize LaheyFCompiler
customize PGroupFCompiler
customize AbsoftFCompiler
customize NAGFCompiler
customize VastFCompiler
customize CompaqFCompiler
customize IntelItaniumFCompiler
customize IntelEM64TFCompiler
customize Gnu95FCompiler
customize Gnu95FCompiler
customize Gnu95FCompiler using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
gcc -pthread _configtest.o -L/usr/local/atlas/lib/ -lptf77blas -lptcblas
-latlas -o _configtest
ATLAS version 3.8.3 built by root on Tue Mar  9 13:52:29 CST 2010:
   UNAME    : Linux vps.network-labs.org 2.6.18-028stab064.7-ent #1 SMP Thu
Aug 20 22:25:02 MSD 2009 i686 i686 i386 GNU/Linux
   INSTFLG  : -1 0 -a 1
   ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=2133 -DATL_SSE3
-DATL_SSE2 -DATL_SSE1 -DATL_GAS_x8632
   F2CDEFS  : -DAdd_ -DF77_INTEGER=int -DStringSunStyle
   CACHEEDGE: 786432
   F77      : gfortran, version GNU Fortran 95 (GCC) 4.1.2
   F77FLAGS : -O -fPIC -m32
   SMC      : gcc, version gcc (GCC) 4.1.2
   SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2
-fno-schedule-insns2 -fPIC -m32
   SKC      : gcc, version gcc (GCC) 4.1.2
   SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2
-fno-schedule-insns2 -fPIC -m32
success!
removing: _configtest.c _configtest.o _configtest
  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib/']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    include_dirs = ['/usr/local/atlas/include']

sfftw_info:
  libraries srfftw,sfftw not found in /usr/local/atlas/lib/
  sfftw not found
  NOT AVAILABLE

xft_info:
  NOT AVAILABLE

fft_opt_info:
fftw2_info:
  libraries rfftw,fftw not found in /usr/local/atlas/lib/
  fftw2 not found
  NOT AVAILABLE

dfftw_info:
  libraries drfftw,dfftw not found in /usr/local/atlas/lib/
  dfftw not found
  NOT AVAILABLE

djbfft_info:
  NOT AVAILABLE

  NOT AVAILABLE

gdk_x11_2_info:
  NOT AVAILABLE

agg2_info:
  NOT AVAILABLE

numarray_info:
  NOT AVAILABLE

blas_src_info:
  NOT AVAILABLE

fftw_threads_info:
  libraries rfftw_threads,fftw_threads not found in /usr/local/atlas/lib/
  fftw threads not found
  NOT AVAILABLE

_numpy_info:
  NOT AVAILABLE

gdk_info:
  NOT AVAILABLE

gtkp_x11_2_info:
  NOT AVAILABLE

sfftw_threads_info:
  libraries srfftw_threads,sfftw_threads not found in /usr/local/atlas/lib/
  sfftw threads not found
  NOT AVAILABLE

boost_python_info:
  NOT AVAILABLE

freetype2_info:
  FOUND:
    libraries = ['freetype']
    define_macros = [('FREETYPE2_INFO', '"\\"9.10.3\\""'),
('FREETYPE2_VERSION_9_10_3', None)]
    include_dirs = ['/usr/include/freetype2']

gdk_2_info:
  NOT AVAILABLE

lapack_src_info:
  NOT AVAILABLE

gtkp_2_info:
  NOT AVAILABLE

gdk_pixbuf_2_info:
  NOT AVAILABLE

amd_info:
  libraries amd not found in /usr/local/atlas/lib/
  NOT AVAILABLE

atlas_info:
  libraries lapack_atlas not found in /usr/local/atlas/lib/
__main__.atlas_info
  FOUND:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/local/atlas/lib/']
    language = f77
    include_dirs = ['/usr/local/atlas/include']

Numeric_info:
  NOT AVAILABLE

numerix_info:
numpy_info:
  FOUND:
    define_macros = [('NUMPY_VERSION', '"\\"2.0.0.dev8291\\""'), ('NUMPY',
None)]

  FOUND:
    define_macros = [('NUMPY_VERSION', '"\\"2.0.0.dev8291\\""'), ('NUMPY',
None)]


Thanks so much for your help!
best regards,
Diederik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100309/eafce19c/attachment.html>

From dwf at cs.toronto.edu  Tue Mar  9 20:32:28 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 9 Mar 2010 20:32:28 -0500
Subject: [SciPy-User] &lt; transferred from Scipy-Dev&gt;
	Re: Seeking help/	advice for applying functions
In-Reply-To: <loom.20100309T224823-787@post.gmane.org>
References: <loom.20100309T204105-840@post.gmane.org>
	<3d375d731003091159u4151c737j65f4b2f2a7ca945@mail.gmail.com>
	<loom.20100309T224823-787@post.gmane.org>
Message-ID: <16D869AF-72D7-4145-A68E-245440FFE2A2@cs.toronto.edu>

On 9-Mar-10, at 4:50 PM, eat wrote:

> Aha, so python does not have a mechanism to optimize bytecode (nor  
> higher
> level code) inorder to get rid off the dynamic nature of code when  
> needed.

CPython does not. Not yet. PyPy has some support for this kind of JIT  
compiling, this is also one of the goals of Unladen Swallow IIRC. It's  
also what the old "Psyco" module used to do, albeit only on one  
platform.

David


From david_baddeley at yahoo.com.au  Tue Mar  9 20:35:00 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Tue, 9 Mar 2010 17:35:00 -0800 (PST)
Subject: [SciPy-User] Shared memory using multiprocessing.sharedctypes
Message-ID: <642596.42150.qm@web33003.mail.mud.yahoo.com>

I'd like shared memory numpy arrays for some code I'm trying to parallelise. The solution should ideally be cross platform. I remember some earlier discussion about this and, as far as I recall Sturla had a working implementation which was (unfortunately) linux only (please correct me if I'm wrong). 

Looking a bit further, multiprocessing.sharedctypes seems to have all the stuff needed to handle the low level bits on at least linux and windows, and it seems as though it should be relatively easy to cobble something together on top of this which let you use shared memory numpy arrays relatively transparently. This would have the advantage that all the platform specific nasty stuff is being handled by the multiprocessing module, which as it's now part of the standard library, should hopefully be well maintained & has a good chance of being ported to additional platforms

Before I jump in and try writing such a wrapper - essentially a helper function for creating shared numpy arrays (easy) and some pickle handlers to make sure what comes out the other end looks like an array (would need to think/read a bit more, but should be doable) I wanted to make sure that there's not something out there already & that I'm not going to be reinventing the wheel.

cheers,
David


      


From dwf at cs.toronto.edu  Tue Mar  9 20:35:58 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 9 Mar 2010 20:35:58 -0500
Subject: [SciPy-User] &amp; lt; transferred from Scipy-Dev&amp; gt;
	Re: Seeking help/ 	advice for applying functions
In-Reply-To: <loom.20100310T000620-731@post.gmane.org>
References: <loom.20100309T204105-840@post.gmane.org>
	<3d375d731003091159u4151c737j65f4b2f2a7ca945@mail.gmail.com>
	<loom.20100309T224823-787@post.gmane.org>
	<ce557a361003091356j4720d919o230c6de32925f763@mail.gmail.com>
	<loom.20100310T000620-731@post.gmane.org>
Message-ID: <A5931E18-3B3E-494E-9C27-14C8A4945540@cs.toronto.edu>


On 9-Mar-10, at 6:06 PM, eat wrote:

> Perhaps giving some kind of exclusive gateway to the underlying  
> libraries.

*cough cough* http://docs.scipy.org/doc/numpy/reference/c-api.html ;)

David



From peridot.faceted at gmail.com  Tue Mar  9 20:46:10 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Tue, 9 Mar 2010 20:46:10 -0500
Subject: [SciPy-User] &amp; lt; transferred from Scipy-Dev&amp; gt;
	Re: 	Seeking help/ advice for applying functions
In-Reply-To: <loom.20100310T000620-731@post.gmane.org>
References: <loom.20100309T204105-840@post.gmane.org>
	<3d375d731003091159u4151c737j65f4b2f2a7ca945@mail.gmail.com>
	<loom.20100309T224823-787@post.gmane.org>
	<ce557a361003091356j4720d919o230c6de32925f763@mail.gmail.com>
	<loom.20100310T000620-731@post.gmane.org>
Message-ID: <ce557a361003091746v64024f05vccd65fdb337a7428@mail.gmail.com>

On 9 March 2010 18:06, eat <e.antero.tammi at gmail.com> wrote:
> Anne Archibald <peridot.faceted <at> gmail.com> writes:
>
>> This kind of optimization is extremely difficult; it involves, for
>> example, inferring that a particular variable will always point to an
>> integer value, so that the type information attached to the integer
>> can be discarded. There are languages/programming environments that
>> are capable of this, but it is a difficult and expensive operation.
>> It's also unlikely to be productive for python, which is very dynamic,
>> as languages go: the value "None" is often used to signify "no answer
>> yet" in variables that would otherwise point to only one type of
>> value, and it's perfectly acceptable to change what function
>> "mymodule.myfunction" points to at any point.
>>
> OK. I kind of understand, but wouldn't it possible to somehow sidetrack this
> whole dynamic nature of python (when really needed)? Perhaps giving some kind
> of exclusive gateway to the underlying libraries. Yes, I'm considering here
> only the mature libraries, where no API changes are expected.

It's not just a question of libraries but also of data representation
- do you annotate every floating point value with a tag that says "I'm
a floating-point value" or not? - but yes, that's exactly what cython,
f2py, swig, and all the other python interfacing tools are for. The
general philosophy with python has always been "use python where you
need flexibility and convenience; where you need speed call into
compiled code". This is great because for many (most?) programs the
part that actually needs to be fast is a small fraction of the total
lines of code, and the rest becomes easier to write and debug when you
do it in python.

Anne


From dwf at cs.toronto.edu  Tue Mar  9 20:56:27 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 9 Mar 2010 20:56:27 -0500
Subject: [SciPy-User] return "full_output" or how to stop throwing away
	already calculated results
In-Reply-To: <1cd32cbb1003091301m7fdd82e5je2cfcd6cf292d0ba@mail.gmail.com>
References: <1cd32cbb1003091301m7fdd82e5je2cfcd6cf292d0ba@mail.gmail.com>
Message-ID: <72427065-39FA-43FD-A745-68499C3A3C17@cs.toronto.edu>


On 9-Mar-10, at 4:01 PM, josef.pktd at gmail.com wrote:

> in statsmodels, I am switching more to the pattern when
> full_output=True (or similar keyword) for a function, then all
> intermediate results are returned, attached to a generic class
> instance ( or could be a struct in matlab or a bunch for Gael,...)

David Cournapeau pointed out (in the thread "anyone to look at #1402?"  
on the numpy-discussion list) that the wider Python community frowns  
on the type of the returned value depending on a boolean flag  
argument, preferring instead different function names that call some  
common (private) helper function. It provided some validation for my  
uncomfortable feelings about scipy.optimize methods that do this.

I do think it's worth thinking about returning some sort of proxy  
object that can have attributes set to either 'None' or the  
appropriate values. It would certainly make code more readable (except  
in the situation where you use argument unpacking, but even then it  
isn't obvious to an outsider how crucial that full_output thing is).

David


From PHobson at Geosyntec.com  Tue Mar  9 21:26:36 2010
From: PHobson at Geosyntec.com (PHobson at Geosyntec.com)
Date: Tue, 9 Mar 2010 21:26:36 -0500
Subject: [SciPy-User] First attempt to import scipy.stats fails
In-Reply-To: <1cd32cbb1003091703m6304694ei510d8db0e61fdfc2@mail.gmail.com>
References: <C97487CD346EEA4E80423F51F6D1EB2FA9769397ED@EXCHANGE-00.GeoSyntec.net>
	<1cd32cbb1003091703m6304694ei510d8db0e61fdfc2@mail.gmail.com>
Message-ID: <C97487CD346EEA4E80423F51F6D1EB2FA976939818@EXCHANGE-00.GeoSyntec.net>

> -----Original Message-----
> From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org]
> On Behalf Of josef.pktd at gmail.com
> scipy 0.7.1 is not compatible with numpy 1.4.0
> 
> new compatible versions are hopefully out soon. the released scipy
> binary is only compatible with numpy 1.3..  scipy needs to be
> recompiled to run with numpy 1.4.0,
> 
> Josef

Ahh. Thanks for the info. Strange how it works fine the second time. 
Cheers,
-paul h.


From robfalck at gmail.com  Tue Mar  9 22:37:32 2010
From: robfalck at gmail.com (Rob Falck)
Date: Tue, 9 Mar 2010 22:37:32 -0500
Subject: [SciPy-User] return "full_output" or how to stop throwing away
	already calculated results
In-Reply-To: <72427065-39FA-43FD-A745-68499C3A3C17@cs.toronto.edu>
References: <1cd32cbb1003091301m7fdd82e5je2cfcd6cf292d0ba@mail.gmail.com>
	<72427065-39FA-43FD-A745-68499C3A3C17@cs.toronto.edu>
Message-ID: <ccb1b5851003091937id01f67fqa1a298390b8b333a@mail.gmail.com>

Returning an object would be my preference as well, it seems more
pythonic.  Most optimizers should be able to return at a minimum

result.x
result.objfun
result.iterations
result.exit_status

Some optimizers have other useful data to return, such as the Lagrange
multiplers from slsqp.  Going down that path means probably breaking
current implementations of the optimizers, but doing it right would be
worth it, in my opinion.  We should also agree upon names for the
common attributes.

On Tue, Mar 9, 2010 at 8:56 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>
> On 9-Mar-10, at 4:01 PM, josef.pktd at gmail.com wrote:
>
>> in statsmodels, I am switching more to the pattern when
>> full_output=True (or similar keyword) for a function, then all
>> intermediate results are returned, attached to a generic class
>> instance ( or could be a struct in matlab or a bunch for Gael,...)
>
> David Cournapeau pointed out (in the thread "anyone to look at #1402?"
> on the numpy-discussion list) that the wider Python community frowns
> on the type of the returned value depending on a boolean flag
> argument, preferring instead different function names that call some
> common (private) helper function. It provided some validation for my
> uncomfortable feelings about scipy.optimize methods that do this.
>
> I do think it's worth thinking about returning some sort of proxy
> object that can have attributes set to either 'None' or the
> appropriate values. It would certainly make code more readable (except
> in the situation where you use argument unpacking, but even then it
> isn't obvious to an outsider how crucial that full_output thing is).
>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
- Rob Falck


From dwf at cs.toronto.edu  Tue Mar  9 23:45:22 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 9 Mar 2010 23:45:22 -0500
Subject: [SciPy-User] return "full_output" or how to stop throwing away
	already calculated results
In-Reply-To: <ccb1b5851003091937id01f67fqa1a298390b8b333a@mail.gmail.com>
References: <1cd32cbb1003091301m7fdd82e5je2cfcd6cf292d0ba@mail.gmail.com>
	<72427065-39FA-43FD-A745-68499C3A3C17@cs.toronto.edu>
	<ccb1b5851003091937id01f67fqa1a298390b8b333a@mail.gmail.com>
Message-ID: <BCD693B6-4AB1-406F-9600-E9BD2DD87AE0@cs.toronto.edu>


On 9-Mar-10, at 10:37 PM, Rob Falck wrote:

> Returning an object would be my preference as well, it seems more
> pythonic.  Most optimizers should be able to return at a minimum
>
> result.x
> result.objfun
> result.iterations
> result.exit_status
>
> Some optimizers have other useful data to return, such as the Lagrange
> multiplers from slsqp.  Going down that path means probably breaking
> current implementations of the optimizers, but doing it right would be
> worth it, in my opinion.  We should also agree upon names for the
> common attributes.

It would require at least one version's worth of deprecation of the  
old interface, probably two or more.

Better to settle on the interface, implement it, and introduce it  
alongside. In my opinion, the more descriptive the names, the better:  
result.minimizer instead of result.x, result.minimum_found or  
result.minimum_value rather than result.objfun, etc. - if we're going  
to add a layer of complexity then we might as well make the names as  
unambiguous as possible.

Also often of interest: number of function evaluations, gradient  
evaluations, Hessian evaluations, etc.

The current report that gets printed could also form a print_summary()  
method for these objects or something like that.

David


From josef.pktd at gmail.com  Tue Mar  9 23:54:15 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 9 Mar 2010 23:54:15 -0500
Subject: [SciPy-User] return "full_output" or how to stop throwing away
	already calculated results
In-Reply-To: <ccb1b5851003091937id01f67fqa1a298390b8b333a@mail.gmail.com>
References: <1cd32cbb1003091301m7fdd82e5je2cfcd6cf292d0ba@mail.gmail.com>
	<72427065-39FA-43FD-A745-68499C3A3C17@cs.toronto.edu>
	<ccb1b5851003091937id01f67fqa1a298390b8b333a@mail.gmail.com>
Message-ID: <1cd32cbb1003092054ged2da2n7b36498c78f728ee@mail.gmail.com>

On Tue, Mar 9, 2010 at 10:37 PM, Rob Falck <robfalck at gmail.com> wrote:
> Returning an object would be my preference as well, it seems more
> pythonic. ?Most optimizers should be able to return at a minimum
>
> result.x
> result.objfun
> result.iterations
> result.exit_status
>
> Some optimizers have other useful data to return, such as the Lagrange
> multiplers from slsqp. ?Going down that path means probably breaking
> current implementations of the optimizers, but doing it right would be
> worth it, in my opinion. ?We should also agree upon names for the
> common attributes.

If we create new functions, as David argued, then the old signature
could still be kept.

I was curious what the increase in call overhead is, and for an
essentially empty function it would be up to 60%. However, since this
is mainly for functions that do heavier work, this will not be really
relevant. (It might add a little bit to my Monte Carlos or bootstrap,
but this should be negligible), and we will gain if we don't have to
redo some calculations.

The main reason I liked the full output option instead of always
returning a result instance, is that, if full_output is true, then
additional results can be calculated, that I don't want when I just
need a fast minimal result.

Josef

>
> On Tue, Mar 9, 2010 at 8:56 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>>
>> On 9-Mar-10, at 4:01 PM, josef.pktd at gmail.com wrote:
>>
>>> in statsmodels, I am switching more to the pattern when
>>> full_output=True (or similar keyword) for a function, then all
>>> intermediate results are returned, attached to a generic class
>>> instance ( or could be a struct in matlab or a bunch for Gael,...)
>>
>> David Cournapeau pointed out (in the thread "anyone to look at #1402?"
>> on the numpy-discussion list) that the wider Python community frowns
>> on the type of the returned value depending on a boolean flag
>> argument, preferring instead different function names that call some
>> common (private) helper function. It provided some validation for my
>> uncomfortable feelings about scipy.optimize methods that do this.
>>
>> I do think it's worth thinking about returning some sort of proxy
>> object that can have attributes set to either 'None' or the
>> appropriate values. It would certainly make code more readable (except
>> in the situation where you use argument unpacking, but even then it
>> isn't obvious to an outsider how crucial that full_output thing is).
>>
>> David
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
>
> --
> - Rob Falck
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------


import numpy as np

class Store(object):
    pass

def function0():
    y = 0
#    x = np.random.randn(20,20)
#    y = np.linalg.inv(x)
#    y = np.corrcoef(x)
    return y

def function1():
    result = Store()
    result.a = np.arange(5)
    result.b = function0()
    return result

def function2():
    return np.arange(5), function0()
    
def function3():
    result = function1()
    return result.a, result.b

def function4(full_output=True):
    if full_output:
        result = Store()
        result.a = np.arange(5)
        result.b = function0()
        return result
    else:
        return np.arange(5), function0()

if __name__=='__main__':
    from timeit import Timer
    n = 10000
    t = Timer("r = function1()", "from __main__ import np,Store,function1")
    t1 = t.timeit(n)
    print 'function1', t1
    t = Timer("r = function2()", "from __main__ import np,Store,function2")
    t2 = t.timeit(n)
    print 'function2', t2
    t = Timer("r = function3()", "from __main__ import np,Store,function3")
    t3 = t.timeit(n)
    print 'function3', t3, t3/t2
    t = Timer("r = function1()", "from __main__ import np,Store,function1")
    t1 = t.timeit(n)  #repeated
    print 'function1', t1, t1/t2
    t = Timer("r = function4()", "from __main__ import np,Store,function4")
    t4a = t.timeit(n)
    print 'function4a', t4a, t4a/t2
    t = Timer("r = function4(full_output=0)", 
              "from __main__ import np,Store,function4")
    t4b = t.timeit(n)
    print 'function4b', t4b, t4b/t2

From Chris.Barker at noaa.gov  Wed Mar 10 00:06:28 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 09 Mar 2010 21:06:28 -0800
Subject: [SciPy-User] fromimage/imread segfaults on my images
In-Reply-To: <hn4q6t$ib0$1@dough.gmane.org>
References: <hn33vh$bgo$1@dough.gmane.org> <4B9556D0.9090409@noaa.gov>
	<hn4q6t$ib0$1@dough.gmane.org>
Message-ID: <4B9728D4.3040109@noaa.gov>

Daniel Lenski wrote:
> Everything works fine as long as I stick with PIL calls only... I can 
> even dump the entire contents of the image to a string with 
> PIL.Image.fromstring().  No problems there.

did you try passing that to numpy.fromstring? And what did it dump? 
single images can be tricky.

>> np.asarray(pil_image)
> 
> I looked at the code for scipy.misc.fromimage, and this is basically all 
> it does.  This is where the segfault occurs.

I don't think that code path has been highly tested -- and maybe never 
for 1-bit images -- it eventually will be deprecated in favor of the new 
buffer protocol anyway.

>> do you have a smaller image in the same format you can experiment with?
>> That might make it easier to figure out.
> 
> Smaller images work fine.  I haven't figured out exactly where the cutoff 
> is, but 500x500 works fine, for instance.

I'd be surprised if size is the issue here -- your images just aren't 
that big (as you pointed out, the MPL functions worked even though they 
were converting to RGB). I'm just guessing that you problem had 
something to do with the bits->bytes conversion.

>> http://netpbm.sourceforge.net/doc/pbm.html
>>
>> read the header, then read the data with np.fromfile(), then convert to
>> a uint8 array with np.unpackbits().
> 
> That's what I've ended up doing... rolled my own loadpbm() function.

Then I guess you're done!

have fun, sorry this was harder than it should have been.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From josef.pktd at gmail.com  Wed Mar 10 00:32:48 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 10 Mar 2010 00:32:48 -0500
Subject: [SciPy-User] return "full_output" or how to stop throwing away
	already calculated results
In-Reply-To: <1cd32cbb1003092054ged2da2n7b36498c78f728ee@mail.gmail.com>
References: <1cd32cbb1003091301m7fdd82e5je2cfcd6cf292d0ba@mail.gmail.com>
	<72427065-39FA-43FD-A745-68499C3A3C17@cs.toronto.edu>
	<ccb1b5851003091937id01f67fqa1a298390b8b333a@mail.gmail.com>
	<1cd32cbb1003092054ged2da2n7b36498c78f728ee@mail.gmail.com>
Message-ID: <1cd32cbb1003092132pfa4322dp9acfe547aeeb416b@mail.gmail.com>

On Tue, Mar 9, 2010 at 11:54 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Mar 9, 2010 at 10:37 PM, Rob Falck <robfalck at gmail.com> wrote:
>> Returning an object would be my preference as well, it seems more
>> pythonic. ?Most optimizers should be able to return at a minimum
>>
>> result.x
>> result.objfun
>> result.iterations
>> result.exit_status
>>
>> Some optimizers have other useful data to return, such as the Lagrange
>> multiplers from slsqp. ?Going down that path means probably breaking
>> current implementations of the optimizers, but doing it right would be
>> worth it, in my opinion. ?We should also agree upon names for the
>> common attributes.
>
> If we create new functions, as David argued, then the old signature
> could still be kept.
>
> I was curious what the increase in call overhead is, and for an
> essentially empty function it would be up to 60%. However, since this
> is mainly for functions that do heavier work, this will not be really
> relevant. (It might add a little bit to my Monte Carlos or bootstrap,
> but this should be negligible), and we will gain if we don't have to
> redo some calculations.
>
> The main reason I liked the full output option instead of always
> returning a result instance, is that, if full_output is true, then
> additional results can be calculated, that I don't want when I just
> need a fast minimal result.

something like this might work, without any undesired calculations and
any break in current API, and return type is independent of
"full_output" flag. (short of going to a class with lazy evaluation)

def function2a():
    '''current function'''
    return np.arange(5)

new:

def function5a(full_output=True):
    '''current function rewritten to return Store'''
    result = Store()
    result.a = np.arange(5)
    if full_output:
        #results that require additional memory or additional calculations
        result.b = function0()
    return result

def function5():
    '''replacement for current function'''
    result = function5a(full_output=False)
    return result.a


The extra calculations might not apply for many functions, but I have
some in statsmodels for which I could also switch to this kind of
pattern instead of making the return type conditional on a keyword.

Josef

>
>>
>> On Tue, Mar 9, 2010 at 8:56 PM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
>>>
>>> On 9-Mar-10, at 4:01 PM, josef.pktd at gmail.com wrote:
>>>
>>>> in statsmodels, I am switching more to the pattern when
>>>> full_output=True (or similar keyword) for a function, then all
>>>> intermediate results are returned, attached to a generic class
>>>> instance ( or could be a struct in matlab or a bunch for Gael,...)
>>>
>>> David Cournapeau pointed out (in the thread "anyone to look at #1402?"
>>> on the numpy-discussion list) that the wider Python community frowns
>>> on the type of the returned value depending on a boolean flag
>>> argument, preferring instead different function names that call some
>>> common (private) helper function. It provided some validation for my
>>> uncomfortable feelings about scipy.optimize methods that do this.
>>>
>>> I do think it's worth thinking about returning some sort of proxy
>>> object that can have attributes set to either 'None' or the
>>> appropriate values. It would certainly make code more readable (except
>>> in the situation where you use argument unpacking, but even then it
>>> isn't obvious to an outsider how crucial that full_output thing is).
>>>
>>> David
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>
>>
>>
>> --
>> - Rob Falck
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From robfalck at gmail.com  Wed Mar 10 07:44:13 2010
From: robfalck at gmail.com (Rob Falck)
Date: Wed, 10 Mar 2010 07:44:13 -0500
Subject: [SciPy-User] return "full_output" or how to stop throwing away
	already calculated results
In-Reply-To: <1cd32cbb1003092132pfa4322dp9acfe547aeeb416b@mail.gmail.com>
References: <1cd32cbb1003091301m7fdd82e5je2cfcd6cf292d0ba@mail.gmail.com>
	<72427065-39FA-43FD-A745-68499C3A3C17@cs.toronto.edu>
	<ccb1b5851003091937id01f67fqa1a298390b8b333a@mail.gmail.com>
	<1cd32cbb1003092054ged2da2n7b36498c78f728ee@mail.gmail.com>
	<1cd32cbb1003092132pfa4322dp9acfe547aeeb416b@mail.gmail.com>
Message-ID: <ccb1b5851003100444o22e153ect8d68de250c467816@mail.gmail.com>

At least in the case of fmin_slsqp (and probably most of the other
optimizers) the attributes returned when full_output=True are things
that are being calculated anyway.  There really shouldn't be much
penalty in terms of speed.  I agree that keeping the old call
signatures in a deprecated mode would be a good idea, though I admit I
don't really like the idea of changing the function names away from
fmin_xxxx.  Perhaps we keep the current function definitions and only
the full_output argument becomes deprecated?

On Wed, Mar 10, 2010 at 12:32 AM,  <josef.pktd at gmail.com> wrote:
> On Tue, Mar 9, 2010 at 11:54 PM, ?<josef.pktd at gmail.com> wrote:
>> On Tue, Mar 9, 2010 at 10:37 PM, Rob Falck <robfalck at gmail.com> wrote:
>>> Returning an object would be my preference as well, it seems more
>>> pythonic. ?Most optimizers should be able to return at a minimum
>>>
>>> result.x
>>> result.objfun
>>> result.iterations
>>> result.exit_status
>>>
>>> Some optimizers have other useful data to return, such as the Lagrange
>>> multiplers from slsqp. ?Going down that path means probably breaking
>>> current implementations of the optimizers, but doing it right would be
>>> worth it, in my opinion. ?We should also agree upon names for the
>>> common attributes.
>>
>> If we create new functions, as David argued, then the old signature
>> could still be kept.
>>
>> I was curious what the increase in call overhead is, and for an
>> essentially empty function it would be up to 60%. However, since this
>> is mainly for functions that do heavier work, this will not be really
>> relevant. (It might add a little bit to my Monte Carlos or bootstrap,
>> but this should be negligible), and we will gain if we don't have to
>> redo some calculations.
>>
>> The main reason I liked the full output option instead of always
>> returning a result instance, is that, if full_output is true, then
>> additional results can be calculated, that I don't want when I just
>> need a fast minimal result.
>
> something like this might work, without any undesired calculations and
> any break in current API, and return type is independent of
> "full_output" flag. (short of going to a class with lazy evaluation)
>
> def function2a():
> ? ?'''current function'''
> ? ?return np.arange(5)
>
> new:
>
> def function5a(full_output=True):
> ? ?'''current function rewritten to return Store'''
> ? ?result = Store()
> ? ?result.a = np.arange(5)
> ? ?if full_output:
> ? ? ? ?#results that require additional memory or additional calculations
> ? ? ? ?result.b = function0()
> ? ?return result
>
> def function5():
> ? ?'''replacement for current function'''
> ? ?result = function5a(full_output=False)
> ? ?return result.a
>
>
> The extra calculations might not apply for many functions, but I have
> some in statsmodels for which I could also switch to this kind of
> pattern instead of making the return type conditional on a keyword.
>
> Josef


-- 
- Rob Falck


From e.antero.tammi at gmail.com  Wed Mar 10 15:04:22 2010
From: e.antero.tammi at gmail.com (eat)
Date: Wed, 10 Mar 2010 20:04:22 +0000 (UTC)
Subject: [SciPy-User]
	=?utf-8?q?=26amp=3Blt=3B_transferred_from_Scipy-Dev?=
	=?utf-8?q?=26amp=3Bgt=3B=09Re=3A_Seeking_help/=09advice_for_applyi?=
	=?utf-8?q?ng_functions?=
References: <loom.20100309T204105-840@post.gmane.org>
	<3d375d731003091159u4151c737j65f4b2f2a7ca945@mail.gmail.com>
	<loom.20100309T224823-787@post.gmane.org>
	<16D869AF-72D7-4145-A68E-245440FFE2A2@cs.toronto.edu>
Message-ID: <loom.20100310T205319-434@post.gmane.org>

David Warde-Farley <dwf <at> cs.toronto.edu> writes:

> 
> On 9-Mar-10, at 4:50 PM, eat wrote:
> 
> > Aha, so python does not have a mechanism to optimize bytecode (nor  
> > higher
> > level code) inorder to get rid off the dynamic nature of code when  
> > needed.
> 
> CPython does not. Not yet. PyPy has some support for this kind of JIT  
> compiling, this is also one of the goals of Unladen Swallow IIRC. It's  
> also what the old "Psyco" module used to do, albeit only on one  
> platform.
> 
> David
> 
Hi,

I just browsed some documents of those. But I'll better to wait when some of 
those (if any) are supported by scipy.


Thanks,
eat



From guilherme at gpfreitas.com  Wed Mar 10 18:20:17 2010
From: guilherme at gpfreitas.com (Guilherme P. de Freitas)
Date: Wed, 10 Mar 2010 15:20:17 -0800
Subject: [SciPy-User] Lagrange Multipliers in optimize.slsqp
In-Reply-To: <ccb1b5851003041755p5c7d1a29qe56081b3bd7ccd74@mail.gmail.com>
References: <a193bbb51002251717j3ab63e73heebcf4f2e229a73e@mail.gmail.com>
	<c048da1c1002252008i124597dbl476c6e6dc95e96b0@mail.gmail.com>
	<1cd32cbb1002260555k3b72a50bsc2e606c9f0e68265@mail.gmail.com>
	<c048da1c1002260735p44263cc3gc394b403f8f2100@mail.gmail.com>
	<a193bbb51003021114l709bbbc9oca3d4941409acd72@mail.gmail.com>
	<ccb1b5851003041755p5c7d1a29qe56081b3bd7ccd74@mail.gmail.com>
Message-ID: <a193bbb51003101520q6667e7e0xc4de750821000d97@mail.gmail.com>

I was wondering if it is worth outputting the multipliers at all...
The low accuracy in a very simple example (see my previous post) makes
me wonder how innacurate the estimated multipliers can be in a more
complex problem.

On Thu, Mar 4, 2010 at 5:55 PM, Rob Falck <robfalck at gmail.com> wrote:
> Any opinions on how this change should be implemented? ?I'd prefer to
> keep the code clean, and the solution below is simple, but I'd rather
> not break current implementations of SLSQP that people have employed,
> as inserting a new element into the output likely would. ?I'm leaning
> towards allowing both boolean or integer types for the full_output
> argument. ?0 or 1 would work as True and False do now, and 2 would
> return the Lagrange multipliers as well as everything else.



-- 
Guilherme P. de Freitas
http://www.gpfreitas.com


From agile.aspect at gmail.com  Wed Mar 10 22:36:11 2010
From: agile.aspect at gmail.com (Agile Aspect)
Date: Wed, 10 Mar 2010 19:36:11 -0800
Subject: [SciPy-User] Compiling scipy.sparse.sparsetools extension
	throws g++ internal error
In-Reply-To: <f99dcccd1003091704q58e1e450taaa58538c339fe16@mail.gmail.com>
References: <f99dcccd1003091704q58e1e450taaa58538c339fe16@mail.gmail.com>
Message-ID: <c6a6f63b1003101936oeaad614mdf072d89f12b0ebe@mail.gmail.com>

On Tue, Mar 9, 2010 at 5:04 PM, Diederik van Liere <dvanliere at gmail.com> wrote:
> As a follow up to the problem I had yesterday with the GCC compiler, I have
> circumvented that by temporarily disabling the sparse module but now I am
> running into the following import error:
>
> ImportError: /usr/local/lib/python2.6/site-packages/scipy/linalg/clapack.so:
> undefined symbol: clapack_sgesv
>

You should use nm to check for the symbol in your ATLAS libraries to
see if it's really missing or if numpy just can't find it.

Try

  nm -A  /usr/local/atlas/lib/lib*.so | grep clapack_sgesv

or just

  nm liblapack.so | grep clapack_sgesv

On my machine, this produces

     liblapack.so:000aff88 T clapack_sgesv

The "T"  symbol means it's defined.

If the symbol is defined in your ATLAS libraries, then you should post
your site.cfg from numpy.

If it isn't defined, then you need to rebuild the ATLAS libraries.

-- 
      Enjoy global warming while it lasts.


From boris.burle at univ-provence.fr  Thu Mar 11 01:42:04 2010
From: boris.burle at univ-provence.fr (=?ISO-8859-1?Q?Bor=EDs_BURLE?=)
Date: Thu, 11 Mar 2010 07:42:04 +0100
Subject: [SciPy-User] Mayavi delaunay3D triangles
Message-ID: <4B9890BC.2070006@univ-provence.fr>

Dear all,

I have a set of points coordinates (x,y,z) representing a surface (in 
3D, but it's not a volume). I would like to create a mesh from those 
points, and then map some values on this mesh (colored surface). Mayavi 
allows to easily represent the mesh. What I have done so far is:
   
pts=mlab.points3d(x,y,z)
delaunay = mlab.pipeline.delaunay3d(pts)
surf = mlab.pipeline.surface(delaunay)

That's create a nice plot of my mesh, but of course without values 
mapped on the surface. Beforehand, I have been using 
mlab.triangular_mesh to create such a colored surface (from a vtk-like 
file), and would be happy to use the same approach. However, 
triangular_mesh requires the x,y,z positions, plus a list  of triplets 
defining the connecting triangles. So, my question is: how can I extract 
those triangles from the delaunay3d object created above? This 
information is very likely stored somewhere, but after hours of 
exploration, I haven't been able to extract it!!

    Thank you in advance for your help !!
          All the best,
             B.

-- 
Bor?s BURLE
Laboratoire de Neurobiologie de la Cognition
CNRS et Universit? de Provence
tel: (+33) 4 88 57 68 79
fax: (+33) 4 88 57 68 72
web page:  http://www.up.univ-mrs.fr/lnc/ACT/act-fr.html



From jgomezdans at gmail.com  Thu Mar 11 06:10:33 2010
From: jgomezdans at gmail.com (Jose Gomez-Dans)
Date: Thu, 11 Mar 2010 11:10:33 +0000
Subject: [SciPy-User] Maximum file size for .npz format?
Message-ID: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>

Hi!
I need to save a fairly large set of arrays to disk. I have saved it using
numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
large ;D). When I try to load it using numpy.load, the zipfile module
compains about
BadZipfile: Bad magic number for file header

I can't open it with the normal zip utility present on the system, but it
could be that it's barfing about files being larger than 2Gb.
Is there some file limit for npzs? Is there anyway I can recover the data (I
guess I could try decompressing the file with 7z and extracting the
individual npy files?)

Thanks!
Jose
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100311/57fb4c4f/attachment.html>

From pierre.lamon at bluebotics.com  Thu Mar 11 08:19:25 2010
From: pierre.lamon at bluebotics.com (Pierre Lamon)
Date: Thu, 11 Mar 2010 14:19:25 +0100
Subject: [SciPy-User] Potential KDTree memory leak
Message-ID: <4B98EDDD.6040608@bluebotics.com>

Hi all,

the memory increases at each iteration of the following program:

from numpy import ndarray
from scipy.spatial import KDTree

class C(ndarray):
     pass

pts = [C((2,)) for i in range(1000)]
for i in xrange(10000):
     tree = KDTree(pts)

It does not occur when using the base class ndarray instead. I've only 
tested on a windows machine. Any idea what happens?

Of course this is a dummy program tailored to emphasis the problem. But 
as I'm using KDTree intensively in my application (both creation and 
query) and I need to fix this memory leak.

Any help is greatly appreciated,
~Pierre




From robert.kern at gmail.com  Thu Mar 11 10:48:41 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 11 Mar 2010 09:48:41 -0600
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
Message-ID: <3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>

On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans <jgomezdans at gmail.com> wrote:
> Hi!
> I need to save a fairly large set of arrays to disk. I have saved it using
> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
> large ;D). When I try to load it using numpy.load, the zipfile module
> compains about
> BadZipfile: Bad magic number for file header
>
> I can't open it with the normal zip utility present on the system, but it
> could be that it's barfing about files being larger than 2Gb.
> Is there some file limit for npzs?

Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
not yet support the ZIP64 format.

> Is there anyway I can recover the data (I
> guess I could try decompressing the file with 7z and extracting the
> individual npy files?)

Possibly. However, if the normal zip utility isn't working, 7z
probably won't, either. Worth a try, though.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Mar 11 10:50:07 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 11 Mar 2010 09:50:07 -0600
Subject: [SciPy-User] Potential KDTree memory leak
In-Reply-To: <4B98EDDD.6040608@bluebotics.com>
References: <4B98EDDD.6040608@bluebotics.com>
Message-ID: <3d375d731003110750h3a6f8bd6se0a91b9cb98ff7b1@mail.gmail.com>

On Thu, Mar 11, 2010 at 07:19, Pierre Lamon <pierre.lamon at bluebotics.com> wrote:
> Hi all,
>
> the memory increases at each iteration of the following program:
>
> from numpy import ndarray
> from scipy.spatial import KDTree
>
> class C(ndarray):
> ? ? pass
>
> pts = [C((2,)) for i in range(1000)]
> for i in xrange(10000):
> ? ? tree = KDTree(pts)
>
> It does not occur when using the base class ndarray instead. I've only
> tested on a windows machine. Any idea what happens?

With recent SVN versions of numpy and scipy, I do not see a leak.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dvanliere at gmail.com  Thu Mar 11 13:10:14 2010
From: dvanliere at gmail.com (Diederik van Liere)
Date: Thu, 11 Mar 2010 13:10:14 -0500
Subject: [SciPy-User] Undefined symbol: clapack_sgesv when doing import in
	Python
Message-ID: <f99dcccd1003111010p7e05517bg480116b9d6a2f620@mail.gmail.com>

>> ImportError:
/usr/local/lib/python2.6/site-packages/scipy/linalg/clapack.so:
>> undefined symbol: clapack_sgesv


> You should use nm to check for the symbol in your ATLAS libraries to
> see if it's really missing or if numpy just can't find it.

> Try

> nm -A  /usr/local/atlas/lib/lib*.so | grep clapack_sgesv

> or just

> nm liblapack.so | grep clapack_sgesv

This is my output:
liblapack.a:clapack_sgesv.o:00000000 r .LC0
liblapack.a:clapack_sgesv.o:00000000 r .LC1
liblapack.a:clapack_sgesv.o:0000002c r .LC2
liblapack.a:clapack_sgesv.o:0000005c r .LC3
liblapack.a:clapack_sgesv.o:00000090 r .LC4
liblapack.a:clapack_sgesv.o:000000b8 r .LC5
liblapack.a:clapack_sgesv.o:         U ATL_sgetrf
liblapack.a:clapack_sgesv.o:         U ATL_sgetrs
liblapack.a:clapack_sgesv.o:         U _GLOBAL_OFFSET_TABLE_
liblapack.a:clapack_sgesv.o:00000000 T __i686.get_pc_thunk.bx
liblapack.a:clapack_sgesv.o:         U cblas_xerbla
liblapack.a:clapack_sgesv.o:00000000 T clapack_sgesv

I don't have .so files, I have .a files, could that be an issue? So it looks
that ATLAS is properly compiled, right?

> If the symbol is defined in your ATLAS libraries, then you should post
> your site.cfg from numpy.

My site.cfg from numpy reads:

[DEFAULT]
library_dirs = /usr/local/atlas/lib/
include_dirs = /usr/local/atlas/include/

[blas_opt]
libraries = lapack, f77blas, cblas, atlas

[lapack_opt]
libraries = lapack, f77blas, cblas, atlas


Thanks so much for your patience and help.
Best,

Diederik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100311/e757d97e/attachment.html>

From ia3n at stanford.edu  Thu Mar 11 15:21:35 2010
From: ia3n at stanford.edu (stair314)
Date: Thu, 11 Mar 2010 12:21:35 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] ATLAS install has no effect
Message-ID: <27869293.post@talk.nabble.com>


I'm following the instructions at:
http://www.scipy.org/Installing_SciPy/Linux#head-40c26a5b93b9afc7e3241e1d7fd84fe9326402e7

After following the instructions for installing ATLAS, I do get the expected
files:
% ls /usr/lib/atlas/sse2
libblas.so.3.0  liblapack.so.3.0


However, the instructions recommend squaring a 1000x1000 matrix with and
without LD_LIBRARY_PATH set to use ATLAS to test whether the installation
worked. I get no difference when setting LD_LIBRARY_PATH to use ATLAS. Also,
this task takes less than a second in Matlab but is taking 10.3 seconds in
numpy. Any ideas how to diagnose whether numpy is using ATLAS or if my
installation of ATLAS is just slow?

Thanks in advance.
-- 
View this message in context: http://old.nabble.com/ATLAS-install-has-no-effect-tp27869293p27869293.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From dwf at cs.toronto.edu  Thu Mar 11 16:06:46 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Thu, 11 Mar 2010 16:06:46 -0500
Subject: [SciPy-User] [SciPy-user] ATLAS install has no effect
In-Reply-To: <27869293.post@talk.nabble.com>
References: <27869293.post@talk.nabble.com>
Message-ID: <A1058F6F-371C-44B9-9ABE-3926F30A53A9@cs.toronto.edu>


On 11-Mar-10, at 3:21 PM, stair314 wrote:

> However, the instructions recommend squaring a 1000x1000 matrix with  
> and
> without LD_LIBRARY_PATH set to use ATLAS to test whether the  
> installation
> worked. I get no difference when setting LD_LIBRARY_PATH to use  
> ATLAS. Also,
> this task takes less than a second in Matlab but is taking 10.3  
> seconds in
> numpy. Any ideas how to diagnose whether numpy is using ATLAS or if my
> installation of ATLAS is just slow?


Can you give us the output of numpy.show_config()?

David



From ia3n at stanford.edu  Thu Mar 11 16:37:50 2010
From: ia3n at stanford.edu (Ian Goodfellow)
Date: Thu, 11 Mar 2010 13:37:50 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] ATLAS install has no effect
In-Reply-To: <A1058F6F-371C-44B9-9ABE-3926F30A53A9@cs.toronto.edu>
Message-ID: <1997623173.2812691268343470110.JavaMail.root@zm09.stanford.edu>

Sure, that's a good call to know about. It looks like it's not using the same setting for library_dirs as what's in my LD_LIBRARY_PATH.

>>> numpy.show_config()
blas_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77

lapack_info:
    libraries = ['lapack']
    library_dirs = ['/usr/lib']
    language = f77

atlas_threads_info:
  NOT AVAILABLE

blas_opt_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]

atlas_blas_threads_info:
  NOT AVAILABLE

lapack_opt_info:
    libraries = ['lapack', 'blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]

atlas_info:
  NOT AVAILABLE

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

atlas_blas_info:
  NOT AVAILABLE

mkl_info:
  NOT AVAILABLE

>>> 
ia3n at gaussian:~/theano/2_atlas% echo $LD_LIBRARY_PATH
/usr/lib/atlas/sse2

----- Original Message -----
From: "David Warde-Farley" <dwf at cs.toronto.edu>
To: "SciPy Users List" <scipy-user at scipy.org>
Sent: Thursday, March 11, 2010 1:06:46 PM GMT -08:00 US/Canada Pacific
Subject: Re: [SciPy-User] [SciPy-user] ATLAS install has no effect


On 11-Mar-10, at 3:21 PM, stair314 wrote:

> However, the instructions recommend squaring a 1000x1000 matrix with  
> and
> without LD_LIBRARY_PATH set to use ATLAS to test whether the  
> installation
> worked. I get no difference when setting LD_LIBRARY_PATH to use  
> ATLAS. Also,
> this task takes less than a second in Matlab but is taking 10.3  
> seconds in
> numpy. Any ideas how to diagnose whether numpy is using ATLAS or if my
> installation of ATLAS is just slow?


Can you give us the output of numpy.show_config()?

David

_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user


From gruben at bigpond.net.au  Thu Mar 11 18:31:06 2010
From: gruben at bigpond.net.au (Gary Ruben)
Date: Fri, 12 Mar 2010 10:31:06 +1100
Subject: [SciPy-User] Mayavi delaunay3D triangles
In-Reply-To: <4B9890BC.2070006@univ-provence.fr>
References: <4B9890BC.2070006@univ-provence.fr>
Message-ID: <4B997D3A.1010206@bigpond.net.au>

Hi Boris,

Take a look at the Delaunay graph example in the online Mayavi 
documentation here:
<http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_delaunay_graph.html#example-delaunay-graph>
I think it does what you're asking. If not, examine the data returned by 
the mlab.pipeline.delaunay3d(vtk_source) call.

Also, there's a Mayavi support list here:
<https://mail.enthought.com/mailman/listinfo/enthought-dev>

Gary R.

Bor?s BURLE wrote:
> Dear all,
> 
> I have a set of points coordinates (x,y,z) representing a surface (in 
> 3D, but it's not a volume). I would like to create a mesh from those 
> points, and then map some values on this mesh (colored surface). Mayavi 
> allows to easily represent the mesh. What I have done so far is:
>    
> pts=mlab.points3d(x,y,z)
> delaunay = mlab.pipeline.delaunay3d(pts)
> surf = mlab.pipeline.surface(delaunay)
> 
> That's create a nice plot of my mesh, but of course without values 
> mapped on the surface. Beforehand, I have been using 
> mlab.triangular_mesh to create such a colored surface (from a vtk-like 
> file), and would be happy to use the same approach. However, 
> triangular_mesh requires the x,y,z positions, plus a list  of triplets 
> defining the connecting triangles. So, my question is: how can I extract 
> those triangles from the delaunay3d object created above? This 
> information is very likely stored somewhere, but after hours of 
> exploration, I haven't been able to extract it!!
> 
>     Thank you in advance for your help !!
>           All the best,
>              B.
> 


From dwf at cs.toronto.edu  Fri Mar 12 01:43:18 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 12 Mar 2010 01:43:18 -0500
Subject: [SciPy-User] [SciPy-user] ATLAS install has no effect
In-Reply-To: <1997623173.2812691268343470110.JavaMail.root@zm09.stanford.edu>
References: <1997623173.2812691268343470110.JavaMail.root@zm09.stanford.edu>
Message-ID: <002455CE-D714-4397-B9E7-F0E0B5F84D4D@cs.toronto.edu>

On 11-Mar-10, at 4:37 PM, Ian Goodfellow wrote:

> Sure, that's a good call to know about. It looks like it's not using  
> the same setting for library_dirs as what's in my LD_LIBRARY_PATH.

LD_LIBRARY_PATH would need to have been set at build time, not just at  
run time. AFAIK NumPy will only link ATLAS in statically (I could be  
wrong about this). At the very least it needs to know what BLAS you  
are using at compile time.

A good bet is to use site.cfg to explicitly tell NumPy's build scripts  
where to look for these things.

David


From lafras at sun.ac.za  Fri Mar 12 02:50:23 2010
From: lafras at sun.ac.za (Lafras Uys)
Date: Fri, 12 Mar 2010 09:50:23 +0200
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
Message-ID: <4B99F23F.1090508@sun.ac.za>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>> I need to save a fairly large set of arrays to disk. I have saved it using
>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>> large ;D). When I try to load it using numpy.load, the zipfile module
>> compains about
>> BadZipfile: Bad magic number for file header
>>
>> I can't open it with the normal zip utility present on the system, but it
>> could be that it's barfing about files being larger than 2Gb.
>> Is there some file limit for npzs?
> 
> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
> not yet support the ZIP64 format.
> 
>> Is there anyway I can recover the data (I
>> guess I could try decompressing the file with 7z and extracting the
>> individual npy files?)
> 
> Possibly. However, if the normal zip utility isn't working, 7z
> probably won't, either. Worth a try, though.

I've had similar problems, my solution was to move to HDF5. There are
two options for accessing and working with HDF files from python: h5py
(http://code.google.com/p/h5py/) and pytables
(http://www.pytables.org/). Both packages have built in numpy support.

Regards,

Lafras
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuZ8jcACgkQKUpCd+bV+kruKwCghfG0yAo/eRXzDZxH6i1eOyfn
bnUAoLsuLB2O9qyvJV7CP3jXT9OcMwye
=1xc3
-----END PGP SIGNATURE-----


From david at silveregg.co.jp  Fri Mar 12 03:05:19 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Fri, 12 Mar 2010 17:05:19 +0900
Subject: [SciPy-User] [SciPy-user] ATLAS install has no effect
In-Reply-To: <002455CE-D714-4397-B9E7-F0E0B5F84D4D@cs.toronto.edu>
References: <1997623173.2812691268343470110.JavaMail.root@zm09.stanford.edu>
	<002455CE-D714-4397-B9E7-F0E0B5F84D4D@cs.toronto.edu>
Message-ID: <4B99F5BF.9030100@silveregg.co.jp>

David Warde-Farley wrote:
> On 11-Mar-10, at 4:37 PM, Ian Goodfellow wrote:
> 
>> Sure, that's a good call to know about. It looks like it's not using  
>> the same setting for library_dirs as what's in my LD_LIBRARY_PATH.
> 
> LD_LIBRARY_PATH would need to have been set at build time, not just at  
> run time.

Not exactly: LD_LIBRARY_PATH only change the loader path (that is which 
shared libraries are loaded when a process is started, as well as dlopen).

> AFAIK NumPy will only link ATLAS in statically (I could be  
> wrong about this).

Atlas may be linked dynamically as well.

On debian, ATLAS is packaged such as it also contains a full blas/lapack 
library, that is -llapack -lblas will link against ATLAS library. In 
particular, you can linked against conventional, slow, blas/lapack when 
building, and switching to a faster, ATLAS-enabled one - but this is 
debian specific, and not supported out of the box by the ATLAS build 
system. You can of course emulate what debian does by yourself if you 
know what you are doing.

cheers,

David


From ia3n at stanford.edu  Fri Mar 12 04:16:03 2010
From: ia3n at stanford.edu (Ian Goodfellow)
Date: Fri, 12 Mar 2010 01:16:03 -0800
Subject: [SciPy-User] [SciPy-user] ATLAS install has no effect
In-Reply-To: <4B99F5BF.9030100@silveregg.co.jp>
References: <1997623173.2812691268343470110.JavaMail.root@zm09.stanford.edu>
	<002455CE-D714-4397-B9E7-F0E0B5F84D4D@cs.toronto.edu>
	<4B99F5BF.9030100@silveregg.co.jp>
Message-ID: <8637424C-1066-42B2-988C-5C7F3EF6EB47@stanford.edu>

Actually, David (Cournapeau), I started off by installing your rpms  
linked at the top of the "Fedora Core 8, openSUSE 10.2, RHEL/Centos 5"  
section, so you're probably the perfect person to answer this. Should  
your rpm installation work with ATLAS or do I need to build numpy from  
source?
Are there any instructions for building atlas from the ashigabou  
repository and then building numpy to use it?
Thanks,
Ian
On Mar 12, 2010, at 12:05 AM, David Cournapeau wrote:

> David Warde-Farley wrote:
>> On 11-Mar-10, at 4:37 PM, Ian Goodfellow wrote:
>>
>>> Sure, that's a good call to know about. It looks like it's not using
>>> the same setting for library_dirs as what's in my LD_LIBRARY_PATH.
>>
>> LD_LIBRARY_PATH would need to have been set at build time, not just  
>> at
>> run time.
>
> Not exactly: LD_LIBRARY_PATH only change the loader path (that is  
> which
> shared libraries are loaded when a process is started, as well as  
> dlopen).
>
>> AFAIK NumPy will only link ATLAS in statically (I could be
>> wrong about this).
>
> Atlas may be linked dynamically as well.
>
> On debian, ATLAS is packaged such as it also contains a full blas/ 
> lapack
> library, that is -llapack -lblas will link against ATLAS library. In
> particular, you can linked against conventional, slow, blas/lapack  
> when
> building, and switching to a faster, ATLAS-enabled one - but this is
> debian specific, and not supported out of the box by the ATLAS build
> system. You can of course emulate what debian does by yourself if you
> know what you are doing.
>
> cheers,
>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From robince at gmail.com  Fri Mar 12 05:38:22 2010
From: robince at gmail.com (Robin)
Date: Fri, 12 Mar 2010 10:38:22 +0000
Subject: [SciPy-User] [SciPy-user] Maximum entropy distribution for
	Ising model - setup?
In-Reply-To: <COL112-W31681BFCC29A87F9AAEFC5F06B0@phx.gbl>
References: <COL112-W17E2252F9AE97A7D4E44BDF0710@phx.gbl>
	<COL112-W30F12BEFD6DAA200F7DABFF0710@phx.gbl>
	<32EE5911-8078-4F1B-887E-00FBE702057E@gmail.com>
	<COL112-W31681BFCC29A87F9AAEFC5F06B0@phx.gbl>
Message-ID: <2d5132a51003120238n593ca70n7a4e080f319f1fa2@mail.gmail.com>

Hi,

I apologise for the extended delay for this reply - I originally wrote
it while I was on holiday, but there was something funny with the
coupling/theta values in my example that I couldn't get to the bottom
of in time and I'm afraid I forgot it about it...

Now I come to ask the scipy list another question of my own I thought
I should really first answer this! Hopefully it is still of some use.

On Wed, Jan 13, 2010 at 3:42 AM, Jordi Molins Coronado
<jordi_molins at hotmail.com> wrote:
> Hello,
> I find all the ideas posted in reply to my message very interesting, thank
> you very much to all who have answered to my question. Especially, I would
> like to know more about Kilian's and Robin's suggestions.
> In particular, I find difficult to understand and translate the ideas posted
> by them into my background. Of course, this is not Kilian's or Robin's
> fault, but my complete fault due to lack of knowledge.
> To Robin:
> - Is there a paper covering your package, but explained in layman's terms,
> not requiring previous knowledge on the subject? Or maybe a simple but
> fully-worked example (ideally closely related to the Ising model) that can
> be used in your package to see how everything works.

The original paper describing the toolbox and the maximum entropy
algorithm is here:
http://www.frontiersin.org/neuroscience/neuroinformatics/paper/10.3389/neuro.11/004.2009/
but I suppose it is pretty focussed on our field. I hope the maximum
entropy section might make sense on its own though.
The algorithm is based on information geometry of Amari which is
covered in this paper, which is a bit heavy going but not particularly
field specific:
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=930911

For an example:

# start with 4 binary variables - 1000 trials
X = np.random.random_integers(0,1,(4,1000))
# convert binary 4-vector to a decimal value for each trial
import pyentropy
X_d = pyentropy.decimalise(X,4,2)
# sample full joint probability distribution (16 possible values)
p = pyentropy.prob(X_d, 16)
# this p vector is ordered so that P(x=(1,0,1,0)) is found at index
with decimal #value of the word, ie p[1] = P(x=(0,0,0,1)), p[10] =
P(x=(1,0,1,0))

# obtain maximum entropy solution
from pyentropy.maxent import AmariSolve
s = AmariSolve(4,2)
# solve for maximum entropy distribution preserving up to second
# order marginals ( = ising for binary variables)
p_me = s.solve(p, k=2)

# theta vector (which is h, J's in different notation)
# this should be covered in the paper
# this is length 15 (no zero entry)
theta = s.theta_from_p(p_me)
# theta[:4] are the first order thetas (the h's)
# theta[4:10] are the second order thetas (the J's) (I would have to think
# a bit harder to get the exact ordering if you are interested in that)
# theta[10:] are the higher order terms which should be numerically
zero (ie #10^-16 or thereabouts)

# marginals
eta = s.eta_from_p(p_me)
# these are the corresponding ordered marginals
# so from contraints (s.eta_from_p(p) - s.eta_from_p(p_me))[:10] is close
# to zero (first and second order marignals are preserved)

If you don't want to sample the full space you can provide just the
vector of marginal values to the solve function... ie:

s.solve(s.eta_from_p(p),k=2, eta_given=True)
Or even
marg_constraints = s.eta_from_p(p)[:10]
s.solve(r_[marg_constraints,zeros(5)],k=2, eta_given=True)
since the higher order marginals don't make any difference.
(here marg_constraints would be a vector <S1>..<S4> then <SiSj> etc.)

Please let me know how you get on. Of course the major limitation with
this is that you can only do small populations of variables (up to
18/20 should be comfortable) - whereas the probabilistic flow method
looks very promising for much larger populations. I would be really
interested to see if the Ising model algorithm would be extendable to
the more general finite alphabet, any order maximum entropy solution.

In my work to date it hasn't been a practical problem though - since I
am interested in entropy and information values, which are impossible
to sample on much bigger spaces. For me the limitation has always been
the available data rather than the computation time for the maximum
entropy solutions.

Cheers

Robin


From josephsmidt at gmail.com  Fri Mar 12 09:59:54 2010
From: josephsmidt at gmail.com (Joseph Smidt)
Date: Fri, 12 Mar 2010 06:59:54 -0800
Subject: [SciPy-User] 3D Interpolation For Uneven Data?
Message-ID: <142682e11003120659h4924e5ebmf41ad599b42bd173@mail.gmail.com>

     I have 3D data *not* equally-spaced.  Is there a routine in scipy
that will interpolate this data?  It seems like the only higher
dimensional routine I can find requires even;y spaced data?
http://www.scipy.org/Cookbook/Interpolation.  Thanks.

           Joseph Smidt

-- 
------------------------------------------------------------------------
Joseph Smidt <josephsmidt at gmail.com>

Physics and Astronomy
4129 Frederick Reines Hall
Irvine, CA 92697-4575
Office: 949-824-3269


From robert.kern at gmail.com  Fri Mar 12 10:08:59 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 12 Mar 2010 09:08:59 -0600
Subject: [SciPy-User] 3D Interpolation For Uneven Data?
In-Reply-To: <142682e11003120659h4924e5ebmf41ad599b42bd173@mail.gmail.com>
References: <142682e11003120659h4924e5ebmf41ad599b42bd173@mail.gmail.com>
Message-ID: <3d375d731003120708nd00954nb0d489387efea9b6@mail.gmail.com>

On Fri, Mar 12, 2010 at 08:59, Joseph Smidt <josephsmidt at gmail.com> wrote:
> ? ? I have 3D data *not* equally-spaced. ?Is there a routine in scipy
> that will interpolate this data? ?It seems like the only higher
> dimensional routine I can find requires even;y spaced data?
> http://www.scipy.org/Cookbook/Interpolation. ?Thanks.

http://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.Rbf.html#scipy.interpolate.Rbf

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From robince at gmail.com  Fri Mar 12 10:46:33 2010
From: robince at gmail.com (Robin)
Date: Fri, 12 Mar 2010 15:46:33 +0000
Subject: [SciPy-User] OT: advice on modelling a time series
Message-ID: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>

Hello,

While not directly Python related I am always impressed with the
quality of scientific advice available on this list, and was hoping I
could receive some...

I have a limited amount of an experimentally obtained time series from
a biological system. I would like to come up with a generative model
which would allow production of large quantities of data with
statistics as similar as possible to the experimental data. The time
series represents a position, and I am particularly interested in
transient high velocity/acceleration events (which are often not very
visible by eye in the position trace), so ideally any model should
reproduce those with particular care.

An example plot of a small section of the data (pos vel and acc) (1s)
is available here:
http://i41.tinypic.com/ou42de.jpg

If it makes any difference it is sampled at 4kHz. I tried fitting a
basic autoregressive model. An order 38 model reproduced the position
signal visually quite well, but velocity and acceleration were far too
regular. I tried fitting one to the velocity, but I think the events
of interest are too far apart in bins so the order required is too
large.

So, could anyone point me to anything that would be helpful in python
(so far I did the AR with a matlab package I found)? Also any
suggestions for how to proceed would be great - other than reading the
wikipedia article I am completely new to this type of AR modelling. So
far the only ideas I have involve either downsampling the signal (to
try to reduce the order of AR model needed), or splitting it in
frequency to low f/high f components and attempting to model them
separately then recombine. Do either of these seem sensible?

Is it likely some non-linear model would be required (pos,vel and acc
all have high kurtosis), or are normal AR models capable of recreating
this kind of fine structure if tweaked sufficiently?

 Thanks in advance for any pointers,

Cheers

Robin


From robert.kern at gmail.com  Fri Mar 12 11:26:27 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 12 Mar 2010 10:26:27 -0600
Subject: [SciPy-User] OT: advice on modelling a time series
In-Reply-To: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>
References: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>
Message-ID: <3d375d731003120826y187a4c3kb5264e0615e54dc8@mail.gmail.com>

On Fri, Mar 12, 2010 at 09:46, Robin <robince at gmail.com> wrote:

> So, could anyone point me to anything that would be helpful in python
> (so far I did the AR with a matlab package I found)? Also any
> suggestions for how to proceed would be great - other than reading the
> wikipedia article I am completely new to this type of AR modelling. So
> far the only ideas I have involve either downsampling the signal (to
> try to reduce the order of AR model needed), or splitting it in
> frequency to low f/high f components and attempting to model them
> separately then recombine. Do either of these seem sensible?

The econometricians here will probably have more to add here, but I
suspect that you could use an ARCH or GARCH model:

  http://en.wikipedia.org/wiki/Autoregressive_conditional_heteroskedasticity

Josef was working on implementing GARCH, but I don't think he
finished. There is MATLAB code:

  http://www.kevinsheppard.com/wiki/MFE_Toolbox

And R code:

  http://cran.r-project.org/web/views/Finance.html

> Is it likely some non-linear model would be required (pos,vel and acc
> all have high kurtosis), or are normal AR models capable of recreating
> this kind of fine structure if tweaked sufficiently?

It is possible that once you allow heterskedasticity, you can account
for the marginal kurtosis.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From agile.aspect at gmail.com  Fri Mar 12 11:28:32 2010
From: agile.aspect at gmail.com (Agile Aspect)
Date: Fri, 12 Mar 2010 08:28:32 -0800
Subject: [SciPy-User] Undefined symbol: clapack_sgesv when doing import
	in Python
In-Reply-To: <f99dcccd1003111010p7e05517bg480116b9d6a2f620@mail.gmail.com>
References: <f99dcccd1003111010p7e05517bg480116b9d6a2f620@mail.gmail.com>
Message-ID: <c6a6f63b1003120828m7655db4ch4fcd6a5fcd2d7d0a@mail.gmail.com>

On Thu, Mar 11, 2010 at 10:10 AM, Diederik van Liere
<dvanliere at gmail.com> wrote:
>>> ImportError:
>>> /usr/local/lib/python2.6/site-packages/scipy/linalg/clapack.so:
>>> undefined symbol: clapack_sgesv
>
>
>> You should use nm to check for the symbol in your ATLAS libraries to
>> see if it's really missing or if numpy just can't find it.
>
>> Try
>
>> nm -A  /usr/local/atlas/lib/lib*.so | grep clapack_sgesv
>
>> or just
>
>> nm liblapack.so | grep clapack_sgesv
>
> This is my output:
> liblapack.a:clapack_sgesv.o:00000000 r .LC0
> liblapack.a:clapack_sgesv.o:00000000 r .LC1
> liblapack.a:clapack_sgesv.o:0000002c r .LC2
> liblapack.a:clapack_sgesv.o:0000005c r .LC3
> liblapack.a:clapack_sgesv.o:00000090 r .LC4
> liblapack.a:clapack_sgesv.o:000000b8 r .LC5
> liblapack.a:clapack_sgesv.o:         U ATL_sgetrf
> liblapack.a:clapack_sgesv.o:         U ATL_sgetrs
> liblapack.a:clapack_sgesv.o:         U _GLOBAL_OFFSET_TABLE_
> liblapack.a:clapack_sgesv.o:00000000 T __i686.get_pc_thunk.bx
> liblapack.a:clapack_sgesv.o:         U cblas_xerbla
> liblapack.a:clapack_sgesv.o:00000000 T clapack_sgesv
> I don't have .so files, I have .a files, could that be an issue? So it looks
> that ATLAS is properly compiled, right?

I presume it has to be shared (it's never worked for me otherwise.)

It looks like it might not have been compiled with

    -fPIC

which is necessary to build shared libraries.

If you built the libraries from source, you should be able to

   make shared
   make ptshared

which would included the -fPIC compile option and create the shared
libraries for you.


>> If the symbol is defined in your ATLAS libraries, then you should post
>> your site.cfg from numpy.
>
> My site.cfg from numpy reads:
> [DEFAULT]
> library_dirs = /usr/local/atlas/lib/
> include_dirs = /usr/local/atlas/include/
> [blas_opt]
> libraries = lapack, f77blas, cblas, atlas
> [lapack_opt]
> libraries = lapack, f77blas, cblas, atlas
>

Looks good.

-- 
      Enjoy global warming while it lasts.


From josef.pktd at gmail.com  Fri Mar 12 11:32:38 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 12 Mar 2010 11:32:38 -0500
Subject: [SciPy-User] OT: advice on modelling a time series
In-Reply-To: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>
References: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>
Message-ID: <1cd32cbb1003120832g3eda14f1g3cda7dfb94bfc740@mail.gmail.com>

On Fri, Mar 12, 2010 at 10:46 AM, Robin <robince at gmail.com> wrote:
> Hello,
>
> While not directly Python related I am always impressed with the
> quality of scientific advice available on this list, and was hoping I
> could receive some...
>
> I have a limited amount of an experimentally obtained time series from
> a biological system. I would like to come up with a generative model
> which would allow production of large quantities of data with
> statistics as similar as possible to the experimental data. The time
> series represents a position, and I am particularly interested in
> transient high velocity/acceleration events (which are often not very
> visible by eye in the position trace), so ideally any model should
> reproduce those with particular care.
>
> An example plot of a small section of the data (pos vel and acc) (1s)
> is available here:
> http://i41.tinypic.com/ou42de.jpg
>
> If it makes any difference it is sampled at 4kHz. I tried fitting a
> basic autoregressive model. An order 38 model reproduced the position
> signal visually quite well, but velocity and acceleration were far too
> regular. I tried fitting one to the velocity, but I think the events
> of interest are too far apart in bins so the order required is too
> large.
>
> So, could anyone point me to anything that would be helpful in python
> (so far I did the AR with a matlab package I found)? Also any
> suggestions for how to proceed would be great - other than reading the
> wikipedia article I am completely new to this type of AR modelling. So
> far the only ideas I have involve either downsampling the signal (to
> try to reduce the order of AR model needed), or splitting it in
> frequency to low f/high f components and attempting to model them
> separately then recombine. Do either of these seem sensible?
>
> Is it likely some non-linear model would be required (pos,vel and acc
> all have high kurtosis), or are normal AR models capable of recreating
> this kind of fine structure if tweaked sufficiently?
>
> ?Thanks in advance for any pointers,

In statsmodels we are working on some time series analysis, but it is
still a bit to early for real use. We have AR, but for this kind of
data I would recommend scikits.talkbox which has a Levinson-Durbin
recursion implemented that gives a more robust estimate of longer AR
polynomial (maybe nitime also has it now.)

I don't know of any implementation of non-linear models for time
series analysis in python, e.g. a markov switching or threshold model,
or of any models that would allow for fat-tailes or asymmetric shock
distributions.

If you just want to generate sample data with similar features, then
this will be much easier than estimation. (I have some tentative
simulation code for continuous time diffusion processes but not
cleaned up)

Your acceleration data looks like a GARCH process, that is the
variance is autocorrelated but not (much) the mean.

There also, I have an initial version but not yet good enough to be reliable.

>From the graph, it also looks like the three observations are strongly
related, so separate (univariate) modeling doesn't look like the most
appropriate choice.

Josef

>
> Cheers
>
> Robin
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Fri Mar 12 11:46:46 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 12 Mar 2010 11:46:46 -0500
Subject: [SciPy-User] OT: advice on modelling a time series
In-Reply-To: <1cd32cbb1003120832g3eda14f1g3cda7dfb94bfc740@mail.gmail.com>
References: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>
	<1cd32cbb1003120832g3eda14f1g3cda7dfb94bfc740@mail.gmail.com>
Message-ID: <1cd32cbb1003120846g1e721f8fp2b978b01149bdf90@mail.gmail.com>

On Fri, Mar 12, 2010 at 11:32 AM,  <josef.pktd at gmail.com> wrote:
> On Fri, Mar 12, 2010 at 10:46 AM, Robin <robince at gmail.com> wrote:
>> Hello,
>>
>> While not directly Python related I am always impressed with the
>> quality of scientific advice available on this list, and was hoping I
>> could receive some...
>>
>> I have a limited amount of an experimentally obtained time series from
>> a biological system. I would like to come up with a generative model
>> which would allow production of large quantities of data with
>> statistics as similar as possible to the experimental data. The time
>> series represents a position, and I am particularly interested in
>> transient high velocity/acceleration events (which are often not very
>> visible by eye in the position trace), so ideally any model should
>> reproduce those with particular care.
>>
>> An example plot of a small section of the data (pos vel and acc) (1s)
>> is available here:
>> http://i41.tinypic.com/ou42de.jpg
>>
>> If it makes any difference it is sampled at 4kHz. I tried fitting a
>> basic autoregressive model. An order 38 model reproduced the position
>> signal visually quite well, but velocity and acceleration were far too
>> regular. I tried fitting one to the velocity, but I think the events
>> of interest are too far apart in bins so the order required is too
>> large.
>>
>> So, could anyone point me to anything that would be helpful in python
>> (so far I did the AR with a matlab package I found)? Also any
>> suggestions for how to proceed would be great - other than reading the
>> wikipedia article I am completely new to this type of AR modelling. So
>> far the only ideas I have involve either downsampling the signal (to
>> try to reduce the order of AR model needed), or splitting it in
>> frequency to low f/high f components and attempting to model them
>> separately then recombine. Do either of these seem sensible?
>>
>> Is it likely some non-linear model would be required (pos,vel and acc
>> all have high kurtosis), or are normal AR models capable of recreating
>> this kind of fine structure if tweaked sufficiently?
>>
>> ?Thanks in advance for any pointers,
>
> In statsmodels we are working on some time series analysis, but it is
> still a bit to early for real use. We have AR, but for this kind of
> data I would recommend scikits.talkbox which has a Levinson-Durbin
> recursion implemented that gives a more robust estimate of longer AR
> polynomial (maybe nitime also has it now.)
>
> I don't know of any implementation of non-linear models for time
> series analysis in python, e.g. a markov switching or threshold model,
> or of any models that would allow for fat-tailes or asymmetric shock
> distributions.
>
> If you just want to generate sample data with similar features, then
> this will be much easier than estimation. (I have some tentative
> simulation code for continuous time diffusion processes but not
> cleaned up)
>
> Your acceleration data looks like a GARCH process, that is the
> variance is autocorrelated but not (much) the mean.
>
> There also, I have an initial version but not yet good enough to be reliable.
>
> From the graph, it also looks like the three observations are strongly
> related, so separate (univariate) modeling doesn't look like the most
> appropriate choice.

from looking at the graphs:

If acceleration is an independent GARCH process, then velocity would
just be the integral (plus noise), isn't it?
Position should also be heteroscedastic, because the change in
position will depend on the velocity, but it looks mean reverting, so
it might be highly autocorrelated in mean with variance given by
velocity, or maybe with an MA error if the position doesn't change
direction too suddenly. (I'm not sure about how the steering wheel
should be modeled)

Josef



>
> Josef
>
>>
>> Cheers
>>
>> Robin
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>


From robert.kern at gmail.com  Fri Mar 12 12:01:45 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 12 Mar 2010 11:01:45 -0600
Subject: [SciPy-User] OT: advice on modelling a time series
In-Reply-To: <1cd32cbb1003120846g1e721f8fp2b978b01149bdf90@mail.gmail.com>
References: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com> 
	<1cd32cbb1003120832g3eda14f1g3cda7dfb94bfc740@mail.gmail.com> 
	<1cd32cbb1003120846g1e721f8fp2b978b01149bdf90@mail.gmail.com>
Message-ID: <3d375d731003120901l3a5cf8d0y4caa4ec16d544bf0@mail.gmail.com>

On Fri, Mar 12, 2010 at 10:46,  <josef.pktd at gmail.com> wrote:
> On Fri, Mar 12, 2010 at 11:32 AM, ?<josef.pktd at gmail.com> wrote:

>> From the graph, it also looks like the three observations are strongly
>> related, so separate (univariate) modeling doesn't look like the most
>> appropriate choice.
>
> from looking at the graphs:
>
> If acceleration is an independent GARCH process, then velocity would
> just be the integral (plus noise), isn't it?

Robin will have to confirm, but I suspect that only the position was
actually measured and that he derived the velocity and acceleration
from the position time series numerically.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Fri Mar 12 12:10:51 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 12 Mar 2010 12:10:51 -0500
Subject: [SciPy-User] OT: advice on modelling a time series
In-Reply-To: <3d375d731003120901l3a5cf8d0y4caa4ec16d544bf0@mail.gmail.com>
References: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>
	<1cd32cbb1003120832g3eda14f1g3cda7dfb94bfc740@mail.gmail.com>
	<1cd32cbb1003120846g1e721f8fp2b978b01149bdf90@mail.gmail.com>
	<3d375d731003120901l3a5cf8d0y4caa4ec16d544bf0@mail.gmail.com>
Message-ID: <1cd32cbb1003120910v74e402a4sbdada3dd9b195882@mail.gmail.com>

On Fri, Mar 12, 2010 at 12:01 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Fri, Mar 12, 2010 at 10:46, ?<josef.pktd at gmail.com> wrote:
>> On Fri, Mar 12, 2010 at 11:32 AM, ?<josef.pktd at gmail.com> wrote:
>
>>> From the graph, it also looks like the three observations are strongly
>>> related, so separate (univariate) modeling doesn't look like the most
>>> appropriate choice.
>>
>> from looking at the graphs:
>>
>> If acceleration is an independent GARCH process, then velocity would
>> just be the integral (plus noise), isn't it?
>
> Robin will have to confirm, but I suspect that only the position was
> actually measured and that he derived the velocity and acceleration
> from the position time series numerically.

In that case it might not be to difficult to go in reverse, from
acceleration to position. From the graph, I would think that
acceleration is the random input for the process. Maybe with some
adjustments to correct for specification errors.

Josef

>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From robince at gmail.com  Fri Mar 12 12:16:47 2010
From: robince at gmail.com (Robin)
Date: Fri, 12 Mar 2010 17:16:47 +0000
Subject: [SciPy-User] OT: advice on modelling a time series
In-Reply-To: <1cd32cbb1003120910v74e402a4sbdada3dd9b195882@mail.gmail.com>
References: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>
	<1cd32cbb1003120832g3eda14f1g3cda7dfb94bfc740@mail.gmail.com>
	<1cd32cbb1003120846g1e721f8fp2b978b01149bdf90@mail.gmail.com>
	<3d375d731003120901l3a5cf8d0y4caa4ec16d544bf0@mail.gmail.com>
	<1cd32cbb1003120910v74e402a4sbdada3dd9b195882@mail.gmail.com>
Message-ID: <2d5132a51003120916l538d6226wa829cf3a8f492129@mail.gmail.com>

On Fri, Mar 12, 2010 at 5:10 PM,  <josef.pktd at gmail.com> wrote:
> On Fri, Mar 12, 2010 at 12:01 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Fri, Mar 12, 2010 at 10:46, ?<josef.pktd at gmail.com> wrote:
>>> On Fri, Mar 12, 2010 at 11:32 AM, ?<josef.pktd at gmail.com> wrote:
>>
>>>> From the graph, it also looks like the three observations are strongly
>>>> related, so separate (univariate) modeling doesn't look like the most
>>>> appropriate choice.
>>>
>>> from looking at the graphs:
>>>
>>> If acceleration is an independent GARCH process, then velocity would
>>> just be the integral (plus noise), isn't it?
>>
>> Robin will have to confirm, but I suspect that only the position was
>> actually measured and that he derived the velocity and acceleration
>> from the position time series numerically.
>
> In that case it might not be to difficult to go in reverse, from
> acceleration to position. From the graph, I would think that
> acceleration is the random input for the process. Maybe with some
> adjustments to correct for specification errors.

Right - I am primarily working with position - vel and acc are shown
just to illustrate the features I am hoping to preserve/replicate (ie
transient high acceleration events which aren't really visible in the
position trace, even when blown up quite a lot). I did the
differentiation with basic finite differences (diff), smoothed with a
1ms (4bin moving average) (this was the method used by others
previously).

Arguably acceleration is the most important of the representations
(this is part of the hypothesis we are planning to test) so I agree
starting from acceleration and integrating to get the actual position
to use could be a good idea.

In the mean time I have to look up a lot of the other things you
mentioned before I have anything sensible to add (heteroscedastic is a
new one on me and I will look up GARCH and ARCH also) .

Thanks very much for the quick and useful responses!

Cheers

Robin


From paul.anton.letnes at gmail.com  Fri Mar 12 12:22:55 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Fri, 12 Mar 2010 09:22:55 -0800
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <4B99F23F.1090508@sun.ac.za>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<4B99F23F.1090508@sun.ac.za>
Message-ID: <6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>


On 11. mars 2010, at 23.50, Lafras Uys wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
>>> I need to save a fairly large set of arrays to disk. I have saved it using
>>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>>> large ;D). When I try to load it using numpy.load, the zipfile module
>>> compains about
>>> BadZipfile: Bad magic number for file header
>>> 
>>> I can't open it with the normal zip utility present on the system, but it
>>> could be that it's barfing about files being larger than 2Gb.
>>> Is there some file limit for npzs?
>> 
>> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
>> not yet support the ZIP64 format.
>> 
>>> Is there anyway I can recover the data (I
>>> guess I could try decompressing the file with 7z and extracting the
>>> individual npy files?)
>> 
>> Possibly. However, if the normal zip utility isn't working, 7z
>> probably won't, either. Worth a try, though.
> 
> I've had similar problems, my solution was to move to HDF5. There are
> two options for accessing and working with HDF files from python: h5py
> (http://code.google.com/p/h5py/) and pytables
> (http://www.pytables.org/). Both packages have built in numpy support.
> 
> Regards,
> Lafras

I've experienced similar issues too, but I moved to NetCDF. The only disadvantage was that I did not find any python modules that work well _and_ support numpy. Hence, I am considering moving to HDF5. Which python module would people here recommend? (Or, alternatively, did I miss a great netCDF python module that someone could tell me about?)

Cheers,
Paul.

From gokhansever at gmail.com  Fri Mar 12 12:29:58 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Fri, 12 Mar 2010 11:29:58 -0600
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
Message-ID: <49d6b3501003120929o610c6ab0j6e2e72e55cdfa93@mail.gmail.com>

On Fri, Mar 12, 2010 at 11:22 AM, Paul Anton Letnes <
paul.anton.letnes at gmail.com> wrote:

>
> On 11. mars 2010, at 23.50, Lafras Uys wrote:
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> >>> I need to save a fairly large set of arrays to disk. I have saved it
> using
> >>> numpy.savez, and the resulting file is around 11Gb (yes, I did say
> fairly
> >>> large ;D). When I try to load it using numpy.load, the zipfile module
> >>> compains about
> >>> BadZipfile: Bad magic number for file header
> >>>
> >>> I can't open it with the normal zip utility present on the system, but
> it
> >>> could be that it's barfing about files being larger than 2Gb.
> >>> Is there some file limit for npzs?
> >>
> >> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
> >> not yet support the ZIP64 format.
> >>
> >>> Is there anyway I can recover the data (I
> >>> guess I could try decompressing the file with 7z and extracting the
> >>> individual npy files?)
> >>
> >> Possibly. However, if the normal zip utility isn't working, 7z
> >> probably won't, either. Worth a try, though.
> >
> > I've had similar problems, my solution was to move to HDF5. There are
> > two options for accessing and working with HDF files from python: h5py
> > (http://code.google.com/p/h5py/) and pytables
> > (http://www.pytables.org/). Both packages have built in numpy support.
> >
> > Regards,
> > Lafras
>
> I've experienced similar issues too, but I moved to NetCDF. The only
> disadvantage was that I did not find any python modules that work well _and_
> support numpy. Hence, I am considering moving to HDF5. Which python module
> would people here recommend? (Or, alternatively, did I miss a great netCDF
> python module that someone could tell me about?)
>
> Cheers,
> Paul.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>

There is http://code.google.com/p/netcdf4-python/

I know netcdf4 is a subset of HDF5. What advantages there to use HDF5 not
NetCDF4 ?


-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100312/2dd4c3ca/attachment.html>

From kwgoodman at gmail.com  Fri Mar 12 12:30:19 2010
From: kwgoodman at gmail.com (Keith Goodman)
Date: Fri, 12 Mar 2010 09:30:19 -0800
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
Message-ID: <f4f93d421003120930y4c5336f2gdf234ac634c24964@mail.gmail.com>

On Fri, Mar 12, 2010 at 9:22 AM, Paul Anton Letnes
<paul.anton.letnes at gmail.com> wrote:
> I've experienced similar issues too, but I moved to NetCDF. The only disadvantage was that I did not find any python modules that work well _and_ support numpy. Hence, I am considering moving to HDF5. Which python module would people here recommend? (Or, alternatively, did I miss a great netCDF python module that someone could tell me about?)
>
> Cheers,
> Paul.

I use h5py. I think it is great. It gives you a dictionary-like
interface to your archive. Here's a quick example:

>> import h5py
>> a = np.random.rand(1000,1000)
>> f = h5py.File('/tmp/myfile.hdf5')
>> f['a'] = a  # <-- Save
>> f.keys()
   ['a']
>> f.filename
   '/tmp/myfile.hdf5'
>> b = f['a']  # <-- Load


From josef.pktd at gmail.com  Fri Mar 12 12:53:12 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 12 Mar 2010 12:53:12 -0500
Subject: [SciPy-User] OT: advice on modelling a time series
In-Reply-To: <2d5132a51003120916l538d6226wa829cf3a8f492129@mail.gmail.com>
References: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>
	<1cd32cbb1003120832g3eda14f1g3cda7dfb94bfc740@mail.gmail.com>
	<1cd32cbb1003120846g1e721f8fp2b978b01149bdf90@mail.gmail.com>
	<3d375d731003120901l3a5cf8d0y4caa4ec16d544bf0@mail.gmail.com>
	<1cd32cbb1003120910v74e402a4sbdada3dd9b195882@mail.gmail.com>
	<2d5132a51003120916l538d6226wa829cf3a8f492129@mail.gmail.com>
Message-ID: <1cd32cbb1003120953o2fb97213j4724d8a422101ca4@mail.gmail.com>

On Fri, Mar 12, 2010 at 12:16 PM, Robin <robince at gmail.com> wrote:
> On Fri, Mar 12, 2010 at 5:10 PM, ?<josef.pktd at gmail.com> wrote:
>> On Fri, Mar 12, 2010 at 12:01 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Fri, Mar 12, 2010 at 10:46, ?<josef.pktd at gmail.com> wrote:
>>>> On Fri, Mar 12, 2010 at 11:32 AM, ?<josef.pktd at gmail.com> wrote:
>>>
>>>>> From the graph, it also looks like the three observations are strongly
>>>>> related, so separate (univariate) modeling doesn't look like the most
>>>>> appropriate choice.
>>>>
>>>> from looking at the graphs:
>>>>
>>>> If acceleration is an independent GARCH process, then velocity would
>>>> just be the integral (plus noise), isn't it?
>>>
>>> Robin will have to confirm, but I suspect that only the position was
>>> actually measured and that he derived the velocity and acceleration
>>> from the position time series numerically.
>>
>> In that case it might not be to difficult to go in reverse, from
>> acceleration to position. From the graph, I would think that
>> acceleration is the random input for the process. Maybe with some
>> adjustments to correct for specification errors.
>
> Right - I am primarily working with position - vel and acc are shown
> just to illustrate the features I am hoping to preserve/replicate (ie
> transient high acceleration events which aren't really visible in the
> position trace, even when blown up quite a lot). I did the
> differentiation with basic finite differences (diff), smoothed with a
> 1ms (4bin moving average) (this was the method used by others
> previously).
>
> Arguably acceleration is the most important of the representations
> (this is part of the hypothesis we are planning to test) so I agree
> starting from acceleration and integrating to get the actual position
> to use could be a good idea.
>
> In the mean time I have to look up a lot of the other things you
> mentioned before I have anything sensible to add (heteroscedastic is a
> new one on me and I will look up GARCH and ARCH also) .
>
> Thanks very much for the quick and useful responses!


If you have rpy and R installed, you can try something like this on
your position data, to see whether GARCH is helpful (ret is my 1d
numpy.array with the time series)

from rpy import r
r.library('fGarch')

# pure garch on mean corrected data
f = r.formula('~garch(1, 1)')
fit = r.garchFit(f, data = ret - ret.mean(), include_mean=False)

#ARMA in mean, GARCH in variance
f = r.formula('~arma(1,1) + ~garch(1, 1)')
fit = r.garchFit(f, data = ret)

I haven't figured out how to do many of the options for GARCH with rpy.

Josef


>
> Cheers
>
> Robin
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From paul.anton.letnes at gmail.com  Fri Mar 12 13:18:18 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Fri, 12 Mar 2010 10:18:18 -0800
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <49d6b3501003120929o610c6ab0j6e2e72e55cdfa93@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
	<49d6b3501003120929o610c6ab0j6e2e72e55cdfa93@mail.gmail.com>
Message-ID: <45A85C73-A7BD-4765-AD9F-0523AF3B7B20@gmail.com>


On 12. mars 2010, at 09.29, G?khan Sever wrote:

> 
> 
> On Fri, Mar 12, 2010 at 11:22 AM, Paul Anton Letnes <paul.anton.letnes at gmail.com> wrote:
> 
> On 11. mars 2010, at 23.50, Lafras Uys wrote:
> 
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> >>> I need to save a fairly large set of arrays to disk. I have saved it using
> >>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
> >>> large ;D). When I try to load it using numpy.load, the zipfile module
> >>> compains about
> >>> BadZipfile: Bad magic number for file header
> >>>
> >>> I can't open it with the normal zip utility present on the system, but it
> >>> could be that it's barfing about files being larger than 2Gb.
> >>> Is there some file limit for npzs?
> >>
> >> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
> >> not yet support the ZIP64 format.
> >>
> >>> Is there anyway I can recover the data (I
> >>> guess I could try decompressing the file with 7z and extracting the
> >>> individual npy files?)
> >>
> >> Possibly. However, if the normal zip utility isn't working, 7z
> >> probably won't, either. Worth a try, though.
> >
> > I've had similar problems, my solution was to move to HDF5. There are
> > two options for accessing and working with HDF files from python: h5py
> > (http://code.google.com/p/h5py/) and pytables
> > (http://www.pytables.org/). Both packages have built in numpy support.
> >
> > Regards,
> > Lafras
> 
> I've experienced similar issues too, but I moved to NetCDF. The only disadvantage was that I did not find any python modules that work well _and_ support numpy. Hence, I am considering moving to HDF5. Which python module would people here recommend? (Or, alternatively, did I miss a great netCDF python module that someone could tell me about?)
> 
> Cheers,
> Paul.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> There is http://code.google.com/p/netcdf4-python/
> 
> I know netcdf4 is a subset of HDF5. What advantages there to use HDF5 not NetCDF4 ?
> 
> 
> -- 
> G?khan
> _______________________________________________

I don't know any particular advantages of the file format itself. There are, however, several python modules for hdf5 that use numpy. Your suggestion for a netcdf module might be a good one, but it does not build on my system: it does not find the netcdf library, only the hdf5 lib - even if they reside in the same folder... I'll see if it works out eventually!

-Paul



From peridot.faceted at gmail.com  Fri Mar 12 13:27:27 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 12 Mar 2010 13:27:27 -0500
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
Message-ID: <ce557a361003121027u7e177763l37018c4b6cb54ed0@mail.gmail.com>

On 11 March 2010 10:48, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans <jgomezdans at gmail.com> wrote:
>> Hi!
>> I need to save a fairly large set of arrays to disk. I have saved it using
>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>> large ;D). When I try to load it using numpy.load, the zipfile module
>> compains about
>> BadZipfile: Bad magic number for file header
>>
>> I can't open it with the normal zip utility present on the system, but it
>> could be that it's barfing about files being larger than 2Gb.
>> Is there some file limit for npzs?
>
> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
> not yet support the ZIP64 format.

Could it be arranged that an exception is raised when creating a >4GB
.npz file, so people do not find themselves with unrecoverable data?

>> Is there anyway I can recover the data (I
>> guess I could try decompressing the file with 7z and extracting the
>> individual npy files?)
>
> Possibly. However, if the normal zip utility isn't working, 7z
> probably won't, either. Worth a try, though.

If your data is valuable enough ? irreplaceable space mission results,
say ? some careful spelunking in the code combined with some knowledge
about your data might allow you to semi-manually reconstruct it from
the damaged zip file. This would be a long and painful labour, but
would probably produce a python module that supported zip64 as an
incidental side effect.

Anne

> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From PHobson at Geosyntec.com  Fri Mar 12 13:32:07 2010
From: PHobson at Geosyntec.com (PHobson at Geosyntec.com)
Date: Fri, 12 Mar 2010 13:32:07 -0500
Subject: [SciPy-User] OT: advice on modelling a time series
In-Reply-To: <1cd32cbb1003120953o2fb97213j4724d8a422101ca4@mail.gmail.com>
References: <2d5132a51003120746g4685e7fby7795c58fe313a9f0@mail.gmail.com>
	<1cd32cbb1003120832g3eda14f1g3cda7dfb94bfc740@mail.gmail.com>
	<1cd32cbb1003120846g1e721f8fp2b978b01149bdf90@mail.gmail.com>
	<3d375d731003120901l3a5cf8d0y4caa4ec16d544bf0@mail.gmail.com>
	<1cd32cbb1003120910v74e402a4sbdada3dd9b195882@mail.gmail.com>
	<2d5132a51003120916l538d6226wa829cf3a8f492129@mail.gmail.com>
	<1cd32cbb1003120953o2fb97213j4724d8a422101ca4@mail.gmail.com>
Message-ID: <C97487CD346EEA4E80423F51F6D1EB2FA976939A32@EXCHANGE-00.GeoSyntec.net>



> -----Original Message-----
> From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org]
> On Behalf Of josef.pktd at gmail.com
> Sent: Friday, March 12, 2010 9:53 AM
> To: SciPy Users List
> Subject: Re: [SciPy-User] OT: advice on modelling a time series
> If you have rpy and R installed, you can try something like this on
> your position data, to see whether GARCH is helpful (ret is my 1d
> numpy.array with the time series)
> 
> from rpy import r
> r.library('fGarch')
> 
> # pure garch on mean corrected data
> f = r.formula('~garch(1, 1)')
> fit = r.garchFit(f, data = ret - ret.mean(), include_mean=False)
> 
> #ARMA in mean, GARCH in variance
> f = r.formula('~arma(1,1) + ~garch(1, 1)')
> fit = r.garchFit(f, data = ret)
> 
> I haven't figured out how to do many of the options for GARCH with rpy.

I don't know about rpy (classic), but in rpy2, you can  input any R expression as a string.

import rpy2.robjects as ro
R = ro.r

[code that defines x and y in python]

# set the variables up in the R environment
R.globalEnv['xR'] = x # needs to be a list?
R.globalEnv['yR'] = y

Fit = R("some pure R expression using xR and yR")

Probably won't be all that fast, but it'll work.

-paul h.


From robert.kern at gmail.com  Fri Mar 12 13:35:48 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 12 Mar 2010 12:35:48 -0600
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <ce557a361003121027u7e177763l37018c4b6cb54ed0@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com> 
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com> 
	<ce557a361003121027u7e177763l37018c4b6cb54ed0@mail.gmail.com>
Message-ID: <3d375d731003121035p6250ec40oe9b20c45ced8d02e@mail.gmail.com>

On Fri, Mar 12, 2010 at 12:27, Anne Archibald <peridot.faceted at gmail.com> wrote:
> On 11 March 2010 10:48, Robert Kern <robert.kern at gmail.com> wrote:
>> On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans <jgomezdans at gmail.com> wrote:
>>> Hi!
>>> I need to save a fairly large set of arrays to disk. I have saved it using
>>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>>> large ;D). When I try to load it using numpy.load, the zipfile module
>>> compains about
>>> BadZipfile: Bad magic number for file header
>>>
>>> I can't open it with the normal zip utility present on the system, but it
>>> could be that it's barfing about files being larger than 2Gb.
>>> Is there some file limit for npzs?
>>
>> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
>> not yet support the ZIP64 format.
>
> Could it be arranged that an exception is raised when creating a >4GB
> .npz file, so people do not find themselves with unrecoverable data?

If you can arrange it, sure.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From Chris.Barker at noaa.gov  Fri Mar 12 13:42:21 2010
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Fri, 12 Mar 2010 10:42:21 -0800
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <45A85C73-A7BD-4765-AD9F-0523AF3B7B20@gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
	<49d6b3501003120929o610c6ab0j6e2e72e55cdfa93@mail.gmail.com>
	<45A85C73-A7BD-4765-AD9F-0523AF3B7B20@gmail.com>
Message-ID: <4B9A8B0D.1010202@noaa.gov>

Paul Anton Letnes wrote:
> On 12. mars 2010, at 09.29, G?khan Sever wrote:
>> I've experienced similar issues too, but I moved to NetCDF. The
>> only
>> disadvantage was that I did not find any python modules that work well
>> _and_ support numpy. Hence, I am considering moving to HDF5. Which
>> python module would people here recommend? (Or, alternatively, did I
>> miss a great netCDF python module that someone could tell me about?)

yes, numpy support is critical -- why anyone would write a netcdf 
wrapper and not use numpy is beyond me.

>> There is http://code.google.com/p/netcdf4-python/
>>
>> I know netcdf4 is a subset of HDF5. What advantages there to use HDF5 not NetCDF4 ?

The way I think about it is that netcdf is a more structured subset of 
hdf5. If the structure imposed by netcdf works well for your needs, it's 
a good option. There are also things like the CF metadata standard that 
make it easier to exchange data with others.

However if you are using it only to save and reload data for your own 
app --  pytables may be a better bet.

I've found the netcdf4-python package to be robust and have a nice 
interface -- and certainly works well with numpy. My only build issue 
with it are actually with getting hd5 built right, the netcdf part has 
been easier, and the python bindings very easy -- at least on OS-X. 
Windows support is not in as good shape, though if you don't' need 
opendap support, I think there are Windows binaries for netcdf4 you 
could use.

Also, I think the python bindings still support netcd3, if you don't 
need the extra stuff netcdf4 gives you (you may, if you need big files 
-- not sure about that).

If netcdf seems like the right way to go for you, then I'm sure you can 
get netcdf4-python working -- and Jeff Whitaker is very helpful if you 
have trouble.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From bsouthey at gmail.com  Fri Mar 12 13:45:55 2010
From: bsouthey at gmail.com (Bruce Southey)
Date: Fri, 12 Mar 2010 12:45:55 -0600
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <3d375d731003121035p6250ec40oe9b20c45ced8d02e@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<ce557a361003121027u7e177763l37018c4b6cb54ed0@mail.gmail.com>
	<3d375d731003121035p6250ec40oe9b20c45ced8d02e@mail.gmail.com>
Message-ID: <4B9A8BE3.40506@gmail.com>

On 03/12/2010 12:35 PM, Robert Kern wrote:
> On Fri, Mar 12, 2010 at 12:27, Anne Archibald<peridot.faceted at gmail.com>  wrote:
>    
>> On 11 March 2010 10:48, Robert Kern<robert.kern at gmail.com>  wrote:
>>      
>>> On Thu, Mar 11, 2010 at 05:10, Jose Gomez-Dans<jgomezdans at gmail.com>  wrote:
>>>        
>>>> Hi!
>>>> I need to save a fairly large set of arrays to disk. I have saved it using
>>>> numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly
>>>> large ;D). When I try to load it using numpy.load, the zipfile module
>>>> compains about
>>>> BadZipfile: Bad magic number for file header
>>>>
>>>> I can't open it with the normal zip utility present on the system, but it
>>>> could be that it's barfing about files being larger than 2Gb.
>>>> Is there some file limit for npzs?
>>>>          
>>> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
>>> not yet support the ZIP64 format.
>>>        
>> Could it be arranged that an exception is raised when creating a>4GB
>> .npz file, so people do not find themselves with unrecoverable data?
>>      
> If you can arrange it, sure.
>
>    
Hi,
Please note that there is the related ticket 991:
http://projects.scipy.org/numpy/ticket/991

Jose, what do Python version are you using and does the ' normal zip 
utility present on the system' actually support zip64?
You should try seeing if 7z sees it.

Bruce


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100312/83d13ef4/attachment.html>

From dwf at cs.toronto.edu  Fri Mar 12 13:54:57 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Fri, 12 Mar 2010 13:54:57 -0500
Subject: [SciPy-User] [SciPy-user] ATLAS install has no effect
In-Reply-To: <4B99F5BF.9030100@silveregg.co.jp>
References: <1997623173.2812691268343470110.JavaMail.root@zm09.stanford.edu>
	<002455CE-D714-4397-B9E7-F0E0B5F84D4D@cs.toronto.edu>
	<4B99F5BF.9030100@silveregg.co.jp>
Message-ID: <6C620C7D-88FC-4915-94EA-EDD331EDD75E@cs.toronto.edu>

On 12-Mar-10, at 3:05 AM, David Cournapeau wrote:

>> LD_LIBRARY_PATH would need to have been set at build time, not just  
>> at
>> run time.
>
> Not exactly: LD_LIBRARY_PATH only change the loader path (that is  
> which
> shared libraries are loaded when a process is started, as well as  
> dlopen).

Yep, but in order for the build to detect ATLAS he'd have to have the  
loader path set at build time too, right?

>> AFAIK NumPy will only link ATLAS in statically (I could be
>> wrong about this).
>
> Atlas may be linked dynamically as well.


Ah okay. Well, regardless, the build doesn't seem to be finding it,  
according to his show_config().

David


From paustin at eos.ubc.ca  Fri Mar 12 16:10:04 2010
From: paustin at eos.ubc.ca (Philip Austin)
Date: Fri, 12 Mar 2010 13:10:04 -0800
Subject: [SciPy-User] [xpyx]  Re:  Maximum file size for .npz format?
In-Reply-To: <4B9A8B0D.1010202@noaa.gov>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
	<49d6b3501003120929o610c6ab0j6e2e72e55cdfa93@mail.gmail.com>
	<45A85C73-A7BD-4765-AD9F-0523AF3B7B20@gmail.com>
	<4B9A8B0D.1010202@noaa.gov>
Message-ID: <4B9AADAC.2040609@eos.ubc.ca>

Chris Barker wrote:
> 
> Also, I think the python bindings still support netcd3, if you don't 
> need the extra stuff netcdf4 gives you (you may, if you need big files 
> -- not sure about that).

You can do netcdf3 without hdf and with big files by
compiling netcdf4-python (on linux) via:

> export NETCDF3_DIR=/home/phil/usr64/netcdf_3.6.3
> ~/usr64/bin/python setup-nc3.py install

then in python open a dataset for writing with:

theFile=Dataset(ncfilename,'w','NETCDF3_64BIT')

  -- Phil


From jsseabold at gmail.com  Fri Mar 12 16:59:39 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Fri, 12 Mar 2010 16:59:39 -0500
Subject: [SciPy-User] usemask kwd in timeseries.tsfromtxt missing?
Message-ID: <c048da1c1003121359k2984c9e3k2dcc100a7f2e3500@mail.gmail.com>

The docs say that usemask is a kwd arg, but it's not implemented.  Is
this an oversight or are the docs wrong and the TimeSeries object must
be a masked array?

Using recent trunk.

In [1]: import scikits.timeseries as ts

In [2]: ts.__version__
Out[2]: '0.91.3'

Skipper


From rmay31 at gmail.com  Fri Mar 12 19:35:35 2010
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 12 Mar 2010 18:35:35 -0600
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com> 
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com> 
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
Message-ID: <c7b95a021003121635y12c877b8g580b22286e3addfe@mail.gmail.com>

> I've experienced similar issues too, but I moved to NetCDF. The only disadvantage was that I did not find
> any python modules that work well _and_ support numpy. Hence, I am considering moving to HDF5.
> Which python module would people here recommend? (Or, alternatively, did I miss a great netCDF
> python module that someone could tell me about?)

You could try pupynere, which is pure python, only a single file
(netcdf 3 only).

http://pypi.python.org/pypi/pupynere/

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From andrew.collette at gmail.com  Fri Mar 12 20:30:25 2010
From: andrew.collette at gmail.com (Andrew Collette)
Date: Fri, 12 Mar 2010 17:30:25 -0800
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <f4f93d421003120930y4c5336f2gdf234ac634c24964@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
	<f4f93d421003120930y4c5336f2gdf234ac634c24964@mail.gmail.com>
Message-ID: <f125f5e1003121730o355a7e7fnf5697e695ae5c72b@mail.gmail.com>

> I use h5py. I think it is great. It gives you a dictionary-like
> interface to your archive. Here's a quick example:

Just to add, h5py (and PyTables also) allows you to read/write subsets
of your data:

>>> f = h5py.File('foo.hdf5','w')
>>> f['a'] = np.random.rand(1000,1000)
>>> subset = f['a'][200:300, 400:500:2] # only reads this slice from the file

H5py also supports transparent compression on a per-dataset basis,
with no limits on the size of the datasets or files.  Slicing is still
efficient for compressed datasets since HDF5 supports a chunked
storage model.  There's a general introduction to h5py here:

http://h5py.alfven.org/docs/guide/quick.html

Andrew Collette


From ia3n at stanford.edu  Fri Mar 12 20:36:11 2010
From: ia3n at stanford.edu (Ian Goodfellow)
Date: Fri, 12 Mar 2010 17:36:11 -0800
Subject: [SciPy-User] [SciPy-user] ATLAS install has no effect
In-Reply-To: <6C620C7D-88FC-4915-94EA-EDD331EDD75E@cs.toronto.edu>
References: <1997623173.2812691268343470110.JavaMail.root@zm09.stanford.edu>
	<002455CE-D714-4397-B9E7-F0E0B5F84D4D@cs.toronto.edu>
	<4B99F5BF.9030100@silveregg.co.jp>
	<6C620C7D-88FC-4915-94EA-EDD331EDD75E@cs.toronto.edu>
Message-ID: <F8EF35CA-0BB2-4381-9541-23E2E46BDB43@stanford.edu>

How should I set the path at build time? I don't see anything about it  
in the numpy INSTALL.txt or in the instructions on the website.
I tried building numpy from source rather than using the rpm and it  
doesn't seem to have detected the presence of ATLAS.
On Mar 12, 2010, at 10:54 AM, David Warde-Farley wrote:

> On 12-Mar-10, at 3:05 AM, David Cournapeau wrote:
>
>>> LD_LIBRARY_PATH would need to have been set at build time, not just
>>> at
>>> run time.
>>
>> Not exactly: LD_LIBRARY_PATH only change the loader path (that is
>> which
>> shared libraries are loaded when a process is started, as well as
>> dlopen).
>
> Yep, but in order for the build to detect ATLAS he'd have to have the
> loader path set at build time too, right?
>
>>> AFAIK NumPy will only link ATLAS in statically (I could be
>>> wrong about this).
>>
>> Atlas may be linked dynamically as well.
>
>
> Ah okay. Well, regardless, the build doesn't seem to be finding it,
> according to his show_config().
>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From paul.anton.letnes at gmail.com  Fri Mar 12 20:47:01 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Fri, 12 Mar 2010 17:47:01 -0800
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <c7b95a021003121635y12c877b8g580b22286e3addfe@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
	<c7b95a021003121635y12c877b8g580b22286e3addfe@mail.gmail.com>
Message-ID: <056687C6-56DC-4F29-A9AC-C52C95634166@gmail.com>



On 12. mars 2010, at 16.35, Ryan May wrote:

>> I've experienced similar issues too, but I moved to NetCDF. The only disadvantage was that I did not find
>> any python modules that work well _and_ support numpy. Hence, I am considering moving to HDF5.
>> Which python module would people here recommend? (Or, alternatively, did I miss a great netCDF
>> python module that someone could tell me about?)
> 
> You could try pupynere, which is pure python, only a single file
> (netcdf 3 only).
> 
> http://pypi.python.org/pypi/pupynere/
> 
> Ryan

pupynere is read-only, which of course is a show-stopper.

Everyone else, thanks for good advice. I still can't get netcdf4-python to work with my netcdf4 library though, which it won't detect, for some mysterious reason... Anyway, h5py seems like a nice module - thanks Keith! I think I might go that route instead.

Paul

From rmay31 at gmail.com  Fri Mar 12 20:54:01 2010
From: rmay31 at gmail.com (Ryan May)
Date: Fri, 12 Mar 2010 19:54:01 -0600
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <056687C6-56DC-4F29-A9AC-C52C95634166@gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com> 
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com> 
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com> 
	<c7b95a021003121635y12c877b8g580b22286e3addfe@mail.gmail.com> 
	<056687C6-56DC-4F29-A9AC-C52C95634166@gmail.com>
Message-ID: <c7b95a021003121754v94e8952j61c102c7c0dbcb6e@mail.gmail.com>

On Fri, Mar 12, 2010 at 7:47 PM, Paul Anton Letnes
<paul.anton.letnes at gmail.com> wrote:
> On 12. mars 2010, at 16.35, Ryan May wrote:
>> You could try pupynere, which is pure python, only a single file
>> (netcdf 3 only).
>>
>> http://pypi.python.org/pypi/pupynere/
>
> pupynere is read-only, which of course is a show-stopper.
>
> Everyone else, thanks for good advice. I still can't get netcdf4-python to work with my netcdf4 library though, which it won't detect, for some mysterious reason... Anyway, h5py seems like a nice module - thanks Keith! I think I might go that route instead.
>

No, it does allow writing.  At the top of the link I sent before
(which I'm guessing you didn't read :)  ):

"Pupynere is a Python module for reading and writing NetCDF files...."

It works pretty well for me.  The only problem is that it doesn't
allow modifying files, but that's not too bad of a limitation. The
pure python part makes it really simple to install (it doesn't even
rely on having the official netcdf libraries installed.)

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From timmichelsen at gmx-topmail.de  Sat Mar 13 03:55:31 2010
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Sat, 13 Mar 2010 09:55:31 +0100
Subject: [SciPy-User] usemask kwd in timeseries.tsfromtxt missing?
In-Reply-To: <c048da1c1003121359k2984c9e3k2dcc100a7f2e3500@mail.gmail.com>
References: <c048da1c1003121359k2984c9e3k2dcc100a7f2e3500@mail.gmail.com>
Message-ID: <hnfju4$8d1$1@dough.gmane.org>

> The docs say that usemask is a kwd arg, but it's not implemented.  Is
> this an oversight or are the docs wrong and the TimeSeries object must
> be a masked array?
do you have masked data?

I suggest to omit the argument.

To my experience:
If you load data with missing values (missing_values) a masked array is
created automatically. If not, the array will be regular.



From jsseabold at gmail.com  Sat Mar 13 11:43:42 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sat, 13 Mar 2010 11:43:42 -0500
Subject: [SciPy-User] usemask kwd in timeseries.tsfromtxt missing?
In-Reply-To: <hnfju4$8d1$1@dough.gmane.org>
References: <c048da1c1003121359k2984c9e3k2dcc100a7f2e3500@mail.gmail.com> 
	<hnfju4$8d1$1@dough.gmane.org>
Message-ID: <c048da1c1003130843g59f5aebbu1fd576f5f1c3d2c@mail.gmail.com>

On Sat, Mar 13, 2010 at 3:55 AM, Tim Michelsen
<timmichelsen at gmx-topmail.de> wrote:
>> The docs say that usemask is a kwd arg, but it's not implemented. ?Is
>> this an oversight or are the docs wrong and the TimeSeries object must
>> be a masked array?
> do you have masked data?
>
> I suggest to omit the argument.
>
> To my experience:
> If you load data with missing values (missing_values) a masked array is
> created automatically. If not, the array will be regular.
>

from StringIO import StringIO
import scikits.timeseries as ts
s = '"dates","GDP","CONS","INV","GOVT"\n"1947-I",100.2,33.4,33.4,33.4\n"1947-II",100.2,33.4,33.4,33.4'
s = StringIO(s)

romans = {"I":1,"II":2,"III":3,"IV":4}
dateconverter = lambda x: ts.Date(freq='Q',
year=int(x.split("-")[0].strip("\"")),
quarter=romans[x.split("-")[1].strip("\"")])

data = ts.tsfromtxt(s, delimiter=",", dtype=None,
dateconverter=dateconverter, names=True, usemask=True)
#TypeError: tsfromtxt() got an unexpected keyword argument 'usemask'

data = ts.tsfromtxt(s, delimiter=",", dtype=None,
dateconverter=dateconverter, names=True)
hasattr(data,'mask')
# True

print ts.tsfromtxt.__doc__
<snip>
    usemask : bool, optional
        If True, return a masked array.
        If False, return a regular array.
<snip>

Skipper


From chris.leemesser at gmail.com  Sat Mar 13 11:45:08 2010
From: chris.leemesser at gmail.com (Christopher Lee-Messer)
Date: Sat, 13 Mar 2010 08:45:08 -0800
Subject: [SciPy-User]  Shared memory using multiprocessing.sharedctypes
Message-ID: <4cccdc8b1003130845j1ab46bfdgdf38ca57510a0f51@mail.gmail.com>

David Baddeley wrote::
> I'd like shared memory numpy arrays for some code I'm trying to parallelise. The solution should ideally be cross platform. I remember some earlier discussion about this and, as far as I recall Sturla had a working implementation which was (unfortunately) linux only (please correct me if I'm wrong).
>
> Looking a bit further, multiprocessing.sharedctypes seems to have all the stuff needed to handle the low level bits on at least linux and windows, and it seems as though it should be relatively easy to cobble something together on top of this which let you use shared memory numpy arrays relatively transparently. This would have the advantage that all the platform specific nasty stuff is being handled by the multiprocessing module, which as it's now part of the standard library, should hopefully be well maintained & has a good chance of being ported to additional platforms
>
> Before I jump in and try writing such a wrapper - essentially a helper function for creating shared numpy arrays (easy) and some pickle handlers to make sure what comes out the other end looks like an array (would need to think/read a bit more, but should be doable) I wanted to make sure that there's not something out there already & that I'm not going to be reinventing the wheel.
>
[Note, I posted this via google groups originally before realizing
that it wasn't the official list. Sorry if you get multiple versions.
-chris]

Hi David,
I was just in the same boat.  I checked out Sturla's a few weeks ago
and tried it out on windows  xp and now linux (32bit, python 2.6 with
numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with numpy 1.2.1)

It's working very well for me in soft-realtime data acquisition
application with sizes of 1 to 800KB.

In case it saves you time, I recently threw together some packaging
for the code and put it on bitbucket as a numpy-sharedmem repository
in case it might save me or someone else some time.

http://bitbucket.org/cleemesser/numpy-sharedmem/

I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
to work on these, but I plan to add testing and save results on
different platforms as my app gets used in different computers in the
lab.  Nadav Horesh mentioned that there are some limitations on the
size of the arrays for "large" arrays and Sturla mentioned that there
may be issue on 64bit systems.
-Chris


From gael.varoquaux at normalesup.org  Sat Mar 13 12:54:04 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 13 Mar 2010 18:54:04 +0100
Subject: [SciPy-User] Shared memory using multiprocessing.sharedctypes
In-Reply-To: <4cccdc8b1003130845j1ab46bfdgdf38ca57510a0f51@mail.gmail.com>
References: <4cccdc8b1003130845j1ab46bfdgdf38ca57510a0f51@mail.gmail.com>
Message-ID: <20100313175404.GA16132@phare.normalesup.org>

On Sat, Mar 13, 2010 at 08:45:08AM -0800, Christopher Lee-Messer wrote:
> http://bitbucket.org/cleemesser/numpy-sharedmem/

> I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
> to work on these, but I plan to add testing and save results on
> different platforms as my app gets used in different computers in the
> lab. 

I am not. I have been wanting to propose this for integration in numpy
for a long while, but I really haven't found the time, and I will not any
time soon.

Before any patch can be written to push in numpy, it does need tests
and packaging, so what you are proposing to do is really great and very
helpful. Also, using it on many platform would help ironing out details.

Thanks for stepping up,

Ga?l


From vanforeest at gmail.com  Sat Mar 13 16:06:10 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Sat, 13 Mar 2010 22:06:10 +0100
Subject: [SciPy-User] caching function values
Message-ID: <fa510ff81003131306g7c30cea4sf4fd2d56fa21ab6@mail.gmail.com>

Hi,

When calling a function for the first time I like to cache some
complicated computations (that only have to be done once), and then
use this cached stuff in all subsequent calls to the functions. I came
across the following trick on the net:

def dummy(x, firstTime = [1]):
    if firstTime[0] == 1:
        A = 3  # typically some lengthy do-once computation to obtain a matrix A
        dummy.A = A # store it
        firstTime[0] = 0 # ensure this piece of code will not be run again
    u = dummy.A*x  # use the cached value of A in the real computations
    return u

Are there other (less tricky) ways to achieve the same effect, without
using a class? (As far as I can see, using classes would involve some
extra lines of code, for instance, I have to instantiate it.)

Thanks for any advice.

Nicky


From ralf.gommers at googlemail.com  Sat Mar 13 22:15:08 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Sun, 14 Mar 2010 11:15:08 +0800
Subject: [SciPy-User] caching function values
In-Reply-To: <fa510ff81003131306g7c30cea4sf4fd2d56fa21ab6@mail.gmail.com>
References: <fa510ff81003131306g7c30cea4sf4fd2d56fa21ab6@mail.gmail.com>
Message-ID: <dde7764a1003131915k3c6ef7haba25bd83e461e12@mail.gmail.com>

On Sun, Mar 14, 2010 at 5:06 AM, nicky van foreest <vanforeest at gmail.com>wrote:

> Hi,
>
> When calling a function for the first time I like to cache some
> complicated computations (that only have to be done once), and then
> use this cached stuff in all subsequent calls to the functions. I came
> across the following trick on the net:
>
> def dummy(x, firstTime = [1]):
>    if firstTime[0] == 1:
>        A = 3  # typically some lengthy do-once computation to obtain a
> matrix A
>        dummy.A = A # store it
>        firstTime[0] = 0 # ensure this piece of code will not be run again
>    u = dummy.A*x  # use the cached value of A in the real computations
>    return u
>
> The above is not too complicated, but what's ugly is the firstTime keyword
in the signature. This looks better to me:
def dummy(x):
   if not hasattr(dummy, '_firsttime'):
       A = 3  # typically some lengthy do-once computation to obtain a
matrix A
       dummy.A = A # store it
   u = dummy.A*x  # use the cached value of A in the real computations
   dummy._firsttime = False
   return u


> Are there other (less tricky) ways to achieve the same effect, without
> using a class? (As far as I can see, using classes would involve some
> extra lines of code, for instance, I have to instantiate it.)
>

As long as the code is readable, why not use a few more lines? If your
calculation of A only has to run once, it doesn't really belong inside the
function. Either use a class, or (even easier) a private object outside the
function to store the result of the computation.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100314/fcf3280a/attachment.html>

From paul.anton.letnes at gmail.com  Sat Mar 13 22:21:50 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Sat, 13 Mar 2010 19:21:50 -0800
Subject: [SciPy-User] caching function values
In-Reply-To: <fa510ff81003131306g7c30cea4sf4fd2d56fa21ab6@mail.gmail.com>
References: <fa510ff81003131306g7c30cea4sf4fd2d56fa21ab6@mail.gmail.com>
Message-ID: <7F7C34C7-D9ED-4135-9CD9-78E34BA2CB44@gmail.com>

On 13. mars 2010, at 13.06, nicky van foreest wrote:

> Hi,
> 
> When calling a function for the first time I like to cache some
> complicated computations (that only have to be done once), and then
> use this cached stuff in all subsequent calls to the functions. I came
> across the following trick on the net:
> 
> def dummy(x, firstTime = [1]):
>    if firstTime[0] == 1:
>        A = 3  # typically some lengthy do-once computation to obtain a matrix A
>        dummy.A = A # store it
>        firstTime[0] = 0 # ensure this piece of code will not be run again
>    u = dummy.A*x  # use the cached value of A in the real computations
>    return u
> 
> Are there other (less tricky) ways to achieve the same effect, without
> using a class? (As far as I can see, using classes would involve some
> extra lines of code, for instance, I have to instantiate it.)
> 
> Thanks for any advice.
> 
> Nicky
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

How about just making the following class?

class DummyClass(object):
	def __init__():
		self.A = lengthy_calculation()
	def __call__(self, x):
		return self.A *x

Somewhere else, type
dummy = DummyClass()
y = dummy(x) # or whatever

I'd say that this is as readable, and in fact, not noticeably longer.

Paul

From vincent at vincentdavis.net  Sun Mar 14 00:01:43 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 13 Mar 2010 22:01:43 -0700
Subject: [SciPy-User] Error: numpy/lib/recfunctions.py line 755,
	in find_duplicates 	sorteddata = sortedbase.filled()
Message-ID: <77e831101003132101q24c9c0f5k4a4b1a3311728c94@mail.gmail.com>

Not sure what is going on here but am am getting an error that I am not sure
how to fix. Any Help world me appreciated.

>>> masteromni
array([ ('AMA1_CDS_st', 'AMA1_CDS_st:389:30', 389, 30, 13, 0, 1,
'AGCCTCCCATGATTGAACAGATCAT', 'Sense', 'PM', 'Tiling', 0.3911,
61.700000000000003, 44.0, -38.661999999999999, -193.69999999999999, -520.0,
'AMA_CDS', 1, 'VIIb', 4880893, 4880869, 4880881,
'TGME49_chrVIIb:4880893..4880869', ' empt'),
       ('AMA1_CDS_at', 'AMA1_CDS_at:45:44', 45, 44, 15, 0, 1,
'GATCTGTTCAATCATGGGAGGCTTG', 'Antisense', 'PM', 'Tiling', 0.437,
61.899999999999999, 48.0, -38.825000000000003, -194.40000000000001,
-521.79999999999995, 'AMA_CDS', 2, 'VIIb', 4880871, 4880895, 4880883,
'TGME49_chrVIIb:4880871..4880895', ' empt'),
       ('AMA1_CDS_st', 'AMA1_CDS_st:116:84', 116, 84, 17, 1, 2,
'CGCAAGCCTCCCATGATTGAACAGA', 'Sense', 'PM', 'Tiling', 0.57669999999999999,
65.200000000000003, 52.0, -41.142000000000003, -199.40000000000001,
-530.79999999999995, 'AMA_CDS', 3, 'VIIb', 4880897, 4880873, 4880885,
'TGME49_chrVIIb:4880897..4880873', ' empt'),
       ...,
       ('25.m02918_AS_at', '25.m02918_AS_at:24:29', 24, 29, 43214, 17, 18,
'ATGGATGTACATTCGCTGCTGACTA', 'Antisense', 'PM', 'Antisense detection', 0.0,
61.700000000000003, 44.0, -39.125, -197.80000000000001, -532.20000000000005,
'-1', -1, 'Ib', 87103, 87079, 87091, 'TGME49_chrIb:87103..87079', ' empt'),
       ('25.m02918_AS_at', '25.m02918_AS_at:208:333', 208, 333, 43310, 18,
19, 'AGAACTTCCGTGTGCAACTGCTTTA', 'Antisense', 'PM', 'Antisense detection',
0.67410000000000003, 62.899999999999999, 44.0, -39.993000000000002,
-200.09999999999999, -537.0, '-1', -1, 'Ib', 87007, 86983, 86995,
'TGME49_chrIb:87007..86983', ' empt'),
       ('25.m02918_AS_at', '25.m02918_AS_at:435:348', 435, 348, 43550, 19,
20, 'AATAGGCATTTCGTGGTTCGCGCAA', 'Antisense', 'PM', 'Antisense detection',
0.62050000000000005, 65.700000000000003, 48.0, -42.009999999999998,
-204.80000000000001, -546.0, '-1', -1, 'Ib', 86767, 86743, 86755,
'TGME49_chrIb:86767..86743', ' empt')],
      dtype=[('probe_set_name', '|S40'), ('ProbeUniqueName', '|S40'), ('x',
'<i8'), ('y', '<i8'), ('Interrogation_position', '<i8'),
('Probeset_position0', '<i8'), ('Probeset_position1', '<i8'), ('Sequence2',
'|S25'), ('Strandedness', '|S40'), ('PM', '|S40'), ('Purpose', '|S40'),
('AffyProbeScore', '<f8'), ('Tm_emboss', '<f8'), ('gc_emboss', '<f8'),
('DeltaG_emboss', '<f8'), ('DeltaH_emboss', '<f8'), ('DeltaS_emboss',
'<f8'), ('TilingSet', '|S40'), ('AbsTileNum', '<i8'), ('Chr', '|S40'),
('ChrProbeStart2', '<i8'), ('ChrProbeStop2', '<i8'), ('ChrMidpoint', '<i8'),
('GenBrowserCoords', '|S40'), ('emptyColumn', '|S5')])


>>> find_duplicates(omni_snps[:5], 'ProbeUniqueName')
Traceback (most recent call last):
  File
"/Applications/WingIDE.app/Contents/MacOS/src/debug/tserver/_sandbox.py",
line 1, in <module>
    # Used internally for debug sandbox under external interpreter
  File
"/Library/Frameworks/EPD64.framework/Versions/6.1/lib/python2.6/site-packages/numpy/lib/recfunctions.py",
line 755, in find_duplicates
    sorteddata = sortedbase.filled()
AttributeError: 'numpy.ndarray' object has no attribute 'filled'


  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100313/48cc2cf1/attachment.html>

From stefan at sun.ac.za  Sun Mar 14 15:43:26 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 14 Mar 2010 21:43:26 +0200
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com> 
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
Message-ID: <9457e7c81003141243v39aa5f63s5c280c9c02796151@mail.gmail.com>

On 11 March 2010 17:48, Robert Kern <robert.kern at gmail.com> wrote:
>> I can't open it with the normal zip utility present on the system, but it
>> could be that it's barfing about files being larger than 2Gb.
>> Is there some file limit for npzs?
>
> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
> not yet support the ZIP64 format.

I see Bruce dug up  http://projects.scipy.org/numpy/ticket/991.  Is
this the right route to go, or do we need a more sophisticated
solution?

Regards
St?fan


From timmichelsen at gmx-topmail.de  Sun Mar 14 18:32:09 2010
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Sun, 14 Mar 2010 23:32:09 +0100
Subject: [SciPy-User] usemask kwd in timeseries.tsfromtxt missing?
In-Reply-To: <c048da1c1003130843g59f5aebbu1fd576f5f1c3d2c@mail.gmail.com>
References: <c048da1c1003121359k2984c9e3k2dcc100a7f2e3500@mail.gmail.com>
	<hnfju4$8d1$1@dough.gmane.org>
	<c048da1c1003130843g59f5aebbu1fd576f5f1c3d2c@mail.gmail.com>
Message-ID: <hnjo5a$st$1@dough.gmane.org>

> data = ts.tsfromtxt(s, delimiter=",", dtype=None,
> dateconverter=dateconverter, names=True, usemask=True)
> #TypeError: tsfromtxt() got an unexpected keyword argument 'usemask'
> 
> data = ts.tsfromtxt(s, delimiter=",", dtype=None,
> dateconverter=dateconverter, names=True)
> hasattr(data,'mask')
> # True
> 
> print ts.tsfromtxt.__doc__
> <snip>
>     usemask : bool, optional
>         If True, return a masked array.
>         If False, return a regular array.
> <snip>
I never really cared about whether a masked array is created or not.
Does this really make a difference in your work?
Let's wait for a comment from Pierre GM?

you may also check:
data.data
data.series



From Chris.Barker at noaa.gov  Sun Mar 14 20:24:38 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Sun, 14 Mar 2010 17:24:38 -0700
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <9457e7c81003141243v39aa5f63s5c280c9c02796151@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<9457e7c81003141243v39aa5f63s5c280c9c02796151@mail.gmail.com>
Message-ID: <4B9D7E46.5030509@noaa.gov>

St?fan van der Walt wrote:
>> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
>> not yet support the ZIP64 format.
> 
> I see Bruce dug up  http://projects.scipy.org/numpy/ticket/991.  Is
> this the right route to go, or do we need a more sophisticated
> solution?

It might be nice to build something on libbzip2 -- It looks like the 
license is right, it's got good compression qualities, supports 64 bit, 
and it getting pretty widely used.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From robert.kern at gmail.com  Sun Mar 14 22:23:48 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 14 Mar 2010 20:23:48 -0600
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <4B9D7E46.5030509@noaa.gov>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com> 
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com> 
	<9457e7c81003141243v39aa5f63s5c280c9c02796151@mail.gmail.com> 
	<4B9D7E46.5030509@noaa.gov>
Message-ID: <3d375d731003141923h22e3d924j6e4a8680b9a16f04@mail.gmail.com>

On Sun, Mar 14, 2010 at 18:24, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> St?fan van der Walt wrote:
>>> Yes, the ZIP file format has a 4GB limit. Unfortunately, Python does
>>> not yet support the ZIP64 format.
>>
>> I see Bruce dug up ?http://projects.scipy.org/numpy/ticket/991. ?Is
>> this the right route to go, or do we need a more sophisticated
>> solution?
>
> It might be nice to build something on libbzip2 -- It looks like the
> license is right, it's got good compression qualities, supports 64 bit,
> and it getting pretty widely used.

bzip2 does not support random access. It just compresses a single
file. It is not a replacement for zipfile.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From david_baddeley at yahoo.com.au  Sun Mar 14 23:09:54 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Sun, 14 Mar 2010 20:09:54 -0700 (PDT)
Subject: [SciPy-User] Shared memory using multiprocessing.sharedctypes
In-Reply-To: <4cccdc8b1003130845j1ab46bfdgdf38ca57510a0f51@mail.gmail.com>
References: <4cccdc8b1003130845j1ab46bfdgdf38ca57510a0f51@mail.gmail.com>
Message-ID: <342100.83421.qm@web33007.mail.mud.yahoo.com>

Thanks!

Will have to give it a spin. Unfortunately  (or fortunately - as it was quite instructive from a learning perspective) I got bored and already knocked up my own version. It delegates all the memory management to multiprocessing so weighs in pretty light at only 80 lines of pure python & works well on 64 bit systems. Just in case anyone wants yet another shared memory array implementation.

cheers,
David


----- Original Message ----
From: Christopher Lee-Messer <chris.leemesser at gmail.com>
To: scipy-user at scipy.org
Sent: Sun, 14 March, 2010 5:45:08 AM
Subject: [SciPy-User]  Shared memory using multiprocessing.sharedctypes

David Baddeley wrote::
> I'd like shared memory numpy arrays for some code I'm trying to parallelise. The solution should ideally be cross platform. I remember some earlier discussion about this and, as far as I recall Sturla had a working implementation which was (unfortunately) linux only (please correct me if I'm wrong).
>
> Looking a bit further, multiprocessing.sharedctypes seems to have all the stuff needed to handle the low level bits on at least linux and windows, and it seems as though it should be relatively easy to cobble something together on top of this which let you use shared memory numpy arrays relatively transparently. This would have the advantage that all the platform specific nasty stuff is being handled by the multiprocessing module, which as it's now part of the standard library, should hopefully be well maintained & has a good chance of being ported to additional platforms
>
> Before I jump in and try writing such a wrapper - essentially a helper function for creating shared numpy arrays (easy) and some pickle handlers to make sure what comes out the other end looks like an array (would need to think/read a bit more, but should be doable) I wanted to make sure that there's not something out there already & that I'm not going to be reinventing the wheel.
>
[Note, I posted this via google groups originally before realizing
that it wasn't the official list. Sorry if you get multiple versions.
-chris]

Hi David,
I was just in the same boat.  I checked out Sturla's a few weeks ago
and tried it out on windows  xp and now linux (32bit, python 2.6 with
numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with numpy 1.2.1)

It's working very well for me in soft-realtime data acquisition
application with sizes of 1 to 800KB.

In case it saves you time, I recently threw together some packaging
for the code and put it on bitbucket as a numpy-sharedmem repository
in case it might save me or someone else some time.

http://bitbucket.org/cleemesser/numpy-sharedmem/

I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
to work on these, but I plan to add testing and save results on
different platforms as my app gets used in different computers in the
lab.  Nadav Horesh mentioned that there are some limitations on the
size of the arrays for "large" arrays and Sturla mentioned that there
may be issue on 64bit systems.
-Chris
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From njs at pobox.com  Mon Mar 15 00:47:56 2010
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 14 Mar 2010 21:47:56 -0700
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <4B9D7E46.5030509@noaa.gov>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<9457e7c81003141243v39aa5f63s5c280c9c02796151@mail.gmail.com>
	<4B9D7E46.5030509@noaa.gov>
Message-ID: <961fa2b41003142147i30a8f114ld97ba17659d90d1@mail.gmail.com>

On Sun, Mar 14, 2010 at 5:24 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> It might be nice to build something on libbzip2 -- It looks like the
> license is right, it's got good compression qualities, supports 64 bit,
> and it getting pretty widely used.

If we're going to pull in a new C library for this purpose, then maybe
we should just use libhdf5? :-).

-- Nathaniel


From vanforeest at gmail.com  Mon Mar 15 04:12:02 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Mon, 15 Mar 2010 09:12:02 +0100
Subject: [SciPy-User] caching function values
In-Reply-To: <7F7C34C7-D9ED-4135-9CD9-78E34BA2CB44@gmail.com>
References: <fa510ff81003131306g7c30cea4sf4fd2d56fa21ab6@mail.gmail.com>
	<7F7C34C7-D9ED-4135-9CD9-78E34BA2CB44@gmail.com>
Message-ID: <fa510ff81003150112j3bb12cbdn2bd9ec96fd589b2a@mail.gmail.com>

Hi Ralf and Paul,

Thanks for sharing your ideas.

Nicky

On 14 March 2010 04:21, Paul Anton Letnes <paul.anton.letnes at gmail.com> wrote:
> On 13. mars 2010, at 13.06, nicky van foreest wrote:
>
>> Hi,
>>
>> When calling a function for the first time I like to cache some
>> complicated computations (that only have to be done once), and then
>> use this cached stuff in all subsequent calls to the functions. I came
>> across the following trick on the net:
>>
>> def dummy(x, firstTime = [1]):
>> ? ?if firstTime[0] == 1:
>> ? ? ? ?A = 3 ?# typically some lengthy do-once computation to obtain a matrix A
>> ? ? ? ?dummy.A = A # store it
>> ? ? ? ?firstTime[0] = 0 # ensure this piece of code will not be run again
>> ? ?u = dummy.A*x ?# use the cached value of A in the real computations
>> ? ?return u
>>
>> Are there other (less tricky) ways to achieve the same effect, without
>> using a class? (As far as I can see, using classes would involve some
>> extra lines of code, for instance, I have to instantiate it.)
>>
>> Thanks for any advice.
>>
>> Nicky
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
> How about just making the following class?
>
> class DummyClass(object):
> ? ? ? ?def __init__():
> ? ? ? ? ? ? ? ?self.A = lengthy_calculation()
> ? ? ? ?def __call__(self, x):
> ? ? ? ? ? ? ? ?return self.A *x
>
> Somewhere else, type
> dummy = DummyClass()
> y = dummy(x) # or whatever
>
> I'd say that this is as readable, and in fact, not noticeably longer.
>
> Paul
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From gruben at bigpond.net.au  Mon Mar 15 04:33:46 2010
From: gruben at bigpond.net.au (Gary Ruben)
Date: Mon, 15 Mar 2010 19:33:46 +1100
Subject: [SciPy-User] caching function values
In-Reply-To: <fa510ff81003150112j3bb12cbdn2bd9ec96fd589b2a@mail.gmail.com>
References: <fa510ff81003131306g7c30cea4sf4fd2d56fa21ab6@mail.gmail.com>	<7F7C34C7-D9ED-4135-9CD9-78E34BA2CB44@gmail.com>
	<fa510ff81003150112j3bb12cbdn2bd9ec96fd589b2a@mail.gmail.com>
Message-ID: <4B9DF0EA.3090503@bigpond.net.au>

Hi Nicky,

I've attached another example which may do what you're after using a 
decorator-based solution. The technique is called memoization and is 
described in lectures 13 and 14 of the MIT OpenCourseWare intro to Comp. 
Sci. and Programming course, which is now based on Python.

Gary R.

nicky van foreest wrote:
> Hi Ralf and Paul,
> 
> Thanks for sharing your ideas.
> 
> Nicky
> 
> On 14 March 2010 04:21, Paul Anton Letnes <paul.anton.letnes at gmail.com> wrote:
>> On 13. mars 2010, at 13.06, nicky van foreest wrote:
>>
>>> Hi,
>>>
>>> When calling a function for the first time I like to cache some
>>> complicated computations (that only have to be done once), and then
>>> use this cached stuff in all subsequent calls to the functions. I came
>>> across the following trick on the net:
>>>
>>> def dummy(x, firstTime = [1]):
>>>    if firstTime[0] == 1:
>>>        A = 3  # typically some lengthy do-once computation to obtain a matrix A
>>>        dummy.A = A # store it
>>>        firstTime[0] = 0 # ensure this piece of code will not be run again
>>>    u = dummy.A*x  # use the cached value of A in the real computations
>>>    return u
>>>
>>> Are there other (less tricky) ways to achieve the same effect, without
>>> using a class? (As far as I can see, using classes would involve some
>>> extra lines of code, for instance, I have to instantiate it.)
>>>
>>> Thanks for any advice.
>>>
>>> Nicky
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> How about just making the following class?
>>
>> class DummyClass(object):
>>        def __init__():
>>                self.A = lengthy_calculation()
>>        def __call__(self, x):
>>                return self.A *x
>>
>> Somewhere else, type
>> dummy = DummyClass()
>> y = dummy(x) # or whatever
>>
>> I'd say that this is as readable, and in fact, not noticeably longer.
>>
>> Paul

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: memoization_example.py
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/c4c18f5b/attachment.ksh>

From pierre.lamon at bluebotics.com  Mon Mar 15 04:39:20 2010
From: pierre.lamon at bluebotics.com (Pierre Lamon)
Date: Mon, 15 Mar 2010 09:39:20 +0100
Subject: [SciPy-User] Potential KDTree memory leak
In-Reply-To: <3d375d731003110750h3a6f8bd6se0a91b9cb98ff7b1@mail.gmail.com>
References: <4B98EDDD.6040608@bluebotics.com>
	<3d375d731003110750h3a6f8bd6se0a91b9cb98ff7b1@mail.gmail.com>
Message-ID: <4B9DF238.3090700@bluebotics.com>

On 2010-03-11 16:50, Robert Kern wrote:
> With recent SVN versions of numpy and scipy, I do not see a leak.
>    
I've tested with compiled sources of numpy 1.3.0 and scipy 0.7.3 and 
there is the leak (test done
under linux). On the other hand there is no leak with the SVN version of 
numpy and scipy. I'm glad
the bug has been fixed. Any idea when the new release will be available ?

-- 
Dr. Pierre Lamon
Research&  Development / Scientific Manager
BlueBotics SA
PSE-C, CH-1015 Lausanne, Switzerland
http://www.bluebotics.com
pierre.lamon at bluebotics.com
+41 21 693 83 14 (tel)
+41 21 693 83 15 (fax)



From jgomezdans at gmail.com  Mon Mar 15 06:32:32 2010
From: jgomezdans at gmail.com (Jose Gomez-Dans)
Date: Mon, 15 Mar 2010 10:32:32 +0000
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <4B9A8BE3.40506@gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<ce557a361003121027u7e177763l37018c4b6cb54ed0@mail.gmail.com>
	<3d375d731003121035p6250ec40oe9b20c45ced8d02e@mail.gmail.com>
	<4B9A8BE3.40506@gmail.com>
Message-ID: <91d218431003150332p67464a87q61d06c5981a5c044@mail.gmail.com>

Hi!

On 12 March 2010 18:45, Bruce Southey <bsouthey at gmail.com> wrote:

> Jose, what do Python version are you using and does the '  normal zip
> utility present on the system' actually support zip64?
> You should try seeing if 7z sees it.
>

I'm using python 2.5 on x86_64. The file is not recognised by either the
local zip ( 2.31), or  7za (v 4.61). The latter lists the files and reports
"Unsupported method".

It appears from this that the pytables/hdf route may be the most promising
for me.

Thanks!
J
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/1a114bb5/attachment.html>

From bsouthey at gmail.com  Mon Mar 15 09:20:17 2010
From: bsouthey at gmail.com (Bruce Southey)
Date: Mon, 15 Mar 2010 08:20:17 -0500
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <91d218431003150332p67464a87q61d06c5981a5c044@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>	<ce557a361003121027u7e177763l37018c4b6cb54ed0@mail.gmail.com>	<3d375d731003121035p6250ec40oe9b20c45ced8d02e@mail.gmail.com>	<4B9A8BE3.40506@gmail.com>
	<91d218431003150332p67464a87q61d06c5981a5c044@mail.gmail.com>
Message-ID: <4B9E3411.5010306@gmail.com>

On 03/15/2010 05:32 AM, Jose Gomez-Dans wrote:
> Hi!
>
> On 12 March 2010 18:45, Bruce Southey <bsouthey at gmail.com 
> <mailto:bsouthey at gmail.com>> wrote:
>
>     Jose, what do Python version are you using and does the ' normal
>     zip utility present on the system' actually support zip64?
>     You should try seeing if 7z sees it.
>
>
> I'm using python 2.5 on x86_64. The file is not recognised by either 
> the local zip ( 2.31), or  7za (v 4.61). The latter lists the files 
> and reports "Unsupported method".
>
> It appears from this that the pytables/hdf route may be the most 
> promising for me.
>
> Thanks!
> J
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>    
Hi,
Good, then you should try the patch for Ticket 991 :
http://projects.scipy.org/numpy/ticket/991.

Note that Python 2.6 documentation says:
"It can handle ZIP files that use the ZIP64 extensions (that is ZIP 
files that are more than 4 GByte in size). "

This is better than the Python 2.5.2 documentation. So can you try 
Python 2.6 ?

It would be great to resolve your numpy issue especially for other users 
- at least to say whether or not the patch or a new Python solves it.

Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/f013a40a/attachment.html>

From ralf.gommers at googlemail.com  Mon Mar 15 11:03:41 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Mon, 15 Mar 2010 23:03:41 +0800
Subject: [SciPy-User] Potential KDTree memory leak
In-Reply-To: <4B9DF238.3090700@bluebotics.com>
References: <4B98EDDD.6040608@bluebotics.com>
	<3d375d731003110750h3a6f8bd6se0a91b9cb98ff7b1@mail.gmail.com>
	<4B9DF238.3090700@bluebotics.com>
Message-ID: <dde7764a1003150803h279f6f85y2e0e42dd59726491@mail.gmail.com>

On Mon, Mar 15, 2010 at 4:39 PM, Pierre Lamon
<pierre.lamon at bluebotics.com>wrote:

> On 2010-03-11 16:50, Robert Kern wrote:
> > With recent SVN versions of numpy and scipy, I do not see a leak.
> >
> I've tested with compiled sources of numpy 1.3.0 and scipy 0.7.3 and
> there is the leak (test done under linux).


Scipy 0.7.3, that's impressive:) 0.7.2 RC1 should be coming soon.


> On the other hand there is no leak with the SVN version of
> numpy and scipy. I'm glad
> the bug has been fixed. Any idea when the new release will be available ?
>

The new release of scipy trunk (0.8.0)? Probably you have to be a bit
patient there. The plan is ATM: first scipy 0.7.2, then numpy 1.4.1, then
numpy 2.0, and only then scipy 0.8.0.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/55fb9219/attachment.html>

From ce at vejnar.eu  Mon Mar 15 11:31:54 2010
From: ce at vejnar.eu (Charles Vejnar)
Date: Mon, 15 Mar 2010 16:31:54 +0100
Subject: [SciPy-User] Shared memory using multiprocessing.sharedctypes
In-Reply-To: <342100.83421.qm@web33007.mail.mud.yahoo.com>
References: <4cccdc8b1003130845j1ab46bfdgdf38ca57510a0f51@mail.gmail.com>
	<342100.83421.qm@web33007.mail.mud.yahoo.com>
Message-ID: <201003151631.54145.ce@vejnar.eu>

Hi David,

I am interested. Can you please show us your code ?

BW,
Charles

On Monday 15 March 2010, David Baddeley wrote:
> Thanks!
> 
> Will have to give it a spin. Unfortunately  (or fortunately - as it was
>  quite instructive from a learning perspective) I got bored and already
>  knocked up my own version. It delegates all the memory management to
>  multiprocessing so weighs in pretty light at only 80 lines of pure python
>  & works well on 64 bit systems. Just in case anyone wants yet another
>  shared memory array implementation.
> 
> cheers,
> David
> 
> 
> ----- Original Message ----
> From: Christopher Lee-Messer <chris.leemesser at gmail.com>
> To: scipy-user at scipy.org
> Sent: Sun, 14 March, 2010 5:45:08 AM
> Subject: [SciPy-User]  Shared memory using multiprocessing.sharedctypes
> 
> David Baddeley wrote::
> > I'd like shared memory numpy arrays for some code I'm trying to
> > parallelise. The solution should ideally be cross platform. I remember
> > some earlier discussion about this and, as far as I recall Sturla had a
> > working implementation which was (unfortunately) linux only (please
> > correct me if I'm wrong).
> >
> > Looking a bit further, multiprocessing.sharedctypes seems to have all the
> > stuff needed to handle the low level bits on at least linux and windows,
> > and it seems as though it should be relatively easy to cobble something
> > together on top of this which let you use shared memory numpy arrays
> > relatively transparently. This would have the advantage that all the
> > platform specific nasty stuff is being handled by the multiprocessing
> > module, which as it's now part of the standard library, should hopefully
> > be well maintained & has a good chance of being ported to additional
> > platforms
> >
> > Before I jump in and try writing such a wrapper - essentially a helper
> > function for creating shared numpy arrays (easy) and some pickle handlers
> > to make sure what comes out the other end looks like an array (would need
> > to think/read a bit more, but should be doable) I wanted to make sure
> > that there's not something out there already & that I'm not going to be
> > reinventing the wheel.
> 
> [Note, I posted this via google groups originally before realizing
> that it wasn't the official list. Sorry if you get multiple versions.
> -chris]
> 
> Hi David,
> I was just in the same boat.  I checked out Sturla's a few weeks ago
> and tried it out on windows  xp and now linux (32bit, python 2.6 with
> numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with numpy 1.2.1)
> 
> It's working very well for me in soft-realtime data acquisition
> application with sizes of 1 to 800KB.
> 
> In case it saves you time, I recently threw together some packaging
> for the code and put it on bitbucket as a numpy-sharedmem repository
> in case it might save me or someone else some time.
> 
> http://bitbucket.org/cleemesser/numpy-sharedmem/
> 
> I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
> to work on these, but I plan to add testing and save results on
> different platforms as my app gets used in different computers in the
> lab.  Nadav Horesh mentioned that there are some limitations on the
> size of the arrays for "large" arrays and Sturla mentioned that there
> may be issue on 64bit systems.
> -Chris
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 
> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 


From pgmdevlist at gmail.com  Mon Mar 15 12:01:26 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Mon, 15 Mar 2010 12:01:26 -0400
Subject: [SciPy-User] usemask kwd in timeseries.tsfromtxt missing?
In-Reply-To: <hnjo5a$st$1@dough.gmane.org>
References: <c048da1c1003121359k2984c9e3k2dcc100a7f2e3500@mail.gmail.com>
	<hnfju4$8d1$1@dough.gmane.org>
	<c048da1c1003130843g59f5aebbu1fd576f5f1c3d2c@mail.gmail.com>
	<hnjo5a$st$1@dough.gmane.org>
Message-ID: <4A98946E-5539-4274-A1B4-778D6D672BE7@gmail.com>

On Mar 14, 2010, at 6:32 PM, Tim Michelsen wrote:
>> data = ts.tsfromtxt(s, delimiter=",", dtype=None,
>> dateconverter=dateconverter, names=True, usemask=True)
>> #TypeError: tsfromtxt() got an unexpected keyword argument 'usemask'
>> 
>> data = ts.tsfromtxt(s, delimiter=",", dtype=None,
>> dateconverter=dateconverter, names=True)
>> hasattr(data,'mask')
>> # True
>> 
>> print ts.tsfromtxt.__doc__
>> <snip>
>>    usemask : bool, optional
>>        If True, return a masked array.
>>        If False, return a regular array.
>> <snip>
> I never really cared about whether a masked array is created or not.
> Does this really make a difference in your work?
> Let's wait for a comment from Pierre GM?


All,
I only have episodic connectability these days, sorry for the lag in answers.
In np.genfromtxt, `usemask` lets you choose whether you want a standard ndarray (False) or a MaskedArray (True). 
In `scikits.timeseries.tsfromtxt`, the output is always a TimeSeries, which is a subclass of MaskedArray. Therefore, `usemask` would always be True. No need for a keyword argument that you can't modify, hence the error message.

From denis-bz-gg at t-online.de  Mon Mar 15 13:32:46 2010
From: denis-bz-gg at t-online.de (denis)
Date: Mon, 15 Mar 2010 10:32:46 -0700 (PDT)
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
Message-ID: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>

Where should plots and other pngs refd from notes in
advice.mechanicalkern.com be put ?
Thanks, cheers
  -- denis


From Chris.Barker at noaa.gov  Mon Mar 15 13:50:30 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 15 Mar 2010 10:50:30 -0700
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <961fa2b41003142147i30a8f114ld97ba17659d90d1@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com>
	<9457e7c81003141243v39aa5f63s5c280c9c02796151@mail.gmail.com>
	<4B9D7E46.5030509@noaa.gov>
	<961fa2b41003142147i30a8f114ld97ba17659d90d1@mail.gmail.com>
Message-ID: <4B9E7366.8060802@noaa.gov>

Nathaniel Smith wrote:
> On Sun, Mar 14, 2010 at 5:24 PM, Christopher Barker
> <Chris.Barker at noaa.gov> wrote:
>> It might be nice to build something on libbzip2 -- It looks like the
>> license is right, it's got good compression qualities, supports 64 bit,
>> and it getting pretty widely used.
> 
> If we're going to pull in a new C library for this purpose, then maybe
> we should just use libhdf5? :-).

except that hdf is one big honking pain in the butt to build.

Robert's right, bzip isn't a great option anyway -- it can do multiple 
files, but it just concatenates them, and doesn't appear to provide an 
index. I used to use afio, which would zip each file first, then put 
them all in an archive -- I liked that approach. We could, of course, 
build something like that with bzip, but it looks like python's zip will 
work for >= 2.6, so no need for something new.

-Chris






-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From dalloliogm at gmail.com  Fri Mar 12 12:44:04 2010
From: dalloliogm at gmail.com (Giovanni Marco Dall'Olio)
Date: Fri, 12 Mar 2010 18:44:04 +0100
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com> 
	<3d375d731003110748q2055b3e1v1cd5af42721a123d@mail.gmail.com> 
	<4B99F23F.1090508@sun.ac.za>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
Message-ID: <5aa3b3571003120944p1e8ced27p9ceac336da1ff5b7@mail.gmail.com>

On Fri, Mar 12, 2010 at 6:22 PM, Paul Anton Letnes <
paul.anton.letnes at gmail.com> wrote:

>
>
>  (Or, alternatively, did I miss a great netCDF python module that someone
> could tell me about?)
>

As for the documentation, PyTables can also read/write netCDF:
- http://www.pytables.org/docs/manual/ch07.html



-- 
Giovanni Dall'Olio, phd student
Department of Biologia Evolutiva at CEXS-UPF (Barcelona, Spain)

My blog on bioinformatics: http://bioinfoblog.it
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100312/c96444b6/attachment.html>

From robert.kern at gmail.com  Mon Mar 15 13:56:01 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 15 Mar 2010 12:56:01 -0500
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
Message-ID: <3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com>

On Mon, Mar 15, 2010 at 12:32, denis <denis-bz-gg at t-online.de> wrote:
> Where should plots and other pngs refd from notes in
> advice.mechanicalkern.com be put ?

Wherever you like. There are many free image hosting sites available.
Try this one:

http://imgur.com/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From faltet at pytables.org  Mon Mar 15 14:20:51 2010
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 15 Mar 2010 19:20:51 +0100
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <5aa3b3571003120944p1e8ced27p9ceac336da1ff5b7@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<6A77BA10-5933-41C4-B837-F1B080C446E7@gmail.com>
	<5aa3b3571003120944p1e8ced27p9ceac336da1ff5b7@mail.gmail.com>
Message-ID: <201003151920.51296.faltet@pytables.org>

A Friday 12 March 2010 18:44:04 Giovanni Marco Dall'Olio escrigu?:
> On Fri, Mar 12, 2010 at 6:22 PM, Paul Anton Letnes <
> 
> paul.anton.letnes at gmail.com> wrote:
> >  (Or, alternatively, did I miss a great netCDF python module that someone
> > could tell me about?)
> 
> As for the documentation, PyTables can also read/write netCDF:
> - http://www.pytables.org/docs/manual/ch07.html

No.  The above is only an API that emulates the NetCDF interface of the 
Scientific package, and do not create (nor can read either) pure NetCDF files.  
For creating/reading NetCDF files, better use the netcdf4-python project:

 http://code.google.com/p/netcdf4-python/

-- 
Francesc Alted


From faltet at pytables.org  Mon Mar 15 14:26:30 2010
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 15 Mar 2010 19:26:30 +0100
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <4B9E7366.8060802@noaa.gov>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<961fa2b41003142147i30a8f114ld97ba17659d90d1@mail.gmail.com>
	<4B9E7366.8060802@noaa.gov>
Message-ID: <201003151926.30498.faltet@pytables.org>

A Monday 15 March 2010 18:50:30 Christopher Barker escrigu?:
> > If we're going to pull in a new C library for this purpose, then maybe
> > we should just use libhdf5? :-).
> 
> except that hdf is one big honking pain in the butt to build.

I wouldn't say that HDF5 it is very difficult to build/install.  In fact, it 
is a matter of "./configure; make install" --and that only if there is not a 
binary package available for your SO, which is usually the case.  It is just 
that it is another dependency to add to numpy/scipy ...and a *big* one.

-- 
Francesc Alted


From denis-bz-gg at t-online.de  Mon Mar 15 14:27:21 2010
From: denis-bz-gg at t-online.de (denis)
Date: Mon, 15 Mar 2010 11:27:21 -0700 (PDT)
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com>
Message-ID: <dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com>

On Mar 15, 6:56?pm, Robert Kern <robert.k... at gmail.com> wrote:
> On Mon, Mar 15, 2010 at 12:32, denis <denis-bz... at t-online.de> wrote:
> > Where should plots and other pngs refd from notes in
> > advice.mechanicalkern.com be put ?
>
> Wherever you like. There are many free image hosting sites available.
> Try this one:
>
> http://imgur.com/

Ok, but most of them delete files after a while. Is there a better
place
that would ease the (planned ?) migration of advice to scipy.org ?



From robert.kern at gmail.com  Mon Mar 15 14:41:03 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 15 Mar 2010 13:41:03 -0500
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com> 
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com>
Message-ID: <3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com>

On Mon, Mar 15, 2010 at 13:27, denis <denis-bz-gg at t-online.de> wrote:
> On Mar 15, 6:56?pm, Robert Kern <robert.k... at gmail.com> wrote:
>> On Mon, Mar 15, 2010 at 12:32, denis <denis-bz... at t-online.de> wrote:
>> > Where should plots and other pngs refd from notes in
>> > advice.mechanicalkern.com be put ?
>>
>> Wherever you like. There are many free image hosting sites available.
>> Try this one:
>>
>> http://imgur.com/
>
> Ok, but most of them delete files after a while.

imgur is nice in that they will not delete images as long as they are
getting at least one view every three months.

> Is there a better
> place
> that would ease the (planned ?) migration of advice to scipy.org ?

There won't be a migration, sorry. The software is too different.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From andrew.collette at gmail.com  Mon Mar 15 14:56:32 2010
From: andrew.collette at gmail.com (Andrew Collette)
Date: Mon, 15 Mar 2010 11:56:32 -0700
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <201003151926.30498.faltet@pytables.org>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<961fa2b41003142147i30a8f114ld97ba17659d90d1@mail.gmail.com>
	<4B9E7366.8060802@noaa.gov> <201003151926.30498.faltet@pytables.org>
Message-ID: <f125f5e1003151156l508e31f8lcc8746f703d9a294@mail.gmail.com>

> I wouldn't say that HDF5 it is very difficult to build/install. ?In fact, it
> is a matter of "./configure; make install" --and that only if there is not a
> binary package available for your SO, which is usually the case. ?It is just
> that it is another dependency to add to numpy/scipy ...and a *big* one.

Except on Windows, of course, where you need to have Visual Studio and
a lot of patience. :) Even on UNIX one of the major support issues
I've had with h5py is that everyone has a slightly different version
of HDF5, built in a slightly different way.

Andrew


From tmp50 at ukr.net  Mon Mar 15 15:11:06 2010
From: tmp50 at ukr.net (Dmitrey)
Date: Mon, 15 Mar 2010 21:11:06 +0200
Subject: [SciPy-User] [ANN] OpenOpt 0.28, FuncDesigner 0.18,
	DerApproximator 0.18
Message-ID: <E1NrFgw-000LDm-AU@ffe15.ukr.net>

Hi all,  
I'm glad to inform you about new release of our free (BSD-licensed) soft OpenOpt 0.28, FuncDesigner 0.18, DerApproximator 0.18  
More details here:  
http://forum.openopt.org/viewtopic.php?id=222  
  
Regards, D.  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/40ed66b7/attachment.html>

From Chris.Barker at noaa.gov  Mon Mar 15 15:25:09 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 15 Mar 2010 12:25:09 -0700
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <201003151926.30498.faltet@pytables.org>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<961fa2b41003142147i30a8f114ld97ba17659d90d1@mail.gmail.com>
	<4B9E7366.8060802@noaa.gov> <201003151926.30498.faltet@pytables.org>
Message-ID: <4B9E8995.9070804@noaa.gov>

Francesc Alted wrote:

> I wouldn't say that HDF5 it is very difficult to build/install.  In fact, it 
> is a matter of "./configure; make install" --and that only if there is not a 
> binary package available for your SO, which is usually the case.

clearly, you've never tired to build a Universal binary on OS-X ;-)

>  It is just 
> that it is another dependency to add to numpy/scipy ...and a *big* one.

yeah, that too.

-Chris




-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From faltet at pytables.org  Mon Mar 15 16:04:20 2010
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 15 Mar 2010 21:04:20 +0100
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <f125f5e1003151156l508e31f8lcc8746f703d9a294@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<201003151926.30498.faltet@pytables.org>
	<f125f5e1003151156l508e31f8lcc8746f703d9a294@mail.gmail.com>
Message-ID: <201003152104.20138.faltet@pytables.org>

A Monday 15 March 2010 19:56:32 Andrew Collette escrigu?:
> > I wouldn't say that HDF5 it is very difficult to build/install.  In fact,
> > it is a matter of "./configure; make install" --and that only if there is
> > not a binary package available for your SO, which is usually the case.
> >  It is just that it is another dependency to add to numpy/scipy ...and a
> > *big* one.
> 
> Except on Windows, of course, where you need to have Visual Studio and
> a lot of patience. :)

Yeah, Windows, but this is a platform where everybody expects to have a 
binary ...and fortunately HDF5 is not an exception there. :)

> Even on UNIX one of the major support issues
> I've had with h5py is that everyone has a slightly different version
> of HDF5, built in a slightly different way.

Well, I cannot say the same with PyTables, but that could be somewhat 
expected, as you try to support much more HDF5 low level features than I do.

-- 
Francesc Alted


From faltet at pytables.org  Mon Mar 15 16:07:44 2010
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 15 Mar 2010 21:07:44 +0100
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <4B9E8995.9070804@noaa.gov>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<201003151926.30498.faltet@pytables.org>
	<4B9E8995.9070804@noaa.gov>
Message-ID: <201003152107.44321.faltet@pytables.org>

A Monday 15 March 2010 20:25:09 Christopher Barker escrigu?:
> Francesc Alted wrote:
> > I wouldn't say that HDF5 it is very difficult to build/install.  In fact,
> > it is a matter of "./configure; make install" --and that only if there is
> > not a binary package available for your SO, which is usually the case.
> 
> clearly, you've never tired to build a Universal binary on OS-X ;-)

Ok, touch?.  But is there any package for which it is easy to build an 
Universal binary on Mac OS-X? ;-)

-- 
Francesc Alted


From robert.kern at gmail.com  Mon Mar 15 16:11:39 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 15 Mar 2010 15:11:39 -0500
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <201003152107.44321.faltet@pytables.org>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com> 
	<201003151926.30498.faltet@pytables.org> <4B9E8995.9070804@noaa.gov> 
	<201003152107.44321.faltet@pytables.org>
Message-ID: <3d375d731003151311o2972b415qa8286126c6a64764@mail.gmail.com>

On Mon, Mar 15, 2010 at 15:07, Francesc Alted <faltet at pytables.org> wrote:
> A Monday 15 March 2010 20:25:09 Christopher Barker escrigu?:
>> Francesc Alted wrote:
>> > I wouldn't say that HDF5 it is very difficult to build/install. ?In fact,
>> > it is a matter of "./configure; make install" --and that only if there is
>> > not a binary package available for your SO, which is usually the case.
>>
>> clearly, you've never tired to build a Universal binary on OS-X ;-)
>
> Ok, touch?. ?But is there any package for which it is easy to build an
> Universal binary on Mac OS-X? ;-)

numpy.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From faltet at pytables.org  Mon Mar 15 16:16:06 2010
From: faltet at pytables.org (Francesc Alted)
Date: Mon, 15 Mar 2010 21:16:06 +0100
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <3d375d731003151311o2972b415qa8286126c6a64764@mail.gmail.com>
References: <91d218431003110310s6b8fa379sfc2bd1c120e264b2@mail.gmail.com>
	<201003152107.44321.faltet@pytables.org>
	<3d375d731003151311o2972b415qa8286126c6a64764@mail.gmail.com>
Message-ID: <201003152116.06249.faltet@pytables.org>

A Monday 15 March 2010 21:11:39 Robert Kern escrigu?:
> On Mon, Mar 15, 2010 at 15:07, Francesc Alted <faltet at pytables.org> wrote:
> > A Monday 15 March 2010 20:25:09 Christopher Barker escrigu?:
> >> Francesc Alted wrote:
> >> > I wouldn't say that HDF5 it is very difficult to build/install.  In
> >> > fact, it is a matter of "./configure; make install" --and that only if
> >> > there is not a binary package available for your SO, which is usually
> >> > the case.
> >>
> >> clearly, you've never tired to build a Universal binary on OS-X ;-)
> >
> > Ok, touch?.  But is there any package for which it is easy to build an
> > Universal binary on Mac OS-X? ;-)
> 
> numpy.

Good!  So, I should start by looking at how it achieves that then :)

-- 
Francesc Alted


From timmichelsen at gmx-topmail.de  Mon Mar 15 18:25:55 2010
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Mon, 15 Mar 2010 23:25:55 +0100
Subject: [SciPy-User] usemask kwd in timeseries.tsfromtxt missing?
In-Reply-To: <4A98946E-5539-4274-A1B4-778D6D672BE7@gmail.com>
References: <c048da1c1003121359k2984c9e3k2dcc100a7f2e3500@mail.gmail.com>	<hnfju4$8d1$1@dough.gmane.org>	<c048da1c1003130843g59f5aebbu1fd576f5f1c3d2c@mail.gmail.com>	<hnjo5a$st$1@dough.gmane.org>
	<4A98946E-5539-4274-A1B4-778D6D672BE7@gmail.com>
Message-ID: <hnmc5k$gm1$1@dough.gmane.org>

> which is a subclass of MaskedArray. Therefore, `usemask` would always
> be True. No need for a keyword argument that you can't modify, hence
> the error message.
Then, why is this argument not removed?
Because of the inheritance from genfromtxt?

Maybe it's worth mentioning this in the docstring.



From david_baddeley at yahoo.com.au  Mon Mar 15 18:40:06 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Mon, 15 Mar 2010 15:40:06 -0700 (PDT)
Subject: [SciPy-User] Shared memory using multiprocessing.sharedctypes
Message-ID: <139766.901.qm@web33001.mail.mud.yahoo.com>

Here's the code, hope it's useful. 

cheers,
David

--- On Tue, 16/3/10, Charles Vejnar <ce at vejnar.eu> wrote:

> From: Charles Vejnar <ce at vejnar.eu>
> Subject: Re: [SciPy-User] Shared memory using multiprocessing.sharedctypes
> To: "SciPy Users List" <scipy-user at scipy.org>
> Cc: "David Baddeley" <david_baddeley at yahoo.com.au>
> Received: Tuesday, 16 March, 2010, 4:31 AM
> Hi David,
> 
> I am interested. Can you please show us your code ?
> 
> BW,
> Charles
> 
> On Monday 15 March 2010, David Baddeley wrote:
> > Thanks!
> > 
> > Will have to give it a spin. Unfortunately? (or
> fortunately - as it was
> >? quite instructive from a learning perspective) I
> got bored and already
> >? knocked up my own version. It delegates all the
> memory management to
> >? multiprocessing so weighs in pretty light at
> only 80 lines of pure python
> >? & works well on 64 bit systems. Just in case
> anyone wants yet another
> >? shared memory array implementation.
> > 
> > cheers,
> > David
> > 
> > 
> > ----- Original Message ----
> > From: Christopher Lee-Messer <chris.leemesser at gmail.com>
> > To: scipy-user at scipy.org
> > Sent: Sun, 14 March, 2010 5:45:08 AM
> > Subject: [SciPy-User]? Shared memory using
> multiprocessing.sharedctypes
> > 
> > David Baddeley wrote::
> > > I'd like shared memory numpy arrays for some code
> I'm trying to
> > > parallelise. The solution should ideally be cross
> platform. I remember
> > > some earlier discussion about this and, as far as
> I recall Sturla had a
> > > working implementation which was (unfortunately)
> linux only (please
> > > correct me if I'm wrong).
> > >
> > > Looking a bit further,
> multiprocessing.sharedctypes seems to have all the
> > > stuff needed to handle the low level bits on at
> least linux and windows,
> > > and it seems as though it should be relatively
> easy to cobble something
> > > together on top of this which let you use shared
> memory numpy arrays
> > > relatively transparently. This would have the
> advantage that all the
> > > platform specific nasty stuff is being handled by
> the multiprocessing
> > > module, which as it's now part of the standard
> library, should hopefully
> > > be well maintained & has a good chance of
> being ported to additional
> > > platforms
> > >
> > > Before I jump in and try writing such a wrapper -
> essentially a helper
> > > function for creating shared numpy arrays (easy)
> and some pickle handlers
> > > to make sure what comes out the other end looks
> like an array (would need
> > > to think/read a bit more, but should be doable) I
> wanted to make sure
> > > that there's not something out there already
> & that I'm not going to be
> > > reinventing the wheel.
> > 
> > [Note, I posted this via google groups originally
> before realizing
> > that it wasn't the official list. Sorry if you get
> multiple versions.
> > -chris]
> > 
> > Hi David,
> > I was just in the same boat.? I checked out
> Sturla's a few weeks ago
> > and tried it out on windows? xp and now linux
> (32bit, python 2.6 with
> > numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with
> numpy 1.2.1)
> > 
> > It's working very well for me in soft-realtime data
> acquisition
> > application with sizes of 1 to 800KB.
> > 
> > In case it saves you time, I recently threw together
> some packaging
> > for the code and put it on bitbucket as a
> numpy-sharedmem repository
> > in case it might save me or someone else some time.
> > 
> > http://bitbucket.org/cleemesser/numpy-sharedmem/
> > 
> > I don't know if Sturla, Gael Varoquaux, or Robert Kern
> are continuing
> > to work on these, but I plan to add testing and save
> results on
> > different platforms as my app gets used in different
> computers in the
> > lab.? Nadav Horesh mentioned that there are some
> limitations on the
> > size of the arrays for "large" arrays and Sturla
> mentioned that there
> > may be issue on 64bit systems.
> > -Chris
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> > 
> > 
> > 
> > 
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> > 
>


      
-------------- next part --------------
A non-text attachment was scrubbed...
Name: shmarray.py
Type: text/x-python
Size: 5246 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/eab3fd02/attachment.py>

From vincent at vincentdavis.net  Tue Mar 16 01:04:39 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Mon, 15 Mar 2010 23:04:39 -0600
Subject: [SciPy-User] Correlation coefficient of large arrays
Message-ID: <77e831101003152204x476b710blfd27a02c8e9ef184@mail.gmail.com>

I have an array 10 observations of 230,000 variables and what to find the
correlation coefficient between each variable.
numpy.corrcef(data) works except I can only do it with about 30,000
variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of
memory.
Is there a better way?

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/1662890d/attachment.html>

From josef.pktd at gmail.com  Tue Mar 16 01:16:14 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Mar 2010 01:16:14 -0400
Subject: [SciPy-User] Correlation coefficient of large arrays
In-Reply-To: <77e831101003152204x476b710blfd27a02c8e9ef184@mail.gmail.com>
References: <77e831101003152204x476b710blfd27a02c8e9ef184@mail.gmail.com>
Message-ID: <1cd32cbb1003152216h67d64af4gccb9e032d6557217@mail.gmail.com>

On Tue, Mar 16, 2010 at 1:04 AM, Vincent Davis <vincent at vincentdavis.net>wrote:

> I have an array 10 observations of 230,000 variables and what to find the
> correlation coefficient between each variable.
> numpy.corrcef(data) works except I can only do it with about 30,000
> variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of
> memory.
> Is there a better way?
>


how much memory does a
>>> 230000**2
52900000000L

float (double) array take ?

Josef
(I'm not going to try)



>
>   *Vincent Davis
> 720-301-3003 *
> vincent at vincentdavis.net
>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100316/bd802c45/attachment.html>

From vincent at vincentdavis.net  Tue Mar 16 01:39:45 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Mon, 15 Mar 2010 23:39:45 -0600
Subject: [SciPy-User] Correlation coefficient of large arrays
In-Reply-To: <1cd32cbb1003152216h67d64af4gccb9e032d6557217@mail.gmail.com>
References: <77e831101003152204x476b710blfd27a02c8e9ef184@mail.gmail.com>
	<1cd32cbb1003152216h67d64af4gccb9e032d6557217@mail.gmail.com>
Message-ID: <77e831101003152239t608852cdif290271585215025@mail.gmail.com>

>
> @Josef

how much memory does a

>>> 230000**2 = 52900000000L  float (double) array take ?



I guess I don't have a real appreciation for how large this is. I can do
this numpy.ones((100000,50000),dtype=np.float64) and it uses about 85% of
the memory I have available. But thats a long ways from 230,000X230,000. Of
course the array is symmetric.

Is it feasible to do writing it to the disk?
The end goal is to find the difference between two correlation arrays and
then calculate the mean of each column. Which then leaves me with an array
1X230,000

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Mon, Mar 15, 2010 at 11:16 PM, <josef.pktd at gmail.com> wrote:

>
>
> On Tue, Mar 16, 2010 at 1:04 AM, Vincent Davis <vincent at vincentdavis.net>wrote:
>
>> I have an array 10 observations of 230,000 variables and what to find the
>> correlation coefficient between each variable.
>> numpy.corrcef(data) works except I can only do it with about 30,000
>> variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of
>> memory.
>> Is there a better way?
>>
>
>
> how much memory does a
> >>> 230000**2
> 52900000000L
>
> float (double) array take ?
>
> Josef
> (I'm not going to try)
>
>
>
>>
>>   *Vincent Davis
>> 720-301-3003 *
>> vincent at vincentdavis.net
>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/1b285d4a/attachment.html>

From josef.pktd at gmail.com  Tue Mar 16 01:56:49 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Mar 2010 01:56:49 -0400
Subject: [SciPy-User] Correlation coefficient of large arrays
In-Reply-To: <77e831101003152239t608852cdif290271585215025@mail.gmail.com>
References: <77e831101003152204x476b710blfd27a02c8e9ef184@mail.gmail.com>
	<1cd32cbb1003152216h67d64af4gccb9e032d6557217@mail.gmail.com>
	<77e831101003152239t608852cdif290271585215025@mail.gmail.com>
Message-ID: <1cd32cbb1003152256iafacc4el7a5028ff0d3080a0@mail.gmail.com>

On Tue, Mar 16, 2010 at 1:39 AM, Vincent Davis <vincent at vincentdavis.net>wrote:

> @Josef
>
> how much memory does a
>
> >>> 230000**2 = 52900000000L  float (double) array take ?
>
>
>
> I guess I don't have a real appreciation for how large this is. I can do
> this numpy.ones((100000,50000),dtype=np.float64) and it uses about 85% of
> the memory I have available. But thats a long ways from 230,000X230,000. Of
> course the array is symmetric.
>
> Is it feasible to do writing it to the disk?
> The end goal is to find the difference between two correlation arrays and
> then calculate the mean of each column. Which then leaves me with an array
> 1X230,000
>

If you don't really care about the correlation matrix itself and only need
the column (or row) sum then I would just loop over it in batches and never
construct the full matrix.
e.g. take the first 1000 variables and calculate the correlation with all
variables (1000 * 230000 -> 1000 for sum)
and loop.
Not using np.corrcoef would avoid some duplicate calculations, but there are
still several intermediate arrays necessary. So maybe using pytables or
similar would still be better to avoid duplicate calculations.

Josef



>
>   *Vincent Davis
> 720-301-3003 *
> vincent at vincentdavis.net
>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>
>
> On Mon, Mar 15, 2010 at 11:16 PM, <josef.pktd at gmail.com> wrote:
>
>>
>>
>> On Tue, Mar 16, 2010 at 1:04 AM, Vincent Davis <vincent at vincentdavis.net>wrote:
>>
>>> I have an array 10 observations of 230,000 variables and what to find the
>>> correlation coefficient between each variable.
>>> numpy.corrcef(data) works except I can only do it with about 30,000
>>> variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of
>>> memory.
>>> Is there a better way?
>>>
>>
>>
>> how much memory does a
>> >>> 230000**2
>> 52900000000L
>>
>> float (double) array take ?
>>
>> Josef
>> (I'm not going to try)
>>
>>
>>
>>>
>>>   *Vincent Davis
>>> 720-301-3003 *
>>> vincent at vincentdavis.net
>>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100316/26a18f15/attachment.html>

From chris.leemesser at gmail.com  Tue Mar 16 02:27:19 2010
From: chris.leemesser at gmail.com (Christopher Lee-Messer)
Date: Mon, 15 Mar 2010 23:27:19 -0700
Subject: [SciPy-User] Shared memory using multiprocessing.sharedctypes
In-Reply-To: <139766.901.qm@web33001.mail.mud.yahoo.com>
References: <139766.901.qm@web33001.mail.mud.yahoo.com>
Message-ID: <4cccdc8b1003152327y6aa5354er7e89970f68b2c515@mail.gmail.com>

Excellent, and many thanks.  I've learned a lot by looking at your
code.  I didn't realize how much good stuff is in multiprocessing.
-chris

On Mon, Mar 15, 2010 at 3:40 PM, David Baddeley
<david_baddeley at yahoo.com.au> wrote:
> Here's the code, hope it's useful.
>
> cheers,
> David
>
> --- On Tue, 16/3/10, Charles Vejnar <ce at vejnar.eu> wrote:
>
>> From: Charles Vejnar <ce at vejnar.eu>
>> Subject: Re: [SciPy-User] Shared memory using multiprocessing.sharedctypes
>> To: "SciPy Users List" <scipy-user at scipy.org>
>> Cc: "David Baddeley" <david_baddeley at yahoo.com.au>
>> Received: Tuesday, 16 March, 2010, 4:31 AM
>> Hi David,
>>
>> I am interested. Can you please show us your code ?
>>
>> BW,
>> Charles
>>
>> On Monday 15 March 2010, David Baddeley wrote:
>> > Thanks!
>> >
>> > Will have to give it a spin. Unfortunately? (or
>> fortunately - as it was
>> >? quite instructive from a learning perspective) I
>> got bored and already
>> >? knocked up my own version. It delegates all the
>> memory management to
>> >? multiprocessing so weighs in pretty light at
>> only 80 lines of pure python
>> >? & works well on 64 bit systems. Just in case
>> anyone wants yet another
>> >? shared memory array implementation.
>> >
>> > cheers,
>> > David
>> >
>> >
>> > ----- Original Message ----
>> > From: Christopher Lee-Messer <chris.leemesser at gmail.com>
>> > To: scipy-user at scipy.org
>> > Sent: Sun, 14 March, 2010 5:45:08 AM
>> > Subject: [SciPy-User]? Shared memory using
>> multiprocessing.sharedctypes
>> >
>> > David Baddeley wrote::
>> > > I'd like shared memory numpy arrays for some code
>> I'm trying to
>> > > parallelise. The solution should ideally be cross
>> platform. I remember
>> > > some earlier discussion about this and, as far as
>> I recall Sturla had a
>> > > working implementation which was (unfortunately)
>> linux only (please
>> > > correct me if I'm wrong).
>> > >
>> > > Looking a bit further,
>> multiprocessing.sharedctypes seems to have all the
>> > > stuff needed to handle the low level bits on at
>> least linux and windows,
>> > > and it seems as though it should be relatively
>> easy to cobble something
>> > > together on top of this which let you use shared
>> memory numpy arrays
>> > > relatively transparently. This would have the
>> advantage that all the
>> > > platform specific nasty stuff is being handled by
>> the multiprocessing
>> > > module, which as it's now part of the standard
>> library, should hopefully
>> > > be well maintained & has a good chance of
>> being ported to additional
>> > > platforms
>> > >
>> > > Before I jump in and try writing such a wrapper -
>> essentially a helper
>> > > function for creating shared numpy arrays (easy)
>> and some pickle handlers
>> > > to make sure what comes out the other end looks
>> like an array (would need
>> > > to think/read a bit more, but should be doable) I
>> wanted to make sure
>> > > that there's not something out there already
>> & that I'm not going to be
>> > > reinventing the wheel.
>> >
>> > [Note, I posted this via google groups originally
>> before realizing
>> > that it wasn't the official list. Sorry if you get
>> multiple versions.
>> > -chris]
>> >
>> > Hi David,
>> > I was just in the same boat.? I checked out
>> Sturla's a few weeks ago
>> > and tried it out on windows? xp and now linux
>> (32bit, python 2.6 with
>> > numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with
>> numpy 1.2.1)
>> >
>> > It's working very well for me in soft-realtime data
>> acquisition
>> > application with sizes of 1 to 800KB.
>> >
>> > In case it saves you time, I recently threw together
>> some packaging
>> > for the code and put it on bitbucket as a
>> numpy-sharedmem repository
>> > in case it might save me or someone else some time.
>> >
>> > http://bitbucket.org/cleemesser/numpy-sharedmem/
>> >
>> > I don't know if Sturla, Gael Varoquaux, or Robert Kern
>> are continuing
>> > to work on these, but I plan to add testing and save
>> results on
>> > different platforms as my app gets used in different
>> computers in the
>> > lab.? Nadav Horesh mentioned that there are some
>> limitations on the
>> > size of the arrays for "large" arrays and Sturla
>> mentioned that there
>> > may be issue on 64bit systems.
>> > -Chris
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>



-- 
Christopher Lee-Messer MD PhD
Instructor in Pediatric Neurology
Postdoctoral Fellow - Deisseroth Lab
Stanford Medical Center
chris (at) lee-messer.net


From seb.haase at gmail.com  Tue Mar 16 04:40:14 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Tue, 16 Mar 2010 09:40:14 +0100
Subject: [SciPy-User] line_profiler and for-loops !?
Message-ID: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>

Hi,

I was starting to use Robert's line_profiler. I seems to work great,
and I already found one easy way do half my execution time.
But now it claims that 33% of the time is spent (directly) in the
"for"-line and another 36% in a very simple "if"-line. See parts of
the output here:
<snip>
Function: doTracing at line 1135
Total time: 23.9171 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
<snip>
  1185                                                       # iterate
over all tracks, and find close points
  1186   3853362      8024186      2.1     33.5              for
tracki,track in enumerate(self.tracks):
  1187   3853063      8639273      2.2     36.1                  if
self.tracks_tLast[tracki] == t-1:
  1188
                  # track went on until t-1 (so far)
  1189
                  #    --  otherwise, skip "old" tracks
  1190     62150       130277      2.1      0.5
   pi_t_1 = track[-1] # index in last time section
<snip>

(The object of the function is to connect closest points found in an
image sequence into tracks connecting the points by shortest steps.)

Anyhow, my question is, is this just an artifact of line_profiler, or
is the fact that those two lines are hit almost 4e6 times really
resulting in more than 50% of the time being spent here !?
(Calculating the actual Euclidean distance matrix over all point pairs
takes supposedly only 15% of the time, for comparison).
I tried to separate out the "enumerate(self.tracks)" into a separate
line before the "for"-line, but the time spent was still unchanged on
the "for".
Does this mean "python is slow" here - and I should try cython (which
i have never done so far ...) ?

Thanks,
Sebastian Haase


From dagss at student.matnat.uio.no  Tue Mar 16 04:49:11 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 16 Mar 2010 09:49:11 +0100
Subject: [SciPy-User] line_profiler and for-loops !?
In-Reply-To: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>
References: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>
Message-ID: <4B9F4607.6050805@student.matnat.uio.no>

Sebastian Haase wrote:
> Hi,
>
> I was starting to use Robert's line_profiler. I seems to work great,
> and I already found one easy way do half my execution time.
> But now it claims that 33% of the time is spent (directly) in the
> "for"-line and another 36% in a very simple "if"-line. See parts of
> the output here:
> <snip>
> Function: doTracing at line 1135
> Total time: 23.9171 s
>
> Line #      Hits         Time  Per Hit   % Time  Line Contents
> ==============================================================
> <snip>
>   1185                                                       # iterate
> over all tracks, and find close points
>   1186   3853362      8024186      2.1     33.5              for
> tracki,track in enumerate(self.tracks):
>   1187   3853063      8639273      2.2     36.1                  if
> self.tracks_tLast[tracki] == t-1:
>   1188
>                   # track went on until t-1 (so far)
>   1189
>                   #    --  otherwise, skip "old" tracks
>   1190     62150       130277      2.1      0.5
>    pi_t_1 = track[-1] # index in last time section
> <snip>
>
> (The object of the function is to connect closest points found in an
> image sequence into tracks connecting the points by shortest steps.)
>
> Anyhow, my question is, is this just an artifact of line_profiler, or
> is the fact that those two lines are hit almost 4e6 times really
> resulting in more than 50% of the time being spent here !?
> (Calculating the actual Euclidean distance matrix over all point pairs
> takes supposedly only 15% of the time, for comparison).
> I tried to separate out the "enumerate(self.tracks)" into a separate
> line before the "for"-line, but the time spent was still unchanged on
> the "for".
> Does this mean "python is slow" here - and I should try cython (which
> i have never done so far ...) ?
>   
Well, that for-line contains enumerate, which is a very non-trivial 
thing (a generator) and results in several functions calls being made. 
And that if-test line contains self.tracks_tLast[...] which is rather 
costly (one attribute lookup to get tracks_tLast (which you can put in a 
function local variable for a minor speedup), and then element indexing 
certainly isn't free either).

At least for the latter, put the value lookup into a temporary variable 
in a seperate line to get better profiling results.

But at one point the running times are so low that line-profiling gets 
very inaccurate, and if you still need more speed that's when you want 
to use Cython. If these lookups indeed dominate, Cython should give good 
results.

Dag Sverre


From meesters at gmx.de  Tue Mar 16 05:16:27 2010
From: meesters at gmx.de (Christian Meesters)
Date: Tue, 16 Mar 2010 10:16:27 +0100
Subject: [SciPy-User] line_profiler and for-loops !?
In-Reply-To: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>
References: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>
Message-ID: <20100316091627.11120@gmx.net>

Hallo Sebastian,

ich kann nicht direkt auf Deine Frage antworten, weil ich weder Deinen Code noch line_profiler kenne. Aber ich habe ein C++-Modul zur Berechnung einer Abstandsverteilungsfunktion (Berechnung aller euklidischen Distanzen innerhalb einer Punktwolke). Das Modul nutzt keinerlei numpy-header (weil ich es in eine reine C++-App mal einbauen wollte) und ist mit SWIG gewrappt. Es ist rel. schnell: ca. 2x so schnell wie ein (noch mit Pyrex gebautes) Vergleichsmodul. Aber Gael sagte ja, da? Cython schneller sein kann und einfacher zu coden.

Wie auch immer: Wenn Du Interesse hast lasse ich es Dir zukommen.

Sehen wir uns in Paris?

Besten Gru?,
Christian


-------- Original-Nachricht --------
> Datum: Tue, 16 Mar 2010 09:40:14 +0100
> Von: Sebastian Haase <seb.haase at gmail.com>
> An: SciPy Users List <SciPy-user at scipy.org>
> Betreff: [SciPy-User] line_profiler and for-loops !?

> Hi,
> 
> I was starting to use Robert's line_profiler. I seems to work great,
> and I already found one easy way do half my execution time.
> But now it claims that 33% of the time is spent (directly) in the
> "for"-line and another 36% in a very simple "if"-line. See parts of
> the output here:
> <snip>
> Function: doTracing at line 1135
> Total time: 23.9171 s
> 
> Line #      Hits         Time  Per Hit   % Time  Line Contents
> ==============================================================
> <snip>
>   1185                                                       # iterate
> over all tracks, and find close points
>   1186   3853362      8024186      2.1     33.5              for
> tracki,track in enumerate(self.tracks):
>   1187   3853063      8639273      2.2     36.1                  if
> self.tracks_tLast[tracki] == t-1:
>   1188
>                   # track went on until t-1 (so far)
>   1189
>                   #    --  otherwise, skip "old" tracks
>   1190     62150       130277      2.1      0.5
>    pi_t_1 = track[-1] # index in last time section
> <snip>
> 
> (The object of the function is to connect closest points found in an
> image sequence into tracks connecting the points by shortest steps.)
> 
> Anyhow, my question is, is this just an artifact of line_profiler, or
> is the fact that those two lines are hit almost 4e6 times really
> resulting in more than 50% of the time being spent here !?
> (Calculating the actual Euclidean distance matrix over all point pairs
> takes supposedly only 15% of the time, for comparison).
> I tried to separate out the "enumerate(self.tracks)" into a separate
> line before the "for"-line, but the time spent was still unchanged on
> the "for".
> Does this mean "python is slow" here - and I should try cython (which
> i have never done so far ...) ?
> 
> Thanks,
> Sebastian Haase
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-- 
GMX DSL: Internet, Telefon und Entertainment f?r nur 19,99 EUR/mtl.!
http://portal.gmx.net/de/go/dsl02


From meesters at gmx.de  Tue Mar 16 05:18:06 2010
From: meesters at gmx.de (Christian Meesters)
Date: Tue, 16 Mar 2010 10:18:06 +0100
Subject: [SciPy-User] line_profiler and for-loops !?
In-Reply-To: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>
References: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>
Message-ID: <20100316091806.11140@gmx.net>

Sorry for that last mail of mine - I intended to address Sebastian directly. 

Ouch!

Christian
-- 
GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01


From david_baddeley at yahoo.com.au  Tue Mar 16 06:27:05 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Tue, 16 Mar 2010 03:27:05 -0700 (PDT)
Subject: [SciPy-User] line_profiler and for-loops !?
In-Reply-To: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>
Message-ID: <586034.79517.qm@web33004.mail.mud.yahoo.com>

I think there are a couple of factors here - python for loops and if statements are bad news performance wise, so if you can remove them using cython / c you'll gain a lot. On the other hand they tend to suffer more of a performance penalty when you've got the line profiling hooks in place than, for example numpy calls which are vectorised and doing all the heavy lifting in places which aren't visible to the profiler. 

You might also be able to improve performance by doing something like:

for tracki, track in enumerate(self.tracks[self.tracks_tlast == t-1]):

if tracks and tracks_tlast are numpy arrays.

Just out of interest, how many points/observations are you trying to track? I've been working on something for stitching together tracks from lists of positions as well and would be interested in comparing strategies.

David


--- On Tue, 16/3/10, Sebastian Haase <seb.haase at gmail.com> wrote:

> From: Sebastian Haase <seb.haase at gmail.com>
> Subject: [SciPy-User] line_profiler and for-loops !?
> To: "SciPy Users List" <SciPy-user at scipy.org>
> Received: Tuesday, 16 March, 2010, 9:40 PM
> Hi,
> 
> I was starting to use Robert's line_profiler. I seems to
> work great,
> and I already found one easy way do half my execution
> time.
> But now it claims that 33% of the time is spent (directly)
> in the
> "for"-line and another 36% in a very simple "if"-line. See
> parts of
> the output here:
> <snip>
> Function: doTracing at line 1135
> Total time: 23.9171 s
> 
> Line #? ? ? Hits? ? ?
> ???Time? Per Hit???%
> Time? Line Contents
> ==============================================================
> <snip>
> ? 1185? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ???# iterate
> over all tracks, and find close points
> ? 1186???3853362? ? ?
> 8024186? ? ? 2.1?
> ???33.5? ? ? ? ?
> ? ? for
> tracki,track in enumerate(self.tracks):
> ? 1187???3853063? ? ?
> 8639273? ? ? 2.2?
> ???36.1? ? ? ? ?
> ? ? ? ? if
> self.tracks_tLast[tracki] == t-1:
> ? 1188
> ? ? ? ? ? ? ? ?
> ? # track went on until t-1 (so far)
> ? 1189
> ? ? ? ? ? ? ? ?
> ? #? ? --? otherwise, skip "old" tracks
> ? 1190? ???62150? ?
> ???130277? ? ? 2.1?
> ? ? 0.5
> ???pi_t_1 = track[-1] # index in last time
> section
> <snip>
> 
> (The object of the function is to connect closest points
> found in an
> image sequence into tracks connecting the points by
> shortest steps.)
> 
> Anyhow, my question is, is this just an artifact of
> line_profiler, or
> is the fact that those two lines are hit almost 4e6 times
> really
> resulting in more than 50% of the time being spent here !?
> (Calculating the actual Euclidean distance matrix over all
> point pairs
> takes supposedly only 15% of the time, for comparison).
> I tried to separate out the "enumerate(self.tracks)" into a
> separate
> line before the "for"-line, but the time spent was still
> unchanged on
> the "for".
> Does this mean "python is slow" here - and I should try
> cython (which
> i have never done so far ...) ?
> 
> Thanks,
> Sebastian Haase
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 


      


From general.mooney at googlemail.com  Tue Mar 16 06:58:16 2010
From: general.mooney at googlemail.com (=?UTF-8?Q?Ciar=C3=A1n_Mooney?=)
Date: Tue, 16 Mar 2010 10:58:16 +0000
Subject: [SciPy-User] [ANN] EuroPython 2010 registration and talk
	submissions now open!
Message-ID: <3e4e51a81003160358i63b5c398m6bdf52b9fdc0eebc@mail.gmail.com>

EuroPython 2010 - 17th to 24th July 2010
----------------------------------------
EuroPython is a conference for the Python programming language
community, including the Django, Zope and Plone communities. It is
aimed at everyone in the Python community, of all skill levels, both
users and programmers.

Last year's conference was the largest open source conference in the
UK and one of the largest community organised software conferences in
Europe.

This year EuroPython will be held from the 17th to 24th July in
Birmingham, UK. It will include over 100 talks, tutorials, sprints and
social events.

Registration
------------
Registration is open now at: http://www.europython.eu/registration/

For the best registration rates, book as soon as you can! Extra Early
Bird closes soon, after which normal Early Bird rate will apply until
10th May

Talks, Activities and Events
----------------------------
Do you have something you wish to present at EuroPython? You want to
give a talk, run a tutorial or sprint?
Go to http://www.europython.eu/talks/cfp/ for information and advice!
Go to http://wiki.europython.eu/Sprints to plan a sprint!

Help Us Out
-----------
EuroPython is run by volunteers, like you! We could use a hand, and
any contribution is welcome.
Go to http://wiki.europython.eu/Helping to join us!
Go to http://www.europython.eu/contact/ to contact us directly!

Sponsors
--------
Sponsoring EuroPython is a unique opportunity to affiliate with this
prestigious conference and to reach a large number of Python users
from computing professionals to academics, from entrepreneurs to
motivated and well-educated job seekers.
http://www.europython.eu/sponsors/

Spread the Word
---------------
We are a community-run not-for-profit conference. Please help to
spread the word by distributing this announcement to colleagues,
project mailing lists, friends, your blog, Web site, and through your
social networking connections. Take a look at our publicity resources:
http://wiki.europython.eu/Publicity

General Information
-------------------
For more information about the conference, please visit the official
site: http://www.europython.eu/

Looking forward to see you!
The EuroPython Team


From vincent at vincentdavis.net  Tue Mar 16 09:18:32 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Tue, 16 Mar 2010 07:18:32 -0600
Subject: [SciPy-User] [R] Correlation coefficient of large data sets
In-Reply-To: <3c29f3e21003152306y349accffr4cf1620e842d66b6@mail.gmail.com>
References: <77e831101003152212n47b2f850l7c62ee0cbd7dbc7a@mail.gmail.com>
	<3c29f3e21003152232l6a467ff9r1a40a0491bd8a522@mail.gmail.com>
	<77e831101003152243p1fe84144ge9132d16171e2cc8@mail.gmail.com>
	<3c29f3e21003152306y349accffr4cf1620e842d66b6@mail.gmail.com>
Message-ID: <77e831101003160618v1fb877bs209530286c7277a2@mail.gmail.com>

>
>  @ Dennis

With 35000 variables at a time, the storage is under

20 Gb; you'd have to compute about 50 such chunks to get the entire matrix.


Is there a way to calculate a column or row of the correlation matrix one at
a time? I ma looking how including an additional set of observation effect
the correlation. For example if I have variables a,b,c,d..... and set of
observations 1-10 if the correlation is calculated for obs 1-5, I then add
observations 6-10 and what to know the average effect of this on the
correlation of c with (a,b,,d,e.....).
So I only need a column or a row at a time. Just not clear to me how I would
do this.

@Joshua Wiley

cor(my.data) # calculate the correlation matrix between all variables
> (columns) of my.data
>

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Tue, Mar 16, 2010 at 12:06 AM, Joshua Wiley <jwiley.psych at gmail.com>wrote:

> I think what you have done should be fine.  read.table() will return a
> data frame, which cor() can handle happily.  For example:
>
> my.data <- read.table("file.csv", header = TRUE, row.names = 1,
> sep=",", strip.white = TRUE) # assign your data to "my.data"
>
> cor(my.data) # calculate the correlation matrix between all variables
> (columns) of my.data
>
> What happens if you try that?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100316/94c076db/attachment.html>

From vincent at vincentdavis.net  Tue Mar 16 09:21:15 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Tue, 16 Mar 2010 07:21:15 -0600
Subject: [SciPy-User] Correlation coefficient of large arrays
In-Reply-To: <1cd32cbb1003152256iafacc4el7a5028ff0d3080a0@mail.gmail.com>
References: <77e831101003152204x476b710blfd27a02c8e9ef184@mail.gmail.com>
	<1cd32cbb1003152216h67d64af4gccb9e032d6557217@mail.gmail.com>
	<77e831101003152239t608852cdif290271585215025@mail.gmail.com>
	<1cd32cbb1003152256iafacc4el7a5028ff0d3080a0@mail.gmail.com>
Message-ID: <77e831101003160621i3c5db34ft577d604c0d379cce@mail.gmail.com>

Is there a way to calculate a column or row of the correlation matrix one at
a time?  I ma looking how including an additional set of observation effect
the correlation. For example if I have variables a,b,c,d..... and set of
observations 1-10 if the correlation is calculated for obs 1-5, I then add
observations 6-10 and what to know the average effect of this on the
correlation of c with (a,b,,d,e.....).
So I only need a column or a row at a time.
Just not clear to me how I would do this. I guess I just need to just DO IT.

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Mon, Mar 15, 2010 at 11:56 PM, <josef.pktd at gmail.com> wrote:

>
>
> On Tue, Mar 16, 2010 at 1:39 AM, Vincent Davis <vincent at vincentdavis.net>wrote:
>
>>  @Josef
>>
>> how much memory does a
>>
>> >>> 230000**2 = 52900000000L  float (double) array take ?
>>
>>
>>
>> I guess I don't have a real appreciation for how large this is. I can do
>> this numpy.ones((100000,50000),dtype=np.float64) and it uses about 85% of
>> the memory I have available. But thats a long ways from 230,000X230,000. Of
>> course the array is symmetric.
>>
>> Is it feasible to do writing it to the disk?
>> The end goal is to find the difference between two correlation arrays and
>> then calculate the mean of each column. Which then leaves me with an array
>> 1X230,000
>>
>
> If you don't really care about the correlation matrix itself and only need
> the column (or row) sum then I would just loop over it in batches and never
> construct the full matrix.
> e.g. take the first 1000 variables and calculate the correlation with all
> variables (1000 * 230000 -> 1000 for sum)
> and loop.
> Not using np.corrcoef would avoid some duplicate calculations, but there
> are still several intermediate arrays necessary. So maybe using pytables or
> similar would still be better to avoid duplicate calculations.
>
> Josef
>
>
>
>>
>>   *Vincent Davis
>> 720-301-3003 *
>> vincent at vincentdavis.net
>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>
>>
>> On Mon, Mar 15, 2010 at 11:16 PM, <josef.pktd at gmail.com> wrote:
>>
>>>
>>>
>>> On Tue, Mar 16, 2010 at 1:04 AM, Vincent Davis <vincent at vincentdavis.net
>>> > wrote:
>>>
>>>> I have an array 10 observations of 230,000 variables and what to find
>>>> the correlation coefficient between each variable.
>>>> numpy.corrcef(data) works except I can only do it with about 30,000
>>>> variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of
>>>> memory.
>>>> Is there a better way?
>>>>
>>>
>>>
>>> how much memory does a
>>> >>> 230000**2
>>> 52900000000L
>>>
>>> float (double) array take ?
>>>
>>> Josef
>>> (I'm not going to try)
>>>
>>>
>>>
>>>>
>>>>   *Vincent Davis
>>>> 720-301-3003 *
>>>> vincent at vincentdavis.net
>>>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100316/2008e996/attachment.html>

From josef.pktd at gmail.com  Tue Mar 16 09:38:17 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 16 Mar 2010 09:38:17 -0400
Subject: [SciPy-User] Correlation coefficient of large arrays
In-Reply-To: <77e831101003160621i3c5db34ft577d604c0d379cce@mail.gmail.com>
References: <77e831101003152204x476b710blfd27a02c8e9ef184@mail.gmail.com>
	<1cd32cbb1003152216h67d64af4gccb9e032d6557217@mail.gmail.com>
	<77e831101003152239t608852cdif290271585215025@mail.gmail.com>
	<1cd32cbb1003152256iafacc4el7a5028ff0d3080a0@mail.gmail.com>
	<77e831101003160621i3c5db34ft577d604c0d379cce@mail.gmail.com>
Message-ID: <1cd32cbb1003160638p59bc90ccsa5609450f7725c45@mail.gmail.com>

On Tue, Mar 16, 2010 at 9:21 AM, Vincent Davis <vincent at vincentdavis.net>wrote:

> Is there a way to calculate a column or row of the correlation matrix one
> at a time?  I ma looking how including an additional set of observation
> effect the correlation. For example if I have variables a,b,c,d..... and set
> of observations 1-10 if the correlation is calculated for obs 1-5, I then
> add observations 6-10 and what to know the average effect of this on the
> correlation of c with (a,b,,d,e.....).
> So I only need a column or a row at a time.
> Just not clear to me how I would do this. I guess I just need to just DO
> IT.
>

I would loop by variable not by observations

example in attachment

Josef



>
>   *Vincent Davis
> 720-301-3003 *
> vincent at vincentdavis.net
>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>
>
> On Mon, Mar 15, 2010 at 11:56 PM, <josef.pktd at gmail.com> wrote:
>
>>
>>
>> On Tue, Mar 16, 2010 at 1:39 AM, Vincent Davis <vincent at vincentdavis.net>wrote:
>>
>>>  @Josef
>>>
>>> how much memory does a
>>>
>>> >>> 230000**2 = 52900000000L  float (double) array take ?
>>>
>>>
>>>
>>> I guess I don't have a real appreciation for how large this is. I can do
>>> this numpy.ones((100000,50000),dtype=np.float64) and it uses about 85% of
>>> the memory I have available. But thats a long ways from 230,000X230,000. Of
>>> course the array is symmetric.
>>>
>>> Is it feasible to do writing it to the disk?
>>> The end goal is to find the difference between two correlation arrays and
>>> then calculate the mean of each column. Which then leaves me with an array
>>> 1X230,000
>>>
>>
>> If you don't really care about the correlation matrix itself and only need
>> the column (or row) sum then I would just loop over it in batches and never
>> construct the full matrix.
>> e.g. take the first 1000 variables and calculate the correlation with all
>> variables (1000 * 230000 -> 1000 for sum)
>> and loop.
>> Not using np.corrcoef would avoid some duplicate calculations, but there
>> are still several intermediate arrays necessary. So maybe using pytables or
>> similar would still be better to avoid duplicate calculations.
>>
>> Josef
>>
>>
>>
>>>
>>>   *Vincent Davis
>>> 720-301-3003 *
>>> vincent at vincentdavis.net
>>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>>
>>>
>>> On Mon, Mar 15, 2010 at 11:16 PM, <josef.pktd at gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Mar 16, 2010 at 1:04 AM, Vincent Davis <
>>>> vincent at vincentdavis.net> wrote:
>>>>
>>>>> I have an array 10 observations of 230,000 variables and what to find
>>>>> the correlation coefficient between each variable.
>>>>> numpy.corrcef(data) works except I can only do it with about 30,000
>>>>> variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of
>>>>> memory.
>>>>> Is there a better way?
>>>>>
>>>>
>>>>
>>>> how much memory does a
>>>> >>> 230000**2
>>>> 52900000000L
>>>>
>>>> float (double) array take ?
>>>>
>>>> Josef
>>>> (I'm not going to try)
>>>>
>>>>
>>>>
>>>>>
>>>>>   *Vincent Davis
>>>>> 720-301-3003 *
>>>>> vincent at vincentdavis.net
>>>>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100316/77550599/attachment.html>
-------------- next part --------------
'''calculating correlation coefficients sequentially for large number of
variables

A: josef-pktd
'''

import numpy as np

x = np.random.randn(4,5)
print np.corrcoef(x, rowvar=False)

xzs = (x-x.mean(0))/x.std(0)  #standardize
nobs, nvars = xzs.shape
rowsum = np.empty(nvars)
# calculate correlation coefficient for each variable with all others
for col in xrange(nvars):
    corr = np.dot(xzs[:,col],xzs)/nobs
    rowsum[col] = corr.sum()
    print corr

print '\nrowsums'
print rowsum
print np.corrcoef(x, rowvar=False).sum(1)
print np.max(np.abs(rowsum - np.corrcoef(x, rowvar=False).sum(1)))

From vincent at vincentdavis.net  Tue Mar 16 09:53:43 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Tue, 16 Mar 2010 07:53:43 -0600
Subject: [SciPy-User] Correlation coefficient of large arrays
In-Reply-To: <1cd32cbb1003160638p59bc90ccsa5609450f7725c45@mail.gmail.com>
References: <77e831101003152204x476b710blfd27a02c8e9ef184@mail.gmail.com>
	<1cd32cbb1003152216h67d64af4gccb9e032d6557217@mail.gmail.com>
	<77e831101003152239t608852cdif290271585215025@mail.gmail.com>
	<1cd32cbb1003152256iafacc4el7a5028ff0d3080a0@mail.gmail.com>
	<77e831101003160621i3c5db34ft577d604c0d379cce@mail.gmail.com>
	<1cd32cbb1003160638p59bc90ccsa5609450f7725c45@mail.gmail.com>
Message-ID: <77e831101003160653u3a25a918k15012b4497aa9991@mail.gmail.com>

>
> @ Josef

I would loop by variable not by observations

example in attachment


Thanks for that example, My wording was poor (rows and col are the equlalent
in a symmetric matrix) but yur example is what I was thinking but not sure
how to do.

Thanks again


  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Tue, Mar 16, 2010 at 7:38 AM, <josef.pktd at gmail.com> wrote:

>
>
> On Tue, Mar 16, 2010 at 9:21 AM, Vincent Davis <vincent at vincentdavis.net>wrote:
>
>> Is there a way to calculate a column or row of the correlation matrix one
>> at a time?  I ma looking how including an additional set of observation
>> effect the correlation. For example if I have variables a,b,c,d..... and set
>> of observations 1-10 if the correlation is calculated for obs 1-5, I then
>> add observations 6-10 and what to know the average effect of this on the
>> correlation of c with (a,b,,d,e.....).
>> So I only need a column or a row at a time.
>> Just not clear to me how I would do this. I guess I just need to just DO
>> IT.
>>
>
> I would loop by variable not by observations
>
> example in attachment
>
> Josef
>
>
>
>>
>>   *Vincent Davis
>> 720-301-3003 *
>> vincent at vincentdavis.net
>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>
>>
>> On Mon, Mar 15, 2010 at 11:56 PM, <josef.pktd at gmail.com> wrote:
>>
>>>
>>>
>>> On Tue, Mar 16, 2010 at 1:39 AM, Vincent Davis <vincent at vincentdavis.net
>>> > wrote:
>>>
>>>>  @Josef
>>>>
>>>> how much memory does a
>>>>
>>>> >>> 230000**2 = 52900000000L  float (double) array take ?
>>>>
>>>>
>>>>
>>>> I guess I don't have a real appreciation for how large this is. I can do
>>>> this numpy.ones((100000,50000),dtype=np.float64) and it uses about 85% of
>>>> the memory I have available. But thats a long ways from 230,000X230,000. Of
>>>> course the array is symmetric.
>>>>
>>>> Is it feasible to do writing it to the disk?
>>>> The end goal is to find the difference between two correlation arrays
>>>> and then calculate the mean of each column. Which then leaves me with an
>>>> array 1X230,000
>>>>
>>>
>>> If you don't really care about the correlation matrix itself and only
>>> need the column (or row) sum then I would just loop over it in batches and
>>> never construct the full matrix.
>>> e.g. take the first 1000 variables and calculate the correlation with all
>>> variables (1000 * 230000 -> 1000 for sum)
>>> and loop.
>>> Not using np.corrcoef would avoid some duplicate calculations, but there
>>> are still several intermediate arrays necessary. So maybe using pytables or
>>> similar would still be better to avoid duplicate calculations.
>>>
>>> Josef
>>>
>>>
>>>
>>>>
>>>>   *Vincent Davis
>>>> 720-301-3003 *
>>>> vincent at vincentdavis.net
>>>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>>>
>>>>
>>>> On Mon, Mar 15, 2010 at 11:16 PM, <josef.pktd at gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 16, 2010 at 1:04 AM, Vincent Davis <
>>>>> vincent at vincentdavis.net> wrote:
>>>>>
>>>>>> I have an array 10 observations of 230,000 variables and what to find
>>>>>> the correlation coefficient between each variable.
>>>>>> numpy.corrcef(data) works except I can only do it with about 30,000
>>>>>> variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of
>>>>>> memory.
>>>>>> Is there a better way?
>>>>>>
>>>>>
>>>>>
>>>>> how much memory does a
>>>>> >>> 230000**2
>>>>> 52900000000L
>>>>>
>>>>> float (double) array take ?
>>>>>
>>>>> Josef
>>>>> (I'm not going to try)
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>   *Vincent Davis
>>>>>> 720-301-3003 *
>>>>>> vincent at vincentdavis.net
>>>>>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100316/46d18839/attachment.html>

From denis-bz-gg at t-online.de  Tue Mar 16 09:54:49 2010
From: denis-bz-gg at t-online.de (denis)
Date: Tue, 16 Mar 2010 06:54:49 -0700 (PDT)
Subject: [SciPy-User] 3D Interpolation For Uneven Data?
In-Reply-To: <142682e11003120659h4924e5ebmf41ad599b42bd173@mail.gmail.com>
References: <142682e11003120659h4924e5ebmf41ad599b42bd173@mail.gmail.com>
Message-ID: <a1cf763d-0ba3-4c41-84f6-f57904b3f7cb@e7g2000yqf.googlegroups.com>

On Mar 12, 3:59?pm, Joseph Smidt <josephsm... at gmail.com> wrote:
> ? ? ?I have 3D data *not* equally-spaced. ?Is there a routine in scipy

Joseph, can you describe your data a bit ?
In: 1000, 1m points, oneshot / production
Out: interpolate 1 value at a time / grid / zoom ...
How smooth / noisy are inputs, how smooth should outputs be
What's the overall job -- visualize, just interpolate, estimate ...
One size does not fit all (TM).
cheers
  -- denis


From seb.haase at gmail.com  Tue Mar 16 09:58:57 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Tue, 16 Mar 2010 14:58:57 +0100
Subject: [SciPy-User] line_profiler and for-loops !?
In-Reply-To: <586034.79517.qm@web33004.mail.mud.yahoo.com>
References: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com> 
	<586034.79517.qm@web33004.mail.mud.yahoo.com>
Message-ID: <bc657ead1003160658h38b63f93l168a724e13dd9585@mail.gmail.com>

On Tue, Mar 16, 2010 at 11:27 AM, David Baddeley
<david_baddeley at yahoo.com.au> wrote:
> I think there are a couple of factors here - python for loops and if statements are bad news performance wise, so if you can remove them using cython / c you'll gain a lot. On the other hand they tend to suffer more of a performance penalty when you've got the line profiling hooks in place than, for example numpy calls which are vectorised and doing all the heavy lifting in places which aren't visible to the profiler.
>
> You might also be able to improve performance by doing something like:
>
> for tracki, track in enumerate(self.tracks[self.tracks_tlast == t-1]):
>
> if tracks and tracks_tlast are numpy arrays.

This did sound like an intriguing idea, but sadly they are all lists,
because they are appended to as new points are one-by-one being added
to the tracks.

>
> Just out of interest, how many points/observations are you trying to track? I've been working on something for stitching together tracks from lists of positions as well and would be interested in comparing strategies.
>

I just finished my first analysis of many of our files.  In total I'm
talking about 300GB of image data, where each "movie" might have 100
to few thousand image frames. At most my detection algorithms found
few hundreds (but sometimes up to 1000) points per frame.
The longest run took about 3 hours for up to 1400 points per frame
over a total of 2000 frames; but here I might have also run into
memory / swapping(?) issues. (I have 6GB memory RAM on a quad-code
64bit linux machine.)

I would really like to try cython on this innermost loop to get
everything re-analysed in a fraction of a day.

-Sebastian


> David
>
>
> --- On Tue, 16/3/10, Sebastian Haase <seb.haase at gmail.com> wrote:
>
>> From: Sebastian Haase <seb.haase at gmail.com>
>> Subject: [SciPy-User] line_profiler and for-loops !?
>> To: "SciPy Users List" <SciPy-user at scipy.org>
>> Received: Tuesday, 16 March, 2010, 9:40 PM
>> Hi,
>>
>> I was starting to use Robert's line_profiler. I seems to
>> work great,
>> and I already found one easy way do half my execution
>> time.
>> But now it claims that 33% of the time is spent (directly)
>> in the
>> "for"-line and another 36% in a very simple "if"-line. See
>> parts of
>> the output here:
>> <snip>
>> Function: doTracing at line 1135
>> Total time: 23.9171 s
>>
>> Line #? ? ? Hits
>> ???Time? Per Hit???%
>> Time? Line Contents
>> ==============================================================
>> <snip>
>> ? 1185
>>
>>
>> ? ? ? ???# iterate
>> over all tracks, and find close points
>> ? 1186???3853362
>> 8024186? ? ? 2.1
>> ???33.5
>> ? ? for
>> tracki,track in enumerate(self.tracks):
>> ? 1187???3853063
>> 8639273? ? ? 2.2
>> ???36.1
>> ? ? ? ? if
>> self.tracks_tLast[tracki] == t-1:
>> ? 1188
>>
>> ? # track went on until t-1 (so far)
>> ? 1189
>>
>> ? #? ? --? otherwise, skip "old" tracks
>> ? 1190? ???62150
>> ???130277? ? ? 2.1
>> ? ? 0.5
>> ???pi_t_1 = track[-1] # index in last time
>> section
>> <snip>
>>
>> (The object of the function is to connect closest points
>> found in an
>> image sequence into tracks connecting the points by
>> shortest steps.)
>>
>> Anyhow, my question is, is this just an artifact of
>> line_profiler, or
>> is the fact that those two lines are hit almost 4e6 times
>> really
>> resulting in more than 50% of the time being spent here !?
>> (Calculating the actual Euclidean distance matrix over all
>> point pairs
>> takes supposedly only 15% of the time, for comparison).
>> I tried to separate out the "enumerate(self.tracks)" into a
>> separate
>> line before the "for"-line, but the time spent was still
>> unchanged on
>> the "for".
>> Does this mean "python is slow" here - and I should try
>> cython (which
>> i have never done so far ...) ?
>>
>> Thanks,
>> Sebastian Haase
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From dagss at student.matnat.uio.no  Tue Mar 16 10:06:02 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 16 Mar 2010 15:06:02 +0100
Subject: [SciPy-User] line_profiler and for-loops !?
In-Reply-To: <bc657ead1003160658h38b63f93l168a724e13dd9585@mail.gmail.com>
References: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>
	<586034.79517.qm@web33004.mail.mud.yahoo.com>
	<bc657ead1003160658h38b63f93l168a724e13dd9585@mail.gmail.com>
Message-ID: <4B9F904A.6010001@student.matnat.uio.no>

Sebastian Haase wrote:
> On Tue, Mar 16, 2010 at 11:27 AM, David Baddeley
> <david_baddeley at yahoo.com.au> wrote:
>   
>> I think there are a couple of factors here - python for loops and if statements are bad news performance wise, so if you can remove them using cython / c you'll gain a lot. On the other hand they tend to suffer more of a performance penalty when you've got the line profiling hooks in place than, for example numpy calls which are vectorised and doing all the heavy lifting in places which aren't visible to the profiler.
>>
>> You might also be able to improve performance by doing something like:
>>
>> for tracki, track in enumerate(self.tracks[self.tracks_tlast == t-1]):
>>
>> if tracks and tracks_tlast are numpy arrays.
>>     
>
> This did sound like an intriguing idea, but sadly they are all lists,
> because they are appended to as new points are one-by-one being added
> to the tracks.
>
>   
>> Just out of interest, how many points/observations are you trying to track? I've been working on something for stitching together tracks from lists of positions as well and would be interested in comparing strategies.
>>
>>     
>
> I just finished my first analysis of many of our files.  In total I'm
> talking about 300GB of image data, where each "movie" might have 100
> to few thousand image frames. At most my detection algorithms found
> few hundreds (but sometimes up to 1000) points per frame.
> The longest run took about 3 hours for up to 1400 points per frame
> over a total of 2000 frames; but here I might have also run into
> memory / swapping(?) issues. (I have 6GB memory RAM on a quad-code
> 64bit linux machine.)
>
> I would really like to try cython on this innermost loop to get
> everything re-analysed in a fraction of a day.
>   
If using Cython, list lookup in itself can be very fast, but the 
comparisons, any arithmetic you might do etc. will be slow because you 
are using Python objects inside the lists.

Consider copying to over-allocated NumPy arrays (and reallocate on need).

Dag Sverre


From dwf at cs.toronto.edu  Tue Mar 16 11:05:05 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 16 Mar 2010 11:05:05 -0400
Subject: [SciPy-User] Maximum file size for .npz format?
Message-ID: <F92F46DD-BCD5-4DA4-A18A-D49B2A31F201@cs.toronto.edu>

Hey Francesc,

On 2010-03-15, at 4:16 PM, Francesc Alted <faltet at pytables.org> wrote:

> Good!  So, I should start by looking at how it achieves that then :)

Actually I did build a 4-way universal binary of HDF5. Long story  
short, because of some quirks of the build process for HDF5 it isn't  
possible to add some -arch flags to CFLAGS -- you actually have to  
build it separately for each architecture and then do something like

	make install DESTDIR=path/for/arch

and then stitch them together manually with otool. I'm also not sure  
of the correct way, if there is one, to handle "hdf5.settings" -- it's  
unclear to me whether programs ever look at this at build time/ 
runtime, or if it's just there for the user's convenience.

An installer for the binaries I made is at

	http://www.cs.toronto.edu/~dwf/mirror/hdf5-1.8.4-quad.pkg

I can write up some instructions if that would be helpful.

David


From pierre.lamon at bluebotics.com  Tue Mar 16 11:51:25 2010
From: pierre.lamon at bluebotics.com (Pierre Lamon)
Date: Tue, 16 Mar 2010 16:51:25 +0100
Subject: [SciPy-User] Potential KDTree memory leak
In-Reply-To: <dde7764a1003150803h279f6f85y2e0e42dd59726491@mail.gmail.com>
References: <4B98EDDD.6040608@bluebotics.com>	<3d375d731003110750h3a6f8bd6se0a91b9cb98ff7b1@mail.gmail.com>	<4B9DF238.3090700@bluebotics.com>
	<dde7764a1003150803h279f6f85y2e0e42dd59726491@mail.gmail.com>
Message-ID: <4B9FA8FD.6020705@bluebotics.com>

On 2010-03-15 16:03, Ralf Gommers wrote:
>
>
> On Mon, Mar 15, 2010 at 4:39 PM, Pierre Lamon 
> <pierre.lamon at bluebotics.com <mailto:pierre.lamon at bluebotics.com>> wrote:
>
>     On 2010-03-11 16:50, Robert Kern wrote:
>     > With recent SVN versions of numpy and scipy, I do not see a leak.
>     >
>     I've tested with compiled sources of numpy 1.3.0 and scipy 0.7.3 and
>     there is the leak (test done under linux).
>
>
> Scipy 0.7.3, that's impressive:) 0.7.2 RC1 should be coming soon.
Sorry for the typo. It was version 0.7.1 of Scipy
>
>     On the other hand there is no leak with the SVN version of
>     numpy and scipy. I'm glad
>     the bug has been fixed. Any idea when the new release will be
>     available ?
>
>
> The new release of scipy trunk (0.8.0)? Probably you have to be a bit 
> patient there. The plan is ATM: first scipy 0.7.2, then numpy 1.4.1, 
> then numpy 2.0, and only then scipy 0.8.0.
Thanks for giving me the schedule. I'll work with the SVN version meanwhile.

Cheers,
~Pierre
>
> Cheers,
> Ralf
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>    


-- 
Dr. Pierre Lamon
Research&  Development / Scientific Manager
BlueBotics SA
PSE-C, CH-1015 Lausanne, Switzerland
http://www.bluebotics.com
pierre.lamon at bluebotics.com
+41 21 693 83 14 (tel)
+41 21 693 83 15 (fax)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100316/838e6b25/attachment.html>

From faltet at pytables.org  Tue Mar 16 12:29:39 2010
From: faltet at pytables.org (Francesc Alted)
Date: Tue, 16 Mar 2010 17:29:39 +0100
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <F92F46DD-BCD5-4DA4-A18A-D49B2A31F201@cs.toronto.edu>
References: <F92F46DD-BCD5-4DA4-A18A-D49B2A31F201@cs.toronto.edu>
Message-ID: <201003161729.39798.faltet@pytables.org>

A Tuesday 16 March 2010 16:05:05 David Warde-Farley escrigu?:
> Hey Francesc,
> 
> On 2010-03-15, at 4:16 PM, Francesc Alted <faltet at pytables.org> wrote:
> > Good!  So, I should start by looking at how it achieves that then :)
> 
> Actually I did build a 4-way universal binary of HDF5. Long story
> short, because of some quirks of the build process for HDF5 it isn't
> possible to add some -arch flags to CFLAGS -- you actually have to
> build it separately for each architecture and then do something like
> 
> 	make install DESTDIR=path/for/arch
> 
> and then stitch them together manually with otool. I'm also not sure
> of the correct way, if there is one, to handle "hdf5.settings" -- it's
> unclear to me whether programs ever look at this at build time/
> runtime, or if it's just there for the user's convenience.

Yes, I think "hdf5.settings" it is for convenience purposes only (a fast way 
to look for configuration of the compiler flags for the library).  So you 
don't need to worry too much about this.

> An installer for the binaries I made is at
> 
> 	http://www.cs.toronto.edu/~dwf/mirror/hdf5-1.8.4-quad.pkg

Hey, that's great.

> I can write up some instructions if that would be helpful.

Please do.  I'm definitely interested!

Thanks,

-- 
Francesc Alted


From Chris.Barker at noaa.gov  Tue Mar 16 12:49:12 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 16 Mar 2010 09:49:12 -0700
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <F92F46DD-BCD5-4DA4-A18A-D49B2A31F201@cs.toronto.edu>
References: <F92F46DD-BCD5-4DA4-A18A-D49B2A31F201@cs.toronto.edu>
Message-ID: <4B9FB688.3070708@noaa.gov>

David Warde-Farley wrote:
> Actually I did build a 4-way universal binary of HDF5.

> An installer for the binaries I made is at
> 
> 	http://www.cs.toronto.edu/~dwf/mirror/hdf5-1.8.4-quad.pkg
> 
> I can write up some instructions if that would be helpful.

Wonderful! Any chance you've tackled netcdf4 as well?

Thanks,

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From denis-bz-gg at t-online.de  Tue Mar 16 12:46:10 2010
From: denis-bz-gg at t-online.de (denis)
Date: Tue, 16 Mar 2010 09:46:10 -0700 (PDT)
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com> 
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com> 
	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com>
Message-ID: <68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com>

On Mar 15, 7:41?pm, Robert Kern <robert.k... at gmail.com> wrote:
> > Is there a better
> > place
> > that would ease the (planned ?) migration of advice to scipy.org ?
>
> There won't be a migration, sorry. The software is too different.

Too bad; I thought advice. a good idea.
Could you find time to post your ideas for a place for snippets and
micro-tutorials ?
    "... covering the whole spectrum dot by dot,
    so that he always began from a blindingly bright nucleus
    and ended at absolute black, or vice versa"
        -- Foucault's Pendulum

cheers
  -- denis


From rmay31 at gmail.com  Tue Mar 16 13:01:48 2010
From: rmay31 at gmail.com (Ryan May)
Date: Tue, 16 Mar 2010 12:01:48 -0500
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com> 
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com> 
	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com> 
	<68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com>
Message-ID: <c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com>

On Tue, Mar 16, 2010 at 11:46 AM, denis <denis-bz-gg at t-online.de> wrote:
> On Mar 15, 7:41?pm, Robert Kern <robert.k... at gmail.com> wrote:
>> > Is there a better
>> > place
>> > that would ease the (planned ?) migration of advice to scipy.org ?
>>
>> There won't be a migration, sorry. The software is too different.
>
> Too bad; I thought advice. a good idea.
> Could you find time to post your ideas for a place for snippets and
> micro-tutorials ?

What I believe Robert is saying is that the old data on
advice.mechanicalkern.com won't be migrated to the new
advice.scipy.org, when it's created.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From robert.kern at gmail.com  Tue Mar 16 13:41:33 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 16 Mar 2010 12:41:33 -0500
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com> 
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com> 
	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com> 
	<68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com> 
	<c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com>
Message-ID: <3d375d731003161041t2af04784o2aa06801ec02fd1a@mail.gmail.com>

On Tue, Mar 16, 2010 at 12:01, Ryan May <rmay31 at gmail.com> wrote:
> On Tue, Mar 16, 2010 at 11:46 AM, denis <denis-bz-gg at t-online.de> wrote:
>> On Mar 15, 7:41?pm, Robert Kern <robert.k... at gmail.com> wrote:
>>> > Is there a better
>>> > place
>>> > that would ease the (planned ?) migration of advice to scipy.org ?
>>>
>>> There won't be a migration, sorry. The software is too different.
>>
>> Too bad; I thought advice. a good idea.
>> Could you find time to post your ideas for a place for snippets and
>> micro-tutorials ?
>
> What I believe Robert is saying is that the old data on
> advice.mechanicalkern.com won't be migrated to the new
> advice.scipy.org, when it's created.

Correct.

That said, the site is not the place for snippets and micro-tutorials
like those which denis has posted. It is a site for asking and
answering questions. Snippets and micro-tutorials should go on the
wiki:

  http://www.scipy.org/Cookbook

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From cournape at gmail.com  Tue Mar 16 15:21:34 2010
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 16 Mar 2010 14:21:34 -0500
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <F92F46DD-BCD5-4DA4-A18A-D49B2A31F201@cs.toronto.edu>
References: <F92F46DD-BCD5-4DA4-A18A-D49B2A31F201@cs.toronto.edu>
Message-ID: <5b8d13221003161221v493fc0abp7beb700a7a031065@mail.gmail.com>

On Tue, Mar 16, 2010 at 10:05 AM, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> Hey Francesc,
>
> On 2010-03-15, at 4:16 PM, Francesc Alted <faltet at pytables.org> wrote:
>
>> Good! ?So, I should start by looking at how it achieves that then :)
>
> Actually I did build a 4-way universal binary of HDF5. Long story
> short, because of some quirks of the build process for HDF5 it isn't
> possible to add some -arch flags to CFLAGS -- you actually have to
> build it separately for each architecture and then do something like
>
> ? ? ? ?make install DESTDIR=path/for/arch
>
> and then stitch them together manually with otool.

Actually, that's a much better way of doing things generally. I wish
this was the standard way of building universal binaries, it is
usually more robust than using multiple -arch arguments, especially
when you export a public API with C headers. I use this technique
myself for all my packages pure C libraries (audiolab and samplerate
in particular),

cheers,

David


From dwf at cs.toronto.edu  Tue Mar 16 16:56:51 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 16 Mar 2010 16:56:51 -0400
Subject: [SciPy-User] Maximum file size for .npz format?
In-Reply-To: <4B9FB688.3070708@noaa.gov>
References: <F92F46DD-BCD5-4DA4-A18A-D49B2A31F201@cs.toronto.edu>
	<4B9FB688.3070708@noaa.gov>
Message-ID: <20100316205650.GA5492@rodimus>

On Tue, Mar 16, 2010 at 09:49:12AM -0700, Christopher Barker wrote:
> David Warde-Farley wrote:
> > Actually I did build a 4-way universal binary of HDF5.
> 
> > An installer for the binaries I made is at
> > 
> > 	http://www.cs.toronto.edu/~dwf/mirror/hdf5-1.8.4-quad.pkg
> > 
> > I can write up some instructions if that would be helpful.
> 
> Wonderful! Any chance you've tackled netcdf4 as well?

Sorry, I haven't, but it should be roughly the same procedure. I'll write up
some instructions.

One thing with HDF5 was that I believe I had to compile i386 and x86_64
binaries on an Intel Mac and ppc/ppc64 on a PowerPC Mac, there was some issue
with cross-compiling that I never quite got sorted out. It may compile a
binary which it then wants to actually *run* for another step of the build
process. Rosetta should take care of this on Intel but the machine I was
sitting in front of was a G5, and the Intel build on that machine failed 
miserably, so I logged in remotely to an Intel Mac. I will try compiling all
4 on an Intel machine and see if that works.

David


From dwf at cs.toronto.edu  Tue Mar 16 17:04:34 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Tue, 16 Mar 2010 17:04:34 -0400
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com>
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com>
	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com>
	<68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com>
	<c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com>
Message-ID: <20100316210434.GB5492@rodimus>

On Tue, Mar 16, 2010 at 12:01:48PM -0500, Ryan May wrote:
> On Tue, Mar 16, 2010 at 11:46 AM, denis <denis-bz-gg at t-online.de> wrote:
> > On Mar 15, 7:41?pm, Robert Kern <robert.k... at gmail.com> wrote:
> >> > Is there a better
> >> > place
> >> > that would ease the (planned ?) migration of advice to scipy.org ?
> >>
> >> There won't be a migration, sorry. The software is too different.
> >
> > Too bad; I thought advice. a good idea.
> > Could you find time to post your ideas for a place for snippets and
> > micro-tutorials ?
> 
> What I believe Robert is saying is that the old data on
> advice.mechanicalkern.com won't be migrated to the new
> advice.scipy.org, when it's created.

Assuming the old site stays up for a while, I'll pitch in to manually migrate
questions and their selected "approved" answers.

Robert, any idea what markup the new site will use?

David


From gokhansever at gmail.com  Tue Mar 16 17:15:45 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Tue, 16 Mar 2010 16:15:45 -0500
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <20100316210434.GB5492@rodimus>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com>
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com>
	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com>
	<68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com>
	<c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com>
	<20100316210434.GB5492@rodimus>
Message-ID: <49d6b3501003161415t45ba9969lc885598e8b43e6fd@mail.gmail.com>

On Tue, Mar 16, 2010 at 4:04 PM, David Warde-Farley <dwf at cs.toronto.edu>wrote:

> On Tue, Mar 16, 2010 at 12:01:48PM -0500, Ryan May wrote:
> > On Tue, Mar 16, 2010 at 11:46 AM, denis <denis-bz-gg at t-online.de> wrote:
> > > On Mar 15, 7:41 pm, Robert Kern <robert.k... at gmail.com> wrote:
> > >> > Is there a better
> > >> > place
> > >> > that would ease the (planned ?) migration of advice to scipy.org ?
> > >>
> > >> There won't be a migration, sorry. The software is too different.
> > >
> > > Too bad; I thought advice. a good idea.
> > > Could you find time to post your ideas for a place for snippets and
> > > micro-tutorials ?
> >
> > What I believe Robert is saying is that the old data on
> > advice.mechanicalkern.com won't be migrated to the new
> > advice.scipy.org, when it's created.
>
> Assuming the old site stays up for a while, I'll pitch in to manually
> migrate
> questions and their selected "approved" answers.
>
> Robert, any idea what markup the new site will use?
>
> David
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>

ask.scipy.org should be in test-drive :)

I can help with the migration if any help needed. It is very easy to access
the old questions in these interfaces plus tagging makes life much easier.
Highly voted - most answered questions listing the advantages can go on and
on etc...

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100316/7a4270df/attachment.html>

From robert.kern at gmail.com  Tue Mar 16 17:49:43 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 16 Mar 2010 16:49:43 -0500
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <20100316210434.GB5492@rodimus>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com> 
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com> 
	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com> 
	<68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com> 
	<c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com> 
	<20100316210434.GB5492@rodimus>
Message-ID: <3d375d731003161449k291ff2b4xfe089762313d84a7@mail.gmail.com>

On Tue, Mar 16, 2010 at 16:04, David Warde-Farley <dwf at cs.toronto.edu> wrote:
> On Tue, Mar 16, 2010 at 12:01:48PM -0500, Ryan May wrote:
>> On Tue, Mar 16, 2010 at 11:46 AM, denis <denis-bz-gg at t-online.de> wrote:
>> > On Mar 15, 7:41?pm, Robert Kern <robert.k... at gmail.com> wrote:
>> >> > Is there a better
>> >> > place
>> >> > that would ease the (planned ?) migration of advice to scipy.org ?
>> >>
>> >> There won't be a migration, sorry. The software is too different.
>> >
>> > Too bad; I thought advice. a good idea.
>> > Could you find time to post your ideas for a place for snippets and
>> > micro-tutorials ?
>>
>> What I believe Robert is saying is that the old data on
>> advice.mechanicalkern.com won't be migrated to the new
>> advice.scipy.org, when it's created.
>
> Assuming the old site stays up for a while, I'll pitch in to manually migrate
> questions and their selected "approved" answers.

Great!

> Robert, any idea what markup the new site will use?

Github-flavored Markdown minus the github-specific URLs.

http://github.github.com/github-flavored-markdown/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From timmichelsen at gmx-topmail.de  Tue Mar 16 18:17:30 2010
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Tue, 16 Mar 2010 23:17:30 +0100
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <49d6b3501003161415t45ba9969lc885598e8b43e6fd@mail.gmail.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com>	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com>	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com>	<68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com>	<c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com>	<20100316210434.GB5492@rodimus>
	<49d6b3501003161415t45ba9969lc885598e8b43e6fd@mail.gmail.com>
Message-ID: <hnp01r$bej$1@dough.gmane.org>


> ask.scipy.org should be in test-drive :)
The RSS
http://ask.scipy.org/en/newest.atom
 leads to

500 Internal Server Error



From robert.kern at gmail.com  Tue Mar 16 18:25:26 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 16 Mar 2010 17:25:26 -0500
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <hnp01r$bej$1@dough.gmane.org>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com> 
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com> 
	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com> 
	<68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com> 
	<c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com> 
	<20100316210434.GB5492@rodimus>
	<49d6b3501003161415t45ba9969lc885598e8b43e6fd@mail.gmail.com> 
	<hnp01r$bej$1@dough.gmane.org>
Message-ID: <3d375d731003161525sdf320dfh8a61acf4d3203a17@mail.gmail.com>

On Tue, Mar 16, 2010 at 17:17, Tim Michelsen
<timmichelsen at gmx-topmail.de> wrote:
>
>> ask.scipy.org should be in test-drive :)
> The RSS
> http://ask.scipy.org/en/newest.atom
> ?leads to
>
> 500 Internal Server Error

Fixed. Thank you.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From dg.gmane at thesamovar.net  Tue Mar 16 21:31:58 2010
From: dg.gmane at thesamovar.net (Dan Goodman)
Date: Wed, 17 Mar 2010 02:31:58 +0100
Subject: [SciPy-User] distributed optimisation package
Message-ID: <hnpbee$d82$1@dough.gmane.org>

Hi all,

Disclaimer: This isn't strictly about Scipy, but I think the people here 
are probably the most likely to have something interesting to say about 
this, as it's about scientific computing with Python.

Myself and a student in our lab have written a package that we're using 
internally to distribute optimisation problems over several CPUs (or 
GPUs using PyCUDA) over the small cluster of computers in our lab using 
multiprocessing. It has two parts: some code for distributing work over 
several computers and over several CPUs/GPUs on those computers. There's 
no load balancing, we assume that each piece of work is roughly the 
same, however it does handle using shared memory arrays with numpy 
automatically (recently discussed on this list and the numpy one). It 
can use named pipes or IP for communications (we had to use named pipes 
because of the firewall setup in our lab).

The other part of the code is a framework for optimisation routines that 
can be used with this. So far we've implemented a particle swarm 
optimisation algorithm which is ideal for this sort of distributed 
processing because it's very local and so splitting amongst several 
machines is quite nice. We're also thinking about implementing a genetic 
algorithm in the same framework.

Anyway, getting to the point. We've written this largely for our own 
use, but it struck us that this might be of interest more widely. The 
code is fairly simple (a few hundred lines), but there are maybe some 
non-obvious things in there (at least for some people). We're thinking 
about releasing it as an independent package, but there's some work 
involved in that, so we were just looking to see if there was enough 
interest to make it worth the effort really. Also, it's very likely that 
our code is less than perfect, so if anyone would be interested in 
working on it that could be good too.

Thanks for any comments!

Dan



From Debabrata.Midya at services.nsw.gov.au  Tue Mar 16 21:59:34 2010
From: Debabrata.Midya at services.nsw.gov.au (Debabrata Midya)
Date: Wed, 17 Mar 2010 12:59:34 +1100
Subject: [SciPy-User] distributed optimisation package
In-Reply-To: <hnpbee$d82$1@dough.gmane.org>
References: <hnpbee$d82$1@dough.gmane.org>
Message-ID: <4BA0D235.5860.00D0.0@services.nsw.gov.au>

Dan,
 
Thanks for the information.
 
I am interested.
 
Thanks,
 
Deb

>>> Dan Goodman <dg.gmane at thesamovar.net> 17/03/2010 12:31 pm >>>
Hi all,

Disclaimer: This isn't strictly about Scipy, but I think the people here 
are probably the most likely to have something interesting to say about 
this, as it's about scientific computing with Python.

Myself and a student in our lab have written a package that we're using 
internally to distribute optimisation problems over several CPUs (or 
GPUs using PyCUDA) over the small cluster of computers in our lab using 
multiprocessing. It has two parts: some code for distributing work over 
several computers and over several CPUs/GPUs on those computers. There's 
no load balancing, we assume that each piece of work is roughly the 
same, however it does handle using shared memory arrays with numpy 
automatically (recently discussed on this list and the numpy one). It 
can use named pipes or IP for communications (we had to use named pipes 
because of the firewall setup in our lab).

The other part of the code is a framework for optimisation routines that 
can be used with this. So far we've implemented a particle swarm 
optimisation algorithm which is ideal for this sort of distributed 
processing because it's very local and so splitting amongst several 
machines is quite nice. We're also thinking about implementing a genetic 
algorithm in the same framework.

Anyway, getting to the point. We've written this largely for our own 
use, but it struck us that this might be of interest more widely. The 
code is fairly simple (a few hundred lines), but there are maybe some 
non-obvious things in there (at least for some people). We're thinking 
about releasing it as an independent package, but there's some work 
involved in that, so we were just looking to see if there was enough 
interest to make it worth the effort really. Also, it's very likely that 
our code is less than perfect, so if anyone would be interested in 
working on it that could be good too.

Thanks for any comments!

Dan

_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org 
http://mail.scipy.org/mailman/listinfo/scipy-user
*************************************************************************************

This email message, including any attached files, is confidential and
intended solely for the use of the individual or entity to whom it is
addressed. 

The Department of Services, Technology and Administration prohibits the 
right to publish, copy, distribute or disclose any information contained in 
this email, or its attachments, by any party other than the intended 
recipient. If you have received this email in error please notify the sender 
and delete it from your system.

No employee or agent is authorised to conclude any binding agreement 
on behalf of the Department of Services, Technology and Administration 
by email. The views or opinions presented in this email are solely those of 
the author and do not necessarily represent those of the Department, 
except where the sender expressly, and with authority, states them to be 
the views of the Department of Services, Technology and Administration.

The Department of Services, Technology and Administration accepts no 
liability for any loss or damage arising from the use of this email and 
recommends that the recipient check this email and any attached files for 
the presence of viruses.

*************************************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100317/2a2e8dbe/attachment.html>

From paul.anton.letnes at gmail.com  Tue Mar 16 22:02:27 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Tue, 16 Mar 2010 19:02:27 -0700
Subject: [SciPy-User] distributed optimisation package
In-Reply-To: <hnpbee$d82$1@dough.gmane.org>
References: <hnpbee$d82$1@dough.gmane.org>
Message-ID: <965B3321-3072-4303-A182-7A9B5AD3B020@gmail.com>

Hi Dan,

I'm not sure if I can use your algorithm, but I'm definitely curious / interested. (We've got a similar (I guess?) setup where I work.)

-Paul

On 16. mars 2010, at 18.31, Dan Goodman wrote:

> Hi all,
> 
> Disclaimer: This isn't strictly about Scipy, but I think the people here 
> are probably the most likely to have something interesting to say about 
> this, as it's about scientific computing with Python.
> 
> Myself and a student in our lab have written a package that we're using 
> internally to distribute optimisation problems over several CPUs (or 
> GPUs using PyCUDA) over the small cluster of computers in our lab using 
> multiprocessing. It has two parts: some code for distributing work over 
> several computers and over several CPUs/GPUs on those computers. There's 
> no load balancing, we assume that each piece of work is roughly the 
> same, however it does handle using shared memory arrays with numpy 
> automatically (recently discussed on this list and the numpy one). It 
> can use named pipes or IP for communications (we had to use named pipes 
> because of the firewall setup in our lab).
> 
> The other part of the code is a framework for optimisation routines that 
> can be used with this. So far we've implemented a particle swarm 
> optimisation algorithm which is ideal for this sort of distributed 
> processing because it's very local and so splitting amongst several 
> machines is quite nice. We're also thinking about implementing a genetic 
> algorithm in the same framework.
> 
> Anyway, getting to the point. We've written this largely for our own 
> use, but it struck us that this might be of interest more widely. The 
> code is fairly simple (a few hundred lines), but there are maybe some 
> non-obvious things in there (at least for some people). We're thinking 
> about releasing it as an independent package, but there's some work 
> involved in that, so we were just looking to see if there was enough 
> interest to make it worth the effort really. Also, it's very likely that 
> our code is less than perfect, so if anyone would be interested in 
> working on it that could be good too.
> 
> Thanks for any comments!
> 
> Dan
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From pgmdevlist at gmail.com  Wed Mar 17 02:14:33 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 17 Mar 2010 02:14:33 -0400
Subject: [SciPy-User] usemask kwd in timeseries.tsfromtxt missing?
In-Reply-To: <hnmc5k$gm1$1@dough.gmane.org>
References: <c048da1c1003121359k2984c9e3k2dcc100a7f2e3500@mail.gmail.com>	<hnfju4$8d1$1@dough.gmane.org>	<c048da1c1003130843g59f5aebbu1fd576f5f1c3d2c@mail.gmail.com>	<hnjo5a$st$1@dough.gmane.org>
	<4A98946E-5539-4274-A1B4-778D6D672BE7@gmail.com>
	<hnmc5k$gm1$1@dough.gmane.org>
Message-ID: <526556CF-70DE-45E2-9537-C575081D0CF2@gmail.com>

On Mar 15, 2010, at 6:25 PM, Tim Michelsen wrote:
>> which is a subclass of MaskedArray. Therefore, `usemask` would always
>> be True. No need for a keyword argument that you can't modify, hence
>> the error message.
> Then, why is this argument not removed?

I don't follow you. The `usemask` argument is not a valid argument for `tsfromtxt` (check the function signature), that's why it's not recognized. There's nothing to remove.

> Because of the inheritance from genfromtxt?

In practice, genfromtxt has more arguments than tsfromtxt.

> 
> Maybe it's worth mentioning this in the docstring.


Ah OK. The docstring is wrong. I'll correct that 

From denis-bz-gg at t-online.de  Wed Mar 17 07:11:38 2010
From: denis-bz-gg at t-online.de (denis)
Date: Wed, 17 Mar 2010 04:11:38 -0700 (PDT)
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <3d375d731003161525sdf320dfh8a61acf4d3203a17@mail.gmail.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<3d375d731003151056k766ff972v6e1376ea90f0cc5e@mail.gmail.com> 
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com> 
	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com> 
	<68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com> 
	<c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com> 
	<20100316210434.GB5492@rodimus>
	<49d6b3501003161415t45ba9969lc885598e8b43e6fd@mail.gmail.com> 
	<hnp01r$bej$1@dough.gmane.org>
	<3d375d731003161525sdf320dfh8a61acf4d3203a17@mail.gmail.com>
Message-ID: <357dbf27-e72a-4f9c-9448-0056899ff36b@k17g2000yqb.googlegroups.com>

Sounds like a plan !
Advice.mechanicalkern has two aspects I really like,
that would be useful for ask.scipy / Cookbook too:

* realtime rendering, to edit and view locally, then post
    (what's the renderer in advice / SO ?)
* anyone can add comments / links (after login)

It would also be useful to collect Q+A examples in heaps
such as "indexing", "vectorizing", "plotting";
one-stop wikis on these would sure beat googling through mails
and would I think help learners and experts alike.  What say ?

cheers
  -- denis


From robert.kern at gmail.com  Wed Mar 17 10:44:31 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 17 Mar 2010 09:44:31 -0500
Subject: [SciPy-User] where to put plots in advice.mechanicalkern ?
In-Reply-To: <357dbf27-e72a-4f9c-9448-0056899ff36b@k17g2000yqb.googlegroups.com>
References: <69b4e3f8-a4f8-4ca9-a90c-45c0907fd19d@19g2000yqu.googlegroups.com>
	<dfda91c4-18c6-4998-9715-78ba703f4903@15g2000yqi.googlegroups.com> 
	<3d375d731003151141x3102b26dy774967f563dd622a@mail.gmail.com> 
	<68cc6b94-b6f9-4aa7-8272-3bc68b4badd4@g11g2000yqe.googlegroups.com> 
	<c7b95a021003161001id71bdb2s2128091c12d16123@mail.gmail.com> 
	<20100316210434.GB5492@rodimus>
	<49d6b3501003161415t45ba9969lc885598e8b43e6fd@mail.gmail.com> 
	<hnp01r$bej$1@dough.gmane.org>
	<3d375d731003161525sdf320dfh8a61acf4d3203a17@mail.gmail.com> 
	<357dbf27-e72a-4f9c-9448-0056899ff36b@k17g2000yqb.googlegroups.com>
Message-ID: <3d375d731003170744j73c66fc9r804f8db43b26306d@mail.gmail.com>

On Wed, Mar 17, 2010 at 06:11, denis <denis-bz-gg at t-online.de> wrote:
> Sounds like a plan !
> Advice.mechanicalkern has two aspects I really like,
> that would be useful for ask.scipy / Cookbook too:
>
> * realtime rendering, to edit and view locally, then post
> ? ?(what's the renderer in advice / SO ?)

ask.scipy.org uses Markdown, like advice, and has realtime preview.

> * anyone can add comments / links (after login)

ask.scipy.org has this.

> It would also be useful to collect Q+A examples in heaps
> such as "indexing", "vectorizing", "plotting";
> one-stop wikis on these would sure beat googling through mails
> and would I think help learners and experts alike. ?What say ?

This is what tags are for.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From SGONG at mdacorporation.com  Wed Mar 17 11:59:33 2010
From: SGONG at mdacorporation.com (Shawn GONG)
Date: Wed, 17 Mar 2010 08:59:33 -0700
Subject: [SciPy-User] 2D interpolation question
Message-ID: <5F76F80A436C1146A7C04C6F844026FB06104712@VMXYVR2.ds.mda.ca>

Hi list,

I'd like to do a 2D interpolation but not sure to use scipy.ndimage.map_coordinates or scipy.interpolate   Your help is greatly appreciated.

I have a large evenly spaced 2D array (say 8000x8000). I need to interpolate over a smaller-but-finer grid (say 500x500). The grid is an evenly spaced rectangular, but it is rotated (i.e., with an angle).

Which scipy function should I call?

Thanks,
Shawn

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100317/a7f66297/attachment.html>

From zachary.pincus at yale.edu  Wed Mar 17 12:12:01 2010
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Wed, 17 Mar 2010 12:12:01 -0400
Subject: [SciPy-User] 2D interpolation question
In-Reply-To: <5F76F80A436C1146A7C04C6F844026FB06104712@VMXYVR2.ds.mda.ca>
References: <5F76F80A436C1146A7C04C6F844026FB06104712@VMXYVR2.ds.mda.ca>
Message-ID: <C7AFD12C-0D2D-4DE0-AF98-D4C7A59B5770@yale.edu>

> I'd like to do a 2D interpolation but not sure to use  
> scipy.ndimage.map_coordinates or scipy.interpolate   Your help is  
> greatly appreciated.
>
> I have a large evenly spaced 2D array (say 8000x8000). I need to  
> interpolate over a smaller-but-finer grid (say 500x500). The grid is  
> an evenly spaced rectangular, but it is rotated (i.e., with an angle).
>
> Which scipy function should I call?
>
map_coordinates is likely going to be the best and simplest for tasks  
like this that are basically "image resampling".

Basically, for each point in your output array, calculate the x,y  
position in the input array (as floating point), and put it into the  
input format described by the map_coordinates documentation (a  
2x500x500 array, IIRC). Then choose the order of spline interpolation:  
sound choices include 0 (nearest neighbor), 1 (linear), and 3 or 5  
(higher order). The higher-order splines will give "nicer looking"  
results on smooth patches, but are susceptible to ringing artifacts  
near sharp edges in the input array.

The rest should be self-explanatory, but if not, ask away.

Zach

PS. if you're doing a lot of resampling of the same input array, and  
speed for these operations is an absolute priority, and linear  
interpolation is acceptable, you could also consider do this on the  
graphics card with opengl. (I like pyglet for these tasks, but I bet  
pyCUDA would work as well?) You'd just have to slice the input texture  
a bit so as to not be over 4096x4096. 


From SGONG at mdacorporation.com  Wed Mar 17 12:18:12 2010
From: SGONG at mdacorporation.com (Shawn GONG)
Date: Wed, 17 Mar 2010 09:18:12 -0700
Subject: [SciPy-User] 2D interpolation question
References: <5F76F80A436C1146A7C04C6F844026FB06104712@VMXYVR2.ds.mda.ca>
	<C7AFD12C-0D2D-4DE0-AF98-D4C7A59B5770@yale.edu>
Message-ID: <5F76F80A436C1146A7C04C6F844026FB06104713@VMXYVR2.ds.mda.ca>

Thanks Zach for your quick reply.
I'll try map_coordinates then.

Shawn



-----Original Message-----
From: scipy-user-bounces at scipy.org on behalf of Zachary Pincus
Sent: Wed 3/17/2010 9:12 AM
To: SciPy Users List
Subject: Re: [SciPy-User] 2D interpolation question
 
> I'd like to do a 2D interpolation but not sure to use  
> scipy.ndimage.map_coordinates or scipy.interpolate   Your help is  
> greatly appreciated.
>
> I have a large evenly spaced 2D array (say 8000x8000). I need to  
> interpolate over a smaller-but-finer grid (say 500x500). The grid is  
> an evenly spaced rectangular, but it is rotated (i.e., with an angle).
>
> Which scipy function should I call?
>
map_coordinates is likely going to be the best and simplest for tasks  
like this that are basically "image resampling".

Basically, for each point in your output array, calculate the x,y  
position in the input array (as floating point), and put it into the  
input format described by the map_coordinates documentation (a  
2x500x500 array, IIRC). Then choose the order of spline interpolation:  
sound choices include 0 (nearest neighbor), 1 (linear), and 3 or 5  
(higher order). The higher-order splines will give "nicer looking"  
results on smooth patches, but are susceptible to ringing artifacts  
near sharp edges in the input array.

The rest should be self-explanatory, but if not, ask away.

Zach

PS. if you're doing a lot of resampling of the same input array, and  
speed for these operations is an absolute priority, and linear  
interpolation is acceptable, you could also consider do this on the  
graphics card with opengl. (I like pyglet for these tasks, but I bet  
pyCUDA would work as well?) You'd just have to slice the input texture  
a bit so as to not be over 4096x4096. 
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user


-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 3683 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100317/c0f1022c/attachment.bin>

From chris.leemesser at gmail.com  Wed Mar 17 12:23:14 2010
From: chris.leemesser at gmail.com (Christopher Lee-Messer)
Date: Wed, 17 Mar 2010 09:23:14 -0700
Subject: [SciPy-User] numpy shared memory api/tests
Message-ID: <4cccdc8b1003170923j55ed2ab4t68a98e9c8b38fec@mail.gmail.com>

I've added David B.'s shmarray and added some tests and packaging to my repo.
It seems useful to have both a sysv-style shared memory and David's
implementation which is mmap based.  I've tested that both work on
winxp, mac OS 10.6 32bit and linux kernel 2.6.x 32bit.

However, what should the api be?

If anyone wants to look at what the tests and make suggestions, I will
try to add them as I get time.

The current tests try creating small shared arrays using a
non-exhaustive list of dtypes.  shmarray.py  fails when trying to
pickle a c_double_Array_4 because multiprocessing.sharedctypes doesn't
support pickling it.  I would guess that that support would be
straightforward to add.

-chris

http://cleemesser at bitbucket.org/cleemesser/numpy-sharedmem/



On Sat, Mar 13, 2010 at 10:54 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sat, Mar 13, 2010 at 08:45:08AM -0800, Christopher Lee-Messer wrote:
>> http://bitbucket.org/cleemesser/numpy-sharedmem/
>
>> I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
>> to work on these, but I plan to add testing and save results on
>> different platforms as my app gets used in different computers in the
>> lab.
>
> I am not. I have been wanting to propose this for integration in numpy
> for a long while, but I really haven't found the time, and I will not any
> time soon.
>
> Before any patch can be written to push in numpy, it does need tests
> and packaging, so what you are proposing to do is really great and very
> helpful. Also, using it on many platform would help ironing out details.
>
> Thanks for stepping up,
>
> Ga?l
>


From david_baddeley at yahoo.com.au  Wed Mar 17 16:51:14 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Wed, 17 Mar 2010 13:51:14 -0700 (PDT)
Subject: [SciPy-User] numpy shared memory api/tests
In-Reply-To: <4cccdc8b1003170923j55ed2ab4t68a98e9c8b38fec@mail.gmail.com>
References: <4cccdc8b1003170923j55ed2ab4t68a98e9c8b38fec@mail.gmail.com>
Message-ID: <205202.82515.qm@web33003.mail.mud.yahoo.com>

Hi Chris,

the shared ctypes arrays in multiprocessing get pickled by a handler attached to multiprocessing.ForkingPickler. This means that they (and consequently also shmarrays) will only pickle properly when pickled through ForkingPickler (ie when passed as arguments to a new process, or e.g. as init arguments to a Pool). It makes shmarray multiprocessing specific, and also means that you can't send them through multiprocessing queues or as arguments to e.g. Pool.map. 

I suspect that this behaviour of sharedcytpes arrays is a design feature, as, due to the fact that multiprocessing queues etc... also support distribution over a network, the only time that you can be sure that the recipient will have access to the shared memory is when you're forking or spawning the recipients on the same machine.

cheers,
David


----- Original Message ----
From: Christopher Lee-Messer <chris.leemesser at gmail.com>
To: Gael Varoquaux <gael.varoquaux at normalesup.org>
Cc: SciPy Users List <scipy-user at scipy.org>
Sent: Thu, 18 March, 2010 5:23:14 AM
Subject: [SciPy-User] numpy shared memory api/tests

I've added David B.'s shmarray and added some tests and packaging to my repo.
It seems useful to have both a sysv-style shared memory and David's
implementation which is mmap based.  I've tested that both work on
winxp, mac OS 10.6 32bit and linux kernel 2.6.x 32bit.

However, what should the api be?

If anyone wants to look at what the tests and make suggestions, I will
try to add them as I get time.

The current tests try creating small shared arrays using a
non-exhaustive list of dtypes.  shmarray.py  fails when trying to
pickle a c_double_Array_4 because multiprocessing.sharedctypes doesn't
support pickling it.  I would guess that that support would be
straightforward to add.

-chris

http://cleemesser at bitbucket.org/cleemesser/numpy-sharedmem/



On Sat, Mar 13, 2010 at 10:54 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sat, Mar 13, 2010 at 08:45:08AM -0800, Christopher Lee-Messer wrote:
>> http://bitbucket.org/cleemesser/numpy-sharedmem/
>
>> I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
>> to work on these, but I plan to add testing and save results on
>> different platforms as my app gets used in different computers in the
>> lab.
>
> I am not. I have been wanting to propose this for integration in numpy
> for a long while, but I really haven't found the time, and I will not any
> time soon.
>
> Before any patch can be written to push in numpy, it does need tests
> and packaging, so what you are proposing to do is really great and very
> helpful. Also, using it on many platform would help ironing out details.
>
> Thanks for stepping up,
>
> Ga?l
>
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From david_baddeley at yahoo.com.au  Wed Mar 17 16:51:14 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Wed, 17 Mar 2010 13:51:14 -0700 (PDT)
Subject: [SciPy-User] numpy shared memory api/tests
In-Reply-To: <4cccdc8b1003170923j55ed2ab4t68a98e9c8b38fec@mail.gmail.com>
References: <4cccdc8b1003170923j55ed2ab4t68a98e9c8b38fec@mail.gmail.com>
Message-ID: <205202.82515.qm@web33003.mail.mud.yahoo.com>

Hi Chris,

the shared ctypes arrays in multiprocessing get pickled by a handler attached to multiprocessing.ForkingPickler. This means that they (and consequently also shmarrays) will only pickle properly when pickled through ForkingPickler (ie when passed as arguments to a new process, or e.g. as init arguments to a Pool). It makes shmarray multiprocessing specific, and also means that you can't send them through multiprocessing queues or as arguments to e.g. Pool.map. 

I suspect that this behaviour of sharedcytpes arrays is a design feature, as, due to the fact that multiprocessing queues etc... also support distribution over a network, the only time that you can be sure that the recipient will have access to the shared memory is when you're forking or spawning the recipients on the same machine.

cheers,
David


----- Original Message ----
From: Christopher Lee-Messer <chris.leemesser at gmail.com>
To: Gael Varoquaux <gael.varoquaux at normalesup.org>
Cc: SciPy Users List <scipy-user at scipy.org>
Sent: Thu, 18 March, 2010 5:23:14 AM
Subject: [SciPy-User] numpy shared memory api/tests

I've added David B.'s shmarray and added some tests and packaging to my repo.
It seems useful to have both a sysv-style shared memory and David's
implementation which is mmap based.  I've tested that both work on
winxp, mac OS 10.6 32bit and linux kernel 2.6.x 32bit.

However, what should the api be?

If anyone wants to look at what the tests and make suggestions, I will
try to add them as I get time.

The current tests try creating small shared arrays using a
non-exhaustive list of dtypes.  shmarray.py  fails when trying to
pickle a c_double_Array_4 because multiprocessing.sharedctypes doesn't
support pickling it.  I would guess that that support would be
straightforward to add.

-chris

http://cleemesser at bitbucket.org/cleemesser/numpy-sharedmem/



On Sat, Mar 13, 2010 at 10:54 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sat, Mar 13, 2010 at 08:45:08AM -0800, Christopher Lee-Messer wrote:
>> http://bitbucket.org/cleemesser/numpy-sharedmem/
>
>> I don't know if Sturla, Gael Varoquaux, or Robert Kern are continuing
>> to work on these, but I plan to add testing and save results on
>> different platforms as my app gets used in different computers in the
>> lab.
>
> I am not. I have been wanting to propose this for integration in numpy
> for a long while, but I really haven't found the time, and I will not any
> time soon.
>
> Before any patch can be written to push in numpy, it does need tests
> and packaging, so what you are proposing to do is really great and very
> helpful. Also, using it on many platform would help ironing out details.
>
> Thanks for stepping up,
>
> Ga?l
>
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



      


From david_baddeley at yahoo.com.au  Thu Mar 18 05:05:01 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Thu, 18 Mar 2010 02:05:01 -0700 (PDT)
Subject: [SciPy-User] line_profiler and for-loops !?
In-Reply-To: <bc657ead1003160658h38b63f93l168a724e13dd9585@mail.gmail.com>
References: <bc657ead1003160140r313d7be4t7927fcfbbee11d63@mail.gmail.com>
	<586034.79517.qm@web33004.mail.mud.yahoo.com>
	<bc657ead1003160658h38b63f93l168a724e13dd9585@mail.gmail.com>
Message-ID: <456652.67832.qm@web33003.mail.mud.yahoo.com>

Your post gave got me motivated to dust off my tracking code (was going to need it anyway in the next month or so) and do a bit of optimisation. Went from about 2 mins to stitch 80k points into tracks to around 5 secs for the best python implementation, which got knocked down to ~.02 secs when I coded it up in c as a module. The main trick was to code the tracks by having a preallocated array the same length as the number of points which got filled with the index of the track each point belonged to (kinda like what you'd get from scipy.ndimage.label) rather than using lists. 

Note that this is all starting from predetermined arrays of positions, and doesn't include object detection and fitting which I do with another program. My problem's slightly different in that I've got more like 20k frames but much fewer (~2-50) points per frame in each image series - a single image series would be ~10GB uncompressed, but I never have all the raw data in memory at one time. My tracks are probably also much shorter - with most only being one or two frames. 

I could give you a copy of the tracking code if you like - may be useful on it's own or potentially just to get ideas,

cheers,
David

----- Original Message ----
From: Sebastian Haase <seb.haase at gmail.com>
To: david_baddeley at yahoo.com.au; SciPy Users List <scipy-user at scipy.org>
Sent: Wed, 17 March, 2010 2:58:57 AM
Subject: Re: [SciPy-User] line_profiler and for-loops !?

On Tue, Mar 16, 2010 at 11:27 AM, David Baddeleyly 
<david_baddeley at yahoo.com.au> wrote:
> I think there are a couple of factors here - python for loops and if statements are bad news performance wise, so if you can remove them using cython / c you'll gain a lot. On the other hand they tend to suffer more of a performance penalty when you've got the line profiling hooks in place than, for example numpy calls which are vectorised and doing all the heavy lifting in places which aren't visible to the profiler.
>
> You might also be able to improve performance by doing something like:
>
> for tracki, track in enumerate(self.tracks[self.tracks_tlast == t-1]):
>
> if tracks and tracks_tlast are numpy arrays.

This did sound like an intriguing idea, but sadly they are all lists,
because they are appended to as new points are one-by-one being added
to the tracks.

>
> Just out of interest, how many points/observations are you trying to track? I've been working on something for stitching together tracks from lists of positions as well and would be interested in comparing strategies.
>

I just finished my first analysis of many of our files.  In total I'm
talking about 300GB of image data, where each "movie" might have 100
to few thousand image frames. At most my detection algorithms found
few hundreds (but sometimes up to 1000) points per frame.
The longest run took about 3 hours for up to 1400 points per frame
over a total of 2000 frames; but here I might have also run into
memory / swapping(?) issues. (I have 6GB memory RAM on a quad-code
64bit linux machine.)

I would really like to try cython on this innermost loop to get
everything re-analysed in a fraction of a day.

-Sebastian


> David
>
>
> --- On Tue, 16/3/10, Sebastian Haase <seb.haase at gmail.com> wrote:
>
>> From: Sebastian Haase <seb.haase at gmail.com>
>> Subject: [SciPy-User] line_profiler and for-loops !?
>> To: "SciPy Users List" <SciPy-user at scipy.org>
>> Received: Tuesday, 16 March, 2010, 9:40 PM
>> Hi,
>>
>> I was starting to use Robert's line_profiler. I seems to
>> work great,
>> and I already found one easy way do half my execution
>> time.
>> But now it claims that 33% of the time is spent (directly)
>> in the
>> "for"-line and another 36% in a very simple "if"-line. See
>> parts of
>> the output here:
>> <snip>
>> Function: doTracing at line 1135
>> Total time: 23.9171 s
>>
>> Line #      Hits
>>    Time  Per Hit   %
>> Time  Line Contents
>> ==============================================================
>> <snip>
>>   1185
>>
>>
>>          # iterate
>> over all tracks, and find close points
>>   1186   3853362
>> 8024186      2.1
>>    33.5
>>     for
>> tracki,track in enumerate(self.tracks):
>>   1187   3853063
>> 8639273      2.2
>>    36.1
>>         if
>> self.tracks_tLast[tracki] == t-1:
>>   1188
>>
>>   # track went on until t-1 (so far)
>>   1189
>>
>>   #    --  otherwise, skip "old" tracks
>>   1190     62150
>>    130277      2.1
>>     0.5
>>    pi_t_1 = track[-1] # index in last time
>> section
>> <snip>
>>
>> (The object of the function is to connect closest points
>> found in an
>> image sequence into tracks connecting the points by
>> shortest steps.)
>>
>> Anyhow, my question is, is this just an artifact of
>> line_profiler, or
>> is the fact that those two lines are hit almost 4e6 times
>> really
>> resulting in more than 50% of the time being spent here !?
>> (Calculating the actual Euclidean distance matrix over all
>> point pairs
>> takes supposedly only 15% of the time, for comparison).
>> I tried to separate out the "enumerate(self.tracks)" into a
>> separate
>> line before the "for"-line, but the time spent was still
>> unchanged on
>> the "for".
>> Does this mean "python is slow" here - and I should try
>> cython (which
>> i have never done so far ...) ?
>>
>> Thanks,
>> Sebastian Haase
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



      


From gokhansever at gmail.com  Thu Mar 18 13:20:52 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Thu, 18 Mar 2010 12:20:52 -0500
Subject: [SciPy-User] ask.scipy.org is online
Message-ID: <49d6b3501003181020x23f3236bp23f4664ed8d2aba2@mail.gmail.com>

Hello,

Please tolerate my impatience for being the first announcing the new
discussion platform :) and my cross-posting over the lists.

The new site is beating at ask.scipy.org . David Warde-Farley is moving the
questions from the old-site at advice.mechanicalkern.com (announced at
SciPy09 by Robert Kern)

We welcome you to join the discussions there. I have kicked-off the new
questions chain by
http://ask.scipy.org/en/topic/11-what-is-your-favorite-numpy-feature
(Also cross-posted at
http://stackoverflow.com/questions/2471872/what-is-your-favorite-numpy-featureto
pull more attention to ask.scipy)

Please share your questions and comments, and have fun with your
discussions.

-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100318/534edc8f/attachment.html>

From nicolas.pinto at gmail.com  Thu Mar 18 23:13:27 2010
From: nicolas.pinto at gmail.com (Nicolas Pinto)
Date: Thu, 18 Mar 2010 23:13:27 -0400
Subject: [SciPy-User] Corrupted data when using io.loadmat with format='5'
	and an array of strings.
Message-ID: <954ae5aa1003182013y3a1bf04aw2c2088bc34ef5b2@mail.gmail.com>

Hello,

Using scipy 0.7.1 I'm getting "corrupted" data when using MATLAB v5
format but not with v4:

In [12]: from scipy import io

In [13]: d = {'str':['test','test2','test34']}

In [14]: io.savemat('bug', d, format='4'); io.loadmat('bug')
Out[14]:
{'__globals__': [],
 'str': array([u'test', u'test2', u'test34'],
      dtype='<U6')}

In [15]: io.savemat('bug', d, format='5'); io.loadmat('bug')
Out[15]:
{'__globals__': [],
 '__header__': 'MATLAB 5.0 MAT-file Platform: posix, Created on: Thu
Jan 14 14:15:11 2010',
 '__version__': '1.0',
 'str': array([u'tttttt', u'e\x00e2e3', u's\x00s\x00s4'],
      dtype='<U6')}


Am I doing something wrong or is this a bug? If so, how can I help fix it ?

Thanks in advance.

Regards,

-- 
Nicolas Pinto
Ph.D. Candidate, Brain & Computer Sciences
Massachusetts Institute of Technology, USA
http://web.mit.edu/pinto


From akoumjian at gmail.com  Fri Mar 19 23:21:05 2010
From: akoumjian at gmail.com (Alec Koumjian)
Date: Fri, 19 Mar 2010 23:21:05 -0400
Subject: [SciPy-User] ask.scipy.org
Message-ID: <61a4c0ba1003192021y6331a6fdv6ef93d9c0ffebbf8@mail.gmail.com>

I'm very much looking forward to using the site. However, I can't seem
to get the sign-up page to work with a Google login. I click the
"Google" icon to use my crudentials, but it simply redirects me back
to the OpenID page.  Is this feature not available yet?


From nico.schloemer at gmail.com  Sat Mar 20 07:51:44 2010
From: nico.schloemer at gmail.com (=?ISO-8859-1?Q?Nico_Schl=F6mer?=)
Date: Sat, 20 Mar 2010 12:51:44 +0100
Subject: [SciPy-User] 2d convolution
Message-ID: <d933725a1003200451u1996c66escb68316201adc266@mail.gmail.com>

Hi,

I'm trying to compute the the convolution if s 2D array, and I see
that there are several ways in SciPy to do that. As the original data
C and the kernel R are about the same size in my case, I'd profit from
an FFT-based implementation, which I see right now is given by

    scipy.signal.fftconvolve( C, R,
                                       mode='same' )

and also

    scipy.stsci.convolve.convolve2d( data=C, kernel=R,
                                                  mode='constant', cval=0.0,
                                                  fft=1 )

The second method is much faster than the first, and as far as I can
see it would spit out the the same results.

Now, when the kernel is actually larger than the data, the resulting
array would have the shape of the kernel. Is there any way to restrict
the computations to the size of the data? At first, I thought that was
what "mode='same'"" is for.
I tried cutting the extra data off of the resulting array, but I'm not
quite sure which is the part that I would like to get rid of.

Any hints here?

Cheers,
Nico


From josef.pktd at gmail.com  Sat Mar 20 08:49:24 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 20 Mar 2010 08:49:24 -0400
Subject: [SciPy-User] 2d convolution
In-Reply-To: <d933725a1003200451u1996c66escb68316201adc266@mail.gmail.com>
References: <d933725a1003200451u1996c66escb68316201adc266@mail.gmail.com>
Message-ID: <1cd32cbb1003200549m75a86fbelea959ad6696dc526@mail.gmail.com>

On Sat, Mar 20, 2010 at 7:51 AM, Nico Schl?mer <nico.schloemer at gmail.com> wrote:
> Hi,
>
> I'm trying to compute the the convolution if s 2D array, and I see
> that there are several ways in SciPy to do that. As the original data
> C and the kernel R are about the same size in my case, I'd profit from
> an FFT-based implementation, which I see right now is given by
>
> ? ?scipy.signal.fftconvolve( C, R,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? mode='same' )
>
> and also
>
> ? ?scipy.stsci.convolve.convolve2d( data=C, kernel=R,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?mode='constant', cval=0.0,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?fft=1 )
>
> The second method is much faster than the first, and as far as I can
> see it would spit out the the same results.
>
> Now, when the kernel is actually larger than the data, the resulting
> array would have the shape of the kernel. Is there any way to restrict
> the computations to the size of the data? At first, I thought that was
> what "mode='same'"" is for.
> I tried cutting the extra data off of the resulting array, but I'm not
> quite sure which is the part that I would like to get rid of.
>
> Any hints here?

It's always good to take a look at the source

fftconvolve

    elif mode == "same":
        if product(s1,axis=0) > product(s2,axis=0):
            osize = s1
        else:
            osize = s2
        return _centered(ret,osize)
    elif mode == "valid":
        return _centered(ret,abs(s2-s1)+1)

from scipy.signal.signaltools import _centered

and it should be possible to  set osize to s1 = array(in1.shape)   (C
in your notation)

scipy.stsci has been removed recently from scipy and will not be in
the scipy 0.8 version

(I didn't try myself whether it works)

Josef

>
> Cheers,
> Nico
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From vincent at vincentdavis.net  Sat Mar 20 11:37:21 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 20 Mar 2010 09:37:21 -0600
Subject: [SciPy-User] reversing or avoiding the need to reverse argsort()
Message-ID: <77e831101003200837j5e93d418m86f44856551b4d58@mail.gmail.com>

What I want to do is simple but not sure the best way to go about it.

I have an array a, shape(a) = rXc I need to sort each column
aargsort = a.argsort(axis=0)  # May use this later
aSort = a.sort(axis=0)

now average each row
aSortRM = asort.mean(axis=1)

now replace each col in a row with the row mean.
is there a better way than this

aWithMeans = ones_like(a)
for ind in range(r)  # r = number of rows
    aWithMeans[ind]* aSortRM[ind]

Now I need to undo the sort I did in the first step.
I think this is where I need the aargsort (argsort of a before any
modification) I am kinda stumped on this step. Everything I think of is
rather convoluted.
Is it possible to do this whole process without actually rearranging the
original array and just using the index info from the argsort() and would I
want to do this?



  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100320/4dc29e14/attachment.html>

From nico.schloemer at gmail.com  Sat Mar 20 12:07:00 2010
From: nico.schloemer at gmail.com (=?ISO-8859-1?Q?Nico_Schl=F6mer?=)
Date: Sat, 20 Mar 2010 17:07:00 +0100
Subject: [SciPy-User] 2d convolution
In-Reply-To: <1cd32cbb1003200549m75a86fbelea959ad6696dc526@mail.gmail.com>
References: <d933725a1003200451u1996c66escb68316201adc266@mail.gmail.com>
	<1cd32cbb1003200549m75a86fbelea959ad6696dc526@mail.gmail.com>
Message-ID: <d933725a1003200907u167a7b40p8cbbca12686ce32b@mail.gmail.com>

Thanks for the hint!

I've now set "mode='full'" and extract the data myself by

            CR = scipy.signal.fftconvolve( C, R, mode='full' )
            from scipy.signal.signaltools import _centered
            CR = _centered( CR , C.shape )

Cheers,
Nico


On Sat, Mar 20, 2010 at 1:49 PM,  <josef.pktd at gmail.com> wrote:
> On Sat, Mar 20, 2010 at 7:51 AM, Nico Schl?mer <nico.schloemer at gmail.com> wrote:
>> Hi,
>>
>> I'm trying to compute the the convolution if s 2D array, and I see
>> that there are several ways in SciPy to do that. As the original data
>> C and the kernel R are about the same size in my case, I'd profit from
>> an FFT-based implementation, which I see right now is given by
>>
>> ? ?scipy.signal.fftconvolve( C, R,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? mode='same' )
>>
>> and also
>>
>> ? ?scipy.stsci.convolve.convolve2d( data=C, kernel=R,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?mode='constant', cval=0.0,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?fft=1 )
>>
>> The second method is much faster than the first, and as far as I can
>> see it would spit out the the same results.
>>
>> Now, when the kernel is actually larger than the data, the resulting
>> array would have the shape of the kernel. Is there any way to restrict
>> the computations to the size of the data? At first, I thought that was
>> what "mode='same'"" is for.
>> I tried cutting the extra data off of the resulting array, but I'm not
>> quite sure which is the part that I would like to get rid of.
>>
>> Any hints here?
>
> It's always good to take a look at the source
>
> fftconvolve
>
> ? ?elif mode == "same":
> ? ? ? ?if product(s1,axis=0) > product(s2,axis=0):
> ? ? ? ? ? ?osize = s1
> ? ? ? ?else:
> ? ? ? ? ? ?osize = s2
> ? ? ? ?return _centered(ret,osize)
> ? ?elif mode == "valid":
> ? ? ? ?return _centered(ret,abs(s2-s1)+1)
>
> from scipy.signal.signaltools import _centered
>
> and it should be possible to ?set osize to s1 = array(in1.shape) ? (C
> in your notation)
>
> scipy.stsci has been removed recently from scipy and will not be in
> the scipy 0.8 version
>
> (I didn't try myself whether it works)
>
> Josef
>
>>
>> Cheers,
>> Nico
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From peridot.faceted at gmail.com  Sat Mar 20 12:09:53 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 20 Mar 2010 12:09:53 -0400
Subject: [SciPy-User] reversing or avoiding the need to reverse argsort()
In-Reply-To: <77e831101003200837j5e93d418m86f44856551b4d58@mail.gmail.com>
References: <77e831101003200837j5e93d418m86f44856551b4d58@mail.gmail.com>
Message-ID: <ce557a361003200909n63674a0evf0853ac034a2094a@mail.gmail.com>

On 20 March 2010 11:37, Vincent Davis <vincent at vincentdavis.net> wrote:

> What I want to do is simple but not sure the best way to go about it.
>
> I have an array a, shape(a) = rXc I need to sort each column
> aargsort = a.argsort(axis=0)  # May use this later
> aSort = a.sort(axis=0)
>
> now average each row
> aSortRM = asort.mean(axis=1)
>
> now replace each col in a row with the row mean.
> is there a better way than this
>
> aWithMeans = ones_like(a)
> for ind in range(r)  # r = number of rows
>     aWithMeans[ind]* aSortRM[ind]
>
> Now I need to undo the sort I did in the first step.
> I think this is where I need the aargsort (argsort of a before any
> modification) I am kinda stumped on this step. Everything I think of is
> rather convoluted.
> Is it possible to do this whole process without actually rearranging the
> original array and just using the index info from the argsort() and would I
> want to do this?
>

First of all, there's no need to use both argsort and sort:

In [2]: A = np.random.randn(1000)

In [3]: I = A.argsort()

In [4]: B = A[I]

In [5]: np.all(np.diff(B)>=0)
Out[5]: True

Now, to invert the permutation I, the easiest way is just to argsort() it;
but this is N log N instead of order N, so there's another, slightly more
cumbersome, way to do it:

In [12]: Ij = np.zeros_like(I)

In [13]: Ij[I] = np.arange(len(I))

In [23]: AA = B[Ij]

In [24]: np.all(AA==A)
Out[24]: True

Alternatively, you can avoid ever creating the inverse permutation:

In [25]: AA[I] = B

In [26]: np.all(AA==A)
Out[26]: True

All of this is somewhat more painful with two-dimensional arrays, since
fancy and normal indexing do not mix well, and argsort returns something
peuliar, but it can be made to work:

In [51]: A = np.random.randn(100,100)

In [52]: I = np.argsort(A, axis=0)

In [54]: B=A[I,np.arange(A.shape[1])]

In [55]: AA = np.zeros_like(A)

In [56]: AA[I,np.arange(A.shape[1])] = B

In [57]: np.all(AA==A)
Out[57]: True

Now for processing B in place, that's a simpler problem:

In [58]: BB = np.zeros_like(B)

In [59]: BB[...] = np.mean(B,axis=1)

Here numpy's broadcasting rules take care of duplicating the entries of B as
needed to fill out BB. In fact we can dispense with BB entirely:

In [60]: AA[I,np.arange(A.shape[1])] = np.mean(B,axis=1)

So your code can be rewritten as the impenetrable three lines:

In [61]: AA = np.zeros_like(A)

In [62]: I = np.argsort(A,axis=0)

In [63]: AA[I,np.arange(A.shape[1])] =
np.mean(A[I,np.arange(A.shape[1])],axis=1)

Or, even less clearly, making I into what I think argsort should probably
return:

In [64]: I = np.argsort(A,axis=0), np.arange(A.shape[1])

In [65]: AA[I] = np.mean(A[I],axis=1)


Anne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100320/8c06cf70/attachment.html>

From vincent at vincentdavis.net  Sat Mar 20 12:41:03 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 20 Mar 2010 10:41:03 -0600
Subject: [SciPy-User] reversing or avoiding the need to reverse argsort()
In-Reply-To: <ce557a361003200909n63674a0evf0853ac034a2094a@mail.gmail.com>
References: <77e831101003200837j5e93d418m86f44856551b4d58@mail.gmail.com>
	<ce557a361003200909n63674a0evf0853ac034a2094a@mail.gmail.com>
Message-ID: <77e831101003200941q72ee94d6pb940fdb6090a260c@mail.gmail.com>

@Anne Archibald
I get an error and not sure way, or I should say first I need to better
understand the indexing then I might be able to fix it.

In [3]: def quantile_normalization(anarray):
   ...:             """ anarray with samples in the columns and probes
across the rows"""
   ...:         A=anarray
   ...:         AA = np.zeros_like(A)
   ...:         I = np.argsort(A,axis=0)
   ...:         AA[I,np.arange(A.shape[1])] =
np.mean(A[I,np.arange(A.shape[1])],axis=1)
   ...:         return AA
   ...:

In [4]: x = np.array([[1,2,3,4],[4,3,2,1],[1,1,1,1]])

In [5]: quantile_normalization(x)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

/Users/vmd/<ipython console> in <module>()

/Users/vmd/<ipython console> in quantile_normalization(anarray)

ValueError: array is not broadcastable to correct shape

In [6]:

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sat, Mar 20, 2010 at 10:09 AM, Anne Archibald
<peridot.faceted at gmail.com>wrote:

>
>
> On 20 March 2010 11:37, Vincent Davis <vincent at vincentdavis.net> wrote:
>
>> What I want to do is simple but not sure the best way to go about it.
>>
>> I have an array a, shape(a) = rXc I need to sort each column
>> aargsort = a.argsort(axis=0)  # May use this later
>> aSort = a.sort(axis=0)
>>
>> now average each row
>> aSortRM = asort.mean(axis=1)
>>
>> now replace each col in a row with the row mean.
>> is there a better way than this
>>
>> aWithMeans = ones_like(a)
>> for ind in range(r)  # r = number of rows
>>     aWithMeans[ind]* aSortRM[ind]
>>
>> Now I need to undo the sort I did in the first step.
>> I think this is where I need the aargsort (argsort of a before any
>> modification) I am kinda stumped on this step. Everything I think of is
>> rather convoluted.
>> Is it possible to do this whole process without actually rearranging the
>> original array and just using the index info from the argsort() and would I
>> want to do this?
>>
>
> First of all, there's no need to use both argsort and sort:
>
> In [2]: A = np.random.randn(1000)
>
> In [3]: I = A.argsort()
>
> In [4]: B = A[I]
>
> In [5]: np.all(np.diff(B)>=0)
> Out[5]: True
>
> Now, to invert the permutation I, the easiest way is just to argsort() it;
> but this is N log N instead of order N, so there's another, slightly more
> cumbersome, way to do it:
>
> In [12]: Ij = np.zeros_like(I)
>
> In [13]: Ij[I] = np.arange(len(I))
>
> In [23]: AA = B[Ij]
>
> In [24]: np.all(AA==A)
> Out[24]: True
>
> Alternatively, you can avoid ever creating the inverse permutation:
>
> In [25]: AA[I] = B
>
> In [26]: np.all(AA==A)
> Out[26]: True
>
> All of this is somewhat more painful with two-dimensional arrays, since
> fancy and normal indexing do not mix well, and argsort returns something
> peuliar, but it can be made to work:
>
> In [51]: A = np.random.randn(100,100)
>
> In [52]: I = np.argsort(A, axis=0)
>
> In [54]: B=A[I,np.arange(A.shape[1])]
>
> In [55]: AA = np.zeros_like(A)
>
> In [56]: AA[I,np.arange(A.shape[1])] = B
>
> In [57]: np.all(AA==A)
> Out[57]: True
>
> Now for processing B in place, that's a simpler problem:
>
> In [58]: BB = np.zeros_like(B)
>
> In [59]: BB[...] = np.mean(B,axis=1)
>
> Here numpy's broadcasting rules take care of duplicating the entries of B
> as needed to fill out BB. In fact we can dispense with BB entirely:
>
> In [60]: AA[I,np.arange(A.shape[1])] = np.mean(B,axis=1)
>
> So your code can be rewritten as the impenetrable three lines:
>
> In [61]: AA = np.zeros_like(A)
>
> In [62]: I = np.argsort(A,axis=0)
>
> In [63]: AA[I,np.arange(A.shape[1])] =
> np.mean(A[I,np.arange(A.shape[1])],axis=1)
>
> Or, even less clearly, making I into what I think argsort should probably
> return:
>
> In [64]: I = np.argsort(A,axis=0), np.arange(A.shape[1])
>
> In [65]: AA[I] = np.mean(A[I],axis=1)
>
>
> Anne
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100320/6a565d17/attachment.html>

From peridot.faceted at gmail.com  Sat Mar 20 12:59:21 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sat, 20 Mar 2010 12:59:21 -0400
Subject: [SciPy-User] reversing or avoiding the need to reverse argsort()
In-Reply-To: <77e831101003200941q72ee94d6pb940fdb6090a260c@mail.gmail.com>
References: <77e831101003200837j5e93d418m86f44856551b4d58@mail.gmail.com>
	<ce557a361003200909n63674a0evf0853ac034a2094a@mail.gmail.com>
	<77e831101003200941q72ee94d6pb940fdb6090a260c@mail.gmail.com>
Message-ID: <ce557a361003200959gcf288c1t3c8637200762e437@mail.gmail.com>

On 20 March 2010 12:41, Vincent Davis <vincent at vincentdavis.net> wrote:

> @Anne Archibald
> I get an error and not sure way, or I should say first I need to better
> understand the indexing then I might be able to fix it.

Ah, oops. I got the indexing wrong. numpy's broadcasting needs a little
help:

AA[I,np.arange(A.shape[1])] =
np.mean(A[I,np.arange(A.shape[1])],axis=1)[:,np.newaxis]

What's going on in this broadcasting is several things.

First, when you argsort a high-dimensional array along one axis, you get an
array the same shape as what you started with, containing the indices along
that axis. But to use fancy indexing on a two-dimensional array, the way it
works is:

A[B, C]

That is, B and C should be arrays the same shape as each other (but not
necessarily the same shape as A). The result will have the same shape as B
and C; what you get is (say B and C are three-dimensional):

(A[B, C]) [i,j,k] = A[B[i,j,k], C[i,j,k]]

So in your case, the argsorted array I is exactly what should go in the
first place. What you really want is:

B[i,j] = A[np.argsort(A,axis=0)[i,j], j]

So you want to make up an array for the second index whose (i,j)th entry is
just j. But I supplied np.arange(A.shape[1]); this is only a one-dimensional
array. Rather than complain that the index arrays don't match up with each
other, numpy tries adding a "fake" axis to the beginning, so that it looks
like a two-dimensional array M that just ignores its first index:  M[i,j] =
np.arange(A.shape[1])[j] =
Which is what you want.

All this was correct in the original email. What I got wrong was the mean:
taking the mean across each row leaves you with a one-dimensional array,
which we try to stuff into a two-dimensional array. So numpy tries its trick
of adding an axis at the beginning, only to find out that that doesn't work.
We needed to add an axis at the end, which numpy won't automatically do for
us. So we do it by using np.newaxis.

Anne

In [3]: def quantile_normalization(anarray):
>    ...:             """ anarray with samples in the columns and probes
> across the rows"""
>    ...:         A=anarray
>    ...:         AA = np.zeros_like(A)
>    ...:         I = np.argsort(A,axis=0)
>    ...:         AA[I,np.arange(A.shape[1])] =
> np.mean(A[I,np.arange(A.shape[1])],axis=1)
>    ...:         return AA
>    ...:
>
> In [4]: x = np.array([[1,2,3,4],[4,3,2,1],[1,1,1,1]])
>
> In [5]: quantile_normalization(x)
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call last)
>
> /Users/vmd/<ipython console> in <module>()
>
> /Users/vmd/<ipython console> in quantile_normalization(anarray)
>
> ValueError: array is not broadcastable to correct shape
>
> In [6]:
>
>   *Vincent Davis
> 720-301-3003 *
> vincent at vincentdavis.net
>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>
>
> On Sat, Mar 20, 2010 at 10:09 AM, Anne Archibald <
> peridot.faceted at gmail.com> wrote:
>
>>
>>
>> On 20 March 2010 11:37, Vincent Davis <vincent at vincentdavis.net> wrote:
>>
>>> What I want to do is simple but not sure the best way to go about it.
>>>
>>> I have an array a, shape(a) = rXc I need to sort each column
>>> aargsort = a.argsort(axis=0)  # May use this later
>>> aSort = a.sort(axis=0)
>>>
>>> now average each row
>>> aSortRM = asort.mean(axis=1)
>>>
>>> now replace each col in a row with the row mean.
>>> is there a better way than this
>>>
>>> aWithMeans = ones_like(a)
>>> for ind in range(r)  # r = number of rows
>>>     aWithMeans[ind]* aSortRM[ind]
>>>
>>> Now I need to undo the sort I did in the first step.
>>> I think this is where I need the aargsort (argsort of a before any
>>> modification) I am kinda stumped on this step. Everything I think of is
>>> rather convoluted.
>>> Is it possible to do this whole process without actually rearranging the
>>> original array and just using the index info from the argsort() and would I
>>> want to do this?
>>>
>>
>> First of all, there's no need to use both argsort and sort:
>>
>> In [2]: A = np.random.randn(1000)
>>
>> In [3]: I = A.argsort()
>>
>> In [4]: B = A[I]
>>
>> In [5]: np.all(np.diff(B)>=0)
>> Out[5]: True
>>
>> Now, to invert the permutation I, the easiest way is just to argsort() it;
>> but this is N log N instead of order N, so there's another, slightly more
>> cumbersome, way to do it:
>>
>> In [12]: Ij = np.zeros_like(I)
>>
>> In [13]: Ij[I] = np.arange(len(I))
>>
>> In [23]: AA = B[Ij]
>>
>> In [24]: np.all(AA==A)
>> Out[24]: True
>>
>> Alternatively, you can avoid ever creating the inverse permutation:
>>
>> In [25]: AA[I] = B
>>
>> In [26]: np.all(AA==A)
>> Out[26]: True
>>
>> All of this is somewhat more painful with two-dimensional arrays, since
>> fancy and normal indexing do not mix well, and argsort returns something
>> peuliar, but it can be made to work:
>>
>> In [51]: A = np.random.randn(100,100)
>>
>> In [52]: I = np.argsort(A, axis=0)
>>
>> In [54]: B=A[I,np.arange(A.shape[1])]
>>
>> In [55]: AA = np.zeros_like(A)
>>
>> In [56]: AA[I,np.arange(A.shape[1])] = B
>>
>> In [57]: np.all(AA==A)
>> Out[57]: True
>>
>> Now for processing B in place, that's a simpler problem:
>>
>> In [58]: BB = np.zeros_like(B)
>>
>> In [59]: BB[...] = np.mean(B,axis=1)
>>
>> Here numpy's broadcasting rules take care of duplicating the entries of B
>> as needed to fill out BB. In fact we can dispense with BB entirely:
>>
>> In [60]: AA[I,np.arange(A.shape[1])] = np.mean(B,axis=1)
>>
>> So your code can be rewritten as the impenetrable three lines:
>>
>> In [61]: AA = np.zeros_like(A)
>>
>> In [62]: I = np.argsort(A,axis=0)
>>
>> In [63]: AA[I,np.arange(A.shape[1])] =
>> np.mean(A[I,np.arange(A.shape[1])],axis=1)
>>
>> Or, even less clearly, making I into what I think argsort should probably
>> return:
>>
>> In [64]: I = np.argsort(A,axis=0), np.arange(A.shape[1])
>>
>> In [65]: AA[I] = np.mean(A[I],axis=1)
>>
>>
>> Anne
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100320/551e0e01/attachment.html>

From vincent at vincentdavis.net  Sat Mar 20 13:08:08 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sat, 20 Mar 2010 11:08:08 -0600
Subject: [SciPy-User] reversing or avoiding the need to reverse argsort()
In-Reply-To: <ce557a361003200959gcf288c1t3c8637200762e437@mail.gmail.com>
References: <77e831101003200837j5e93d418m86f44856551b4d58@mail.gmail.com>
	<ce557a361003200909n63674a0evf0853ac034a2094a@mail.gmail.com>
	<77e831101003200941q72ee94d6pb940fdb6090a260c@mail.gmail.com>
	<ce557a361003200959gcf288c1t3c8637200762e437@mail.gmail.com>
Message-ID: <77e831101003201008x1db37424qbc9be92cf25ae2d8@mail.gmail.com>

@Anne Archibald
Thanks for that long reply, The code worked and now I have a better
understanding of fancy indexing although I need to ready your reply several
 more times. Is fancy indexing documented with lots of examples somewhere.

Thanks Again

 *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sat, Mar 20, 2010 at 10:59 AM, Anne Archibald
<peridot.faceted at gmail.com>wrote:

>
>
> On 20 March 2010 12:41, Vincent Davis <vincent at vincentdavis.net> wrote:
>
>> @Anne Archibald
>> I get an error and not sure way, or I should say first I need to better
>> understand the indexing then I might be able to fix it.
>>
> Ah, oops. I got the indexing wrong. numpy's broadcasting needs a little
> help:
>
> AA[I,np.arange(A.shape[1])] =
> np.mean(A[I,np.arange(A.shape[1])],axis=1)[:,np.newaxis]
>
> What's going on in this broadcasting is several things.
>
> First, when you argsort a high-dimensional array along one axis, you get an
> array the same shape as what you started with, containing the indices along
> that axis. But to use fancy indexing on a two-dimensional array, the way it
> works is:
>
> A[B, C]
>
> That is, B and C should be arrays the same shape as each other (but not
> necessarily the same shape as A). The result will have the same shape as B
> and C; what you get is (say B and C are three-dimensional):
>
> (A[B, C]) [i,j,k] = A[B[i,j,k], C[i,j,k]]
>
> So in your case, the argsorted array I is exactly what should go in the
> first place. What you really want is:
>
> B[i,j] = A[np.argsort(A,axis=0)[i,j], j]
>
> So you want to make up an array for the second index whose (i,j)th entry is
> just j. But I supplied np.arange(A.shape[1]); this is only a one-dimensional
> array. Rather than complain that the index arrays don't match up with each
> other, numpy tries adding a "fake" axis to the beginning, so that it looks
> like a two-dimensional array M that just ignores its first index:  M[i,j] =
> np.arange(A.shape[1])[j] =
> Which is what you want.
>
> All this was correct in the original email. What I got wrong was the mean:
> taking the mean across each row leaves you with a one-dimensional array,
> which we try to stuff into a two-dimensional array. So numpy tries its trick
> of adding an axis at the beginning, only to find out that that doesn't work.
> We needed to add an axis at the end, which numpy won't automatically do for
> us. So we do it by using np.newaxis.
>
> Anne
>
> In [3]: def quantile_normalization(anarray):
>>    ...:             """ anarray with samples in the columns and probes
>> across the rows"""
>>    ...:         A=anarray
>>    ...:         AA = np.zeros_like(A)
>>    ...:         I = np.argsort(A,axis=0)
>>    ...:         AA[I,np.arange(A.shape[1])] =
>> np.mean(A[I,np.arange(A.shape[1])],axis=1)
>>    ...:         return AA
>>    ...:
>>
>> In [4]: x = np.array([[1,2,3,4],[4,3,2,1],[1,1,1,1]])
>>
>> In [5]: quantile_normalization(x)
>>
>> ---------------------------------------------------------------------------
>> ValueError                                Traceback (most recent call
>> last)
>>
>> /Users/vmd/<ipython console> in <module>()
>>
>> /Users/vmd/<ipython console> in quantile_normalization(anarray)
>>
>> ValueError: array is not broadcastable to correct shape
>>
>> In [6]:
>>
>>   *Vincent Davis
>> 720-301-3003 *
>> vincent at vincentdavis.net
>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>
>>
>> On Sat, Mar 20, 2010 at 10:09 AM, Anne Archibald <
>> peridot.faceted at gmail.com> wrote:
>>
>>>
>>>
>>> On 20 March 2010 11:37, Vincent Davis <vincent at vincentdavis.net> wrote:
>>>
>>>> What I want to do is simple but not sure the best way to go about it.
>>>>
>>>> I have an array a, shape(a) = rXc I need to sort each column
>>>> aargsort = a.argsort(axis=0)  # May use this later
>>>> aSort = a.sort(axis=0)
>>>>
>>>> now average each row
>>>> aSortRM = asort.mean(axis=1)
>>>>
>>>> now replace each col in a row with the row mean.
>>>> is there a better way than this
>>>>
>>>> aWithMeans = ones_like(a)
>>>> for ind in range(r)  # r = number of rows
>>>>     aWithMeans[ind]* aSortRM[ind]
>>>>
>>>> Now I need to undo the sort I did in the first step.
>>>> I think this is where I need the aargsort (argsort of a before any
>>>> modification) I am kinda stumped on this step. Everything I think of is
>>>> rather convoluted.
>>>> Is it possible to do this whole process without actually rearranging the
>>>> original array and just using the index info from the argsort() and would I
>>>> want to do this?
>>>>
>>>
>>> First of all, there's no need to use both argsort and sort:
>>>
>>> In [2]: A = np.random.randn(1000)
>>>
>>> In [3]: I = A.argsort()
>>>
>>> In [4]: B = A[I]
>>>
>>> In [5]: np.all(np.diff(B)>=0)
>>> Out[5]: True
>>>
>>> Now, to invert the permutation I, the easiest way is just to argsort()
>>> it; but this is N log N instead of order N, so there's another, slightly
>>> more cumbersome, way to do it:
>>>
>>> In [12]: Ij = np.zeros_like(I)
>>>
>>> In [13]: Ij[I] = np.arange(len(I))
>>>
>>> In [23]: AA = B[Ij]
>>>
>>> In [24]: np.all(AA==A)
>>> Out[24]: True
>>>
>>> Alternatively, you can avoid ever creating the inverse permutation:
>>>
>>> In [25]: AA[I] = B
>>>
>>> In [26]: np.all(AA==A)
>>> Out[26]: True
>>>
>>> All of this is somewhat more painful with two-dimensional arrays, since
>>> fancy and normal indexing do not mix well, and argsort returns something
>>> peuliar, but it can be made to work:
>>>
>>> In [51]: A = np.random.randn(100,100)
>>>
>>> In [52]: I = np.argsort(A, axis=0)
>>>
>>> In [54]: B=A[I,np.arange(A.shape[1])]
>>>
>>> In [55]: AA = np.zeros_like(A)
>>>
>>> In [56]: AA[I,np.arange(A.shape[1])] = B
>>>
>>> In [57]: np.all(AA==A)
>>> Out[57]: True
>>>
>>> Now for processing B in place, that's a simpler problem:
>>>
>>> In [58]: BB = np.zeros_like(B)
>>>
>>> In [59]: BB[...] = np.mean(B,axis=1)
>>>
>>> Here numpy's broadcasting rules take care of duplicating the entries of B
>>> as needed to fill out BB. In fact we can dispense with BB entirely:
>>>
>>> In [60]: AA[I,np.arange(A.shape[1])] = np.mean(B,axis=1)
>>>
>>> So your code can be rewritten as the impenetrable three lines:
>>>
>>> In [61]: AA = np.zeros_like(A)
>>>
>>> In [62]: I = np.argsort(A,axis=0)
>>>
>>> In [63]: AA[I,np.arange(A.shape[1])] =
>>> np.mean(A[I,np.arange(A.shape[1])],axis=1)
>>>
>>> Or, even less clearly, making I into what I think argsort should probably
>>> return:
>>>
>>> In [64]: I = np.argsort(A,axis=0), np.arange(A.shape[1])
>>>
>>> In [65]: AA[I] = np.mean(A[I],axis=1)
>>>
>>>
>>> Anne
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100320/91346a41/attachment.html>

From robert.kern at gmail.com  Sat Mar 20 15:56:06 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 20 Mar 2010 13:56:06 -0600
Subject: [SciPy-User] ask.scipy.org
In-Reply-To: <61a4c0ba1003192021y6331a6fdv6ef93d9c0ffebbf8@mail.gmail.com>
References: <61a4c0ba1003192021y6331a6fdv6ef93d9c0ffebbf8@mail.gmail.com>
Message-ID: <3d375d731003201256i7e040369r10c82edeaf020c84@mail.gmail.com>

On Fri, Mar 19, 2010 at 21:21, Alec Koumjian <akoumjian at gmail.com> wrote:
> I'm very much looking forward to using the site. However, I can't seem
> to get the sign-up page to work with a Google login. I click the
> "Google" icon to use my crudentials, but it simply redirects me back
> to the OpenID page. ?Is this feature not available yet?

Other people have reported this problem, but I've always been able to
log in using my Google account. Someone has suggested that already
being logged in through GMail or whatever Google service will help.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From seb.haase at gmail.com  Sat Mar 20 16:23:37 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Sat, 20 Mar 2010 21:23:37 +0100
Subject: [SciPy-User] ask.scipy.org
In-Reply-To: <3d375d731003201256i7e040369r10c82edeaf020c84@mail.gmail.com>
References: <61a4c0ba1003192021y6331a6fdv6ef93d9c0ffebbf8@mail.gmail.com> 
	<3d375d731003201256i7e040369r10c82edeaf020c84@mail.gmail.com>
Message-ID: <bc657ead1003201323v2a631222y6c0956fdd2a5ac6d@mail.gmail.com>

On Sat, Mar 20, 2010 at 8:56 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Fri, Mar 19, 2010 at 21:21, Alec Koumjian <akoumjian at gmail.com> wrote:
>> I'm very much looking forward to using the site. However, I can't seem
>> to get the sign-up page to work with a Google login. I click the
>> "Google" icon to use my crudentials, but it simply redirects me back
>> to the OpenID page. ?Is this feature not available yet?
>
> Other people have reported this problem, but I've always been able to
> log in using my Google account. Someone has suggested that already
> being logged in through GMail or whatever Google service will help.
>
I am currently logged into gmail, and I can confirm that clicking on
any icon it just jumps back to OpenID.
I use google chrome.
 -Sebastian


From robert.kern at gmail.com  Sat Mar 20 16:32:09 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 20 Mar 2010 14:32:09 -0600
Subject: [SciPy-User] ask.scipy.org
In-Reply-To: <bc657ead1003201323v2a631222y6c0956fdd2a5ac6d@mail.gmail.com>
References: <61a4c0ba1003192021y6331a6fdv6ef93d9c0ffebbf8@mail.gmail.com> 
	<3d375d731003201256i7e040369r10c82edeaf020c84@mail.gmail.com> 
	<bc657ead1003201323v2a631222y6c0956fdd2a5ac6d@mail.gmail.com>
Message-ID: <3d375d731003201332y4b294875pffa27eac3ed030b5@mail.gmail.com>

On Sat, Mar 20, 2010 at 14:23, Sebastian Haase <seb.haase at gmail.com> wrote:
> On Sat, Mar 20, 2010 at 8:56 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> On Fri, Mar 19, 2010 at 21:21, Alec Koumjian <akoumjian at gmail.com> wrote:
>>> I'm very much looking forward to using the site. However, I can't seem
>>> to get the sign-up page to work with a Google login. I click the
>>> "Google" icon to use my crudentials, but it simply redirects me back
>>> to the OpenID page. ?Is this feature not available yet?
>>
>> Other people have reported this problem, but I've always been able to
>> log in using my Google account. Someone has suggested that already
>> being logged in through GMail or whatever Google service will help.
>>
> I am currently logged into gmail, and I can confirm that clicking on
> any icon it just jumps back to OpenID.
> I use google chrome.

Okay, *now* I'm having the problem, too. I'll take a look at it tomorrow.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Sat Mar 20 16:52:40 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 20 Mar 2010 16:52:40 -0400
Subject: [SciPy-User] 2d convolution
In-Reply-To: <d933725a1003200907u167a7b40p8cbbca12686ce32b@mail.gmail.com>
References: <d933725a1003200451u1996c66escb68316201adc266@mail.gmail.com>
	<1cd32cbb1003200549m75a86fbelea959ad6696dc526@mail.gmail.com>
	<d933725a1003200907u167a7b40p8cbbca12686ce32b@mail.gmail.com>
Message-ID: <1cd32cbb1003201352g14bf6bc4tf018d7e5bcde07fb@mail.gmail.com>

On Sat, Mar 20, 2010 at 12:07 PM, Nico Schl?mer
<nico.schloemer at gmail.com> wrote:
> Thanks for the hint!
>
> I've now set "mode='full'" and extract the data myself by
>
> ? ? ? ? ? ?CR = scipy.signal.fftconvolve( C, R, mode='full' )
> ? ? ? ? ? ?from scipy.signal.signaltools import _centered
> ? ? ? ? ? ?CR = _centered( CR , C.shape )

glad it works,

I forgot to mention earlier:
scipy.stsci.convolve.convolve2d   uses numpy.fft.fft2

If it is much faster than the n-dimensional fft convolution, it might
be worth writing a fftconvolve2

Josef

>
> Cheers,
> Nico
>
>
> On Sat, Mar 20, 2010 at 1:49 PM, ?<josef.pktd at gmail.com> wrote:
>> On Sat, Mar 20, 2010 at 7:51 AM, Nico Schl?mer <nico.schloemer at gmail.com> wrote:
>>> Hi,
>>>
>>> I'm trying to compute the the convolution if s 2D array, and I see
>>> that there are several ways in SciPy to do that. As the original data
>>> C and the kernel R are about the same size in my case, I'd profit from
>>> an FFT-based implementation, which I see right now is given by
>>>
>>> ? ?scipy.signal.fftconvolve( C, R,
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? mode='same' )
>>>
>>> and also
>>>
>>> ? ?scipy.stsci.convolve.convolve2d( data=C, kernel=R,
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?mode='constant', cval=0.0,
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?fft=1 )
>>>
>>> The second method is much faster than the first, and as far as I can
>>> see it would spit out the the same results.
>>>
>>> Now, when the kernel is actually larger than the data, the resulting
>>> array would have the shape of the kernel. Is there any way to restrict
>>> the computations to the size of the data? At first, I thought that was
>>> what "mode='same'"" is for.
>>> I tried cutting the extra data off of the resulting array, but I'm not
>>> quite sure which is the part that I would like to get rid of.
>>>
>>> Any hints here?
>>
>> It's always good to take a look at the source
>>
>> fftconvolve
>>
>> ? ?elif mode == "same":
>> ? ? ? ?if product(s1,axis=0) > product(s2,axis=0):
>> ? ? ? ? ? ?osize = s1
>> ? ? ? ?else:
>> ? ? ? ? ? ?osize = s2
>> ? ? ? ?return _centered(ret,osize)
>> ? ?elif mode == "valid":
>> ? ? ? ?return _centered(ret,abs(s2-s1)+1)
>>
>> from scipy.signal.signaltools import _centered
>>
>> and it should be possible to ?set osize to s1 = array(in1.shape) ? (C
>> in your notation)
>>
>> scipy.stsci has been removed recently from scipy and will not be in
>> the scipy 0.8 version
>>
>> (I didn't try myself whether it works)
>>
>> Josef
>>
>>>
>>> Cheers,
>>> Nico
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From seb.haase at gmail.com  Sat Mar 20 16:53:02 2010
From: seb.haase at gmail.com (Sebastian Haase)
Date: Sat, 20 Mar 2010 21:53:02 +0100
Subject: [SciPy-User] ask.scipy.org
In-Reply-To: <3d375d731003201332y4b294875pffa27eac3ed030b5@mail.gmail.com>
References: <61a4c0ba1003192021y6331a6fdv6ef93d9c0ffebbf8@mail.gmail.com> 
	<3d375d731003201256i7e040369r10c82edeaf020c84@mail.gmail.com> 
	<bc657ead1003201323v2a631222y6c0956fdd2a5ac6d@mail.gmail.com> 
	<3d375d731003201332y4b294875pffa27eac3ed030b5@mail.gmail.com>
Message-ID: <bc657ead1003201353r12b9d59fndc6cb3f8993db19d@mail.gmail.com>

On Sat, Mar 20, 2010 at 9:32 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Sat, Mar 20, 2010 at 14:23, Sebastian Haase <seb.haase at gmail.com> wrote:
>> On Sat, Mar 20, 2010 at 8:56 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>> On Fri, Mar 19, 2010 at 21:21, Alec Koumjian <akoumjian at gmail.com> wrote:
>>>> I'm very much looking forward to using the site. However, I can't seem
>>>> to get the sign-up page to work with a Google login. I click the
>>>> "Google" icon to use my crudentials, but it simply redirects me back
>>>> to the OpenID page. ?Is this feature not available yet?
>>>
>>> Other people have reported this problem, but I've always been able to
>>> log in using my Google account. Someone has suggested that already
>>> being logged in through GMail or whatever Google service will help.
>>>
>> I am currently logged into gmail, and I can confirm that clicking on
>> any icon it just jumps back to OpenID.
>> I use google chrome.
>
> Okay, *now* I'm having the problem, too. I'll take a look at it tomorrow.
>
Apparently I don't understand any of this openID stuff, but (as
someone on ask.scipy.org suggested)
copy/paste of
https://www.google.com/accounts/o8/id
into the login text field got me in ....
??
-Sebastian


From harald.schilly at gmail.com  Sat Mar 20 20:41:22 2010
From: harald.schilly at gmail.com (Harald Schilly)
Date: Sun, 21 Mar 2010 01:41:22 +0100
Subject: [SciPy-User] ask.scipy.org
In-Reply-To: <bc657ead1003201353r12b9d59fndc6cb3f8993db19d@mail.gmail.com>
References: <61a4c0ba1003192021y6331a6fdv6ef93d9c0ffebbf8@mail.gmail.com> 
	<3d375d731003201256i7e040369r10c82edeaf020c84@mail.gmail.com> 
	<bc657ead1003201323v2a631222y6c0956fdd2a5ac6d@mail.gmail.com> 
	<3d375d731003201332y4b294875pffa27eac3ed030b5@mail.gmail.com> 
	<bc657ead1003201353r12b9d59fndc6cb3f8993db19d@mail.gmail.com>
Message-ID: <20548feb1003201741v67a8ed77h686a52152275d63a@mail.gmail.com>

On Sat, Mar 20, 2010 at 21:53, Sebastian Haase <seb.haase at gmail.com> wrote:
> copy/paste of
> https://www.google.com/accounts/o8/id
> into the login text field got me in ....

that worked for me, too. seems like it's something with the html/form values.

h


From jsseabold at gmail.com  Sun Mar 21 13:12:35 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sun, 21 Mar 2010 13:12:35 -0400
Subject: [SciPy-User] determining if structured/rec array
Message-ID: <c048da1c1003211012u3c0c601ci6bb64db18ca45a2a@mail.gmail.com>

Is there anywhere in scipy that works on structured arrays
dynamically?  If not, is this a good way to determine if a structured/
rec array was given to a function or is there a better/easier way?

import numpy as np

def isstructarray(X):
    """
    Returns a bool indicating if X is a structured or recarray.

    Parameters
    -----------
    X : ndarray or subclass

    Returns
    -------
    True or False, indicating if X is a structured or recarray.
    """
    if X.__class__ is np.recarray or (isinstance(X, np.ndarray) and\
            X.dtype.names):
        return True
    else:
        return False

X = np.array([('1', 1.0), ('1', 1.0), ('1', 1.0)],
      dtype=[('foo', 'a1'), ('bar', '<f8')])

Y = np.rec.array(X)

Z = np.ones((2,3))

L = [1,2,3]

isstructarray(X)
# True
isstructarray(Y)
# True
isstructarray(Z)
# False
isstructarray(L)
# False

-Skipper


From jsseabold at gmail.com  Sun Mar 21 13:14:45 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sun, 21 Mar 2010 13:14:45 -0400
Subject: [SciPy-User] determining types in dtype
Message-ID: <c048da1c1003211014j6255a0e0j71cb0c51c9fd078f@mail.gmail.com>

Is there an easy way to determine if the dtype of a (non-nested)
structured-array is homogenous/similar?  Basically, I want to know if
all elements are int and/or float so it can be safely cast to an
ndarray or if it contains strings (or, more generally, other objects)
so it needs some more processing.

I thought issctype might provide what I'm looking for but

import numpy as np
X = np.array([('1', 1.0), ('1', 1.0), ('1', 1.0)],
      dtype=[('foo', 'a1'), ('bar', '<f8')])
np.issctype(X.dtype)
# True

Similar result for nested structs of string type

Y = np.array([(('1','2'), 1.0), (('1','2'), 1.0), (('1','2'), 1.0)],
      dtype=[('foo', 'a1', (1,2)), ('bar', '<f8')])
np.issctype(Y.dtype)
# True

In particular, I find this odd

np.issctype(str)
# True

I would expect this to be like

np.issctype(object)
# False

Unless there is something I don't understand about types, which is
probably the case.

The only other way I can think of is to use X.dtype.descr and actually
parse the list to determine the types.  Any thoughts?

Skipper


From eadrogue at gmx.net  Sun Mar 21 13:41:11 2010
From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=)
Date: Sun, 21 Mar 2010 18:41:11 +0100
Subject: [SciPy-User] determining types in dtype
In-Reply-To: <c048da1c1003211014j6255a0e0j71cb0c51c9fd078f@mail.gmail.com>
References: <c048da1c1003211014j6255a0e0j71cb0c51c9fd078f@mail.gmail.com>
Message-ID: <20100321174111.GA22640@doriath.local>

21/03/10 @ 13:14 (-0400), thus spake Skipper Seabold:
> Is there an easy way to determine if the dtype of a (non-nested)
> structured-array is homogenous/similar?  Basically, I want to know if
> all elements are int and/or float so it can be safely cast to an
> ndarray or if it contains strings (or, more generally, other objects)
> so it needs some more processing.
> 
> I thought issctype might provide what I'm looking for but
> 
> import numpy as np
> X = np.array([('1', 1.0), ('1', 1.0), ('1', 1.0)],
>       dtype=[('foo', 'a1'), ('bar', '<f8')])
> np.issctype(X.dtype)
> # True
> 
> Similar result for nested structs of string type
> 
> Y = np.array([(('1','2'), 1.0), (('1','2'), 1.0), (('1','2'), 1.0)],
>       dtype=[('foo', 'a1', (1,2)), ('bar', '<f8')])
> np.issctype(Y.dtype)
> # True
> 
> In particular, I find this odd
> 
> np.issctype(str)
> # True
> 
> I would expect this to be like
> 
> np.issctype(object)
> # False
> 
> Unless there is something I don't understand about types, which is
> probably the case.
> 
> The only other way I can think of is to use X.dtype.descr and actually
> parse the list to determine the types.  Any thoughts?

Yes, this is what I do. From X.dtype.fields you get the dtype
of each field:

dtypes = set(i[0] for i in X.dtype.fields.values())
if len(dtypes) == 1:
	print 'homogenous'
	view = X.view(dtypes.pop())
else:
	print 'heterogenous'

Bye.

> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From rmay31 at gmail.com  Sun Mar 21 13:48:25 2010
From: rmay31 at gmail.com (Ryan May)
Date: Sun, 21 Mar 2010 12:48:25 -0500
Subject: [SciPy-User] determining if structured/rec array
In-Reply-To: <c048da1c1003211012u3c0c601ci6bb64db18ca45a2a@mail.gmail.com>
References: <c048da1c1003211012u3c0c601ci6bb64db18ca45a2a@mail.gmail.com>
Message-ID: <c7b95a021003211048v638df9a6hfc9d1e3f4803c402@mail.gmail.com>

On Sun, Mar 21, 2010 at 12:12 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> Is there anywhere in scipy that works on structured arrays
> dynamically? ?If not, is this a good way to determine if a structured/
> rec array was given to a function or is there a better/easier way?

Not commenting on the utility, but numpy as lib/recfunctions.py that
would seem to be a sensible spot for such a function.  Certainly,
since it deals just with numpy arrays, it should live in numpy and not
scipy.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From jsseabold at gmail.com  Sun Mar 21 16:00:51 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sun, 21 Mar 2010 16:00:51 -0400
Subject: [SciPy-User] determining if structured/rec array
In-Reply-To: <c7b95a021003211048v638df9a6hfc9d1e3f4803c402@mail.gmail.com>
References: <c048da1c1003211012u3c0c601ci6bb64db18ca45a2a@mail.gmail.com> 
	<c7b95a021003211048v638df9a6hfc9d1e3f4803c402@mail.gmail.com>
Message-ID: <c048da1c1003211300h259dc372l61d5fbf466a26bf5@mail.gmail.com>

On Sun, Mar 21, 2010 at 1:48 PM, Ryan May <rmay31 at gmail.com> wrote:
> On Sun, Mar 21, 2010 at 12:12 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
>> Is there anywhere in scipy that works on structured arrays
>> dynamically? ?If not, is this a good way to determine if a structured/
>> rec array was given to a function or is there a better/easier way?
>
> Not commenting on the utility, but numpy as lib/recfunctions.py that
> would seem to be a sensible spot for such a function. ?Certainly,
> since it deals just with numpy arrays, it should live in numpy and not
> scipy.
>

Agreed.  Is there general interest in something like this?  It's
really simply obviously, but the function is convenient and more
transparent than what I've been doing.

Skipper


From vincent at vincentdavis.net  Sun Mar 21 21:19:37 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sun, 21 Mar 2010 19:19:37 -0600
Subject: [SciPy-User] ask.scipy.org, google open id not working
Message-ID: <77e831101003211819k10732ec3k6c12f6a3b676e71f@mail.gmail.com>

Not sure whats up but I can click on the google icon all day and I still
can't create an account. it just refreshes the page. Maybe I am doing
something wrong but I have signed into many websites using my google email
as an open id but it does not seem to work here.
I would like to ask a question but I will post it to the mail list instead
:(

Let me know if I can help

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100321/7d5ef1b4/attachment.html>

From jsseabold at gmail.com  Sun Mar 21 21:27:50 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sun, 21 Mar 2010 21:27:50 -0400
Subject: [SciPy-User] ask.scipy.org, google open id not working
In-Reply-To: <77e831101003211819k10732ec3k6c12f6a3b676e71f@mail.gmail.com>
References: <77e831101003211819k10732ec3k6c12f6a3b676e71f@mail.gmail.com>
Message-ID: <c048da1c1003211827q680cf6ahb7465171ec5e633f@mail.gmail.com>

On Sun, Mar 21, 2010 at 9:19 PM, Vincent Davis <vincent at vincentdavis.net> wrote:
>
> Not sure whats up but I can click on the google icon all day and I still can't create an account. it just refreshes the page. Maybe I am doing something wrong but I have signed into many websites using my google email as an open id but it does not seem to work here.
> I would like to ask a question but I will post it to the mail list instead :(
> Let me know if I can help
>

I think there's something borked in the html?  If you copy/paste
https://www.google.com/accounts/o8/id into the login, it should work.
Worked for me and others.

Skipper


From jsseabold at gmail.com  Sun Mar 21 21:30:14 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sun, 21 Mar 2010 21:30:14 -0400
Subject: [SciPy-User] determining types in dtype
In-Reply-To: <20100321174111.GA22640@doriath.local>
References: <c048da1c1003211014j6255a0e0j71cb0c51c9fd078f@mail.gmail.com> 
	<20100321174111.GA22640@doriath.local>
Message-ID: <c048da1c1003211830l31cd7e78ma462ad54b7ad168f@mail.gmail.com>

2010/3/21 Ernest Adrogu? <eadrogue at gmx.net>:
> 21/03/10 @ 13:14 (-0400), thus spake Skipper Seabold:
>> Is there an easy way to determine if the dtype of a (non-nested)
>> structured-array is homogenous/similar? ?Basically, I want to know if
>> all elements are int and/or float so it can be safely cast to an
>> ndarray or if it contains strings (or, more generally, other objects)
>> so it needs some more processing.
>>
>> I thought issctype might provide what I'm looking for but
>>
>> import numpy as np
>> X = np.array([('1', 1.0), ('1', 1.0), ('1', 1.0)],
>> ? ? ? dtype=[('foo', 'a1'), ('bar', '<f8')])
>> np.issctype(X.dtype)
>> # True
>>
>> Similar result for nested structs of string type
>>
>> Y = np.array([(('1','2'), 1.0), (('1','2'), 1.0), (('1','2'), 1.0)],
>> ? ? ? dtype=[('foo', 'a1', (1,2)), ('bar', '<f8')])
>> np.issctype(Y.dtype)
>> # True
>>
>> In particular, I find this odd
>>
>> np.issctype(str)
>> # True
>>
>> I would expect this to be like
>>
>> np.issctype(object)
>> # False
>>
>> Unless there is something I don't understand about types, which is
>> probably the case.
>>
>> The only other way I can think of is to use X.dtype.descr and actually
>> parse the list to determine the types. ?Any thoughts?
>
> Yes, this is what I do. From X.dtype.fields you get the dtype
> of each field:
>
> dtypes = set(i[0] for i in X.dtype.fields.values())
> if len(dtypes) == 1:
> ? ? ? ?print 'homogenous'
> ? ? ? ?view = X.view(dtypes.pop())
> else:
> ? ? ? ?print 'heterogenous'
>

Thanks, I think I will take a similar approach.  np.can_cast also
looks very helpful.

Skipper


From vincent at vincentdavis.net  Sun Mar 21 21:36:31 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sun, 21 Mar 2010 19:36:31 -0600
Subject: [SciPy-User] ask.scipy.org, google open id not working
In-Reply-To: <c048da1c1003211827q680cf6ahb7465171ec5e633f@mail.gmail.com>
References: <77e831101003211819k10732ec3k6c12f6a3b676e71f@mail.gmail.com>
	<c048da1c1003211827q680cf6ahb7465171ec5e633f@mail.gmail.com>
Message-ID: <77e831101003211836m1e920e96kb948fd99c7573e1e@mail.gmail.com>

That did it, thanks
Obviously it should be fixed, but until than maybe making note of this
workaround on the login page would be good.

*Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sun, Mar 21, 2010 at 7:27 PM, Skipper Seabold <jsseabold at gmail.com>wrote:

> On Sun, Mar 21, 2010 at 9:19 PM, Vincent Davis <vincent at vincentdavis.net>
> wrote:
> >
> > Not sure whats up but I can click on the google icon all day and I still
> can't create an account. it just refreshes the page. Maybe I am doing
> something wrong but I have signed into many websites using my google email
> as an open id but it does not seem to work here.
> > I would like to ask a question but I will post it to the mail list
> instead :(
> > Let me know if I can help
> >
>
> I think there's something borked in the html?  If you copy/paste
> https://www.google.com/accounts/o8/id into the login, it should work.
> Worked for me and others.
>
> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100321/cec2518e/attachment.html>

From pgmdevlist at gmail.com  Sun Mar 21 21:43:33 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Sun, 21 Mar 2010 20:43:33 -0500
Subject: [SciPy-User] determining if structured/rec array
In-Reply-To: <c048da1c1003211012u3c0c601ci6bb64db18ca45a2a@mail.gmail.com>
References: <c048da1c1003211012u3c0c601ci6bb64db18ca45a2a@mail.gmail.com>
Message-ID: <F5C9BB75-3E3F-4133-96C2-CBFED435AEA8@gmail.com>

On Mar 21, 2010, at 12:12 PM, Skipper Seabold wrote:
> Is there anywhere in scipy that works on structured arrays
> dynamically?  If not, is this a good way to determine if a structured/
> rec array was given to a function or is there a better/easier way?
> 
> import numpy as np
> 
> def isstructarray(X):
>    """
>    Returns a bool indicating if X is a structured or recarray.
> 
>    Parameters
>    -----------
>    X : ndarray or subclass
> 
>    Returns
>    -------
>    True or False, indicating if X is a structured or recarray.
>    """
>    if X.__class__ is np.recarray or (isinstance(X, np.ndarray) and\
>            X.dtype.names):
>        return True
>    else:
>        return False


Meh. As long as dtype.names is not None, you have a structured array (recarrays are by definition structured arrays). Looks like a function that doesn't really do anything...
-1

From vincent at vincentdavis.net  Sun Mar 21 21:52:20 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sun, 21 Mar 2010 19:52:20 -0600
Subject: [SciPy-User] accessing a set of columns from a recarray
Message-ID: <77e831101003211852n109c26e7q1f5b289aa1cc36de@mail.gmail.com>

I have a few things I am trying to understand.
I have a record array and and I list of columns for which I would like to
get the row means. My current solution is to iterate though my list of
column names and make a new "normal" array. I would like to do something
like np.mean(A['x','y','z'])
It would be great to append the results as an additional column to the
array, but I think I have seen how to do this.

Next, to make it a little more interesting I have a column that has values
0-14. I would like to get the mean of a row conditional on the value of this
column. Ideally I get a new array with columns 0-14 with missing values
where needed.

I hope this is clear, please ask for clarification if not.


  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100321/bda896ee/attachment.html>

From vincent at vincentdavis.net  Sun Mar 21 22:20:32 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sun, 21 Mar 2010 20:20:32 -0600
Subject: [SciPy-User] accessing a set of columns from a recarray
In-Reply-To: <77e831101003211852n109c26e7q1f5b289aa1cc36de@mail.gmail.com>
References: <77e831101003211852n109c26e7q1f5b289aa1cc36de@mail.gmail.com>
Message-ID: <77e831101003211920h435efae2n505c1c7175b1a694@mail.gmail.com>

To many distractions let me try to write that a little better.

I have a record array and a list of columns for which I would like to get
the row means. My current solution is to iterate though the list of column
names and make a new "normal" array. then calculate the row means. I would
like to do something like np.mean(A['x','y','z']) where x,y,z are the tiles
of the columns

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sun, Mar 21, 2010 at 7:52 PM, Vincent Davis <vincent at vincentdavis.net>wrote:

> I have a few things I am trying to understand.
> I have a record array and and I list of columns for which I would like to
> get the row means. My current solution is to iterate though my list of
> column names and make a new "normal" array. I would like to do something
> like np.mean(A['x','y','z'])
> It would be great to append the results as an additional column to the
> array, but I think I have seen how to do this.
>
> Next, to make it a little more interesting I have a column that has values
> 0-14. I would like to get the mean of a row conditional on the value of this
> column. Ideally I get a new array with columns 0-14 with missing values
> where needed.
>
> I hope this is clear, please ask for clarification if not.
>
>
>   *Vincent Davis
> 720-301-3003 *
> vincent at vincentdavis.net
>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100321/b9fcb5da/attachment.html>

From jsseabold at gmail.com  Sun Mar 21 22:24:12 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sun, 21 Mar 2010 22:24:12 -0400
Subject: [SciPy-User] determining if structured/rec array
In-Reply-To: <F5C9BB75-3E3F-4133-96C2-CBFED435AEA8@gmail.com>
References: <c048da1c1003211012u3c0c601ci6bb64db18ca45a2a@mail.gmail.com> 
	<F5C9BB75-3E3F-4133-96C2-CBFED435AEA8@gmail.com>
Message-ID: <c048da1c1003211924o186801d4i887e5c5165da9ee9@mail.gmail.com>

On Sun, Mar 21, 2010 at 9:43 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
> On Mar 21, 2010, at 12:12 PM, Skipper Seabold wrote:
>> Is there anywhere in scipy that works on structured arrays
>> dynamically? ?If not, is this a good way to determine if a structured/
>> rec array was given to a function or is there a better/easier way?
>>
>> import numpy as np
>>
>> def isstructarray(X):
>> ? ?"""
>> ? ?Returns a bool indicating if X is a structured or recarray.
>>
>> ? ?Parameters
>> ? ?-----------
>> ? ?X : ndarray or subclass
>>
>> ? ?Returns
>> ? ?-------
>> ? ?True or False, indicating if X is a structured or recarray.
>> ? ?"""
>> ? ?if X.__class__ is np.recarray or (isinstance(X, np.ndarray) and\
>> ? ? ? ? ? ?X.dtype.names):
>> ? ? ? ?return True
>> ? ?else:
>> ? ? ? ?return False
>
>
> Meh. As long as dtype.names is not None, you have a structured array (recarrays are by definition structured arrays). Looks like a function that doesn't really do anything...
> -1

Hmm, so that's definitely simpler.  I thought there was a corner case
that returned an error, so I used this if statement, but I wrote this
a while ago and never bothered messing with it since.

Thanks,

Skipper


From jsseabold at gmail.com  Sun Mar 21 22:33:06 2010
From: jsseabold at gmail.com (Skipper Seabold)
Date: Sun, 21 Mar 2010 22:33:06 -0400
Subject: [SciPy-User] accessing a set of columns from a recarray
In-Reply-To: <77e831101003211920h435efae2n505c1c7175b1a694@mail.gmail.com>
References: <77e831101003211852n109c26e7q1f5b289aa1cc36de@mail.gmail.com> 
	<77e831101003211920h435efae2n505c1c7175b1a694@mail.gmail.com>
Message-ID: <c048da1c1003211933m7fe4248ap4a252694737b63b@mail.gmail.com>

On Sun, Mar 21, 2010 at 10:20 PM, Vincent Davis
<vincent at vincentdavis.net> wrote:
>
> To many distractions let me try to write that a little better.
> I have a record array and a list of columns for which I would like to get the row means. My current solution is to iterate though the list of column names and make a new "normal" array. then calculate the row means. I would like to do something like np.mean(A['x','y','z']) where x,y,z are the tiles of the columns
>

If you have a rec array

Y = np.rec.array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0), (7.0, 8.0, 9.0)],
      dtype=[('var1', '<f8'), ('var2', '<f8'), ('var3', '<f8')])

You can access the rows like,

Y[['var1','var2','var3']]

Note the list within [].

If you want a "normal" array, I like this way that Pierre recently
pointed out.  3 is the number of columns, and it fills in the number
of rows.

Y[['var1','var2','var3']].view((float,3))

note the tuple for the view, if they're all floats.  Taking a view
might not work if var# have different types, like ints and floats.

If you want the mean of the rows (mean over the columns axis = 1)

Y[['var1','var2','var3']].view((float,3)).mean(1)

Some shortcuts.

Y[list(Y.dtype.names)].view((float,len(Y.dtype))).mean(1)

Also, for now, the columns will given back to you in the order they're
in in the array no matter which way you ask for them.  A patch has
been submitted for returning the order you ask that I hope gets picked
up...

Skipper


From vincent at vincentdavis.net  Mon Mar 22 00:00:27 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sun, 21 Mar 2010 22:00:27 -0600
Subject: [SciPy-User] accessing a set of columns from a recarray
In-Reply-To: <c048da1c1003211933m7fe4248ap4a252694737b63b@mail.gmail.com>
References: <77e831101003211852n109c26e7q1f5b289aa1cc36de@mail.gmail.com>
	<77e831101003211920h435efae2n505c1c7175b1a694@mail.gmail.com>
	<c048da1c1003211933m7fe4248ap4a252694737b63b@mail.gmail.com>
Message-ID: <77e831101003212100i7aeaacc6x88d0cca739e0b38a@mail.gmail.com>

Thanks, Thats what I needed

*Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sun, Mar 21, 2010 at 8:33 PM, Skipper Seabold <jsseabold at gmail.com>wrote:

> On Sun, Mar 21, 2010 at 10:20 PM, Vincent Davis
> <vincent at vincentdavis.net> wrote:
> >
> > To many distractions let me try to write that a little better.
> > I have a record array and a list of columns for which I would like to get
> the row means. My current solution is to iterate though the list of column
> names and make a new "normal" array. then calculate the row means. I would
> like to do something like np.mean(A['x','y','z']) where x,y,z are the tiles
> of the columns
> >
>
> If you have a rec array
>
> Y = np.rec.array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0), (7.0, 8.0, 9.0)],
>      dtype=[('var1', '<f8'), ('var2', '<f8'), ('var3', '<f8')])
>
> You can access the rows like,
>
> Y[['var1','var2','var3']]
>
> Note the list within [].
>
> If you want a "normal" array, I like this way that Pierre recently
> pointed out.  3 is the number of columns, and it fills in the number
> of rows.
>
> Y[['var1','var2','var3']].view((float,3))
>
> note the tuple for the view, if they're all floats.  Taking a view
> might not work if var# have different types, like ints and floats.
>
> If you want the mean of the rows (mean over the columns axis = 1)
>
> Y[['var1','var2','var3']].view((float,3)).mean(1)
>
> Some shortcuts.
>
> Y[list(Y.dtype.names)].view((float,len(Y.dtype))).mean(1)
>
> Also, for now, the columns will given back to you in the order they're
> in in the array no matter which way you ask for them.  A patch has
> been submitted for returning the order you ask that I hope gets picked
> up...
>
> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100321/85ad5f87/attachment.html>

From vincent at vincentdavis.net  Mon Mar 22 00:18:03 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sun, 21 Mar 2010 22:18:03 -0600
Subject: [SciPy-User] accessing a set of columns from a recarray
In-Reply-To: <c048da1c1003211933m7fe4248ap4a252694737b63b@mail.gmail.com>
References: <77e831101003211852n109c26e7q1f5b289aa1cc36de@mail.gmail.com>
	<77e831101003211920h435efae2n505c1c7175b1a694@mail.gmail.com>
	<c048da1c1003211933m7fe4248ap4a252694737b63b@mail.gmail.com>
Message-ID: <77e831101003212118g371c3ae9x31ea530056792fd5@mail.gmail.com>

I am a little puzzeled by this
>>> data2[list(typeallnames)][0][0]
171.0
>>> data2[list(typeallnames)][0][0]=0
>>> data2[list(typeallnames)][0][0]
171.0

How do I change the value?

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sun, Mar 21, 2010 at 8:33 PM, Skipper Seabold <jsseabold at gmail.com>wrote:

> On Sun, Mar 21, 2010 at 10:20 PM, Vincent Davis
> <vincent at vincentdavis.net> wrote:
> >
> > To many distractions let me try to write that a little better.
> > I have a record array and a list of columns for which I would like to get
> the row means. My current solution is to iterate though the list of column
> names and make a new "normal" array. then calculate the row means. I would
> like to do something like np.mean(A['x','y','z']) where x,y,z are the tiles
> of the columns
> >
>
> If you have a rec array
>
> Y = np.rec.array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0), (7.0, 8.0, 9.0)],
>      dtype=[('var1', '<f8'), ('var2', '<f8'), ('var3', '<f8')])
>
> You can access the rows like,
>
> Y[['var1','var2','var3']]
>
> Note the list within [].
>
> If you want a "normal" array, I like this way that Pierre recently
> pointed out.  3 is the number of columns, and it fills in the number
> of rows.
>
> Y[['var1','var2','var3']].view((float,3))
>
> note the tuple for the view, if they're all floats.  Taking a view
> might not work if var# have different types, like ints and floats.
>
> If you want the mean of the rows (mean over the columns axis = 1)
>
> Y[['var1','var2','var3']].view((float,3)).mean(1)
>
> Some shortcuts.
>
> Y[list(Y.dtype.names)].view((float,len(Y.dtype))).mean(1)
>
> Also, for now, the columns will given back to you in the order they're
> in in the array no matter which way you ask for them.  A patch has
> been submitted for returning the order you ask that I hope gets picked
> up...
>
> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100321/04ae1abf/attachment.html>

From vincent at vincentdavis.net  Mon Mar 22 01:09:22 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Sun, 21 Mar 2010 23:09:22 -0600
Subject: [SciPy-User] accessing a set of columns from a recarray
In-Reply-To: <c048da1c1003211933m7fe4248ap4a252694737b63b@mail.gmail.com>
References: <77e831101003211852n109c26e7q1f5b289aa1cc36de@mail.gmail.com>
	<77e831101003211920h435efae2n505c1c7175b1a694@mail.gmail.com>
	<c048da1c1003211933m7fe4248ap4a252694737b63b@mail.gmail.com>
Message-ID: <77e831101003212209m6b25acd4l8451fbe6dbcf3d0e@mail.gmail.com>

I posted the question on ask.scipy.org, I just posted your response, If you
are using ask.scipy feel feel to post this reply and I will vote it the
correct answer ,if you care :)
http://ask.scipy.org/en/topic/16-accessing-a-list-of-columns-in-a-recarray#reply-48

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sun, Mar 21, 2010 at 8:33 PM, Skipper Seabold <jsseabold at gmail.com>wrote:

> On Sun, Mar 21, 2010 at 10:20 PM, Vincent Davis
> <vincent at vincentdavis.net> wrote:
> >
> > To many distractions let me try to write that a little better.
> > I have a record array and a list of columns for which I would like to get
> the row means. My current solution is to iterate though the list of column
> names and make a new "normal" array. then calculate the row means. I would
> like to do something like np.mean(A['x','y','z']) where x,y,z are the tiles
> of the columns
> >
>
> If you have a rec array
>
> Y = np.rec.array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0), (7.0, 8.0, 9.0)],
>      dtype=[('var1', '<f8'), ('var2', '<f8'), ('var3', '<f8')])
>
> You can access the rows like,
>
> Y[['var1','var2','var3']]
>
> Note the list within [].
>
> If you want a "normal" array, I like this way that Pierre recently
> pointed out.  3 is the number of columns, and it fills in the number
> of rows.
>
> Y[['var1','var2','var3']].view((float,3))
>
> note the tuple for the view, if they're all floats.  Taking a view
> might not work if var# have different types, like ints and floats.
>
> If you want the mean of the rows (mean over the columns axis = 1)
>
> Y[['var1','var2','var3']].view((float,3)).mean(1)
>
> Some shortcuts.
>
> Y[list(Y.dtype.names)].view((float,len(Y.dtype))).mean(1)
>
> Also, for now, the columns will given back to you in the order they're
> in in the array no matter which way you ask for them.  A patch has
> been submitted for returning the order you ask that I hope gets picked
> up...
>
> Skipper
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100321/2cfdfbcb/attachment.html>

From J.Anderson at hull.ac.uk  Mon Mar 22 04:05:29 2010
From: J.Anderson at hull.ac.uk (Joseph Anderson)
Date: Mon, 22 Mar 2010 08:05:29 -0000
Subject: [SciPy-User] EPD and fast FFT?
References: <c7009a551003080044w656f4fc2o66fe9a417fc42247@mail.gmail.com>
	<c7009a551003082129g69d89a31na4eb595743ec5156@mail.gmail.com>
Message-ID: <D0C13D29BDA2C84A9F9DC7BD978B67E76FC113@dougal.scar.hull.ac.uk>

Hello All,

Anyone have any hints on speeding up the FFTs on EPD?

I've just started to do some larger 1-d convolutions, and am finding the crunching to be somewhat slow for my data sets (Room Impulse Responses * Audio Files).

I don't seem to think the EPD installation comes set up with FFTW... and, my first thought is that's what's needed to speed things up.

Any advice?

Thanks in advance.



My best,

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dr Joseph Anderson
Lecturer in Music

School of Arts and New Media
University of Hull, Scarborough Campus,
Scarborough, North Yorkshire, YO11 3AZ, UK

T: +44.(0)1723.357341 T: +44.(0)1723.357370 F: +44.(0)1723.350815
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 2790 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100322/9e5c07b9/attachment.bin>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100322/9e5c07b9/attachment.ksh>

From dagss at student.matnat.uio.no  Mon Mar 22 05:02:57 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 22 Mar 2010 10:02:57 +0100
Subject: [SciPy-User] EPD and fast FFT?
In-Reply-To: <D0C13D29BDA2C84A9F9DC7BD978B67E76FC113@dougal.scar.hull.ac.uk>
References: <c7009a551003080044w656f4fc2o66fe9a417fc42247@mail.gmail.com>	<c7009a551003082129g69d89a31na4eb595743ec5156@mail.gmail.com>
	<D0C13D29BDA2C84A9F9DC7BD978B67E76FC113@dougal.scar.hull.ac.uk>
Message-ID: <4BA73241.8010601@student.matnat.uio.no>

Joseph Anderson wrote:
> Hello All,
>
> Anyone have any hints on speeding up the FFTs on EPD?
>
> I've just started to do some larger 1-d convolutions, and am finding the crunching to be somewhat slow for my data sets (Room Impulse Responses * Audio Files).
>
> I don't seem to think the EPD installation comes set up with FFTW... and, my first thought is that's what's needed to speed things up.
>
> Any advice?
>   
The latest EPD comes with Intel MKL; so that's likely what you want to 
use. It uses the FFTW 3 API, so any code you write would be usable with 
either MKL (costs money) or FFTW (GPL).

Personally I'd use Cython to start wrap the FFTW 3 API... at least if 
you know some C that shouldn't take long.

Dag Sverre


From cycomanic at gmail.com  Mon Mar 22 07:59:44 2010
From: cycomanic at gmail.com (Jochen Schroeder)
Date: Mon, 22 Mar 2010 22:59:44 +1100
Subject: [SciPy-User] EPD and fast FFT?
In-Reply-To: <4BA73241.8010601@student.matnat.uio.no>
References: <c7009a551003080044w656f4fc2o66fe9a417fc42247@mail.gmail.com>
	<c7009a551003082129g69d89a31na4eb595743ec5156@mail.gmail.com>
	<D0C13D29BDA2C84A9F9DC7BD978B67E76FC113@dougal.scar.hull.ac.uk>
	<4BA73241.8010601@student.matnat.uio.no>
Message-ID: <20100322115941.GA3166@jschrod-laptop>

I've written an fftw3 wrapper using ctypes (not cython). In contrast to the
numpy and scipy fft functions it uses plans like the fftw3 c-api. However I've
tried to make it more usable from Python, i.e. types are detected automatically
etc.. You can find the fftw3 wrapper at http://launchpad.net/pyfftw.

Cheers
Jochen


On 03/22/10 10:02, Dag Sverre Seljebotn wrote:
> Joseph Anderson wrote:
> > Hello All,
> >
> > Anyone have any hints on speeding up the FFTs on EPD?
> >
> > I've just started to do some larger 1-d convolutions, and am finding the crunching to be somewhat slow for my data sets (Room Impulse Responses * Audio Files).
> >
> > I don't seem to think the EPD installation comes set up with FFTW... and, my first thought is that's what's needed to speed things up.
> >
> > Any advice?
> >   
> The latest EPD comes with Intel MKL; so that's likely what you want to 
> use. It uses the FFTW 3 API, so any code you write would be usable with 
> either MKL (costs money) or FFTW (GPL).
> 
> Personally I'd use Cython to start wrap the FFTW 3 API... at least if 
> you know some C that shouldn't take long.
> 
> Dag Sverre
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From wkerzendorf at googlemail.com  Mon Mar 22 09:15:00 2010
From: wkerzendorf at googlemail.com (Wolfgang Kerzendorf)
Date: Tue, 23 Mar 2010 00:15:00 +1100
Subject: [SciPy-User] using sunperf for compiling scipy and numpy
Message-ID: <4BA76D54.5070008@gmail.com>

Dear all,

I would like to use the sunperf libraries when compiling scipy and 
numpy. I tried using setupscons.py which seems to check from SUNPERF 
libraries, but it didnt recognize where mine are: here is a listing of 
/pkg/linux/SS12/sunstudio12.1 (thats where the sunperf library lives):

wkerzend at mosura:/home/wkerzend>ls /pkg/linux/SS12/sunstudio12.1/lib/
CCios/            libdbx_agent.so@    libsunperf.so.3@
amd64/            libfcollector.so@   libtha.so@
collector.jar@        libfsu.so@        libtha.so.1@
dbxrc@            libfsu.so.1@        locale/
debugging.so@        libfui.so@        make.rules@
er.rc@            libfui.so.1@        rw7/
libblacs_openmpi.so@    librtc.so@        sse2/
libblacs_openmpi.so.1@    libscalapack.so@    stlport4/
libcollectorAPI.so@    libscalapack.so.1@  svr4.make.rules@
libcollectorAPI.so.1@    libsunperf.so@        tools_svc_mgr@

-------------------------------

I tried to specify this directory in sites.cfg, but I still get the 
following errors:

Checking if g77 needs dummy main - MAIN__.
Checking g77 name mangling - '_', '', lower-case.
Checking g77 C compatibility runtime 
...-L/usr/lib/gcc/x86_64-redhat-linux/3.4.6 
-L/usr/lib/gcc/x86_64-redhat-linux/3.4.6 
-L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 
-L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../.. -L/lib/../lib64 
-L/usr/lib/../lib64 -lfrtbegin -lg2c -lm
Checking MKL ... Failed (could not check header(s) : check config.log in 
build/scons/scipy/integrate for more details)
Checking ATLAS ... Failed (could not check header(s) : check config.log 
in build/scons/scipy/integrate for more details)
Checking SUNPERF ... Failed (could not check symbol cblas_sgemm : check 
config.log in build/scons/scipy/integrate for more details))
Checking Generic BLAS ... yes
Checking for BLAS (Generic BLAS) ... Failed: BLAS (Generic BLAS) test 
could not be linked and run
Exception: Could not find F77 BLAS, needed for integrate package:
   File 
"/priv/manana1/wkerzend/install_dir/scipy-0.7.1/scipy/integrate/SConstruct", 
line 2:
     GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript')
   File 
"/home/wkerzend/python_coala/numscons-0.10.1-py2.6.egg/numscons/core/numpyenv.py", 
line 108:
     build_dir = '$build_dir', src_dir = '$src_dir')
   File 
"/priv/manana1/wkerzend/python_coala/numscons-0.10.1-py2.6.egg/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py", 
line 549:
     return apply(_SConscript, [self.fs,] + files, subst_kw)
   File 
"/priv/manana1/wkerzend/python_coala/numscons-0.10.1-py2.6.egg/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py", 
line 259:
     exec _file_ in call_stack[-1].globals
   File 
"/priv/manana1/wkerzend/install_dir/scipy-0.7.1/build/scons/scipy/integrate/SConscript", 
line 15:
     raise Exception("Could not find F77 BLAS, needed for integrate 
package")
error: Error while executing scons command. See above for more information.
If you think it is a problem in numscons, you can also try executing the 
scons
command with --log-level option for more detailed output of what numscons is
doing, for example --log-level=0; the lowest the level is, the more detailed
the output it.-----


--------------------------

any help is appreciated

     Wolfgang



From Deil.Christoph at googlemail.com  Mon Mar 22 10:44:41 2010
From: Deil.Christoph at googlemail.com (Christoph Deil)
Date: Mon, 22 Mar 2010 15:44:41 +0100
Subject: [SciPy-User] Request for usage examples on
	scipy.stats.rv_continuous and scipy.ndimage.grey_dilate(structure)
Message-ID: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>

Hi,

I am having some trouble figuring out how to use the following two aspects of scipy:

1) How can I  create a random distribution given by some python formula, e.g. pdf(x) = x**2 in the range x = 1 to 2.

The example given in the rv_continuous docstring doesn't work for me:
In [1]: import matplotlib.pyplot as plt
In [2]: numargs = generic.numargs
AttributeError: type object 'numpy.generic' has no attribute 'numargs'

Is it also possible to use rv_continuous for multidimensional distributions, say pdf(x,y)=x*y in the range x = 1 to 2 and y = 0 to 3?
Or for pdfs that are just given numerically on a grid (say as the result of a gaussian_kde() ), not by a python function?

It would be nice if someone could add a few examples that show rv_continuous usage to the docstring or tutorial.

2) How to use the 'structure' argument of scipy.ndimage.grey_dilate() correctly?
There are some explanations at
http://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html
but I think they are inconsistent with the implementation:

--------------------------------------------------------------
In [98]: a = zeros((7,7)); a[3,3]=1; a
Out[98]: 
array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  1.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.]])
In [100]: s = generate_binary_structure(2,2); s
Out[100]: 
array([[ True,  True,  True],
       [ True,  True,  True],
       [ True,  True,  True]], dtype=bool)

In [101]: grey_dilation(a,size=(3,3))
Out[101]: 
array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.]])

In [102]: grey_dilation(a,size=(3,3),structure=s)
Out[102]: 
array([[ 1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  2.,  2.,  2.,  1.,  1.],
       [ 1.,  1.,  2.,  2.,  2.,  1.,  1.],
       [ 1.,  1.,  2.,  2.,  2.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.],
       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.]])

In [103]: grey_dilation(a,structure=s)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

/Users/deil/<ipython console> in <module>()

/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/ndimage/morphology.pyc in grey_dilation(input, size, footprint, structure, output, mode, cval, origin)
    354             sz = footprint.shape[ii]
    355         else:
--> 356             sz = size[ii]
    357         if not sz & 1:
    358             origin[ii] -= 1

TypeError: 'NoneType' object is unsubscriptable

In [104]: grey_dilation(a,footprint=s)
Out[104]: 
array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.]])
--------------------------------------------------------------

Why doesn't  grey_dilation(a,structure=s) work?
Why doesn't grey_dilation(a,size=(3,3),structure=s) give the same result as grey_dilation(a,size=(3,3),footprint=s)?

I am running on numpy version 1.4.0 and scipy version 0.7.1

Thanks!
--Christoph





From robert.kern at gmail.com  Mon Mar 22 10:50:23 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 22 Mar 2010 09:50:23 -0500
Subject: [SciPy-User] Request for usage examples on
	scipy.stats.rv_continuous and scipy.ndimage.grey_dilate(structure)
In-Reply-To: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>
References: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>
Message-ID: <3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com>

On Mon, Mar 22, 2010 at 09:44, Christoph Deil
<Deil.Christoph at googlemail.com> wrote:
> Hi,
>
> I am having some trouble figuring out how to use the following two aspects of scipy:
>
> 1) How can I ?create a random distribution given by some python formula, e.g. pdf(x) = x**2 in the range x = 1 to 2.
>
> The example given in the rv_continuous docstring doesn't work for me:
> In [1]: import matplotlib.pyplot as plt
> In [2]: numargs = generic.numargs
> AttributeError: type object 'numpy.generic' has no attribute 'numargs'
>
> Is it also possible to use rv_continuous for multidimensional distributions, say pdf(x,y)=x*y in the range x = 1 to 2 and y = 0 to 3?
> Or for pdfs that are just given numerically on a grid (say as the result of a gaussian_kde() ), not by a python function?
>
> It would be nice if someone could add a few examples that show rv_continuous usage to the docstring or tutorial.

rv_continuous is a base class. You do not use it directly. The reason
the docstring doesn't work is because it is used as a template to
build the docstrings for the subclasses. If you want to subclass from
it to make your own distribution class, read the source of the other
distribution classes for examples.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From ralf.gommers at googlemail.com  Mon Mar 22 11:17:35 2010
From: ralf.gommers at googlemail.com (Ralf Gommers)
Date: Mon, 22 Mar 2010 23:17:35 +0800
Subject: [SciPy-User] Request for usage examples on
	scipy.stats.rv_continuous and scipy.ndimage.grey_dilate(structure)
In-Reply-To: <3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com>
References: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>
	<3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com>
Message-ID: <dde7764a1003220817kac0bc64j491a309feac89b94@mail.gmail.com>

On Mon, Mar 22, 2010 at 10:50 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Mon, Mar 22, 2010 at 09:44, Christoph Deil
> > It would be nice if someone could add a few examples that show
> rv_continuous usage to the docstring or tutorial.
>
> rv_continuous is a base class. You do not use it directly. The reason
> the docstring doesn't work is because it is used as a template to
> build the docstrings for the subclasses. If you want to subclass from
> it to make your own distribution class, read the source of the other
> distribution classes for examples.
>
> That reminds me of http://projects.scipy.org/scipy/ticket/1055. That will
enable a proper docstring for rv_continuous that explains this.

Josef, if I split up the patch into a couple of logical units, will you
accept it?

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100322/37c167fe/attachment.html>

From emmanuelle.gouillart at normalesup.org  Mon Mar 22 11:37:34 2010
From: emmanuelle.gouillart at normalesup.org (Emmanuelle Gouillart)
Date: Mon, 22 Mar 2010 16:37:34 +0100
Subject: [SciPy-User] Request for usage examples
	on	scipy.stats.rv_continuous and
	scipy.ndimage.grey_dilate(structure)
In-Reply-To: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>
References: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>
Message-ID: <20100322153734.GB7211@phare.normalesup.org>

Hi,

The documentation of ndimage.grey_dilate is much more detailed on the doc
wiki (including the use of structure, footprint etc.)
http://docs.scipy.org/scipy/docs/scipy.ndimage.morphology.grey_dilation/
(like most other functions of ndimage.morphology, the docstring has
been edited quite recently and the changes haven't made their way to the
"official" doc yet).

Cheers,

Emmanuelle

On Mon, Mar 22, 2010 at 03:44:41PM +0100, Christoph Deil wrote:
> Hi,

> I am having some trouble figuring out how to use the following two aspects of scipy:

> 1) How can I  create a random distribution given by some python formula, e.g. pdf(x) = x**2 in the range x = 1 to 2.

> The example given in the rv_continuous docstring doesn't work for me:
> In [1]: import matplotlib.pyplot as plt
> In [2]: numargs = generic.numargs
> AttributeError: type object 'numpy.generic' has no attribute 'numargs'

> Is it also possible to use rv_continuous for multidimensional distributions, say pdf(x,y)=x*y in the range x = 1 to 2 and y = 0 to 3?
> Or for pdfs that are just given numerically on a grid (say as the result of a gaussian_kde() ), not by a python function?

> It would be nice if someone could add a few examples that show rv_continuous usage to the docstring or tutorial.

> 2) How to use the 'structure' argument of scipy.ndimage.grey_dilate() correctly?
> There are some explanations at
> http://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html
> but I think they are inconsistent with the implementation:

> --------------------------------------------------------------
> In [98]: a = zeros((7,7)); a[3,3]=1; a
> Out[98]: 
> array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  1.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.]])
> In [100]: s = generate_binary_structure(2,2); s
> Out[100]: 
> array([[ True,  True,  True],
>        [ True,  True,  True],
>        [ True,  True,  True]], dtype=bool)

> In [101]: grey_dilation(a,size=(3,3))
> Out[101]: 
> array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>        [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>        [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.]])

> In [102]: grey_dilation(a,size=(3,3),structure=s)
> Out[102]: 
> array([[ 1.,  1.,  1.,  1.,  1.,  1.,  1.],
>        [ 1.,  1.,  1.,  1.,  1.,  1.,  1.],
>        [ 1.,  1.,  2.,  2.,  2.,  1.,  1.],
>        [ 1.,  1.,  2.,  2.,  2.,  1.,  1.],
>        [ 1.,  1.,  2.,  2.,  2.,  1.,  1.],
>        [ 1.,  1.,  1.,  1.,  1.,  1.,  1.],
>        [ 1.,  1.,  1.,  1.,  1.,  1.,  1.]])

> In [103]: grey_dilation(a,structure=s)
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call last)

> /Users/deil/<ipython console> in <module>()

> /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/ndimage/morphology.pyc in grey_dilation(input, size, footprint, structure, output, mode, cval, origin)
>     354             sz = footprint.shape[ii]
>     355         else:
> --> 356             sz = size[ii]
>     357         if not sz & 1:
>     358             origin[ii] -= 1

> TypeError: 'NoneType' object is unsubscriptable

> In [104]: grey_dilation(a,footprint=s)
> Out[104]: 
> array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>        [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>        [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>        [ 0.,  0.,  0.,  0.,  0.,  0.,  0.]])
> --------------------------------------------------------------

> Why doesn't  grey_dilation(a,structure=s) work?
> Why doesn't grey_dilation(a,size=(3,3),structure=s) give the same result as grey_dilation(a,size=(3,3),footprint=s)?

> I am running on numpy version 1.4.0 and scipy version 0.7.1

> Thanks!
> --Christoph



> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From zachary.pincus at yale.edu  Mon Mar 22 12:12:37 2010
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Mon, 22 Mar 2010 12:12:37 -0400
Subject: [SciPy-User] Bug in scipy.ndimage.grey_dilate(structure)
In-Reply-To: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>
References: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>
Message-ID: <4FB535D5-54E8-43C0-A34F-5BA8BA511572@yale.edu>

> 2) How to use the 'structure' argument of  
> scipy.ndimage.grey_dilate() correctly?
> There are some explanations at
> http://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html
> but I think they are inconsistent with the implementation:

Oh dear, I think this is a bug that was previously reported by Tony Yu  
months ago (along with a patch!), which I said I would open a ticket  
about and look over the patch provided, but I seem to have entirely  
forgotten about (because I'm a flake).

http://projects.scipy.org/scipy/ticket/1135

Could someone who uses grey_dilation frequently (any cellprofiler  
people?) check the proposed changes over and make sure it makes sense  
in the grey case? The tests are only for binary cases.

Zach



On Mar 22, 2010, at 10:44 AM, Christoph Deil wrote:

> Hi,
>
> I am having some trouble figuring out how to use the following two  
> aspects of scipy:
>
> 1) How can I  create a random distribution given by some python  
> formula, e.g. pdf(x) = x**2 in the range x = 1 to 2.
>
> The example given in the rv_continuous docstring doesn't work for me:
> In [1]: import matplotlib.pyplot as plt
> In [2]: numargs = generic.numargs
> AttributeError: type object 'numpy.generic' has no attribute 'numargs'
>
> Is it also possible to use rv_continuous for multidimensional  
> distributions, say pdf(x,y)=x*y in the range x = 1 to 2 and y = 0 to  
> 3?
> Or for pdfs that are just given numerically on a grid (say as the  
> result of a gaussian_kde() ), not by a python function?
>
> It would be nice if someone could add a few examples that show  
> rv_continuous usage to the docstring or tutorial.
>
> 2) How to use the 'structure' argument of  
> scipy.ndimage.grey_dilate() correctly?
> There are some explanations at
> http://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html
> but I think they are inconsistent with the implementation:
>
> --------------------------------------------------------------
> In [98]: a = zeros((7,7)); a[3,3]=1; a
> Out[98]:
> array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  1.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.]])
> In [100]: s = generate_binary_structure(2,2); s
> Out[100]:
> array([[ True,  True,  True],
>       [ True,  True,  True],
>       [ True,  True,  True]], dtype=bool)
>
> In [101]: grey_dilation(a,size=(3,3))
> Out[101]:
> array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.]])
>
> In [102]: grey_dilation(a,size=(3,3),structure=s)
> Out[102]:
> array([[ 1.,  1.,  1.,  1.,  1.,  1.,  1.],
>       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.],
>       [ 1.,  1.,  2.,  2.,  2.,  1.,  1.],
>       [ 1.,  1.,  2.,  2.,  2.,  1.,  1.],
>       [ 1.,  1.,  2.,  2.,  2.,  1.,  1.],
>       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.],
>       [ 1.,  1.,  1.,  1.,  1.,  1.,  1.]])
>
> In [103]: grey_dilation(a,structure=s)
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent  
> call last)
>
> /Users/deil/<ipython console> in <module>()
>
> /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/site-packages/scipy/ndimage/morphology.pyc in  
> grey_dilation(input, size, footprint, structure, output, mode, cval,  
> origin)
>    354             sz = footprint.shape[ii]
>    355         else:
> --> 356             sz = size[ii]
>    357         if not sz & 1:
>    358             origin[ii] -= 1
>
> TypeError: 'NoneType' object is unsubscriptable
>
> In [104]: grey_dilation(a,footprint=s)
> Out[104]:
> array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>       [ 0.,  0.,  1.,  1.,  1.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.],
>       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.]])
> --------------------------------------------------------------
>
> Why doesn't  grey_dilation(a,structure=s) work?
> Why doesn't grey_dilation(a,size=(3,3),structure=s) give the same  
> result as grey_dilation(a,size=(3,3),footprint=s)?
>
> I am running on numpy version 1.4.0 and scipy version 0.7.1
>
> Thanks!
> --Christoph
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From Deil.Christoph at googlemail.com  Mon Mar 22 12:58:47 2010
From: Deil.Christoph at googlemail.com (Christoph Deil)
Date: Mon, 22 Mar 2010 17:58:47 +0100
Subject: [SciPy-User] Request for usage examples on
	scipy.stats.rv_continuous and scipy.ndimage.grey_dilate(structure)
In-Reply-To: <3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com>
References: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>
	<3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com>
Message-ID: <088968F9-41CA-4D99-8E9D-86FB2F8DB56F@googlemail.com>

Dear Robert,

thanks for the tip. I tried understanding the examples in scipy/stats/distributions.py, but being a python / scipy newbie I find the mechanism hard to understand and couldn't implement the simple examples I suggested below.

Maybe it would be possible to add an example to the tutorial? At http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/stats.rst/#stats there is an example on how to use rv_discrete, but none on how to use rv_continuous.

Would it be possible to add a convenience function to scipy.stats that makes it easy to construct a distribution from a function:
>>> p = lambda x: x**2
>>> pdist = scipy.stats.rv_continuous_from_function(pdf=p, lim=[0,2]) # a suggestion, doesn't exist at the moment
>>> samples = pdist.rvs(size=10)

I would guess that getting random numbers from a user defined distribution function is such a common usage that it would be nice (at least for newbies like me :-) to being able to do it from the command line, without having to derive a class.

Cheers, --Christoph

On Mar 22, 2010, at 3:50 PM, Robert Kern wrote:

> On Mon, Mar 22, 2010 at 09:44, Christoph Deil
> <Deil.Christoph at googlemail.com> wrote:
>> Hi,
>> 
>> I am having some trouble figuring out how to use the following two aspects of scipy:
>> 
>> 1) How can I  create a random distribution given by some python formula, e.g. pdf(x) = x**2 in the range x = 1 to 2.
>> 
>> The example given in the rv_continuous docstring doesn't work for me:
>> In [1]: import matplotlib.pyplot as plt
>> In [2]: numargs = generic.numargs
>> AttributeError: type object 'numpy.generic' has no attribute 'numargs'
>> 
>> Is it also possible to use rv_continuous for multidimensional distributions, say pdf(x,y)=x*y in the range x = 1 to 2 and y = 0 to 3?
>> Or for pdfs that are just given numerically on a grid (say as the result of a gaussian_kde() ), not by a python function?
>> 
>> It would be nice if someone could add a few examples that show rv_continuous usage to the docstring or tutorial.
> 
> rv_continuous is a base class. You do not use it directly. The reason
> the docstring doesn't work is because it is used as a template to
> build the docstrings for the subclasses. If you want to subclass from
> it to make your own distribution class, read the source of the other
> distribution classes for examples.
> 
> -- 
> Robert Kern
> 
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From robert.kern at gmail.com  Mon Mar 22 13:13:56 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 22 Mar 2010 12:13:56 -0500
Subject: [SciPy-User] Request for usage examples on
	scipy.stats.rv_continuous and scipy.ndimage.grey_dilate(structure)
In-Reply-To: <088968F9-41CA-4D99-8E9D-86FB2F8DB56F@googlemail.com>
References: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com> 
	<3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com> 
	<088968F9-41CA-4D99-8E9D-86FB2F8DB56F@googlemail.com>
Message-ID: <3d375d731003221013p50c81bdekc750060e10436cf@mail.gmail.com>

On Mon, Mar 22, 2010 at 11:58, Christoph Deil
<Deil.Christoph at googlemail.com> wrote:
> Dear Robert,
>
> thanks for the tip. I tried understanding the examples in scipy/stats/distributions.py, but being a python / scipy newbie I find the mechanism hard to understand and couldn't implement the simple examples I suggested below.

What confused you?

> Maybe it would be possible to add an example to the tutorial? At http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/stats.rst/#stats there is an example on how to use rv_discrete, but none on how to use rv_continuous.
>
> Would it be possible to add a convenience function to scipy.stats that makes it easy to construct a distribution from a function:
>>>> p = lambda x: x**2
>>>> pdist = scipy.stats.rv_continuous_from_function(pdf=p, lim=[0,2]) # a suggestion, doesn't exist at the moment
>>>> samples = pdist.rvs(size=10)

class x2_gen(rv_continuous):
    def _pdf(self, x):
        return x * x * 0.375

x2 = x2_gen(a=0.0, b=2.0, name='x2')

> I would guess that getting random numbers from a user defined distribution function is such a common usage that it would be nice (at least for newbies like me :-) to being able to do it from the command line, without having to derive a class.

It's not particularly common, no.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From kwmsmith at gmail.com  Mon Mar 22 13:20:05 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 22 Mar 2010 11:20:05 -0600
Subject: [SciPy-User] Bug in ndimage.map_coordinates with mode='wrap' ?
Message-ID: <c266a7931003221020j272337abn51a323b1653699d3@mail.gmail.com>

Hi,

Testing the example code in ndimage.map_coordinate's docstring, I
can't get things to work with mode='wrap'.  What am I doing wrong?

In [31]: a
Out[31]:
array([[  0.,   1.,   2.],
       [  3.,   4.,   5.],
       [  6.,   7.,   8.],
       [  9.,  10.,  11.]])

In [32]: ndimage.map_coordinates(a, [range(5), [0]*5], order=1,
mode='wrap')  # should be 0, 3, 6, 9, 0 -- right?
Out[32]: array([ 0.,  3.,  6.,  9.,  3.])

In [33]: ndimage.map_coordinates(a, [[0]*4, range(4)], order=1,
mode='wrap')  # should be 0, 1, 2, 0 -- right?
Out[33]: array([ 0.,  1.,  2.,  1.])

Here's the output when extending the sampling range:

In [36]: ndimage.map_coordinates(a, [range(10), [0]*10], order=1,
mode='wrap')  # should be 0, 3, 6, 9, 0, 3, 6, 9, ...
Out[36]: array([ 0.,  3.,  6.,  9.,  3.,  6.,  0.,  3.,  6.,  0.])

In [37]: ndimage.map_coordinates(a, [[0]*8, range(8)], order=1, mode='wrap')
Out[37]: array([ 0.,  1.,  2.,  1.,  0.,  1.,  0.,  1.])


If it's a bug, where can I file a report, and what can I do to help
fix it?  Looks like the wrapping code is in a compiled extension
module -- I'll take a look.

Kurt


From kwmsmith at gmail.com  Mon Mar 22 13:24:03 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 22 Mar 2010 11:24:03 -0600
Subject: [SciPy-User] Bug in ndimage.map_coordinates with mode='wrap' ?
In-Reply-To: <c266a7931003221020j272337abn51a323b1653699d3@mail.gmail.com>
References: <c266a7931003221020j272337abn51a323b1653699d3@mail.gmail.com>
Message-ID: <c266a7931003221024u1a1d8209k974e2d644ab27203@mail.gmail.com>

On Mon, Mar 22, 2010 at 11:20 AM, Kurt Smith <kwmsmith at gmail.com> wrote:
> Hi,
>
> Testing the example code in ndimage.map_coordinate's docstring, I
> can't get things to work with mode='wrap'. ?What am I doing wrong?
>
> In [31]: a
> Out[31]:
> array([[ ?0., ? 1., ? 2.],
> ? ? ? [ ?3., ? 4., ? 5.],
> ? ? ? [ ?6., ? 7., ? 8.],
> ? ? ? [ ?9., ?10., ?11.]])
>
> In [32]: ndimage.map_coordinates(a, [range(5), [0]*5], order=1,
> mode='wrap') ?# should be 0, 3, 6, 9, 0 -- right?
> Out[32]: array([ 0., ?3., ?6., ?9., ?3.])
>
> In [33]: ndimage.map_coordinates(a, [[0]*4, range(4)], order=1,
> mode='wrap') ?# should be 0, 1, 2, 0 -- right?
> Out[33]: array([ 0., ?1., ?2., ?1.])
>
> Here's the output when extending the sampling range:
>
> In [36]: ndimage.map_coordinates(a, [range(10), [0]*10], order=1,
> mode='wrap') ?# should be 0, 3, 6, 9, 0, 3, 6, 9, ...
> Out[36]: array([ 0., ?3., ?6., ?9., ?3., ?6., ?0., ?3., ?6., ?0.])
>
> In [37]: ndimage.map_coordinates(a, [[0]*8, range(8)], order=1, mode='wrap')
> Out[37]: array([ 0., ?1., ?2., ?1., ?0., ?1., ?0., ?1.])
>
>
> If it's a bug, where can I file a report, and what can I do to help
> fix it? ?Looks like the wrapping code is in a compiled extension
> module -- I'll take a look.

I forgot to include:

In [39]: sp.version.version
Out[39]: '0.8.0.dev6120'


From Deil.Christoph at googlemail.com  Mon Mar 22 13:47:42 2010
From: Deil.Christoph at googlemail.com (Christoph Deil)
Date: Mon, 22 Mar 2010 18:47:42 +0100
Subject: [SciPy-User] Request for usage examples on
	scipy.stats.rv_continuous and scipy.ndimage.grey_dilate(structure)
In-Reply-To: <3d375d731003221013p50c81bdekc750060e10436cf@mail.gmail.com>
References: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com>
	<3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com>
	<088968F9-41CA-4D99-8E9D-86FB2F8DB56F@googlemail.com>
	<3d375d731003221013p50c81bdekc750060e10436cf@mail.gmail.com>
Message-ID: <21BB2F85-48A0-446D-A675-36FDB71A07E6@googlemail.com>

On Mar 22, 2010, at 6:13 PM, Robert Kern wrote:

> On Mon, Mar 22, 2010 at 11:58, Christoph Deil
> <Deil.Christoph at googlemail.com> wrote:
>> Dear Robert,
>> 
>> thanks for the tip. I tried understanding the examples in scipy/stats/distributions.py, but being a python / scipy newbie I find the mechanism hard to understand and couldn't implement the simple examples I suggested below.
> 
> What confused you?

I didn't know how to specify the limits. The rv_continuous docstring says:

Constructor information:
Definition:       scipy.stats.rv_continuous(self, momtype=1, a=None, b=None, xa=-10.0, xb=10.0, xtol=1e-14, badvalue=None, name=None, longname=None, shapes=None, extradoc=None)

The meaning of the parameters a, b, xa, xb doesn't seem to be documented. Now that I had a look at the source code it's obvious.

> 
>> Maybe it would be possible to add an example to the tutorial? At http://docs.scipy.org/scipy/docs/scipy-docs/tutorial/stats.rst/#stats there is an example on how to use rv_discrete, but none on how to use rv_continuous.
>> 
>> Would it be possible to add a convenience function to scipy.stats that makes it easy to construct a distribution from a function:
>>>>> p = lambda x: x**2
>>>>> pdist = scipy.stats.rv_continuous_from_function(pdf=p, lim=[0,2]) # a suggestion, doesn't exist at the moment
>>>>> samples = pdist.rvs(size=10)
> 
> class x2_gen(rv_continuous):
>    def _pdf(self, x):
>        return x * x * 0.375
> 
> x2 = x2_gen(a=0.0, b=2.0, name='x2')

Thanks! This works.

Multidimensional correlated parameter distributions like pdf(x,y) = x*y cannot be implemented using rv_continuous, right?
If yes, is there a python module to work with correlated multidimensional pdfs? 

> 
>> I would guess that getting random numbers from a user defined distribution function is such a common usage that it would be nice (at least for newbies like me :-) to being able to do it from the command line, without having to derive a class.
> 
> It's not particularly common, no.
> 
> -- 
> Robert Kern
> 
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From robert.kern at gmail.com  Mon Mar 22 13:51:11 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 22 Mar 2010 12:51:11 -0500
Subject: [SciPy-User] Request for usage examples on
	scipy.stats.rv_continuous and scipy.ndimage.grey_dilate(structure)
In-Reply-To: <21BB2F85-48A0-446D-A675-36FDB71A07E6@googlemail.com>
References: <6C22962B-98C8-4BA0-B1F1-E0ED878D653E@googlemail.com> 
	<3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com> 
	<088968F9-41CA-4D99-8E9D-86FB2F8DB56F@googlemail.com>
	<3d375d731003221013p50c81bdekc750060e10436cf@mail.gmail.com> 
	<21BB2F85-48A0-446D-A675-36FDB71A07E6@googlemail.com>
Message-ID: <3d375d731003221051t4259e1a4j651047185b358bef@mail.gmail.com>

On Mon, Mar 22, 2010 at 12:47, Christoph Deil
<Deil.Christoph at googlemail.com> wrote:

> Multidimensional correlated parameter distributions like pdf(x,y) = x*y cannot be implemented using rv_continuous, right?

Correct; rv_continuous does not handle multidimensional distributions.

> If yes, is there a python module to work with correlated multidimensional pdfs?

I don't think there are any frameworks out there along the lines of
rv_continuous. There is code for sampling certain multivariate
distributions in numpy.random. If you need to sample from an arbitrary
multidimensional distribution, you will need to look at PyMC for doing
Markov-chain Monte Carlo.

  http://code.google.com/p/pymc/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From d.l.goldsmith at gmail.com  Mon Mar 22 15:08:34 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Mon, 22 Mar 2010 12:08:34 -0700
Subject: [SciPy-User] Opportunity to "plug" scipy
Message-ID: <4BA7C032.9040806@gmail.com>

Hi, folks.  In trying to drum up funding for the doc project, one of the 
orgs I contacted - The Society for Imaging Informatics in Medicine - 
invited us to present at the SIIM 2010 open source plugfest at their 
Annual Meeting in June.  If people are interested, they should probably 
coordinate efforts, i.e., assemble one team and one presentation/booth. 
  Anyway, here's the URL:

http://www.siimweb.org/open

and here's the contact info:

Anna Marie Mason, MS,CAE
Executive Director
Society for Imaging Informatics in Medicine (SIIM)
19440 Golf Vista Plaza, Suite 330
Leesburg, VA 20176
703-723-0432 x 306
amason at siimweb.org

"Tell her David Goldsmith sent you," as they say.  (I'll email her 
separately to let her know that someones may be emailing her separately 
about this.)

DG


From gael.varoquaux at normalesup.org  Mon Mar 22 15:18:42 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 22 Mar 2010 20:18:42 +0100
Subject: [SciPy-User] Opportunity to "plug" scipy
In-Reply-To: <4BA7C032.9040806@gmail.com>
References: <4BA7C032.9040806@gmail.com>
Message-ID: <20100322191842.GJ956@phare.normalesup.org>

On Mon, Mar 22, 2010 at 12:08:34PM -0700, David Goldsmith wrote:
> Hi, folks.  In trying to drum up funding for the doc project, one of the 
> orgs I contacted - The Society for Imaging Informatics in Medicine - 
> invited us to present at the SIIM 2010 open source plugfest at their 
> Annual Meeting in June.  

I can't help, but if you are interested in usecases, there is a growing
community around Python in neuroimaging, http://nipy.sourceforge.net/
making use of numpy and scipy.

Ga?l


From d.l.goldsmith at gmail.com  Mon Mar 22 16:19:43 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Mon, 22 Mar 2010 13:19:43 -0700
Subject: [SciPy-User] Opportunity to "plug" scipy
In-Reply-To: <20100322191842.GJ956@phare.normalesup.org>
References: <4BA7C032.9040806@gmail.com>
	<20100322191842.GJ956@phare.normalesup.org>
Message-ID: <4BA7D0DF.9040709@gmail.com>

On 3/22/2010 12:18 PM, Gael Varoquaux wrote:
> On Mon, Mar 22, 2010 at 12:08:34PM -0700, David Goldsmith wrote:
>    
>> Hi, folks.  In trying to drum up funding for the doc project, one of the
>> orgs I contacted - The Society for Imaging Informatics in Medicine -
>> invited us to present at the SIIM 2010 open source plugfest at their
>> Annual Meeting in June.
>>      
> I can't help, but if you are interested in usecases, there is a growing
> community around Python in neuroimaging, http://nipy.sourceforge.net/
> making use of numpy and scipy.
>
> Ga?l
>    
Good to know.  (Unfortunately, I'm too distant from the field to feel 
comfortable being a presenter - though it occurs to me that I could 
certainly coordinate a team and/or be the POC between such a team and 
SIIM - but hopefully your observation means that a team will come 
forward; do you happen to know how well said "growing community" is 
represented on this list?  Should I cross-post on the numpy list?)

DG


From gael.varoquaux at normalesup.org  Mon Mar 22 17:12:49 2010
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 22 Mar 2010 22:12:49 +0100
Subject: [SciPy-User] Opportunity to "plug" scipy
In-Reply-To: <4BA7D0DF.9040709@gmail.com>
References: <4BA7C032.9040806@gmail.com>
	<20100322191842.GJ956@phare.normalesup.org>
	<4BA7D0DF.9040709@gmail.com>
Message-ID: <20100322211249.GB29507@phare.normalesup.org>

On Mon, Mar 22, 2010 at 01:19:43PM -0700, David Goldsmith wrote:
> Good to know.  (Unfortunately, I'm too distant from the field to feel 
> comfortable being a presenter - though it occurs to me that I could 
> certainly coordinate a team and/or be the POC between such a team and 
> SIIM - but hopefully your observation means that a team will come 
> forward; do you happen to know how well said "growing community" is 
> represented on this list? 

It's a bunch of overworked researchers trying to develop quality software
that their scientific community can more or less apprehend.

Some of them do follow this mailing list, but I won't give them up, as I
understand that they might not have much time.

Ga?l


From tatiana.alchueyr at gmail.com  Mon Mar 22 17:20:54 2010
From: tatiana.alchueyr at gmail.com (Tatiana Al-Chueyr)
Date: Mon, 22 Mar 2010 18:20:54 -0300
Subject: [SciPy-User] Opportunity to "plug" scipy
In-Reply-To: <20100322211249.GB29507@phare.normalesup.org>
References: <4BA7C032.9040806@gmail.com>
	<20100322191842.GJ956@phare.normalesup.org> 
	<4BA7D0DF.9040709@gmail.com>
	<20100322211249.GB29507@phare.normalesup.org>
Message-ID: <e4b8e4a61003221420r1986d7cbt6c6f206be58d413d@mail.gmail.com>

In addition to NIPY,  InVesalius [1] could be presented....

InVesalius is an open source 3D medical imaging reconstruction
software. It was developed using Python and NumPy, among other open
source technologies.

For the last 6 years I've been working on this project, with medical
imaging and Python... It would be a pleasure to contribute to the
presentation.

Kind regards,

Tatiana

[1] http://svn.softwarepublico.gov.br/trac/invesalius


On 22 March 2010 18:12, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:
> On Mon, Mar 22, 2010 at 01:19:43PM -0700, David Goldsmith wrote:
>> Good to know. ?(Unfortunately, I'm too distant from the field to feel
>> comfortable being a presenter - though it occurs to me that I could
>> certainly coordinate a team and/or be the POC between such a team and
>> SIIM - but hopefully your observation means that a team will come
>> forward; do you happen to know how well said "growing community" is
>> represented on this list?
>
> It's a bunch of overworked researchers trying to develop quality software
> that their scientific community can more or less apprehend.
>
> Some of them do follow this mailing list, but I won't give them up, as I
> understand that they might not have much time.
>
> Ga?l
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Tatiana Al-Chueyr


From nico.schloemer at gmail.com  Mon Mar 22 20:17:19 2010
From: nico.schloemer at gmail.com (=?ISO-8859-1?Q?Nico_Schl=F6mer?=)
Date: Tue, 23 Mar 2010 01:17:19 +0100
Subject: [SciPy-User] 2d convolution
In-Reply-To: <1cd32cbb1003201352g14bf6bc4tf018d7e5bcde07fb@mail.gmail.com>
References: <d933725a1003200451u1996c66escb68316201adc266@mail.gmail.com>
	<1cd32cbb1003200549m75a86fbelea959ad6696dc526@mail.gmail.com>
	<d933725a1003200907u167a7b40p8cbbca12686ce32b@mail.gmail.com>
	<1cd32cbb1003201352g14bf6bc4tf018d7e5bcde07fb@mail.gmail.com>
Message-ID: <d933725a1003221717s4bae9790ndf825fc39ae141df@mail.gmail.com>

> If it is much faster than the n-dimensional fft convolution

For me it is about 60(!) times faster, see the attached graph (mind
the log scaling).
I had NxN data with NxN kernels convolved.

The source code for producing the figure is attached as well.

> be worth writing a fftconvolve2
What remains to be checked is the ratio for the case where the kernel
is a lot smaller than the data. If that turns out to be equally fast,
I don't see any reason to keep the current implementation of
scipy.signal.fftconvolve.

Anyway, this may also be related to that other discussion going on
about FFTW. I'm not sure what the current status about FFT
implementations in SciPy is, but at first glance there seem to be
quite a few really, which to me seems redundant and unhelpful.

Does anyone have more insight here?

Cheers,
Nico
-------------- next part --------------
A non-text attachment was scrubbed...
Name: convol-comp.pdf
Type: application/pdf
Size: 18486 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100323/17bcc01e/attachment.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ctest.py
Type: text/x-python
Size: 1826 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100323/17bcc01e/attachment.py>

From mdekauwe at gmail.com  Mon Mar 22 21:05:28 2010
From: mdekauwe at gmail.com (Martin De Kauwe)
Date: Tue, 23 Mar 2010 01:05:28 +0000
Subject: [SciPy-User] Remove the need for the loop
Message-ID: <63aa63301003221805y5e162598qb7169e249a77f0e1@mail.gmail.com>

Hi,

I would like to transfer the contents of a 1D array to a 2D array based on
two arrays listing the relevant locations (the arrays rows and cols).
Ideally I would like to solve my problem without the need for the current
for loop that it quite slow due to the array size. Any help doing this more
elegantly would be much appreciated.

What I currently have...

for fname in glob.glob(os.path.join(path, 'Snowf_P3_*.nc')):
    year = int(fname.split('/')[9].split('_')[2].split('.')[0][0:4])
    if year >= year_jules_run_starts and year <= year_jules_run_ends:
        month = int(fname.split('/')[9].split('_')[2].split('.')[0][4:6])
        f = Dataset(fname, 'r')
        snow = f.variables['Snowf']
        landpts = len(f.dimensions['land'])

        # sum monthly vals
        gswp_snow = (snow[:] * 1000.0 * (3.0 * 60.0 * 60.0) /
1000.0).sum(axis=0)
        f.close()

        # fill up the holding array, this is the bit where I would like to
avoid the for loop
        for point in xrange(numpts):
            snowfall[month-1,(180-(rows[point]-1)),cols[point]-1] +=
gswp_snow[point]
 Many Thanks
Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100323/d541c571/attachment.html>

From josef.pktd at gmail.com  Mon Mar 22 21:08:11 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 22 Mar 2010 21:08:11 -0400
Subject: [SciPy-User] 2d convolution
In-Reply-To: <d933725a1003221717s4bae9790ndf825fc39ae141df@mail.gmail.com>
References: <d933725a1003200451u1996c66escb68316201adc266@mail.gmail.com>
	<1cd32cbb1003200549m75a86fbelea959ad6696dc526@mail.gmail.com>
	<d933725a1003200907u167a7b40p8cbbca12686ce32b@mail.gmail.com>
	<1cd32cbb1003201352g14bf6bc4tf018d7e5bcde07fb@mail.gmail.com>
	<d933725a1003221717s4bae9790ndf825fc39ae141df@mail.gmail.com>
Message-ID: <1cd32cbb1003221808i4fc0179bhcdb669fdc42f92e0@mail.gmail.com>

On Mon, Mar 22, 2010 at 8:17 PM, Nico Schl?mer <nico.schloemer at gmail.com> wrote:
>> If it is much faster than the n-dimensional fft convolution
>
> For me it is about 60(!) times faster, see the attached graph (mind
> the log scaling).
> I had NxN data with NxN kernels convolved.
>
> The source code for producing the figure is attached as well.
>
>> be worth writing a fftconvolve2
> What remains to be checked is the ratio for the case where the kernel
> is a lot smaller than the data. If that turns out to be equally fast,
> I don't see any reason to keep the current implementation of
> scipy.signal.fftconvolve.

I'm using it (or tried to use it) also for 3d data, so we still want
to keep the nd version.

You could file an enhancement ticket for a fftconvolve2d

A few weeks ago there was also the remark in a thread that
signal.convolve2d is faster than the nd version.

I'm not an image processing person, so I don't know what the optimal
convolution behavior for this is, and what could be a replacement for
stsci.convolve.

There is also a convolution in ndimage but I have no idea about it's
performance, and I don't know if scikits.image has any special
convolutions.

>
> Anyway, this may also be related to that other discussion going on
> about FFTW. I'm not sure what the current status about FFT
> implementations in SciPy is, but at first glance there seem to be
> quite a few really, which to me seems redundant and unhelpful.
>
> Does anyone have more insight here?

fftw has been removed from scipy.  scipy.fft and numpy.fft are plain vanilla.

cheers,

Josef

>
> Cheers,
> Nico
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From josef.pktd at gmail.com  Mon Mar 22 21:19:54 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 22 Mar 2010 21:19:54 -0400
Subject: [SciPy-User] Remove the need for the loop
In-Reply-To: <63aa63301003221805y5e162598qb7169e249a77f0e1@mail.gmail.com>
References: <63aa63301003221805y5e162598qb7169e249a77f0e1@mail.gmail.com>
Message-ID: <1cd32cbb1003221819j2f163899ya1aa5321b6475f46@mail.gmail.com>

On Mon, Mar 22, 2010 at 9:05 PM, Martin De Kauwe <mdekauwe at gmail.com> wrote:
> Hi,
> I would like to transfer the contents of a 1D array to a 2D array based on
> two arrays listing the relevant locations (the arrays rows and cols).
> Ideally I would like to solve my problem without the need for the current
> for loop that it quite slow due to the array size. Any help doing this more
> elegantly would be much appreciated.
> What I currently have...
> for fname in glob.glob(os.path.join(path, 'Snowf_P3_*.nc')):
> ?? ?year = int(fname.split('/')[9].split('_')[2].split('.')[0][0:4])
> ?? ?if year >= year_jules_run_starts and year <= year_jules_run_ends:
> ?? ? ? ?month = int(fname.split('/')[9].split('_')[2].split('.')[0][4:6])
> ?? ? ? ?f = Dataset(fname, 'r')
> ?? ? ? ?snow = f.variables['Snowf']
> ?? ? ? ?landpts = len(f.dimensions['land'])
> ?? ? ? ?# sum monthly vals
> ?? ? ? ?gswp_snow = (snow[:] * 1000.0 * (3.0 * 60.0 * 60.0) /
> 1000.0).sum(axis=0)
> ?? ? ? ?f.close()
>
> ?? ? ? ?# fill up the holding array, this is the bit where I would like to
> avoid the for loop
> ?? ? ? ?for point in xrange(numpts):
> ?? ? ? ? ? ?snowfall[month-1,(180-(rows[point]-1)),cols[point]-1] +=
> gswp_snow[point]

this should work if rows, cols and gswp_snow are all 1d (not clear from snippet)

snowfall[month-1,(180-(rows-1)),cols-1] += gswp_snow

otherwise, I think we need an example from which the dimensions/shapes
can be inferred

Josef

> Many Thanks
> Martin
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From d.l.goldsmith at gmail.com  Mon Mar 22 22:13:15 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Mon, 22 Mar 2010 19:13:15 -0700
Subject: [SciPy-User] Opportunity to "plug" scipy
In-Reply-To: <e4b8e4a61003221420r1986d7cbt6c6f206be58d413d@mail.gmail.com>
References: <4BA7C032.9040806@gmail.com>
	<20100322191842.GJ956@phare.normalesup.org>
	<4BA7D0DF.9040709@gmail.com>
	<20100322211249.GB29507@phare.normalesup.org>
	<e4b8e4a61003221420r1986d7cbt6c6f206be58d413d@mail.gmail.com>
Message-ID: <45d1ab481003221913i7081870dpc426b801fc3f994@mail.gmail.com>

On Mon, Mar 22, 2010 at 2:20 PM, Tatiana Al-Chueyr
<tatiana.alchueyr at gmail.com> wrote:
> In addition to NIPY, ?InVesalius [1] could be presented....
>
> InVesalius is an open source 3D medical imaging reconstruction
> software. It was developed using Python and NumPy, among other open
> source technologies.
>
> For the last 6 years I've been working on this project, with medical
> imaging and Python... It would be a pleasure to contribute to the
> presentation.
>
> Kind regards,
>
> Tatiana
>
> [1] http://svn.softwarepublico.gov.br/trac/invesalius

Let me contact Ms. Mason and try to get an idea of a commitment level
(at this point, I don't even know if she was just offering the
opportunity to make a presentation, to have a booth, or what.)

DG

> On 22 March 2010 18:12, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:
>> On Mon, Mar 22, 2010 at 01:19:43PM -0700, David Goldsmith wrote:
>>> Good to know. ?(Unfortunately, I'm too distant from the field to feel
>>> comfortable being a presenter - though it occurs to me that I could
>>> certainly coordinate a team and/or be the POC between such a team and
>>> SIIM - but hopefully your observation means that a team will come
>>> forward; do you happen to know how well said "growing community" is
>>> represented on this list?
>>
>> It's a bunch of overworked researchers trying to develop quality software
>> that their scientific community can more or less apprehend.
>>
>> Some of them do follow this mailing list, but I won't give them up, as I
>> understand that they might not have much time.
>>
>> Ga?l
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
>
> --
> Tatiana Al-Chueyr
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From vincent at vincentdavis.net  Mon Mar 22 23:00:04 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Mon, 22 Mar 2010 21:00:04 -0600
Subject: [SciPy-User] Choosing sub array based on values in a column
Message-ID: <77e831101003222000i6ff466b5j4be78983ac8262fe@mail.gmail.com>

I feel kinda stupid as I think this must be easier than I am making it.
Below my question you will see an answer I got to a question that I thought
I would be able to complete the last steps my self I was wrong :)

 So if I have an array

Y = np.rec.array([(1.0, 0.0, 3.0, 3.5), (0.0, 0.0, 6.0, 6.5), (1.0, 1.0,
9.0, 9.5)], dtype=[('var1', '<f8'), ('var2', '<f8'), ('var3',
'<f8'), ('var4', '<f8')])

do this works like I would expect

Y[['var3','var4']][Y['var1']==1]

>>>array([(3.0, 3.5), (9.0, 9.5)],

      dtype=[('var3', '<f8'), ('var4', '<f8')])

But I would like to do this,

>>> Y[['var3','var4']][Y['var1']==0 and Y['var2']==0]

Traceback (most recent call last):

  File "<string>", line 1, in <fragment>

ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()

I tried some any() and all() combination but nothing worked. What s the
right way to go about this?


Thanks

Vincent


This answer was received on the mail list from Skipper Seabold

If you have a rec array

Y = np.rec.array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0), (7.0, 8.0, 9.0)],

dtype=[('var1', ''var2','var3'])

You can access the rows like,

Y[['var1','var2','var3']]

Note the list within [].

If you want a "normal" array, I like this way that Pierre recently

pointed out. 3 is the number of columns, and it fills in the number

of rows.

Y[['var1','var2','var3']].view((float,3))

note the tuple for the view, if they're all floats. Taking a view

might not work if var# have different types, like ints and floats.

If you want the mean of the rows (mean over the columns axis = 1)

Y[['var1','var2','var3']].view((float,3)).mean(1)

Some shortcuts.

Y[list(Y.dtype.names)].view((float,len(Y.dtype))).mean(1)

Also, for now, the columns will given back to you in the order they're

in in the array no matter which way you ask for them. A patch has

been submitted for returning the order you ask that I hope gets picked

up...

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100322/54d02eaf/attachment.html>

From kshipras at packtpub.com  Tue Mar 23 04:32:50 2010
From: kshipras at packtpub.com (Kshipra Singh)
Date: Tue, 23 Mar 2010 14:02:50 +0530
Subject: [SciPy-User] Author SciPy Books- Packt Publishing.
Message-ID: <27E032775E8E4618AEDD8B039A825731@sonyPC>

Hi All, 

I am writing to you for Packt Publishing, the publishers of computer related books. 

We are planning to extend our catalogue of books based on Scientific Computing Tools and are currently inviting authors interested in writing for Packt. This doesn't need any previous writing experience. Just an expert knowledge of your subject and a passion to share it with others is all that we require. 

So, if you love SciPy and are interested in authoring a book, here's an opportunity knocking your door. Write to us with your SciPy book ideas at author at packtpub.com. Even if you don't have a book idea and are simply interested in authoring a book, we are still keen to hear from you. 

More details about the opportunity are available at: http://authors.packtpub.com/content/scientific-compting-tools-write-Packt

Thanks
Kshipra Singh
Author Relationship Manager
Packt Publishing
www.PacktPub.com
 
Skype: kshiprasingh15
Twitter: http://twitter.com/kshipras
 
Interested in becoming an author? Visit http://authors.packtpub.com for all the information you need about writing for Packt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100323/e19cd47e/attachment.html>

From wizzard028wise at gmail.com  Tue Mar 23 05:53:44 2010
From: wizzard028wise at gmail.com (Dorian)
Date: Tue, 23 Mar 2010 10:53:44 +0100
Subject: [SciPy-User] Author SciPy Books- Packt Publishing.
In-Reply-To: <27E032775E8E4618AEDD8B039A825731@sonyPC>
References: <27E032775E8E4618AEDD8B039A825731@sonyPC>
Message-ID: <674a602a1003230253v62c89ea2r6b1f9f80a075d159@mail.gmail.com>

Yesterday  David asked :

"Would it be possible to release the book under a free documentation
license? "

I'm also interested to know your answer.

Dorian





On Tue, Mar 23, 2010 at 9:32 AM, Kshipra Singh <kshipras at packtpub.com>wrote:

>  Hi All,
>
> I am writing to you for Packt Publishing, the publishers of computer
> related books.
>
> We are planning to extend our catalogue of books based on Scientific
> Computing Tools and are currently inviting authors interested in writing for
> Packt. This doesn't need any previous writing experience. Just an expert
> knowledge of your subject and a passion to share it with others is all that
> we require.
>
> So, if you love SciPy and are interested in authoring a book, here's an
> opportunity knocking your door. Write to us with your SciPy book ideas at
> author at packtpub.com. Even if you don't have a book idea and are simply
> interested in authoring a book, we are still keen to hear from you.
>
> More details about the opportunity are available at:
> http://authors.packtpub.com/content/scientific-compting-tools-write-Packt
>
> Thanks
> Kshipra Singh
> Author Relationship Manager
> Packt Publishing
> www.PacktPub.com
>
> Skype: kshiprasingh15
> Twitter: http://twitter.com/kshipras
>
> Interested in becoming an author? Visit http://authors.packtpub.com for
> all the information you need about writing for Packt.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100323/0b03a795/attachment.html>

From mdekauwe at gmail.com  Tue Mar 23 06:16:19 2010
From: mdekauwe at gmail.com (Martin De Kauwe)
Date: Tue, 23 Mar 2010 10:16:19 +0000 (UTC)
Subject: [SciPy-User] Remove the need for the loop
References: <63aa63301003221805y5e162598qb7169e249a77f0e1@mail.gmail.com>
	<1cd32cbb1003221819j2f163899ya1aa5321b6475f46@mail.gmail.com>
Message-ID: <loom.20100323T110908-37@post.gmane.org>


Hmm OK that didn't work, not quite sure why? 

Error message was...

Traceback (most recent call last):
File "./spatial_forcing_map.py", line 115, in <module>
snowfall[month-1,(180-(rows-1)),cols-1] += gswp_snow
IndexError: arrays used as indices must be of integer (or boolean) type

the shapes for the arrays rows, cols and gswp_snow:

(15238,)
(15238,)
(15238,)

Thanks





From mdekauwe at gmail.com  Tue Mar 23 06:42:13 2010
From: mdekauwe at gmail.com (Martin De Kauwe)
Date: Tue, 23 Mar 2010 10:42:13 +0000 (UTC)
Subject: [SciPy-User] Remove the need for the loop
References: <63aa63301003221805y5e162598qb7169e249a77f0e1@mail.gmail.com>
	<1cd32cbb1003221819j2f163899ya1aa5321b6475f46@mail.gmail.com>
	<loom.20100323T110908-37@post.gmane.org>
Message-ID: <loom.20100323T114129-879@post.gmane.org>


Solved...the rows, cols arrays were not of type int.




From zachary.pincus at yale.edu  Tue Mar 23 07:06:03 2010
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Tue, 23 Mar 2010 07:06:03 -0400
Subject: [SciPy-User] Choosing sub array based on values in a column
In-Reply-To: <77e831101003222000i6ff466b5j4be78983ac8262fe@mail.gmail.com>
References: <77e831101003222000i6ff466b5j4be78983ac8262fe@mail.gmail.com>
Message-ID: <B846B2F5-C446-4484-AEBB-114F4F8F1414@yale.edu>



Y[['var3','var4']][Y['var1']==0 and Y['var2']==0]
should be

Y[['var3','var4']][(Y['var1']==0) & (Y['var2']==0)]


Long story, explained elsewhere, but python's 'and' is expected to  
coerce into a single boolean True or False, which is ambiguous for an  
array ('any' vs. 'all' interpretation), so it just raises an error  
instead. To do elementwise logical operations like you want, use the  
bitwise operators & | ^ on boolean arrays, but note the == operator  
has lower precedence so parenthesization is a must. Or use  
logical_and(), logical_or(), etc.

Zach


On Mar 22, 2010, at 11:00 PM, Vincent Davis wrote:

> I feel kinda stupid as I think this must be easier than I am making  
> it. Below my question you will see an answer I got to a question  
> that I thought I would be able to complete the last steps my self I  
> was wrong :)
>
>  So if I have an array
>
>
> Y = np.rec.array([(1.0, 0.0, 3.0, 3.5), (0.0, 0.0, 6.0, 6.5), (1.0,  
> 1.0, 9.0, 9.5)], dtype=[('var1', '<f8'), ('var2', '<f8'), ('var3',  
> '<f8'), ('var4', '<f8')])
>
>
> do this works like I would expect
>
> Y[['var3','var4']][Y['var1']==1]
>
> >>>array([(3.0, 3.5), (9.0, 9.5)],
>
>       dtype=[('var3', '<f8'), ('var4', '<f8')])
>
> But I would like to do this,
>
>
> >>> Y[['var3','var4']][Y['var1']==0 and Y['var2']==0]
>
> Traceback (most recent call last):
>
>   File "<string>", line 1, in <fragment>
>
> ValueError: The truth value of an array with more than one element  
> is ambiguous. Use a.any() or a.all()
>
> I tried some any() and all() combination but nothing worked. What s  
> the right way to go about this?
>
>
>
> Thanks
>
> Vincent
>
>
>
>
>
> This answer was received on the mail list from Skipper Seabold If  
> you have a rec array
>
> Y = np.rec.array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0), (7.0, 8.0,  
> 9.0)],dtype=[('var1', ''var2','var3'])
>
> You can access the rows like,Y[['var1','var2','var3']]Note the list  
> within [].
>
> If you want a "normal" array, I like this way that Pierre recently  
> pointed out. 3 is the number of columns, and it fills in the  
> numberof rows.
>
> Y[['var1','var2','var3']].view((float,3))
>
> note the tuple for the view, if they're all floats. Taking a  
> viewmight not work if var# have different types, like ints and floats.
>
> If you want the mean of the rows (mean over the columns axis = 1)  
> Y[['var1','var2','var3']].view((float,3)).mean(1)Some  
> shortcuts.Y[list(Y.dtype.names)].view((float,len(Y.dtype))).mean(1)
>
> Also, for now, the columns will given back to you in the order  
> they'rein in the array no matter which way you ask for them. A patch  
> hasbeen submitted for returning the order you ask that I hope gets  
> picked up...
>
>
> Vincent Davis
> 720-301-3003
> vincent at vincentdavis.net
>
> my blog | LinkedIn
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From j.anderson at hull.ac.uk  Tue Mar 23 11:28:18 2010
From: j.anderson at hull.ac.uk (Joseph Anderson)
Date: Tue, 23 Mar 2010 15:28:18 +0000
Subject: [SciPy-User] EPD and fast FFT?
In-Reply-To: <20100322115941.GA3166@jschrod-laptop>
Message-ID: <C7CE8E92.194C5%j.anderson@hull.ac.uk>

Hello Jochen,

Ah... Ok, I'll have a look.

And... I'm hoping that there will be a simple way to stand in for
SciPy.signal.signaltools.fftconvolve.

Ideally I'd just like to replace the above convolve with something
faster--so I don't have to do much, if any re-writing!


My best,

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dr Joseph Anderson
Lecturer in Music

School of Arts and New Media
University of Hull, Scarborough Campus,
Scarborough, North Yorkshire, YO11 3AZ, UK

T: +44.(0)1723.362392 T: +44.(0)1723.357370 F: +44.(0)1723.350815

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On 22/03/2010 11:59 am, "Jochen Schroeder" <cycomanic at gmail.com> wrote:

> I've written an fftw3 wrapper using ctypes (not cython). In contrast to the
> numpy and scipy fft functions it uses plans like the fftw3 c-api. However I've
> tried to make it more usable from Python, i.e. types are detected
> automatically
> etc.. You can find the fftw3 wrapper at http://launchpad.net/pyfftw.
> 
> Cheers
> Jochen



*****************************************************************************************
To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html
*****************************************************************************************


From j.anderson at hull.ac.uk  Tue Mar 23 11:25:46 2010
From: j.anderson at hull.ac.uk (Joseph Anderson)
Date: Tue, 23 Mar 2010 15:25:46 +0000
Subject: [SciPy-User] EPD and fast FFT?
In-Reply-To: <4BA73241.8010601@student.matnat.uio.no>
Message-ID: <C7CE8DFA.194C3%j.anderson@hull.ac.uk>

Hello Dag,



> The latest EPD comes with Intel MKL; so that's likely what you want to
> use. It uses the FFTW 3 API, so any code you write would be usable with
> either MKL (costs money) or FFTW (GPL).

Ah... I forgot to mention I'm using a OSX on a G5 PPC. From my reading, MKL
isn't available for this.

> Personally I'd use Cython to start wrap the FFTW 3 API... at least if
> you know some C that shouldn't take long.
> 
> Dag Sverre

Hmm... I'm a bit rough on my C...

I have some code written calling scipy.signal.signaltools.fftconvolve. With
small arrays, this is fine, but I have some larger RIRs (room impulse
responses) that just grind to a halt.

I've figured that I can write an overlap-add routine to step through both my
input signal (doing this already) AND the kernel.

However, my feeling is that a nicely written convolution routine will 'play
nicely' and do that for me.

Was hoping that there was a simple way to do that, by just setting up FFTW
or something else to stand in for scipy.signal.signaltools.fftconvolve.




~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dr Joseph Anderson
Lecturer in Music

School of Arts and New Media
University of Hull, Scarborough Campus,
Scarborough, North Yorkshire, YO11 3AZ, UK

T: +44.(0)1723.362392 T: +44.(0)1723.357370 F: +44.(0)1723.350815
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


*****************************************************************************************
To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html
*****************************************************************************************


From j.anderson at hull.ac.uk  Tue Mar 23 11:48:27 2010
From: j.anderson at hull.ac.uk (Joseph Anderson)
Date: Tue, 23 Mar 2010 15:48:27 +0000
Subject: [SciPy-User] 2d convolution
In-Reply-To: <d933725a1003221717s4bae9790ndf825fc39ae141df@mail.gmail.com>
Message-ID: <C7CE934B.194CB%j.anderson@hull.ac.uk>

>> If it is much faster than the n-dimensional fft convolution
> 
> For me it is about 60(!) times faster, see the attached graph (mind
> the log scaling).
> I had NxN data with NxN kernels convolved.

This is interesting.

>> be worth writing a fftconvolve2
> What remains to be checked is the ratio for the case where the kernel
> is a lot smaller than the data. If that turns out to be equally fast,
> I don't see any reason to keep the current implementation of
> scipy.signal.fftconvolve.
> 
> Anyway, this may also be related to that other discussion going on
> about FFTW. I'm not sure what the current status about FFT
> implementations in SciPy is, but at first glance there seem to be
> quite a few really, which to me seems redundant and unhelpful.

On this point, I can echo. A quick look turns up:

scipy.fftpack.convolve
scipy.signal.convolve
scipy.signal.fftconvolve
scipy.stsci.convolve
numpy.convolve


I've used scipy.signal.fftconvolve as that's where other signal processing
tools useful to me have been found.

IMHO, ideally, there would be one 'fast' convolve that does the right thing.

I can understand that different convolves in different name spaces exist for
historical and practical reasons. So, maybe there's an argument to keep all
these numerous convolves. (Some appear to be direct, rather than fft
convolves--reason enough to keep those.)

Maybe a thing to do would be a page someplace clarifying which are the
fastest?

Or... 'rewrite' so that all use a fast convolution routine, but preserve the
separate interfaces (if they exist) as necessary?

******



My regards,

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dr Joseph Anderson
Lecturer in Music

School of Arts and New Media
University of Hull, Scarborough Campus,
Scarborough, North Yorkshire, YO11 3AZ, UK

T: +44.(0)1723.362392 T: +44.(0)1723.357370 F: +44.(0)1723.350815
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


*****************************************************************************************
To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html
*****************************************************************************************


From j.anderson at hull.ac.uk  Tue Mar 23 11:51:33 2010
From: j.anderson at hull.ac.uk (Joseph Anderson)
Date: Tue, 23 Mar 2010 15:51:33 +0000
Subject: [SciPy-User] 2d convolution
In-Reply-To: <1cd32cbb1003221808i4fc0179bhcdb669fdc42f92e0@mail.gmail.com>
Message-ID: <C7CE9405.194CF%j.anderson@hull.ac.uk>


> fftw has been removed from scipy.  scipy.fft and numpy.fft are plain vanilla.

Interesting...

Curious to know the reason... Whether for speed or logistics?

*******

In any case, I suppose I still have the problem of looking for a fast
convolution that I can use--with minimal effort--to replace
scipy.signal.signaltools.fftconvolve.



My regards,

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dr Joseph Anderson
Lecturer in Music

School of Arts and New Media
University of Hull, Scarborough Campus,
Scarborough, North Yorkshire, YO11 3AZ, UK

T: +44.(0)1723.362392 T: +44.(0)1723.357370 F: +44.(0)1723.350815
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


*****************************************************************************************
To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html
*****************************************************************************************


From nico.schloemer at gmail.com  Tue Mar 23 12:26:42 2010
From: nico.schloemer at gmail.com (=?ISO-8859-1?Q?Nico_Schl=F6mer?=)
Date: Tue, 23 Mar 2010 17:26:42 +0100
Subject: [SciPy-User] 2d convolution
In-Reply-To: <C7CE9405.194CF%j.anderson@hull.ac.uk>
References: <1cd32cbb1003221808i4fc0179bhcdb669fdc42f92e0@mail.gmail.com>
	<C7CE9405.194CF%j.anderson@hull.ac.uk>
Message-ID: <d933725a1003230926m3d887d3h230151ad3df0caf9@mail.gmail.com>

>> fftw has been removed from scipy. ?scipy.fft and numpy.fft are plain vanilla.
> Interesting...
> Curious to know the reason... Whether for speed or logistics?

I read at <http://cens.ioc.ee/~pearu/scipy/FFTPACK_NOTES.html> that
SciPy's FFTPACK actually interfaces FFTW as well. Apparently they also
do some speed heuristics there. -- FFTPACK devs, is that still
up-to-date?

--Nico


From nico.schloemer at gmail.com  Tue Mar 23 12:22:02 2010
From: nico.schloemer at gmail.com (=?ISO-8859-1?Q?Nico_Schl=F6mer?=)
Date: Tue, 23 Mar 2010 17:22:02 +0100
Subject: [SciPy-User] 2d convolution
In-Reply-To: <C7CE934B.194CB%j.anderson@hull.ac.uk>
References: <d933725a1003221717s4bae9790ndf825fc39ae141df@mail.gmail.com>
	<C7CE934B.194CB%j.anderson@hull.ac.uk>
Message-ID: <d933725a1003230922o3393a3f9n622e32a231d97d94@mail.gmail.com>

> IMHO, ideally, there would be one 'fast' convolve that does the right thing.

I fully and wholeheartedly agree. Eventually the user shouldn't be
bothered too much which implementation is actually the fastest one.
And as far as I understand there are two general ways: with and
without FFT, where the FFT-less implementation seems to excel when
data and kernel differ very much in size. IMHO, it should be possible
for the user to make this choice, overwriting the sane defaults.

Unfortunately, there is no such default right now.

> Or... 'rewrite' so that all use a fast convolution routine, but preserve the
> separate interfaces (if they exist) as necessary?
Indeed. If there are five separate implementations of convolution (or
FFT for that matter) within SciPy alone, this seems like a big waste
of energy.

Would that be something for GSOC maybe?

--Nico


From dg.gmane at thesamovar.net  Tue Mar 23 19:52:14 2010
From: dg.gmane at thesamovar.net (Dan Goodman)
Date: Wed, 24 Mar 2010 00:52:14 +0100
Subject: [SciPy-User] distributed optimisation package
In-Reply-To: <hnpbee$d82$1@dough.gmane.org>
References: <hnpbee$d82$1@dough.gmane.org>
Message-ID: <hobk7h$k3o$1@dough.gmane.org>

Hi all and thanks for comments. We've now released this package, 
available on Google code and soon on PyPI. For various reasons we ended 
up calling it playdoh (hope we don't get in trouble for that).

http://code.google.com/p/playdoh/

Please give it a go and let us know what you think. If you're interested 
in contributing, that would be great too! ;-)

Dan

On 17/03/2010 02:31, Dan Goodman wrote:
> Hi all,
>
> Disclaimer: This isn't strictly about Scipy, but I think the people here
> are probably the most likely to have something interesting to say about
> this, as it's about scientific computing with Python.
>
> Myself and a student in our lab have written a package that we're using
> internally to distribute optimisation problems over several CPUs (or
> GPUs using PyCUDA) over the small cluster of computers in our lab using
> multiprocessing. It has two parts: some code for distributing work over
> several computers and over several CPUs/GPUs on those computers. There's
> no load balancing, we assume that each piece of work is roughly the
> same, however it does handle using shared memory arrays with numpy
> automatically (recently discussed on this list and the numpy one). It
> can use named pipes or IP for communications (we had to use named pipes
> because of the firewall setup in our lab).
>
> The other part of the code is a framework for optimisation routines that
> can be used with this. So far we've implemented a particle swarm
> optimisation algorithm which is ideal for this sort of distributed
> processing because it's very local and so splitting amongst several
> machines is quite nice. We're also thinking about implementing a genetic
> algorithm in the same framework.
>
> Anyway, getting to the point. We've written this largely for our own
> use, but it struck us that this might be of interest more widely. The
> code is fairly simple (a few hundred lines), but there are maybe some
> non-obvious things in there (at least for some people). We're thinking
> about releasing it as an independent package, but there's some work
> involved in that, so we were just looking to see if there was enough
> interest to make it worth the effort really. Also, it's very likely that
> our code is less than perfect, so if anyone would be interested in
> working on it that could be good too.
>
> Thanks for any comments!
>
> Dan



From cycomanic at gmail.com  Tue Mar 23 20:55:22 2010
From: cycomanic at gmail.com (Jochen Schroeder)
Date: Wed, 24 Mar 2010 11:55:22 +1100
Subject: [SciPy-User] 2d convolution
In-Reply-To: <C7CE9405.194CF%j.anderson@hull.ac.uk>
References: <1cd32cbb1003221808i4fc0179bhcdb669fdc42f92e0@mail.gmail.com>
	<C7CE9405.194CF%j.anderson@hull.ac.uk>
Message-ID: <20100324005520.GA2135@cudos0803>

On 03/23/10 15:51, Joseph Anderson wrote:
> 
> > fftw has been removed from scipy.  scipy.fft and numpy.fft are plain vanilla.
> 
> Interesting...
> 
> Curious to know the reason... Whether for speed or logistics?
> 

Firstly fftw in scipy and numpy was fftw2 IIRC, and switching to fftw3 is a bit
more involved, due to the plan semantics (I'm actually planning to implement a
"old style" api, i.e. y=fft(x) in pyfftw at some point). The other thing is
that fftw3 (I don't know about 2) is GPL which does not fit with scipy which is
BSD-like. 
Note I'm just guessing as I was not involved in the decision to remove fftw
from scipy.

> *******
> 
> In any case, I suppose I still have the problem of looking for a fast
> convolution that I can use--with minimal effort--to replace
> scipy.signal.signaltools.fftconvolve.
> 
> 
> 
> My regards,
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dr Joseph Anderson
> Lecturer in Music
> 
> School of Arts and New Media
> University of Hull, Scarborough Campus,
> Scarborough, North Yorkshire, YO11 3AZ, UK
> 
> T: +44.(0)1723.362392 T: +44.(0)1723.357370 F: +44.(0)1723.350815
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 
> *****************************************************************************************
> To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html
> *****************************************************************************************
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From david at silveregg.co.jp  Tue Mar 23 23:10:25 2010
From: david at silveregg.co.jp (David Cournapeau)
Date: Wed, 24 Mar 2010 12:10:25 +0900
Subject: [SciPy-User] 2d convolution
In-Reply-To: <d933725a1003230926m3d887d3h230151ad3df0caf9@mail.gmail.com>
References: <1cd32cbb1003221808i4fc0179bhcdb669fdc42f92e0@mail.gmail.com>
	<C7CE9405.194CF%j.anderson@hull.ac.uk>
	<d933725a1003230926m3d887d3h230151ad3df0caf9@mail.gmail.com>
Message-ID: <4BA982A1.9040105@silveregg.co.jp>

Nico Schl?mer wrote:
>>> fftw has been removed from scipy.  scipy.fft and numpy.fft are plain vanilla.
>> Interesting...
>> Curious to know the reason... Whether for speed or logistics?
> 
> I read at <http://cens.ioc.ee/~pearu/scipy/FFTPACK_NOTES.html> that
> SciPy's FFTPACK actually interfaces FFTW as well.

This is not true anymore, only the original, fortran-based FFTPACK code 
is used by scipy.fftpack since 0.7.0,

David


From spons at utm.csic.es  Wed Mar 24 10:43:50 2010
From: spons at utm.csic.es (Sergi Pons Freixes)
Date: Wed, 24 Mar 2010 15:43:50 +0100
Subject: [SciPy-User] Scikits TimeSeries modifies names of features (dtypes
Message-ID: <2d88eb101003240743y12893e24iff3ad29ad52f5866@mail.gmail.com>

I have this code, which simply creates a TimeSeries object from a datafile:

data_labels = ["Temp", "Sal", "O2"] + self.species.keys()
tmpdata = ts.tsfromtxt(join(dir,f), freq='D', delimiter='\t', \
              usecols=range(len(data_labels)), datecols=0, \
              names=data_labels, dateconverter=self.__dateconverter)

where self.species.keys() is a list of strings, some of them with
spaces, points, or parenthesis. A pair of examples: "Phaeocystis
(colonies)", "Agr.fil. G. corsicum 200x".

But, If I check the resulting tmpdata.dtypes.names (or simply
tmpdata.dtypes), all the spaces has been changed to underscore, and
the points and parenthesis removed. Examples: "Phaeocystis_colonies",
"Agrfil_G_corsicum_200x".

I've checked the documentation from timeseries and from record arrays
of numpy, and cannot found the reason... Any hint abou it?

Regards,
Sergi

--
Sergi Pons Freixes
Ph.D. Student
Marine Technology Unit
http://www.utm.csic.es


From j.anderson at hull.ac.uk  Wed Mar 24 10:58:32 2010
From: j.anderson at hull.ac.uk (Joseph Anderson)
Date: Wed, 24 Mar 2010 14:58:32 +0000
Subject: [SciPy-User] 2d convolution
In-Reply-To: <20100324005520.GA2135@cudos0803>
Message-ID: <C7CFD918.19522%j.anderson@hull.ac.uk>


> Firstly fftw in scipy and numpy was fftw2 IIRC, and switching to fftw3 is a
> bit
> more involved, due to the plan semantics (I'm actually planning to implement a
> "old style" api, i.e. y=fft(x) in pyfftw at some point). The other thing is
> that fftw3 (I don't know about 2) is GPL which does not fit with scipy which
> is
> BSD-like. 
> Note I'm just guessing as I was not involved in the decision to remove fftw
> from scipy.

So... Back to my 1st question...

Is there an easy way for me to get a fast y=fft(x) on G5 PPC OSX?


(BTW--last night I tried a quick enpkg upgrade on my G5 PPC OSX, with little
apparent success. Need to look into that.)


Thanks for your advice.



My regards,

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dr Joseph Anderson
Lecturer in Music

School of Arts and New Media
University of Hull, Scarborough Campus,
Scarborough, North Yorkshire, YO11 3AZ, UK

T: +44.(0)1723.362392 T: +44.(0)1723.357370 F: +44.(0)1723.350815
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


*****************************************************************************************
To view the terms under which this email is distributed, please go to http://www.hull.ac.uk/legal/email_disclaimer.html
*****************************************************************************************


From eadrogue at gmx.net  Wed Mar 24 12:11:29 2010
From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=)
Date: Wed, 24 Mar 2010 17:11:29 +0100
Subject: [SciPy-User] Terminology question
Message-ID: <20100324161129.GA29841@doriath.local>

Hi,

Sorry for this slightly off-topic question..

In a statistical model, what do you call the equation that
defines the independent variable in terms of one or more
dependent variables?

Is it the "model equation"? The "regression function"?
I have very good names for everything else, except for the
bloody equation :(

Ernest


From robert.kern at gmail.com  Wed Mar 24 12:26:15 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 24 Mar 2010 11:26:15 -0500
Subject: [SciPy-User] Terminology question
In-Reply-To: <20100324161129.GA29841@doriath.local>
References: <20100324161129.GA29841@doriath.local>
Message-ID: <3d375d731003240926r102cd8aap39471a6403977a81@mail.gmail.com>

2010/3/24 Ernest Adrogu? <eadrogue at gmx.net>:
> Hi,
>
> Sorry for this slightly off-topic question..
>
> In a statistical model, what do you call the equation that
> defines the independent variable in terms of one or more
> dependent variables?
>
> Is it the "model equation"? The "regression function"?
> I have very good names for everything else, except for the
> bloody equation :(

"The model" is usually fine. For multidimensional linear regression,
this is often called "the design matrix".

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From balarsen at lanl.gov  Wed Mar 24 12:59:30 2010
From: balarsen at lanl.gov (Brian Larsen)
Date: Wed, 24 Mar 2010 10:59:30 -0600
Subject: [SciPy-User] scipy.signal not installing (mac)
Message-ID: <463A1836-46F7-443E-AF42-5D39922DD05C@lanl.gov>

Hello all, 

I didn't see an answer on this in my google searching.  I installed scipy from the latest svn and for some reason scipy.signal does not exist (or isnt installed).

Files like scipy/scipy/signal/waveforms.py do seem to exist but aren't being compiled.

Is there a way to tell if I am missing prerequisites or something?

Sorry I am not sure what more info to pass along, aside from that I followed http://blog.hyperjeff.net/?p=160 to do my install so I have numpy, fftw, etc installed.

Thanks much, 

Brian











-- 

Brian A. Larsen
Space Science and Applications
Group ISR-1, MS D466
Los Alamos National Laboratory
Los Alamos, NM 87545
USA

(For overnight add:
SM-30, Bikini Atoll Road)

Phone: TBD
FAX:   505-665-7395
email: balarsen at lanl.gov

Correspondence /
Technical data or Software Publicly Available




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100324/cbf82331/attachment.html>

From josef.pktd at gmail.com  Wed Mar 24 13:16:59 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 24 Mar 2010 13:16:59 -0400
Subject: [SciPy-User] Terminology question
In-Reply-To: <3d375d731003240926r102cd8aap39471a6403977a81@mail.gmail.com>
References: <20100324161129.GA29841@doriath.local>
	<3d375d731003240926r102cd8aap39471a6403977a81@mail.gmail.com>
Message-ID: <1cd32cbb1003241016icbbdebakd5aa887609823c57@mail.gmail.com>

On Wed, Mar 24, 2010 at 12:26 PM, Robert Kern <robert.kern at gmail.com> wrote:
> 2010/3/24 Ernest Adrogu? <eadrogue at gmx.net>:
>> Hi,
>>
>> Sorry for this slightly off-topic question..
>>
>> In a statistical model, what do you call the equation that
>> defines the independent variable in terms of one or more
>> dependent variables?
>>
>> Is it the "model equation"? The "regression function"?
>> I have very good names for everything else, except for the
>> bloody equation :(
>
> "The model" is usually fine. For multidimensional linear regression,
> this is often called "the design matrix".

just the equation is also often called the regression equation

For the full specification, with distribution of the error term and
other statistical assumptions, we often use "statistical model" (to
distinguish it from a model that we might get from the topic theory
e.g. economics)

Josef

>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> ?-- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From Chris.Barker at noaa.gov  Wed Mar 24 13:48:19 2010
From: Chris.Barker at noaa.gov (Chris Barker)
Date: Wed, 24 Mar 2010 10:48:19 -0700
Subject: [SciPy-User] 2d convolution
In-Reply-To: <C7CFD918.19522%j.anderson@hull.ac.uk>
References: <C7CFD918.19522%j.anderson@hull.ac.uk>
Message-ID: <4BAA5063.2050504@noaa.gov>

Joseph Anderson wrote:
> (BTW--last night I tried a quick enpkg upgrade on my G5 PPC OSX, with little
> apparent success. Need to look into that.)

A while back, Enthought said they were no longer properly supporting PPC :-(

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From kuiper at jpl.nasa.gov  Wed Mar 24 14:25:17 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Wed, 24 Mar 2010 11:25:17 -0700
Subject: [SciPy-User] milliseconds in matplotlib.dates?
Message-ID: <4BAA590D.1080705@jpl.nasa.gov>

Greetings, everyone.

I have some time tagged data spanning about three minutes with 4 ms 
resolution.  If I plot these with plot_data and focus in on some event 
at high resolution I lose the tick marks.  It seems that the minimum 
tick spacing is 1 sec.  That appears consistent with strftime which does 
not provide for fractional seconds.

I thought I saw this issue go by not long ago but I don't recall the 
solution, if any.

I'm using python 2.5.2 and matplotlib 1.1.0.

Thanks and regards,

Tom




From david_baddeley at yahoo.com.au  Wed Mar 24 14:43:42 2010
From: david_baddeley at yahoo.com.au (David Baddeley)
Date: Wed, 24 Mar 2010 11:43:42 -0700 (PDT)
Subject: [SciPy-User] scipy.signal not installing (mac)
In-Reply-To: <463A1836-46F7-443E-AF42-5D39922DD05C@lanl.gov>
References: <463A1836-46F7-443E-AF42-5D39922DD05C@lanl.gov>
Message-ID: <234476.35107.qm@web33003.mail.mud.yahoo.com>

Have you checked to see if there's an __init__.py under scipy.signal? If the folders are there with files in them then this would be the most obvious reason for not being able to import scipy.signal. Have you tried explicitly importing scipy.signal and not just scipy? A while back scipy was changed so that most of the sub modules needed to be imported explicitly so as to reduce the overhead when only using one part of scipy.

cheers,
David



________________________________
From: Brian Larsen <balarsen at lanl.gov>
To: scipy-user at scipy.org
Sent: Thu, 25 March, 2010 5:59:30 AM
Subject: [SciPy-User] scipy.signal not installing (mac)


Hello all, 

I didn't see an answer on this in my google searching.  I installed scipy from the latest svn and for some reason scipy.signal does not exist (or isnt installed).

Files like scipy/scipy/signal/waveforms.py do seem to exist but aren't being compiled.

Is there a way to tell if I am missing prerequisites or something?

Sorry I am not sure what more info to pass along, aside from that I followed http://blog.hyperjeff.net/?p=160 to do my install so I have numpy, fftw, etc installed.

Thanks much, 

Brian












-- 

Brian A. Larsen
Space Science and Applications
Group ISR-1, MS D466
Los Alamos National Laboratory
Los Alamos, NM 87545
USA

(For overnight add:
SM-30, Bikini Atoll Road)

Phone: TBD
FAX:   505-665-7395
email: balarsen at lanl.gov

Correspondence /
Technical data or Software Publicly Available


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100324/5c766499/attachment.html>

From d.l.goldsmith at gmail.com  Wed Mar 24 17:11:49 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Wed, 24 Mar 2010 14:11:49 -0700
Subject: [SciPy-User] Terminology question
In-Reply-To: <1cd32cbb1003241016icbbdebakd5aa887609823c57@mail.gmail.com>
References: <20100324161129.GA29841@doriath.local>
	<3d375d731003240926r102cd8aap39471a6403977a81@mail.gmail.com>
	<1cd32cbb1003241016icbbdebakd5aa887609823c57@mail.gmail.com>
Message-ID: <45d1ab481003241411w56fb8a24t2d6db673bc74513f@mail.gmail.com>

On Wed, Mar 24, 2010 at 10:16 AM,  <josef.pktd at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 12:26 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> 2010/3/24 Ernest Adrogu? <eadrogue at gmx.net>:
>>> Hi,
>>>
>>> Sorry for this slightly off-topic question..
>>>
>>> In a statistical model, what do you call the equation that
>>> defines the independent variable in terms of one or more
>>> dependent variables?

What the others said, except that we usually think of "the model" as
expressing the dependent variable(s) in terms of the independent
variable(s) (the opposite of what you wrote), so strictly speaking, I
guess we'd call what you were asking for (if that's what you really
meant) "the inverse model."

DG

>>>
>>> Is it the "model equation"? The "regression function"?
>>> I have very good names for everything else, except for the
>>> bloody equation :(
>>
>> "The model" is usually fine. For multidimensional linear regression,
>> this is often called "the design matrix".
>
> just the equation is also often called the regression equation
>
> For the full specification, with distribution of the error term and
> other statistical assumptions, we often use "statistical model" (to
> distinguish it from a model that we might get from the topic theory
> e.g. economics)
>
> Josef
>
>>
>> --
>> Robert Kern
>>
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>> ?-- Umberto Eco
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From balarsen at lanl.gov  Wed Mar 24 17:32:34 2010
From: balarsen at lanl.gov (Brian Larsen)
Date: Wed, 24 Mar 2010 15:32:34 -0600
Subject: [SciPy-User] scipy.signal not installing (mac)
Message-ID: <267B79C1-D3DC-4D54-BB19-8F9E11C777CC@lanl.gov>

David, 

I found part of the issue, I had some bogus env variables that I had to kill.  The issue was the c was not compiling, but...

I have a new error that has an odd name _aswfa_...

Below is the error:

Cheers, 

Brian








In [4]: import scipy.signal
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/Users/balarsen/<ipython console> in <module>()

/Library/Python/2.6/site-packages/scipy/signal/__init__.py in <module>()
      7 import sigtools
      8 from waveforms import *
----> 9 from bsplines import *
     10 from filter_design import *
     11 from ltisys import *

/Library/Python/2.6/site-packages/scipy/signal/bsplines.py in <module>()
      2 
----> 3 import scipy.special
      4 from numpy import logical_and, asarray, pi, zeros_like, \
      5      piecewise, array, arctan2, tan, zeros, arange, floor
      6 from numpy.core.umath import sqrt, exp, greater, less, cos, add, sin, \

/Library/Python/2.6/site-packages/scipy/special/__init__.py in <module>()
      6 #from special_version import special_version as __version__

      7 
----> 8 from basic import *
      9 import specfun
     10 import orthogonal

/Library/Python/2.6/site-packages/scipy/special/basic.py in <module>()
      6 
      7 from numpy import *
----> 8 from _cephes import *
      9 import types
     10 import specfun

ImportError: dlopen(/Library/Python/2.6/site-packages/scipy/special/_cephes.so, 2): Symbol not found: _aswfa_
  Referenced from: /Library/Python/2.6/site-packages/scipy/special/_cephes.so
  Expected in: flat namespace
 in /Library/Python/2.6/site-packages/scipy/special/_cephes.so










-- 

Brian A. Larsen
Space Science and Applications
Group ISR-1, MS D466
Los Alamos National Laboratory
Los Alamos, NM 87545
USA

(For overnight add:
SM-30, Bikini Atoll Road)

Phone: TBD
FAX:   505-665-7395
email: balarsen at lanl.gov

Correspondence /
Technical data or Software Publicly Available




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100324/59ecac3a/attachment.html>

From eadrogue at gmx.net  Wed Mar 24 18:02:19 2010
From: eadrogue at gmx.net (Ernest =?iso-8859-1?Q?Adrogu=E9?=)
Date: Wed, 24 Mar 2010 23:02:19 +0100
Subject: [SciPy-User] Terminology question
In-Reply-To: <45d1ab481003241411w56fb8a24t2d6db673bc74513f@mail.gmail.com>
References: <20100324161129.GA29841@doriath.local>
	<3d375d731003240926r102cd8aap39471a6403977a81@mail.gmail.com>
	<1cd32cbb1003241016icbbdebakd5aa887609823c57@mail.gmail.com>
	<45d1ab481003241411w56fb8a24t2d6db673bc74513f@mail.gmail.com>
Message-ID: <20100324220219.GA1891@doriath.local>

24/03/10 @ 14:11 (-0700), thus spake David Goldsmith:
> On Wed, Mar 24, 2010 at 10:16 AM,  <josef.pktd at gmail.com> wrote:
> > On Wed, Mar 24, 2010 at 12:26 PM, Robert Kern <robert.kern at gmail.com> wrote:
> >> 2010/3/24 Ernest Adrogu? <eadrogue at gmx.net>:
> >>> Hi,
> >>>
> >>> Sorry for this slightly off-topic question..
> >>>
> >>> In a statistical model, what do you call the equation that
> >>> defines the independent variable in terms of one or more
> >>> dependent variables?
> 
> What the others said, except that we usually think of "the model" as
> expressing the dependent variable(s) in terms of the independent
> variable(s) (the opposite of what you wrote), so strictly speaking, I
> guess we'd call what you were asking for (if that's what you really
> meant) "the inverse model."

No, I didn't mean the inverse model, sorry about that, I wrote
it the other way around by mistake :)

Thanks for your comments, David, Josef & Robert.

> 
> DG
> 
> >>>
> >>> Is it the "model equation"? The "regression function"?
> >>> I have very good names for everything else, except for the
> >>> bloody equation :(
> >>
> >> "The model" is usually fine. For multidimensional linear regression,
> >> this is often called "the design matrix".
> >
> > just the equation is also often called the regression equation
> >
> > For the full specification, with distribution of the error term and
> > other statistical assumptions, we often use "statistical model" (to
> > distinguish it from a model that we might get from the topic theory
> > e.g. economics)
> >
> > Josef
> >
> >>
> >> --
> >> Robert Kern
> >>
> >> "I have come to believe that the whole world is an enigma, a harmless
> >> enigma that is made terrible by our own mad attempt to interpret it as
> >> though it had an underlying truth."
> >> ?-- Umberto Eco
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From amenity at enthought.com  Wed Mar 24 19:44:34 2010
From: amenity at enthought.com (Amenity Applewhite)
Date: Wed, 24 Mar 2010 18:44:34 -0500
Subject: [SciPy-User] Hiring: Software Developer, Scientific Applications
Message-ID: <6C2FD6F0-A710-4AF2-BF27-19BFDAE9C046@enthought.com>


Enthought is hiring a Software Developer.
See the description below, or on our website: http://www.enthought.com/company/sd-scientific-app.php

Best,
Amenity
--
Amenity Applewhite
Enthought, Inc.
Scientific Computing Solutions
www.enthought.com




Software Developer

The Software Developer at Enthought, Inc. participates in the  
development of scientific and technical applications involving GUIs, 2- 
D and 3-D graphics, workflow and pipeline architecture, and numerical  
algorithms.   The position also involves some interaction with  
clients. Some travel may be required.

We are interested both in experienced applicants as well as in recent  
graduates. Applicants should have a BS, MS, or PhD degree with a  
strong background in science and mathematics, as well as real  
experience developing quality software, either commercial or open  
source. More experienced applicants should also have demonstrated  
project management skills and the ability to lead a team of strong  
developers with highly technical backgrounds.

Applicants will be measured against the following qualifications:

	? (Required) Bachelor's Degree in Computer Science or other  
scientific or engineering field with preferably an M.S. or Ph.D. degree.
	? (Required) Minimum 2 years of technical lead or development  
experience with 4 or more years preferred.
	? Ability to understand a problem domain and then conceive of and  
implement an intuitive user interface geared toward the scientist or  
engineer user.
	? Discipline, pride, and professionalism to write readable,  
documented, and unit-tested code that serves as an example to others  
who later study your work.
	? Strong work ethic and commitment to satisfying the customer.
	? Experience with Python, and a strong understanding of how to apply  
its capabilities to develop GUI frameworks, work flow frameworks, and  
elegant scientific applications.
	? Strong understanding of statistics, optimization, image processing,  
signal processing, or other technical area.
	? Experience with the following:
		? GUI frameworks such as NetBeans or Eclipse
		? wxPython, Qt
		? Low-level 2-D graphics APIs such as Quartz or GDI+
		? 3-D graphics, preferably using VTK
		? Developing or working with plotting APIs
		? Experience using (and interest in contributing to) SciPy
		? numeric algorithms

Enthought offers competitive salaries and the opportunity to work on  
varied and interesting technical projects. We are located in Austin,  
TX,  consistently rated as one of the best places to live in the US.   
Benefits include health, dental, vision and a 401k plan.

If you are interested in applying, submit a resume to  
jobs at enthought.com.   Code samples and links to previous work are  
encouraged but not required.




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100324/4c0f5ec9/attachment.html>

From gunbuster363 at gmail.com  Thu Mar 25 05:31:27 2010
From: gunbuster363 at gmail.com (Lam Waiman)
Date: Thu, 25 Mar 2010 02:31:27 -0700
Subject: [SciPy-User] How to install scipy for opensuse 10.3?
Message-ID: <fc3b9ae01003250231x40b2515fi8fcca84793e98f51@mail.gmail.com>

Dear all,
              I want to install scipy for python usage. I see the
installation isreally complex, can someone give on some advice on it?

My system, opensuse 10.3 (i586)
I am using python 2.6.4

when I see the doc, it said it require LAPACK to be installed,
then I download lapack and see inside, it also require lots of things
including fortran compiler
and I don't know whether the gcc42-fortran compiler in my comp is ok.
I try to dl the g77 compiler, which someone said I will need it for LAPACK
then it g77 only works with a gcc compiler of version 2.8, but I am using a
gcc compiler of version 4.2.

SO COMPLEX!

Looking forwards to your reply, thanks.

Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/c459776e/attachment.html>

From graziano.mirata at lu.unisi.ch  Thu Mar 25 09:29:18 2010
From: graziano.mirata at lu.unisi.ch (Graziano Mirata)
Date: Thu, 25 Mar 2010 14:29:18 +0100
Subject: [SciPy-User] A newbie question on matrix multiplication
Message-ID: <128F2B49-38DB-4A09-893B-C76C020DC102@lu.unisi.ch>

	Hi to everybody, 
i am new here and above all I am new in python. After years of matlab programming I finally have switched to Python. 

Coming to question: i am trying to multiply a matrix transpose by itself but i've got an error on the shape. But it is impossible. Here there is the code: 

type(f) 
Out[213]: <type 'numpy.ndarray'> 

In [214]: type(ft) 
Out[214]: <type 'numpy.ndarray'> 

In [215]: f.size 
Out[215]: 12 

In [216]: f.shape 
Out[216]: (4, 3) 

In [217]: ft = f.transpose() 

In [218]: ft.shape 
Out[218]: (3, 4) 

In [219]: ft*f 
--------------------------------------------------------------------------- 
ValueError                                Traceback (most recent call last) 

/Users/Graz/Documents/PhD researchs/Outsample Perf KT/PythonCodesOSPKT/<ipython console> in <module>() 

ValueError: shape mismatch: objects cannot be broadcast to a single shape 


Any idea where i am wrong? 

Thanx 

G 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/e57e8345/attachment.html>

From amcmorl at gmail.com  Thu Mar 25 09:41:57 2010
From: amcmorl at gmail.com (Angus McMorland)
Date: Thu, 25 Mar 2010 09:41:57 -0400
Subject: [SciPy-User] A newbie question on matrix multiplication
In-Reply-To: <128F2B49-38DB-4A09-893B-C76C020DC102@lu.unisi.ch>
References: <128F2B49-38DB-4A09-893B-C76C020DC102@lu.unisi.ch>
Message-ID: <e85f13c41003250641y6f78e591x5ab77d92cd843061@mail.gmail.com>

On 25 March 2010 09:29, Graziano Mirata <graziano.mirata at lu.unisi.ch> wrote:
> Hi to everybody,
> i am new here and above all I am new in python. After years of matlab
> programming I finally have switched to Python.
>
> Coming to question: i am trying to multiply a matrix transpose by itself but
> i've got an error on the shape. But it is impossible. Here there is the
> code:
>
> type(f)
> Out[213]: <type 'numpy.ndarray'>
>
> In [214]: type(ft)
> Out[214]: <type 'numpy.ndarray'>
>
> In [215]: f.size
> Out[215]: 12
>
> In [216]: f.shape
> Out[216]: (4, 3)
>
> In [217]: ft = f.transpose()
>
> In [218]: ft.shape
> Out[218]: (3, 4)
>
> In [219]: ft*f

In python the * operator isn't overloaded to mean matrix
multiplication, meaning that ft * f tries to do element-wise
multiplication, and (3,4) * (4,3) aren't compatible shapes for that.
Try np.dot(ft, f) instead.

Angus.
-- 
AJC McMorland
Post-doctoral research fellow
Neurobiology, University of Pittsburgh


From graziano.mirata at lu.unisi.ch  Thu Mar 25 09:49:39 2010
From: graziano.mirata at lu.unisi.ch (Graziano Mirata)
Date: Thu, 25 Mar 2010 14:49:39 +0100
Subject: [SciPy-User] A newbie question on matrix multiplication
In-Reply-To: <e85f13c41003250641y6f78e591x5ab77d92cd843061@mail.gmail.com>
References: <128F2B49-38DB-4A09-893B-C76C020DC102@lu.unisi.ch>
	<e85f13c41003250641y6f78e591x5ab77d92cd843061@mail.gmail.com>
Message-ID: <CBB3DB57-63A9-4B23-8E76-5BFAFA37D3CC@lu.unisi.ch>

Great! Thanx

On Mar 25, 2010, at 2:41 PM, Angus McMorland wrote:

> On 25 March 2010 09:29, Graziano Mirata <graziano.mirata at lu.unisi.ch> wrote:
>> Hi to everybody,
>> i am new here and above all I am new in python. After years of matlab
>> programming I finally have switched to Python.
>> 
>> Coming to question: i am trying to multiply a matrix transpose by itself but
>> i've got an error on the shape. But it is impossible. Here there is the
>> code:
>> 
>> type(f)
>> Out[213]: <type 'numpy.ndarray'>
>> 
>> In [214]: type(ft)
>> Out[214]: <type 'numpy.ndarray'>
>> 
>> In [215]: f.size
>> Out[215]: 12
>> 
>> In [216]: f.shape
>> Out[216]: (4, 3)
>> 
>> In [217]: ft = f.transpose()
>> 
>> In [218]: ft.shape
>> Out[218]: (3, 4)
>> 
>> In [219]: ft*f
> 
> In python the * operator isn't overloaded to mean matrix
> multiplication, meaning that ft * f tries to do element-wise
> multiplication, and (3,4) * (4,3) aren't compatible shapes for that.
> Try np.dot(ft, f) instead.
> 
> Angus.
> -- 
> AJC McMorland
> Post-doctoral research fellow
> Neurobiology, University of Pittsburgh
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From ivo.maljevic at gmail.com  Thu Mar 25 10:13:05 2010
From: ivo.maljevic at gmail.com (Ivo Maljevic)
Date: Thu, 25 Mar 2010 10:13:05 -0400
Subject: [SciPy-User] A newbie question on matrix multiplication
In-Reply-To: <CBB3DB57-63A9-4B23-8E76-5BFAFA37D3CC@lu.unisi.ch>
References: <128F2B49-38DB-4A09-893B-C76C020DC102@lu.unisi.ch>
	<e85f13c41003250641y6f78e591x5ab77d92cd843061@mail.gmail.com>
	<CBB3DB57-63A9-4B23-8E76-5BFAFA37D3CC@lu.unisi.ch>
Message-ID: <826c64da1003250713s78e9b004o940693e4ee692167@mail.gmail.com>

You may find that, and some other useful tips here:

http://www.scipy.org/NumPy_for_Matlab_Users



On 25 March 2010 09:49, Graziano Mirata <graziano.mirata at lu.unisi.ch> wrote:

> Great! Thanx
>
> On Mar 25, 2010, at 2:41 PM, Angus McMorland wrote:
>
> > On 25 March 2010 09:29, Graziano Mirata <graziano.mirata at lu.unisi.ch>
> wrote:
> >> Hi to everybody,
> >> i am new here and above all I am new in python. After years of matlab
> >> programming I finally have switched to Python.
> >>
> >> Coming to question: i am trying to multiply a matrix transpose by itself
> but
> >> i've got an error on the shape. But it is impossible. Here there is the
> >> code:
> >>
> >> type(f)
> >> Out[213]: <type 'numpy.ndarray'>
> >>
> >> In [214]: type(ft)
> >> Out[214]: <type 'numpy.ndarray'>
> >>
> >> In [215]: f.size
> >> Out[215]: 12
> >>
> >> In [216]: f.shape
> >> Out[216]: (4, 3)
> >>
> >> In [217]: ft = f.transpose()
> >>
> >> In [218]: ft.shape
> >> Out[218]: (3, 4)
> >>
> >> In [219]: ft*f
> >
> > In python the * operator isn't overloaded to mean matrix
> > multiplication, meaning that ft * f tries to do element-wise
> > multiplication, and (3,4) * (4,3) aren't compatible shapes for that.
> > Try np.dot(ft, f) instead.
> >
> > Angus.
> > --
> > AJC McMorland
> > Post-doctoral research fellow
> > Neurobiology, University of Pittsburgh
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/ad0ca053/attachment.html>

From jkington at wisc.edu  Thu Mar 25 10:40:26 2010
From: jkington at wisc.edu (Joe Kington)
Date: Thu, 25 Mar 2010 09:40:26 -0500
Subject: [SciPy-User] How to install scipy for opensuse 10.3?
In-Reply-To: <fc3b9ae01003250231x40b2515fi8fcca84793e98f51@mail.gmail.com>
References: <fc3b9ae01003250231x40b2515fi8fcca84793e98f51@mail.gmail.com>
Message-ID: <f6951adf1003250740sf893ceav88f5fc44b943268f@mail.gmail.com>

I'm pretty certain scipy and numpy are in the repos for OpenSuse 10.3.  (or
at least they were a few years ago... Is there some reason you can't upgrade
to 11.2? They're definitely in the repos for 11.0, 11.1, and 11.2)

Do you have the science repos enabled?  They may have dropped them for 10.x,
but you can always add the SEL_10 versions.  They should be compatible.
http://download.opensuse.org/repositories/science/SLE_10/

Hope that helps,
-Joe

On Thu, Mar 25, 2010 at 4:31 AM, Lam Waiman <gunbuster363 at gmail.com> wrote:

> Dear all,
>               I want to install scipy for python usage. I see the
> installation isreally complex, can someone give on some advice on it?
>
> My system, opensuse 10.3 (i586)
> I am using python 2.6.4
>
> when I see the doc, it said it require LAPACK to be installed,
> then I download lapack and see inside, it also require lots of things
> including fortran compiler
> and I don't know whether the gcc42-fortran compiler in my comp is ok.
> I try to dl the g77 compiler, which someone said I will need it for LAPACK
> then it g77 only works with a gcc compiler of version 2.8, but I am using a
> gcc compiler of version 4.2.
>
> SO COMPLEX!
>
> Looking forwards to your reply, thanks.
>
> Raymond
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/03e05fd6/attachment.html>

From paul.anton.letnes at gmail.com  Thu Mar 25 11:16:13 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Thu, 25 Mar 2010 08:16:13 -0700
Subject: [SciPy-User] A newbie question on matrix multiplication
In-Reply-To: <128F2B49-38DB-4A09-893B-C76C020DC102@lu.unisi.ch>
References: <128F2B49-38DB-4A09-893B-C76C020DC102@lu.unisi.ch>
Message-ID: <29DD5851-BFF8-4B08-9313-F7B03113FF44@gmail.com>

Hi.

On 25. mars 2010, at 06.29, Graziano Mirata wrote:

> 	Hi to everybody, 
> i am new here and above all I am new in python. After years of matlab programming I finally have switched to Python. 
> 
> Coming to question: i am trying to multiply a matrix transpose by itself but i've got an error on the shape. But it is impossible. Here there is the code: 
> 
> type(f) 
> Out[213]: <type 'numpy.ndarray'> 
> 
In numpy, you have 2 distinct classes representing arrays and matrices. In general, arrays are n-dimensional and use element-wise multiplication. Matrices are 2-dimensional and use matrix multiplication, matlab-style.
>>> import numpy
>>> a = numpy.eye(3)
>>> type(a)
<type 'numpy.ndarray'>
>>> a = numpy.mat(a)
>>> type(a)
<class 'numpy.matrixlib.defmatrix.matrix'>

So take your arrays and put them inside a numpy.mat() function call, and you're basically back to matlab-style matrices. Be careful which type you have - as in matlab, it matters a lot whether you are doing matrix or element-wise multiplication...

-Paul



> In [214]: type(ft) 
> Out[214]: <type 'numpy.ndarray'> 
> 
> In [215]: f.size 
> Out[215]: 12 
> 
> In [216]: f.shape 
> Out[216]: (4, 3) 
> 
> In [217]: ft = f.transpose() 
> 
> In [218]: ft.shape 
> Out[218]: (3, 4) 
> 
> In [219]: ft*f 
> --------------------------------------------------------------------------- 
> ValueError                                Traceback (most recent call last) 
> 
> /Users/Graz/Documents/PhD researchs/Outsample Perf KT/PythonCodesOSPKT/<ipython console> in <module>() 
> 
> ValueError: shape mismatch: objects cannot be broadcast to a single shape 
> 
> 
> Any idea where i am wrong? 
> 
> Thanx 
> 
> G 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From gunbuster363 at gmail.com  Thu Mar 25 11:29:28 2010
From: gunbuster363 at gmail.com (Lam Waiman)
Date: Thu, 25 Mar 2010 23:29:28 +0800
Subject: [SciPy-User] How to install scipy for opensuse 10.3?
In-Reply-To: <f6951adf1003250740sf893ceav88f5fc44b943268f@mail.gmail.com>
References: <fc3b9ae01003250231x40b2515fi8fcca84793e98f51@mail.gmail.com>
	<f6951adf1003250740sf893ceav88f5fc44b943268f@mail.gmail.com>
Message-ID: <fc3b9ae01003250829w36fcff9ahaf48548ac59d3d34@mail.gmail.com>

But I cannot see scipy in that folder........
And I don't want to upgrade to 11.2 because I am not sure if my package in
10.3 still works.....I don't want to risk now....

2010/3/25 Joe Kington <jkington at wisc.edu>

> I'm pretty certain scipy and numpy are in the repos for OpenSuse 10.3.  (or
> at least they were a few years ago... Is there some reason you can't upgrade
> to 11.2? They're definitely in the repos for 11.0, 11.1, and 11.2)
>
> Do you have the science repos enabled?  They may have dropped them for
> 10.x, but you can always add the SEL_10 versions.  They should be
> compatible. http://download.opensuse.org/repositories/science/SLE_10/
>
> Hope that helps,
> -Joe
>
> On Thu, Mar 25, 2010 at 4:31 AM, Lam Waiman <gunbuster363 at gmail.com>wrote:
>
>> Dear all,
>>               I want to install scipy for python usage. I see the
>> installation isreally complex, can someone give on some advice on it?
>>
>> My system, opensuse 10.3 (i586)
>> I am using python 2.6.4
>>
>> when I see the doc, it said it require LAPACK to be installed,
>> then I download lapack and see inside, it also require lots of things
>> including fortran compiler
>> and I don't know whether the gcc42-fortran compiler in my comp is ok.
>> I try to dl the g77 compiler, which someone said I will need it for LAPACK
>> then it g77 only works with a gcc compiler of version 2.8, but I am using
>> a gcc compiler of version 4.2.
>>
>> SO COMPLEX!
>>
>> Looking forwards to your reply, thanks.
>>
>> Raymond
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/c2f024c7/attachment.html>

From gokhansever at gmail.com  Thu Mar 25 14:17:01 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Thu, 25 Mar 2010 13:17:01 -0500
Subject: [SciPy-User] milliseconds in matplotlib.dates?
In-Reply-To: <4BAA590D.1080705@jpl.nasa.gov>
References: <4BAA590D.1080705@jpl.nasa.gov>
Message-ID: <49d6b3501003251117v2e332a88t18bd52efc2d2cc4d@mail.gmail.com>

On Wed, Mar 24, 2010 at 1:25 PM, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:

> Greetings, everyone.
>
> I have some time tagged data spanning about three minutes with 4 ms
> resolution.  If I plot these with plot_data and focus in on some event
> at high resolution I lose the tick marks.  It seems that the minimum
> tick spacing is 1 sec.  That appears consistent with strftime which does
> not provide for fractional seconds.
>
> I thought I saw this issue go by not long ago but I don't recall the
> solution, if any.
>
> I'm using python 2.5.2 and matplotlib 1.1.0.
>
> Thanks and regards,
>
> Tom
>

Hi,

As far as I know SecondLocator() is the smallest it gets for time-date
plotting in matplotlib. Probably a new class needs to be written for
MilliSecondLocator() in dates.py

Alternatively, you might use just floating-point version of your time values
and with a little adjustment (Ryan mentioned this in the recently, and will
forward there for other suggestions) millisecond resolutions should be
visible when zoomed in furthest.

Something like:

sci_fmt = plt.FormatStrFormatter("%.2f")
plt.gca().xaxis.set_major_formatter(sci_fmt)

By the way, where did you get mpl v1.1.0? Mine says:

In[19]: matplotlib.__version__
Out[19]: '1.0.svn'


-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/7f44a69b/attachment.html>

From jkington at wisc.edu  Thu Mar 25 15:00:20 2010
From: jkington at wisc.edu (Joe Kington)
Date: Thu, 25 Mar 2010 14:00:20 -0500
Subject: [SciPy-User] How to install scipy for opensuse 10.3?
In-Reply-To: <fc3b9ae01003250829w36fcff9ahaf48548ac59d3d34@mail.gmail.com>
References: <fc3b9ae01003250231x40b2515fi8fcca84793e98f51@mail.gmail.com>
	<f6951adf1003250740sf893ceav88f5fc44b943268f@mail.gmail.com>
	<fc3b9ae01003250829w36fcff9ahaf48548ac59d3d34@mail.gmail.com>
Message-ID: <f6951adf1003251200v69622a3dj701ae73e2081148a@mail.gmail.com>

It's rather dated (scipy 0.6) , but I think this repo definitely has scipy
and should be compatible with 10.3.
http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/<http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/x86_64/>

<http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/x86_64/>Hope
that helps,
-Joe

On Thu, Mar 25, 2010 at 10:29 AM, Lam Waiman <gunbuster363 at gmail.com> wrote:

> But I cannot see scipy in that folder........
> And I don't want to upgrade to 11.2 because I am not sure if my package in
> 10.3 still works.....I don't want to risk now....
>
> 2010/3/25 Joe Kington <jkington at wisc.edu>
>
> I'm pretty certain scipy and numpy are in the repos for OpenSuse 10.3.  (or
>> at least they were a few years ago... Is there some reason you can't upgrade
>> to 11.2? They're definitely in the repos for 11.0, 11.1, and 11.2)
>>
>> Do you have the science repos enabled?  They may have dropped them for
>> 10.x, but you can always add the SEL_10 versions.  They should be
>> compatible. http://download.opensuse.org/repositories/science/SLE_10/
>>
>> Hope that helps,
>> -Joe
>>
>> On Thu, Mar 25, 2010 at 4:31 AM, Lam Waiman <gunbuster363 at gmail.com>wrote:
>>
>>> Dear all,
>>>               I want to install scipy for python usage. I see the
>>> installation isreally complex, can someone give on some advice on it?
>>>
>>> My system, opensuse 10.3 (i586)
>>> I am using python 2.6.4
>>>
>>> when I see the doc, it said it require LAPACK to be installed,
>>> then I download lapack and see inside, it also require lots of things
>>> including fortran compiler
>>> and I don't know whether the gcc42-fortran compiler in my comp is ok.
>>> I try to dl the g77 compiler, which someone said I will need it for
>>> LAPACK
>>> then it g77 only works with a gcc compiler of version 2.8, but I am using
>>> a gcc compiler of version 4.2.
>>>
>>> SO COMPLEX!
>>>
>>> Looking forwards to your reply, thanks.
>>>
>>> Raymond
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/54cb9506/attachment.html>

From jkington at wisc.edu  Thu Mar 25 15:02:35 2010
From: jkington at wisc.edu (Joe Kington)
Date: Thu, 25 Mar 2010 14:02:35 -0500
Subject: [SciPy-User] How to install scipy for opensuse 10.3?
In-Reply-To: <15106_1269543643_ZZg0U6IyMPQs8.00_f6951adf1003251200v69622a3dj701ae73e2081148a@mail.gmail.com>
References: <fc3b9ae01003250231x40b2515fi8fcca84793e98f51@mail.gmail.com>
	<f6951adf1003250740sf893ceav88f5fc44b943268f@mail.gmail.com>
	<fc3b9ae01003250829w36fcff9ahaf48548ac59d3d34@mail.gmail.com>
	<15106_1269543643_ZZg0U6IyMPQs8.00_f6951adf1003251200v69622a3dj701ae73e2081148a@mail.gmail.com>
Message-ID: <f6951adf1003251202s47ea8a17xcc618cd649cba0dc@mail.gmail.com>

Gah, meant to link to the base repo, but I managed to have the text show the
base repo, and the link have the 64-bit repo...
Here you go:
http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/


On Thu, Mar 25, 2010 at 2:00 PM, Joe Kington <jkington at wisc.edu> wrote:

> It's rather dated (scipy 0.6) , but I think this repo definitely has scipy
> and should be compatible with 10.3.
> http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/<http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/x86_64/>
>
>
> <http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/x86_64/>Hope
> that helps,
> -Joe
>
>
> On Thu, Mar 25, 2010 at 10:29 AM, Lam Waiman <gunbuster363 at gmail.com>wrote:
>
>> But I cannot see scipy in that folder........
>> And I don't want to upgrade to 11.2 because I am not sure if my package in
>> 10.3 still works.....I don't want to risk now....
>>
>> 2010/3/25 Joe Kington <jkington at wisc.edu>
>>
>> I'm pretty certain scipy and numpy are in the repos for OpenSuse 10.3.
>>> (or at least they were a few years ago... Is there some reason you can't
>>> upgrade to 11.2? They're definitely in the repos for 11.0, 11.1, and 11.2)
>>>
>>> Do you have the science repos enabled?  They may have dropped them for
>>> 10.x, but you can always add the SEL_10 versions.  They should be
>>> compatible. http://download.opensuse.org/repositories/science/SLE_10/
>>>
>>> Hope that helps,
>>> -Joe
>>>
>>> On Thu, Mar 25, 2010 at 4:31 AM, Lam Waiman <gunbuster363 at gmail.com>wrote:
>>>
>>>> Dear all,
>>>>               I want to install scipy for python usage. I see the
>>>> installation isreally complex, can someone give on some advice on it?
>>>>
>>>> My system, opensuse 10.3 (i586)
>>>> I am using python 2.6.4
>>>>
>>>> when I see the doc, it said it require LAPACK to be installed,
>>>> then I download lapack and see inside, it also require lots of things
>>>> including fortran compiler
>>>> and I don't know whether the gcc42-fortran compiler in my comp is ok.
>>>> I try to dl the g77 compiler, which someone said I will need it for
>>>> LAPACK
>>>> then it g77 only works with a gcc compiler of version 2.8, but I am
>>>> using a gcc compiler of version 4.2.
>>>>
>>>> SO COMPLEX!
>>>>
>>>> Looking forwards to your reply, thanks.
>>>>
>>>> Raymond
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/323de0a3/attachment.html>

From jlconlin at gmail.com  Thu Mar 25 16:32:39 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Thu, 25 Mar 2010 14:32:39 -0600
Subject: [SciPy-User] How to estimate error in polynomial coefficients from
	scipy.polyfit?
Message-ID: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>

I am using scipy.polyfit to fit a curve to my data.  Members of this
list have been integral in my understanding of how to use this
function.  Now I would like to know how I can get the uncertainties
(standard deviations) of polynomial coefficients from the returned
values from scipy.polyfit.  If I understand correctly, the residuals
are sometimes called the R^2 error, right?  That gives an estimate of
how well we have fit the data.  I don't know how to use rank or any of
the other returned values to get the uncertainties.

Can someone please help?
Thanks,
Jeremy


From dwf at cs.toronto.edu  Thu Mar 25 16:49:51 2010
From: dwf at cs.toronto.edu (David Warde-Farley)
Date: Thu, 25 Mar 2010 16:49:51 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
Message-ID: <6E426866-35BA-4919-9ABD-465DAD62FB19@cs.toronto.edu>

On 25-Mar-10, at 4:32 PM, Jeremy Conlin wrote:

> Now I would like to know how I can get the uncertainties
> (standard deviations) of polynomial coefficients from the returned
> values from scipy.polyfit.  If I understand correctly, the residuals
> are sometimes called the R^2 error, right?

No, that's not what R^2 is. See http://en.wikipedia.org/wiki/Coefficient_of_determination

The residuals plus some assumptions about the kind of noise present in  
your data can tell you something about the standard deviation of a  
given predicted function value, but I'm not sure how to turn that into  
a measure of uncertainty on the coefficient itself. My first instinct  
would be to do it with bootstrap samples, i.e. if you have N data  
points, sample with replacement from that data to generate, say, 100  
sets of size N, fit the coefficients on each of them, and take the  
empirical standard deviation. I am not sure how polyfit responds to  
duplicated points, however, though it should reweight your least  
squares estimate a bit.

David


From charlesr.harris at gmail.com  Thu Mar 25 16:52:32 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 25 Mar 2010 14:52:32 -0600
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
Message-ID: <e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>

On Thu, Mar 25, 2010 at 2:32 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:

> I am using scipy.polyfit to fit a curve to my data.  Members of this
> list have been integral in my understanding of how to use this
> function.  Now I would like to know how I can get the uncertainties
> (standard deviations) of polynomial coefficients from the returned
> values from scipy.polyfit.  If I understand correctly, the residuals
> are sometimes called the R^2 error, right?  That gives an estimate of
> how well we have fit the data.  I don't know how to use rank or any of
> the other returned values to get the uncertainties.
>
> Can someone please help?
>

You want the covariance of the coefficients, (A.T * A)^-1/var, where A is
the design matrix. I'd have to see what the scipy fit returns to tell you
more. In anycase, from that you can plot curves at +/- sigma to show the
error bounds on the result. I can be more explicit if you want.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/4ca018e3/attachment.html>

From josef.pktd at gmail.com  Thu Mar 25 16:58:32 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 25 Mar 2010 16:58:32 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
Message-ID: <1cd32cbb1003251358n3c7b6096nff0e6dc4f0a6353a@mail.gmail.com>

On Thu, Mar 25, 2010 at 4:52 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Thu, Mar 25, 2010 at 2:32 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>
>> I am using scipy.polyfit to fit a curve to my data. ?Members of this
>> list have been integral in my understanding of how to use this
>> function. ?Now I would like to know how I can get the uncertainties
>> (standard deviations) of polynomial coefficients from the returned
>> values from scipy.polyfit. ?If I understand correctly, the residuals
>> are sometimes called the R^2 error, right? ?That gives an estimate of
>> how well we have fit the data. ?I don't know how to use rank or any of
>> the other returned values to get the uncertainties.
>>
>> Can someone please help?
>
> You want the covariance of the coefficients, (A.T * A)^-1/var, where A is
> the design matrix. I'd have to see what the scipy fit returns to tell you
> more. In anycase, from that you can plot curves at +/- sigma to show the
> error bounds on the result. I can be more explicit if you want.
>
> Chuck

the easiest way to get the full statistical results for the fit is to
use statsmodels

something like
import scikits.statsmodels as sm
results = sm.OLS(y, np.vandermonde(x, order)).fit()
dir(results)
with standard deviations on parameter estimates, ...

results.params should be the same as np.polyfit
results.model.predict(xnew)
and there are somewhere the prediction errors

without statsmodels it is a few lines of code, but requires "thinking"

Josef



>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From kuiper at jpl.nasa.gov  Thu Mar 25 16:59:38 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Thu, 25 Mar 2010 13:59:38 -0700
Subject: [SciPy-User] milliseconds in matplotlib.dates?
In-Reply-To: <mailman.6066.1269550202.2398.scipy-user@scipy.org>
References: <mailman.6066.1269550202.2398.scipy-user@scipy.org>
Message-ID: <4BABCEBA.2020809@jpl.nasa.gov>

scipy-user-request at scipy.org wrote:
> Message: 1
> Date: Thu, 25 Mar 2010 13:17:01 -0500
> From: G?khan Sever <gokhansever at gmail.com>
> Subject: Re: [SciPy-User] milliseconds in matplotlib.dates?
> To: SciPy Users List <scipy-user at scipy.org>
> Cc: Matplotlib Users <matplotlib-users at lists.sourceforge.net>
> Message-ID:
> 	<49d6b3501003251117v2e332a88t18bd52efc2d2cc4d at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Wed, Mar 24, 2010 at 1:25 PM, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:
>
>   
>> I'm using python 2.5.2 and matplotlib 1.1.0.
>>     
I got mpl confused with pylab.
In [2]: pylab.__version__
Out[2]: '1.1.0'
> By the way, where did you get mpl v1.1.0? Mine says:
>
> In[19]: matplotlib.__version__
> Out[19]: '1.0.svn'
>   
In [3]: print matplotlib.__version__
Out[3]: '0.98.1'

Thanks for your help.

Tom


From kuiper at jpl.nasa.gov  Thu Mar 25 17:17:17 2010
From: kuiper at jpl.nasa.gov (Tom Kuiper)
Date: Thu, 25 Mar 2010 14:17:17 -0700
Subject: [SciPy-User] milliseconds in matplotlib.dates?
In-Reply-To: <mailman.6066.1269550202.2398.scipy-user@scipy.org>
References: <mailman.6066.1269550202.2398.scipy-user@scipy.org>
Message-ID: <4BABD2DD.2040702@jpl.nasa.gov>


> Date: Thu, 25 Mar 2010 13:17:01 -0500
> From: G?khan Sever <gokhansever at gmail.com>
> Subject: Re: [SciPy-User] milliseconds in matplotlib.dates?
> To: SciPy Users List <scipy-user at scipy.org>
> Cc: Matplotlib Users <matplotlib-users at lists.sourceforge.net>
>   
...
> Alternatively, you might use just floating-point version of your time values
> and with a little adjustment (Ryan mentioned this in the recently, and will
> forward there for other suggestions) millisecond resolutions should be
> visible when zoomed in furthest.
>
> Something like:
>
> sci_fmt = plt.FormatStrFormatter("%.2f")
> plt.gca().xaxis.set_major_formatter(sci_fmt)
>   
Here's how I modified my code:
sci_fmt = FormatStrFormatter("%.2f")
fig = figure()
top_axes = subplot(211)
top_axes.xaxis.set_major_formatter(sci_fmt)
#plot_date(times,kurts,fmt='b-')
#plot_date(times,kurt_sm,fmt='r-',label=("Hamming 1-sec FWHM"))
plot(times,kurts,'b-')
plot(times,kurt_sm,'r-',label=("Hamming 1-sec FWHM"))
I then restarted 'ipython -pylab' because I've noticed that date 
formatting problems seems to linger even after a code change.  However, 
I see no change in behaviour.  I can get six 1-sec ticks across my 
plot.  If I expand the plot any more, the ticks disappear.

Regards

Tom


From jlconlin at gmail.com  Thu Mar 25 17:20:24 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Thu, 25 Mar 2010 15:20:24 -0600
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
Message-ID: <2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>

On Thu, Mar 25, 2010 at 2:52 PM, Charles R Harris
<charlesr.harris at gmail.com> wrote:
>
>
> On Thu, Mar 25, 2010 at 2:32 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>
>> I am using scipy.polyfit to fit a curve to my data. ?Members of this
>> list have been integral in my understanding of how to use this
>> function. ?Now I would like to know how I can get the uncertainties
>> (standard deviations) of polynomial coefficients from the returned
>> values from scipy.polyfit. ?If I understand correctly, the residuals
>> are sometimes called the R^2 error, right? ?That gives an estimate of
>> how well we have fit the data. ?I don't know how to use rank or any of
>> the other returned values to get the uncertainties.
>>
>> Can someone please help?
>
> You want the covariance of the coefficients, (A.T * A)^-1/var, where A is
> the design matrix. I'd have to see what the scipy fit returns to tell you
> more. In anycase, from that you can plot curves at +/- sigma to show the
> error bounds on the result. I can be more explicit if you want.
>
> Chuck

Thanks Chuck.  That seems to get closer to what I need.  I am just
fitting data to a polynomial, nothing too fancy.  I would like the
variance (not the covariance) of the coeffficients.  As far as what is
returned from scipy.polyfit this is what the documentation says:

residuals, rank, singular_values, rcond : present only if `full` = True
        Residuals of the least-squares fit, the effective rank of the scaled
        Vandermonde coefficient matrix, its singular values, and the specified
        value of `rcond`. For more details, see `linalg.lstsq`.

I don't think any of these things are "design matrix" as you have
indicated I need.  The documentation for linalg.lstsq does not say
what rcond is unfortunately.   Any ideas?

Jeremy


From gokhansever at gmail.com  Thu Mar 25 17:25:32 2010
From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=)
Date: Thu, 25 Mar 2010 16:25:32 -0500
Subject: [SciPy-User] milliseconds in matplotlib.dates?
In-Reply-To: <4BABD2DD.2040702@jpl.nasa.gov>
References: <mailman.6066.1269550202.2398.scipy-user@scipy.org>
	<4BABD2DD.2040702@jpl.nasa.gov>
Message-ID: <49d6b3501003251425r792ff367oc889e85b21744f25@mail.gmail.com>

On Thu, Mar 25, 2010 at 4:17 PM, Tom Kuiper <kuiper at jpl.nasa.gov> wrote:

>
> > Date: Thu, 25 Mar 2010 13:17:01 -0500
> > From: G?khan Sever <gokhansever at gmail.com>
> > Subject: Re: [SciPy-User] milliseconds in matplotlib.dates?
> > To: SciPy Users List <scipy-user at scipy.org>
> > Cc: Matplotlib Users <matplotlib-users at lists.sourceforge.net>
> >
> ...
> > Alternatively, you might use just floating-point version of your time
> values
> > and with a little adjustment (Ryan mentioned this in the recently, and
> will
> > forward there for other suggestions) millisecond resolutions should be
> > visible when zoomed in furthest.
> >
> > Something like:
> >
> > sci_fmt = plt.FormatStrFormatter("%.2f")
> > plt.gca().xaxis.set_major_formatter(sci_fmt)
> >
> Here's how I modified my code:
> sci_fmt = FormatStrFormatter("%.2f")
> fig = figure()
> top_axes = subplot(211)
> top_axes.xaxis.set_major_formatter(sci_fmt)
> #plot_date(times,kurts,fmt='b-')
> #plot_date(times,kurt_sm,fmt='r-',label=("Hamming 1-sec FWHM"))
> plot(times,kurts,'b-')
> plot(times,kurt_sm,'r-',label=("Hamming 1-sec FWHM"))
> I then restarted 'ipython -pylab' because I've noticed that date
> formatting problems seems to linger even after a code change.  However,
> I see no change in behaviour.  I can get six 1-sec ticks across my
> plot.  If I expand the plot any more, the ticks disappear.
>
> Regards
>
> Tom
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>

Here is how it looks when I run your code till plot(times...)

http://img202.imageshack.us/img202/5415/ss1i.png

and one with zoom:

http://img411.imageshack.us/img411/8292/ss2el.png

I am not sure this could be related to the mpl version you have.


-- 
G?khan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/a6aca923/attachment.html>

From peridot.faceted at gmail.com  Thu Mar 25 18:21:31 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 25 Mar 2010 18:21:31 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
Message-ID: <ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>

On 25 March 2010 17:20, Jeremy Conlin <jlconlin at gmail.com> wrote:
> On Thu, Mar 25, 2010 at 2:52 PM, Charles R Harris
> <charlesr.harris at gmail.com> wrote:
>>
>>
>> On Thu, Mar 25, 2010 at 2:32 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>>
>>> I am using scipy.polyfit to fit a curve to my data. ?Members of this
>>> list have been integral in my understanding of how to use this
>>> function. ?Now I would like to know how I can get the uncertainties
>>> (standard deviations) of polynomial coefficients from the returned
>>> values from scipy.polyfit. ?If I understand correctly, the residuals
>>> are sometimes called the R^2 error, right? ?That gives an estimate of
>>> how well we have fit the data. ?I don't know how to use rank or any of
>>> the other returned values to get the uncertainties.
>>>
>>> Can someone please help?
>>
>> You want the covariance of the coefficients, (A.T * A)^-1/var, where A is
>> the design matrix. I'd have to see what the scipy fit returns to tell you
>> more. In anycase, from that you can plot curves at +/- sigma to show the
>> error bounds on the result. I can be more explicit if you want.
>>
>> Chuck
>
> Thanks Chuck. ?That seems to get closer to what I need. ?I am just
> fitting data to a polynomial, nothing too fancy. ?I would like the
> variance (not the covariance) of the coeffficients. ?As far as what is
> returned from scipy.polyfit this is what the documentation says:

Just an important warning: for polynomials, especially high degree
polynomials, the coefficients are an awful way to specify them. If
what you really need is the coefficients, then you're stuck with it,
but if what you need is to estimate interpolated values, there are
better approaches.

I mention this in particular because you talk about needing "variance
rather than covariance". When you do a linear fit of a
several-parameter model to data with normal errors, the uncertainty in
the fitted parameters is a multidimensional Gaussian - that is, the
(say) one-sigma error region is a multidimensional ellipsoid. If you
chose a good parameterization for your model, this ellipsoid will line
up nicely with the axes, and you can describe it by its size along
each axis: that is, you only need a variance in each parameter.

But if your parameterization is not good, then this ellipse will be
some tilted long skinny shape, and taking its size along one axis can
drastically underestimate its size. You also have the problem that
there are correlations in your parameters: if one is high, say, then
the other is also likely to be high. The covariance matrix captures
all this information along with the variances you asked for. As you
might expect, life is vastly simpler if the ellipsoid is aligned with
the axes, so that this matrix is nearly diagonal and you can just read
the variances off the diagonal.

Unfortunately, the parameterization of polynomials by their
coefficients is not "good" in this sense: you almost always get very
long skinny non-aligned error ellipses. An easy example is if you're
fitting positive x,y pairs with a straight line y=mx+b. Then if m is
too low, b is almost certainly too high, since the uncertainty pivots
the line around the middle of the data set. With linear data you can
help this by using y = m(x-x_middle)+b, but when you go to
higher-order polynomials it all becomes messier and you can't usually
cure the problems this easily.

I would say, look closely at your problem and think about whether you
really need the polynomial coefficients.

Anne

> residuals, rank, singular_values, rcond : present only if `full` = True
> ? ? ? ?Residuals of the least-squares fit, the effective rank of the scaled
> ? ? ? ?Vandermonde coefficient matrix, its singular values, and the specified
> ? ? ? ?value of `rcond`. For more details, see `linalg.lstsq`.
>
> I don't think any of these things are "design matrix" as you have
> indicated I need. ?The documentation for linalg.lstsq does not say
> what rcond is unfortunately. ? Any ideas?
>
> Jeremy
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From jlconlin at gmail.com  Thu Mar 25 18:40:06 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Thu, 25 Mar 2010 16:40:06 -0600
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
Message-ID: <2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>

On Thu, Mar 25, 2010 at 4:21 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> On 25 March 2010 17:20, Jeremy Conlin <jlconlin at gmail.com> wrote:
>> On Thu, Mar 25, 2010 at 2:52 PM, Charles R Harris
>> <charlesr.harris at gmail.com> wrote:
>>>
>>>
>>> On Thu, Mar 25, 2010 at 2:32 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>>>
>>>> I am using scipy.polyfit to fit a curve to my data. ?Members of this
>>>> list have been integral in my understanding of how to use this
>>>> function. ?Now I would like to know how I can get the uncertainties
>>>> (standard deviations) of polynomial coefficients from the returned
>>>> values from scipy.polyfit. ?If I understand correctly, the residuals
>>>> are sometimes called the R^2 error, right? ?That gives an estimate of
>>>> how well we have fit the data. ?I don't know how to use rank or any of
>>>> the other returned values to get the uncertainties.
>>>>
>>>> Can someone please help?
>>>
>>> You want the covariance of the coefficients, (A.T * A)^-1/var, where A is
>>> the design matrix. I'd have to see what the scipy fit returns to tell you
>>> more. In anycase, from that you can plot curves at +/- sigma to show the
>>> error bounds on the result. I can be more explicit if you want.
>>>
>>> Chuck
>>
>> Thanks Chuck. ?That seems to get closer to what I need. ?I am just
>> fitting data to a polynomial, nothing too fancy. ?I would like the
>> variance (not the covariance) of the coeffficients. ?As far as what is
>> returned from scipy.polyfit this is what the documentation says:
>
> Just an important warning: for polynomials, especially high degree
> polynomials, the coefficients are an awful way to specify them. If
> what you really need is the coefficients, then you're stuck with it,
> but if what you need is to estimate interpolated values, there are
> better approaches.
>
> I mention this in particular because you talk about needing "variance
> rather than covariance". When you do a linear fit of a
> several-parameter model to data with normal errors, the uncertainty in
> the fitted parameters is a multidimensional Gaussian - that is, the
> (say) one-sigma error region is a multidimensional ellipsoid. If you
> chose a good parameterization for your model, this ellipsoid will line
> up nicely with the axes, and you can describe it by its size along
> each axis: that is, you only need a variance in each parameter.
>
> But if your parameterization is not good, then this ellipse will be
> some tilted long skinny shape, and taking its size along one axis can
> drastically underestimate its size. You also have the problem that
> there are correlations in your parameters: if one is high, say, then
> the other is also likely to be high. The covariance matrix captures
> all this information along with the variances you asked for. As you
> might expect, life is vastly simpler if the ellipsoid is aligned with
> the axes, so that this matrix is nearly diagonal and you can just read
> the variances off the diagonal.
>
> Unfortunately, the parameterization of polynomials by their
> coefficients is not "good" in this sense: you almost always get very
> long skinny non-aligned error ellipses. An easy example is if you're
> fitting positive x,y pairs with a straight line y=mx+b. Then if m is
> too low, b is almost certainly too high, since the uncertainty pivots
> the line around the middle of the data set. With linear data you can
> help this by using y = m(x-x_middle)+b, but when you go to
> higher-order polynomials it all becomes messier and you can't usually
> cure the problems this easily.
>
> I would say, look closely at your problem and think about whether you
> really need the polynomial coefficients.

Yikes!  This sounds like it may be more trouble than it's worth.  I
have a few sets of statistical data that each need to have curves fit
to them.  I can see what the parameters are and they are similar
between each set.  I want to know if the coefficients are within a few
standard deviations of each other.  That's what I was hoping to get
at, but it seems to be more trouble than I'm willing to accept.
Unless anyone has a simpler solution.

Thanks,
Jeremy
>
> Anne
>
>> residuals, rank, singular_values, rcond : present only if `full` = True
>> ? ? ? ?Residuals of the least-squares fit, the effective rank of the scaled
>> ? ? ? ?Vandermonde coefficient matrix, its singular values, and the specified
>> ? ? ? ?value of `rcond`. For more details, see `linalg.lstsq`.
>>
>> I don't think any of these things are "design matrix" as you have
>> indicated I need. ?The documentation for linalg.lstsq does not say
>> what rcond is unfortunately. ? Any ideas?
>>
>> Jeremy
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From peridot.faceted at gmail.com  Thu Mar 25 18:45:56 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 25 Mar 2010 18:45:56 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
Message-ID: <ce557a361003251545m3e75a45cv82b941472c7e48c7@mail.gmail.com>

On 25 March 2010 18:40, Jeremy Conlin <jlconlin at gmail.com> wrote:
> On Thu, Mar 25, 2010 at 4:21 PM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
>> On 25 March 2010 17:20, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>> On Thu, Mar 25, 2010 at 2:52 PM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>>>
>>>>
>>>> On Thu, Mar 25, 2010 at 2:32 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>>>>
>>>>> I am using scipy.polyfit to fit a curve to my data. ?Members of this
>>>>> list have been integral in my understanding of how to use this
>>>>> function. ?Now I would like to know how I can get the uncertainties
>>>>> (standard deviations) of polynomial coefficients from the returned
>>>>> values from scipy.polyfit. ?If I understand correctly, the residuals
>>>>> are sometimes called the R^2 error, right? ?That gives an estimate of
>>>>> how well we have fit the data. ?I don't know how to use rank or any of
>>>>> the other returned values to get the uncertainties.
>>>>>
>>>>> Can someone please help?
>>>>
>>>> You want the covariance of the coefficients, (A.T * A)^-1/var, where A is
>>>> the design matrix. I'd have to see what the scipy fit returns to tell you
>>>> more. In anycase, from that you can plot curves at +/- sigma to show the
>>>> error bounds on the result. I can be more explicit if you want.
>>>>
>>>> Chuck
>>>
>>> Thanks Chuck. ?That seems to get closer to what I need. ?I am just
>>> fitting data to a polynomial, nothing too fancy. ?I would like the
>>> variance (not the covariance) of the coeffficients. ?As far as what is
>>> returned from scipy.polyfit this is what the documentation says:
>>
>> Just an important warning: for polynomials, especially high degree
>> polynomials, the coefficients are an awful way to specify them. If
>> what you really need is the coefficients, then you're stuck with it,
>> but if what you need is to estimate interpolated values, there are
>> better approaches.
>>
>> I mention this in particular because you talk about needing "variance
>> rather than covariance". When you do a linear fit of a
>> several-parameter model to data with normal errors, the uncertainty in
>> the fitted parameters is a multidimensional Gaussian - that is, the
>> (say) one-sigma error region is a multidimensional ellipsoid. If you
>> chose a good parameterization for your model, this ellipsoid will line
>> up nicely with the axes, and you can describe it by its size along
>> each axis: that is, you only need a variance in each parameter.
>>
>> But if your parameterization is not good, then this ellipse will be
>> some tilted long skinny shape, and taking its size along one axis can
>> drastically underestimate its size. You also have the problem that
>> there are correlations in your parameters: if one is high, say, then
>> the other is also likely to be high. The covariance matrix captures
>> all this information along with the variances you asked for. As you
>> might expect, life is vastly simpler if the ellipsoid is aligned with
>> the axes, so that this matrix is nearly diagonal and you can just read
>> the variances off the diagonal.
>>
>> Unfortunately, the parameterization of polynomials by their
>> coefficients is not "good" in this sense: you almost always get very
>> long skinny non-aligned error ellipses. An easy example is if you're
>> fitting positive x,y pairs with a straight line y=mx+b. Then if m is
>> too low, b is almost certainly too high, since the uncertainty pivots
>> the line around the middle of the data set. With linear data you can
>> help this by using y = m(x-x_middle)+b, but when you go to
>> higher-order polynomials it all becomes messier and you can't usually
>> cure the problems this easily.
>>
>> I would say, look closely at your problem and think about whether you
>> really need the polynomial coefficients.
>
> Yikes! ?This sounds like it may be more trouble than it's worth. ?I
> have a few sets of statistical data that each need to have curves fit
> to them. ?I can see what the parameters are and they are similar
> between each set. ?I want to know if the coefficients are within a few
> standard deviations of each other. ?That's what I was hoping to get
> at, but it seems to be more trouble than I'm willing to accept.
> Unless anyone has a simpler solution.

You could try using the chebyshev polynomials; they're in the new
numpy, or you can just use the formulas (the degree-n one is cos(n
arccos(x)), and they're sensible on [0,1]). They're not perfect, but
they're a much better representation of polynomials that gets rid of
most of the problems I described above (the covariances tend to be
much less).

Anne

> Thanks,
> Jeremy
>>
>> Anne
>>
>>> residuals, rank, singular_values, rcond : present only if `full` = True
>>> ? ? ? ?Residuals of the least-squares fit, the effective rank of the scaled
>>> ? ? ? ?Vandermonde coefficient matrix, its singular values, and the specified
>>> ? ? ? ?value of `rcond`. For more details, see `linalg.lstsq`.
>>>
>>> I don't think any of these things are "design matrix" as you have
>>> indicated I need. ?The documentation for linalg.lstsq does not say
>>> what rcond is unfortunately. ? Any ideas?
>>>
>>> Jeremy
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From pgmdevlist at gmail.com  Wed Mar 24 12:27:59 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Wed, 24 Mar 2010 11:27:59 -0500
Subject: [SciPy-User] Scikits TimeSeries modifies names of features
	(dtypes
In-Reply-To: <2d88eb101003240743y12893e24iff3ad29ad52f5866@mail.gmail.com>
References: <2d88eb101003240743y12893e24iff3ad29ad52f5866@mail.gmail.com>
Message-ID: <5517C8CE-6024-41AC-929D-391D5BF48A48@gmail.com>

On Mar 24, 2010, at 9:43 AM, Sergi Pons Freixes wrote:
> I have this code, which simply creates a TimeSeries object from a datafile:
> 
> data_labels = ["Temp", "Sal", "O2"] + self.species.keys()
> tmpdata = ts.tsfromtxt(join(dir,f), freq='D', delimiter='\t', \
>              usecols=range(len(data_labels)), datecols=0, \
>              names=data_labels, dateconverter=self.__dateconverter)
> 
> where self.species.keys() is a list of strings, some of them with
> spaces, points, or parenthesis. A pair of examples: "Phaeocystis
> (colonies)", "Agr.fil. G. corsicum 200x".
> 
> But, If I check the resulting tmpdata.dtypes.names (or simply
> tmpdata.dtypes), all the spaces has been changed to underscore, and
> the points and parenthesis removed. Examples: "Phaeocystis_colonies",
> "Agrfil_G_corsicum_200x".
> 
> I've checked the documentation from timeseries and from record arrays
> of numpy, and cannot found the reason... Any hint abou it?


Actually, this is not a tsfromtxt issue, but a np.genfromtxt one (the main function tsfromtxt is derived from). By giving names in the input, you indicate that you want a structured array (with named fields). Structured arrays can also be viewed as recarrays, where named fields can be accessed as attributes. Obviously, having a space, a comma, a period or any other non-alphanumeric character in the field names will prevent the access by attribute to work. For that reason, np.genfromtxt checks whether the names of the fields are valid, and modify them if they're not (by replacing any non-alphanumeric character by an underscore). Think of it as a feature, not a bug.
Now, I can agree that force-feeding this attempt of foolproof behavior to the user is a bit controlling. After all, if you decided to use a name like "Agr.fil. G. corsicum 200x" as field name, you must have a very good reason, and won't mind not being able to use recarrays. So, what I can do is leave the automatic validation of names as the default, but introduce an option to bypass it. That'd be OK for everybody ?

From charlesr.harris at gmail.com  Thu Mar 25 20:17:02 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 25 Mar 2010 18:17:02 -0600
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
Message-ID: <e06186141003251717o56f8d549q482b5fe2742221bd@mail.gmail.com>

On Thu, Mar 25, 2010 at 4:40 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:

> On Thu, Mar 25, 2010 at 4:21 PM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
> > On 25 March 2010 17:20, Jeremy Conlin <jlconlin at gmail.com> wrote:
> >> On Thu, Mar 25, 2010 at 2:52 PM, Charles R Harris
> >> <charlesr.harris at gmail.com> wrote:
> >>>
> >>>
> >>> On Thu, Mar 25, 2010 at 2:32 PM, Jeremy Conlin <jlconlin at gmail.com>
> wrote:
> >>>>
> >>>> I am using scipy.polyfit to fit a curve to my data.  Members of this
> >>>> list have been integral in my understanding of how to use this
> >>>> function.  Now I would like to know how I can get the uncertainties
> >>>> (standard deviations) of polynomial coefficients from the returned
> >>>> values from scipy.polyfit.  If I understand correctly, the residuals
> >>>> are sometimes called the R^2 error, right?  That gives an estimate of
> >>>> how well we have fit the data.  I don't know how to use rank or any of
> >>>> the other returned values to get the uncertainties.
> >>>>
> >>>> Can someone please help?
> >>>
> >>> You want the covariance of the coefficients, (A.T * A)^-1/var, where A
> is
> >>> the design matrix. I'd have to see what the scipy fit returns to tell
> you
> >>> more. In anycase, from that you can plot curves at +/- sigma to show
> the
> >>> error bounds on the result. I can be more explicit if you want.
> >>>
> >>> Chuck
> >>
> >> Thanks Chuck.  That seems to get closer to what I need.  I am just
> >> fitting data to a polynomial, nothing too fancy.  I would like the
> >> variance (not the covariance) of the coeffficients.  As far as what is
> >> returned from scipy.polyfit this is what the documentation says:
> >
> > Just an important warning: for polynomials, especially high degree
> > polynomials, the coefficients are an awful way to specify them. If
> > what you really need is the coefficients, then you're stuck with it,
> > but if what you need is to estimate interpolated values, there are
> > better approaches.
> >
> > I mention this in particular because you talk about needing "variance
> > rather than covariance". When you do a linear fit of a
> > several-parameter model to data with normal errors, the uncertainty in
> > the fitted parameters is a multidimensional Gaussian - that is, the
> > (say) one-sigma error region is a multidimensional ellipsoid. If you
> > chose a good parameterization for your model, this ellipsoid will line
> > up nicely with the axes, and you can describe it by its size along
> > each axis: that is, you only need a variance in each parameter.
> >
> > But if your parameterization is not good, then this ellipse will be
> > some tilted long skinny shape, and taking its size along one axis can
> > drastically underestimate its size. You also have the problem that
> > there are correlations in your parameters: if one is high, say, then
> > the other is also likely to be high. The covariance matrix captures
> > all this information along with the variances you asked for. As you
> > might expect, life is vastly simpler if the ellipsoid is aligned with
> > the axes, so that this matrix is nearly diagonal and you can just read
> > the variances off the diagonal.
> >
> > Unfortunately, the parameterization of polynomials by their
> > coefficients is not "good" in this sense: you almost always get very
> > long skinny non-aligned error ellipses. An easy example is if you're
> > fitting positive x,y pairs with a straight line y=mx+b. Then if m is
> > too low, b is almost certainly too high, since the uncertainty pivots
> > the line around the middle of the data set. With linear data you can
> > help this by using y = m(x-x_middle)+b, but when you go to
> > higher-order polynomials it all becomes messier and you can't usually
> > cure the problems this easily.
> >
> > I would say, look closely at your problem and think about whether you
> > really need the polynomial coefficients.
>
> Yikes!  This sounds like it may be more trouble than it's worth.  I
> have a few sets of statistical data that each need to have curves fit
> to them.  I can see what the parameters are and they are similar
> between each set.  I want to know if the coefficients are within a few
> standard deviations of each other.  That's what I was hoping to get
> at, but it seems to be more trouble than I'm willing to accept.
> Unless anyone has a simpler solution.
>
> Thanks,
> Jeremy
> >
> > Anne
> >
> >> residuals, rank, singular_values, rcond : present only if `full` = True
> >>        Residuals of the least-squares fit, the effective rank of the
> scaled
> >>        Vandermonde coefficient matrix, its singular values, and the
> specified
> >>        value of `rcond`. For more details, see `linalg.lstsq`.
> >>
> >> I don't think any of these things are "design matrix" as you have
> >> indicated I need.  The documentation for linalg.lstsq does not say
> >> what rcond is unfortunately.   Any ideas?
> >>
>

lstsq is a bit deficient in that regard, it doesn't return all the info you
need. The design matrix is the matrix A in the equation Ac=y that you want
to solve using least squares. In the case of polynomial fits it is the
Vandermode matrix of the desired degree given by vander(x, ndeg + 1), where
x is the vector of the sample points. To get error estimates you can either
just use the package Josef mentioned or you can go ahead and roll your own,
which is what I generally do. If your sample points are reasonably
distributed and the fit is of low degree you can explicitly compute dot(A.T,
A).inv and multiply it by the residual/(number of data points - number of
coefficients). That gives the covariance, the square root of its diagonal is
the estimated standard deviation of the coefficients. Note that this method
of estimating the variance from the residual isn't very accurate unless you
have a lot of data points, it's better to have independent estimates of the
measurement errors if you can manage that.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/6ac9676c/attachment.html>

From josef.pktd at gmail.com  Thu Mar 25 20:23:51 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 25 Mar 2010 20:23:51 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <ce557a361003251545m3e75a45cv82b941472c7e48c7@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<ce557a361003251545m3e75a45cv82b941472c7e48c7@mail.gmail.com>
Message-ID: <1cd32cbb1003251723m5804d8d0o4d02d0359e3fb31e@mail.gmail.com>

On Thu, Mar 25, 2010 at 6:45 PM, Anne Archibald
<peridot.faceted at gmail.com> wrote:
> On 25 March 2010 18:40, Jeremy Conlin <jlconlin at gmail.com> wrote:
>> On Thu, Mar 25, 2010 at 4:21 PM, Anne Archibald
>> <peridot.faceted at gmail.com> wrote:
>>> On 25 March 2010 17:20, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>>> On Thu, Mar 25, 2010 at 2:52 PM, Charles R Harris
>>>> <charlesr.harris at gmail.com> wrote:
>>>>>
>>>>>
>>>>> On Thu, Mar 25, 2010 at 2:32 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>>>>>
>>>>>> I am using scipy.polyfit to fit a curve to my data. ?Members of this
>>>>>> list have been integral in my understanding of how to use this
>>>>>> function. ?Now I would like to know how I can get the uncertainties
>>>>>> (standard deviations) of polynomial coefficients from the returned
>>>>>> values from scipy.polyfit. ?If I understand correctly, the residuals
>>>>>> are sometimes called the R^2 error, right? ?That gives an estimate of
>>>>>> how well we have fit the data. ?I don't know how to use rank or any of
>>>>>> the other returned values to get the uncertainties.
>>>>>>
>>>>>> Can someone please help?
>>>>>
>>>>> You want the covariance of the coefficients, (A.T * A)^-1/var, where A is
>>>>> the design matrix. I'd have to see what the scipy fit returns to tell you
>>>>> more. In anycase, from that you can plot curves at +/- sigma to show the
>>>>> error bounds on the result. I can be more explicit if you want.
>>>>>
>>>>> Chuck
>>>>
>>>> Thanks Chuck. ?That seems to get closer to what I need. ?I am just
>>>> fitting data to a polynomial, nothing too fancy. ?I would like the
>>>> variance (not the covariance) of the coeffficients. ?As far as what is
>>>> returned from scipy.polyfit this is what the documentation says:
>>>
>>> Just an important warning: for polynomials, especially high degree
>>> polynomials, the coefficients are an awful way to specify them. If
>>> what you really need is the coefficients, then you're stuck with it,
>>> but if what you need is to estimate interpolated values, there are
>>> better approaches.
>>>
>>> I mention this in particular because you talk about needing "variance
>>> rather than covariance". When you do a linear fit of a
>>> several-parameter model to data with normal errors, the uncertainty in
>>> the fitted parameters is a multidimensional Gaussian - that is, the
>>> (say) one-sigma error region is a multidimensional ellipsoid. If you
>>> chose a good parameterization for your model, this ellipsoid will line
>>> up nicely with the axes, and you can describe it by its size along
>>> each axis: that is, you only need a variance in each parameter.
>>>
>>> But if your parameterization is not good, then this ellipse will be
>>> some tilted long skinny shape, and taking its size along one axis can
>>> drastically underestimate its size. You also have the problem that
>>> there are correlations in your parameters: if one is high, say, then
>>> the other is also likely to be high. The covariance matrix captures
>>> all this information along with the variances you asked for. As you
>>> might expect, life is vastly simpler if the ellipsoid is aligned with
>>> the axes, so that this matrix is nearly diagonal and you can just read
>>> the variances off the diagonal.
>>>
>>> Unfortunately, the parameterization of polynomials by their
>>> coefficients is not "good" in this sense: you almost always get very
>>> long skinny non-aligned error ellipses. An easy example is if you're
>>> fitting positive x,y pairs with a straight line y=mx+b. Then if m is
>>> too low, b is almost certainly too high, since the uncertainty pivots
>>> the line around the middle of the data set. With linear data you can
>>> help this by using y = m(x-x_middle)+b, but when you go to
>>> higher-order polynomials it all becomes messier and you can't usually
>>> cure the problems this easily.
>>>
>>> I would say, look closely at your problem and think about whether you
>>> really need the polynomial coefficients.
>>
>> Yikes! ?This sounds like it may be more trouble than it's worth. ?I
>> have a few sets of statistical data that each need to have curves fit
>> to them. ?I can see what the parameters are and they are similar
>> between each set. ?I want to know if the coefficients are within a few
>> standard deviations of each other. ?That's what I was hoping to get
>> at, but it seems to be more trouble than I'm willing to accept.
>> Unless anyone has a simpler solution.
>
> You could try using the chebyshev polynomials; they're in the new
> numpy, or you can just use the formulas (the degree-n one is cos(n
> arccos(x)), and they're sensible on [0,1]). They're not perfect, but
> they're a much better representation of polynomials that gets rid of
> most of the problems I described above (the covariances tend to be
> much less).

I don't think that multicollinearity matters so much when all
coefficients included in the null hypothesis, because the test is
based on the entire covariance of the parameter estimates and not just
the variance, e.g. with F-test.

One of the first references for testing equality of regression
coefficients across regressions that google found is

http://www.stata.com/support/faqs/stat/testing.html

Josef
(google is swallowing my sent emails today, so it will come up double
or not at all)

>
> Anne
>
>> Thanks,
>> Jeremy
>>>
>>> Anne
>>>
>>>> residuals, rank, singular_values, rcond : present only if `full` = True
>>>> ? ? ? ?Residuals of the least-squares fit, the effective rank of the scaled
>>>> ? ? ? ?Vandermonde coefficient matrix, its singular values, and the specified
>>>> ? ? ? ?value of `rcond`. For more details, see `linalg.lstsq`.
>>>>
>>>> I don't think any of these things are "design matrix" as you have
>>>> indicated I need. ?The documentation for linalg.lstsq does not say
>>>> what rcond is unfortunately. ? Any ideas?
>>>>
>>>> Jeremy
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From d.l.goldsmith at gmail.com  Thu Mar 25 23:40:46 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Thu, 25 Mar 2010 20:40:46 -0700
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
Message-ID: <45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>

On Thu, Mar 25, 2010 at 3:40 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:

> Yikes!  This sounds like it may be more trouble than it's worth.  I
> have a few sets of statistical data that each need to have curves fit
> to them.


That's an awfully generic need - it may be obvious from examination of the
data that a line is inappropriate, but besides polynomials there are many
other non-linear models (which can be linearly fit to data by means of data
transformation) which possess fewer parameters (and thus are simpler from a
parameter analysis perspective).  So, the question is: why are you fitting
to polynomials?  If it's just to get a good fit to the data, you might be
getting "more fit" than your data warrants (and even if that isn't a
problem, you probably want to use a polynomial form different from "standard
form," e.g., Lagrange interpolators).  Are you sure something like an
exponential growth/decay or power law model (both of which are "more
natural," linearizable, two-parameter models) wouldn't be more appropriate -
it would almost certainly be simpler to analyze (and perhaps easier to
justify to a referee).

On this note, perhaps some of our experts might care to comment: what
"physics" (in a generalized sense) gives rise to a polynomial dependency of
degree higher than two?  The only generic thing I can think of is something
where third or higher order derivatives proportional to the independent
variable are important, and those are pretty uncommon.

DG


> I can see what the parameters are and they are similar
> between each set.  I want to know if the coefficients are within a few
> standard deviations of each other.  That's what I was hoping to get
> at, but it seems to be more trouble than I'm willing to accept.
> Unless anyone has a simpler solution.
>
> Thanks,
> Jeremy
> >
> > Anne
> >
> >> residuals, rank, singular_values, rcond : present only if `full` = True
> >>        Residuals of the least-squares fit, the effective rank of the
> scaled
> >>        Vandermonde coefficient matrix, its singular values, and the
> specified
> >>        value of `rcond`. For more details, see `linalg.lstsq`.
> >>
> >> I don't think any of these things are "design matrix" as you have
> >> indicated I need.  The documentation for linalg.lstsq does not say
> >> what rcond is unfortunately.   Any ideas?
> >>
> >> Jeremy
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/77111824/attachment.html>

From josef.pktd at gmail.com  Fri Mar 26 00:10:37 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 26 Mar 2010 00:10:37 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
Message-ID: <1cd32cbb1003252110h3432366dyc2f8f8105c7cd096@mail.gmail.com>

On Thu, Mar 25, 2010 at 11:40 PM, David Goldsmith
<d.l.goldsmith at gmail.com> wrote:
> On Thu, Mar 25, 2010 at 3:40 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>
>> Yikes! ?This sounds like it may be more trouble than it's worth. ?I
>> have a few sets of statistical data that each need to have curves fit
>> to them.
>
> That's an awfully generic need - it may be obvious from examination of the
> data that a line is inappropriate, but besides polynomials there are many
> other non-linear models (which can be linearly fit to data by means of data
> transformation) which possess fewer parameters (and thus are simpler from a
> parameter analysis perspective).? So, the question is: why are you fitting
> to polynomials?? If it's just to get a good fit to the data, you might be
> getting "more fit" than your data warrants (and even if that isn't a
> problem, you probably want to use a polynomial form different from "standard
> form," e.g., Lagrange interpolators).? Are you sure something like an
> exponential growth/decay or power law model (both of which are "more
> natural," linearizable, two-parameter models) wouldn't be more appropriate -
> it would almost certainly be simpler to analyze (and perhaps easier to
> justify to a referee).
>
> On this note, perhaps some of our experts might care to comment: what
> "physics" (in a generalized sense) gives rise to a polynomial dependency of
> degree higher than two?? The only generic thing I can think of is something
> where third or higher order derivatives proportional to the independent
> variable are important, and those are pretty uncommon.

Taylor series expansion is a "natural" source for higher order
polynomials, although as Anne pointed out, it's not a "nice" base for
the space of functions. In most of the examples I have seen, 3rd order
was the highest, after that non-parametric, Fourier, Chebychev or
Hermite.

I'm attaching the statsmodels equivalent of most of the example in the
Stata FAQ. A few parts are (still) a bit more complicated than the
Stata equivalent.
Final result: F-test rejects null of equal coefficients (small
p-value) if the coefficients are far enough apart, and does not reject
if the two regression equations are samples from the same data
generating process.

Josef


>
> DG
>
>>
>> I can see what the parameters are and they are similar
>> between each set. ?I want to know if the coefficients are within a few
>> standard deviations of each other. ?That's what I was hoping to get
>> at, but it seems to be more trouble than I'm willing to accept.
>> Unless anyone has a simpler solution.
>>
>> Thanks,
>> Jeremy
>> >
>> > Anne
>> >
>> >> residuals, rank, singular_values, rcond : present only if `full` = True
>> >> ? ? ? ?Residuals of the least-squares fit, the effective rank of the
>> >> scaled
>> >> ? ? ? ?Vandermonde coefficient matrix, its singular values, and the
>> >> specified
>> >> ? ? ? ?value of `rcond`. For more details, see `linalg.lstsq`.
>> >>
>> >> I don't think any of these things are "design matrix" as you have
>> >> indicated I need. ?The documentation for linalg.lstsq does not say
>> >> what rcond is unfortunately. ? Any ideas?
>> >>
>> >> Jeremy
>> >> _______________________________________________
>> >> SciPy-User mailing list
>> >> SciPy-User at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/scipy-user
>> >>
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
# -*- coding: utf-8 -*-
"""F test for null hypothesis that coefficients in two regressions are the same

Created on Thu Mar 25 22:56:45 2010
Author: josef-pktd
"""

import numpy as np
from numpy.testing import assert_almost_equal
import scikits.statsmodels as sm

nobs = 10 #100
x1 = np.random.randn(nobs)
y1 = 10 + 15*x1 + 2*np.random.randn(nobs)

x1 = sm.add_constant(x1) #, prepend=True)
assert_almost_equal(x1, np.vander(x1[:,0],2), 16)
res1 = sm.OLS(y1, x1).fit()
print res1.params
print np.polyfit(x1[:,0], y1, 1)
assert_almost_equal(res1.params, np.polyfit(x1[:,0], y1, 1), 14)
print res1.summary(xname=['x1','const1'])

#regression 2
x2 = np.random.randn(nobs)
y2 = 19 + 17*x2 + 2*np.random.randn(nobs)
#y2 = 10 + 15*x2 + 2*np.random.randn(nobs)  # if H0 is true

x2 = sm.add_constant(x2) #, prepend=True)
assert_almost_equal(x2, np.vander(x2[:,0],2), 16)

res2 = sm.OLS(y2, x2).fit()
print res2.params
print np.polyfit(x2[:,0], y2, 1)
assert_almost_equal(res2.params, np.polyfit(x2[:,0], y2, 1), 14)
print res2.summary(xname=['x2','const2'])


# joint regression

x = np.concatenate((x1,x2),0)
y = np.concatenate((y1,y2))
dummy = np.arange(2*nobs)>nobs-1
x = np.column_stack((x,x*dummy[:,None]))

res = sm.OLS(y, x).fit()
print res.summary(xname=['x','const','x2','const2'])

print '\nF test for equal coefficients in 2 regression equations'
#effect of dummy times second regression is zero
#is equivalent to 3rd and 4th coefficient are both zero
print res.f_test([[0,0,1,0],[0,0,0,1]])

print '\nchecking coefficients individual versus joint'
print res1.params, res2.params
print res.params[:2], res.params[:2]+res.params[2:]
assert_almost_equal(res1.params, res.params[:2], 13)
assert_almost_equal(res2.params, res.params[:2]+res.params[2:], 13)

From charlesr.harris at gmail.com  Fri Mar 26 00:14:25 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 25 Mar 2010 22:14:25 -0600
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
Message-ID: <e06186141003252114x3157fae4ye5b6ac60e5d5bbdb@mail.gmail.com>

On Thu, Mar 25, 2010 at 9:40 PM, David Goldsmith <d.l.goldsmith at gmail.com>wrote:

> On Thu, Mar 25, 2010 at 3:40 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>
>> Yikes!  This sounds like it may be more trouble than it's worth.  I
>> have a few sets of statistical data that each need to have curves fit
>> to them.
>
>
> That's an awfully generic need - it may be obvious from examination of the
> data that a line is inappropriate, but besides polynomials there are many
> other non-linear models (which can be linearly fit to data by means of data
> transformation) which possess fewer parameters (and thus are simpler from a
> parameter analysis perspective).  So, the question is: why are you fitting
> to polynomials?  If it's just to get a good fit to the data, you might be
> getting "more fit" than your data warrants (and even if that isn't a
> problem, you probably want to use a polynomial form different from "standard
> form," e.g., Lagrange interpolators).  Are you sure something like an
> exponential growth/decay or power law model (both of which are "more
> natural," linearizable, two-parameter models) wouldn't be more appropriate -
> it would almost certainly be simpler to analyze (and perhaps easier to
> justify to a referee).
>
> On this note, perhaps some of our experts might care to comment: what
> "physics" (in a generalized sense) gives rise to a polynomial dependency of
> degree higher than two?  The only generic thing I can think of is something
> where third or higher order derivatives proportional to the independent
> variable are important, and those are pretty uncommon.
>
>
Oh, there are lots of things. Detector responses, platinum resistance
thermometers, optical aberrations, so on and so forth. Polynomials aren't
always appropriate and they are usually useless for extrapolation, but they
are often invaluable for getting a compact representation of the data.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100325/843029b9/attachment.html>

From graziano.mirata at lu.unisi.ch  Fri Mar 26 07:10:32 2010
From: graziano.mirata at lu.unisi.ch (Graziano Mirata)
Date: Fri, 26 Mar 2010 12:10:32 +0100
Subject: [SciPy-User] Again with matrices multiplication
Message-ID: <189976BA-0A25-480D-9970-C283152D6B2F@lu.unisi.ch>

Hi guys, 
it's me again. Still problems with matrices multiplication. 

So i have used dot(w,R) to multiply two arrays but now i cannot access to elements. I.e 

In [224]: w.shape
Out[224]: (3,)
In [226]: R.shape
Out[226]: (24000, 3)

In [227]: r = dot(R,w)

In [228]: type(r)
Out[228]: <type 'numpy.ndarray'>

In [229]: r.shape
Out[229]: (24000,)

In [230]: r[0,0]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)

/<ipython console> in <module>()

IndexError: invalid index

In [231]: r[:,0]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)

/<ipython console> in <module>()

IndexError: invalid index

Thanx

G




From josef.pktd at gmail.com  Fri Mar 26 07:43:09 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 26 Mar 2010 07:43:09 -0400
Subject: [SciPy-User] Again with matrices multiplication
In-Reply-To: <189976BA-0A25-480D-9970-C283152D6B2F@lu.unisi.ch>
References: <189976BA-0A25-480D-9970-C283152D6B2F@lu.unisi.ch>
Message-ID: <1cd32cbb1003260443s3d17fe78v820abfb0691540d1@mail.gmail.com>

On Fri, Mar 26, 2010 at 7:10 AM, Graziano Mirata
<graziano.mirata at lu.unisi.ch> wrote:
> Hi guys,
> it's me again. Still problems with matrices multiplication.
>
> So i have used dot(w,R) to multiply two arrays but now i cannot access to elements. I.e
>
> In [224]: w.shape
> Out[224]: (3,)
> In [226]: R.shape
> Out[226]: (24000, 3)
>
> In [227]: r = dot(R,w)
>
> In [228]: type(r)
> Out[228]: <type 'numpy.ndarray'>
>
> In [229]: r.shape
> Out[229]: (24000,)

the shape of r means it is a 1 dimensional array

r[0] to get first element
r[:5] to get first five elements

Josef

>
> In [230]: r[0,0]
> ---------------------------------------------------------------------------
> IndexError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call last)
>
> /<ipython console> in <module>()
>
> IndexError: invalid index
>
> In [231]: r[:,0]
> ---------------------------------------------------------------------------
> IndexError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Traceback (most recent call last)
>
> /<ipython console> in <module>()
>
> IndexError: invalid index
>
> Thanx
>
> G
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From gunbuster363 at gmail.com  Fri Mar 26 08:33:52 2010
From: gunbuster363 at gmail.com (Lam Waiman)
Date: Fri, 26 Mar 2010 20:33:52 +0800
Subject: [SciPy-User] How to install scipy for opensuse 10.3?
In-Reply-To: <f6951adf1003251202s47ea8a17xcc618cd649cba0dc@mail.gmail.com>
References: <fc3b9ae01003250231x40b2515fi8fcca84793e98f51@mail.gmail.com>
	<f6951adf1003250740sf893ceav88f5fc44b943268f@mail.gmail.com>
	<fc3b9ae01003250829w36fcff9ahaf48548ac59d3d34@mail.gmail.com>
	<15106_1269543643_ZZg0U6IyMPQs8.00_f6951adf1003251200v69622a3dj701ae73e2081148a@mail.gmail.com>
	<f6951adf1003251202s47ea8a17xcc618cd649cba0dc@mail.gmail.com>
Message-ID: <fc3b9ae01003260533k3f606e72j91eff63b82f8dd48@mail.gmail.com>

I added the above linkas a repository.
I tried to installed scipy, then hell lots of dependencies problems
come out........
one of them is scipy need python 2.5........non-sense, I am using
python 2.6 already.......

what is your opinion?

2010/3/26 Joe Kington <jkington at wisc.edu>:
> Gah, meant to link to the base repo, but I managed to have the text show the
> base repo, and the link have the 64-bit repo...
> Here you go:
> http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/
>
> On Thu, Mar 25, 2010 at 2:00 PM, Joe Kington <jkington at wisc.edu> wrote:
>>
>> It's rather dated (scipy 0.6) , but I think this repo definitely has scipy
>> and should be compatible with 10.3.
>> http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/
>> Hope that helps,
>> -Joe
>>
>> On Thu, Mar 25, 2010 at 10:29 AM, Lam Waiman <gunbuster363 at gmail.com>
>> wrote:
>>>
>>> But I cannot see scipy in that folder........
>>> And I don't want to upgrade to 11.2 because I am not sure if my package
>>> in 10.3 still works.....I don't want to risk now....
>>>
>>> 2010/3/25 Joe Kington <jkington at wisc.edu>
>>>>
>>>> I'm pretty certain scipy and numpy are in the repos for OpenSuse 10.3.
>>>> (or at least they were a few years ago... Is there some reason you can't
>>>> upgrade to 11.2? They're definitely in the repos for 11.0, 11.1, and 11.2)
>>>>
>>>> Do you have the science repos enabled?? They may have dropped them for
>>>> 10.x, but you can always add the SEL_10 versions.? They should be
>>>> compatible. http://download.opensuse.org/repositories/science/SLE_10/
>>>>
>>>> Hope that helps,
>>>> -Joe
>>>>
>>>> On Thu, Mar 25, 2010 at 4:31 AM, Lam Waiman <gunbuster363 at gmail.com>
>>>> wrote:
>>>>>
>>>>> Dear all,
>>>>> ?? ? ? ? ? ? ?I want to install scipy for python usage. I see the
>>>>> installation isreally complex, can someone give on some advice on it?
>>>>> My system, opensuse 10.3 (i586)
>>>>> I am using python 2.6.4
>>>>> when I see the doc, it said it require LAPACK to be installed,
>>>>> then I download lapack and see inside, it also require lots of things
>>>>> including fortran compiler
>>>>> and I don't know whether the gcc42-fortran compiler in my comp is ok.
>>>>> I try to dl the g77 compiler, which someone said I will need it for
>>>>> LAPACK
>>>>> then it g77 only works with a gcc compiler of version 2.8, but I am
>>>>> using a gcc compiler of version 4.2.
>>>>> SO COMPLEX!
>>>>> Looking forwards to your reply, thanks.
>>>>> Raymond
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From gunbuster363 at gmail.com  Fri Mar 26 08:38:11 2010
From: gunbuster363 at gmail.com (Lam Waiman)
Date: Fri, 26 Mar 2010 20:38:11 +0800
Subject: [SciPy-User] How to install scipy for opensuse 10.3?
In-Reply-To: <fc3b9ae01003260533k3f606e72j91eff63b82f8dd48@mail.gmail.com>
References: <fc3b9ae01003250231x40b2515fi8fcca84793e98f51@mail.gmail.com>
	<f6951adf1003250740sf893ceav88f5fc44b943268f@mail.gmail.com>
	<fc3b9ae01003250829w36fcff9ahaf48548ac59d3d34@mail.gmail.com>
	<15106_1269543643_ZZg0U6IyMPQs8.00_f6951adf1003251200v69622a3dj701ae73e2081148a@mail.gmail.com>
	<f6951adf1003251202s47ea8a17xcc618cd649cba0dc@mail.gmail.com>
	<fc3b9ae01003260533k3f606e72j91eff63b82f8dd48@mail.gmail.com>
Message-ID: <fc3b9ae01003260538n262a2f23wc3ed672c6e765034@mail.gmail.com>

See the conflict list:
-------------------------------------------------
#### YaST2 conflicts list - generated 2010-03-26 20:47:14 ####

libblas3 cannot be installed due to missing dependencies
    There are no installable providers of libgfortran.so.1 for
libblas3-3.2.1-10.1.i586[rere]
    === libblas3-3.2.1-10.1.i586[rere] ===
        libblas3-3.2.1-10.1.i586[rere] will be installed by the user.
        libgcc42-4.2.1_20070724-17.i586 is needed by
libblas3-3.2.1-10.1.i586[rere] (libgcc_s.so.1)
        libblas3-3.2.1-10.1.i586[rere] is needed by
liblapack3-3.2.1-10.1.i586[rere] (libblas.so.3)
    (null)
    Conflict Resolution:
        ( ) do not install libblas3
        ( ) Ignore this requirement just here
liblapack3 cannot be installed due to missing dependencies
    There are no installable providers of libgfortran.so.1 for
liblapack3-3.2.1-10.1.i586[rere]
    === liblapack3-3.2.1-10.1.i586[rere] ===
        liblapack3-3.2.1-10.1.i586[rere] will be installed by the user.
        libblas3-3.2.1-10.1.i586[rere] is needed by
liblapack3-3.2.1-10.1.i586[rere] (libblas.so.3)
        liblapack3-3.2.1-10.1.i586[rere] is needed by
python-scipy-0.6.0-6.4.i586[rere] (liblapack.so.3)
    (null)
    Conflict Resolution:
        ( ) do not install liblapack3
        ( ) Ignore this requirement just here
python-scipy cannot be installed due to missing dependencies
    There are no installable providers of libgfortran.so.1 for
python-scipy-0.6.0-6.4.i586[rere]
    === python-scipy-0.6.0-6.4.i586[rere] ===
        python-scipy-0.6.0-6.4.i586[rere] will be installed by the user.
        libstdc++42-4.2.1_20070724-17.i586 is needed by
python-scipy-0.6.0-6.4.i586[rere] (libstdc++.so.6)
        fftw3-3.1.2-62.i586 is needed by
python-scipy-0.6.0-6.4.i586[rere] (libfftw3.so.3)
        liblapack3-3.2.1-10.1.i586[rere] is needed by
python-scipy-0.6.0-6.4.i586[rere] (liblapack.so.3)
        python-2.5.1-39.8.i586 is needed by
python-scipy-0.6.0-6.4.i586[rere] (python == 2.5.1-39.8)
    (null)
    Conflict Resolution:
        ( ) do not install python-scipy
        ( ) Ignore this requirement just here

#### YaST2 conflicts list END ###

2010/3/26 Lam Waiman <gunbuster363 at gmail.com>:
> I added the above linkas a repository.
> I tried to installed scipy, then hell lots of dependencies problems
> come out........
> one of them is scipy need python 2.5........non-sense, I am using
> python 2.6 already.......
>
> what is your opinion?
>
> 2010/3/26 Joe Kington <jkington at wisc.edu>:
>> Gah, meant to link to the base repo, but I managed to have the text show the
>> base repo, and the link have the 64-bit repo...
>> Here you go:
>> http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/
>>
>> On Thu, Mar 25, 2010 at 2:00 PM, Joe Kington <jkington at wisc.edu> wrote:
>>>
>>> It's rather dated (scipy 0.6) , but I think this repo definitely has scipy
>>> and should be compatible with 10.3.
>>> http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/
>>> Hope that helps,
>>> -Joe
>>>
>>> On Thu, Mar 25, 2010 at 10:29 AM, Lam Waiman <gunbuster363 at gmail.com>
>>> wrote:
>>>>
>>>> But I cannot see scipy in that folder........
>>>> And I don't want to upgrade to 11.2 because I am not sure if my package
>>>> in 10.3 still works.....I don't want to risk now....
>>>>
>>>> 2010/3/25 Joe Kington <jkington at wisc.edu>
>>>>>
>>>>> I'm pretty certain scipy and numpy are in the repos for OpenSuse 10.3.
>>>>> (or at least they were a few years ago... Is there some reason you can't
>>>>> upgrade to 11.2? They're definitely in the repos for 11.0, 11.1, and 11.2)
>>>>>
>>>>> Do you have the science repos enabled?? They may have dropped them for
>>>>> 10.x, but you can always add the SEL_10 versions.? They should be
>>>>> compatible. http://download.opensuse.org/repositories/science/SLE_10/
>>>>>
>>>>> Hope that helps,
>>>>> -Joe
>>>>>
>>>>> On Thu, Mar 25, 2010 at 4:31 AM, Lam Waiman <gunbuster363 at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Dear all,
>>>>>> ?? ? ? ? ? ? ?I want to install scipy for python usage. I see the
>>>>>> installation isreally complex, can someone give on some advice on it?
>>>>>> My system, opensuse 10.3 (i586)
>>>>>> I am using python 2.6.4
>>>>>> when I see the doc, it said it require LAPACK to be installed,
>>>>>> then I download lapack and see inside, it also require lots of things
>>>>>> including fortran compiler
>>>>>> and I don't know whether the gcc42-fortran compiler in my comp is ok.
>>>>>> I try to dl the g77 compiler, which someone said I will need it for
>>>>>> LAPACK
>>>>>> then it g77 only works with a gcc compiler of version 2.8, but I am
>>>>>> using a gcc compiler of version 4.2.
>>>>>> SO COMPLEX!
>>>>>> Looking forwards to your reply, thanks.
>>>>>> Raymond
>>>>>> _______________________________________________
>>>>>> SciPy-User mailing list
>>>>>> SciPy-User at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SciPy-User mailing list
>>>>> SciPy-User at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>


From m.abdollahi at gmail.com  Fri Mar 26 10:39:04 2010
From: m.abdollahi at gmail.com (Mohammad Abdollahi)
Date: Fri, 26 Mar 2010 15:39:04 +0100
Subject: [SciPy-User] MATLAB accumarray equivalent ?
Message-ID: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>

Hi,

Does anyone know what is the equivalent function of MATLAB's "accumarray()"
in scipy/numpy,  with the exact same functionality ?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100326/3488e742/attachment.html>

From josef.pktd at gmail.com  Fri Mar 26 10:47:33 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 26 Mar 2010 10:47:33 -0400
Subject: [SciPy-User] MATLAB accumarray equivalent ?
In-Reply-To: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>
References: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>
Message-ID: <1cd32cbb1003260747m89830a1vaec81702d8fb2581@mail.gmail.com>

On Fri, Mar 26, 2010 at 10:39 AM, Mohammad Abdollahi
<m.abdollahi at gmail.com> wrote:
> Hi,
>
> Does anyone know what is the equivalent function of MATLAB's "accumarray()"
> in scipy/numpy,? with the exact same functionality ?

A description or link would be helpful.
I never used accumarray
http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html

but it looks similar to np.bincount(labels, weights=values)

I don't know what the equivalent options are?

Josef

>
> Thanks
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From jkington at wisc.edu  Fri Mar 26 11:08:11 2010
From: jkington at wisc.edu (Joe Kington)
Date: Fri, 26 Mar 2010 10:08:11 -0500
Subject: [SciPy-User] How to install scipy for opensuse 10.3?
In-Reply-To: <fc3b9ae01003260538n262a2f23wc3ed672c6e765034@mail.gmail.com>
References: <fc3b9ae01003250231x40b2515fi8fcca84793e98f51@mail.gmail.com>
	<f6951adf1003250740sf893ceav88f5fc44b943268f@mail.gmail.com>
	<fc3b9ae01003250829w36fcff9ahaf48548ac59d3d34@mail.gmail.com>
	<15106_1269543643_ZZg0U6IyMPQs8.00_f6951adf1003251200v69622a3dj701ae73e2081148a@mail.gmail.com>
	<f6951adf1003251202s47ea8a17xcc618cd649cba0dc@mail.gmail.com>
	<fc3b9ae01003260533k3f606e72j91eff63b82f8dd48@mail.gmail.com>
	<fc3b9ae01003260538n262a2f23wc3ed672c6e765034@mail.gmail.com>
Message-ID: <f6951adf1003260808nea531c9lbceb996d1c478ae3@mail.gmail.com>

Well, if you're using a non-standard python (The default for suse 10.3 is
2.5, I believe), none of the binaries in the repositories are going to work,
as they're built with python 2.5.

If you want to be up to date, you really need to upgrade to a reasonably
up-to-date distribution (Try 11.2 in a virtual machine and see if it works
for you... If you're worried about compatibility with python code, and
you're already using 2.6, there really shouldn't be any problems.)

If you really don't want to upgrade, you can also try installing binaries
thorough easy_install.  It should work.

If that doesn't work, you'll have to build from source.

Hope that helps,
-Joe
P.S. I'll be out of touch with the world for the next three weeks (field
work in New Zealand). Good luck!


On Fri, Mar 26, 2010 at 7:38 AM, Lam Waiman <gunbuster363 at gmail.com> wrote:

> See the conflict list:
> -------------------------------------------------
> #### YaST2 conflicts list - generated 2010-03-26 20:47:14 ####
>
> libblas3 cannot be installed due to missing dependencies
>    There are no installable providers of libgfortran.so.1 for
> libblas3-3.2.1-10.1.i586[rere]
>    === libblas3-3.2.1-10.1.i586[rere] ===
>        libblas3-3.2.1-10.1.i586[rere] will be installed by the user.
>        libgcc42-4.2.1_20070724-17.i586 is needed by
> libblas3-3.2.1-10.1.i586[rere] (libgcc_s.so.1)
>        libblas3-3.2.1-10.1.i586[rere] is needed by
> liblapack3-3.2.1-10.1.i586[rere] (libblas.so.3)
>    (null)
>    Conflict Resolution:
>        ( ) do not install libblas3
>        ( ) Ignore this requirement just here
> liblapack3 cannot be installed due to missing dependencies
>    There are no installable providers of libgfortran.so.1 for
> liblapack3-3.2.1-10.1.i586[rere]
>    === liblapack3-3.2.1-10.1.i586[rere] ===
>        liblapack3-3.2.1-10.1.i586[rere] will be installed by the user.
>        libblas3-3.2.1-10.1.i586[rere] is needed by
> liblapack3-3.2.1-10.1.i586[rere] (libblas.so.3)
>        liblapack3-3.2.1-10.1.i586[rere] is needed by
> python-scipy-0.6.0-6.4.i586[rere] (liblapack.so.3)
>    (null)
>    Conflict Resolution:
>        ( ) do not install liblapack3
>        ( ) Ignore this requirement just here
> python-scipy cannot be installed due to missing dependencies
>    There are no installable providers of libgfortran.so.1 for
> python-scipy-0.6.0-6.4.i586[rere]
>    === python-scipy-0.6.0-6.4.i586[rere] ===
>        python-scipy-0.6.0-6.4.i586[rere] will be installed by the user.
>        libstdc++42-4.2.1_20070724-17.i586 is needed by
> python-scipy-0.6.0-6.4.i586[rere] (libstdc++.so.6)
>        fftw3-3.1.2-62.i586 is needed by
> python-scipy-0.6.0-6.4.i586[rere] (libfftw3.so.3)
>        liblapack3-3.2.1-10.1.i586[rere] is needed by
> python-scipy-0.6.0-6.4.i586[rere] (liblapack.so.3)
>        python-2.5.1-39.8.i586 is needed by
> python-scipy-0.6.0-6.4.i586[rere] (python == 2.5.1-39.8)
>    (null)
>    Conflict Resolution:
>        ( ) do not install python-scipy
>        ( ) Ignore this requirement just here
>
> #### YaST2 conflicts list END ###
>
> 2010/3/26 Lam Waiman <gunbuster363 at gmail.com>:
> > I added the above linkas a repository.
> > I tried to installed scipy, then hell lots of dependencies problems
> > come out........
> > one of them is scipy need python 2.5........non-sense, I am using
> > python 2.6 already.......
> >
> > what is your opinion?
> >
> > 2010/3/26 Joe Kington <jkington at wisc.edu>:
> >> Gah, meant to link to the base repo, but I managed to have the text show
> the
> >> base repo, and the link have the 64-bit repo...
> >> Here you go:
> >>
> http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/
> >>
> >> On Thu, Mar 25, 2010 at 2:00 PM, Joe Kington <jkington at wisc.edu> wrote:
> >>>
> >>> It's rather dated (scipy 0.6) , but I think this repo definitely has
> scipy
> >>> and should be compatible with 10.3.
> >>>
> http://download.opensuse.org/repositories/science:/ScientificLinux/SLE_10/
> >>> Hope that helps,
> >>> -Joe
> >>>
> >>> On Thu, Mar 25, 2010 at 10:29 AM, Lam Waiman <gunbuster363 at gmail.com>
> >>> wrote:
> >>>>
> >>>> But I cannot see scipy in that folder........
> >>>> And I don't want to upgrade to 11.2 because I am not sure if my
> package
> >>>> in 10.3 still works.....I don't want to risk now....
> >>>>
> >>>> 2010/3/25 Joe Kington <jkington at wisc.edu>
> >>>>>
> >>>>> I'm pretty certain scipy and numpy are in the repos for OpenSuse
> 10.3.
> >>>>> (or at least they were a few years ago... Is there some reason you
> can't
> >>>>> upgrade to 11.2? They're definitely in the repos for 11.0, 11.1, and
> 11.2)
> >>>>>
> >>>>> Do you have the science repos enabled?  They may have dropped them
> for
> >>>>> 10.x, but you can always add the SEL_10 versions.  They should be
> >>>>> compatible.
> http://download.opensuse.org/repositories/science/SLE_10/
> >>>>>
> >>>>> Hope that helps,
> >>>>> -Joe
> >>>>>
> >>>>> On Thu, Mar 25, 2010 at 4:31 AM, Lam Waiman <gunbuster363 at gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Dear all,
> >>>>>>               I want to install scipy for python usage. I see the
> >>>>>> installation isreally complex, can someone give on some advice on
> it?
> >>>>>> My system, opensuse 10.3 (i586)
> >>>>>> I am using python 2.6.4
> >>>>>> when I see the doc, it said it require LAPACK to be installed,
> >>>>>> then I download lapack and see inside, it also require lots of
> things
> >>>>>> including fortran compiler
> >>>>>> and I don't know whether the gcc42-fortran compiler in my comp is
> ok.
> >>>>>> I try to dl the g77 compiler, which someone said I will need it for
> >>>>>> LAPACK
> >>>>>> then it g77 only works with a gcc compiler of version 2.8, but I am
> >>>>>> using a gcc compiler of version 4.2.
> >>>>>> SO COMPLEX!
> >>>>>> Looking forwards to your reply, thanks.
> >>>>>> Raymond
> >>>>>> _______________________________________________
> >>>>>> SciPy-User mailing list
> >>>>>> SciPy-User at scipy.org
> >>>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> SciPy-User mailing list
> >>>>> SciPy-User at scipy.org
> >>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> SciPy-User mailing list
> >>>> SciPy-User at scipy.org
> >>>> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> SciPy-User mailing list
> >>> SciPy-User at scipy.org
> >>> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>>
> >>
> >>
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >>
> >>
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100326/5cc3d43c/attachment.html>

From aisaac at american.edu  Fri Mar 26 11:25:57 2010
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 26 Mar 2010 11:25:57 -0400
Subject: [SciPy-User] MATLAB accumarray equivalent ?
In-Reply-To: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>
References: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>
Message-ID: <4BACD205.607@american.edu>

On 3/26/2010 10:39 AM, Mohammad Abdollahi wrote:
> Does anyone know what is the equivalent function of MATLAB's
> "accumarray()" in scipy/numpy,  with the exact same functionality ?

That certainly gets the basic accumarray functionality:

         >>> vals = 101+np.arange(5)
         >>> subs = np.array([1,2,4,2,4])-1
         >>> np.bincount(subs,vals)
         array([ 101.,  206.,    0.,  208.])

Here's another way that maybe easier
to adapt to whatever the OP needs:

         >>> from collections import defaultdict
         >>> d = defaultdict(float)
         >>> for i, cat in enumerate(subs):
         ...   d[cat] += vals[i]
         ...
         >>> d
         defaultdict(<type 'float'>, {0: 101.0, 1: 206.0, 3: 208.0})

That should actually be pretty fast,
though not as fast as bincount of course.

What's the use case?

hth,
Alan Isaac



From spons at utm.csic.es  Fri Mar 26 11:38:25 2010
From: spons at utm.csic.es (Sergi Pons Freixes)
Date: Fri, 26 Mar 2010 16:38:25 +0100
Subject: [SciPy-User] Scikits TimeSeries modifies names of features
	(dtypes
In-Reply-To: <5517C8CE-6024-41AC-929D-391D5BF48A48@gmail.com>
References: <2d88eb101003240743y12893e24iff3ad29ad52f5866@mail.gmail.com>
	<5517C8CE-6024-41AC-929D-391D5BF48A48@gmail.com>
Message-ID: <2d88eb101003260838i4e00d540l5daecd9a62f9833d@mail.gmail.com>

On Wed, Mar 24, 2010 at 5:27 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> Actually, this is not a tsfromtxt issue, but a np.genfromtxt one (the main function tsfromtxt is derived from). By giving names in the input, you indicate that you want a structured array (with named fields). Structured arrays can also be viewed as recarrays, where named fields can be accessed as attributes. Obviously, having a space, a comma, a period or any other non-alphanumeric character in the field names will prevent the access by attribute to work. For that reason, np.genfromtxt checks whether the names of the fields are valid, and modify them if they're not (by replacing any non-alphanumeric character by an underscore). Think of it as a feature, not a bug.

Ok. At first, I also thought about the possibility of "they're
removeng strange characters for the sake of simplicity", but I was
shoked when creating the array directly worked:

In [9]: x = np.array([(1,2),(3,4)], dtype=[('Label.1', '<i4'), ('Label
(2)', '<i4')])

In [10]: x
Out[10]:
array([(1, 2), (3, 4)],
      dtype=[('Label.1', '<i4'), ('Label (2)', '<i4')])

In [11]: x['Label (2)']
Out[11]: array([2, 4])

So, no cleaning in this case... :S. It's funny that it has been
implemented on np.genfromtxt.


> Now, I can agree that force-feeding this attempt of foolproof behavior to the user is a bit controlling. After all, if you decided to use a name like "Agr.fil. G. corsicum 200x" as field name, you must have a very good reason, and won't mind not being able to use recarrays.

I agree that the field names are not "the best choice", but they are
automatically created from a txt file, and are automatically used in
the code (i.e.,  I never need to type "Agr.fil. G. corsicum 200x"), so
I didn't care much about it. Maybe I implementd a temporal workaround
to clean the names before passing it as labels.

> So, what I can do is leave the automatic validation of names as the default, but introduce an option to bypass it. That'd be OK for everybody ?

It's ok for me.


From pgmdevlist at gmail.com  Fri Mar 26 13:12:24 2010
From: pgmdevlist at gmail.com (Pierre GM)
Date: Fri, 26 Mar 2010 13:12:24 -0400
Subject: [SciPy-User] Scikits TimeSeries modifies names of features
	(dtypes
In-Reply-To: <2d88eb101003260838i4e00d540l5daecd9a62f9833d@mail.gmail.com>
References: <2d88eb101003240743y12893e24iff3ad29ad52f5866@mail.gmail.com>
	<5517C8CE-6024-41AC-929D-391D5BF48A48@gmail.com>
	<2d88eb101003260838i4e00d540l5daecd9a62f9833d@mail.gmail.com>
Message-ID: <9199551B-6C8B-4BDE-B695-B92B327920F2@gmail.com>

On Mar 26, 2010, at 11:38 AM, Sergi Pons Freixes wrote:
> On Wed, Mar 24, 2010 at 5:27 PM, Pierre GM <pgmdevlist at gmail.com> wrote:
>> 
>> Actually, this is not a tsfromtxt issue, but a np.genfromtxt one (the main function tsfromtxt is derived from). By giving names in the input, you indicate that you want a structured array (with named fields). Structured arrays can also be viewed as recarrays, where named fields can be accessed as attributes. Obviously, having a space, a comma, a period or any other non-alphanumeric character in the field names will prevent the access by attribute to work. For that reason, np.genfromtxt checks whether the names of the fields are valid, and modify them if they're not (by replacing any non-alphanumeric character by an underscore). Think of it as a feature, not a bug.
> 
> Ok. At first, I also thought about the possibility of "they're
> removeng strange characters for the sake of simplicity", but I was
> shoked when creating the array directly worked:
> 
> In [9]: x = np.array([(1,2),(3,4)], dtype=[('Label.1', '<i4'), ('Label
> (2)', '<i4')])
> 
> In [10]: x
> Out[10]:
> array([(1, 2), (3, 4)],
>      dtype=[('Label.1', '<i4'), ('Label (2)', '<i4')])
> 
> In [11]: x['Label (2)']
> Out[11]: array([2, 4])
> 
> So, no cleaning in this case... :S. It's funny that it has been
> implemented on np.genfromtxt.

Yes, you can define a strucutred array with non-alphanumeric characters in the field names. However, the names will be corrected if you attempt to create a recarray. np.genfromtxt follows the latter convention.


>> Now, I can agree that force-feeding this attempt of foolproof behavior to the user is a bit controlling. After all, if you decided to use a name like "Agr.fil. G. corsicum 200x" as field name, you must have a very good reason, and won't mind not being able to use recarrays.
> 
> I agree that the field names are not "the best choice", but they are
> automatically created from a txt file, and are automatically used in
> the code (i.e.,  I never need to type "Agr.fil. G. corsicum 200x"),

Which is exactly the case where an automatic validation of the fields names comes handy, if you decide to make a recarray out of your structured array...


> 
>> So, what I can do is leave the automatic validation of names as the default, but introduce an option to bypass it. That'd be OK for everybody ?
> 
> It's ok for me.

'K then, I'll work on that this afternoon.



From PHobson at Geosyntec.com  Fri Mar 26 13:33:36 2010
From: PHobson at Geosyntec.com (PHobson at Geosyntec.com)
Date: Fri, 26 Mar 2010 13:33:36 -0400
Subject: [SciPy-User] How to estimate error in polynomial
	coefficients	from scipy.polyfit?
In-Reply-To: <45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
Message-ID: <C97487CD346EEA4E80423F51F6D1EB2FA98322E7DE@EXCHANGE-00.GeoSyntec.net>

I'm no expert, but the power required to overcome aerodynamic drag varies with the cube of speed -- though the physics behind that is pretty well understood.

I guess if you were doing a bulk estimate of all of the other factors (fluid density & viscosity, drag coeff, etc) this would be an applicable use case.

Like I said, I'm hardly an expert...
-paul h.

# -------------------------
From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of David Goldsmith
Sent: Thursday, March 25, 2010 8:41 PM
To: SciPy Users List
Subject: Re: [SciPy-User] How to estimate error in polynomial coefficients from scipy.polyfit?

[snip]

On this note, perhaps some of our experts might care to comment: what "physics" (in a generalized sense) gives rise to a polynomial dependency of degree higher than two?? The only generic thing I can think of is something where third or higher order derivatives proportional to the independent variable are important, and those are pretty uncommon.



From ellisonbg.net at gmail.com  Sat Mar 27 00:25:22 2010
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 26 Mar 2010 21:25:22 -0700
Subject: [SciPy-User] SciPy 2010 Tutorials: brainstorming and call for
	proposals
Message-ID: <6ce0ac131003262125n38a74697i25dab45c1bf69b57@mail.gmail.com>

Greetings everyone,

This year, there will be two days of tutorials (June 28th and 29th) before the
main SciPy 2010 conference. Each of the two tutorial tracks (intro, advanced)
will have a 3-4 hour morning and afternoon session both days, for a total of 4
intro sessions and 4 advanced sessions.

The main tutorial web page for SciPy 2010 is here:

http://conference.scipy.org/scipy2010/tutorials.html

We are currently in the process of planning the tutorial sessions. You
can help us in two ways:

Brainstorm/vote on potential tutorial topics
============================================

To help us plan the tutorials, we have setup a web site that allow everyone in
the community to brainstorm and vote on tutorial ideas/topics.

The website for brainstorming/voting is here:

http://conference.scipy.org/scipy2010/tutorialsUV.html

The tutorial committee will use this information to help select the tutorials.
Please jump in and let us know what tutorial topics you would like to see.

Tutorial proposal submissions
=============================

We are now accepting tutorial proposals from individuals or teams that would
like to present a tutorial. Tutorials should be focused on covering a well
defined topic in a hands on manner. We want to see tutorial attendees coding!

We are pleased to offer tutorial presenters stipends this year for the first
time:

    * 1 Session: $1,000 (half day)
    * 2 Sessions: $1,500 (full day)

Optionally, part of this stipend can be applied to the presenter's
registration costs.

To submit a tutorial proposal please submit the following materials
to 2010tutorials at scipy.org by April 15:

* A short bio of the presenter or team members.
* Which track the tutorial would be in (intro or advanced).
* A short description and/or outline of the tutorial content.
* A list of Python packages that attendees will need to have installed to
  follow along.


From timmichelsen at gmx-topmail.de  Sat Mar 27 08:27:17 2010
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Sat, 27 Mar 2010 13:27:17 +0100
Subject: [SciPy-User] experiences with Numpy+Scipy+IronPython?
Message-ID: <hoktj6$bbd$1@dough.gmane.org>

Dear all,
are there any users which have projects that use the Numpy/Scipy
libraries on IronPython?

Ironclad (http://code.google.com/p/ironclad/)claims to make bothe
accessible under IronPython

There exist then a possibility to plug this into Excel...

Can I install IronPython alongside normal Python without conflicts?

Best regards,
Tim



From dagss at student.matnat.uio.no  Sat Mar 27 12:17:13 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 27 Mar 2010 17:17:13 +0100
Subject: [SciPy-User] experiences with Numpy+Scipy+IronPython?
In-Reply-To: <hoktj6$bbd$1@dough.gmane.org>
References: <hoktj6$bbd$1@dough.gmane.org>
Message-ID: <4BAE2F89.10901@student.matnat.uio.no>

Tim Michelsen wrote:
> Dear all,
> are there any users which have projects that use the Numpy/Scipy
> libraries on IronPython?
> 
> Ironclad (http://code.google.com/p/ironclad/)claims to make bothe
> accessible under IronPython
> 
> There exist then a possibility to plug this into Excel...

I can't answer, but just to make conversation: Interestingly, Ironclad 
development appears to be supported by a "Python-in-spreadsheet" program 
which appears very interesting:

http://www.resolversystems.com/products/resolver-one/

It says that it can directly import NumPy.

Dag Sverre


From jeremy at jeremysanders.net  Sun Mar 28 12:37:52 2010
From: jeremy at jeremysanders.net (Jeremy Sanders)
Date: Sun, 28 Mar 2010 17:37:52 +0100
Subject: [SciPy-User] ANN: Veusz 1.7
Message-ID: <hoo0l0$nja$1@dough.gmane.org>

Veusz 1.7
---------
Velvet Ember Under Sky Zenith
-----------------------------
http://home.gna.org/veusz/

Veusz is Copyright (C) 2003-2010 Jeremy Sanders <jeremy at jeremysanders.net>
Licenced under the GPL (version 2 or greater).

Veusz is a Qt4 based scientific plotting package. It is written in
Python, using PyQt4 for display and user-interfaces, and numpy for
handling the numeric data. Veusz is designed to produce
publication-ready Postscript/PDF/SVG output. The user interface aims
to be simple, consistent and powerful.

Veusz provides a GUI, command line, embedding and scripting interface
(based on Python) to its plotting facilities. It also allows for
manipulation and editing of datasets. Data can be captured from
external sources such as internet sockets or other programs.

Changes in 1.7:
 * Widgets can be moved by dragged and dropped in the widget tree, or copied
   by holding down ctrl at the same time
 * Tick labels are centred if possible at the start and ends of axes
 * When putting graphs in grid, axis labels and tick labels are placed
   in much better positions
 * Embedding module is shipped in binary versions
 * Grid lines can be drawn on axis minor ticks
 * Contour widget can draw minor (dotted) contours between main contours
 * Logarithmic contours have proper logarithmic spacing
 * Fixes for widget names and dataset names with Unicode characters, 
including
   copy and paste
 * Optional smoothing in the image widget

Minor changes:
 * Errors in evaluating expressions are logged in the console window, and
   do not show exceptions
 * Fix problems when importing multiple symbols from python modules in the
   custom import dialog
 * Use minus sign for negative numbers in tick labels, rather than hyphens
 * Contour widget lines can have transparency
 * Datasets are sorted by name when writing to saved document
 * Use correct status for paste button when starting application
 * Add option for extra space between axes and tick labels, and axis labels
 * Preference added for background color of exported bitmaps
 * Add IsClosed() and WaitForClose() embedding functions to check whether 
plot
   window is closed, or to wait for closing of plot window.

Features of package:
 * X-Y plots (with errorbars)
 * Line and function plots
 * Contour plots
 * Images (with colour mappings and colorbars)
 * Stepped plots (for histograms)
 * Bar graphs
 * Plotting dates
 * Fitting functions to data
 * Stacked plots and arrays of plots
 * Plot keys
 * Plot labels
 * Shapes and arrows on plots
 * LaTeX-like formatting for text
 * EPS/PDF/PNG/SVG/EMF export
 * Scripting interface
 * Dataset creation/manipulation
 * Embed Veusz within other programs
 * Text, CSV and FITS importing
 * Data can be captured from external sources
 * User defined functions, constants and can import external Python 
functions

Requirements for source install:
 Python (2.4 or greater required)
   http://www.python.org/
 Qt >= 4.3 (free edition)
   http://www.trolltech.com/products/qt/  
 PyQt >= 4.3 (SIP is required to be installed first)
   http://www.riverbankcomputing.co.uk/pyqt/
   http://www.riverbankcomputing.co.uk/sip/
 numpy >= 1.0
   http://numpy.scipy.org/

Optional:
 Microsoft Core Fonts (recommended for nice output)
   http://corefonts.sourceforge.net/
 PyFITS >= 1.1 (optional for FITS import)
   http://www.stsci.edu/resources/software_hardware/pyfits
 pyemf >= 2.0.0 (optional for EMF export)
   http://pyemf.sourceforge.net/
 For EMF and better SVG export, PyQt >= 4.6 or better is
   required, to fix a bug in the C++ wrapping

For documentation on using Veusz, see the "Documents" directory. The
manual is in PDF, HTML and text format (generated from docbook). The
examples are also useful documentation.

Issues with the current version:

 * Due to Qt, hatched regions sometimes look rather poor when exported
   to PostScript, PDF or SVG.

 * Due to a bug in Qt, some long lines, or using log scales, can lead
   to very slow plot times under X11. It is fixed by upgrading to
   Qt-4.5.1 (or using a binary). Switching off antialiasing in the options
   may help.

If you enjoy using Veusz, I would love to hear from you. Please join
the mailing lists at

https://gna.org/mail/?group=veusz

to discuss new features or if you'd like to contribute code. The
latest code can always be found in the SVN repository.

Jeremy Sanders




From alan at ajackson.org  Sun Mar 28 14:36:53 2010
From: alan at ajackson.org (alan at ajackson.org)
Date: Sun, 28 Mar 2010 13:36:53 -0500
Subject: [SciPy-User] How to estimate error in polynomial coefficients
 from scipy.polyfit?
In-Reply-To: <C97487CD346EEA4E80423F51F6D1EB2FA98322E7DE@EXCHANGE-00.GeoSyntec.net>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
	<C97487CD346EEA4E80423F51F6D1EB2FA98322E7DE@EXCHANGE-00.GeoSyntec.net>
Message-ID: <20100328133653.35b8cbfd@ajackson.org>

Crack permeability goes like the third power of the opening (that is,
fluid flow through cracks - think gas or oil in a fractured rock). 

And for many problems, the Taylor expansion might appropriately be taken out to
third order. Sometimes the even orders cancel out so to get the first
non-linear effects you have to go to third order. Can't think of a specific
offhand, but I've seen it quite a few times.

>I'm no expert, but the power required to overcome aerodynamic drag varies with the cube of speed -- though the physics behind that is pretty well understood.
>
>I guess if you were doing a bulk estimate of all of the other factors (fluid density & viscosity, drag coeff, etc) this would be an applicable use case.
>
>Like I said, I'm hardly an expert...
>-paul h.
>
># -------------------------
>From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of David Goldsmith
>Sent: Thursday, March 25, 2010 8:41 PM
>To: SciPy Users List
>Subject: Re: [SciPy-User] How to estimate error in polynomial coefficients from scipy.polyfit?
>
>[snip]
>
>On this note, perhaps some of our experts might care to comment: what "physics" (in a generalized sense) gives rise to a polynomial dependency of degree higher than two?? The only generic thing I can think of is something where third or higher order derivatives proportional to the independent variable are important, and those are pretty uncommon.
>
>_______________________________________________
>SciPy-User mailing list
>SciPy-User at scipy.org
>http://mail.scipy.org/mailman/listinfo/scipy-user


-- 
-----------------------------------------------------------------------
| Alan K. Jackson            | To see a World in a Grain of Sand      |
| alan at ajackson.org          | And a Heaven in a Wild Flower,         |
| www.ajackson.org           | Hold Infinity in the palm of your hand |
| Houston, Texas             | And Eternity in an hour. - Blake       |
-----------------------------------------------------------------------


From peridot.faceted at gmail.com  Sun Mar 28 19:46:29 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Sun, 28 Mar 2010 19:46:29 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com> 
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com> 
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com> 
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com> 
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com> 
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
Message-ID: <ce557a361003281646v194f753cvef264cb3b01a6dff@mail.gmail.com>

On 25 March 2010 23:40, David Goldsmith <d.l.goldsmith at gmail.com> wrote:
> On Thu, Mar 25, 2010 at 3:40 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>
>> Yikes! ?This sounds like it may be more trouble than it's worth. ?I
>> have a few sets of statistical data that each need to have curves fit
>> to them.
>
> That's an awfully generic need - it may be obvious from examination of the
> data that a line is inappropriate, but besides polynomials there are many
> other non-linear models (which can be linearly fit to data by means of data
> transformation) which possess fewer parameters (and thus are simpler from a
> parameter analysis perspective).? So, the question is: why are you fitting
> to polynomials?? If it's just to get a good fit to the data, you might be
> getting "more fit" than your data warrants (and even if that isn't a
> problem, you probably want to use a polynomial form different from "standard
> form," e.g., Lagrange interpolators).? Are you sure something like an
> exponential growth/decay or power law model (both of which are "more
> natural," linearizable, two-parameter models) wouldn't be more appropriate -
> it would almost certainly be simpler to analyze (and perhaps easier to
> justify to a referee).
>
> On this note, perhaps some of our experts might care to comment: what
> "physics" (in a generalized sense) gives rise to a polynomial dependency of
> degree higher than two?? The only generic thing I can think of is something
> where third or higher order derivatives proportional to the independent
> variable are important, and those are pretty uncommon.

In pulsar timing, what you measure is phase (actually pulse arrival
time); pulse period/frequency is its derivative, and spin-down is the
derivative of period and measures the power available for producing
radiation. These two parameters are also used in standard formulas to
infer magnetic field and age. The second derivative of period (third
derivative of the measured quantity) is generally too small to be
measured, but when it can be it tells you something about the
mechanism by which the spin-down is happening.

Beyond the plain physics, there is typically some low-frequency noise
(nobody's sure where it comes from, possibly the turbulence in the
superfluid layers of the neutron star) which is typically managed by
subtracting a best-fit polynomial (polynomial subtraction has the
property of killing off noise spectra that diverge at zero frequency,
which this noise appears to).

I suppose power laws, though ubiquitous, do not exactly count as
polynomials; generally their exponents are not integers.

Anne

> DG
>
>>
>> I can see what the parameters are and they are similar
>> between each set. ?I want to know if the coefficients are within a few
>> standard deviations of each other. ?That's what I was hoping to get
>> at, but it seems to be more trouble than I'm willing to accept.
>> Unless anyone has a simpler solution.
>>
>> Thanks,
>> Jeremy
>> >
>> > Anne
>> >
>> >> residuals, rank, singular_values, rcond : present only if `full` = True
>> >> ? ? ? ?Residuals of the least-squares fit, the effective rank of the
>> >> scaled
>> >> ? ? ? ?Vandermonde coefficient matrix, its singular values, and the
>> >> specified
>> >> ? ? ? ?value of `rcond`. For more details, see `linalg.lstsq`.
>> >>
>> >> I don't think any of these things are "design matrix" as you have
>> >> indicated I need. ?The documentation for linalg.lstsq does not say
>> >> what rcond is unfortunately. ? Any ideas?
>> >>
>> >> Jeremy
>> >> _______________________________________________
>> >> SciPy-User mailing list
>> >> SciPy-User at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/scipy-user
>> >>
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From d.l.goldsmith at gmail.com  Sun Mar 28 23:26:17 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Sun, 28 Mar 2010 20:26:17 -0700
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <20100328133653.35b8cbfd@ajackson.org>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
	<C97487CD346EEA4E80423F51F6D1EB2FA98322E7DE@EXCHANGE-00.GeoSyntec.net>
	<20100328133653.35b8cbfd@ajackson.org>
Message-ID: <45d1ab481003282026y3824c114r7cea2948c90e5c4@mail.gmail.com>

On Sun, Mar 28, 2010 at 11:36 AM, <alan at ajackson.org> wrote:

> Crack permeability goes like the third power of the opening (that is,
> fluid flow through cracks - think gas or oil in a fractured rock).
>

Power law or polynomial: from a regression stand point, there's quite a big
difference.

And for many problems, the Taylor expansion might appropriately be taken out
> to
> third order. Sometimes the even orders cancel out so to get the first
> non-linear effects you have to go to third order. Can't think of a specific
> offhand, but I've seen it quite a few times.
>

A slightly less-idealized pendulum.


> >I'm no expert, but the power required to overcome aerodynamic drag varies
> with the cube of speed


Same comment as above.

In my experience, compared to multi-term polynomial models, power law models
are comparatively common (though I acknowledge that, as the power is one of
the fit parameters, there's no a priori guarantee that it'll be an integer;
indeed, since the integers have measure zero rel. the reals, the odds of
getting an integer are zero, *unless* there is some "physical" reason why it
should be an integer).

DG


> -- though the physics behind that is pretty well understood.
> >
> >I guess if you were doing a bulk estimate of all of the other factors
> (fluid density & viscosity, drag coeff, etc) this would be an applicable use
> case.
> >
> >Like I said, I'm hardly an expert...
> >-paul h.
> >
> ># -------------------------
> >From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org]
> On Behalf Of David Goldsmith
> >Sent: Thursday, March 25, 2010 8:41 PM
> >To: SciPy Users List
> >Subject: Re: [SciPy-User] How to estimate error in polynomial coefficients
> from scipy.polyfit?
> >
> >[snip]
> >
> >On this note, perhaps some of our experts might care to comment: what
> "physics" (in a generalized sense) gives rise to a polynomial dependency of
> degree higher than two?  The only generic thing I can think of is something
> where third or higher order derivatives proportional to the independent
> variable are important, and those are pretty uncommon.
> >
> >_______________________________________________
> >SciPy-User mailing list
> >SciPy-User at scipy.org
> >http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> --
> -----------------------------------------------------------------------
> | Alan K. Jackson            | To see a World in a Grain of Sand      |
> | alan at ajackson.org          | And a Heaven in a Wild Flower,         |
> | www.ajackson.org           | Hold Infinity in the palm of your hand |
> | Houston, Texas             | And Eternity in an hour. - Blake       |
> -----------------------------------------------------------------------
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100328/c1b4948f/attachment.html>

From lanceboyle at qwest.net  Mon Mar 29 01:02:44 2010
From: lanceboyle at qwest.net (Jerry)
Date: Sun, 28 Mar 2010 22:02:44 -0700
Subject: [SciPy-User] [OT] Re:  ANN: Veusz 1.7
In-Reply-To: <hoo0l0$nja$1@dough.gmane.org>
References: <hoo0l0$nja$1@dough.gmane.org>
Message-ID: <5D3B55B0-D5D7-40CB-86D1-676FBC6F317F@qwest.net>

Does anyone know why QT apps on OS X refuse to understand Command-W?  
This has been default behavior for closing a window since around, oh,  
1984.

Jerry

On Mar 28, 2010, at 9:37 AM, Jeremy Sanders wrote:

> Veusz 1.7
> ---------
> Velvet Ember Under Sky Zenith
> -----------------------------
> http://home.gna.org/veusz/
>
> Veusz is Copyright (C) 2003-2010 Jeremy Sanders <jeremy at jeremysanders.net 
> >
> Licenced under the GPL (version 2 or greater).
>
> Veusz is a Qt4 based scientific plotting package. It is written in
> Python, using PyQt4 for display and user-interfaces, and numpy for
> handling the numeric data. Veusz is designed to produce
> publication-ready Postscript/PDF/SVG output. The user interface aims
> to be simple, consistent and powerful.
>
> Veusz provides a GUI, command line, embedding and scripting interface
> (based on Python) to its plotting facilities. It also allows for
> manipulation and editing of datasets. Data can be captured from
> external sources such as internet sockets or other programs.
>
> Changes in 1.7:
> * Widgets can be moved by dragged and dropped in the widget tree, or  
> copied
>   by holding down ctrl at the same time
> * Tick labels are centred if possible at the start and ends of axes
> * When putting graphs in grid, axis labels and tick labels are placed
>   in much better positions
> * Embedding module is shipped in binary versions
> * Grid lines can be drawn on axis minor ticks
> * Contour widget can draw minor (dotted) contours between main  
> contours
> * Logarithmic contours have proper logarithmic spacing
> * Fixes for widget names and dataset names with Unicode characters,
> including
>   copy and paste
> * Optional smoothing in the image widget
>
> Minor changes:
> * Errors in evaluating expressions are logged in the console window,  
> and
>   do not show exceptions
> * Fix problems when importing multiple symbols from python modules  
> in the
>   custom import dialog
> * Use minus sign for negative numbers in tick labels, rather than  
> hyphens
> * Contour widget lines can have transparency
> * Datasets are sorted by name when writing to saved document
> * Use correct status for paste button when starting application
> * Add option for extra space between axes and tick labels, and axis  
> labels
> * Preference added for background color of exported bitmaps
> * Add IsClosed() and WaitForClose() embedding functions to check  
> whether
> plot
>   window is closed, or to wait for closing of plot window.
>
> Features of package:
> * X-Y plots (with errorbars)
> * Line and function plots
> * Contour plots
> * Images (with colour mappings and colorbars)
> * Stepped plots (for histograms)
> * Bar graphs
> * Plotting dates
> * Fitting functions to data
> * Stacked plots and arrays of plots
> * Plot keys
> * Plot labels
> * Shapes and arrows on plots
> * LaTeX-like formatting for text
> * EPS/PDF/PNG/SVG/EMF export
> * Scripting interface
> * Dataset creation/manipulation
> * Embed Veusz within other programs
> * Text, CSV and FITS importing
> * Data can be captured from external sources
> * User defined functions, constants and can import external Python
> functions
>
> Requirements for source install:
> Python (2.4 or greater required)
>   http://www.python.org/
> Qt >= 4.3 (free edition)
>   http://www.trolltech.com/products/qt/
> PyQt >= 4.3 (SIP is required to be installed first)
>   http://www.riverbankcomputing.co.uk/pyqt/
>   http://www.riverbankcomputing.co.uk/sip/
> numpy >= 1.0
>   http://numpy.scipy.org/
>
> Optional:
> Microsoft Core Fonts (recommended for nice output)
>   http://corefonts.sourceforge.net/
> PyFITS >= 1.1 (optional for FITS import)
>   http://www.stsci.edu/resources/software_hardware/pyfits
> pyemf >= 2.0.0 (optional for EMF export)
>   http://pyemf.sourceforge.net/
> For EMF and better SVG export, PyQt >= 4.6 or better is
>   required, to fix a bug in the C++ wrapping
>
> For documentation on using Veusz, see the "Documents" directory. The
> manual is in PDF, HTML and text format (generated from docbook). The
> examples are also useful documentation.
>
> Issues with the current version:
>
> * Due to Qt, hatched regions sometimes look rather poor when exported
>   to PostScript, PDF or SVG.
>
> * Due to a bug in Qt, some long lines, or using log scales, can lead
>   to very slow plot times under X11. It is fixed by upgrading to
>   Qt-4.5.1 (or using a binary). Switching off antialiasing in the  
> options
>   may help.
>
> If you enjoy using Veusz, I would love to hear from you. Please join
> the mailing lists at
>
> https://gna.org/mail/?group=veusz
>
> to discuss new features or if you'd like to contribute code. The
> latest code can always be found in the SVN repository.
>
> Jeremy Sanders
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



From robert.kern at gmail.com  Mon Mar 29 10:34:15 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 29 Mar 2010 09:34:15 -0500
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <45d1ab481003282026y3824c114r7cea2948c90e5c4@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com> 
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com> 
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com> 
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com> 
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com> 
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com> 
	<C97487CD346EEA4E80423F51F6D1EB2FA98322E7DE@EXCHANGE-00.GeoSyntec.net> 
	<20100328133653.35b8cbfd@ajackson.org>
	<45d1ab481003282026y3824c114r7cea2948c90e5c4@mail.gmail.com>
Message-ID: <3d375d731003290734w6234071dke3a093e97dc0ac0a@mail.gmail.com>

On Sun, Mar 28, 2010 at 22:26, David Goldsmith <d.l.goldsmith at gmail.com> wrote:
> On Sun, Mar 28, 2010 at 11:36 AM, <alan at ajackson.org> wrote:
>>
>> Crack permeability goes like the third power of the opening (that is,
>> fluid flow through cracks - think gas or oil in a fractured rock).
>
> Power law or polynomial: from a regression stand point, there's quite a big
> difference.

"Third power" == "x**3". He's not talking about a power law.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From paul.anton.letnes at gmail.com  Mon Mar 29 10:57:42 2010
From: paul.anton.letnes at gmail.com (Paul Anton Letnes)
Date: Mon, 29 Mar 2010 07:57:42 -0700
Subject: [SciPy-User] Again with matrices multiplication
In-Reply-To: <189976BA-0A25-480D-9970-C283152D6B2F@lu.unisi.ch>
References: <189976BA-0A25-480D-9970-C283152D6B2F@lu.unisi.ch>
Message-ID: <9B6DF534-E8B4-4358-B875-BEA6195EE732@gmail.com>


On 26. mars 2010, at 04.10, Graziano Mirata wrote:

> Hi guys, 
> it's me again. Still problems with matrices multiplication. 
> 
> So i have used dot(w,R) to multiply two arrays but now i cannot access to elements. I.e 
> 
> In [224]: w.shape
> Out[224]: (3,)
> In [226]: R.shape
> Out[226]: (24000, 3)
> 
It is easier to work with matrices. To convert arrays to matrices, just type:
>>> R = numpy.mat(R)
>>> w = numpy.mat(w)
>>> r = R * w
Notice that arrays can have _any_ number of dimensions, while matrices _always_ are 2-dimensional (2 indices). Use the matrix class, and you'll always be sure that matrix multiplication is performed as desired for any shape of matrices, and also, you'll always end up with a 2-dimensional array-like object, which is what you want. You can always convert back to arrays later, using
>>> myarray = numpy.array(mymatrix)

Cheers,
Paul

> Thanx
> 
> G
> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user




From graziano.mirata at lu.unisi.ch  Mon Mar 29 11:01:25 2010
From: graziano.mirata at lu.unisi.ch (Graziano Mirata)
Date: Mon, 29 Mar 2010 17:01:25 +0200
Subject: [SciPy-User] Again with matrices multiplication
In-Reply-To: <9B6DF534-E8B4-4358-B875-BEA6195EE732@gmail.com>
References: <189976BA-0A25-480D-9970-C283152D6B2F@lu.unisi.ch>
	<9B6DF534-E8B4-4358-B875-BEA6195EE732@gmail.com>
Message-ID: <D47787B3-973D-4B73-892D-434093A10670@lu.unisi.ch>

Thanx to everybody! 

On Mar 29, 2010, at 4:57 PM, Paul Anton Letnes wrote:

> 
> On 26. mars 2010, at 04.10, Graziano Mirata wrote:
> 
>> Hi guys, 
>> it's me again. Still problems with matrices multiplication. 
>> 
>> So i have used dot(w,R) to multiply two arrays but now i cannot access to elements. I.e 
>> 
>> In [224]: w.shape
>> Out[224]: (3,)
>> In [226]: R.shape
>> Out[226]: (24000, 3)
>> 
> It is easier to work with matrices. To convert arrays to matrices, just type:
>>>> R = numpy.mat(R)
>>>> w = numpy.mat(w)
>>>> r = R * w
> Notice that arrays can have _any_ number of dimensions, while matrices _always_ are 2-dimensional (2 indices). Use the matrix class, and you'll always be sure that matrix multiplication is performed as desired for any shape of matrices, and also, you'll always end up with a 2-dimensional array-like object, which is what you want. You can always convert back to arrays later, using
>>>> myarray = numpy.array(mymatrix)
> 
> Cheers,
> Paul
> 
>> Thanx
>> 
>> G
>> 
>> 
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From jlconlin at gmail.com  Mon Mar 29 11:08:48 2010
From: jlconlin at gmail.com (Jeremy Conlin)
Date: Mon, 29 Mar 2010 09:08:48 -0600
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
Message-ID: <2588da421003290808t4f2e185cy5b07a35002f85e87@mail.gmail.com>

On Thu, Mar 25, 2010 at 9:40 PM, David Goldsmith
<d.l.goldsmith at gmail.com> wrote:
> On Thu, Mar 25, 2010 at 3:40 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>
>> Yikes! ?This sounds like it may be more trouble than it's worth. ?I
>> have a few sets of statistical data that each need to have curves fit
>> to them.
>
> That's an awfully generic need - it may be obvious from examination of the
> data that a line is inappropriate, but besides polynomials there are many
> other non-linear models (which can be linearly fit to data by means of data
> transformation) which possess fewer parameters (and thus are simpler from a
> parameter analysis perspective).? So, the question is: why are you fitting
> to polynomials?? If it's just to get a good fit to the data, you might be
> getting "more fit" than your data warrants (and even if that isn't a
> problem, you probably want to use a polynomial form different from "standard
> form," e.g., Lagrange interpolators).? Are you sure something like an
> exponential growth/decay or power law model (both of which are "more
> natural," linearizable, two-parameter models) wouldn't be more appropriate -
> it would almost certainly be simpler to analyze (and perhaps easier to
> justify to a referee).
>
> On this note, perhaps some of our experts might care to comment: what
> "physics" (in a generalized sense) gives rise to a polynomial dependency of
> degree higher than two?? The only generic thing I can think of is something
> where third or higher order derivatives proportional to the independent
> variable are important, and those are pretty uncommon.

I will only be fitting data to a first or second degree polynomial.
Eventually I would like to fit my data to an exponential or a power
law, just to see how it compares to a low-order polynomial.  Choosing
these functions was based on qualitative analysis (i.e. "it looks
quadratic").

The best case scenario would be that I take what I learn from this
"simple" example and apply it to more difficult problems as they come
along down the road.  It appears, however, that it's not so simple to
apply it to other problems.  I wish I had more time to learn about
fitting data to curves.  I'm sure there are a lot of powerful tools
that can help.

Jeremy


From josef.pktd at gmail.com  Mon Mar 29 11:21:45 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 29 Mar 2010 11:21:45 -0400
Subject: [SciPy-User] tests for equality of coefficients across regression
	or groups
Message-ID: <1cd32cbb1003290821u1fa7f609n3d973e73307faac9@mail.gmail.com>

I wrote a class that is more convenient to test whether all
coefficient are the same across different regressions or groups. This
can also be used to test for structural breaks at predefined change
points, and it is a bit similar to ANCOVA (as far as I understand)
however currently all coefficients are tested for changes across
groups. (While ANCOVA assumes that some slope coefficients are the
same. But I'm not sure about all the ANCOVA, ANOVA and MANCOVA
terminology).

If the only regressor is a constant, then the result is identical to
scipy.stats.f_oneway.

The original will be in the statsmodels sandbox, until some missing
enhancements are finished. The attached file contains a standalone
version (but requires scikits.statsmodels). Several examples are
included.

Below is par of the example output.
4 groups, groups 0 and 1 and groups 2 and 3 have identical
coefficients, but large difference between 0,1 and 2,3
sample size is 100 where the results are pretty clear cut.

I haven't thought much about the API for this kind of convenience
class. I will finish it up one of the next weekends.
Any comments are welcome, in case someone finds this useful.

Josef

Test for equality of coefficients for all exogenous variables
-------------------------------------------------------------

Table of F-tests for overall or pairwise equality of coefficients
hypothesis F-statistic         p-value  df_denom df_num  reject
('all', (212.94813033530886, 5.0118071432076266e-082, 192.0, 6)) *
((0, 1), (1.8208707506170163, 0.16466906994879624, 192.0, 2))
((0, 2), (281.4877465209409, 8.2293196773132505e-058, 192.0, 2)) *
((0, 3), (322.21218144617899, 4.4051148030359891e-062, 192.0, 2)) *
((1, 2), (313.90670953958153, 3.0218699484829141e-061, 192.0, 2)) *
((1, 3), (356.39739587117498, 2.3340444544965008e-065, 192.0, 2)) *
((2, 3), (1.4442741180990379, 0.23846859348160215, 192.0, 2))
Notes: p-values are not corrected for many tests
       (no Bonferroni correction)
       * : reject at 5% uncorrected confidence level
Null hypothesis: all or pairwise coefficient are the same
Alternative hypothesis: all coefficients are different

Comparison with stats.f_oneway
(5.4853751213497954, 0.0012219835303810941)

Likelihood Ratio Test
likelihood ratio    p-value       df
(407.06211448271051, 8.4739101836496682e-085, 6.0)
Null model: pooled all coefficients are the same across groups,
Alternative model: all coefficients are allowed to be different
not verified but looks close to f-test result

Ols parameters by group from individual, separate ols regressions
0 [ 14.89986056  10.49887409]
1 [ 14.56954028   9.8411222 ]
2 [ 16.95335652  18.83403447]
3 [ 17.03709822  19.43980093]

Check for heteroscedasticity,
variance and standard deviation for individual regressions
             group 0          group 1          group 2          group 3
variance     [ 2.615224    2.73526096  4.38683135  2.91666872]
standard dev [ 1.61716542  1.65386244  2.09447639  1.70782573]
-------------- next part --------------
# -*- coding: utf-8 -*-
"""
F test for null hypothesis that coefficients in several regressions are the same

* implemented by creating groupdummies*exog and testing appropriate contrast
  matrices
* similar to test for structural change in all variables at predefined break points
* allows only one group variable
* currently tests for change in all exog variables
* allows for heterogscedasticity, error variance varies across groups (not tried out)


TODO
----

* generalize anova structure, 
  - structural break in only some variables
  - compare structural breaks in several exog versus constant only
  - fast way to construct comparisons
* print anova style results
* add all pairwise comparison tests with and without Bonferroni correction
* add additional test, likelihood-ratio, lagrange-multiplier, wald ?
* test for heteroscedasticity, equality of variances 
  - how? 
  - like lagrange-multiplier in stattools heteroscedasticity tests

Created on Sat Mar 27 01:48:01 2010
Author: josef-pktd
"""
import numpy as np
from scipy import stats
from scikits.statsmodels.regression import OLS, WLS


class OneWayLS(object):
    '''Class to test equality of regression coefficients across groups
    
    This class performs tests whether the linear regression coefficients are
    the same across pre-specified groups. This can be used to test for 
    structural breaks at given change points, or for ANOVA style analysis of
    differences in the effect of explanatory variables across groups.
    
    Notes
    -----
    The test is implemented by regression on the original pooled exogenous
    variables and on group dummies times the exogenous regressors.
    
    y_i = X_i beta_i + u_i  for all groups i
    
    The test is for the null hypothesis: beta_i = beta for all i
    against the alternative that at least one beta_i is different.
    
    By default it is assumed that all u_i have the same variance. If the
    keyword option het is True, then it is assumed that the variance is
    group specific. This uses WLS with weights given by the standard errors
    from separate regressions for each group.
    Note: het=True is not sufficiently tested
    
    The F-test assumes that the errors are normally distributed.
    
    
    
    original question from mailing list for equality of coefficients 
    across regressions, and example in Stata FAQ
    
    *testing*:
    
    * if constant is the only regressor then the result for the F-test is
      the same as scipy.stats.f_oneway 
      (which in turn is verified against NIST for not badly scaled problems) 
    * f-test for simple structural break is the same as in original script
    * power and size of test look ok in examples
    * not checked/verified for heteroscedastic case
      - for constant only: ftest result is the same with WLS as with OLS - check?
    
    check: I might be mixing up group names (unique) 
           and group id (integers in arange(ngroups) 
           not tested for groups that are not arange(ngroups)
           make sure groupnames are always consistently sorted/ordered
    '''
    def __init__(self, y, x, groups=None, het=False, data=None, meta=None):
        if groups is None:
            raise ValueError('use OLS if there are no groups')
            #maybe replace by dispatch to OLS
        if data:
            y = data[y]
            x = [data[v] for v in x]
            try:
                groups = data[groups]
            except [KeyError, ValueError]:
                pass
        self.endog = np.asarray(y)
        self.exog = np.asarray(x)
        if self.exog.ndim == 1:
            self.exog = self.exog[:,None]
        self.groups = np.asarray(groups)
        self.het = het        
        
        self.groupsint = None
        if np.issubdtype(self.groups.dtype, int):
            self.unique = np.unique(self.groups)
            if (self.unique == np.arange(len(self.unique))).all():
                self.groupsint = self.groups
        
        if self.groupsint is None: # groups are not consecutive integers
            self.unique, self.groupsint = np.unique(self.groupsint, return_inverse=True)
    
    def fitbygroups(self):
        olsbygroup = {}
        sigmabygroup = []
        for gi, group in enumerate(self.unique):
            groupmask = self.groupsint == group
            res = OLS(self.endog[groupmask], self.exog[groupmask]).fit()
            olsbygroup[group] = res
            sigmabygroup.append(res.mse_resid)
        self.olsbygroup = olsbygroup
        self.sigmabygroup = np.array(sigmabygroup)
        self.weights = np.sqrt(self.sigmabygroup[self.groupsint]) #TODO:chk sqrt
        
    def fitjoint(self):
        if not hasattr(self, 'weights'):
            self.fitbygroups()
        groupdummy = (self.groupsint[:,None] == self.unique).astype(int)
        #order of dummy variables by grous - used
        dummyexog = self.exog[:,None,:]*groupdummy[:,1:,None]
        exog = np.c_[self.exog, dummyexog.reshape(self.exog.shape[0],-1)] #self.nobs ??
        #Notes: I changed to drop first group from dummy 
        #instead I want one full set dummies
        if self.het:
            weights = self.weights
            res = WLS(self.endog, exog, weights=weights).fit()
        else:
            res = OLS(self.endog, exog).fit()
        self.lsjoint = res
        contrasts = {}
        nvars = self.exog.shape[1]
        nparams = exog.shape[1]
        ndummies = nparams - nvars
        contrasts['all'] = np.c_[np.zeros((ndummies, nvars)), np.eye(ndummies)]
        for groupind,group in enumerate(self.unique[1:]):  #need enumerate if groups != groupsint
            groupind = groupind + 1 
            contr = np.zeros((nvars, nparams))
            contr[:,nvars*groupind:nvars*(groupind+1)] = np.eye(nvars)
            contrasts[group] = contr
            #save also for pairs, see next
            contrasts[(self.unique[0], group)] = contr
        
        #Note: I'm keeping some duplication for testing
        pairs = np.triu_indices(len(self.unique),1)
        for ind1,ind2 in zip(*pairs):  #replace with group1, group2 in sorted(keys)
            if ind1 == 0:
                continue # need comparison with benchmark/normalization group separate
            g1 = self.unique[ind1]
            g2 = self.unique[ind2]
            group = (g1, g2)
            contr = np.zeros((nvars, nparams))
            contr[:,nvars*ind1:nvars*(ind1+1)] = np.eye(nvars)
            contr[:,nvars*ind2:nvars*(ind2+1)] = -np.eye(nvars)
            contrasts[group] = contr
        self.contrasts = contrasts
    
    def fitpooled(self):
        if self.het:
            if not hasattr(self, 'weights'):
                self.fitbygroups()
            weights = self.weights
            res = WLS(self.endog, self.exog, weights=weights).fit()
        else:
            res = OLS(self.endog, self.exog).fit()
        self.lspooled = res
        
    def ftest_summary(self):
        if not hasattr(self, 'lsjoint'):
            self.fitjoint()
        txt = []
        summarytable = []
        
        txt.append('F-test for equality of coefficients across groups')
        fres = self.lsjoint.f_test(self.contrasts['all'])
        txt.append(fres.__str__())
        summarytable.append(('all',(fres.fvalue, fres.pvalue, fres.df_denom, fres.df_num)))
        
#        for group in self.unique[1:]:  #replace with group1, group2 in sorted(keys)
#            txt.append('F-test for equality of coefficients between group'
#                       ' %s and group %s' % (group, '0'))
#            fres = self.lsjoint.f_test(self.contrasts[group])
#            txt.append(fres.__str__())
#            summarytable.append((group,(fres.fvalue, fres.pvalue, fres.df_denom, fres.df_num)))
        pairs = np.triu_indices(len(self.unique),1)
        for ind1,ind2 in zip(*pairs):  #replace with group1, group2 in sorted(keys)
            g1 = self.unique[ind1]
            g2 = self.unique[ind2]
            txt.append('F-test for equality of coefficients between group'
                       ' %s and group %s' % (g1, g2))
            group = (g1, g2)
            fres = self.lsjoint.f_test(self.contrasts[group])
            txt.append(fres.__str__())
            summarytable.append((group,(fres.fvalue, fres.pvalue, fres.df_denom, fres.df_num)))
        
        return '\n'.join(txt), summarytable
                            
        
    def lr_test(self):
        '''generic likelihood ration test between nested models
        
            \begin{align} D & = -2(\ln(\text{likelihood for null model}) - \ln(\text{likelihood for alternative model})) \\ & = -2\ln\left( \frac{\text{likelihood for null model}}{\text{likelihood for alternative model}} \right). \end{align}
            
            is distributed as chisquare with df equal to difference in number of parameters or equivalently
            difference in residual degrees of freedom  (sign?) 
        
        TODO: put into separate function
        '''
        if not hasattr(self, 'lsjoint'):
            self.fitjoint()
        if not hasattr(self, 'lspooled'):
            self.fitpooled()
        loglikejoint = self.lsjoint.llf
        loglikepooled = self.lspooled.llf
        lrstat = -2*(loglikepooled - loglikejoint)   #??? check sign
        lrdf = self.lspooled.df_resid - self.lsjoint.df_resid
        lrpval = stats.chi2.sf(lrstat, lrdf)
        
        return lrstat, lrpval, lrdf
        
        
if __name__ == '__main__':

    """Example: Test for equality of coefficients across groups/regressions
    
    Created on Sat Mar 27 22:36:51 2010
    Author: josef-pktd
    """
    
    import numpy as np
    from scipy import stats
    #from numpy.testing import assert_almost_equal
    import scikits.statsmodels as sm
    #from scikits.statsmodels.sandbox.regression.try_onewaygls import OneWayLS
    
    #choose example
    #--------------
    example = ['null', 'diff'][1]
    example_size = [10, 100][1]
    example_groups = ['2', '2-2'][1]
    #'2-2': 4 groups, 
    #       groups 0 and 1 and groups 2 and 3 have identical parameters in DGP
    
    #generate example
    #----------------
    np.random.seed(87654589)
    nobs = example_size
    
    #group/regression 1
    x1 = np.random.randn(nobs)
    y1 = 10 + 15*x1 + 2*np.random.randn(nobs)
    x1 = sm.add_constant(x1) #, prepend=True)
    
    #group/regression 2
    x2 = np.random.randn(nobs)
    if example == 'null':
        y2 = 10 + 15*x2 + 2*np.random.randn(nobs)  # if H0 is true
    else:   
        y2 = 19 + 17*x2 + 2*np.random.randn(nobs)
    
    x2 = sm.add_constant(x2)
    
    # stack
    x = np.concatenate((x1,x2),0)
    y = np.concatenate((y1,y2))
    if example_groups == '2':
        groupind = (np.arange(2*nobs)>nobs-1).astype(int)
    else:
        groupind = np.mod(np.arange(2*nobs),4) 
        groupind.sort()
    
    
    def print_results(res):
        groupind = res.groups
        #res.fitjoint()  #not really necessary, because called by ftest_summary
        ft = res.ftest_summary()
        #print ft[0]  #skip because table is nicer
        print '\nTable of F-tests for overall or pairwise equality of coefficients'
        print 'hypothesis F-statistic         p-value  df_denom df_num  reject'
        for row in ft[1]: 
            print row,
            if row[1][1]<0.05:
                print '*'
            else:
                print ''
        print 'Notes: p-values are not corrected for many tests'
        print '       (no Bonferroni correction)'
        print '       * : reject at 5% uncorrected confidence level'
        print 'Null hypothesis: all or pairwise coefficient are the same'
        print 'Alternative hypothesis: all coefficients are different'
        
        print '\nComparison with stats.f_oneway'
        print stats.f_oneway(*[y[groupind==gr] for gr in res.unique])
        print '\nLikelihood Ratio Test' 
        print 'likelihood ratio    p-value       df'
        print res.lr_test()
        print 'Null model: pooled all coefficients are the same across groups,' 
        print 'Alternative model: all coefficients are allowed to be different'
        print 'not verified but looks close to f-test result'
        
        print '\nOls parameters by group from individual, separate ols regressions'
        for group in sorted(res.olsbygroup):
            r = res.olsbygroup[group]
            print group, r.params 
        
        print '\nCheck for heteroscedasticity, '
        print 'variance and standard deviation for individual regressions'
        print ' '*12, ' '.join('group %-10s' %(gr) for gr in res.unique)
        print 'variance    ', res.sigmabygroup
        print 'standard dev', np.sqrt(res.sigmabygroup)
    
    
    #get results for example
    #-----------------------
    
    print '\nTest for equality of coefficients for all exogenous variables'
    print   '-------------------------------------------------------------'
    res = OneWayLS(y,x, groups=groupind.astype(int))
    print_results(res)
    
    print '\nOne way ANOVA, constant is the only regressor'
    print   '---------------------------------------------'
    
    print 'this is the same as scipy.stats.f_oneway'
    res = OneWayLS(y,np.ones(len(y)), groups=groupind)
    print_results(res)
    
    
    print '\nOne way ANOVA, constant is the only regressor with het is true'
    print   '--------------------------------------------------------------'
    
    print 'this is similar to scipy.stats.f_oneway,' 
    print 'but variance is not assumed to be the same across groups'
    res = OneWayLS(y,np.ones(len(y)), groups=groupind, het=True)
    print_results(res)



From christian at prinoth.name  Mon Mar 29 11:35:52 2010
From: christian at prinoth.name (Christian Prinoth)
Date: Mon, 29 Mar 2010 17:35:52 +0200
Subject: [SciPy-User] question about timeseries indexing
Message-ID: <89536841003290835s4c363a54h76de81e28f59384e@mail.gmail.com>

Hi, I am experimenting with scikits.timeseries, and have found a behaviour
that I do not understand.

I have the following time series:
s2 = ts.time_series(np.arange(10),start_date=ts.Date('B','2010-01-01'))

Now, I get the following:

In [79]: s2['2010-01-08']
Out[79]: 5

In [80]: s2['2010-01-09']
Out[80]: 6

In [81]: s2['2010-01-10']
Out[81]: 5

In [82]: s2['2010-01-11']
Out[82]: 6

for lines 80 and 81, I would expect either an error message or 5, but not
two different results.
What is the reason for this behaviour?

Also, more generally, is there some way to index a time series with dates so
that I get the latest available value?

Thanks
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100329/e8678c36/attachment.html>

From mattknox.ca at gmail.com  Mon Mar 29 12:23:10 2010
From: mattknox.ca at gmail.com (Matt Knox)
Date: Mon, 29 Mar 2010 16:23:10 +0000 (UTC)
Subject: [SciPy-User] question about timeseries indexing
References: <89536841003290835s4c363a54h76de81e28f59384e@mail.gmail.com>
Message-ID: <loom.20100329T181822-370@post.gmane.org>

Christian Prinoth <christian <at> prinoth.name> writes:

> for lines 80 and 81, I would expect either an error message or 5, but not two
> different results. What is the reason for this behaviour?

The reason for this is becausee Jan 9 and 10 are Saturday and Sunday and the
series you created is "Business" frequency, so behind the scenes it is
implicitly converting those weekend dates to Friday or Monday. If we were
starting from scratch I would probably raise an error in this situation, but
for now we are preserving that behaviour. Although I am somewhat surprised that
one of the days returns the value from the preceding Friday, and the other from
the following Monday, I'll have to look at that.

> Also, more generally, is there some way to index a time series with dates so
> that I get the latest available value?

Assuming your time series is an ordered series, just index the series with -1
to get the last value. Same as for a regular numpy array.

- Matt



From peridot.faceted at gmail.com  Mon Mar 29 12:24:33 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 29 Mar 2010 12:24:33 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <2588da421003290808t4f2e185cy5b07a35002f85e87@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com> 
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com> 
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com> 
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com> 
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com> 
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com> 
	<2588da421003290808t4f2e185cy5b07a35002f85e87@mail.gmail.com>
Message-ID: <ce557a361003290924q2471e583sa38af2847c6a70fb@mail.gmail.com>

On 29 March 2010 11:08, Jeremy Conlin <jlconlin at gmail.com> wrote:
> On Thu, Mar 25, 2010 at 9:40 PM, David Goldsmith
> <d.l.goldsmith at gmail.com> wrote:
>> On Thu, Mar 25, 2010 at 3:40 PM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>>
>>> Yikes! ?This sounds like it may be more trouble than it's worth. ?I
>>> have a few sets of statistical data that each need to have curves fit
>>> to them.
>>
>> That's an awfully generic need - it may be obvious from examination of the
>> data that a line is inappropriate, but besides polynomials there are many
>> other non-linear models (which can be linearly fit to data by means of data
>> transformation) which possess fewer parameters (and thus are simpler from a
>> parameter analysis perspective).? So, the question is: why are you fitting
>> to polynomials?? If it's just to get a good fit to the data, you might be
>> getting "more fit" than your data warrants (and even if that isn't a
>> problem, you probably want to use a polynomial form different from "standard
>> form," e.g., Lagrange interpolators).? Are you sure something like an
>> exponential growth/decay or power law model (both of which are "more
>> natural," linearizable, two-parameter models) wouldn't be more appropriate -
>> it would almost certainly be simpler to analyze (and perhaps easier to
>> justify to a referee).
>>
>> On this note, perhaps some of our experts might care to comment: what
>> "physics" (in a generalized sense) gives rise to a polynomial dependency of
>> degree higher than two?? The only generic thing I can think of is something
>> where third or higher order derivatives proportional to the independent
>> variable are important, and those are pretty uncommon.
>
> I will only be fitting data to a first or second degree polynomial.
> Eventually I would like to fit my data to an exponential or a power
> law, just to see how it compares to a low-order polynomial. ?Choosing
> these functions was based on qualitative analysis (i.e. "it looks
> quadratic").

I should say, then, that most of my dire-sounding comments can be
ignored in this case as long as you rescale the data to (-1,1) before
fitting your quadratic. (Which, come to think of it, gives you very
nearly the Chebyshev basis, but never mind.)

Exponentials and power laws are even easier, since they're linear when
you take logs of the appropriate coordinates. (Though here too, it
helps to center your data before fitting.)

> The best case scenario would be that I take what I learn from this
> "simple" example and apply it to more difficult problems as they come
> along down the road. ?It appears, however, that it's not so simple to
> apply it to other problems. ?I wish I had more time to learn about
> fitting data to curves. ?I'm sure there are a lot of powerful tools
> that can help.

This is a fine approach; in fact even if you end up doing non-linear
fitting (i.e. fits where the function is not a linear combination of
the parameters), the standard approach to getting errors is to just
use the derivatives of the fitting function at the best fit to treat
the problem as locally linear. (There are more elaborate alternatives
as well, including Monte Carlo approaches, which may be worth looking
into.)

Anne


From christian at prinoth.name  Mon Mar 29 12:44:17 2010
From: christian at prinoth.name (Christian Prinoth)
Date: Mon, 29 Mar 2010 18:44:17 +0200
Subject: [SciPy-User] question about timeseries indexing
In-Reply-To: <loom.20100329T181822-370@post.gmane.org>
References: <89536841003290835s4c363a54h76de81e28f59384e@mail.gmail.com>
	<loom.20100329T181822-370@post.gmane.org>
Message-ID: <89536841003290944i7d7d5fbtf19c2a43f4c83215@mail.gmail.com>

>
> > Also, more generally, is there some way to index a time series with dates
> so
> > that I get the latest available value?
>
> Assuming your time series is an ordered series, just index the series with
> -1
> to get the last value. Same as for a regular numpy array.
>
> Sorry, I was referring to a situation where I want the latest available
value _before_ a given date. for example, If I do

series[sunday]

I want it to return the value from series[friday]

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100329/9003f2cf/attachment.html>

From d.l.goldsmith at gmail.com  Mon Mar 29 13:13:41 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Mon, 29 Mar 2010 10:13:41 -0700
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <2588da421003290808t4f2e185cy5b07a35002f85e87@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
	<2588da421003290808t4f2e185cy5b07a35002f85e87@mail.gmail.com>
Message-ID: <45d1ab481003291013s48e4590bv51fdcea9fe2c1600@mail.gmail.com>

On Mon, Mar 29, 2010 at 8:08 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:

> On Thu, Mar 25, 2010 at 9:40 PM, David Goldsmith
> <d.l.goldsmith at gmail.com> wrote:
> > On Thu, Mar 25, 2010 at 3:40 PM, Jeremy Conlin <jlconlin at gmail.com>
> wrote:
> >>
> >> Yikes!  This sounds like it may be more trouble than it's worth.  I
> >> have a few sets of statistical data that each need to have curves fit
> >> to them.
> >
> > That's an awfully generic need - it may be obvious from examination of
> the
> > data that a line is inappropriate, but besides polynomials there are many
> > other non-linear models (which can be linearly fit to data by means of
> data
> > transformation) which possess fewer parameters (and thus are simpler from
> a
> > parameter analysis perspective).  So, the question is: why are you
> fitting
> > to polynomials?  If it's just to get a good fit to the data, you might be
> > getting "more fit" than your data warrants (and even if that isn't a
> > problem, you probably want to use a polynomial form different from
> "standard
> > form," e.g., Lagrange interpolators).  Are you sure something like an
> > exponential growth/decay or power law model (both of which are "more
> > natural," linearizable, two-parameter models) wouldn't be more
> appropriate -
> > it would almost certainly be simpler to analyze (and perhaps easier to
> > justify to a referee).
> >
> > On this note, perhaps some of our experts might care to comment: what
> > "physics" (in a generalized sense) gives rise to a polynomial dependency
> of
> > degree higher than two?  The only generic thing I can think of is
> something
> > where third or higher order derivatives proportional to the independent
> > variable are important, and those are pretty uncommon.
>
> I will only be fitting data to a first or second degree polynomial.
> Eventually I would like to fit my data to an exponential or a power
> law, just to see how it compares to a low-order polynomial.  Choosing
> these functions was based on qualitative analysis (i.e. "it looks
> quadratic").
>

This is a good approach if applied to the right data: the residuals, i.e.,
the (signed, not absolute value) differences between the predicted and
corresponding measured dependent variable values.  For example, if you fit a
line to quadratic data and then plot the residuals v. the independent
variable, you'll see systematic error: most of the residuals at the
extremities will be on one side of zero, while most of those around the
center will be on the other side of zero, i.e., the residuals will look like
a parabola.  If you then fit the original data to a quadratic model (and a
quadratic model is "physically" appropriate) and plot the residuals, these
should appear to be randomly (and rather symmetrically) distributed around
zero.  This generalizes: systematic behavior of residuals reveals
deficiencies in the assumed form of your model - the "best" (most
"physical") model produces residuals randomly (though not necessarily
uniformly) distributed around zero (because the "best" model captures all
non-randomness in your data).  Moral: when doing regression where you're not
a priori certain of your model, always plot the residuals.

DG




>
> The best case scenario would be that I take what I learn from this
> "simple" example and apply it to more difficult problems as they come
> along down the road.  It appears, however, that it's not so simple to
> apply it to other problems.  I wish I had more time to learn about
> fitting data to curves.  I'm sure there are a lot of powerful tools
> that can help.
>
> Jeremy
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100329/181e8116/attachment.html>

From d.l.goldsmith at gmail.com  Mon Mar 29 13:31:52 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Mon, 29 Mar 2010 10:31:52 -0700
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <3d375d731003290734w6234071dke3a093e97dc0ac0a@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
	<C97487CD346EEA4E80423F51F6D1EB2FA98322E7DE@EXCHANGE-00.GeoSyntec.net>
	<20100328133653.35b8cbfd@ajackson.org>
	<45d1ab481003282026y3824c114r7cea2948c90e5c4@mail.gmail.com>
	<3d375d731003290734w6234071dke3a093e97dc0ac0a@mail.gmail.com>
Message-ID: <45d1ab481003291031h76306f29xc53017f557a9bbf8@mail.gmail.com>

On Mon, Mar 29, 2010 at 7:34 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Sun, Mar 28, 2010 at 22:26, David Goldsmith <d.l.goldsmith at gmail.com>
> wrote:
> > On Sun, Mar 28, 2010 at 11:36 AM, <alan at ajackson.org> wrote:
> >>
> >> Crack permeability goes like the third power of the opening (that is,
> >> fluid flow through cracks - think gas or oil in a fractured rock).
> >
> > Power law or polynomial: from a regression stand point, there's quite a
> big
> > difference.
>
> "Third power" == "x**3". He's not talking about a power law.
>

Yes, I know that, but from a regression stand point, unless there's an
offset (constant) term (in which case a two parameter polynomial fit is what
you'll be doing) if your model is simply y = ax**3, aren't you better off
doing the regression as if you were doing a power law, albeit w/ a fixed
power (i.e., log transforming the data first, fixing the slope parameter at
three, and then regressing to find the constant term, i.e., log(a))?

In other words, I was soliciting examples of situations where a true
polynomial (as opposed to a monomial) model was appropriate - I maintain
that if your model is a monomial (integer power law model, w/ only one
term), then, as far as regression is concerned, it is more appropriate to
think of it as a power law model w/ a fixed parameter, not as a polynomial
model.  From this perspective, the number of "naturally occurring"
polynomial models is greatly reduced.

DG


>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100329/7ae13ac8/attachment.html>

From josef.pktd at gmail.com  Mon Mar 29 13:47:15 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 29 Mar 2010 13:47:15 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <45d1ab481003291031h76306f29xc53017f557a9bbf8@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
	<C97487CD346EEA4E80423F51F6D1EB2FA98322E7DE@EXCHANGE-00.GeoSyntec.net>
	<20100328133653.35b8cbfd@ajackson.org>
	<45d1ab481003282026y3824c114r7cea2948c90e5c4@mail.gmail.com>
	<3d375d731003290734w6234071dke3a093e97dc0ac0a@mail.gmail.com>
	<45d1ab481003291031h76306f29xc53017f557a9bbf8@mail.gmail.com>
Message-ID: <1cd32cbb1003291047i62a599e9wb22119a37125173a@mail.gmail.com>

On Mon, Mar 29, 2010 at 1:31 PM, David Goldsmith
<d.l.goldsmith at gmail.com> wrote:
> On Mon, Mar 29, 2010 at 7:34 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Sun, Mar 28, 2010 at 22:26, David Goldsmith <d.l.goldsmith at gmail.com>
>> wrote:
>> > On Sun, Mar 28, 2010 at 11:36 AM, <alan at ajackson.org> wrote:
>> >>
>> >> Crack permeability goes like the third power of the opening (that is,
>> >> fluid flow through cracks - think gas or oil in a fractured rock).
>> >
>> > Power law or polynomial: from a regression stand point, there's quite a
>> > big
>> > difference.
>>
>> "Third power" == "x**3". He's not talking about a power law.
>
> Yes, I know that, but from a regression stand point, unless there's an
> offset (constant) term (in which case a two parameter polynomial fit is what
> you'll be doing) if your model is simply y = ax**3, aren't you better off
> doing the regression as if you were doing a power law, albeit w/ a fixed
> power (i.e., log transforming the data first, fixing the slope parameter at
> three, and then regressing to find the constant term, i.e., log(a))?
>
> In other words, I was soliciting examples of situations where a true
> polynomial (as opposed to a monomial) model was appropriate - I maintain
> that if your model is a monomial (integer power law model, w/ only one
> term), then, as far as regression is concerned, it is more appropriate to
> think of it as a power law model w/ a fixed parameter, not as a polynomial
> model.? From this perspective, the number of "naturally occurring"
> polynomial models is greatly reduced.

That looks to me like splitting hairs.

It depends on the statistical model for the regression error, e.g.

y = ax**3 + u   where u is normal, additive noise,
or
y = ax**3 *z   where z is log-normal, multiplicative noise
ln(y) = ln(a) + 3*ln(x) + u  with u = ln(z)

I would do it if I want to estimate or test if 3 is the correct power,
but not if 3 is known.

Josef

>
> DG
>
>>
>> --
>> Robert Kern
>>
>> "I have come to believe that the whole world is an enigma, a harmless
>> enigma that is made terrible by our own mad attempt to interpret it as
>> though it had an underlying truth."
>> ?-- Umberto Eco
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From d.l.goldsmith at gmail.com  Mon Mar 29 13:55:52 2010
From: d.l.goldsmith at gmail.com (David Goldsmith)
Date: Mon, 29 Mar 2010 10:55:52 -0700
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <1cd32cbb1003291047i62a599e9wb22119a37125173a@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
	<C97487CD346EEA4E80423F51F6D1EB2FA98322E7DE@EXCHANGE-00.GeoSyntec.net>
	<20100328133653.35b8cbfd@ajackson.org>
	<45d1ab481003282026y3824c114r7cea2948c90e5c4@mail.gmail.com>
	<3d375d731003290734w6234071dke3a093e97dc0ac0a@mail.gmail.com>
	<45d1ab481003291031h76306f29xc53017f557a9bbf8@mail.gmail.com>
	<1cd32cbb1003291047i62a599e9wb22119a37125173a@mail.gmail.com>
Message-ID: <45d1ab481003291055w337064f0kfdf7915f983e0025@mail.gmail.com>

On Mon, Mar 29, 2010 at 10:47 AM, <josef.pktd at gmail.com> wrote:

> On Mon, Mar 29, 2010 at 1:31 PM, David Goldsmith
> <d.l.goldsmith at gmail.com> wrote:
> > On Mon, Mar 29, 2010 at 7:34 AM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Sun, Mar 28, 2010 at 22:26, David Goldsmith <d.l.goldsmith at gmail.com
> >
> >> wrote:
> >> > On Sun, Mar 28, 2010 at 11:36 AM, <alan at ajackson.org> wrote:
> >> >>
> >> >> Crack permeability goes like the third power of the opening (that is,
> >> >> fluid flow through cracks - think gas or oil in a fractured rock).
> >> >
> >> > Power law or polynomial: from a regression stand point, there's quite
> a
> >> > big
> >> > difference.
> >>
> >> "Third power" == "x**3". He's not talking about a power law.
> >
> > Yes, I know that, but from a regression stand point, unless there's an
> > offset (constant) term (in which case a two parameter polynomial fit is
> what
> > you'll be doing) if your model is simply y = ax**3, aren't you better off
> > doing the regression as if you were doing a power law, albeit w/ a fixed
> > power (i.e., log transforming the data first, fixing the slope parameter
> at
> > three, and then regressing to find the constant term, i.e., log(a))?
> >
> > In other words, I was soliciting examples of situations where a true
> > polynomial (as opposed to a monomial) model was appropriate - I maintain
> > that if your model is a monomial (integer power law model, w/ only one
> > term), then, as far as regression is concerned, it is more appropriate to
> > think of it as a power law model w/ a fixed parameter, not as a
> polynomial
> > model.  From this perspective, the number of "naturally occurring"
> > polynomial models is greatly reduced.
>
> That looks to me like splitting hairs.
>
> It depends on the statistical model for the regression error, e.g.
>
> y = ax**3 + u   where u is normal, additive noise,
> or
> y = ax**3 *z   where z is log-normal, multiplicative noise
> ln(y) = ln(a) + 3*ln(x) + u  with u = ln(z)
>
> I would do it


Do what?

DG


> if I want to estimate or test if 3 is the correct power,
> but not if 3 is known.
>
> Josef
>
> >
> > DG
> >
> >>
> >> --
> >> Robert Kern
> >>
> >> "I have come to believe that the whole world is an enigma, a harmless
> >> enigma that is made terrible by our own mad attempt to interpret it as
> >> though it had an underlying truth."
> >>  -- Umberto Eco
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100329/05e45491/attachment.html>

From josef.pktd at gmail.com  Mon Mar 29 13:59:58 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 29 Mar 2010 13:59:58 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <45d1ab481003291055w337064f0kfdf7915f983e0025@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com>
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com>
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com>
	<C97487CD346EEA4E80423F51F6D1EB2FA98322E7DE@EXCHANGE-00.GeoSyntec.net>
	<20100328133653.35b8cbfd@ajackson.org>
	<45d1ab481003282026y3824c114r7cea2948c90e5c4@mail.gmail.com>
	<3d375d731003290734w6234071dke3a093e97dc0ac0a@mail.gmail.com>
	<45d1ab481003291031h76306f29xc53017f557a9bbf8@mail.gmail.com>
	<1cd32cbb1003291047i62a599e9wb22119a37125173a@mail.gmail.com>
	<45d1ab481003291055w337064f0kfdf7915f983e0025@mail.gmail.com>
Message-ID: <1cd32cbb1003291059v2258bf34m4f8ed65f9bb68ddb@mail.gmail.com>

On Mon, Mar 29, 2010 at 1:55 PM, David Goldsmith
<d.l.goldsmith at gmail.com> wrote:
> On Mon, Mar 29, 2010 at 10:47 AM, <josef.pktd at gmail.com> wrote:
>>
>> On Mon, Mar 29, 2010 at 1:31 PM, David Goldsmith
>> <d.l.goldsmith at gmail.com> wrote:
>> > On Mon, Mar 29, 2010 at 7:34 AM, Robert Kern <robert.kern at gmail.com>
>> > wrote:
>> >>
>> >> On Sun, Mar 28, 2010 at 22:26, David Goldsmith
>> >> <d.l.goldsmith at gmail.com>
>> >> wrote:
>> >> > On Sun, Mar 28, 2010 at 11:36 AM, <alan at ajackson.org> wrote:
>> >> >>
>> >> >> Crack permeability goes like the third power of the opening (that
>> >> >> is,
>> >> >> fluid flow through cracks - think gas or oil in a fractured rock).
>> >> >
>> >> > Power law or polynomial: from a regression stand point, there's quite
>> >> > a
>> >> > big
>> >> > difference.
>> >>
>> >> "Third power" == "x**3". He's not talking about a power law.
>> >
>> > Yes, I know that, but from a regression stand point, unless there's an
>> > offset (constant) term (in which case a two parameter polynomial fit is
>> > what
>> > you'll be doing) if your model is simply y = ax**3, aren't you better
>> > off
>> > doing the regression as if you were doing a power law, albeit w/ a fixed
>> > power (i.e., log transforming the data first, fixing the slope parameter
>> > at
>> > three, and then regressing to find the constant term, i.e., log(a))?
>> >
>> > In other words, I was soliciting examples of situations where a true
>> > polynomial (as opposed to a monomial) model was appropriate - I maintain
>> > that if your model is a monomial (integer power law model, w/ only one
>> > term), then, as far as regression is concerned, it is more appropriate
>> > to
>> > think of it as a power law model w/ a fixed parameter, not as a
>> > polynomial
>> > model.? From this perspective, the number of "naturally occurring"
>> > polynomial models is greatly reduced.
>>
>> That looks to me like splitting hairs.
>>
>> It depends on the statistical model for the regression error, e.g.
>>
>> y = ax**3 + u ? where u is normal, additive noise,
>> or
>> y = ax**3 *z ? where z is log-normal, multiplicative noise
>> ln(y) = ln(a) + 3*ln(x) + u ?with u = ln(z)
>>
>> I would do it
>
> Do what?

"it" = (i.e., log transforming the data first, fixing the slope parameter at
       three, and then regressing to find the constant term, i.e., log(a))?

Josef

>
> DG
>
>>
>> if I want to estimate or test if 3 is the correct power,
>> but not if 3 is known.
>>
>> Josef
>>
>> >
>> > DG
>> >
>> >>
>> >> --
>> >> Robert Kern
>> >>
>> >> "I have come to believe that the whole world is an enigma, a harmless
>> >> enigma that is made terrible by our own mad attempt to interpret it as
>> >> though it had an underlying truth."
>> >> ?-- Umberto Eco
>> >> _______________________________________________
>> >> SciPy-User mailing list
>> >> SciPy-User at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> >
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> >
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From peridot.faceted at gmail.com  Mon Mar 29 14:02:09 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 29 Mar 2010 14:02:09 -0400
Subject: [SciPy-User] How to estimate error in polynomial coefficients
	from scipy.polyfit?
In-Reply-To: <45d1ab481003291013s48e4590bv51fdcea9fe2c1600@mail.gmail.com>
References: <2588da421003251332x288b7520y11e6aa0931631e09@mail.gmail.com> 
	<e06186141003251352o27a5c25amdc199bf34d1162cb@mail.gmail.com> 
	<2588da421003251420y2fcb3afn55f3f5e543ce542c@mail.gmail.com> 
	<ce557a361003251521p51a12f49nbf9ef7aef0532642@mail.gmail.com> 
	<2588da421003251540p2f6bab49lc189b964c65ca45b@mail.gmail.com> 
	<45d1ab481003252040m3ae67a15ucc8af3c7aa60cda2@mail.gmail.com> 
	<2588da421003290808t4f2e185cy5b07a35002f85e87@mail.gmail.com> 
	<45d1ab481003291013s48e4590bv51fdcea9fe2c1600@mail.gmail.com>
Message-ID: <ce557a361003291102n28e1b028k4f9092eff4b7d970@mail.gmail.com>

On 29 March 2010 13:13, David Goldsmith <d.l.goldsmith at gmail.com> wrote:
> On Mon, Mar 29, 2010 at 8:08 AM, Jeremy Conlin <jlconlin at gmail.com> wrote:
>>
>> On Thu, Mar 25, 2010 at 9:40 PM, David Goldsmith
>> <d.l.goldsmith at gmail.com> wrote:
>> > On Thu, Mar 25, 2010 at 3:40 PM, Jeremy Conlin <jlconlin at gmail.com>
>> > wrote:
>> >>
>> >> Yikes!  This sounds like it may be more trouble than it's worth.  I
>> >> have a few sets of statistical data that each need to have curves fit
>> >> to them.
>> >
>> > That's an awfully generic need - it may be obvious from examination of
>> > the
>> > data that a line is inappropriate, but besides polynomials there are
>> > many
>> > other non-linear models (which can be linearly fit to data by means of
>> > data
>> > transformation) which possess fewer parameters (and thus are simpler
>> > from a
>> > parameter analysis perspective).  So, the question is: why are you
>> > fitting
>> > to polynomials?  If it's just to get a good fit to the data, you might
>> > be
>> > getting "more fit" than your data warrants (and even if that isn't a
>> > problem, you probably want to use a polynomial form different from
>> > "standard
>> > form," e.g., Lagrange interpolators).  Are you sure something like an
>> > exponential growth/decay or power law model (both of which are "more
>> > natural," linearizable, two-parameter models) wouldn't be more
>> > appropriate -
>> > it would almost certainly be simpler to analyze (and perhaps easier to
>> > justify to a referee).
>> >
>> > On this note, perhaps some of our experts might care to comment: what
>> > "physics" (in a generalized sense) gives rise to a polynomial dependency
>> > of
>> > degree higher than two?  The only generic thing I can think of is
>> > something
>> > where third or higher order derivatives proportional to the independent
>> > variable are important, and those are pretty uncommon.
>>
>> I will only be fitting data to a first or second degree polynomial.
>> Eventually I would like to fit my data to an exponential or a power
>> law, just to see how it compares to a low-order polynomial.  Choosing
>> these functions was based on qualitative analysis (i.e. "it looks
>> quadratic").
>
> This is a good approach if applied to the right data: the residuals, i.e.,
> the (signed, not absolute value) differences between the predicted and
> corresponding measured dependent variable values.  For example, if you fit a
> line to quadratic data and then plot the residuals v. the independent
> variable, you'll see systematic error: most of the residuals at the
> extremities will be on one side of zero, while most of those around the
> center will be on the other side of zero, i.e., the residuals will look like
> a parabola.  If you then fit the original data to a quadratic model (and a
> quadratic model is "physically" appropriate) and plot the residuals, these
> should appear to be randomly (and rather symmetrically) distributed around
> zero.  This generalizes: systematic behavior of residuals reveals
> deficiencies in the assumed form of your model - the "best" (most
> "physical") model produces residuals randomly (though not necessarily
> uniformly) distributed around zero (because the "best" model captures all
> non-randomness in your data).  Moral: when doing regression where you're not
> a priori certain of your model, always plot the residuals.

I would shorten that to "always plot the residuals". In fact, that's
one of the problems with unbinned maximum-likelihood methods: there
often isn't any feasible notion of "residual".

As a quality-of-fit measure, the standard tool is the chi-squared
distribution: if you (linearly) fit an n-parameter model to an m-point
data set, and the model captures the true behaviour of your system,
then the sum of the squared errors divided by the variances is
distributed according to a chi-squared distribution with m-n degrees
of freedom. So you can calculate this number and evaluate the
likelihood that the Gaussian noise would produce a quality-of-fit this
bad. If you get a tiny probability, you can be pretty sure your model
is not fitting the data adequately.

Also worth noting is the highly annoying situation where you have what
purports to be a good fit: the chi-squared gives you a decent
probability of noise producing a fit this bad, and the error bars on
the residuals all nicely include zero, but there's *still* a
discernible trend in the residuals. I interpret this to mean that not
only does my model not capture the system's behaviour, but I have also
overestimated my error bars: the fact that there's a good fit in spite
of the trend means that there is not as much scatter in the residuals
as there should be.

Anne

> DG
>
>
>
>>
>> The best case scenario would be that I take what I learn from this
>> "simple" example and apply it to more difficult problems as they come
>> along down the road.  It appears, however, that it's not so simple to
>> apply it to other problems.  I wish I had more time to learn about
>> fitting data to curves.  I'm sure there are a lot of powerful tools
>> that can help.
>>
>> Jeremy
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From moritz.maus at uni-jena.de  Mon Mar 29 15:31:31 2010
From: moritz.maus at uni-jena.de (Moritz Maus)
Date: Mon, 29 Mar 2010 21:31:31 +0200
Subject: [SciPy-User] Zero crossing detection
Message-ID: <4BB10013.505@uni-jena.de>

Is there a way for the ODE-integrators to
a.) stop the simulation when a specific event occurred or
b.) doing a MatLab (Simulink)-like "zero crossing detection", (when 
these timepoints are not pre-defined?)



From mattknox.ca at gmail.com  Mon Mar 29 15:45:08 2010
From: mattknox.ca at gmail.com (Matt Knox)
Date: Mon, 29 Mar 2010 19:45:08 +0000 (UTC)
Subject: [SciPy-User] question about timeseries indexing
References: <89536841003290835s4c363a54h76de81e28f59384e@mail.gmail.com>
	<loom.20100329T181822-370@post.gmane.org>
	<89536841003290944i7d7d5fbtf19c2a43f4c83215@mail.gmail.com>
Message-ID: <loom.20100329T213929-940@post.gmane.org>

Christian Prinoth <christian <at> prinoth.name> writes:

> Sorry, I was referring to a situation where I want the latest available value
> _before_ a given date. for example, If I do
> 
> series[sunday]
> 
> I want it to return the value from series[friday]

Then in that case it would be better to explicitly convert the date and specify
the behaviour you want.

For example:

input_date_D = ts.Date('D', '2010-01-10')
input_date_B = input_date_D.asfreq('B', 'BEFORE')
value = series[input_date_B]

In the case where input_date_D is already a week day, input_date_B will end up
being the same as the original input date, if input_date_D was a weekend, it
would be converted to the Friday before.

- Matt




From bouloumag at gmail.com  Mon Mar 29 16:54:10 2010
From: bouloumag at gmail.com (Darcoux Christine)
Date: Mon, 29 Mar 2010 16:54:10 -0400
Subject: [SciPy-User] preconditioner for GMRES
Message-ID: <477ac73a1003291354n25cf2d2an8f481c40cedb7dae@mail.gmail.com>

I am developping a Jacobian-Free Newton-Krylov code to solve the Navier
Stokes equations. In this code, the product of the Jacobian matrix with a
given vector is represented by the matvec method of a
"LinearOperator"object. The
scipy.sparse.linalg.isolve.gmres method can takes an optional preconditioner
as parameter which is either an object "LinearOperator" or a matrix. However,
the scipy documentation does not explain how one can form this
preconditioner in a (ideally) matrix-free manner given only the matvec
method of the jacobian.

Any suggestions on this would be very appreciated.

Thanks,

Christine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100329/c0d6b614/attachment.html>

From peridot.faceted at gmail.com  Mon Mar 29 16:56:13 2010
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Mon, 29 Mar 2010 16:56:13 -0400
Subject: [SciPy-User] Zero crossing detection
In-Reply-To: <4BB10013.505@uni-jena.de>
References: <4BB10013.505@uni-jena.de>
Message-ID: <ce557a361003291356p1ee03708y133fba78cd36d104@mail.gmail.com>

On 29 March 2010 15:31, Moritz Maus <moritz.maus at uni-jena.de> wrote:
> Is there a way for the ODE-integrators to
> a.) stop the simulation when a specific event occurred or
> b.) doing a MatLab (Simulink)-like "zero crossing detection", (when
> these timepoints are not pre-defined?)

The short answer is no. (And we apologize for this.)

There's a much more powerful set of tools available for working with
ODEs provided in the python package pydstool; they provide many
features including this one. Alternatively, there are a number of
wrappers for the ODEPACK integrator LSODAR (which is included but not
wrapped in scipy) which provides exactly this capability. Finally, you
can get some approximation of this behviour using scipy's integrators;
they can be made to advance "one step" of the underlying integrator at
a time, and while in this mode they can be used to "backtrack" as far
as the beginning of the step. So you can just keep advancing until you
meet your criterion then backtrack.

This brings me to a warning: no ODE solver I know of is able to deal
with "here there be dragons"-type events, that is, situations where
you not only want the integrator to stop, but where it is not allowed
to evaluate anything past that point (perhaps because your RHS
function yields nonsense). The standard approach is to write RHS
functions so that they detect out-of-bounds conditions and return some
not-too-offensive value, hoping that this won't send the integrator
into a frenzy.

Anne

> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From timmichelsen at gmx-topmail.de  Mon Mar 29 17:41:44 2010
From: timmichelsen at gmx-topmail.de (Tim Michelsen)
Date: Mon, 29 Mar 2010 23:41:44 +0200
Subject: [SciPy-User] ANN: Veusz 1.7
In-Reply-To: <hoo0l0$nja$1@dough.gmane.org>
References: <hoo0l0$nja$1@dough.gmane.org>
Message-ID: <hor6qo$bmn$1@dough.gmane.org>

>  * Text, CSV and FITS importing
Are you planning to support hdf5 in the future?
What about XLS/ODS tables?



From joshua.stults at gmail.com  Mon Mar 29 18:19:24 2010
From: joshua.stults at gmail.com (Joshua Stults)
Date: Mon, 29 Mar 2010 18:19:24 -0400
Subject: [SciPy-User] preconditioner for GMRES
In-Reply-To: <477ac73a1003291354n25cf2d2an8f481c40cedb7dae@mail.gmail.com>
References: <477ac73a1003291354n25cf2d2an8f481c40cedb7dae@mail.gmail.com>
Message-ID: <c5ae9cf01003291519i23d9e082ved1a2e1471ecdf9b@mail.gmail.com>

On Mon, Mar 29, 2010 at 4:54 PM, Darcoux Christine <bouloumag at gmail.com> wrote:
> I am developping a Jacobian-Free Newton-Krylov code to solve the Navier
> Stokes equations. In this code, the product of the Jacobian matrix with a
> given vector is represented by the matvec method of a "LinearOperator"
> object. The scipy.sparse.linalg.isolve.gmres method can takes an optional
> preconditioner as parameter which is either an object "LinearOperator" or a
> matrix. However, the scipy documentation does not explain how one can form
> this preconditioner in a (ideally) matrix-free manner given only the matvec
> method of the jacobian.
>
> Any suggestions on this would be very appreciated.
>

A pretty straight-forward way to precondition these types of problems
in a matrix-free way is with symmetric successive over relaxation (the
symmetric part is important, you need to do a 'forward' and a
'backward' sweep or things won't converge):
http://www.netlib.org/linalg/old_html_templates/subsection2.6.2.4.html

You'll have to derive the update formulas yourself based on the type
of discretization you are using;  if you are using a transform based
method, then you can still precondition based on a low-order scheme.
IIRC Boyd has a pretty decent discussion of this in his spectral
methods book.

Maybe you could construct one of the approximate factorization
preconditioners using only the matvec of your Jacobian, but I've never
gone with this approach, maybe someone else could chime in with
recommendations along those lines?
http://www.netlib.org/linalg/old_html_templates/chapter2.7.html

What sort of spatial discretization are you using?

> Thanks,
>
> Christine
>

HtH,
-- 
Joshua Stults
Website: http://j-stults.blogspot.com


From bouloumag at gmail.com  Mon Mar 29 18:59:37 2010
From: bouloumag at gmail.com (Darcoux Christine)
Date: Mon, 29 Mar 2010 18:59:37 -0400
Subject: [SciPy-User] preconditioner for GMRES
In-Reply-To: <c5ae9cf01003291519i23d9e082ved1a2e1471ecdf9b@mail.gmail.com>
References: <477ac73a1003291354n25cf2d2an8f481c40cedb7dae@mail.gmail.com>
	<c5ae9cf01003291519i23d9e082ved1a2e1471ecdf9b@mail.gmail.com>
Message-ID: <477ac73a1003291559v64cea8d3i9eb133f81da832e3@mail.gmail.com>

Thnak you. I will look at these links.

I am using the finite volume method, with QUICK advection (maybe I will try
flux limiters in the future if there are too much oscillations).

2010/3/29 Joshua Stults <joshua.stults at gmail.com>

> On Mon, Mar 29, 2010 at 4:54 PM, Darcoux Christine <bouloumag at gmail.com>
> wrote:
> > I am developping a Jacobian-Free Newton-Krylov code to solve the Navier
> > Stokes equations. In this code, the product of the Jacobian matrix with a
> > given vector is represented by the matvec method of a "LinearOperator"
> > object. The scipy.sparse.linalg.isolve.gmres method can takes an optional
> > preconditioner as parameter which is either an object "LinearOperator" or
> a
> > matrix. However, the scipy documentation does not explain how one can
> form
> > this preconditioner in a (ideally) matrix-free manner given only the
> matvec
> > method of the jacobian.
> >
> > Any suggestions on this would be very appreciated.
> >
>
> A pretty straight-forward way to precondition these types of problems
> in a matrix-free way is with symmetric successive over relaxation (the
> symmetric part is important, you need to do a 'forward' and a
> 'backward' sweep or things won't converge):
> http://www.netlib.org/linalg/old_html_templates/subsection2.6.2.4.html
>
> You'll have to derive the update formulas yourself based on the type
> of discretization you are using;  if you are using a transform based
> method, then you can still precondition based on a low-order scheme.
> IIRC Boyd has a pretty decent discussion of this in his spectral
> methods book.
>
> Maybe you could construct one of the approximate factorization
> preconditioners using only the matvec of your Jacobian, but I've never
> gone with this approach, maybe someone else could chime in with
> recommendations along those lines?
> http://www.netlib.org/linalg/old_html_templates/chapter2.7.html
>
> What sort of spatial discretization are you using?
>
> > Thanks,
> >
> > Christine
> >
>
> HtH,
> --
> Joshua Stults
> Website: http://j-stults.blogspot.com
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100329/dec5e3fd/attachment.html>

From lafras at sun.ac.za  Tue Mar 30 03:45:58 2010
From: lafras at sun.ac.za (Lafras Uys)
Date: Tue, 30 Mar 2010 09:45:58 +0200
Subject: [SciPy-User] Zero crossing detection
In-Reply-To: <ce557a361003291356p1ee03708y133fba78cd36d104@mail.gmail.com>
References: <4BB10013.505@uni-jena.de>
	<ce557a361003291356p1ee03708y133fba78cd36d104@mail.gmail.com>
Message-ID: <4BB1AC36.5000601@sun.ac.za>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> On 29 March 2010 15:31, Moritz Maus <moritz.maus at uni-jena.de> wrote:
>> Is there a way for the ODE-integrators to
>> a.) stop the simulation when a specific event occurred or
>> b.) doing a MatLab (Simulink)-like "zero crossing detection", (when
>> these timepoints are not pre-defined?)
> 
> The short answer is no. (And we apologize for this.)
> 
> There's a much more powerful set of tools available for working with
> ODEs provided in the python package pydstool; they provide many
> features including this one. Alternatively, there are a number of
> wrappers for the ODEPACK integrator LSODAR (which is included but not
> wrapped in scipy) which provides exactly this capability. Finally, you
> can get some approximation of this behviour using scipy's integrators;
> they can be made to advance "one step" of the underlying integrator at
> a time, and while in this mode they can be used to "backtrack" as far
> as the beginning of the step. So you can just keep advancing until you
> meet your criterion then backtrack.
> 
> This brings me to a warning: no ODE solver I know of is able to deal
> with "here there be dragons"-type events, that is, situations where
> you not only want the integrator to stop, but where it is not allowed
> to evaluate anything past that point (perhaps because your RHS
> function yields nonsense). The standard approach is to write RHS
> functions so that they detect out-of-bounds conditions and return some
> not-too-offensive value, hoping that this won't send the integrator
> into a frenzy.

Not SciPy either, but PySUNDIALS (http://pysundials.sourceforge.net/) is
a set of python bindings for SUNDIALS. SUNDIALS can handle triggers,
events, delays and so forth.

Lafras
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuxrCwACgkQKUpCd+bV+kpYwACgkJ25xeI4OtDTtPAn52o9SCJt
abcAoKAjJnWnNDk4NrVXamXkUhFTzk5n
=BDRY
-----END PGP SIGNATURE-----


From jeremy at jeremysanders.net  Tue Mar 30 05:18:56 2010
From: jeremy at jeremysanders.net (Jeremy Sanders)
Date: Tue, 30 Mar 2010 10:18:56 +0100
Subject: [SciPy-User] ANN: Veusz 1.7
References: <hoo0l0$nja$1@dough.gmane.org> <hor6qo$bmn$1@dough.gmane.org>
Message-ID: <hosfm0$rj7$1@dough.gmane.org>

Tim Michelsen wrote:

>>  * Text, CSV and FITS importing
> Are you planning to support hdf5 in the future?
> What about XLS/ODS tables?

Maybe HDF5 at least. Someone kindly made me an HDF5 patch a while ago that I 
dropped because of other pressures. I'll have to hunt for it and fix it for 
the latest version. I think the problem was that I couldn't find any example 
HDF5 tables to try it on.

I haven't thought about XLS and ODS. If there's a python module that makes 
them look like a CSV file to my code then adding them should be pretty easy.

Jeremy



From g.statkute at gmail.com  Tue Mar 30 07:52:17 2010
From: g.statkute at gmail.com (gintare statkute)
Date: Tue, 30 Mar 2010 14:52:17 +0300
Subject: [SciPy-User] installation, Espoo-Helsinki
Message-ID: <8e7295c71003300452w6c59f86cgb6528f5be6851340@mail.gmail.com>

Hello,

I tried to install scientific-python one - two months before, but got in
problems with ATLAS installation.
I tried several times, thus now LAPACK maybe also has to be reinstalled.
I really would like to try some simulations with scientific python, maybe
somebody could help me in installation.

regards,
gintare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100330/827806dd/attachment.html>

From almar.klein at gmail.com  Tue Mar 30 08:16:21 2010
From: almar.klein at gmail.com (Almar Klein)
Date: Tue, 30 Mar 2010 14:16:21 +0200
Subject: [SciPy-User] ANN: visvis 1.2
Message-ID: <h2scc38d75f1003300516ua49048c3l3577ffdbba0a6ec@mail.gmail.com>

Hi all,

I am pleased to announce version 1.2 of visvis, an object oriented approach
to visualization of 1D to 4D data.

Website: http://code.google.com/p/visvis/
Discussion group: http://groups.google.com/group/visvis/
Documentation: http://code.google.com/p/visvis/wiki/Visvis_basics

Since the last release, much work has been done on the interaction with the
backend GUI toolkits, making visvis work (better) on Linux. As a result of
these changes, the FLTK backend is now also stable. For more changes see the
release notes <http://code.google.com/p/visvis/wiki/releaseNotes>.


=== Description ===
Visvis is a pure Python library for visualization of 1D to 4D data in an
object oriented way. It can visualize plots (lines with markers), show
images, and render 3D volumes. These data can also be moved in time. Visvis
is cross-platform, lightweight, and is designed to be easily extendable.

Essentially, visvis is an object oriented layer of Python on top of OpenGl,
thereby combining the power of OpenGl with the usability of Python. A
Matlab-like interface in the form of a set of functions allows easy creation
of objects (e.g. plot(), imshow(), volshow()).

Visvis can be used in Python scripts, interactive Python shells (as with
IPython or IEP) and can be embedded in GUI applications.

Regards,
  Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100330/404a061c/attachment.html>

From amenity at enthought.com  Tue Mar 30 10:07:49 2010
From: amenity at enthought.com (Amenity Applewhite)
Date: Tue, 30 Mar 2010 09:07:49 -0500
Subject: [SciPy-User] SciPy 2010: Vote on Tutorials & Sprints
References: <e91b4574d5d1709a9dc4f7ab71ddd26b9f5.20100330140005@mcsv117.net>
Message-ID: <81C97727-3E4E-43BA-B4D0-6D7C95C6BCB4@enthought.com>

Email not displaying correctly? View it in your browser.

Hello Amenity,

Spring is upon us and arrangements for SciPy 2010 are in full swing.   
We're already nearing on some important deadlines for conference  
participants: April 11th is the deadline for submitting an abstract  
for a paper, and April 15th is the deadline for submitting a tutorial  
proposal.

Help choose tutorials for SciPy 2010...
We set up a UserVoice page to brainstorm tutorial topics last week and  
we already have some great ideas. The top ones at the moment are:

Effective multi-core programming with Cython and Python
Building your own app with Mayavi
High performance computing with Python
Propose your own or vote on the existing suggestions here.

...Or instruct a tutorial and cover your conference costs.

Did you know that we're awarding generous stipends to tutorial  
instructors this year?  So if you believe you could lead a tutorial,  
by all means submit your proposal ? soon! They're due April 15th.

Call for Papers Continues
Submitting a paper for to present at SciPy 2010 is easy, so remember  
to prepare one and have your friends and colleagues follow suit. Send  
us your abstract before April 11th and let us know whether you'd like  
to speak at the main conference or one of the specialized tracks.   
Details here.

Have you registered?

Booking your tickets early should save you money ? not to mention the  
early registration prices you will qualify for if you register before  
May 10th.

Best,

The SciPy 2010 Team
@SciPy2010 on Twitter

You are receiving this email because you have registered for the SciPy  
2010 conference in Austin, TX.

Unsubscribe amenity at enthought.com from this list | Forward to a friend  
| Update your profile
Our mailing address is:
Enthought, Inc.
515 Congress Ave.
Austin, TX 78701

Add us to your address book

Copyright (C) 2010 Enthought, Inc. All rights reserved.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100330/13f71646/attachment.html>

From charlesr.harris at gmail.com  Tue Mar 30 10:38:22 2010
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 30 Mar 2010 08:38:22 -0600
Subject: [SciPy-User] installation, Espoo-Helsinki
In-Reply-To: <8e7295c71003300452w6c59f86cgb6528f5be6851340@mail.gmail.com>
References: <8e7295c71003300452w6c59f86cgb6528f5be6851340@mail.gmail.com>
Message-ID: <e06186141003300738g16147dd6k2f04680316ed9e17@mail.gmail.com>

On Tue, Mar 30, 2010 at 5:52 AM, gintare statkute <g.statkute at gmail.com>wrote:

> Hello,
>
> I tried to install scientific-python one - two months before, but got in
> problems with ATLAS installation.
> I tried several times, thus now LAPACK maybe also has to be reinstalled.
> I really would like to try some simulations with scientific python, maybe
> somebody could help me in installation.
>
>
What OS are you using? What were the problems you saw?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100330/68a6531d/attachment.html>

From bapeters at terastat.com  Tue Mar 30 14:20:33 2010
From: bapeters at terastat.com (Bruce Peterson)
Date: Tue, 30 Mar 2010 11:20:33 -0700
Subject: [SciPy-User] How to estimate error in polynomial coefficients
In-Reply-To: <mailman.6278.1269901171.2398.scipy-user@scipy.org>
References: <mailman.6278.1269901171.2398.scipy-user@scipy.org>
Message-ID: <20100330182042.C430339CAEC@scipy.org>


Another way to estimate the variability of model parameters is to use 
resampling statistics such as the Jackknife or Bootstrap approach. 
Basically a Jackknife estimate of the parameter error is obtained by 
removing one point from your data set,fitting your model to the 
remaining data, record the estimated parameters, replacing the data 
point, remove another and refit the model and recording the new 
estimates--repeat until all data points are removed and replaced. The 
set of estimated parameter values can be to create the cumulative 
frequency distribution and thus the variability of the parameter estimates.
see http://en.wikipedia.org/wiki/Resampling_(statistics)



Bruce Peterson, Ph.D.
Terastat, Inc
Information Access Systems
Voice (425) 466 7344



From itsdho at ucla.edu  Tue Mar 30 15:10:11 2010
From: itsdho at ucla.edu (David Ho)
Date: Tue, 30 Mar 2010 12:10:11 -0700
Subject: [SciPy-User] How do I use vonmises.fit()?
Message-ID: <469ba0f41003301210x5bfb6bd7w364123db34f1f54a@mail.gmail.com>

Hi all!

I want to fit some data with a Von Mises distribution, and get the mean and
"kappa" parameters for that distribution.
I'm a little confused about scipy.stats.distributions.vonmises.fit().

Just as a test, I tried this:

>>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000) # the first
argument appears to be "kappa", and the second argument is the mean.
>>> scipy.stats.distributions.vonmises.fit(vm_rvs)
array([  1.17643696e-01,   3.38956854e-03,   1.27331662e-27])

I got an array of 3 values, none of which seem to be equal to the mean or
kappa of the distribution.
I looked around in some of the docstrings, but I couldn't find any clear
documentation of what these values are supposed to correspond to.

I would've expected vonmises.fit() to return something like:
array([  1.2,   2.3])

What am I doing wrong?
Thanks for your help,

--David Ho

PS: I also don't fully understand the "scale" and "loc" parameters for all
of the continuous random variables.
Does "loc" always correspond to the mean, and "scale" always correspond to
the square root of the variance, or something else?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100330/7ff30ae9/attachment.html>

From robert.kern at gmail.com  Tue Mar 30 15:21:35 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 30 Mar 2010 14:21:35 -0500
Subject: [SciPy-User] How do I use vonmises.fit()?
In-Reply-To: <469ba0f41003301210x5bfb6bd7w364123db34f1f54a@mail.gmail.com>
References: <469ba0f41003301210x5bfb6bd7w364123db34f1f54a@mail.gmail.com>
Message-ID: <3d375d731003301221w209d2245rf34dbfcdd08ab445@mail.gmail.com>

On Tue, Mar 30, 2010 at 14:10, David Ho <itsdho at ucla.edu> wrote:

> PS: I also don't fully understand the "scale" and "loc" parameters for all
> of the continuous random variables.
> Does "loc" always correspond to the mean, and "scale" always correspond to
> the square root of the variance, or something else?

No, they are location and scale parameters. Sometimes they correspond
with particular parameters in the canonical representation of the
distribution, but sometimes not. For some distributions (particularly
those usually presented with restricted domains like von Mises), we
add them in order to have a uniform implementation for all
distributions.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From bouloumag at gmail.com  Tue Mar 30 16:07:51 2010
From: bouloumag at gmail.com (Darcoux Christine)
Date: Tue, 30 Mar 2010 16:07:51 -0400
Subject: [SciPy-User] preconditioner for GMRES
In-Reply-To: <477ac73a1003291354n25cf2d2an8f481c40cedb7dae@mail.gmail.com>
References: <477ac73a1003291354n25cf2d2an8f481c40cedb7dae@mail.gmail.com>
Message-ID: <477ac73a1003301307h586751f0t1c079b7fd31f4034@mail.gmail.com>

I guess that replacing the QUICK advection and Crank-Nicolson by low-order
schemes (like central difference advection and Laasonen time integration
scheme) in the derivation of the preconditioner should be enough to improve
the krylov solver.

There are two approach to devise a preconditioner :
   1)  Find a matrix M that approximates A, and for which solving a system
is easier than solving with A.
   2)  Find a matrix M that approximates A^{-1}, so that only multiplication
by M is needed.
Which approach should be used with the krylov solvers provided by scipy ?
For example, if I want to use Jacobi preconditioner, should I use  M =
diag(A) or M = 1/diag(A) ?

For the moment, I am working with regular cartesian grid, so SSOR is fine
and could be used because the matrix is SPD. In the future, I would like to
extend my work to unstructured grid and I am not sure that the jacobian will
still be symetric. Which preconditioner could be used in this case in a
matrix-free manner ?

Thanks

Christine

2010/3/29 Darcoux Christine <bouloumag at gmail.com>

> I am developping a Jacobian-Free Newton-Krylov code to solve the Navier
> Stokes equations. In this code, the product of the Jacobian matrix with a
> given vector is represented by the matvec method of a "LinearOperator"object. The
> scipy.sparse.linalg.isolve.gmres method can takes an optional
> preconditioner as parameter which is either an object "LinearOperator" or a
> matrix. However, the scipy documentation does not explain how one can form
> this preconditioner in a (ideally) matrix-free manner given only the
> matvec method of the jacobian.
>
> Any suggestions on this would be very appreciated.
>
> Thanks,
>
> Christine
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100330/034c8666/attachment.html>

From josef.pktd at gmail.com  Tue Mar 30 16:34:56 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 30 Mar 2010 16:34:56 -0400
Subject: [SciPy-User] How do I use vonmises.fit()?
In-Reply-To: <469ba0f41003301210x5bfb6bd7w364123db34f1f54a@mail.gmail.com>
References: <469ba0f41003301210x5bfb6bd7w364123db34f1f54a@mail.gmail.com>
Message-ID: <1cd32cbb1003301334q6b990ff5yc31482d4e1e28a87@mail.gmail.com>

On Tue, Mar 30, 2010 at 3:10 PM, David Ho <itsdho at ucla.edu> wrote:
> Hi all!
>
> I want to fit some data with a Von Mises distribution, and get the mean and
> "kappa" parameters for that distribution.
> I'm a little confused about scipy.stats.distributions.vonmises.fit().
>
> Just as a test, I tried this:
>
>>>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000) # the first
>>>> argument appears to be "kappa", and the second argument is the mean.
>>>> scipy.stats.distributions.vonmises.fit(vm_rvs)
> array([? 1.17643696e-01,?? 3.38956854e-03,?? 1.27331662e-27])
>
> I got an array of 3 values, none of which seem to be equal to the mean or
> kappa of the distribution.
> I looked around in some of the docstrings, but I couldn't find any clear
> documentation of what these values are supposed to correspond to.
>
> I would've expected vonmises.fit() to return something like:
> array([? 1.2,?? 2.3])
>
> What am I doing wrong?
> Thanks for your help,

When I did I check of the fit method for all distributions, then
vonmises most of the time didn't produce any useful results,
especially estimated scale of almost zero.

The problem is that vonmises doesn't have a well defined pdf on the real line

>>> import matplotlib.pyplot as plt
>>> plt.plot(scipy.stats.vonmises.pdf(np.linspace(-10,10),1.2, loc=2.3))
>>> plt.show()

Since vonmises is intended for circular data, and I don't know much
about circular statistics (except for what Anne explained on the
mailing list), I never tried to get it to work like the other
distributions.

It might be possible to patch vonmises to work on the real line but I
never tried.

Josef

>
> --David Ho
>
> PS: I also don't fully understand the "scale" and "loc" parameters for all
> of the continuous random variables.
> Does "loc" always correspond to the mean, and "scale" always correspond to
> the square root of the variance, or something else?

loc=0, scale=1 is the standard distribution, the loc and scale
parameters transform the distribution of the transformation x~ =
(x-loc)/scale

If the standard distribution has mean or variance different from 0, 1
then the transformed distribution has mean and variance different from
loc, scale.
the stats method shows the implied variance scale:

>>> scipy.stats.vonmises.stats(1.2, loc=2.3, scale=2)
(array(2.2999999999999998), array(5.4900668161122423))
>>> scipy.stats.vonmises.stats(1.2, loc=0, scale=1)
(array(1.5832118030775403e-017), array(1.3725167040280606))


>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From josef.pktd at gmail.com  Tue Mar 30 16:51:07 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 30 Mar 2010 16:51:07 -0400
Subject: [SciPy-User] How do I use vonmises.fit()?
In-Reply-To: <1cd32cbb1003301334q6b990ff5yc31482d4e1e28a87@mail.gmail.com>
References: <469ba0f41003301210x5bfb6bd7w364123db34f1f54a@mail.gmail.com>
	<1cd32cbb1003301334q6b990ff5yc31482d4e1e28a87@mail.gmail.com>
Message-ID: <1cd32cbb1003301351q3a865878u1c4775c68a8547e3@mail.gmail.com>

On Tue, Mar 30, 2010 at 4:34 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Mar 30, 2010 at 3:10 PM, David Ho <itsdho at ucla.edu> wrote:
>> Hi all!
>>
>> I want to fit some data with a Von Mises distribution, and get the mean and
>> "kappa" parameters for that distribution.
>> I'm a little confused about scipy.stats.distributions.vonmises.fit().
>>
>> Just as a test, I tried this:
>>
>>>>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000) # the first
>>>>> argument appears to be "kappa", and the second argument is the mean.
>>>>> scipy.stats.distributions.vonmises.fit(vm_rvs)
>> array([? 1.17643696e-01,?? 3.38956854e-03,?? 1.27331662e-27])
>>
>> I got an array of 3 values, none of which seem to be equal to the mean or
>> kappa of the distribution.
>> I looked around in some of the docstrings, but I couldn't find any clear
>> documentation of what these values are supposed to correspond to.
>>
>> I would've expected vonmises.fit() to return something like:
>> array([? 1.2,?? 2.3])
>>
>> What am I doing wrong?
>> Thanks for your help,
>
> When I did I check of the fit method for all distributions, then
> vonmises most of the time didn't produce any useful results,
> especially estimated scale of almost zero.
>
> The problem is that vonmises doesn't have a well defined pdf on the real line
>
>>>> import matplotlib.pyplot as plt
>>>> plt.plot(scipy.stats.vonmises.pdf(np.linspace(-10,10),1.2, loc=2.3))
>>>> plt.show()
>
> Since vonmises is intended for circular data, and I don't know much
> about circular statistics (except for what Anne explained on the
> mailing list), I never tried to get it to work like the other
> distributions.
>
> It might be possible to patch vonmises to work on the real line but I
> never tried.

Here is a way that seems to work
* set bounds a,b
* use mean of random variable as starting value to start inside the
support of the distribution

>>> vm_rvs = scipy.stats.vonmises.rvs(1.2, 2.3, size=10000)
>>> stats.vonmises.a = -np.pi
>>> stats.vonmises.b = np.pi
>>> stats.vonmises.fit(vm_rvs,1,loc=vm_rvs.mean(),scale=1)
array([ 1.19767227,  2.30077064,  0.99916372])

Josef


>
> Josef
>
>>
>> --David Ho
>>
>> PS: I also don't fully understand the "scale" and "loc" parameters for all
>> of the continuous random variables.
>> Does "loc" always correspond to the mean, and "scale" always correspond to
>> the square root of the variance, or something else?
>
> loc=0, scale=1 is the standard distribution, the loc and scale
> parameters transform the distribution of the transformation x~ =
> (x-loc)/scale
>
> If the standard distribution has mean or variance different from 0, 1
> then the transformed distribution has mean and variance different from
> loc, scale.
> the stats method shows the implied variance scale:
>
>>>> scipy.stats.vonmises.stats(1.2, loc=2.3, scale=2)
> (array(2.2999999999999998), array(5.4900668161122423))
>>>> scipy.stats.vonmises.stats(1.2, loc=0, scale=1)
> (array(1.5832118030775403e-017), array(1.3725167040280606))
>
>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>


From d.boles at hotmail.com  Tue Mar 30 17:04:24 2010
From: d.boles at hotmail.com (Dan bole)
Date: Tue, 30 Mar 2010 14:04:24 -0700
Subject: [SciPy-User] Kurtosis/Skewness
Message-ID: <BAY126-W2324D47B86E9895227916CF21F0@phx.gbl>


Hi all,
I am trying to create a series of random variables selected from a distribution.  I would like this distribution to start as a normal distribution, but then be altered based on assumptions of skewness and kurtosis (so I am not calculating skewness/kurtosis from a dataset, but instead creating the probability density function from assumptions of skewness/kurtosis).  I can create a normal distribution and then pull random variables from this, and was wondering if it is possible to create a distribution based on assumptions of skewness and kurtosis?
Many thanks,Dan 		 	   		  
_________________________________________________________________
Hotmail: Trusted email with Microsoft?s powerful SPAM protection.
http://clk.atdmt.com/GBL/go/210850552/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100330/04d9afc8/attachment.html>

From josef.pktd at gmail.com  Tue Mar 30 17:19:43 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 30 Mar 2010 17:19:43 -0400
Subject: [SciPy-User] Kurtosis/Skewness
In-Reply-To: <BAY126-W2324D47B86E9895227916CF21F0@phx.gbl>
References: <BAY126-W2324D47B86E9895227916CF21F0@phx.gbl>
Message-ID: <1cd32cbb1003301419s466450e3mc064139d58cfad74@mail.gmail.com>

On Tue, Mar 30, 2010 at 5:04 PM, Dan bole <d.boles at hotmail.com> wrote:
> Hi all,
> I am trying to create a series of random variables selected from a
> distribution. ?I would like this distribution to start as a normal
> distribution, but then be altered based on assumptions of skewness and
> kurtosis (so I am not calculating skewness/kurtosis from a dataset, but
> instead creating the probability density function from assumptions of
> skewness/kurtosis). ?I can create a normal distribution and then pull random
> variables from this, and was wondering if it is possible to create a
> distribution based on assumptions of skewness and kurtosis?
> Many thanks,
> Dan

For creating random variables, I would look into pymc, I remember it
has a skew-normal, but I don't know if it also has a distribution for
changed kurtosis.

A warning: don't use scipy.stats.pdf_approx and friends for estimation
because they don't produce correct results.

I wrote a Gram-Charlier normal expansion function, but I don't
remember if it only does estimation or has all distribution methods.
If I did an rvs method, then it would be only generic and possibly
pretty slow.
If you are interested, then I can dig up my scripts.

When I googled this last year, Gram-Charlier expansion was the best
search term for this.

Josef

> ________________________________
> Hotmail: Trusted email with Microsoft?s powerful SPAM protection. Sign up
> now.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


From robert.kern at gmail.com  Tue Mar 30 17:21:24 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 30 Mar 2010 16:21:24 -0500
Subject: [SciPy-User] Kurtosis/Skewness
In-Reply-To: <BAY126-W2324D47B86E9895227916CF21F0@phx.gbl>
References: <BAY126-W2324D47B86E9895227916CF21F0@phx.gbl>
Message-ID: <3d375d731003301421r3f6ba6e7s880b3b0a4dbc4b14@mail.gmail.com>

On Tue, Mar 30, 2010 at 16:04, Dan bole <d.boles at hotmail.com> wrote:
> Hi all,
> I am trying to create a series of random variables selected from a
> distribution. ?I would like this distribution to start as a normal
> distribution, but then be altered based on assumptions of skewness and
> kurtosis (so I am not calculating skewness/kurtosis from a dataset, but
> instead creating the probability density function from assumptions of
> skewness/kurtosis). ?I can create a normal distribution and then pull random
> variables from this, and was wondering if it is possible to create a
> distribution based on assumptions of skewness and kurtosis?

There are an infinite number of distributions that will have the same
skewness and kurtosis. However, it is reasonable to search for the
maximum entropy distribution satisfying those constraints. The normal
distribution is the maximum entropy distribution for a fixed mean and
variance.

http://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution

The PDF will have the form:

  pdf(x) = c * exp(- lagrange * (x ** arange(1, 5)))

c is just the normalizing constant. You will have to find the lagrange
parameters that satisfy the mean, variance, skewness and kurtosis.
Sampling from this distribution will be tricky, though. You will have
to resort to general methods that are going to be pretty slow.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From vanforeest at gmail.com  Tue Mar 30 17:25:16 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Tue, 30 Mar 2010 23:25:16 +0200
Subject: [SciPy-User] implementing recursions
Message-ID: <fa510ff81003301425n4b53b4fcj82535f2acf1d289d@mail.gmail.com>

Hi,

Just out of curiosity, is there a smarter way to implement the
following recursion?

grid =delta = 0.001
end = 20
np.arange(0,end+delta,delta)
n = len(grid)

gamma = 3*grid
for j in range(n-1, -1, -1):
     gamma[j] += dot(gamma[j+1:],G[1:n-j])  # G is a precomputed np.array

I still dislike the remaining for loop, but I do not see an easy way
out. Of course, it is possible to write the above as Ax = b, and use
np.solve, but then I have to build the matrix A, which takes more time
as much more memory is claimed.
(I also realize that I can use Z-transforms, but this appears somewhat
of a hassle, to say the least.)

Thanks

Nicky


From vanforeest at gmail.com  Tue Mar 30 17:32:24 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Tue, 30 Mar 2010 23:32:24 +0200
Subject: [SciPy-User] Kurtosis/Skewness
In-Reply-To: <3d375d731003301421r3f6ba6e7s880b3b0a4dbc4b14@mail.gmail.com>
References: <BAY126-W2324D47B86E9895227916CF21F0@phx.gbl>
	<3d375d731003301421r3f6ba6e7s880b3b0a4dbc4b14@mail.gmail.com>
Message-ID: <fa510ff81003301432u108cbfc8ua3e196f947861910@mail.gmail.com>

> There are an infinite number of distributions that will have the same
> skewness and kurtosis. However, it is reasonable to search for the
> maximum entropy distribution satisfying those constraints. The normal
> distribution is the maximum entropy distribution for a fixed mean and
> variance.
>
> http://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution
>
> The PDF will have the form:
>
> ?pdf(x) = c * exp(- lagrange * (x ** arange(1, 5)))
>
> c is just the normalizing constant. You will have to find the lagrange
> parameters that satisfy the mean, variance, skewness and kurtosis.
> Sampling from this distribution will be tricky, though. You will have
> to resort to general methods that are going to be pretty slow.


This is of course a very good suggestion. However, mind that this
claim is only true if the support of your desired distribution is the
entire real axis. I recall that I once tried to find the maximum
entropy distribution with given mean and variance, but such that the
support was the positive reals (including 0), rather then the entire
real line. This was less easy then I initially thought, and it is
certainly not the normal distribution.

bye

Nicky


From robert.kern at gmail.com  Tue Mar 30 17:36:33 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 30 Mar 2010 16:36:33 -0500
Subject: [SciPy-User] Kurtosis/Skewness
In-Reply-To: <fa510ff81003301432u108cbfc8ua3e196f947861910@mail.gmail.com>
References: <BAY126-W2324D47B86E9895227916CF21F0@phx.gbl>
	<3d375d731003301421r3f6ba6e7s880b3b0a4dbc4b14@mail.gmail.com> 
	<fa510ff81003301432u108cbfc8ua3e196f947861910@mail.gmail.com>
Message-ID: <3d375d731003301436r33299a1btbc48aa106e1779aa@mail.gmail.com>

On Tue, Mar 30, 2010 at 16:32, nicky van foreest <vanforeest at gmail.com> wrote:
>> There are an infinite number of distributions that will have the same
>> skewness and kurtosis. However, it is reasonable to search for the
>> maximum entropy distribution satisfying those constraints. The normal
>> distribution is the maximum entropy distribution for a fixed mean and
>> variance.
>>
>> http://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution
>>
>> The PDF will have the form:
>>
>> ?pdf(x) = c * exp(- lagrange * (x ** arange(1, 5)))
>>
>> c is just the normalizing constant. You will have to find the lagrange
>> parameters that satisfy the mean, variance, skewness and kurtosis.
>> Sampling from this distribution will be tricky, though. You will have
>> to resort to general methods that are going to be pretty slow.
>
>
> This is of course a very good suggestion. However, mind that this
> claim is only true if the support of your desired distribution is the
> entire real axis. I recall that I once tried to find the maximum
> entropy distribution with given mean and variance, but such that the
> support was the positive reals (including 0), rather then the entire
> real line. This was less easy then I initially thought, and it is
> certainly not the normal distribution.

Right. Since he was starting with the normal distribution, I assume he
wanted something that went over the entire real axis.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


From josef.pktd at gmail.com  Tue Mar 30 17:40:41 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 30 Mar 2010 17:40:41 -0400
Subject: [SciPy-User] Kurtosis/Skewness
In-Reply-To: <fa510ff81003301432u108cbfc8ua3e196f947861910@mail.gmail.com>
References: <BAY126-W2324D47B86E9895227916CF21F0@phx.gbl>
	<3d375d731003301421r3f6ba6e7s880b3b0a4dbc4b14@mail.gmail.com>
	<fa510ff81003301432u108cbfc8ua3e196f947861910@mail.gmail.com>
Message-ID: <1cd32cbb1003301440j6bf34ca0ybf73b1249fc9b8e9@mail.gmail.com>

On Tue, Mar 30, 2010 at 5:32 PM, nicky van foreest <vanforeest at gmail.com> wrote:
>> There are an infinite number of distributions that will have the same
>> skewness and kurtosis. However, it is reasonable to search for the
>> maximum entropy distribution satisfying those constraints. The normal
>> distribution is the maximum entropy distribution for a fixed mean and
>> variance.
>>
>> http://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution
>>
>> The PDF will have the form:
>>
>> ?pdf(x) = c * exp(- lagrange * (x ** arange(1, 5)))
>>
>> c is just the normalizing constant. You will have to find the lagrange
>> parameters that satisfy the mean, variance, skewness and kurtosis.
>> Sampling from this distribution will be tricky, though. You will have
>> to resort to general methods that are going to be pretty slow.
>
>
> This is of course a very good suggestion. However, mind that this
> claim is only true if the support of your desired distribution is the
> entire real axis. I recall that I once tried to find the maximum
> entropy distribution with given mean and variance, but such that the
> support was the positive reals (including 0), rather then the entire
> real line. This was less easy then I initially thought, and it is
> certainly not the normal distribution.

If you want to stay close to the normal distribution, then the main
two general expansions that I have seen, are the Edgeworth expansion
(I think based on Taylor series expansion but I'm not sure) and the
Gram-Charlier expansion which is based on Hermite polynomials.

For skew distributions, e.g. skew-normal, skew-t there are several
version in the literature (Azalini?).

Josef

>
> bye
>
> Nicky
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From josef.pktd at gmail.com  Tue Mar 30 18:40:56 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 30 Mar 2010 18:40:56 -0400
Subject: [SciPy-User] implementing recursions
In-Reply-To: <fa510ff81003301425n4b53b4fcj82535f2acf1d289d@mail.gmail.com>
References: <fa510ff81003301425n4b53b4fcj82535f2acf1d289d@mail.gmail.com>
Message-ID: <1cd32cbb1003301540m7e992bf7x5939ca84423d2637@mail.gmail.com>

On Tue, Mar 30, 2010 at 5:25 PM, nicky van foreest <vanforeest at gmail.com> wrote:
> Hi,
>
> Just out of curiosity, is there a smarter way to implement the
> following recursion?
>
> grid =delta = 0.001
> end = 20
> np.arange(0,end+delta,delta)
> n = len(grid)
>
> gamma = 3*grid
> for j in range(n-1, -1, -1):
> ? ? gamma[j] += dot(gamma[j+1:],G[1:n-j]) ?# G is a precomputed np.array
>
> I still dislike the remaining for loop, but I do not see an easy way
> out. Of course, it is possible to write the above as Ax = b, and use
> np.solve, but then I have to build the matrix A, which takes more time
> as much more memory is claimed.
> (I also realize that I can use Z-transforms, but this appears somewhat
> of a hassle, to say the least.)

It looks like a case for lfilter to me, but G is full length so maybe
it works with padding gamma with zeros. if lfilter  works, then only
in the case when G is 1d.

(I have a hard time thinking in reverse indexed loops.)

Josef

>
> Thanks
>
> Nicky
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From axel.kowald at gmail.com  Wed Mar 31 06:16:50 2010
From: axel.kowald at gmail.com (Axel Kowald)
Date: Wed, 31 Mar 2010 12:16:50 +0200
Subject: [SciPy-User] No Wilcoxon test in scipy ?
Message-ID: <4BB32112.7080708@gmail.com>

Hello everybody,

I'm using scipy 0.7.1 and try to find the Wilcoxon test, i.e. a 
non-parametric test for paired observations.
Am I right, that there is no Wilcoxon test in SciPy ?!
In the function summary of 
http://www.scipy.org/doc/api_docs/SciPy.stats.stats.html I could not 
find anything with Wilcoxon.

I'm only a bit confused, because at the beginning of this website the 
Wilcoxon test is listet:

INFERENTIAL STATS:  ttest_1samp
                    ttest_ind
                    ttest_rel
                    chisquare
                    ks_2samp
                    mannwhitneyu
                    ranksums
                    wilcoxon   <===============
                    kruskal
                    friedmanchisquare


So, is the Wilcoxon test implemented in SciPy or not ?

Many thanks,

 Axel





From m.abdollahi at gmail.com  Wed Mar 31 06:28:20 2010
From: m.abdollahi at gmail.com (Mohammad Abdollahi)
Date: Wed, 31 Mar 2010 12:28:20 +0200
Subject: [SciPy-User] MATLAB accumarray equivalent ?
In-Reply-To: <4BACD205.607@american.edu>
References: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>
	<4BACD205.607@american.edu>
Message-ID: <j2pf72d1a261003310328g7422bdc7kda82bcd82460df9d@mail.gmail.com>

thank you for your replys, but both of these methods are for the case when
the subs argument is a vector. but apparently they can not be used when subs
is a matrix. i.e. the output of the function is meant to be a matrix too.
right ?

On Fri, Mar 26, 2010 at 4:25 PM, Alan G Isaac <aisaac at american.edu> wrote:

> On 3/26/2010 10:39 AM, Mohammad Abdollahi wrote:
> > Does anyone know what is the equivalent function of MATLAB's
> > "accumarray()" in scipy/numpy,  with the exact same functionality ?
>
> That certainly gets the basic accumarray functionality:
>
>         >>> vals = 101+np.arange(5)
>         >>> subs = np.array([1,2,4,2,4])-1
>         >>> np.bincount(subs,vals)
>         array([ 101.,  206.,    0.,  208.])
>
> Here's another way that maybe easier
> to adapt to whatever the OP needs:
>
>         >>> from collections import defaultdict
>         >>> d = defaultdict(float)
>         >>> for i, cat in enumerate(subs):
>         ...   d[cat] += vals[i]
>         ...
>         >>> d
>         defaultdict(<type 'float'>, {0: 101.0, 1: 206.0, 3: 208.0})
>
> That should actually be pretty fast,
> though not as fast as bincount of course.
>
> What's the use case?
>
> hth,
> Alan Isaac
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100331/f2c809f8/attachment.html>

From harald.schilly at gmail.com  Wed Mar 31 06:33:02 2010
From: harald.schilly at gmail.com (Harald Schilly)
Date: Wed, 31 Mar 2010 12:33:02 +0200
Subject: [SciPy-User] MATLAB accumarray equivalent ?
In-Reply-To: <j2pf72d1a261003310328g7422bdc7kda82bcd82460df9d@mail.gmail.com>
References: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com> 
	<4BACD205.607@american.edu>
	<j2pf72d1a261003310328g7422bdc7kda82bcd82460df9d@mail.gmail.com>
Message-ID: <v2l20548feb1003310333v6092b251zf7795e113b90fcf9@mail.gmail.com>

On Wed, Mar 31, 2010 at 12:28, Mohammad Abdollahi <m.abdollahi at gmail.com> wrote:
> but apparently they can not be used when subs
> is a matrix. i.e. the output of the function is meant to be a matrix too.
> right ?

Can you please explain what you wanna do? Maybe there is another
function that does what you want or it is done in another way. Numpy
is not a 1:1 matlab clone!

greetings harald


From josef.pktd at gmail.com  Wed Mar 31 07:16:24 2010
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 31 Mar 2010 07:16:24 -0400
Subject: [SciPy-User] No Wilcoxon test in scipy ?
In-Reply-To: <4BB32112.7080708@gmail.com>
References: <4BB32112.7080708@gmail.com>
Message-ID: <n2y1cd32cbb1003310416k486e6ea4kbbdf698748e11fc5@mail.gmail.com>

On Wed, Mar 31, 2010 at 6:16 AM, Axel Kowald <axel.kowald at gmail.com> wrote:
> Hello everybody,
>
> I'm using scipy 0.7.1 and try to find the Wilcoxon test, i.e. a
> non-parametric test for paired observations.
> Am I right, that there is no Wilcoxon test in SciPy ?!
> In the function summary of
> http://www.scipy.org/doc/api_docs/SciPy.stats.stats.html I could not
> find anything with Wilcoxon.
>
> I'm only a bit confused, because at the beginning of this website the
> Wilcoxon test is listet:
>
> INFERENTIAL STATS: ?ttest_1samp
> ? ? ? ? ? ? ? ? ? ?ttest_ind
> ? ? ? ? ? ? ? ? ? ?ttest_rel
> ? ? ? ? ? ? ? ? ? ?chisquare
> ? ? ? ? ? ? ? ? ? ?ks_2samp
> ? ? ? ? ? ? ? ? ? ?mannwhitneyu
> ? ? ? ? ? ? ? ? ? ?ranksums
> ? ? ? ? ? ? ? ? ? ?wilcoxon ? <===============
> ? ? ? ? ? ? ? ? ? ?kruskal
> ? ? ? ? ? ? ? ? ? ?friedmanchisquare
>
>
> So, is the Wilcoxon test implemented in SciPy or not ?
>
> Many thanks,
>
> ?Axel

the new documentation is here:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html?highlight=wilcoxon#scipy.stats.wilcoxon

the api_docs are outdated

>>> help(scipy.stats.wilcoxon)
Help on function wilcoxon in module scipy.stats.morestats:

wilcoxon(x, y=None)
    Calculate the Wilcoxon signed-rank test

    The Wilcoxon signed-rank test tests the null hypothesis that two
    related samples come from the same distribution. It is a a
    non-parametric version of the paired T-test.

a good comparison of the non-parametric tests is in this ticket
http://projects.scipy.org/scipy/ticket/901

Josef

>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From m.abdollahi at gmail.com  Wed Mar 31 07:20:58 2010
From: m.abdollahi at gmail.com (Mohammad Abdollahi)
Date: Wed, 31 Mar 2010 13:20:58 +0200
Subject: [SciPy-User] MATLAB accumarray equivalent ?
In-Reply-To: <v2l20548feb1003310333v6092b251zf7795e113b90fcf9@mail.gmail.com>
References: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>
	<4BACD205.607@american.edu>
	<j2pf72d1a261003310328g7422bdc7kda82bcd82460df9d@mail.gmail.com>
	<v2l20548feb1003310333v6092b251zf7795e113b90fcf9@mail.gmail.com>
Message-ID: <k2vf72d1a261003310420g933e54d2l8419c977831e7be6@mail.gmail.com>

Dera Herlad

Thank you for your reply. so what i'm trying to do is to create a
re-assigned spectrogram. which means that i have a 2-d matrix representing
my weighted spectrogram that i want to operate on this matrix in a way that
the value of each element in matrix will be reassigned based on the value of
the neighbouring elemnts of it. (i.e the sum of the values of some
neighbours). for that purpose i was using accumarray in matlab which is very
efficient that the trivial way of using for loops that comes to mind first.
and the way i used the accumarray in matlab was sth like this : (very simple
example)

>> subs = [1:6;6:-1:1]

subs =

     1     2     3     4     5     6
     6     5     4     3     2     1

>> val = 10:15

val =

    10    11    12    13    14    15

>> accumarray(subs',val)

ans =

     0     0     0     0     0    10
     0     0     0     0    11     0
     0     0     0    12     0     0
     0     0    13     0     0     0
     0    14     0     0     0     0
    15     0     0     0     0     0


regards
Mohammad

On Wed, Mar 31, 2010 at 12:33 PM, Harald Schilly
<harald.schilly at gmail.com>wrote:

> On Wed, Mar 31, 2010 at 12:28, Mohammad Abdollahi <m.abdollahi at gmail.com>
> wrote:
> > but apparently they can not be used when subs
> > is a matrix. i.e. the output of the function is meant to be a matrix too.
> > right ?
>
> Can you please explain what you wanna do? Maybe there is another
> function that does what you want or it is done in another way. Numpy
> is not a 1:1 matlab clone!
>
> greetings harald
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100331/673a6c17/attachment.html>

From daniel.wheeler2 at gmail.com  Wed Mar 31 09:51:46 2010
From: daniel.wheeler2 at gmail.com (Daniel Wheeler)
Date: Wed, 31 Mar 2010 09:51:46 -0400
Subject: [SciPy-User] preconditioner for GMRES
In-Reply-To: <477ac73a1003291559v64cea8d3i9eb133f81da832e3@mail.gmail.com>
References: <477ac73a1003291354n25cf2d2an8f481c40cedb7dae@mail.gmail.com>
	<c5ae9cf01003291519i23d9e082ved1a2e1471ecdf9b@mail.gmail.com>
	<477ac73a1003291559v64cea8d3i9eb133f81da832e3@mail.gmail.com>
Message-ID: <q2t80b160a1003310651mf0a2f106r411503864dea27fa@mail.gmail.com>

You might want to look at fipy <http://www.ctcms.nist.gov/fipy>, it
has the Van Leer flux limiting schemes for convection terms as well as
other options. Also, it uses either pysparse or trilinos, which both
have a number or preconditioning options.

On Mon, Mar 29, 2010 at 6:59 PM, Darcoux Christine <bouloumag at gmail.com> wrote:
> Thnak you. I will look at these links.
>
> I am using the finite volume method, with QUICK advection (maybe I will try
> flux limiters in the future if there are too much oscillations).
>
> 2010/3/29 Joshua Stults <joshua.stults at gmail.com>
>>
>> On Mon, Mar 29, 2010 at 4:54 PM, Darcoux Christine <bouloumag at gmail.com>
>> wrote:
>> > I am developping a Jacobian-Free Newton-Krylov code to solve the Navier
>> > Stokes equations. In this code, the product of the Jacobian matrix with
>> > a
>> > given vector is represented by the matvec method of a "LinearOperator"
>> > object. The scipy.sparse.linalg.isolve.gmres method can takes an
>> > optional
>> > preconditioner as parameter which is either an object "LinearOperator"
>> > or a
>> > matrix. However, the scipy documentation does not explain how one can
>> > form
>> > this preconditioner in a (ideally) matrix-free manner given only the
>> > matvec
>> > method of the jacobian.
>> >
>> > Any suggestions on this would be very appreciated.
>> >
>>
>> A pretty straight-forward way to precondition these types of problems
>> in a matrix-free way is with symmetric successive over relaxation (the
>> symmetric part is important, you need to do a 'forward' and a
>> 'backward' sweep or things won't converge):
>> http://www.netlib.org/linalg/old_html_templates/subsection2.6.2.4.html
>>
>> You'll have to derive the update formulas yourself based on the type
>> of discretization you are using; ?if you are using a transform based
>> method, then you can still precondition based on a low-order scheme.
>> IIRC Boyd has a pretty decent discussion of this in his spectral
>> methods book.
>>
>> Maybe you could construct one of the approximate factorization
>> preconditioners using only the matvec of your Jacobian, but I've never
>> gone with this approach, maybe someone else could chime in with
>> recommendations along those lines?
>> http://www.netlib.org/linalg/old_html_templates/chapter2.7.html
>>
>> What sort of spatial discretization are you using?
>>
>> > Thanks,
>> >
>> > Christine
>> >
>>
>> HtH,
>> --
>> Joshua Stults
>> Website: http://j-stults.blogspot.com
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>



-- 
Daniel Wheeler


From tsyu80 at gmail.com  Wed Mar 31 10:27:10 2010
From: tsyu80 at gmail.com (Tony S Yu)
Date: Wed, 31 Mar 2010 10:27:10 -0400
Subject: [SciPy-User] MATLAB accumarray equivalent ?
In-Reply-To: <k2vf72d1a261003310420g933e54d2l8419c977831e7be6@mail.gmail.com>
References: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>
	<4BACD205.607@american.edu>
	<j2pf72d1a261003310328g7422bdc7kda82bcd82460df9d@mail.gmail.com>
	<v2l20548feb1003310333v6092b251zf7795e113b90fcf9@mail.gmail.com>
	<k2vf72d1a261003310420g933e54d2l8419c977831e7be6@mail.gmail.com>
Message-ID: <815C15E7-00C2-458B-AED5-30D0B67C00E1@gmail.com>


On Mar 31, 2010, at 7:20 AM, Mohammad Abdollahi wrote:

> Dera Herlad 
> 
> Thank you for your reply. so what i'm trying to do is to create a re-assigned spectrogram. which means that i have a 2-d matrix representing my weighted spectrogram that i want to operate on this matrix in a way that the value of each element in matrix will be reassigned based on the value of the neighbouring elemnts of it. (i.e the sum of the values of some neighbours). for that purpose i was using accumarray in matlab which is very efficient that the trivial way of using for loops that comes to mind first. and the way i used the accumarray in matlab was sth like this : (very simple example) 
> 
> >> subs = [1:6;6:-1:1]
> 
> subs =
> 
>      1     2     3     4     5     6
>      6     5     4     3     2     1
> 
> >> val = 10:15
> 
> val =
> 
>     10    11    12    13    14    15
> 
> >> accumarray(subs',val)
> 
> ans =
> 
>      0     0     0     0     0    10
>      0     0     0     0    11     0
>      0     0     0    12     0     0
>      0     0    13     0     0     0
>      0    14     0     0     0     0
>     15     0     0     0     0     0

Hey Mohammad,

You can reproduce the above example with the following numpy code:

>>> import numpy as np
>>> 
>>> def accumarray(idx, val):
>>>     num_rows, num_cols = idx.max(axis=1) + 1
>>>     new_array = np.zeros((num_rows, num_cols))
>>>     new_array[tuple(idx)] = val
>>>     return new_array
>>> 
>>> idx = np.array([np.arange(6), np.arange(6)[::-1]])
>>> val = np.arange(10,16)
>>> print accumarray(idx, val)

Your actual use of accumarray may not be satisfied by the above, but it shouldn't be too difficult to modify it to suit your needs.

Cheers,
-Tony



From axel.kowald at gmail.com  Wed Mar 31 10:48:12 2010
From: axel.kowald at gmail.com (Axel Kowald)
Date: Wed, 31 Mar 2010 16:48:12 +0200
Subject: [SciPy-User] New official SciPy release ?
In-Reply-To: <n2y1cd32cbb1003310416k486e6ea4kbbdf698748e11fc5@mail.gmail.com>
References: <4BB32112.7080708@gmail.com>
	<n2y1cd32cbb1003310416k486e6ea4kbbdf698748e11fc5@mail.gmail.com>
Message-ID: <4BB360AC.80809@gmail.com>

Hello,

I wonder when there will be a new official SciPy release ?
Documentation is already available for 0.8 but the current release is 0.7.1
I'm not really able to build SciPy from the sources for Windows and so I 
wanted to hear if there are any planned release dates ?

Many thanks,

axel



From warren.weckesser at enthought.com  Wed Mar 31 12:04:28 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Wed, 31 Mar 2010 11:04:28 -0500
Subject: [SciPy-User] MATLAB accumarray equivalent ?
In-Reply-To: <815C15E7-00C2-458B-AED5-30D0B67C00E1@gmail.com>
References: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>	<4BACD205.607@american.edu>	<j2pf72d1a261003310328g7422bdc7kda82bcd82460df9d@mail.gmail.com>	<v2l20548feb1003310333v6092b251zf7795e113b90fcf9@mail.gmail.com>	<k2vf72d1a261003310420g933e54d2l8419c977831e7be6@mail.gmail.com>
	<815C15E7-00C2-458B-AED5-30D0B67C00E1@gmail.com>
Message-ID: <4BB3728C.70403@enthought.com>

I've added an accumarray-like function to the Cookbook:

    http://www.scipy.org/Cookbook/AccumarrayLike


Warren


Tony S Yu wrote:
> On Mar 31, 2010, at 7:20 AM, Mohammad Abdollahi wrote:
>
>   
>> Dera Herlad 
>>
>> Thank you for your reply. so what i'm trying to do is to create a re-assigned spectrogram. which means that i have a 2-d matrix representing my weighted spectrogram that i want to operate on this matrix in a way that the value of each element in matrix will be reassigned based on the value of the neighbouring elemnts of it. (i.e the sum of the values of some neighbours). for that purpose i was using accumarray in matlab which is very efficient that the trivial way of using for loops that comes to mind first. and the way i used the accumarray in matlab was sth like this : (very simple example) 
>>
>>     
>>>> subs = [1:6;6:-1:1]
>>>>         
>> subs =
>>
>>      1     2     3     4     5     6
>>      6     5     4     3     2     1
>>
>>     
>>>> val = 10:15
>>>>         
>> val =
>>
>>     10    11    12    13    14    15
>>
>>     
>>>> accumarray(subs',val)
>>>>         
>> ans =
>>
>>      0     0     0     0     0    10
>>      0     0     0     0    11     0
>>      0     0     0    12     0     0
>>      0     0    13     0     0     0
>>      0    14     0     0     0     0
>>     15     0     0     0     0     0
>>     
>
> Hey Mohammad,
>
> You can reproduce the above example with the following numpy code:
>
>   
>>>> import numpy as np
>>>>
>>>> def accumarray(idx, val):
>>>>     num_rows, num_cols = idx.max(axis=1) + 1
>>>>     new_array = np.zeros((num_rows, num_cols))
>>>>     new_array[tuple(idx)] = val
>>>>     return new_array
>>>>
>>>> idx = np.array([np.arange(6), np.arange(6)[::-1]])
>>>> val = np.arange(10,16)
>>>> print accumarray(idx, val)
>>>>         
>
> Your actual use of accumarray may not be satisfied by the above, but it shouldn't be too difficult to modify it to suit your needs.
>
> Cheers,
> -Tony
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   



From vanforeest at gmail.com  Wed Mar 31 15:03:05 2010
From: vanforeest at gmail.com (nicky van foreest)
Date: Wed, 31 Mar 2010 21:03:05 +0200
Subject: [SciPy-User] implementing recursions
In-Reply-To: <1cd32cbb1003301540m7e992bf7x5939ca84423d2637@mail.gmail.com>
References: <fa510ff81003301425n4b53b4fcj82535f2acf1d289d@mail.gmail.com>
	<1cd32cbb1003301540m7e992bf7x5939ca84423d2637@mail.gmail.com>
Message-ID: <h2vfa510ff81003311203m6831491fxe747d0eb36c4d38b@mail.gmail.com>

Hi Josef,

I'll try it. Thanks for the hint.

On 31 March 2010 00:40,  <josef.pktd at gmail.com> wrote:
> On Tue, Mar 30, 2010 at 5:25 PM, nicky van foreest <vanforeest at gmail.com> wrote:
>> Hi,
>>
>> Just out of curiosity, is there a smarter way to implement the
>> following recursion?
>>
>> grid =delta = 0.001
>> end = 20
>> np.arange(0,end+delta,delta)
>> n = len(grid)
>>
>> gamma = 3*grid
>> for j in range(n-1, -1, -1):
>> ? ? gamma[j] += dot(gamma[j+1:],G[1:n-j]) ?# G is a precomputed np.array
>>
>> I still dislike the remaining for loop, but I do not see an easy way
>> out. Of course, it is possible to write the above as Ax = b, and use
>> np.solve, but then I have to build the matrix A, which takes more time
>> as much more memory is claimed.
>> (I also realize that I can use Z-transforms, but this appears somewhat
>> of a hassle, to say the least.)
>
> It looks like a case for lfilter to me, but G is full length so maybe
> it works with padding gamma with zeros. if lfilter ?works, then only
> in the case when G is 1d.
>
> (I have a hard time thinking in reverse indexed loops.)
>
> Josef
>
>>
>> Thanks
>>
>> Nicky
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From e.antero.tammi at gmail.com  Wed Mar 31 17:12:21 2010
From: e.antero.tammi at gmail.com (eat)
Date: Wed, 31 Mar 2010 21:12:21 +0000 (UTC)
Subject: [SciPy-User] MATLAB accumarray equivalent ?
References: <f72d1a261003260739s30afa12axb1fb2ffc3fe5e2ba@mail.gmail.com>	<4BACD205.607@american.edu>	<j2pf72d1a261003310328g7422bdc7kda82bcd82460df9d@mail.gmail.com>	<v2l20548feb1003310333v6092b251zf7795e113b90fcf9@mail.gmail.com>	<k2vf72d1a261003310420g933e54d2l8419c977831e7be6@mail.gmail.com>
	<815C15E7-00C2-458B-AED5-30D0B67C00E1@gmail.com>
	<4BB3728C.70403@enthought.com>
Message-ID: <loom.20100331T231022-310@post.gmane.org>

Hi all,

Warren Weckesser <warren.weckesser <at> enthought.com> writes:

> 
> I've added an accumarray-like function to the Cookbook:
> 
>     http://www.scipy.org/Cookbook/AccumarrayLike
> 
> Warren
Excellent accumulator!
Would it be possible to add support for sparse arrays as well?

Along the lines:
def accum(..., dtype=None):
=> def accum(..., dtype=None, sparse= False):

# Create the output array.
out= np.empty(size, dtype=dtype)
...
=> # Create the output array.
if sparse:
    out= sparse(size, dtype=dtype) # just an idea, caller may provide more 
details
    ...
else:
    out= np.empty(size, dtype=dtype)
    ...


Regards,
eat