From oliverh at promanthan.com  Sat Dec  1 15:15:55 2012
From: oliverh at promanthan.com (Oliver M. Haynold)
Date: Sat, 1 Dec 2012 14:15:55 -0600
Subject: [SciPy-User]  skew-normal distribution and Owen's T function
Message-ID: <B7B49361-2435-44BD-A18F-2CFDC02F3C6C@promanthan.com>

I've coded up the skew-normal probability distribution, following closely the logic from the sn package in CRAN, for SciPy. I've put a very early version (skew-t isn't implemented yet, which is the main purpose of the package) on

	http://promanthan.com/randomstuff/skewt-0.0.1.tgz 

There are also unit tests that compare the results of my package to those of the sn package in R. Skew-t should be ready later this month. As always, any feedback and bug reports are highly appreciated.



From josef.pktd at gmail.com  Sat Dec  1 16:53:10 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sat, 1 Dec 2012 16:53:10 -0500
Subject: [SciPy-User] skew-normal distribution and Owen's T function
In-Reply-To: <B7B49361-2435-44BD-A18F-2CFDC02F3C6C@promanthan.com>
References: <B7B49361-2435-44BD-A18F-2CFDC02F3C6C@promanthan.com>
Message-ID: <CAMMTP+Avj6EXYOrU9dEM=9_8yJnTB5gXR7f2TgY+nJ+sPM39Rg@mail.gmail.com>

On Sat, Dec 1, 2012 at 3:15 PM, Oliver M. Haynold
<oliverh at promanthan.com> wrote:
> I've coded up the skew-normal probability distribution, following closely the logic from the sn package in CRAN, for SciPy. I've put a very early version (skew-t isn't implemented yet, which is the main purpose of the package) on
>
>         http://promanthan.com/randomstuff/skewt-0.0.1.tgz
>
> There are also unit tests that compare the results of my package to those of the sn package in R. Skew-t should be ready later this month. As always, any feedback and bug reports are highly appreciated.

Very good, thank you

I was looking for this. I just bumped recently again into Owen's T for
skewed distributions.

After a first look:
I would outsource _tOwen into a standalone function, since it might
also be useful in other places. scipy.special might also be a good
location for this function, if the implementation is not specific to
the current usage.

Josef





>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From takowl at gmail.com  Sun Dec  2 17:12:03 2012
From: takowl at gmail.com (Thomas Kluyver)
Date: Sun, 2 Dec 2012 22:12:03 +0000
Subject: [SciPy-User] Fwd: New Scipy website
In-Reply-To: <CAOvn4qg3RaVA3khKJy0YOANhK+uNrYL_4qwLxKKspo1TgRH8rw@mail.gmail.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
	<CABL7CQh9aYkTh6ZCsm_z_i17AuWj=vaCEoVpz0Ufs7CesSA9dg@mail.gmail.com>
	<CAOvn4qg3RaVA3khKJy0YOANhK+uNrYL_4qwLxKKspo1TgRH8rw@mail.gmail.com>
Message-ID: <CAOvn4qhk9KOA7Jjp+bSEU5Tn1uFutqZoqhkXFWd2jUcP0_muLg@mail.gmail.com>

On 27 November 2012 20:11, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> You're right that we can't have broken links, so this should be done.
> Before deciding to move things around, it would be good to have a complete
> overview of all pages that are at scipy.org/xxx or xxx.scipy.org. Then
> for each one you can decide if it has to be included in the Sphinx based
> site, or put somewhere else with a redirect. Cleaning up some content at
> the same time would be even better. The overview of all pages will also
> allow you to write a test script to check no links break.
>
> Moving things to Sphinx is a task that parallelizes nicely and can
> gradually be done after flipping the switch on the new site, but I think we
> do need the complete overview beforehand.
>

The complete (long) list of pages on the current site is at
http://www.scipy.org/TitleIndex

I've pulled out the pages I think should have redirects. This list can be
collaboratively edited if I've missed things:
http://piratepad.net/tnY97djjTC

Thanks,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121202/421fb21c/attachment.html>

From zhushazang at gmail.com  Sun Dec  2 19:18:26 2012
From: zhushazang at gmail.com (Rodolfo Timoteo da Silva)
Date: Sun, 02 Dec 2012 22:18:26 -0200
Subject: [SciPy-User] New Scipy website
In-Reply-To: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
Message-ID: <50BBEFEB.3000500@gmail.com>

Hi, just to report how to install into Gentoo

/emerge scipy/

Best regards

Rodolfo

Em 26-11-2012 12:14, Thomas Kluyver escreveu:
> I'd like to discuss how we can move the new SciPy website (currently
> at http://scipy.github.com/ ) to be the homepage of scipy.org
> <http://scipy.org>.
>
> The new site is still something of a work in progress, but it's been
> there for years, and we can't continue to maintain two sites
> indefinitely. We've already been bitten by people referring to out of
> date FAQs, and I think the aesthetics of the new site are a marked
> improvement. So, first, what does the new site need before it can go
> live?/
>
> /Two things I'm already aware of:
>
> - Installation instructions for other package managers (so far we
> cover apt and macports). Please can users of other Linux distros send
> me the corresponding installation commands?
> http://scipy.github.com/install.html
>
> - The 'Getting started' page should be rewritten to describe the Scipy
> stack, and from a more newcomer-ish perspective.
>
> Of course, there's a lot of useful information on the existing site -
> such as the Cookbook - that it's impractical to move to the new site.
> So, I suggest that we move the current MoinMoin-based site to
> wiki.scipy.org <http://wiki.scipy.org>, and provide redirects so we
> don't break existing links. Unfortunately, Github pages can't do
> proper HTTP-based redirects. So we can either use hackish HTML/JS
> redirects, or host the new site on a server where we can set up proper
> redirects.
>
> Thanks,
> Thomas
>
>
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121202/1ceeacf1/attachment.html>

From takowl at gmail.com  Sun Dec  2 19:38:06 2012
From: takowl at gmail.com (Thomas Kluyver)
Date: Mon, 3 Dec 2012 00:38:06 +0000
Subject: [SciPy-User] New Scipy website
In-Reply-To: <50BBEFEB.3000500@gmail.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
	<50BBEFEB.3000500@gmail.com>
Message-ID: <CAOvn4qit8wKOUzV96SSOLzHh_Q0xSq74RQEBw78ZvY_ERwv3dA@mail.gmail.com>

On 3 December 2012 00:18, Rodolfo Timoteo da Silva <zhushazang at gmail.com>wrote:

> Hi, just to report how to install into Gentoo
>
> *emerge scipy*
>

Thanks, Rodolfo,

I'd like to provide instructions for how to install the entire Scipy Stack
- the set of core packages we've chosen. That's numpy, scipy, matplotlib,
ipython (including the notebook), sympy, pandas and nose. Can the package
names be chained together in a single emerge command? And what versions
will the user get by default?

You can see the Scipy Stack specification here:
http://scipy.github.com/stackspec.html

Best wishes,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121203/419f8b44/attachment.html>

From oliverh at promanthan.com  Sat Dec  1 15:15:55 2012
From: oliverh at promanthan.com (Oliver M. Haynold)
Date: Sat, 1 Dec 2012 14:15:55 -0600
Subject: [SciPy-User]  skew-normal distribution and Owen's T function
Message-ID: <B7B49361-2435-44BD-A18F-2CFDC02F3C6C@promanthan.com>

I've coded up the skew-normal probability distribution, following closely the logic from the sn package in CRAN, for SciPy. I've put a very early version (skew-t isn't implemented yet, which is the main purpose of the package) on

	http://promanthan.com/randomstuff/skewt-0.0.1.tgz 

There are also unit tests that compare the results of my package to those of the sn package in R. Skew-t should be ready later this month. As always, any feedback and bug reports are highly appreciated.



From takowl at gmail.com  Mon Dec  3 13:02:40 2012
From: takowl at gmail.com (Thomas Kluyver)
Date: Mon, 3 Dec 2012 18:02:40 +0000
Subject: [SciPy-User] New Scipy website
In-Reply-To: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
Message-ID: <CAOvn4qiMKtU1vgUJsQWXxo6z7n66+5k3snMSbcj78f47hELvfg@mail.gmail.com>

>From the lack of offers for a new server to host on, I assume that we're
going to keep using Github pages. So I'll put something together to do a
"poor man's redirect" on the pages I listed:

http://piratepad.net/tnY97djjTC

Thanks,
Thomas


On 26 November 2012 14:14, Thomas Kluyver <takowl at gmail.com> wrote:

> I'd like to discuss how we can move the new SciPy website (currently at
> http://scipy.github.com/ ) to be the homepage of scipy.org.
>
> The new site is still something of a work in progress, but it's been there
> for years, and we can't continue to maintain two sites indefinitely. We've
> already been bitten by people referring to out of date FAQs, and I think
> the aesthetics of the new site are a marked improvement. So, first, what
> does the new site need before it can go live?*
>
> *Two things I'm already aware of:
>
> - Installation instructions for other package managers (so far we cover
> apt and macports). Please can users of other Linux distros send me the
> corresponding installation commands?
> http://scipy.github.com/install.html
>
> - The 'Getting started' page should be rewritten to describe the Scipy
> stack, and from a more newcomer-ish perspective.
>
> Of course, there's a lot of useful information on the existing site - such
> as the Cookbook - that it's impractical to move to the new site. So, I
> suggest that we move the current MoinMoin-based site to wiki.scipy.org,
> and provide redirects so we don't break existing links. Unfortunately,
> Github pages can't do proper HTTP-based redirects. So we can either use
> hackish HTML/JS redirects, or host the new site on a server where we can
> set up proper redirects.
>
> Thanks,
> Thomas
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121203/c7e69671/attachment.html>

From pdeepak at instem.res.in  Sun Dec  2 23:14:07 2012
From: pdeepak at instem.res.in (Poduval Deepak Balakrishnan)
Date: Mon, 3 Dec 2012 09:44:07 +0530
Subject: [SciPy-User] Help Scipy Installation
Message-ID: <7bab8c8a4df595dbe43ecef657178089.squirrel@webmail.instem.res.in>

Hi,

I am deepak. I installed numpy but when i try and install scipy I get the
following error. Please help

Traceback (most recent call last):
  File "setup.py", line 208, in <module>
    setup_package()
  File "setup.py", line 145, in setup_package
    from numpy.distutils.core import setup
  File
"/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/__init__.py",
line 137, in <module>
    import add_newdocs
  File
"/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/add_newdocs.py",
line 9, in <module>
    from numpy.lib import add_newdoc
  File
"/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/lib/__init__.py",
line 4, in <module>
    from type_check import *
  File
"/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/lib/type_check.py",
line 8, in <module>
    import numpy.core.numeric as _nx
  File
"/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/core/__init__.py",
line 5, in <module>
    import multiarray
ImportError: libpython2.7.so.1.0: cannot open shared object file: No such
file or directory

Thank you in advance.

Best regards
Poduval Deepak Balakrishnan,
Junior Research Fellow,
Das's Lab,
inStem, NCBS,
GKVK Campus,
Bellary Road,
Bangalore
Karnataka - 560065






From aanderso at med.wayne.edu  Mon Dec  3 11:29:35 2012
From: aanderso at med.wayne.edu (Anderson, Amy)
Date: Mon, 3 Dec 2012 16:29:35 +0000
Subject: [SciPy-User] running script error (eigen symmetric)
Message-ID: <13BFE00357AE78459CE7894943A31CA91BD932BE@MED-CORE07A.med.wayne.edu>

Dear Scipy users,

I am a new python and scipy user and I am looking for some help to troubleshoot an error I have been getting when trying to run a script in python.  The script I am trying to run is called pyCluster ROI and it needs python, pynifti, scipy, and numpy to run. I have installed all these programs successfully using macports on Mac OS 10.8.

I unarchived the scripts and ran the test script getting this error:

$ python pyClusterROI_test.py

Traceback (most recent call last):

  File "pyClusterROI_test.py", line 48, in <module>

    from make_local_connectivity_scorr import *

  File "/Users/matthewnye/Downloads/pyClusterROI/make_local_connectivity_scorr.py", line 40, in <module>

    from scipy.sparse.linalg.eigen.arpack import eigen_symmetric

ImportError: cannot import name eigen_symmetric


I have been trying to hunt down what this error may be and it seems that older versions of scipy might have used  ?scipy.sparse.linalg.eigen.arpack.eigen symmetric()?  where as new versions it has been renamed as "scipy.sparse.linalg.eigen.arpack.eigs()."

Does anyone know what might be causing this error or if it is a version problem is there any way to get an older version of scipy through macports? Also I have considered installing all these modules through a manual install but since i am a novice user I did not want to attempt that if there was a potentially simpler solution.

Any help would be much appreciated!

-Amy




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121203/732b39c6/attachment.html>

From matthew.brett at gmail.com  Mon Dec  3 14:56:31 2012
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 3 Dec 2012 11:56:31 -0800
Subject: [SciPy-User] New Scipy website
In-Reply-To: <CAOvn4qiMKtU1vgUJsQWXxo6z7n66+5k3snMSbcj78f47hELvfg@mail.gmail.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
	<CAOvn4qiMKtU1vgUJsQWXxo6z7n66+5k3snMSbcj78f47hELvfg@mail.gmail.com>
Message-ID: <CAH6Pt5rWohhD_rUpO=b5Y+f2s-ZW2ErLs-0F6YAn_1Aq5UQMqw@mail.gmail.com>

Hi,

On Mon, Dec 3, 2012 at 10:02 AM, Thomas Kluyver <takowl at gmail.com> wrote:
> >From the lack of offers for a new server to host on, I assume that we're
> going to keep using Github pages. So I'll put something together to do a
> "poor man's redirect" on the pages I listed:

Is there an advantage on hosting somewhere other than github pages?
I'm very happy to host on my space if that's an advantage.

Sourceforge is an obvious free alternative for html page hosting.

Best,

Matthew


From davidmenhur at gmail.com  Mon Dec  3 14:58:14 2012
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Mon, 3 Dec 2012 20:58:14 +0100
Subject: [SciPy-User] running script error (eigen symmetric)
In-Reply-To: <13BFE00357AE78459CE7894943A31CA91BD932BE@MED-CORE07A.med.wayne.edu>
References: <13BFE00357AE78459CE7894943A31CA91BD932BE@MED-CORE07A.med.wayne.edu>
Message-ID: <CAJhcF=1wV3i1yJ=-MYAL=Qvg4w1g1V9jD622RuGo9=VboLj6Hw@mail.gmail.com>

In Mac, the python command corresponds to the python that come shipped
with it, not the one installed by macports, and therefore, the one
that has scipy installed. It is something like python-macports2.6 or
so (tupe python and tab/double tab on the shell to  make sure).

On Mon, Dec 3, 2012 at 5:29 PM, Anderson, Amy <aanderso at med.wayne.edu> wrote:
> Dear Scipy users,
>
> I am a new python and scipy user and I am looking for some help to
> troubleshoot an error I have been getting when trying to run a script in
> python.  The script I am trying to run is called pyCluster ROI and it needs
> python, pynifti, scipy, and numpy to run. I have installed all these
> programs successfully using macports on Mac OS 10.8.
>
> I unarchived the scripts and ran the test script getting this error:
>
> $ python pyClusterROI_test.py
>
> Traceback (most recent call last):
>
>   File "pyClusterROI_test.py", line 48, in <module>
>
>     from make_local_connectivity_scorr import *
>
>   File
> "/Users/matthewnye/Downloads/pyClusterROI/make_local_connectivity_scorr.py",
> line 40, in <module>
>
>     from scipy.sparse.linalg.eigen.arpack import eigen_symmetric
>
> ImportError: cannot import name eigen_symmetric
>
>
> I have been trying to hunt down what this error may be and it seems that
> older versions of scipy might have used
> ?scipy.sparse.linalg.eigen.arpack.eigen symmetric()?  where as new versions
> it has been renamed as "scipy.sparse.linalg.eigen.arpack.eigs()."
>
>
> Does anyone know what might be causing this error or if it is a version
> problem is there any way to get an older version of scipy through macports?
> Also I have considered installing all these modules through a manual install
> but since i am a novice user I did not want to attempt that if there was a
> potentially simpler solution.
>
> Any help would be much appreciated!
>
> -Amy
>
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From cournape at gmail.com  Mon Dec  3 14:59:04 2012
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 3 Dec 2012 20:59:04 +0100
Subject: [SciPy-User] Help Scipy Installation
In-Reply-To: <7bab8c8a4df595dbe43ecef657178089.squirrel@webmail.instem.res.in>
References: <7bab8c8a4df595dbe43ecef657178089.squirrel@webmail.instem.res.in>
Message-ID: <CAGY4rcXGLeLYLqfNbHoYtHWgRW0g+DC+9dT5BSm+E93dGS_W+Q@mail.gmail.com>

Hi Deepak

On Mon, Dec 3, 2012 at 5:14 AM, Poduval Deepak Balakrishnan
<pdeepak at instem.res.in> wrote:
> Hi,
>
> I am deepak. I installed numpy but when i try and install scipy I get the
> following error. Please help
>
> Traceback (most recent call last):
>   File "setup.py", line 208, in <module>
>     setup_package()
>   File "setup.py", line 145, in setup_package
>     from numpy.distutils.core import setup
>   File
> "/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/__init__.py",
> line 137, in <module>
>     import add_newdocs
>   File
> "/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/add_newdocs.py",
> line 9, in <module>
>     from numpy.lib import add_newdoc
>   File
> "/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/lib/__init__.py",
> line 4, in <module>
>     from type_check import *
>   File
> "/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/lib/type_check.py",
> line 8, in <module>
>     import numpy.core.numeric as _nx
>   File
> "/usr/lib/python2.6/site-packages/numpy-1.6.1-0.egg/numpy/core/__init__.py",
> line 5, in <module>
>     import multiarray
> ImportError: libpython2.7.so.1.0: cannot open shared object file: No such
> file or directory
>

It looks like you installed numpy in the directory expected by python
2.6 (/usr/lib/python2.6/...), but built it with python 2.7 (since it
is looking for the python 2.7 .so file).

How exactly did you install numpy ? Also, what's the value of your
PYTHONPATH variable ?

thanks,
David


From amyla333 at gmail.com  Mon Dec  3 16:19:15 2012
From: amyla333 at gmail.com (Amy Anderson)
Date: Mon, 3 Dec 2012 16:19:15 -0500
Subject: [SciPy-User] running script error (eigen symmetric)
In-Reply-To: <CAJhcF=1wV3i1yJ=-MYAL=Qvg4w1g1V9jD622RuGo9=VboLj6Hw@mail.gmail.com>
References: <13BFE00357AE78459CE7894943A31CA91BD932BE@MED-CORE07A.med.wayne.edu>
	<CAJhcF=1wV3i1yJ=-MYAL=Qvg4w1g1V9jD622RuGo9=VboLj6Hw@mail.gmail.com>
Message-ID: <CAP_FHc8h9PtqcC74CeELNu6SdRc8TwpRRaPi1pmBooSsST9cPg@mail.gmail.com>

I am actually having an issue getting an older version of Scipy to install
through macports.  I need a version of scipy before 0.9 because the eigen
symmetric module was not changed yet. I have been using macports to install
scipy etc because I thought it was the easier way but at this point I
really just need to get this script working and need an older version of
scipy.  if anyone has any ideas please let me know.

On Mon, Dec 3, 2012 at 2:58 PM, Da?id <davidmenhur at gmail.com> wrote:

> In Mac, the python command corresponds to the python that come shipped
> with it, not the one installed by macports, and therefore, the one
> that has scipy installed. It is something like python-macports2.6 or
> so (tupe python and tab/double tab on the shell to  make sure).
>
> On Mon, Dec 3, 2012 at 5:29 PM, Anderson, Amy <aanderso at med.wayne.edu>
> wrote:
> > Dear Scipy users,
> >
> > I am a new python and scipy user and I am looking for some help to
> > troubleshoot an error I have been getting when trying to run a script in
> > python.  The script I am trying to run is called pyCluster ROI and it
> needs
> > python, pynifti, scipy, and numpy to run. I have installed all these
> > programs successfully using macports on Mac OS 10.8.
> >
> > I unarchived the scripts and ran the test script getting this error:
> >
> > $ python pyClusterROI_test.py
> >
> > Traceback (most recent call last):
> >
> >   File "pyClusterROI_test.py", line 48, in <module>
> >
> >     from make_local_connectivity_scorr import *
> >
> >   File
> >
> "/Users/matthewnye/Downloads/pyClusterROI/make_local_connectivity_scorr.py",
> > line 40, in <module>
> >
> >     from scipy.sparse.linalg.eigen.arpack import eigen_symmetric
> >
> > ImportError: cannot import name eigen_symmetric
> >
> >
> > I have been trying to hunt down what this error may be and it seems that
> > older versions of scipy might have used
> > ?scipy.sparse.linalg.eigen.arpack.eigen symmetric()?  where as new
> versions
> > it has been renamed as "scipy.sparse.linalg.eigen.arpack.eigs()."
> >
> >
> > Does anyone know what might be causing this error or if it is a version
> > problem is there any way to get an older version of scipy through
> macports?
> > Also I have considered installing all these modules through a manual
> install
> > but since i am a novice user I did not want to attempt that if there was
> a
> > potentially simpler solution.
> >
> > Any help would be much appreciated!
> >
> > -Amy
> >
> >
> >
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121203/113d5132/attachment.html>

From deil.christoph at googlemail.com  Mon Dec  3 16:29:27 2012
From: deil.christoph at googlemail.com (Christoph Deil)
Date: Mon, 3 Dec 2012 22:29:27 +0100
Subject: [SciPy-User] running script error (eigen symmetric)
In-Reply-To: <CAP_FHc8h9PtqcC74CeELNu6SdRc8TwpRRaPi1pmBooSsST9cPg@mail.gmail.com>
References: <13BFE00357AE78459CE7894943A31CA91BD932BE@MED-CORE07A.med.wayne.edu>
	<CAJhcF=1wV3i1yJ=-MYAL=Qvg4w1g1V9jD622RuGo9=VboLj6Hw@mail.gmail.com>
	<CAP_FHc8h9PtqcC74CeELNu6SdRc8TwpRRaPi1pmBooSsST9cPg@mail.gmail.com>
Message-ID: <8304C676-A87C-4B63-9554-9744C435D4E6@gmail.com>

Dear Amy,

the scipy you installed via Macports is version 0.11 :

$ port installed py27-scipy
The following ports are currently installed:
  py27-scipy @0.11.0_0+gcc45 (active)

Indeed for that version of scipy there is no function eigen_symmetric in scipy.sparse.linalg.eigen.arpack, as the ImportError in your script said:

$ python
Python 2.7.3 (default, Oct 22 2012, 20:01:15) 
[GCC 4.2.1 Compatible Apple Clang 4.0 ((tags/Apple/clang-421.0.60))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy.sparse.linalg.eigen.arpack import eigen_symmetric
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name eigen_symmetric
>>> from scipy.sparse.linalg.eigen.arpack import eigs           
>>> exit()
$ 

It is possible to install an older version of scipy with Macports, but it looks complicated (I've never done this):
https://trac.macports.org/wiki/howto/InstallingOlderPort

Installing scipy 0.9 on a modern Mac outside Macports is also not simple.
You could try a binary or source install from http://sourceforge.net/projects/scipy/files/scipy/ .

Is updating your script to use eigs instead of eigen_symmetric an option?
Or ask the guy that wrote it do update it for you?

Christoph


On Dec 3, 2012, at 10:19 PM, Amy Anderson <amyla333 at gmail.com> wrote:

> I am actually having an issue getting an older version of Scipy to install through macports.  I need a version of scipy before 0.9 because the eigen symmetric module was not changed yet. I have been using macports to install scipy etc because I thought it was the easier way but at this point I really just need to get this script working and need an older version of scipy.  if anyone has any ideas please let me know.
> 
> On Mon, Dec 3, 2012 at 2:58 PM, Da?id <davidmenhur at gmail.com> wrote:
> In Mac, the python command corresponds to the python that come shipped
> with it, not the one installed by macports, and therefore, the one
> that has scipy installed. It is something like python-macports2.6 or
> so (tupe python and tab/double tab on the shell to  make sure).
> 
> On Mon, Dec 3, 2012 at 5:29 PM, Anderson, Amy <aanderso at med.wayne.edu> wrote:
> > Dear Scipy users,
> >
> > I am a new python and scipy user and I am looking for some help to
> > troubleshoot an error I have been getting when trying to run a script in
> > python.  The script I am trying to run is called pyCluster ROI and it needs
> > python, pynifti, scipy, and numpy to run. I have installed all these
> > programs successfully using macports on Mac OS 10.8.
> >
> > I unarchived the scripts and ran the test script getting this error:
> >
> > $ python pyClusterROI_test.py
> >
> > Traceback (most recent call last):
> >
> >   File "pyClusterROI_test.py", line 48, in <module>
> >
> >     from make_local_connectivity_scorr import *
> >
> >   File
> > "/Users/matthewnye/Downloads/pyClusterROI/make_local_connectivity_scorr.py",
> > line 40, in <module>
> >
> >     from scipy.sparse.linalg.eigen.arpack import eigen_symmetric
> >
> > ImportError: cannot import name eigen_symmetric
> >
> >
> > I have been trying to hunt down what this error may be and it seems that
> > older versions of scipy might have used
> > ?scipy.sparse.linalg.eigen.arpack.eigen symmetric()?  where as new versions
> > it has been renamed as "scipy.sparse.linalg.eigen.arpack.eigs()."
> >
> >
> > Does anyone know what might be causing this error or if it is a version
> > problem is there any way to get an older version of scipy through macports?
> > Also I have considered installing all these modules through a manual install
> > but since i am a novice user I did not want to attempt that if there was a
> > potentially simpler solution.
> >
> > Any help would be much appreciated!
> >
> > -Amy
> >
> >
> >
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121203/860430c7/attachment.html>

From pav at iki.fi  Mon Dec  3 16:39:37 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 03 Dec 2012 23:39:37 +0200
Subject: [SciPy-User] running script error (eigen symmetric)
In-Reply-To: <8304C676-A87C-4B63-9554-9744C435D4E6@gmail.com>
References: <13BFE00357AE78459CE7894943A31CA91BD932BE@MED-CORE07A.med.wayne.edu>
	<CAJhcF=1wV3i1yJ=-MYAL=Qvg4w1g1V9jD622RuGo9=VboLj6Hw@mail.gmail.com>
	<CAP_FHc8h9PtqcC74CeELNu6SdRc8TwpRRaPi1pmBooSsST9cPg@mail.gmail.com>
	<8304C676-A87C-4B63-9554-9744C435D4E6@gmail.com>
Message-ID: <k9j66n$cs9$1@ger.gmane.org>

03.12.2012 23:29, Christoph Deil kirjoitti:
[clip]
>>>> from scipy.sparse.linalg.eigen.arpack import eigs           
>>>> exit()

Or, better, as documented:

>>> from scipy.sparse.linalg import eigs

http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.eigs.html

-- 
Pauli Virtanen



From takowl at gmail.com  Mon Dec  3 17:13:19 2012
From: takowl at gmail.com (Thomas Kluyver)
Date: Mon, 3 Dec 2012 22:13:19 +0000
Subject: [SciPy-User] New Scipy website
In-Reply-To: <1101fb90-1df0-413e-883b-fa0b9cb3e334@googlegroups.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
	<CAOvn4qiMKtU1vgUJsQWXxo6z7n66+5k3snMSbcj78f47hELvfg@mail.gmail.com>
	<CAKdwa3nx2gna9uxAgHG4V9kQ+pVDFZqqQNO8QWkX-8Bvk=sOww@mail.gmail.com>
	<1101fb90-1df0-413e-883b-fa0b9cb3e334@googlegroups.com>
Message-ID: <CAOvn4qhLCq5+coHE8ykwBUs-Gb3z6tu1eyVZbty_QOaehKpOng@mail.gmail.com>

On 3 December 2012 21:44, BrettRMurphy <bmurphy at enthought.com> wrote:

> Sorry it took us at Enthought a while to get back to this thread.
>
> In addition to the server, we've got some people who can help out with
> setting things up and getting the new site up and running. To this end,
> Ognen D has set up a private mailing list for people who will be working on
> the effort. The list is private as they'll be discussing data security and
> privacy issues. The current list is Ognen D, Prabhu R, Thomas K, Pauli V,
> David C, Dhruv B and Parth B. If anyone is interested in joining, please
> send a note to ognen at enthought,com
>
> We're looking forward to helping get SciPy.org updated.
>

Thanks Brett, Robert. Should I have received an invitation or notification
about this mailing list?

Matthew, the disadvantage of Github pages is that you can't create proper
HTTP redirects. Thanks for the offer of hosting, although we'll probably
use Enthought's offer.

Best wishes,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121203/50fa3b33/attachment.html>

From travis at continuum.io  Mon Dec  3 21:16:30 2012
From: travis at continuum.io (Travis Oliphant)
Date: Mon, 3 Dec 2012 21:16:30 -0500
Subject: [SciPy-User] New Scipy website
In-Reply-To: <CAOvn4qhLCq5+coHE8ykwBUs-Gb3z6tu1eyVZbty_QOaehKpOng@mail.gmail.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
	<CAOvn4qiMKtU1vgUJsQWXxo6z7n66+5k3snMSbcj78f47hELvfg@mail.gmail.com>
	<CAKdwa3nx2gna9uxAgHG4V9kQ+pVDFZqqQNO8QWkX-8Bvk=sOww@mail.gmail.com>
	<1101fb90-1df0-413e-883b-fa0b9cb3e334@googlegroups.com>
	<CAOvn4qhLCq5+coHE8ykwBUs-Gb3z6tu1eyVZbty_QOaehKpOng@mail.gmail.com>
Message-ID: <A0042025-AA2F-48B6-B1B6-1D28EE8D3946@continuum.io>

NumFocus can also easily host and administer an EC2 server if that is needed.   If Enthought is willing to continue to host the SciPy server then that is great. 

This is also something that NumFocus is able to do at any time.    If there are other projects that would like a server setup for them --- if github pages is not enough ---  just contact admin at numfocus.org.  


On Dec 3, 2012, at 5:13 PM, Thomas Kluyver <takowl at gmail.com> wrote:

> On 3 December 2012 21:44, BrettRMurphy <bmurphy at enthought.com> wrote:
> Sorry it took us at Enthought a while to get back to this thread.
> 
> In addition to the server, we've got some people who can help out with setting things up and getting the new site up and running. To this end, Ognen D has set up a private mailing list for people who will be working on the effort. The list is private as they'll be discussing data security and privacy issues. The current list is Ognen D, Prabhu R, Thomas K, Pauli V, David C, Dhruv B and Parth B. If anyone is interested in joining, please send a note to ognen at enthought,com
> 
> We're looking forward to helping get SciPy.org updated.
> 
> Thanks Brett, Robert. Should I have received an invitation or notification about this mailing list?
> 
> Matthew, the disadvantage of Github pages is that you can't create proper HTTP redirects. Thanks for the offer of hosting, although we'll probably use Enthought's offer.
> 
> Best wishes,
> Thomas
> -- 
>  
>  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121203/efaf63eb/attachment.html>

From josef.pktd at gmail.com  Mon Dec  3 22:30:10 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 3 Dec 2012 22:30:10 -0500
Subject: [SciPy-User] distributions - who got the most ?
Message-ID: <CAMMTP+BLWAjpbfbQ78TdP4WgnMoXAS3668WGDyY7xYx1U+8_fQ@mail.gmail.com>

scipy.stats has more than 90 distributions.

Do we want to increase it by almost a factor of 10? :)

While looking for the cdf of a distribution, I found this :
http://www.mathworks.com/matlabcentral/fileexchange/35008-generation-of-random-variates

He collected 870 distributions (under BSD license). Includes generic
random number generation.

Even though there are some variations of distributions counted
separately, given my quick browsing this looks impressive and a good
source for code and references.
Coding style is not great but it's 10 years or so of collecting distributions.

Josef


From travis at continuum.io  Mon Dec  3 22:50:43 2012
From: travis at continuum.io (Travis Oliphant)
Date: Mon, 3 Dec 2012 21:50:43 -0600
Subject: [SciPy-User] distributions - who got the most ?
In-Reply-To: <CAMMTP+BLWAjpbfbQ78TdP4WgnMoXAS3668WGDyY7xYx1U+8_fQ@mail.gmail.com>
References: <CAMMTP+BLWAjpbfbQ78TdP4WgnMoXAS3668WGDyY7xYx1U+8_fQ@mail.gmail.com>
Message-ID: <504F8B26-F48D-41CA-A049-D010460A97FA@continuum.io>


On Dec 3, 2012, at 9:30 PM, josef.pktd at gmail.com wrote:

> scipy.stats has more than 90 distributions.
> 
> Do we want to increase it by almost a factor of 10? :)
> 

That is pretty cool.   I would personally love to see this sort of thing.  This might be a good project to propose to NumFOCUS for an enterprising young student / mentor combination under the Technical Fellowship Program.   Josef would be a good mentor for this sort of thing. 

The John Hunter Technical Fellowship is a 3 month to 18 month program where a student / post-doc works with a mentor (both of whom receive funding) on some project of interest to the scientific computing community.    Proposals are reviewed by a grant team.     We are just getting this program started and need good projects and mentors.      Students can submit proposals and can often be matched with a mentor.  

Contact numfocus at googlegroups.org to inquire or send a proposal.   

-Travis




> While looking for the cdf of a distribution, I found this :
> http://www.mathworks.com/matlabcentral/fileexchange/35008-generation-of-random-variates
> 
> He collected 870 distributions (under BSD license). Includes generic
> random number generation.
> 
> Even though there are some variations of distributions counted
> separately, given my quick browsing this looks impressive and a good
> source for code and references.
> Coding style is not great but it's 10 years or so of collecting distributions.
> 
> Josef
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From kmichael.aye at gmail.com  Tue Dec  4 02:33:17 2012
From: kmichael.aye at gmail.com (Michael Aye)
Date: Mon, 3 Dec 2012 23:33:17 -0800
Subject: [SciPy-User] distributions - who got the most ?
References: <CAMMTP+BLWAjpbfbQ78TdP4WgnMoXAS3668WGDyY7xYx1U+8_fQ@mail.gmail.com>
	<504F8B26-F48D-41CA-A049-D010460A97FA@continuum.io>
Message-ID: <k9k8vm$gm4$1@ger.gmane.org>

On 2012-12-04 03:50:43 +0000, Travis Oliphant said:

> On Dec 3, 2012, at 9:30 PM, josef.pktd at gmail.com wrote:
> 
>> scipy.stats has more than 90 distributions.
>> 
>> Do we want to increase it by almost a factor of 10? :)
>> 
> 
> That is pretty cool.   I would personally love to see this sort of 
> thing.  This might be a good project to propose to NumFOCUS for an 
> enterprising young student / mentor combination under the Technical 
> Fellowship Program.   Josef would be a good mentor for this sort of 
> thing.
> 
> The John Hunter Technical Fellowship is a 3 month to 18 month program 
> where a student / post-doc works with a mentor (both of whom receive 
> funding) on some project of interest to the scientific computing 
> community.    Proposals are reviewed by a grant team.     We are just 
> getting this program started and need good projects and mentors.      
> Students can submit proposals and can often be matched with a mentor.
> 
> Contact numfocus at googlegroups.org to inquire or send a proposal.

Interesting, I just thought about something like this. Will part-time 
be possible? I am asking here, because the list might want to know this 
as well.

Michael


> 
> -Travis
> 
> 
> 
> 
>> While looking for the cdf of a distribution, I found this :
>> http://www.mathworks.com/matlabcentral/fileexchange/35008-generation-of-random-variates 
>> 
>> 
>> He collected 870 distributions (under BSD license). Includes generic
>> random number generation.
>> 
>> Even though there are some variations of distributions counted
>> separately, given my quick browsing this looks impressive and a good
>> source for code and references.
>> Coding style is not great but it's 10 years or so of collecting distributions.
>> 
>> Josef
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user





From rkern at enthought.com  Mon Dec  3 15:17:11 2012
From: rkern at enthought.com (Robert Kern)
Date: Mon, 3 Dec 2012 20:17:11 +0000
Subject: [SciPy-User] New Scipy website
In-Reply-To: <CAOvn4qiMKtU1vgUJsQWXxo6z7n66+5k3snMSbcj78f47hELvfg@mail.gmail.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
	<CAOvn4qiMKtU1vgUJsQWXxo6z7n66+5k3snMSbcj78f47hELvfg@mail.gmail.com>
Message-ID: <CAKdwa3nx2gna9uxAgHG4V9kQ+pVDFZqqQNO8QWkX-8Bvk=sOww@mail.gmail.com>

On Mon, Dec 3, 2012 at 6:02 PM, Thomas Kluyver <takowl at gmail.com> wrote:
> From the lack of offers for a new server to host on, I assume that we're
> going to keep using Github pages.

Enthought is happy to continue to donate a server for the SciPy
website (on EC2 infrastructure most likely).

--
Robert Kern
Enthought


From bmurphy at enthought.com  Mon Dec  3 16:44:43 2012
From: bmurphy at enthought.com (BrettRMurphy)
Date: Mon, 3 Dec 2012 13:44:43 -0800 (PST)
Subject: [SciPy-User] New Scipy website
In-Reply-To: <CAKdwa3nx2gna9uxAgHG4V9kQ+pVDFZqqQNO8QWkX-8Bvk=sOww@mail.gmail.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
	<CAOvn4qiMKtU1vgUJsQWXxo6z7n66+5k3snMSbcj78f47hELvfg@mail.gmail.com>
	<CAKdwa3nx2gna9uxAgHG4V9kQ+pVDFZqqQNO8QWkX-8Bvk=sOww@mail.gmail.com>
Message-ID: <1101fb90-1df0-413e-883b-fa0b9cb3e334@googlegroups.com>

Sorry it took us at Enthought a while to get back to this thread.

In addition to the server, we've got some people who can help out with 
setting things up and getting the new site up and running. To this end, 
Ognen D has set up a private mailing list for people who will be working on 
the effort. The list is private as they'll be discussing data security and 
privacy issues. The current list is Ognen D, Prabhu R, Thomas K, Pauli V, 
David C, Dhruv B and Parth B. If anyone is interested in joining, please 
send a note to ognen at enthought,com

We're looking forward to helping get SciPy.org updated.

Brett Murphy
Enthought

On Monday, December 3, 2012 2:17:11 PM UTC-6, Robert Kern wrote:
>
> On Mon, Dec 3, 2012 at 6:02 PM, Thomas Kluyver <tak... at gmail.com<javascript:>> 
> wrote: 
> > From the lack of offers for a new server to host on, I assume that we're 
> > going to keep using Github pages. 
>
> Enthought is happy to continue to donate a server for the SciPy 
> website (on EC2 infrastructure most likely). 
>
> -- 
> Robert Kern 
> Enthought 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121203/ec34a2e9/attachment.html>

From bmurphy at enthought.com  Mon Dec  3 17:42:47 2012
From: bmurphy at enthought.com (BrettRMurphy)
Date: Mon, 3 Dec 2012 14:42:47 -0800 (PST)
Subject: [SciPy-User] New Scipy website
In-Reply-To: <CAOvn4qhLCq5+coHE8ykwBUs-Gb3z6tu1eyVZbty_QOaehKpOng@mail.gmail.com>
References: <CAOvn4qjFtFe6jqOUR-TcQ5rb-bPsghdizWsQEGb_hik+Bv3XBw@mail.gmail.com>
	<CAOvn4qiMKtU1vgUJsQWXxo6z7n66+5k3snMSbcj78f47hELvfg@mail.gmail.com>
	<CAKdwa3nx2gna9uxAgHG4V9kQ+pVDFZqqQNO8QWkX-8Bvk=sOww@mail.gmail.com>
	<1101fb90-1df0-413e-883b-fa0b9cb3e334@googlegroups.com>
	<CAOvn4qhLCq5+coHE8ykwBUs-Gb3z6tu1eyVZbty_QOaehKpOng@mail.gmail.com>
Message-ID: <9b096976-1734-43a1-aea1-f2162abec511@googlegroups.com>

I assume Ognen will update the list members soon.

-- Brett
Enthought

On Monday, December 3, 2012 4:13:19 PM UTC-6, Thomas Kluyver wrote:
>
> On 3 December 2012 21:44, BrettRMurphy <bmu... at enthought.com <javascript:>
> > wrote:
>
>> Sorry it took us at Enthought a while to get back to this thread.
>>
>> In addition to the server, we've got some people who can help out with 
>> setting things up and getting the new site up and running. To this end, 
>> Ognen D has set up a private mailing list for people who will be working on 
>> the effort. The list is private as they'll be discussing data security and 
>> privacy issues. The current list is Ognen D, Prabhu R, Thomas K, Pauli V, 
>> David C, Dhruv B and Parth B. If anyone is interested in joining, please 
>> send a note to ognen at enthought,com
>>
>> We're looking forward to helping get SciPy.org updated.
>>
>
> Thanks Brett, Robert. Should I have received an invitation or notification 
> about this mailing list?
>
> Matthew, the disadvantage of Github pages is that you can't create proper 
> HTTP redirects. Thanks for the offer of hosting, although we'll probably 
> use Enthought's offer.
>
> Best wishes,
> Thomas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121203/9a7460aa/attachment.html>

From servant.mathieu at gmail.com  Tue Dec  4 10:55:37 2012
From: servant.mathieu at gmail.com (servant mathieu)
Date: Tue, 4 Dec 2012 16:55:37 +0100
Subject: [SciPy-User] equivalent of R quantile() function in scipy
Message-ID: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>

Dear list,

>From an array X of values, the quantile () function in R can return the
score at any given specified quantile : e.g., quant_values = quantile (X,
probs = c(.1, .3, .5, .7, .9)).

The scoreatpercentile() function in scipy seems to to the same stuff.
However, you can specify only one quantile per function call e.g.
quant_value = scoreatpercentile (X, per = 10) etc..

Is it possible to return more than one quantile per function call?

Cheers,
Mathieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121204/eb74cd41/attachment.html>

From jsseabold at gmail.com  Tue Dec  4 11:14:48 2012
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 4 Dec 2012 11:14:48 -0500
Subject: [SciPy-User] equivalent of R quantile() function in scipy
In-Reply-To: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>
References: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>
Message-ID: <CAKF=DjvTwS9odrVUJ-yUsm6_knvSE3eQvMdgYcahOoufqO_q_A@mail.gmail.com>

On Tue, Dec 4, 2012 at 10:55 AM, servant mathieu
<servant.mathieu at gmail.com> wrote:
> Dear list,
>
> From an array X of values, the quantile () function in R can return the
> score at any given specified quantile : e.g., quant_values = quantile (X,
> probs = c(.1, .3, .5, .7, .9)).
>
> The scoreatpercentile() function in scipy seems to to the same stuff.
> However, you can specify only one quantile per function call e.g.
> quant_value = scoreatpercentile (X, per = 10) etc..
>

Yes, this is pretty annoying.

> Is it possible to return more than one quantile per function call?

You can use percentile in numpy. Note that it expects the percentiles in 0-100.

quants = np.percentile(np.random.randn(25), [10, 30, 50, 70, 90])

Skipper


From jjhelmus at gmail.com  Tue Dec  4 11:20:16 2012
From: jjhelmus at gmail.com (Jonathan Helmus)
Date: Tue, 04 Dec 2012 11:20:16 -0500
Subject: [SciPy-User] equivalent of R quantile() function in scipy
In-Reply-To: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>
References: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>
Message-ID: <50BE22C0.6040103@gmail.com>

Mathieu,

numpy.percentile can can accept a sequence of percentiles as the second 
parameter:

In [8]: probs = [10.0, 30.0, 50.0, 70.0, 90.0]

In [9]: a = np.arange(100)

In [10]: np.percentile(a, probs)
Out[10]:
[9.9000000000000004,
  29.699999999999999,
  49.5,
  69.299999999999997,
  89.100000000000009]


Or you could use a list comprehension with scipy.stat.scoreatpercentile:

In [11]: [scipy.stats.scoreatpercentile(a, i) for i in probs]
Out[11]:
[9.9000000000000004,
  29.699999999999999,
  49.5,
  69.299999999999997,
  89.100000000000009]


The first solution is probably faster as the sequence is only sorted 
once. Hope this helps,

     - Jonathan Helmus



On 12/04/2012 10:55 AM, servant mathieu wrote:
> Dear list,
> From an array X of values, the quantile () function in R can return 
> the score at any given specified quantile : e.g., quant_values = 
> quantile (X, probs = c(.1, .3, .5, .7, .9)).
> The scoreatpercentile() function in scipy seems to to the same stuff. 
> However, you can specify only one quantile per function call e.g.  
> quant_value = scoreatpercentile (X, per = 10) etc..
> Is it possible to return more than one quantile per function call?
> Cheers,
> Mathieu
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121204/68eb6ba9/attachment.html>

From pav at iki.fi  Tue Dec  4 11:26:55 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 04 Dec 2012 18:26:55 +0200
Subject: [SciPy-User] equivalent of R quantile() function in scipy
In-Reply-To: <50BE22C0.6040103@gmail.com>
References: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>
	<50BE22C0.6040103@gmail.com>
Message-ID: <k9l88d$a4q$1@ger.gmane.org>

04.12.2012 18:20, Jonathan Helmus kirjoitti:
> numpy.percentile can can accept a sequence of percentiles as the second
> parameter:
> 
> In [8]: probs = [10.0, 30.0, 50.0, 70.0, 90.0]
> 
> In [9]: a = np.arange(100)
> 
> In [10]: np.percentile(a, probs)
[clip]
> In [11]: [scipy.stats.scoreatpercentile(a, i) for i in probs]
> Out[11]:
[clip]

It could be useful if someone would take a look if that the
implementation in scipy can be easily adapted for that.

-- 
Pauli Virtanen




From jjhelmus at gmail.com  Tue Dec  4 11:44:47 2012
From: jjhelmus at gmail.com (Jonathan Helmus)
Date: Tue, 04 Dec 2012 11:44:47 -0500
Subject: [SciPy-User] equivalent of R quantile() function in scipy
In-Reply-To: <k9l88d$a4q$1@ger.gmane.org>
References: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>
	<50BE22C0.6040103@gmail.com> <k9l88d$a4q$1@ger.gmane.org>
Message-ID: <50BE287F.1020800@gmail.com>

On 12/04/2012 11:26 AM, Pauli Virtanen wrote:
> 04.12.2012 18:20, Jonathan Helmus kirjoitti:
>> numpy.percentile can can accept a sequence of percentiles as the second
>> parameter:
>>
>> In [8]: probs = [10.0, 30.0, 50.0, 70.0, 90.0]
>>
>> In [9]: a = np.arange(100)
>>
>> In [10]: np.percentile(a, probs)
> [clip]
>> In [11]: [scipy.stats.scoreatpercentile(a, i) for i in probs]
>> Out[11]:
> [clip]
>
> It could be useful if someone would take a look if that the
> implementation in scipy can be easily adapted for that.
>
NumPy's implementation is pure python and quite straightforward, I'll 
put together a pull request to include similar functionality in SciPy.  
Adding the limit optional parameter shouldn't be too hard.

- Jonathan Helmus


From jsseabold at gmail.com  Tue Dec  4 11:43:08 2012
From: jsseabold at gmail.com (Skipper Seabold)
Date: Tue, 4 Dec 2012 11:43:08 -0500
Subject: [SciPy-User] equivalent of R quantile() function in scipy
In-Reply-To: <50BE287F.1020800@gmail.com>
References: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>
	<50BE22C0.6040103@gmail.com> <k9l88d$a4q$1@ger.gmane.org>
	<50BE287F.1020800@gmail.com>
Message-ID: <CAKF=DjvpKqgShs=scQXJ+b-U=qV+d+z33LZieiqv4HiOOA1rNA@mail.gmail.com>

On Tue, Dec 4, 2012 at 11:44 AM, Jonathan Helmus <jjhelmus at gmail.com> wrote:
> On 12/04/2012 11:26 AM, Pauli Virtanen wrote:
>> 04.12.2012 18:20, Jonathan Helmus kirjoitti:
>>> numpy.percentile can can accept a sequence of percentiles as the second
>>> parameter:
>>>
>>> In [8]: probs = [10.0, 30.0, 50.0, 70.0, 90.0]
>>>
>>> In [9]: a = np.arange(100)
>>>
>>> In [10]: np.percentile(a, probs)
>> [clip]
>>> In [11]: [scipy.stats.scoreatpercentile(a, i) for i in probs]
>>> Out[11]:
>> [clip]
>>
>> It could be useful if someone would take a look if that the
>> implementation in scipy can be easily adapted for that.
>>
> NumPy's implementation is pure python and quite straightforward, I'll
> put together a pull request to include similar functionality in SciPy.
> Adding the limit optional parameter shouldn't be too hard.
>

Great thanks for looking at this. Could you also look at adding an
axis keyword to scoreatpercentile? Should be simple, but IIRC this was
missing the last time I looked at this.

Skipper


From pav at iki.fi  Tue Dec  4 11:45:23 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 04 Dec 2012 18:45:23 +0200
Subject: [SciPy-User] equivalent of R quantile() function in scipy
In-Reply-To: <50BE287F.1020800@gmail.com>
References: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>
	<50BE22C0.6040103@gmail.com> <k9l88d$a4q$1@ger.gmane.org>
	<50BE287F.1020800@gmail.com>
Message-ID: <k9l9b1$mtj$1@ger.gmane.org>

04.12.2012 18:44, Jonathan Helmus kirjoitti:
> On 12/04/2012 11:26 AM, Pauli Virtanen wrote:
[clip]
>> It could be useful if someone would take a look if that the
>> implementation in scipy can be easily adapted for that.
>>
> NumPy's implementation is pure python and quite straightforward, I'll 
> put together a pull request to include similar functionality in SciPy.  
> Adding the limit optional parameter shouldn't be too hard.

Great, that's fast action :)

Cheers,
Pauli




From ralf.gommers at gmail.com  Tue Dec  4 16:01:36 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 4 Dec 2012 22:01:36 +0100
Subject: [SciPy-User] distributions - who got the most ?
In-Reply-To: <CAMMTP+BLWAjpbfbQ78TdP4WgnMoXAS3668WGDyY7xYx1U+8_fQ@mail.gmail.com>
References: <CAMMTP+BLWAjpbfbQ78TdP4WgnMoXAS3668WGDyY7xYx1U+8_fQ@mail.gmail.com>
Message-ID: <CABL7CQjbDD06buRaS67iAbi0=khzEvxDuyt1_xFjgUvAe2pbgA@mail.gmail.com>

On Tue, Dec 4, 2012 at 4:30 AM, <josef.pktd at gmail.com> wrote:

> scipy.stats has more than 90 distributions.
>
> Do we want to increase it by almost a factor of 10? :)
>
> While looking for the cdf of a distribution, I found this :
>
> http://www.mathworks.com/matlabcentral/fileexchange/35008-generation-of-random-variates
>
> He collected 870 distributions (under BSD license). Includes generic
> random number generation.
>
> Even though there are some variations of distributions counted
> separately, given my quick browsing this looks impressive and a good
> source for code and references.
> Coding style is not great but it's 10 years or so of collecting
> distributions.
>

Adding a lot of distributions sounds fine to me. That many distributions
would need to go into a separate namespace.

Any additions should be complete though (the Matlab code only has pdf/cdf)
and well tested. The Matlab code doesn't look all that useful except for
the references ("coding style is not great" is really too kind). I also
don't trust the BSD license that's put on it, many files have different
author names in them with no mention of where they came from.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121204/fbb3057b/attachment.html>

From josef.pktd at gmail.com  Tue Dec  4 20:15:56 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 4 Dec 2012 20:15:56 -0500
Subject: [SciPy-User] distributions - who got the most ?
In-Reply-To: <CABL7CQjbDD06buRaS67iAbi0=khzEvxDuyt1_xFjgUvAe2pbgA@mail.gmail.com>
References: <CAMMTP+BLWAjpbfbQ78TdP4WgnMoXAS3668WGDyY7xYx1U+8_fQ@mail.gmail.com>
	<CABL7CQjbDD06buRaS67iAbi0=khzEvxDuyt1_xFjgUvAe2pbgA@mail.gmail.com>
Message-ID: <CAMMTP+CrGZPNoJobc1Bu5VqP_EmbRD-iL9WXGKOy+6mfCEGzVQ@mail.gmail.com>

On Tue, Dec 4, 2012 at 4:01 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Tue, Dec 4, 2012 at 4:30 AM, <josef.pktd at gmail.com> wrote:
>>
>> scipy.stats has more than 90 distributions.
>>
>> Do we want to increase it by almost a factor of 10? :)
>>
>> While looking for the cdf of a distribution, I found this :
>>
>> http://www.mathworks.com/matlabcentral/fileexchange/35008-generation-of-random-variates
>>
>> He collected 870 distributions (under BSD license). Includes generic
>> random number generation.
>>
>> Even though there are some variations of distributions counted
>> separately, given my quick browsing this looks impressive and a good
>> source for code and references.
>> Coding style is not great but it's 10 years or so of collecting
>> distributions.
>
>
> Adding a lot of distributions sounds fine to me. That many distributions
> would need to go into a separate namespace.
>
> Any additions should be complete though (the Matlab code only has pdf/cdf)
> and well tested. The Matlab code doesn't look all that useful except for the
> references ("coding style is not great" is really too kind). I also don't
> trust the BSD license that's put on it, many files have different author
> names in them with no mention of where they came from.

The matlab code includes several "special" functions that look mostly
copied from other authors.
This would need checking, but I doubt we need many of those since we
have scipy.special.
We are missing some special functions for distributions, but I didn't
check whether he has any of those.
The pdfs, and the cdfs when available, look like they were implemented
by the author, at least it looks that way for the small sample that I
checked.
(code quality varies a lot, but many distributions are vectorized or
can be easily vectorized from his code.

Given the pdf, the rest could all be derived generically. But it won't
be efficient.

Also, I just saw that sympy could become useful to derive extra properties
http://matthewrocklin.com/blog/work/2012/12/03/Characteristic-Functions/
sympy.stats also works based only on the pdf (from what I have seen).

I'm a bit skeptical about the number of distributions that are
actually generally useful and not just used once in a journal article.
 My impression from tracking several statistics journals is that there
are at least 10 new distributions each year.

As an example, he has a long list of poisson mixture distributions
that I never heard of except for negative binomial. They might be
useful in some cases, but a more general class might cover it better.
>From a brief look at his reference
http://scholar.google.com/scholar?cluster=6061641765696455790&hl=en&as_sdt=0,5&as_vis=1
I think it might not be necessary to implement all details for 5 or
more distributions separately.
According to Google the paper has only 4 citations.  see also 1)

But there are a lot of distributions, or classes/categories of
distributions that scipy is missing, and are for example available in
R, but in R they are spread out over many packages.

Josef

1) another reference for poisson mixtures (technical, not a quick
read, but a funny table)

Karlis, D. and Xekalaki, E. (2005), Mixed Poisson Distributions.
International Statistical Review, 73: 35?58. doi:
10.1111/j.1751-5823.2005.tb00250.x
http://scholar.google.com/scholar?cluster=4455890634693542956&hl=en&as_sdt=2005&sciodt=0,5

--------------------------
Table 1
Some mixed Poisson distributions.
Mixed Poisson Distribution Mixing Distribution A Key Reference
Negative Binomial Gamma Greenwood & Yule (1920)
Geometric Exponential Johnson et al. (1992)
Poisson-Linear Exponential Family Linear Exponential Family Sankaran (1969)
Poisson?Lindley Lindley Sankaran (1970)
Poisson-Linear Exponential Linear Exponential Kling & Goovaerts (1993)
Poisson-Lognormal Lognormal Bulmer (1974)
Poisson-Confluent Hypergeometric Series Confluent Hypergeometric
Series Bhattacharya (1966)
Poisson-Generalized Inverse Gaussian Generalized Inverse Gaussian Sichel (1974)
Sichel Inverse Gaussian Sichel (1975)
Poisson-Inverse Gamma Inverse Gamma Willmot (1993)
Poisson-Truncated Normal Truncated Normal Patil (1964)
Generalized Waring Gamma Product Ratio Irwin (1975)
Simple Waring Exponential 
 Beta Pielou (1962)
Yule Beta with Specific Parameter Values Simon (1955)
Poisson-Generalized Pareto Generalized Pareto Kempton (1975)
Poisson-Beta I Beta Type I Holla & Bhattacharya (1965)
Poisson-Beta II Beta Type II Gurland (1958)
Poisson-Truncated Beta II Truncated Beta Type II Willmot (1986)
Poisson-Uniform Uniform Bhattacharya (1966)
Poisson-Truncated Gamma Truncated Gamma Willmot (1993)
Poisson-Generalized Gamma Generalized Gamma Albrecht (1984)
Dellaporte Shifted Gamma Ruohonen (1988)
Poisson-Modified Bessel of the 3rd Kind Modified Bessel of the 3rd
Kind Ong & Muthaloo (1995)
Poisson?Pareto Pareto Willmot (1993)
Poisson-Shifted Pareto Shifted Pareto Willmot (1993)
Poisson?Pearson Family Pearson?s Family of Distributions Albrecht (1982)
Poisson-Log-Student Log-Student Gaver & O?Muircheartaigh (1987)
Poisson-Power Function Power Function Distribution Rai (1971)
Poisson?Lomax Lomax Al-Awadhi & Ghitany (2001)
Poisson-Power Variance Power Variance Family Hougaard et al. (1997)
Neyman Poisson Douglas (1980)
Other Discrete Distributions Johnson et al. (1992)
-------------------------------------------------------------------

>
> Ralf
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From sepideh at isc.ac.uk  Wed Dec  5 10:11:28 2012
From: sepideh at isc.ac.uk (Sepideh Rastin)
Date: Wed, 05 Dec 2012 15:11:28 +0000
Subject: [SciPy-User] Need help to use rv_continuous.fit
References: 3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com
Message-ID: <50BF6420.20307@isc.ac.uk>

Hi there,
I have histograms that will form the likelihood function and need to 
find the best normal distributions.
I would like to know if it would be possible to create my input array to 
rv_continuous.fit using the data of my histograms.
Kind regards,
Sep


From malandrea80 at gmail.com  Wed Dec  5 11:12:05 2012
From: malandrea80 at gmail.com (MalAndrea)
Date: Wed, 5 Dec 2012 17:12:05 +0100
Subject: [SciPy-User]  Parallel Differential Evolution
Message-ID: <CAKuNVDb094_OgDC4GGLrASsB21zTiYA1d2cHjZqamgVKBtUnaQ@mail.gmail.com>

Hi,

I have followed Rob suggestions on this, i.e. to substute:

(1). *numpy.zeros(X)* instead of *flex.double(X, 0)*
(2). *1000*numpy.ones(X) *instead of *flex.double(X, 1000)*
(3). *numpy.min(X) *for *flex.min(X)*, etc. (mean, sum)
(4). *numpy.random.uniform(size=N)* for *flex.random_double(N) *

However, to get this work, I also had to modify the following:

- modification (4). only works when floats are meant to be used, I guess,
so in the only case: *rnd =
numpy.random.uniform(size=self.vector_length)*instead of
*flex.random_double(N)*. In the other two cases it is *random_values =
numpy.random.random_integers(low=0.0, high=1.0, size=N)*  instead of  *
flex.random_double(N)*
- Also,* numpy.nanargmin* instead of *flex.min_index*
- Also, *numpy.argsort* instead of *flex.sort_permutation*
- Also, *.copy()* instead of *.deep_copy()*
- Finally, *numpy.random.seed(0)* instead of *flex.set_random_seed(0)*

Now, it works (at least for the Rosenbrock function).
Thank you,
Andrea
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121205/871c67b6/attachment.html>

From josef.pktd at gmail.com  Wed Dec  5 11:44:53 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 5 Dec 2012 11:44:53 -0500
Subject: [SciPy-User] Need help to use rv_continuous.fit
In-Reply-To: <50BF6420.20307@isc.ac.uk>
References: <50BF6420.20307@isc.ac.uk>
Message-ID: <CAMMTP+CSmPpzUkrzmTRy2h8F2ygsHOb3QJxL5C6NqZiMu_NZCQ@mail.gmail.com>

On Wed, Dec 5, 2012 at 10:11 AM, Sepideh Rastin <sepideh at isc.ac.uk> wrote:
> Hi there,
> I have histograms that will form the likelihood function and need to
> find the best normal distributions.
> I would like to know if it would be possible to create my input array to
> rv_continuous.fit using the data of my histograms.

several answers

No, you cannot use the histogram for fit() directly. stats.norm.fit()
expects the original data with individual observations. It's possible
to create an artifical dataset by just creating observations based on
the histogramm.
np.repeat(bin_centers, bin_counts) or something like this.

fitting a normal distribution is "boring": sample mean and standard
deviation are estimates of loc and scale.

There are several scripts on the web, scipy cookbook, scipy central,
... ? how to fit a normal pdf directly to a histogram.

If you have only a small number of bins, then using the above will
cause a discretization bias (reference ?).
In this case, I would fit either the histogram or the cumulative
histogram to the discrete probabilities from the discretization, for
example something like minimizing
def fun(cumhist, bins_edges, loc, scale):
    diff = cumhist - stats.norm.cdf(bin_edges, loc=loc, scale=scale)
#fix or drop first element
    return (diff**2).sum()

Josef





> Kind regards,
> Sep
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From jjhelmus at gmail.com  Wed Dec  5 12:33:20 2012
From: jjhelmus at gmail.com (Jonathan Helmus)
Date: Wed, 05 Dec 2012 12:33:20 -0500
Subject: [SciPy-User] equivalent of R quantile() function in scipy
In-Reply-To: <CAKF=DjvpKqgShs=scQXJ+b-U=qV+d+z33LZieiqv4HiOOA1rNA@mail.gmail.com>
References: <CALnu5bNX2kk+oSE_Uu=QOigoE7vC58hyg3GgZ-GOviL2Z=vFfw@mail.gmail.com>
	<50BE22C0.6040103@gmail.com> <k9l88d$a4q$1@ger.gmane.org>
	<50BE287F.1020800@gmail.com>
	<CAKF=DjvpKqgShs=scQXJ+b-U=qV+d+z33LZieiqv4HiOOA1rNA@mail.gmail.com>
Message-ID: <50BF8560.4050702@gmail.com>

On 12/04/2012 11:43 AM, Skipper Seabold wrote:
> On Tue, Dec 4, 2012 at 11:44 AM, Jonathan Helmus<jjhelmus at gmail.com>  wrote:
>> On 12/04/2012 11:26 AM, Pauli Virtanen wrote:
>>> 04.12.2012 18:20, Jonathan Helmus kirjoitti:
>>>> numpy.percentile can can accept a sequence of percentiles as the second
>>>> parameter:
>>>>
>>>> In [8]: probs = [10.0, 30.0, 50.0, 70.0, 90.0]
>>>>
>>>> In [9]: a = np.arange(100)
>>>>
>>>> In [10]: np.percentile(a, probs)
>>> [clip]
>>>> In [11]: [scipy.stats.scoreatpercentile(a, i) for i in probs]
>>>> Out[11]:
>>> [clip]
>>>
>>> It could be useful if someone would take a look if that the
>>> implementation in scipy can be easily adapted for that.
>>>
>> NumPy's implementation is pure python and quite straightforward, I'll
>> put together a pull request to include similar functionality in SciPy.
>> Adding the limit optional parameter shouldn't be too hard.
>>
> Great thanks for looking at this. Could you also look at adding an
> axis keyword to scoreatpercentile? Should be simple, but IIRC this was
> missing the last time I looked at this.
>
> Skipper
I made a pull request that adds sequences of percentiles and an axis 
keyword to the scoreatpercentile function. 
https://github.com/scipy/scipy/pull/374

We can move any further discussion to the PR comments.

     - Jonathan Helmus


From ndbecker2 at gmail.com  Wed Dec  5 14:01:44 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 05 Dec 2012 14:01:44 -0500
Subject: [SciPy-User] typo in stats tutorial?
Message-ID: <k9o5mm$j82$1@ger.gmane.org>

The text says there's very little difference between Scott's Rule and 
Silverman's Rule.  With the code, I don't think there'd be any difference at 
all:

from scipy import stats

x1 = np.array([-7, -5, 1, 4, 5], dtype=np.float)
kde1 = stats.gaussian_kde(x1)
kde2 = stats.gaussian_kde(x1, bw_method='silverman')

fig = plt.figure()
ax = fig.add_subplot(111)

ax.plot(x1, np.zeros(x1.shape), 'b+', ms=20)  # rug plot
x_eval = np.linspace(-10, 10, num=200)
ax.plot(x_eval, kde1(x_eval), 'k-', label="Scott's Rule")
ax.plot(x_eval, kde1(x_eval), 'r-', label="Silverman's Rule")

plt.show()



From josef.pktd at gmail.com  Wed Dec  5 14:20:00 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 5 Dec 2012 14:20:00 -0500
Subject: [SciPy-User] typo in stats tutorial?
In-Reply-To: <k9o5mm$j82$1@ger.gmane.org>
References: <k9o5mm$j82$1@ger.gmane.org>
Message-ID: <CAMMTP+Bc0LF09SbPkiHhG09RjuzxQX_a8G=JTkknNu9-LD6g3g@mail.gmail.com>

On Wed, Dec 5, 2012 at 2:01 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> The text says there's very little difference between Scott's Rule and
> Silverman's Rule.  With the code, I don't think there'd be any difference at
> all:
>
> from scipy import stats
>
> x1 = np.array([-7, -5, 1, 4, 5], dtype=np.float)
> kde1 = stats.gaussian_kde(x1)
> kde2 = stats.gaussian_kde(x1, bw_method='silverman')
>
> fig = plt.figure()
> ax = fig.add_subplot(111)
>
> ax.plot(x1, np.zeros(x1.shape), 'b+', ms=20)  # rug plot
> x_eval = np.linspace(-10, 10, num=200)
> ax.plot(x_eval, kde1(x_eval), 'k-', label="Scott's Rule")
> ax.plot(x_eval, kde1(x_eval), 'r-', label="Silverman's Rule")

kde2

what are we supposed to see?

nice, example runs without changes in python 3.2.3 (the only python on
my computer that seems to have a recent scipy and matplotlib)

Josef

>
> plt.show()
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From jrocher at enthought.com  Wed Dec  5 16:48:57 2012
From: jrocher at enthought.com (Jonathan Rocher)
Date: Wed, 5 Dec 2012 15:48:57 -0600
Subject: [SciPy-User] SciPy 2013 in Austin TX, June 24th-29th
Message-ID: <CAOzk5Qeqrq7eTTMmvUMi1pr5DPja3c7UTtyqbiWPu2v2uaFwdQ@mail.gmail.com>

Dear all,

We are extremely excited to announce that the 12th SciPy conference will
take place in Austin TX from June 24th to June 29th 2013.

http://conference.scipy.org/scipy2013/

This year again, we will have tutorials, BOFs, sprints in addition to the
conference. We also have many ideas to make this coming edition even more
exciting and productive than the past years and will be probing the
community for ideas and volunteers.

In the meanwhile, please save the dates, signup for the
newsletter<http://enthought.us1.list-manage2.com/subscribe?u=e91b4574d5d1709a9dc4f7ab7&id=069dcb6ee4>
and
pass the word around.

Hope to see many of you then.

Andy Terrel, Co-Chair of Scipy2013
Jonathan Rocher, Co-Chair of Scipy2013
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121205/9c504d9a/attachment.html>

From rmain at gmx.de  Wed Dec  5 17:23:49 2012
From: rmain at gmx.de (rmain at gmx.de)
Date: Wed, 05 Dec 2012 23:23:49 +0100
Subject: [SciPy-User] How to convolve with transfer function
Message-ID: <50BFC975.1090905@gmx.de>

Hi everyone. I want to do the following:
I have an image and want to apply the transfer function 
http://www.abload.de/img/tfbiif7.png to my image. However, in the 
article I am using it said the following:
"Due to the singularity in the log function at the origin, one cannot 
construct an analytic expression for the shape
of the Log-Gabor function in the spatial domain. Therefore, the filters 
are constructed in the frequency domain."
So it has to be used in the frequency domain. I first have to transform 
my image via FFT. For that, I found the function numpy.fft.rfft2 which 
does the transformation. However I'm at a loss on how to apply the 
transfer function to the array. I found the scipy.signal.convolve 
method, however that one seems to work with arrays. But I can't figure 
out how to convolve with this transfer function if it can't be defined 
as an array.
Would you please help me with my problem?

Cheers
Robert


From malandrea80 at gmail.com  Wed Dec  5 18:48:58 2012
From: malandrea80 at gmail.com (MalAndrea)
Date: Thu, 6 Dec 2012 00:48:58 +0100
Subject: [SciPy-User] Parallel Differential Evolution
In-Reply-To: <CAKuNVDb094_OgDC4GGLrASsB21zTiYA1d2cHjZqamgVKBtUnaQ@mail.gmail.com>
References: <CAKuNVDb094_OgDC4GGLrASsB21zTiYA1d2cHjZqamgVKBtUnaQ@mail.gmail.com>
Message-ID: <CAKuNVDbSkV4RynPgLBk3z_WMtdG_oWpLvyu80tbEa0bwpqLtpw@mail.gmail.com>

Hi,

I have followed Rob suggestions on this, i.e. to substute:

(1). *numpy.zeros(X)* instead of *flex.double(X, 0)*
(2). *1000*numpy.ones(X) *instead of *flex.double(X, 1000)*
(3). *numpy.min(X) *for *flex.min(X)*, etc. (mean, sum)
(4). *numpy.random.uniform(size=N)* for *flex.random_double(N) *

However, to get this work, I also had to modify the following:

- modification (4). only works when floats are meant to be used, I guess,
so in the only case: *rnd =
numpy.random.uniform(size=self.vector_length)* instead
of  *flex.random_double(N)*. In the other two cases it is *random_values =
numpy.random.random_integers(low=0.0, high=1.0, size=N)*  instead of  *
flex.random_double(N)*
- Also,* numpy.nanargmin* instead of *flex.min_index*
- Also, *numpy.argsort* instead of *flex.sort_permutation*
- Also, *.copy()* instead of *.deep_copy()*
- Finally, *numpy.random.seed(0)* instead of *flex.set_random_seed(0)*

Now, it works (at least for the Rosenbrock function).
Thank you,
Andrea

P.S. Sorry, this is the (old) message I referred to:

Hi,
>
> Maybe the following is a useful implementation for you:
> http://cci.lbl.gov/cctbx_sources/scitbx/differential_evolution.py
>
> According to this file you need only the scitbx library, which
> unfortunately looks non-trivial to install (it seems to be related to
> Boost). However, this looks like an old code (stdlib no longer exists,
> you'd just write "import random" now) and all the work done with
> arrays using the scitbx library (index of minimum value, mean value,
> etc.) can easily be done with the much more easily-installed numpy
> library from scipy.org. You probably just need that, a simple graphing
> package such as matplotlib, and a working environment such as Ipython.
> Binary windows installers are available for all these from a simple
> google search for "install X windows" where X is any of the above
> packages (they will be found at sourceforge.net). Don't forget to
> match the binary installer's version to your version of Python.
>
> So instead of importing flex you'd import numpy and then make the
> following substitutions (not tested):
>
> numpy.zeros(X) instead of flex.double(X, 0)
>
> 1000*numpy.ones(X) instead of flex.double(X, 1000)
>
> numpy.min(X) for flex.min(X), etc.
>
> numpy.random.uniform(size=N) for flex.random_double(N)
>
> If you get stuck post your code back here and someone can take a look.
> Hope this helps,
> Rob
>
> On Wed, Feb 24, 2010 at 6:02 PM, Debabrata Midya
> <Debabrata.Midya at services.nsw.gov.au <http://mail.scipy.org/mailman/listinfo/scipy-user>> wrote:
> >* Hi SciPy-Users,*>**>* Thanks in advance.*>**>* I am Deb, Statistician at NSW Department of Services, Technology and*>* Administration, Sydney, Australia.*>**>* I am new to this user group as well as to Python and Parallel Python.*>**>* I am interested to use Differential Evolution (DE) using Parallel Python on*>* Windows XP. I have some experience in DE using GNU GCC compiler.*>**>* Is there any software for it? If yes, anyone can assist me what are the*>* softwares I need to install and their locations please.*>**>* Once again, thank you very much for the time you have given.*>**>* I am looking forward for your reply.*>**>* Regards,*>**>* Deb*>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121206/1a9643c2/attachment.html>

From josef.pktd at gmail.com  Wed Dec  5 20:19:49 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 5 Dec 2012 20:19:49 -0500
Subject: [SciPy-User] How to convolve with transfer function
In-Reply-To: <50BFC975.1090905@gmx.de>
References: <50BFC975.1090905@gmx.de>
Message-ID: <CAMMTP+CTqfpsnN433bHMWd8ufBk==csDk4FCjqYOkNFwE0TNHQ@mail.gmail.com>

On Wed, Dec 5, 2012 at 5:23 PM,  <rmain at gmx.de> wrote:
> Hi everyone. I want to do the following:
> I have an image and want to apply the transfer function
> http://www.abload.de/img/tfbiif7.png to my image. However, in the
> article I am using it said the following:
> "Due to the singularity in the log function at the origin, one cannot
> construct an analytic expression for the shape
> of the Log-Gabor function in the spatial domain. Therefore, the filters
> are constructed in the frequency domain."
> So it has to be used in the frequency domain. I first have to transform
> my image via FFT. For that, I found the function numpy.fft.rfft2 which
> does the transformation. However I'm at a loss on how to apply the
> transfer function to the array. I found the scipy.signal.convolve
> method, however that one seems to work with arrays. But I can't figure
> out how to convolve with this transfer function if it can't be defined
> as an array.
> Would you please help me with my problem?

Hopefully there will be a more specific answer.

What I did in a similar case (but not with images) is to take fftconvolve
https://github.com/scipy/scipy/blob/v0.11.0/scipy/signal/signaltools.py#L134
replace
IN1 *= fftn(in2, fsize)
with
IN1 *= G(w)

where G(w) is the Fourier transform of the transfer function that you
have given in your formula.

The problem that I usually had was figuring out what the frequencies,
w, are supposed to be, and what shape G(w) is supposed to have.

(recipe how to cannibalize scipy source, not for your specific problem)

Josef

>
> Cheers
> Robert
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From paul.kienzle at nist.gov  Wed Dec  5 11:31:03 2012
From: paul.kienzle at nist.gov (Paul Kienzle)
Date: Wed, 5 Dec 2012 11:31:03 -0500
Subject: [SciPy-User] Need help to use rv_continuous.fit
In-Reply-To: <50BF6420.20307@isc.ac.uk>
References: 3d375d731003220750i8801ed7o3dfaaa2990f7b057@mail.gmail.com
	<50BF6420.20307@isc.ac.uk>
Message-ID: <D5A2B1DE-DF3B-47D1-A806-39F71C7C5341@nist.gov>


On Dec 5, 2012, at 10:11 AM, Sepideh Rastin <sepideh at isc.ac.uk> wrote:

> Hi there,
> I have histograms that will form the likelihood function and need to 
> find the best normal distributions.
> I would like to know if it would be possible to create my input array to 
> rv_continuous.fit using the data of my histograms.

If you have the raw data, you can just use mean() and std() functions.

E.g.,

from pylab import *

n = 1000
mu = 2
sigma = 0.3

def G(x,m,s): return exp(-((x-m)/s)**2/2)/sqrt(2*pi*s**2)

# fake data
s = randn(n)*sigma + mu

# estimates
mu_hat, sigma_hat = mean(s), std(s)

# plot
t = linspace(mu-3*sigma,mu+3*sigma,200)
bins = linspace(t[0],t[-1],21)
step = bins[1]-bins[0]
hist(s,bins=bins)
plot(t,G(t,mu_hat,sigma_hat)*n*step,'g',hold=True)
show()

  - Paul

Paul Kienzle
pkienzle at nist.gov


From monocongo at gmail.com  Thu Dec  6 15:10:26 2012
From: monocongo at gmail.com (James Adams)
Date: Thu, 6 Dec 2012 15:10:26 -0500
Subject: [SciPy-User] Hydroclimpy: unable to find sqlite module
Message-ID: <CACafSVn3hUqTreTNo-56KH2gP-J7c04QOe_nRO7vyBGGya=iMg@mail.gmail.com>

I have installed the scikits.hydroclimpy package but when I try to
verify that it's installed correctly via an import command within the
python interpreter I get an error message telling me that the sqlite
module is not present:

$ python
Python 2.7.3 (default, Dec  6 2012, 14:20:34)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import scikits.hydroclimpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/scikits.hydroclimpy-0.67.1-py2.7-linux-x86_64.egg/scikits/hydroclimpy/__init__.py",
line 19, in <module>
    import io
  File "/usr/local/lib/python2.7/site-packages/scikits.hydroclimpy-0.67.1-py2.7-linux-x86_64.egg/scikits/hydroclimpy/io/__init__.py",
line 7, in <module>
    import sqlite
ImportError: No module named sqlite


I do however have the sqlite3 module installed.

Can anyone suggest a work around for this?

--James


From cgohlke at uci.edu  Thu Dec  6 15:34:49 2012
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Thu, 06 Dec 2012 12:34:49 -0800
Subject: [SciPy-User] Hydroclimpy: unable to find sqlite module
In-Reply-To: <CACafSVn3hUqTreTNo-56KH2gP-J7c04QOe_nRO7vyBGGya=iMg@mail.gmail.com>
References: <CACafSVn3hUqTreTNo-56KH2gP-J7c04QOe_nRO7vyBGGya=iMg@mail.gmail.com>
Message-ID: <50C10169.6000200@uci.edu>

On 12/6/2012 12:10 PM, James Adams wrote:
> I have installed the scikits.hydroclimpy package but when I try to
> verify that it's installed correctly via an import command within the
> python interpreter I get an error message telling me that the sqlite
> module is not present:
>
> $ python
> Python 2.7.3 (default, Dec  6 2012, 14:20:34)
> [GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import scikits.hydroclimpy
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "/usr/local/lib/python2.7/site-packages/scikits.hydroclimpy-0.67.1-py2.7-linux-x86_64.egg/scikits/hydroclimpy/__init__.py",
> line 19, in <module>
>      import io
>    File "/usr/local/lib/python2.7/site-packages/scikits.hydroclimpy-0.67.1-py2.7-linux-x86_64.egg/scikits/hydroclimpy/io/__init__.py",
> line 7, in <module>
>      import sqlite
> ImportError: No module named sqlite
>
>
> I do however have the sqlite3 module installed.
>
> Can anyone suggest a work around for this?
>
> --James


sqlite.py is missing in trunk. You could try to copy the file from the 
pierregm branch

<http://projects.scipy.org/scikits/browser/branches/pierregm/hydroclimpy/scikits/hydroclimpy/io/sqlite.py>

Christoph


From monocongo at gmail.com  Thu Dec  6 16:03:24 2012
From: monocongo at gmail.com (James Adams)
Date: Thu, 6 Dec 2012 16:03:24 -0500
Subject: [SciPy-User] Hydroclimpy: unable to find sqlite module
Message-ID: <CACafSV=xbKTPWufOhaoXnSx5Eq0JDfpQBsOPn_e96MOF8VyZrg@mail.gmail.com>

Copying the sqlite.py file you referenced into the io directory and
then rebuilding/reinstalling hydroclimpy did the trick.  Thanks!

--James


From ralf.gommers at gmail.com  Sat Dec  8 05:36:44 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 8 Dec 2012 11:36:44 +0100
Subject: [SciPy-User] distributions - who got the most ?
In-Reply-To: <CAMMTP+CrGZPNoJobc1Bu5VqP_EmbRD-iL9WXGKOy+6mfCEGzVQ@mail.gmail.com>
References: <CAMMTP+BLWAjpbfbQ78TdP4WgnMoXAS3668WGDyY7xYx1U+8_fQ@mail.gmail.com>
	<CABL7CQjbDD06buRaS67iAbi0=khzEvxDuyt1_xFjgUvAe2pbgA@mail.gmail.com>
	<CAMMTP+CrGZPNoJobc1Bu5VqP_EmbRD-iL9WXGKOy+6mfCEGzVQ@mail.gmail.com>
Message-ID: <CABL7CQhELcZGSWnggpDCpMC3OeCo6Qa=_PY1wBJLVSb6NOEo9A@mail.gmail.com>

On Wed, Dec 5, 2012 at 2:15 AM, <josef.pktd at gmail.com> wrote:

> On Tue, Dec 4, 2012 at 4:01 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >
> >
> >
> > On Tue, Dec 4, 2012 at 4:30 AM, <josef.pktd at gmail.com> wrote:
> >>
> >> scipy.stats has more than 90 distributions.
> >>
> >> Do we want to increase it by almost a factor of 10? :)
> >>
> >> While looking for the cdf of a distribution, I found this :
> >>
> >>
> http://www.mathworks.com/matlabcentral/fileexchange/35008-generation-of-random-variates
> >>
> >> He collected 870 distributions (under BSD license). Includes generic
> >> random number generation.
> >>
> >> Even though there are some variations of distributions counted
> >> separately, given my quick browsing this looks impressive and a good
> >> source for code and references.
> >> Coding style is not great but it's 10 years or so of collecting
> >> distributions.
> >
> >
> > Adding a lot of distributions sounds fine to me. That many distributions
> > would need to go into a separate namespace.
> >
> > Any additions should be complete though (the Matlab code only has
> pdf/cdf)
> > and well tested. The Matlab code doesn't look all that useful except for
> the
> > references ("coding style is not great" is really too kind). I also don't
> > trust the BSD license that's put on it, many files have different author
> > names in them with no mention of where they came from.
>
> The matlab code includes several "special" functions that look mostly
> copied from other authors.
> This would need checking, but I doubt we need many of those since we
> have scipy.special.
> We are missing some special functions for distributions, but I didn't
> check whether he has any of those.
> The pdfs, and the cdfs when available, look like they were implemented
> by the author, at least it looks that way for the small sample that I
> checked.
> (code quality varies a lot, but many distributions are vectorized or
> can be easily vectorized from his code.
>
> Given the pdf, the rest could all be derived generically. But it won't
> be efficient.
>

True, but that doesn't feel quite right. Tickets are being opened regularly
about precision issues due to using the generic methods. Same for speed,
but that's perhaps less critical. Generic methods are often not good enough.


> Also, I just saw that sympy could become useful to derive extra properties
> http://matthewrocklin.com/blog/work/2012/12/03/Characteristic-Functions/
> sympy.stats also works based only on the pdf (from what I have seen).
>
> I'm a bit skeptical about the number of distributions that are
> actually generally useful and not just used once in a journal article.
>  My impression from tracking several statistics journals is that there
> are at least 10 new distributions each year.
>
> As an example, he has a long list of poisson mixture distributions
> that I never heard of except for negative binomial. They might be
> useful in some cases, but a more general class might cover it better.
> >From a brief look at his reference
>
> http://scholar.google.com/scholar?cluster=6061641765696455790&hl=en&as_sdt=0,5&as_vis=1
> I think it might not be necessary to implement all details for 5 or
> more distributions separately.
> According to Google the paper has only 4 citations.  see also 1)
>
> But there are a lot of distributions, or classes/categories of
> distributions that scipy is missing, and are for example available in
> R, but in R they are spread out over many packages.
>

Keeping a list of those in a ticket could be useful.

Ralf


>
> Josef
>
> 1) another reference for poisson mixtures (technical, not a quick
> read, but a funny table)
>
> Karlis, D. and Xekalaki, E. (2005), Mixed Poisson Distributions.
> International Statistical Review, 73: 35?58. doi:
> 10.1111/j.1751-5823.2005.tb00250.x
>
> http://scholar.google.com/scholar?cluster=4455890634693542956&hl=en&as_sdt=2005&sciodt=0,5
>
> --------------------------
> Table 1
> Some mixed Poisson distributions.
> Mixed Poisson Distribution Mixing Distribution A Key Reference
> Negative Binomial Gamma Greenwood & Yule (1920)
> Geometric Exponential Johnson et al. (1992)
> Poisson-Linear Exponential Family Linear Exponential Family Sankaran (1969)
> Poisson?Lindley Lindley Sankaran (1970)
> Poisson-Linear Exponential Linear Exponential Kling & Goovaerts (1993)
> Poisson-Lognormal Lognormal Bulmer (1974)
> Poisson-Confluent Hypergeometric Series Confluent Hypergeometric
> Series Bhattacharya (1966)
> Poisson-Generalized Inverse Gaussian Generalized Inverse Gaussian Sichel
> (1974)
> Sichel Inverse Gaussian Sichel (1975)
> Poisson-Inverse Gamma Inverse Gamma Willmot (1993)
> Poisson-Truncated Normal Truncated Normal Patil (1964)
> Generalized Waring Gamma Product Ratio Irwin (1975)
> Simple Waring Exponential   Beta Pielou (1962)
> Yule Beta with Specific Parameter Values Simon (1955)
> Poisson-Generalized Pareto Generalized Pareto Kempton (1975)
> Poisson-Beta I Beta Type I Holla & Bhattacharya (1965)
> Poisson-Beta II Beta Type II Gurland (1958)
> Poisson-Truncated Beta II Truncated Beta Type II Willmot (1986)
> Poisson-Uniform Uniform Bhattacharya (1966)
> Poisson-Truncated Gamma Truncated Gamma Willmot (1993)
> Poisson-Generalized Gamma Generalized Gamma Albrecht (1984)
> Dellaporte Shifted Gamma Ruohonen (1988)
> Poisson-Modified Bessel of the 3rd Kind Modified Bessel of the 3rd
> Kind Ong & Muthaloo (1995)
> Poisson?Pareto Pareto Willmot (1993)
> Poisson-Shifted Pareto Shifted Pareto Willmot (1993)
> Poisson?Pearson Family Pearson?s Family of Distributions Albrecht (1982)
> Poisson-Log-Student Log-Student Gaver & O?Muircheartaigh (1987)
> Poisson-Power Function Power Function Distribution Rai (1971)
> Poisson?Lomax Lomax Al-Awadhi & Ghitany (2001)
> Poisson-Power Variance Power Variance Family Hougaard et al. (1997)
> Neyman Poisson Douglas (1980)
> Other Discrete Distributions Johnson et al. (1992)
> -------------------------------------------------------------------
>
> >
> > Ralf
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121208/e9ec6e37/attachment.html>

From fgnu32 at yahoo.com  Sun Dec  9 23:01:30 2012
From: fgnu32 at yahoo.com (Fg Nu)
Date: Sun, 9 Dec 2012 20:01:30 -0800 (PST)
Subject: [SciPy-User] Logistic regression using SciPy
Message-ID: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>



I am trying to code up logistic regression in Python using the SciPy "fmin_bfgs" function, but am running into some issues. I wrote functions for the logistic (sigmoid) transformation ?function, and the cost function, and those work fine (I have used the optimized values of the parameter vector found via canned software to test the functions, and those match up). I am not that sure of my implementation of the gradient function, but it looks reasonable.?

Here is the code:

#==================================================

? ? # purpose: logistic regression?
? ? import numpy as np
? ? import scipy as sp
? ? import scipy.optimize
? ??
? ? import matplotlib as mpl
? ? import os
? ??
? ? # prepare the data
? ? data = np.loadtxt('data.csv', delimiter=',', skiprows=1)
? ? vY = data[:, 0]
? ? mX = data[:, 1:]
? ? intercept = np.ones(mX.shape[0]).reshape(mX.shape[0], 1)
? ? mX = np.concatenate((intercept, mX), axis = 1)
? ? iK = mX.shape[1]
? ? iN = mX.shape[0]
? ? ?
? ? # logistic transformation
? ? def logit(mX, vBeta):
? ? ? ? return((1/(1.0 + np.exp(-np.dot(mX, vBeta)))))
? ??
? ? # test function call
? ? vBeta0 = np.array([-.10296645, -.0332327, -.01209484, .44626211, .92554137, .53973828,?
? ? ? ? 1.7993371, .7148045 ?])
? ? logit(mX, vBeta0)
? ??
? ? # cost function
? ? def logLikelihoodLogit(vBeta, mX, vY):
? ? ? ? return(-(np.sum(vY*np.log(logit(mX, vBeta)) + (1-vY)*(np.log(1-logit(mX, vBeta))))))
? ? logLikelihoodLogit(vBeta0, mX, vY) # test function call
? ??
? ? # gradient function
? ? def likelihoodScore(vBeta, mX, vY):
? ? ? ? return(np.dot(mX.T,?
? ? ? ? ? ? ? ? ? ? ? ((np.dot(mX, vBeta) - vY)/
? ? ? ? ? ? ? ? ? ? ? ?np.dot(mX, vBeta)).reshape(iN, 1)).reshape(iK, 1))
? ??
? ? likelihoodScore(vBeta0, mX, vY).shape # test function call
? ? ?
? ? # optimize the function (without gradient)
? ? optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1.8, .71]),?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? args = (mX, vY), gtol = 1e-3)
? ??
? ? # optimize the function (with gradient)
? ? optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1.8, .71]), fprime = likelihoodScore,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? args = (mX, vY), gtol = 1e-3)
#=====================================================

* The first optimization (without gradient) ?ends with a whole lot of stuff about division by zero.

* The second optimization (with gradient) ends with a matrices not aligned error, which probably means I have got the way the gradient is to be returned wrong.

Any help with this is appreciated. If anyone wants to try this, the data is included below.

? ? low,age,lwt,race,smoke,ptl,ht,ui
? ? 0,19,182,2,0,0,0,1
? ? 0,33,155,3,0,0,0,0
? ? 0,20,105,1,1,0,0,0
? ? 0,21,108,1,1,0,0,1
? ? 0,18,107,1,1,0,0,1
? ? 0,21,124,3,0,0,0,0
? ? 0,22,118,1,0,0,0,0
? ? 0,17,103,3,0,0,0,0
? ? 0,29,123,1,1,0,0,0
? ? 0,26,113,1,1,0,0,0
? ? 0,19,95,3,0,0,0,0
? ? 0,19,150,3,0,0,0,0
? ? 0,22,95,3,0,0,1,0
? ? 0,30,107,3,0,1,0,1
? ? 0,18,100,1,1,0,0,0
? ? 0,18,100,1,1,0,0,0
? ? 0,15,98,2,0,0,0,0
? ? 0,25,118,1,1,0,0,0
? ? 0,20,120,3,0,0,0,1
? ? 0,28,120,1,1,0,0,0
? ? 0,32,121,3,0,0,0,0
? ? 0,31,100,1,0,0,0,1
? ? 0,36,202,1,0,0,0,0
? ? 0,28,120,3,0,0,0,0
? ? 0,25,120,3,0,0,0,1
? ? 0,28,167,1,0,0,0,0
? ? 0,17,122,1,1,0,0,0
? ? 0,29,150,1,0,0,0,0
? ? 0,26,168,2,1,0,0,0
? ? 0,17,113,2,0,0,0,0
? ? 0,17,113,2,0,0,0,0
? ? 0,24,90,1,1,1,0,0
? ? 0,35,121,2,1,1,0,0
? ? 0,25,155,1,0,0,0,0
? ? 0,25,125,2,0,0,0,0
? ? 0,29,140,1,1,0,0,0
? ? 0,19,138,1,1,0,0,0
? ? 0,27,124,1,1,0,0,0
? ? 0,31,215,1,1,0,0,0
? ? 0,33,109,1,1,0,0,0
? ? 0,21,185,2,1,0,0,0
? ? 0,19,189,1,0,0,0,0
? ? 0,23,130,2,0,0,0,0
? ? 0,21,160,1,0,0,0,0
? ? 0,18,90,1,1,0,0,1
? ? 0,18,90,1,1,0,0,1
? ? 0,32,132,1,0,0,0,0
? ? 0,19,132,3,0,0,0,0
? ? 0,24,115,1,0,0,0,0
? ? 0,22,85,3,1,0,0,0
? ? 0,22,120,1,0,0,1,0
? ? 0,23,128,3,0,0,0,0
? ? 0,22,130,1,1,0,0,0
? ? 0,30,95,1,1,0,0,0
? ? 0,19,115,3,0,0,0,0
? ? 0,16,110,3,0,0,0,0
? ? 0,21,110,3,1,0,0,1
? ? 0,30,153,3,0,0,0,0
? ? 0,20,103,3,0,0,0,0
? ? 0,17,119,3,0,0,0,0
? ? 0,17,119,3,0,0,0,0
? ? 0,23,119,3,0,0,0,0
? ? 0,24,110,3,0,0,0,0
? ? 0,28,140,1,0,0,0,0
? ? 0,26,133,3,1,2,0,0
? ? 0,20,169,3,0,1,0,1
? ? 0,24,115,3,0,0,0,0
? ? 0,28,250,3,1,0,0,0
? ? 0,20,141,1,0,2,0,1
? ? 0,22,158,2,0,1,0,0
? ? 0,22,112,1,1,2,0,0
? ? 0,31,150,3,1,0,0,0
? ? 0,23,115,3,1,0,0,0
? ? 0,16,112,2,0,0,0,0
? ? 0,16,135,1,1,0,0,0
? ? 0,18,229,2,0,0,0,0
? ? 0,25,140,1,0,0,0,0
? ? 0,32,134,1,1,1,0,0
? ? 0,20,121,2,1,0,0,0
? ? 0,23,190,1,0,0,0,0
? ? 0,22,131,1,0,0,0,0
? ? 0,32,170,1,0,0,0,0
? ? 0,30,110,3,0,0,0,0
? ? 0,20,127,3,0,0,0,0
? ? 0,23,123,3,0,0,0,0
? ? 0,17,120,3,1,0,0,0
? ? 0,19,105,3,0,0,0,0
? ? 0,23,130,1,0,0,0,0
? ? 0,36,175,1,0,0,0,0
? ? 0,22,125,1,0,0,0,0
? ? 0,24,133,1,0,0,0,0
? ? 0,21,134,3,0,0,0,0
? ? 0,19,235,1,1,0,1,0
? ? 0,25,95,1,1,3,0,1
? ? 0,16,135,1,1,0,0,0
? ? 0,29,135,1,0,0,0,0
? ? 0,29,154,1,0,0,0,0
? ? 0,19,147,1,1,0,0,0
? ? 0,19,147,1,1,0,0,0
? ? 0,30,137,1,0,0,0,0
? ? 0,24,110,1,0,0,0,0
? ? 0,19,184,1,1,0,1,0
? ? 0,24,110,3,0,1,0,0
? ? 0,23,110,1,0,0,0,0
? ? 0,20,120,3,0,0,0,0
? ? 0,25,241,2,0,0,1,0
? ? 0,30,112,1,0,0,0,0
? ? 0,22,169,1,0,0,0,0
? ? 0,18,120,1,1,0,0,0
? ? 0,16,170,2,0,0,0,0
? ? 0,32,186,1,0,0,0,0
? ? 0,18,120,3,0,0,0,0
? ? 0,29,130,1,1,0,0,0
? ? 0,33,117,1,0,0,0,1
? ? 0,20,170,1,1,0,0,0
? ? 0,28,134,3,0,0,0,0
? ? 0,14,135,1,0,0,0,0
? ? 0,28,130,3,0,0,0,0
? ? 0,25,120,1,0,0,0,0
? ? 0,16,95,3,0,0,0,0
? ? 0,20,158,1,0,0,0,0
? ? 0,26,160,3,0,0,0,0
? ? 0,21,115,1,0,0,0,0
? ? 0,22,129,1,0,0,0,0
? ? 0,25,130,1,0,0,0,0
? ? 0,31,120,1,0,0,0,0
? ? 0,35,170,1,0,1,0,0
? ? 0,19,120,1,1,0,0,0
? ? 0,24,116,1,0,0,0,0
? ? 0,45,123,1,0,0,0,0
? ? 1,28,120,3,1,1,0,1
? ? 1,29,130,1,0,0,0,1
? ? 1,34,187,2,1,0,1,0
? ? 1,25,105,3,0,1,1,0
? ? 1,25,85,3,0,0,0,1
? ? 1,27,150,3,0,0,0,0
? ? 1,23,97,3,0,0,0,1
? ? 1,24,128,2,0,1,0,0
? ? 1,24,132,3,0,0,1,0
? ? 1,21,165,1,1,0,1,0
? ? 1,32,105,1,1,0,0,0
? ? 1,19,91,1,1,2,0,1
? ? 1,25,115,3,0,0,0,0
? ? 1,16,130,3,0,0,0,0
? ? 1,25,92,1,1,0,0,0
? ? 1,20,150,1,1,0,0,0
? ? 1,21,200,2,0,0,0,1
? ? 1,24,155,1,1,1,0,0
? ? 1,21,103,3,0,0,0,0
? ? 1,20,125,3,0,0,0,1
? ? 1,25,89,3,0,2,0,0
? ? 1,19,102,1,0,0,0,0
? ? 1,19,112,1,1,0,0,1
? ? 1,26,117,1,1,1,0,0
? ? 1,24,138,1,0,0,0,0
? ? 1,17,130,3,1,1,0,1
? ? 1,20,120,2,1,0,0,0
? ? 1,22,130,1,1,1,0,1
? ? 1,27,130,2,0,0,0,1
? ? 1,20,80,3,1,0,0,1
? ? 1,17,110,1,1,0,0,0
? ? 1,25,105,3,0,1,0,0
? ? 1,20,109,3,0,0,0,0
? ? 1,18,148,3,0,0,0,0
? ? 1,18,110,2,1,1,0,0
? ? 1,20,121,1,1,1,0,1
? ? 1,21,100,3,0,1,0,0
? ? 1,26,96,3,0,0,0,0
? ? 1,31,102,1,1,1,0,0
? ? 1,15,110,1,0,0,0,0
? ? 1,23,187,2,1,0,0,0
? ? 1,20,122,2,1,0,0,0
? ? 1,24,105,2,1,0,0,0
? ? 1,15,115,3,0,0,0,1
? ? 1,23,120,3,0,0,0,0
? ? 1,30,142,1,1,1,0,0
? ? 1,22,130,1,1,0,0,0
? ? 1,17,120,1,1,0,0,0
? ? 1,23,110,1,1,1,0,0
? ? 1,17,120,2,0,0,0,0
? ? 1,26,154,3,0,1,1,0
? ? 1,20,106,3,0,0,0,0
? ? 1,26,190,1,1,0,0,0
? ? 1,14,101,3,1,1,0,0
? ? 1,28,95,1,1,0,0,0
? ? 1,14,100,3,0,0,0,0
? ? 1,23,94,3,1,0,0,0
? ? 1,17,142,2,0,0,1,0
? ? 1,21,130,1,1,0,1,0

Thanks.



From d.warde.farley at gmail.com  Sun Dec  9 23:22:35 2012
From: d.warde.farley at gmail.com (David Warde-Farley)
Date: Sun, 9 Dec 2012 23:22:35 -0500
Subject: [SciPy-User] Logistic regression using SciPy
In-Reply-To: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
References: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
Message-ID: <CALc6Xo6A-RqpfFArsW5oueFa53UM=b1E8vCwPubfcdtPvz6n+A@mail.gmail.com>

First, the way you've written the log likelihood is numerically
unstable. Consider simplifying the expression (using logarithm laws
and breaking apart logistic function) and using the log1p function
where appropriate.

Second, the optimization problem is going to be extremely ill
conditioned given the very different scales of your different
predictors. You should probably mean-center and divide by the standard
deviation.

Third, there's a check_grad function in scipy.optimize that can be
used to troubleshoot gradient issues.

Fourth, there's a pre-rolled of this in scikit-learn that will
probably be a good deal faster (it wraps a C library) and certainly
better tested than home-rolling it.

David

On Sun, Dec 9, 2012 at 11:01 PM, Fg Nu <fgnu32 at yahoo.com> wrote:
>
>
> I am trying to code up logistic regression in Python using the SciPy "fmin_bfgs" function, but am running into some issues. I wrote functions for the logistic (sigmoid) transformation  function, and the cost function, and those work fine (I have used the optimized values of the parameter vector found via canned software to test the functions, and those match up). I am not that sure of my implementation of the gradient function, but it looks reasonable.
>
> Here is the code:
>
> #==================================================
>
>     # purpose: logistic regression
>     import numpy as np
>     import scipy as sp
>     import scipy.optimize
>
>     import matplotlib as mpl
>     import os
>
>     # prepare the data
>     data = np.loadtxt('data.csv', delimiter=',', skiprows=1)
>     vY = data[:, 0]
>     mX = data[:, 1:]
>     intercept = np.ones(mX.shape[0]).reshape(mX.shape[0], 1)
>     mX = np.concatenate((intercept, mX), axis = 1)
>     iK = mX.shape[1]
>     iN = mX.shape[0]
>
>     # logistic transformation
>     def logit(mX, vBeta):
>         return((1/(1.0 + np.exp(-np.dot(mX, vBeta)))))
>
>     # test function call
>     vBeta0 = np.array([-.10296645, -.0332327, -.01209484, .44626211, .92554137, .53973828,
>         1.7993371, .7148045  ])
>     logit(mX, vBeta0)
>
>     # cost function
>     def logLikelihoodLogit(vBeta, mX, vY):
>         return(-(np.sum(vY*np.log(logit(mX, vBeta)) + (1-vY)*(np.log(1-logit(mX, vBeta))))))
>     logLikelihoodLogit(vBeta0, mX, vY) # test function call
>
>     # gradient function
>     def likelihoodScore(vBeta, mX, vY):
>         return(np.dot(mX.T,
>                       ((np.dot(mX, vBeta) - vY)/
>                        np.dot(mX, vBeta)).reshape(iN, 1)).reshape(iK, 1))
>
>     likelihoodScore(vBeta0, mX, vY).shape # test function call
>
>     # optimize the function (without gradient)
>     optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,
>                                       x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
>                                                 1.8, .71]),
>                                       args = (mX, vY), gtol = 1e-3)
>
>     # optimize the function (with gradient)
>     optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,
>                                       x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
>                                                 1.8, .71]), fprime = likelihoodScore,
>                                       args = (mX, vY), gtol = 1e-3)
> #=====================================================
>
> * The first optimization (without gradient)  ends with a whole lot of stuff about division by zero.
>
> * The second optimization (with gradient) ends with a matrices not aligned error, which probably means I have got the way the gradient is to be returned wrong.
>
> Any help with this is appreciated. If anyone wants to try this, the data is included below.
>
>     low,age,lwt,race,smoke,ptl,ht,ui
>     0,19,182,2,0,0,0,1
>     0,33,155,3,0,0,0,0
>     0,20,105,1,1,0,0,0
>     0,21,108,1,1,0,0,1
>     0,18,107,1,1,0,0,1
>     0,21,124,3,0,0,0,0
>     0,22,118,1,0,0,0,0
>     0,17,103,3,0,0,0,0
>     0,29,123,1,1,0,0,0
>     0,26,113,1,1,0,0,0
>     0,19,95,3,0,0,0,0
>     0,19,150,3,0,0,0,0
>     0,22,95,3,0,0,1,0
>     0,30,107,3,0,1,0,1
>     0,18,100,1,1,0,0,0
>     0,18,100,1,1,0,0,0
>     0,15,98,2,0,0,0,0
>     0,25,118,1,1,0,0,0
>     0,20,120,3,0,0,0,1
>     0,28,120,1,1,0,0,0
>     0,32,121,3,0,0,0,0
>     0,31,100,1,0,0,0,1
>     0,36,202,1,0,0,0,0
>     0,28,120,3,0,0,0,0
>     0,25,120,3,0,0,0,1
>     0,28,167,1,0,0,0,0
>     0,17,122,1,1,0,0,0
>     0,29,150,1,0,0,0,0
>     0,26,168,2,1,0,0,0
>     0,17,113,2,0,0,0,0
>     0,17,113,2,0,0,0,0
>     0,24,90,1,1,1,0,0
>     0,35,121,2,1,1,0,0
>     0,25,155,1,0,0,0,0
>     0,25,125,2,0,0,0,0
>     0,29,140,1,1,0,0,0
>     0,19,138,1,1,0,0,0
>     0,27,124,1,1,0,0,0
>     0,31,215,1,1,0,0,0
>     0,33,109,1,1,0,0,0
>     0,21,185,2,1,0,0,0
>     0,19,189,1,0,0,0,0
>     0,23,130,2,0,0,0,0
>     0,21,160,1,0,0,0,0
>     0,18,90,1,1,0,0,1
>     0,18,90,1,1,0,0,1
>     0,32,132,1,0,0,0,0
>     0,19,132,3,0,0,0,0
>     0,24,115,1,0,0,0,0
>     0,22,85,3,1,0,0,0
>     0,22,120,1,0,0,1,0
>     0,23,128,3,0,0,0,0
>     0,22,130,1,1,0,0,0
>     0,30,95,1,1,0,0,0
>     0,19,115,3,0,0,0,0
>     0,16,110,3,0,0,0,0
>     0,21,110,3,1,0,0,1
>     0,30,153,3,0,0,0,0
>     0,20,103,3,0,0,0,0
>     0,17,119,3,0,0,0,0
>     0,17,119,3,0,0,0,0
>     0,23,119,3,0,0,0,0
>     0,24,110,3,0,0,0,0
>     0,28,140,1,0,0,0,0
>     0,26,133,3,1,2,0,0
>     0,20,169,3,0,1,0,1
>     0,24,115,3,0,0,0,0
>     0,28,250,3,1,0,0,0
>     0,20,141,1,0,2,0,1
>     0,22,158,2,0,1,0,0
>     0,22,112,1,1,2,0,0
>     0,31,150,3,1,0,0,0
>     0,23,115,3,1,0,0,0
>     0,16,112,2,0,0,0,0
>     0,16,135,1,1,0,0,0
>     0,18,229,2,0,0,0,0
>     0,25,140,1,0,0,0,0
>     0,32,134,1,1,1,0,0
>     0,20,121,2,1,0,0,0
>     0,23,190,1,0,0,0,0
>     0,22,131,1,0,0,0,0
>     0,32,170,1,0,0,0,0
>     0,30,110,3,0,0,0,0
>     0,20,127,3,0,0,0,0
>     0,23,123,3,0,0,0,0
>     0,17,120,3,1,0,0,0
>     0,19,105,3,0,0,0,0
>     0,23,130,1,0,0,0,0
>     0,36,175,1,0,0,0,0
>     0,22,125,1,0,0,0,0
>     0,24,133,1,0,0,0,0
>     0,21,134,3,0,0,0,0
>     0,19,235,1,1,0,1,0
>     0,25,95,1,1,3,0,1
>     0,16,135,1,1,0,0,0
>     0,29,135,1,0,0,0,0
>     0,29,154,1,0,0,0,0
>     0,19,147,1,1,0,0,0
>     0,19,147,1,1,0,0,0
>     0,30,137,1,0,0,0,0
>     0,24,110,1,0,0,0,0
>     0,19,184,1,1,0,1,0
>     0,24,110,3,0,1,0,0
>     0,23,110,1,0,0,0,0
>     0,20,120,3,0,0,0,0
>     0,25,241,2,0,0,1,0
>     0,30,112,1,0,0,0,0
>     0,22,169,1,0,0,0,0
>     0,18,120,1,1,0,0,0
>     0,16,170,2,0,0,0,0
>     0,32,186,1,0,0,0,0
>     0,18,120,3,0,0,0,0
>     0,29,130,1,1,0,0,0
>     0,33,117,1,0,0,0,1
>     0,20,170,1,1,0,0,0
>     0,28,134,3,0,0,0,0
>     0,14,135,1,0,0,0,0
>     0,28,130,3,0,0,0,0
>     0,25,120,1,0,0,0,0
>     0,16,95,3,0,0,0,0
>     0,20,158,1,0,0,0,0
>     0,26,160,3,0,0,0,0
>     0,21,115,1,0,0,0,0
>     0,22,129,1,0,0,0,0
>     0,25,130,1,0,0,0,0
>     0,31,120,1,0,0,0,0
>     0,35,170,1,0,1,0,0
>     0,19,120,1,1,0,0,0
>     0,24,116,1,0,0,0,0
>     0,45,123,1,0,0,0,0
>     1,28,120,3,1,1,0,1
>     1,29,130,1,0,0,0,1
>     1,34,187,2,1,0,1,0
>     1,25,105,3,0,1,1,0
>     1,25,85,3,0,0,0,1
>     1,27,150,3,0,0,0,0
>     1,23,97,3,0,0,0,1
>     1,24,128,2,0,1,0,0
>     1,24,132,3,0,0,1,0
>     1,21,165,1,1,0,1,0
>     1,32,105,1,1,0,0,0
>     1,19,91,1,1,2,0,1
>     1,25,115,3,0,0,0,0
>     1,16,130,3,0,0,0,0
>     1,25,92,1,1,0,0,0
>     1,20,150,1,1,0,0,0
>     1,21,200,2,0,0,0,1
>     1,24,155,1,1,1,0,0
>     1,21,103,3,0,0,0,0
>     1,20,125,3,0,0,0,1
>     1,25,89,3,0,2,0,0
>     1,19,102,1,0,0,0,0
>     1,19,112,1,1,0,0,1
>     1,26,117,1,1,1,0,0
>     1,24,138,1,0,0,0,0
>     1,17,130,3,1,1,0,1
>     1,20,120,2,1,0,0,0
>     1,22,130,1,1,1,0,1
>     1,27,130,2,0,0,0,1
>     1,20,80,3,1,0,0,1
>     1,17,110,1,1,0,0,0
>     1,25,105,3,0,1,0,0
>     1,20,109,3,0,0,0,0
>     1,18,148,3,0,0,0,0
>     1,18,110,2,1,1,0,0
>     1,20,121,1,1,1,0,1
>     1,21,100,3,0,1,0,0
>     1,26,96,3,0,0,0,0
>     1,31,102,1,1,1,0,0
>     1,15,110,1,0,0,0,0
>     1,23,187,2,1,0,0,0
>     1,20,122,2,1,0,0,0
>     1,24,105,2,1,0,0,0
>     1,15,115,3,0,0,0,1
>     1,23,120,3,0,0,0,0
>     1,30,142,1,1,1,0,0
>     1,22,130,1,1,0,0,0
>     1,17,120,1,1,0,0,0
>     1,23,110,1,1,1,0,0
>     1,17,120,2,0,0,0,0
>     1,26,154,3,0,1,1,0
>     1,20,106,3,0,0,0,0
>     1,26,190,1,1,0,0,0
>     1,14,101,3,1,1,0,0
>     1,28,95,1,1,0,0,0
>     1,14,100,3,0,0,0,0
>     1,23,94,3,1,0,0,0
>     1,17,142,2,0,0,1,0
>     1,21,130,1,1,0,1,0
>
> Thanks.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From josef.pktd at gmail.com  Sun Dec  9 23:42:20 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 9 Dec 2012 23:42:20 -0500
Subject: [SciPy-User] Logistic regression using SciPy
In-Reply-To: <CALc6Xo6A-RqpfFArsW5oueFa53UM=b1E8vCwPubfcdtPvz6n+A@mail.gmail.com>
References: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
	<CALc6Xo6A-RqpfFArsW5oueFa53UM=b1E8vCwPubfcdtPvz6n+A@mail.gmail.com>
Message-ID: <CAMMTP+A_6zMTzsxacdfHYtG3jZUeRVOJZVuQVPs-+Tq4tRVcGA@mail.gmail.com>

On Sun, Dec 9, 2012 at 11:22 PM, David Warde-Farley
<d.warde.farley at gmail.com> wrote:
> First, the way you've written the log likelihood is numerically
> unstable. Consider simplifying the expression (using logarithm laws
> and breaking apart logistic function) and using the log1p function
> where appropriate.
>
> Second, the optimization problem is going to be extremely ill
> conditioned given the very different scales of your different
> predictors. You should probably mean-center and divide by the standard
> deviation.
>
> Third, there's a check_grad function in scipy.optimize that can be
> used to troubleshoot gradient issues.
>
> Fourth, there's a pre-rolled of this in scikit-learn that will
> probably be a good deal faster (it wraps a C library) and certainly
> better tested than home-rolling it.

or statsmodels, which uses analytical gradient and hessians, and gives
you additional result statistics and (statistical) tests.

Josef

>
> David
>
> On Sun, Dec 9, 2012 at 11:01 PM, Fg Nu <fgnu32 at yahoo.com> wrote:
>>
>>
>> I am trying to code up logistic regression in Python using the SciPy "fmin_bfgs" function, but am running into some issues. I wrote functions for the logistic (sigmoid) transformation  function, and the cost function, and those work fine (I have used the optimized values of the parameter vector found via canned software to test the functions, and those match up). I am not that sure of my implementation of the gradient function, but it looks reasonable.
>>
>> Here is the code:
>>
>> #==================================================
>>
>>     # purpose: logistic regression
>>     import numpy as np
>>     import scipy as sp
>>     import scipy.optimize
>>
>>     import matplotlib as mpl
>>     import os
>>
>>     # prepare the data
>>     data = np.loadtxt('data.csv', delimiter=',', skiprows=1)
>>     vY = data[:, 0]
>>     mX = data[:, 1:]
>>     intercept = np.ones(mX.shape[0]).reshape(mX.shape[0], 1)
>>     mX = np.concatenate((intercept, mX), axis = 1)
>>     iK = mX.shape[1]
>>     iN = mX.shape[0]
>>
>>     # logistic transformation
>>     def logit(mX, vBeta):
>>         return((1/(1.0 + np.exp(-np.dot(mX, vBeta)))))
>>
>>     # test function call
>>     vBeta0 = np.array([-.10296645, -.0332327, -.01209484, .44626211, .92554137, .53973828,
>>         1.7993371, .7148045  ])
>>     logit(mX, vBeta0)
>>
>>     # cost function
>>     def logLikelihoodLogit(vBeta, mX, vY):
>>         return(-(np.sum(vY*np.log(logit(mX, vBeta)) + (1-vY)*(np.log(1-logit(mX, vBeta))))))
>>     logLikelihoodLogit(vBeta0, mX, vY) # test function call
>>
>>     # gradient function
>>     def likelihoodScore(vBeta, mX, vY):
>>         return(np.dot(mX.T,
>>                       ((np.dot(mX, vBeta) - vY)/
>>                        np.dot(mX, vBeta)).reshape(iN, 1)).reshape(iK, 1))
>>
>>     likelihoodScore(vBeta0, mX, vY).shape # test function call
>>
>>     # optimize the function (without gradient)
>>     optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,
>>                                       x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
>>                                                 1.8, .71]),
>>                                       args = (mX, vY), gtol = 1e-3)
>>
>>     # optimize the function (with gradient)
>>     optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,
>>                                       x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
>>                                                 1.8, .71]), fprime = likelihoodScore,
>>                                       args = (mX, vY), gtol = 1e-3)
>> #=====================================================
>>
>> * The first optimization (without gradient)  ends with a whole lot of stuff about division by zero.
>>
>> * The second optimization (with gradient) ends with a matrices not aligned error, which probably means I have got the way the gradient is to be returned wrong.
>>
>> Any help with this is appreciated. If anyone wants to try this, the data is included below.
>>
>>     low,age,lwt,race,smoke,ptl,ht,ui
>>     0,19,182,2,0,0,0,1
>>     0,33,155,3,0,0,0,0
>>     0,20,105,1,1,0,0,0
>>     0,21,108,1,1,0,0,1
>>     0,18,107,1,1,0,0,1
>>     0,21,124,3,0,0,0,0
>>     0,22,118,1,0,0,0,0
>>     0,17,103,3,0,0,0,0
>>     0,29,123,1,1,0,0,0
>>     0,26,113,1,1,0,0,0
>>     0,19,95,3,0,0,0,0
>>     0,19,150,3,0,0,0,0
>>     0,22,95,3,0,0,1,0
>>     0,30,107,3,0,1,0,1
>>     0,18,100,1,1,0,0,0
>>     0,18,100,1,1,0,0,0
>>     0,15,98,2,0,0,0,0
>>     0,25,118,1,1,0,0,0
>>     0,20,120,3,0,0,0,1
>>     0,28,120,1,1,0,0,0
>>     0,32,121,3,0,0,0,0
>>     0,31,100,1,0,0,0,1
>>     0,36,202,1,0,0,0,0
>>     0,28,120,3,0,0,0,0
>>     0,25,120,3,0,0,0,1
>>     0,28,167,1,0,0,0,0
>>     0,17,122,1,1,0,0,0
>>     0,29,150,1,0,0,0,0
>>     0,26,168,2,1,0,0,0
>>     0,17,113,2,0,0,0,0
>>     0,17,113,2,0,0,0,0
>>     0,24,90,1,1,1,0,0
>>     0,35,121,2,1,1,0,0
>>     0,25,155,1,0,0,0,0
>>     0,25,125,2,0,0,0,0
>>     0,29,140,1,1,0,0,0
>>     0,19,138,1,1,0,0,0
>>     0,27,124,1,1,0,0,0
>>     0,31,215,1,1,0,0,0
>>     0,33,109,1,1,0,0,0
>>     0,21,185,2,1,0,0,0
>>     0,19,189,1,0,0,0,0
>>     0,23,130,2,0,0,0,0
>>     0,21,160,1,0,0,0,0
>>     0,18,90,1,1,0,0,1
>>     0,18,90,1,1,0,0,1
>>     0,32,132,1,0,0,0,0
>>     0,19,132,3,0,0,0,0
>>     0,24,115,1,0,0,0,0
>>     0,22,85,3,1,0,0,0
>>     0,22,120,1,0,0,1,0
>>     0,23,128,3,0,0,0,0
>>     0,22,130,1,1,0,0,0
>>     0,30,95,1,1,0,0,0
>>     0,19,115,3,0,0,0,0
>>     0,16,110,3,0,0,0,0
>>     0,21,110,3,1,0,0,1
>>     0,30,153,3,0,0,0,0
>>     0,20,103,3,0,0,0,0
>>     0,17,119,3,0,0,0,0
>>     0,17,119,3,0,0,0,0
>>     0,23,119,3,0,0,0,0
>>     0,24,110,3,0,0,0,0
>>     0,28,140,1,0,0,0,0
>>     0,26,133,3,1,2,0,0
>>     0,20,169,3,0,1,0,1
>>     0,24,115,3,0,0,0,0
>>     0,28,250,3,1,0,0,0
>>     0,20,141,1,0,2,0,1
>>     0,22,158,2,0,1,0,0
>>     0,22,112,1,1,2,0,0
>>     0,31,150,3,1,0,0,0
>>     0,23,115,3,1,0,0,0
>>     0,16,112,2,0,0,0,0
>>     0,16,135,1,1,0,0,0
>>     0,18,229,2,0,0,0,0
>>     0,25,140,1,0,0,0,0
>>     0,32,134,1,1,1,0,0
>>     0,20,121,2,1,0,0,0
>>     0,23,190,1,0,0,0,0
>>     0,22,131,1,0,0,0,0
>>     0,32,170,1,0,0,0,0
>>     0,30,110,3,0,0,0,0
>>     0,20,127,3,0,0,0,0
>>     0,23,123,3,0,0,0,0
>>     0,17,120,3,1,0,0,0
>>     0,19,105,3,0,0,0,0
>>     0,23,130,1,0,0,0,0
>>     0,36,175,1,0,0,0,0
>>     0,22,125,1,0,0,0,0
>>     0,24,133,1,0,0,0,0
>>     0,21,134,3,0,0,0,0
>>     0,19,235,1,1,0,1,0
>>     0,25,95,1,1,3,0,1
>>     0,16,135,1,1,0,0,0
>>     0,29,135,1,0,0,0,0
>>     0,29,154,1,0,0,0,0
>>     0,19,147,1,1,0,0,0
>>     0,19,147,1,1,0,0,0
>>     0,30,137,1,0,0,0,0
>>     0,24,110,1,0,0,0,0
>>     0,19,184,1,1,0,1,0
>>     0,24,110,3,0,1,0,0
>>     0,23,110,1,0,0,0,0
>>     0,20,120,3,0,0,0,0
>>     0,25,241,2,0,0,1,0
>>     0,30,112,1,0,0,0,0
>>     0,22,169,1,0,0,0,0
>>     0,18,120,1,1,0,0,0
>>     0,16,170,2,0,0,0,0
>>     0,32,186,1,0,0,0,0
>>     0,18,120,3,0,0,0,0
>>     0,29,130,1,1,0,0,0
>>     0,33,117,1,0,0,0,1
>>     0,20,170,1,1,0,0,0
>>     0,28,134,3,0,0,0,0
>>     0,14,135,1,0,0,0,0
>>     0,28,130,3,0,0,0,0
>>     0,25,120,1,0,0,0,0
>>     0,16,95,3,0,0,0,0
>>     0,20,158,1,0,0,0,0
>>     0,26,160,3,0,0,0,0
>>     0,21,115,1,0,0,0,0
>>     0,22,129,1,0,0,0,0
>>     0,25,130,1,0,0,0,0
>>     0,31,120,1,0,0,0,0
>>     0,35,170,1,0,1,0,0
>>     0,19,120,1,1,0,0,0
>>     0,24,116,1,0,0,0,0
>>     0,45,123,1,0,0,0,0
>>     1,28,120,3,1,1,0,1
>>     1,29,130,1,0,0,0,1
>>     1,34,187,2,1,0,1,0
>>     1,25,105,3,0,1,1,0
>>     1,25,85,3,0,0,0,1
>>     1,27,150,3,0,0,0,0
>>     1,23,97,3,0,0,0,1
>>     1,24,128,2,0,1,0,0
>>     1,24,132,3,0,0,1,0
>>     1,21,165,1,1,0,1,0
>>     1,32,105,1,1,0,0,0
>>     1,19,91,1,1,2,0,1
>>     1,25,115,3,0,0,0,0
>>     1,16,130,3,0,0,0,0
>>     1,25,92,1,1,0,0,0
>>     1,20,150,1,1,0,0,0
>>     1,21,200,2,0,0,0,1
>>     1,24,155,1,1,1,0,0
>>     1,21,103,3,0,0,0,0
>>     1,20,125,3,0,0,0,1
>>     1,25,89,3,0,2,0,0
>>     1,19,102,1,0,0,0,0
>>     1,19,112,1,1,0,0,1
>>     1,26,117,1,1,1,0,0
>>     1,24,138,1,0,0,0,0
>>     1,17,130,3,1,1,0,1
>>     1,20,120,2,1,0,0,0
>>     1,22,130,1,1,1,0,1
>>     1,27,130,2,0,0,0,1
>>     1,20,80,3,1,0,0,1
>>     1,17,110,1,1,0,0,0
>>     1,25,105,3,0,1,0,0
>>     1,20,109,3,0,0,0,0
>>     1,18,148,3,0,0,0,0
>>     1,18,110,2,1,1,0,0
>>     1,20,121,1,1,1,0,1
>>     1,21,100,3,0,1,0,0
>>     1,26,96,3,0,0,0,0
>>     1,31,102,1,1,1,0,0
>>     1,15,110,1,0,0,0,0
>>     1,23,187,2,1,0,0,0
>>     1,20,122,2,1,0,0,0
>>     1,24,105,2,1,0,0,0
>>     1,15,115,3,0,0,0,1
>>     1,23,120,3,0,0,0,0
>>     1,30,142,1,1,1,0,0
>>     1,22,130,1,1,0,0,0
>>     1,17,120,1,1,0,0,0
>>     1,23,110,1,1,1,0,0
>>     1,17,120,2,0,0,0,0
>>     1,26,154,3,0,1,1,0
>>     1,20,106,3,0,0,0,0
>>     1,26,190,1,1,0,0,0
>>     1,14,101,3,1,1,0,0
>>     1,28,95,1,1,0,0,0
>>     1,14,100,3,0,0,0,0
>>     1,23,94,3,1,0,0,0
>>     1,17,142,2,0,0,1,0
>>     1,21,130,1,1,0,1,0
>>
>> Thanks.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From fgnu32 at yahoo.com  Mon Dec 10 00:00:39 2012
From: fgnu32 at yahoo.com (Fg Nu)
Date: Sun, 9 Dec 2012 21:00:39 -0800 (PST)
Subject: [SciPy-User] Logistic regression using SciPy
In-Reply-To: <CAMMTP+A_6zMTzsxacdfHYtG3jZUeRVOJZVuQVPs-+Tq4tRVcGA@mail.gmail.com>
References: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
	<CALc6Xo6A-RqpfFArsW5oueFa53UM=b1E8vCwPubfcdtPvz6n+A@mail.gmail.com>
	<CAMMTP+A_6zMTzsxacdfHYtG3jZUeRVOJZVuQVPs-+Tq4tRVcGA@mail.gmail.com>
Message-ID: <1355115639.66117.YahooMailNeo@web160106.mail.bf1.yahoo.com>



Thanks Josef, but I am aware of the alternatives but am interested in rolling my own.



----- Original Message -----
From: "josef.pktd at gmail.com" <josef.pktd at gmail.com>
To: SciPy Users List <scipy-user at scipy.org>
Cc: 
Sent: Monday, December 10, 2012 10:12 AM
Subject: Re: [SciPy-User] Logistic regression using SciPy

On Sun, Dec 9, 2012 at 11:22 PM, David Warde-Farley
<d.warde.farley at gmail.com> wrote:
> First, the way you've written the log likelihood is numerically
> unstable. Consider simplifying the expression (using logarithm laws
> and breaking apart logistic function) and using the log1p function
> where appropriate.
>
> Second, the optimization problem is going to be extremely ill
> conditioned given the very different scales of your different
> predictors. You should probably mean-center and divide by the standard
> deviation.
>
> Third, there's a check_grad function in scipy.optimize that can be
> used to troubleshoot gradient issues.
>
> Fourth, there's a pre-rolled of this in scikit-learn that will
> probably be a good deal faster (it wraps a C library) and certainly
> better tested than home-rolling it.

or statsmodels, which uses analytical gradient and hessians, and gives
you additional result statistics and (statistical) tests.

Josef

>
> David
>
> On Sun, Dec 9, 2012 at 11:01 PM, Fg Nu <fgnu32 at yahoo.com> wrote:
>>
>>
>> I am trying to code up logistic regression in Python using the SciPy "fmin_bfgs" function, but am running into some issues. I wrote functions for the logistic (sigmoid) transformation? function, and the cost function, and those work fine (I have used the optimized values of the parameter vector found via canned software to test the functions, and those match up). I am not that sure of my implementation of the gradient function, but it looks reasonable.
>>
>> Here is the code:
>>
>> #==================================================
>>
>>? ?  # purpose: logistic regression
>>? ?  import numpy as np
>>? ?  import scipy as sp
>>? ?  import scipy.optimize
>>
>>? ?  import matplotlib as mpl
>>? ?  import os
>>
>>? ?  # prepare the data
>>? ?  data = np.loadtxt('data.csv', delimiter=',', skiprows=1)
>>? ?  vY = data[:, 0]
>>? ?  mX = data[:, 1:]
>>? ?  intercept = np.ones(mX.shape[0]).reshape(mX.shape[0], 1)
>>? ?  mX = np.concatenate((intercept, mX), axis = 1)
>>? ?  iK = mX.shape[1]
>>? ?  iN = mX.shape[0]
>>
>>? ?  # logistic transformation
>>? ?  def logit(mX, vBeta):
>>? ? ? ?  return((1/(1.0 + np.exp(-np.dot(mX, vBeta)))))
>>
>>? ?  # test function call
>>? ?  vBeta0 = np.array([-.10296645, -.0332327, -.01209484, .44626211, .92554137, .53973828,
>>? ? ? ?  1.7993371, .7148045? ])
>>? ?  logit(mX, vBeta0)
>>
>>? ?  # cost function
>>? ?  def logLikelihoodLogit(vBeta, mX, vY):
>>? ? ? ?  return(-(np.sum(vY*np.log(logit(mX, vBeta)) + (1-vY)*(np.log(1-logit(mX, vBeta))))))
>>? ?  logLikelihoodLogit(vBeta0, mX, vY) # test function call
>>
>>? ?  # gradient function
>>? ?  def likelihoodScore(vBeta, mX, vY):
>>? ? ? ?  return(np.dot(mX.T,
>>? ? ? ? ? ? ? ? ? ? ?  ((np.dot(mX, vBeta) - vY)/
>>? ? ? ? ? ? ? ? ? ? ? ? np.dot(mX, vBeta)).reshape(iN, 1)).reshape(iK, 1))
>>
>>? ?  likelihoodScore(vBeta0, mX, vY).shape # test function call
>>
>>? ?  # optimize the function (without gradient)
>>? ?  optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,
>>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?  x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
>>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?  1.8, .71]),
>>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?  args = (mX, vY), gtol = 1e-3)
>>
>>? ?  # optimize the function (with gradient)
>>? ?  optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,
>>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?  x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
>>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?  1.8, .71]), fprime = likelihoodScore,
>>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?  args = (mX, vY), gtol = 1e-3)
>> #=====================================================
>>
>> * The first optimization (without gradient)? ends with a whole lot of stuff about division by zero.
>>
>> * The second optimization (with gradient) ends with a matrices not aligned error, which probably means I have got the way the gradient is to be returned wrong.
>>
>> Any help with this is appreciated. If anyone wants to try this, the data is included below.
>>
>>? ?  low,age,lwt,race,smoke,ptl,ht,ui
>>? ?  0,19,182,2,0,0,0,1
>>? ?  0,33,155,3,0,0,0,0
>>? ?  0,20,105,1,1,0,0,0
>>? ?  0,21,108,1,1,0,0,1
>>? ?  0,18,107,1,1,0,0,1
>>? ?  0,21,124,3,0,0,0,0
>>? ?  0,22,118,1,0,0,0,0
>>? ?  0,17,103,3,0,0,0,0
>>? ?  0,29,123,1,1,0,0,0
>>? ?  0,26,113,1,1,0,0,0
>>? ?  0,19,95,3,0,0,0,0
>>? ?  0,19,150,3,0,0,0,0
>>? ?  0,22,95,3,0,0,1,0
>>? ?  0,30,107,3,0,1,0,1
>>? ?  0,18,100,1,1,0,0,0
>>? ?  0,18,100,1,1,0,0,0
>>? ?  0,15,98,2,0,0,0,0
>>? ?  0,25,118,1,1,0,0,0
>>? ?  0,20,120,3,0,0,0,1
>>? ?  0,28,120,1,1,0,0,0
>>? ?  0,32,121,3,0,0,0,0
>>? ?  0,31,100,1,0,0,0,1
>>? ?  0,36,202,1,0,0,0,0
>>? ?  0,28,120,3,0,0,0,0
>>? ?  0,25,120,3,0,0,0,1
>>? ?  0,28,167,1,0,0,0,0
>>? ?  0,17,122,1,1,0,0,0
>>? ?  0,29,150,1,0,0,0,0
>>? ?  0,26,168,2,1,0,0,0
>>? ?  0,17,113,2,0,0,0,0
>>? ?  0,17,113,2,0,0,0,0
>>? ?  0,24,90,1,1,1,0,0
>>? ?  0,35,121,2,1,1,0,0
>>? ?  0,25,155,1,0,0,0,0
>>? ?  0,25,125,2,0,0,0,0
>>? ?  0,29,140,1,1,0,0,0
>>? ?  0,19,138,1,1,0,0,0
>>? ?  0,27,124,1,1,0,0,0
>>? ?  0,31,215,1,1,0,0,0
>>? ?  0,33,109,1,1,0,0,0
>>? ?  0,21,185,2,1,0,0,0
>>? ?  0,19,189,1,0,0,0,0
>>? ?  0,23,130,2,0,0,0,0
>>? ?  0,21,160,1,0,0,0,0
>>? ?  0,18,90,1,1,0,0,1
>>? ?  0,18,90,1,1,0,0,1
>>? ?  0,32,132,1,0,0,0,0
>>? ?  0,19,132,3,0,0,0,0
>>? ?  0,24,115,1,0,0,0,0
>>? ?  0,22,85,3,1,0,0,0
>>? ?  0,22,120,1,0,0,1,0
>>? ?  0,23,128,3,0,0,0,0
>>? ?  0,22,130,1,1,0,0,0
>>? ?  0,30,95,1,1,0,0,0
>>? ?  0,19,115,3,0,0,0,0
>>? ?  0,16,110,3,0,0,0,0
>>? ?  0,21,110,3,1,0,0,1
>>? ?  0,30,153,3,0,0,0,0
>>? ?  0,20,103,3,0,0,0,0
>>? ?  0,17,119,3,0,0,0,0
>>? ?  0,17,119,3,0,0,0,0
>>? ?  0,23,119,3,0,0,0,0
>>? ?  0,24,110,3,0,0,0,0
>>? ?  0,28,140,1,0,0,0,0
>>? ?  0,26,133,3,1,2,0,0
>>? ?  0,20,169,3,0,1,0,1
>>? ?  0,24,115,3,0,0,0,0
>>? ?  0,28,250,3,1,0,0,0
>>? ?  0,20,141,1,0,2,0,1
>>? ?  0,22,158,2,0,1,0,0
>>? ?  0,22,112,1,1,2,0,0
>>? ?  0,31,150,3,1,0,0,0
>>? ?  0,23,115,3,1,0,0,0
>>? ?  0,16,112,2,0,0,0,0
>>? ?  0,16,135,1,1,0,0,0
>>? ?  0,18,229,2,0,0,0,0
>>? ?  0,25,140,1,0,0,0,0
>>? ?  0,32,134,1,1,1,0,0
>>? ?  0,20,121,2,1,0,0,0
>>? ?  0,23,190,1,0,0,0,0
>>? ?  0,22,131,1,0,0,0,0
>>? ?  0,32,170,1,0,0,0,0
>>? ?  0,30,110,3,0,0,0,0
>>? ?  0,20,127,3,0,0,0,0
>>? ?  0,23,123,3,0,0,0,0
>>? ?  0,17,120,3,1,0,0,0
>>? ?  0,19,105,3,0,0,0,0
>>? ?  0,23,130,1,0,0,0,0
>>? ?  0,36,175,1,0,0,0,0
>>? ?  0,22,125,1,0,0,0,0
>>? ?  0,24,133,1,0,0,0,0
>>? ?  0,21,134,3,0,0,0,0
>>? ?  0,19,235,1,1,0,1,0
>>? ?  0,25,95,1,1,3,0,1
>>? ?  0,16,135,1,1,0,0,0
>>? ?  0,29,135,1,0,0,0,0
>>? ?  0,29,154,1,0,0,0,0
>>? ?  0,19,147,1,1,0,0,0
>>? ?  0,19,147,1,1,0,0,0
>>? ?  0,30,137,1,0,0,0,0
>>? ?  0,24,110,1,0,0,0,0
>>? ?  0,19,184,1,1,0,1,0
>>? ?  0,24,110,3,0,1,0,0
>>? ?  0,23,110,1,0,0,0,0
>>? ?  0,20,120,3,0,0,0,0
>>? ?  0,25,241,2,0,0,1,0
>>? ?  0,30,112,1,0,0,0,0
>>? ?  0,22,169,1,0,0,0,0
>>? ?  0,18,120,1,1,0,0,0
>>? ?  0,16,170,2,0,0,0,0
>>? ?  0,32,186,1,0,0,0,0
>>? ?  0,18,120,3,0,0,0,0
>>? ?  0,29,130,1,1,0,0,0
>>? ?  0,33,117,1,0,0,0,1
>>? ?  0,20,170,1,1,0,0,0
>>? ?  0,28,134,3,0,0,0,0
>>? ?  0,14,135,1,0,0,0,0
>>? ?  0,28,130,3,0,0,0,0
>>? ?  0,25,120,1,0,0,0,0
>>? ?  0,16,95,3,0,0,0,0
>>? ?  0,20,158,1,0,0,0,0
>>? ?  0,26,160,3,0,0,0,0
>>? ?  0,21,115,1,0,0,0,0
>>? ?  0,22,129,1,0,0,0,0
>>? ?  0,25,130,1,0,0,0,0
>>? ?  0,31,120,1,0,0,0,0
>>? ?  0,35,170,1,0,1,0,0
>>? ?  0,19,120,1,1,0,0,0
>>? ?  0,24,116,1,0,0,0,0
>>? ?  0,45,123,1,0,0,0,0
>>? ?  1,28,120,3,1,1,0,1
>>? ?  1,29,130,1,0,0,0,1
>>? ?  1,34,187,2,1,0,1,0
>>? ?  1,25,105,3,0,1,1,0
>>? ?  1,25,85,3,0,0,0,1
>>? ?  1,27,150,3,0,0,0,0
>>? ?  1,23,97,3,0,0,0,1
>>? ?  1,24,128,2,0,1,0,0
>>? ?  1,24,132,3,0,0,1,0
>>? ?  1,21,165,1,1,0,1,0
>>? ?  1,32,105,1,1,0,0,0
>>? ?  1,19,91,1,1,2,0,1
>>? ?  1,25,115,3,0,0,0,0
>>? ?  1,16,130,3,0,0,0,0
>>? ?  1,25,92,1,1,0,0,0
>>? ?  1,20,150,1,1,0,0,0
>>? ?  1,21,200,2,0,0,0,1
>>? ?  1,24,155,1,1,1,0,0
>>? ?  1,21,103,3,0,0,0,0
>>? ?  1,20,125,3,0,0,0,1
>>? ?  1,25,89,3,0,2,0,0
>>? ?  1,19,102,1,0,0,0,0
>>? ?  1,19,112,1,1,0,0,1
>>? ?  1,26,117,1,1,1,0,0
>>? ?  1,24,138,1,0,0,0,0
>>? ?  1,17,130,3,1,1,0,1
>>? ?  1,20,120,2,1,0,0,0
>>? ?  1,22,130,1,1,1,0,1
>>? ?  1,27,130,2,0,0,0,1
>>? ?  1,20,80,3,1,0,0,1
>>? ?  1,17,110,1,1,0,0,0
>>? ?  1,25,105,3,0,1,0,0
>>? ?  1,20,109,3,0,0,0,0
>>? ?  1,18,148,3,0,0,0,0
>>? ?  1,18,110,2,1,1,0,0
>>? ?  1,20,121,1,1,1,0,1
>>? ?  1,21,100,3,0,1,0,0
>>? ?  1,26,96,3,0,0,0,0
>>? ?  1,31,102,1,1,1,0,0
>>? ?  1,15,110,1,0,0,0,0
>>? ?  1,23,187,2,1,0,0,0
>>? ?  1,20,122,2,1,0,0,0
>>? ?  1,24,105,2,1,0,0,0
>>? ?  1,15,115,3,0,0,0,1
>>? ?  1,23,120,3,0,0,0,0
>>? ?  1,30,142,1,1,1,0,0
>>? ?  1,22,130,1,1,0,0,0
>>? ?  1,17,120,1,1,0,0,0
>>? ?  1,23,110,1,1,1,0,0
>>? ?  1,17,120,2,0,0,0,0
>>? ?  1,26,154,3,0,1,1,0
>>? ?  1,20,106,3,0,0,0,0
>>? ?  1,26,190,1,1,0,0,0
>>? ?  1,14,101,3,1,1,0,0
>>? ?  1,28,95,1,1,0,0,0
>>? ?  1,14,100,3,0,0,0,0
>>? ?  1,23,94,3,1,0,0,0
>>? ?  1,17,142,2,0,0,1,0
>>? ?  1,21,130,1,1,0,1,0
>>
>> Thanks.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user



From josef.pktd at gmail.com  Mon Dec 10 00:28:58 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 10 Dec 2012 00:28:58 -0500
Subject: [SciPy-User] Logistic regression using SciPy
In-Reply-To: <1355115639.66117.YahooMailNeo@web160106.mail.bf1.yahoo.com>
References: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
	<CALc6Xo6A-RqpfFArsW5oueFa53UM=b1E8vCwPubfcdtPvz6n+A@mail.gmail.com>
	<CAMMTP+A_6zMTzsxacdfHYtG3jZUeRVOJZVuQVPs-+Tq4tRVcGA@mail.gmail.com>
	<1355115639.66117.YahooMailNeo@web160106.mail.bf1.yahoo.com>
Message-ID: <CAMMTP+D3V3x2WbRnXfE5e2z0EEXfFe8X5aC39gu02-aVsYXOXQ@mail.gmail.com>

On Mon, Dec 10, 2012 at 12:00 AM, Fg Nu <fgnu32 at yahoo.com> wrote:
>
>
> Thanks Josef, but I am aware of the alternatives but am interested in rolling my own.

That's always a good exercise :)

I think your last reshape in likelihoodScore is wrong. The return
should be 1-dimensional.
for example .ravel()

Josef

>
>
>
> ----- Original Message -----
> From: "josef.pktd at gmail.com" <josef.pktd at gmail.com>
> To: SciPy Users List <scipy-user at scipy.org>
> Cc:
> Sent: Monday, December 10, 2012 10:12 AM
> Subject: Re: [SciPy-User] Logistic regression using SciPy
>
> On Sun, Dec 9, 2012 at 11:22 PM, David Warde-Farley
> <d.warde.farley at gmail.com> wrote:
>> First, the way you've written the log likelihood is numerically
>> unstable. Consider simplifying the expression (using logarithm laws
>> and breaking apart logistic function) and using the log1p function
>> where appropriate.
>>
>> Second, the optimization problem is going to be extremely ill
>> conditioned given the very different scales of your different
>> predictors. You should probably mean-center and divide by the standard
>> deviation.
>>
>> Third, there's a check_grad function in scipy.optimize that can be
>> used to troubleshoot gradient issues.
>>
>> Fourth, there's a pre-rolled of this in scikit-learn that will
>> probably be a good deal faster (it wraps a C library) and certainly
>> better tested than home-rolling it.
>
> or statsmodels, which uses analytical gradient and hessians, and gives
> you additional result statistics and (statistical) tests.
>
> Josef
>
>>
>> David
>>
>> On Sun, Dec 9, 2012 at 11:01 PM, Fg Nu <fgnu32 at yahoo.com> wrote:
>>>
>>>
>>> I am trying to code up logistic regression in Python using the SciPy "fmin_bfgs" function, but am running into some issues. I wrote functions for the logistic (sigmoid) transformation  function, and the cost function, and those work fine (I have used the optimized values of the parameter vector found via canned software to test the functions, and those match up). I am not that sure of my implementation of the gradient function, but it looks reasonable.
>>>
>>> Here is the code:
>>>
>>> #==================================================
>>>
>>>     # purpose: logistic regression
>>>     import numpy as np
>>>     import scipy as sp
>>>     import scipy.optimize
>>>
>>>     import matplotlib as mpl
>>>     import os
>>>
>>>     # prepare the data
>>>     data = np.loadtxt('data.csv', delimiter=',', skiprows=1)
>>>     vY = data[:, 0]
>>>     mX = data[:, 1:]
>>>     intercept = np.ones(mX.shape[0]).reshape(mX.shape[0], 1)
>>>     mX = np.concatenate((intercept, mX), axis = 1)
>>>     iK = mX.shape[1]
>>>     iN = mX.shape[0]
>>>
>>>     # logistic transformation
>>>     def logit(mX, vBeta):
>>>         return((1/(1.0 + np.exp(-np.dot(mX, vBeta)))))
>>>
>>>     # test function call
>>>     vBeta0 = np.array([-.10296645, -.0332327, -.01209484, .44626211, .92554137, .53973828,
>>>         1.7993371, .7148045  ])
>>>     logit(mX, vBeta0)
>>>
>>>     # cost function
>>>     def logLikelihoodLogit(vBeta, mX, vY):
>>>         return(-(np.sum(vY*np.log(logit(mX, vBeta)) + (1-vY)*(np.log(1-logit(mX, vBeta))))))
>>>     logLikelihoodLogit(vBeta0, mX, vY) # test function call
>>>
>>>     # gradient function
>>>     def likelihoodScore(vBeta, mX, vY):
>>>         return(np.dot(mX.T,
>>>                       ((np.dot(mX, vBeta) - vY)/
>>>                        np.dot(mX, vBeta)).reshape(iN, 1)).reshape(iK, 1))
>>>
>>>     likelihoodScore(vBeta0, mX, vY).shape # test function call
>>>
>>>     # optimize the function (without gradient)
>>>     optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,
>>>                                       x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
>>>                                                 1.8, .71]),
>>>                                       args = (mX, vY), gtol = 1e-3)
>>>
>>>     # optimize the function (with gradient)
>>>     optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,
>>>                                       x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
>>>                                                 1.8, .71]), fprime = likelihoodScore,
>>>                                       args = (mX, vY), gtol = 1e-3)
>>> #=====================================================
>>>
>>> * The first optimization (without gradient)  ends with a whole lot of stuff about division by zero.
>>>
>>> * The second optimization (with gradient) ends with a matrices not aligned error, which probably means I have got the way the gradient is to be returned wrong.
>>>
>>> Any help with this is appreciated. If anyone wants to try this, the data is included below.
>>>
>>>     low,age,lwt,race,smoke,ptl,ht,ui
>>>     0,19,182,2,0,0,0,1
>>>     0,33,155,3,0,0,0,0
>>>     0,20,105,1,1,0,0,0
>>>     0,21,108,1,1,0,0,1
>>>     0,18,107,1,1,0,0,1
>>>     0,21,124,3,0,0,0,0
>>>     0,22,118,1,0,0,0,0
>>>     0,17,103,3,0,0,0,0
>>>     0,29,123,1,1,0,0,0
>>>     0,26,113,1,1,0,0,0
>>>     0,19,95,3,0,0,0,0
>>>     0,19,150,3,0,0,0,0
>>>     0,22,95,3,0,0,1,0
>>>     0,30,107,3,0,1,0,1
>>>     0,18,100,1,1,0,0,0
>>>     0,18,100,1,1,0,0,0
>>>     0,15,98,2,0,0,0,0
>>>     0,25,118,1,1,0,0,0
>>>     0,20,120,3,0,0,0,1
>>>     0,28,120,1,1,0,0,0
>>>     0,32,121,3,0,0,0,0
>>>     0,31,100,1,0,0,0,1
>>>     0,36,202,1,0,0,0,0
>>>     0,28,120,3,0,0,0,0
>>>     0,25,120,3,0,0,0,1
>>>     0,28,167,1,0,0,0,0
>>>     0,17,122,1,1,0,0,0
>>>     0,29,150,1,0,0,0,0
>>>     0,26,168,2,1,0,0,0
>>>     0,17,113,2,0,0,0,0
>>>     0,17,113,2,0,0,0,0
>>>     0,24,90,1,1,1,0,0
>>>     0,35,121,2,1,1,0,0
>>>     0,25,155,1,0,0,0,0
>>>     0,25,125,2,0,0,0,0
>>>     0,29,140,1,1,0,0,0
>>>     0,19,138,1,1,0,0,0
>>>     0,27,124,1,1,0,0,0
>>>     0,31,215,1,1,0,0,0
>>>     0,33,109,1,1,0,0,0
>>>     0,21,185,2,1,0,0,0
>>>     0,19,189,1,0,0,0,0
>>>     0,23,130,2,0,0,0,0
>>>     0,21,160,1,0,0,0,0
>>>     0,18,90,1,1,0,0,1
>>>     0,18,90,1,1,0,0,1
>>>     0,32,132,1,0,0,0,0
>>>     0,19,132,3,0,0,0,0
>>>     0,24,115,1,0,0,0,0
>>>     0,22,85,3,1,0,0,0
>>>     0,22,120,1,0,0,1,0
>>>     0,23,128,3,0,0,0,0
>>>     0,22,130,1,1,0,0,0
>>>     0,30,95,1,1,0,0,0
>>>     0,19,115,3,0,0,0,0
>>>     0,16,110,3,0,0,0,0
>>>     0,21,110,3,1,0,0,1
>>>     0,30,153,3,0,0,0,0
>>>     0,20,103,3,0,0,0,0
>>>     0,17,119,3,0,0,0,0
>>>     0,17,119,3,0,0,0,0
>>>     0,23,119,3,0,0,0,0
>>>     0,24,110,3,0,0,0,0
>>>     0,28,140,1,0,0,0,0
>>>     0,26,133,3,1,2,0,0
>>>     0,20,169,3,0,1,0,1
>>>     0,24,115,3,0,0,0,0
>>>     0,28,250,3,1,0,0,0
>>>     0,20,141,1,0,2,0,1
>>>     0,22,158,2,0,1,0,0
>>>     0,22,112,1,1,2,0,0
>>>     0,31,150,3,1,0,0,0
>>>     0,23,115,3,1,0,0,0
>>>     0,16,112,2,0,0,0,0
>>>     0,16,135,1,1,0,0,0
>>>     0,18,229,2,0,0,0,0
>>>     0,25,140,1,0,0,0,0
>>>     0,32,134,1,1,1,0,0
>>>     0,20,121,2,1,0,0,0
>>>     0,23,190,1,0,0,0,0
>>>     0,22,131,1,0,0,0,0
>>>     0,32,170,1,0,0,0,0
>>>     0,30,110,3,0,0,0,0
>>>     0,20,127,3,0,0,0,0
>>>     0,23,123,3,0,0,0,0
>>>     0,17,120,3,1,0,0,0
>>>     0,19,105,3,0,0,0,0
>>>     0,23,130,1,0,0,0,0
>>>     0,36,175,1,0,0,0,0
>>>     0,22,125,1,0,0,0,0
>>>     0,24,133,1,0,0,0,0
>>>     0,21,134,3,0,0,0,0
>>>     0,19,235,1,1,0,1,0
>>>     0,25,95,1,1,3,0,1
>>>     0,16,135,1,1,0,0,0
>>>     0,29,135,1,0,0,0,0
>>>     0,29,154,1,0,0,0,0
>>>     0,19,147,1,1,0,0,0
>>>     0,19,147,1,1,0,0,0
>>>     0,30,137,1,0,0,0,0
>>>     0,24,110,1,0,0,0,0
>>>     0,19,184,1,1,0,1,0
>>>     0,24,110,3,0,1,0,0
>>>     0,23,110,1,0,0,0,0
>>>     0,20,120,3,0,0,0,0
>>>     0,25,241,2,0,0,1,0
>>>     0,30,112,1,0,0,0,0
>>>     0,22,169,1,0,0,0,0
>>>     0,18,120,1,1,0,0,0
>>>     0,16,170,2,0,0,0,0
>>>     0,32,186,1,0,0,0,0
>>>     0,18,120,3,0,0,0,0
>>>     0,29,130,1,1,0,0,0
>>>     0,33,117,1,0,0,0,1
>>>     0,20,170,1,1,0,0,0
>>>     0,28,134,3,0,0,0,0
>>>     0,14,135,1,0,0,0,0
>>>     0,28,130,3,0,0,0,0
>>>     0,25,120,1,0,0,0,0
>>>     0,16,95,3,0,0,0,0
>>>     0,20,158,1,0,0,0,0
>>>     0,26,160,3,0,0,0,0
>>>     0,21,115,1,0,0,0,0
>>>     0,22,129,1,0,0,0,0
>>>     0,25,130,1,0,0,0,0
>>>     0,31,120,1,0,0,0,0
>>>     0,35,170,1,0,1,0,0
>>>     0,19,120,1,1,0,0,0
>>>     0,24,116,1,0,0,0,0
>>>     0,45,123,1,0,0,0,0
>>>     1,28,120,3,1,1,0,1
>>>     1,29,130,1,0,0,0,1
>>>     1,34,187,2,1,0,1,0
>>>     1,25,105,3,0,1,1,0
>>>     1,25,85,3,0,0,0,1
>>>     1,27,150,3,0,0,0,0
>>>     1,23,97,3,0,0,0,1
>>>     1,24,128,2,0,1,0,0
>>>     1,24,132,3,0,0,1,0
>>>     1,21,165,1,1,0,1,0
>>>     1,32,105,1,1,0,0,0
>>>     1,19,91,1,1,2,0,1
>>>     1,25,115,3,0,0,0,0
>>>     1,16,130,3,0,0,0,0
>>>     1,25,92,1,1,0,0,0
>>>     1,20,150,1,1,0,0,0
>>>     1,21,200,2,0,0,0,1
>>>     1,24,155,1,1,1,0,0
>>>     1,21,103,3,0,0,0,0
>>>     1,20,125,3,0,0,0,1
>>>     1,25,89,3,0,2,0,0
>>>     1,19,102,1,0,0,0,0
>>>     1,19,112,1,1,0,0,1
>>>     1,26,117,1,1,1,0,0
>>>     1,24,138,1,0,0,0,0
>>>     1,17,130,3,1,1,0,1
>>>     1,20,120,2,1,0,0,0
>>>     1,22,130,1,1,1,0,1
>>>     1,27,130,2,0,0,0,1
>>>     1,20,80,3,1,0,0,1
>>>     1,17,110,1,1,0,0,0
>>>     1,25,105,3,0,1,0,0
>>>     1,20,109,3,0,0,0,0
>>>     1,18,148,3,0,0,0,0
>>>     1,18,110,2,1,1,0,0
>>>     1,20,121,1,1,1,0,1
>>>     1,21,100,3,0,1,0,0
>>>     1,26,96,3,0,0,0,0
>>>     1,31,102,1,1,1,0,0
>>>     1,15,110,1,0,0,0,0
>>>     1,23,187,2,1,0,0,0
>>>     1,20,122,2,1,0,0,0
>>>     1,24,105,2,1,0,0,0
>>>     1,15,115,3,0,0,0,1
>>>     1,23,120,3,0,0,0,0
>>>     1,30,142,1,1,1,0,0
>>>     1,22,130,1,1,0,0,0
>>>     1,17,120,1,1,0,0,0
>>>     1,23,110,1,1,1,0,0
>>>     1,17,120,2,0,0,0,0
>>>     1,26,154,3,0,1,1,0
>>>     1,20,106,3,0,0,0,0
>>>     1,26,190,1,1,0,0,0
>>>     1,14,101,3,1,1,0,0
>>>     1,28,95,1,1,0,0,0
>>>     1,14,100,3,0,0,0,0
>>>     1,23,94,3,1,0,0,0
>>>     1,17,142,2,0,0,1,0
>>>     1,21,130,1,1,0,1,0
>>>
>>> Thanks.
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From fgnu32 at yahoo.com  Mon Dec 10 01:26:12 2012
From: fgnu32 at yahoo.com (Fg Nu)
Date: Sun, 9 Dec 2012 22:26:12 -0800 (PST)
Subject: [SciPy-User] Logistic regression using SciPy
In-Reply-To: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
References: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
Message-ID: <1355120772.80318.YahooMailNeo@web160102.mail.bf1.yahoo.com>



For any one interested, here is the final code, where everything works. Basically, I reparametrized the likelihood function. Also, added a call to the check_grad function.

#=====================================================
# purpose: logistic regression?
import numpy as np
import scipy as sp
import scipy.optimize

import matplotlib as mpl
import os

# prepare the data
data = np.loadtxt('data.csv', delimiter=',', skiprows=1)
vY = data[:, 0]
mX = data[:, 1:]
# mX = (mX - np.mean(mX))/np.std(mX) ?# standardize the data; if required

intercept = np.ones(mX.shape[0]).reshape(mX.shape[0], 1)
mX = np.concatenate((intercept, mX), axis = 1)
iK = mX.shape[1]
iN = mX.shape[0]
?
# logistic transformation
def logit(mX, vBeta):
? ? return((np.exp(np.dot(mX, vBeta))/(1.0 + np.exp(np.dot(mX, vBeta)))))

# test function call
vBeta0 = np.array([-.10296645, -.0332327, -.01209484, .44626211, .92554137, .53973828,?
? ? 1.7993371, .7148045 ?])
logit(mX, vBeta0)

# cost function
def logLikelihoodLogit(vBeta, mX, vY):
? ? return(-(np.sum(vY*np.log(logit(mX, vBeta)) + (1-vY)*(np.log(1-logit(mX, vBeta))))))
logLikelihoodLogit(vBeta0, mX, vY) # test function call

# different parametrization of the cost function
def logLikelihoodLogitVerbose(vBeta, mX, vY):
? ? return(-(np.sum(vY*(np.dot(mX, vBeta) - np.log((1.0 + np.exp(np.dot(mX, vBeta))))) +
? ? ? ? ? ? ? ? ? ? (1-vY)*(-np.log((1.0 + np.exp(np.dot(mX, vBeta))))))))
logLikelihoodLogitVerbose(vBeta0, mX, vY) ?# test function call

# gradient function
def likelihoodScore(vBeta, mX, vY):
? ? return(np.dot(mX.T,?
? ? ? ? ? ? ? ? ? (logit(mX, vBeta) - vY)))
likelihoodScore(vBeta0, mX, vY).shape # test function call
sp.optimize.check_grad(logLikelihoodLogitVerbose, likelihoodScore,?
? ? ? ? ? ? ? ? ? ? ? ?vBeta0, mX, vY) ?# check that the analytical gradient is close to?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? # numerical gradient

# optimize the function (without gradient)
optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogitVerbose,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1.8, .71]),?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? args = (mX, vY), gtol = 1e-3)

# optimize the function (with gradient)
optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogitVerbose,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1.8, .71]), fprime = likelihoodScore,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? args = (mX, vY), gtol = 1e-3)
#=====================================================


Thanks for the helpful suggestions from everyone on- and off-list.


----- Original Message -----
From: Fg Nu <fgnu32 at yahoo.com>
To: "scipy-user at scipy.org" <scipy-user at scipy.org>
Cc: 
Sent: Monday, December 10, 2012 9:31 AM
Subject: Logistic regression using SciPy



I am trying to code up logistic regression in Python using the SciPy "fmin_bfgs" function, but am running into some issues. I wrote functions for the logistic (sigmoid) transformation ?function, and the cost function, and those work fine (I have used the optimized values of the parameter vector found via canned software to test the functions, and those match up). I am not that sure of my implementation of the gradient function, but it looks reasonable.?

Here is the code:

#==================================================

? ? # purpose: logistic regression?
? ? import numpy as np
? ? import scipy as sp
? ? import scipy.optimize
? ??
? ? import matplotlib as mpl
? ? import os
? ??
? ? # prepare the data
? ? data = np.loadtxt('data.csv', delimiter=',', skiprows=1)
? ? vY = data[:, 0]
? ? mX = data[:, 1:]
? ? intercept = np.ones(mX.shape[0]).reshape(mX.shape[0], 1)
? ? mX = np.concatenate((intercept, mX), axis = 1)
? ? iK = mX.shape[1]
? ? iN = mX.shape[0]
? ? ?
? ? # logistic transformation
? ? def logit(mX, vBeta):
? ? ? ? return((1/(1.0 + np.exp(-np.dot(mX, vBeta)))))
? ??
? ? # test function call
? ? vBeta0 = np.array([-.10296645, -.0332327, -.01209484, .44626211, .92554137, .53973828,?
? ? ? ? 1.7993371, .7148045 ?])
? ? logit(mX, vBeta0)
? ??
? ? # cost function
? ? def logLikelihoodLogit(vBeta, mX, vY):
? ? ? ? return(-(np.sum(vY*np.log(logit(mX, vBeta)) + (1-vY)*(np.log(1-logit(mX, vBeta))))))
? ? logLikelihoodLogit(vBeta0, mX, vY) # test function call
? ??
? ? # gradient function
? ? def likelihoodScore(vBeta, mX, vY):
? ? ? ? return(np.dot(mX.T,?
? ? ? ? ? ? ? ? ? ? ? ((np.dot(mX, vBeta) - vY)/
? ? ? ? ? ? ? ? ? ? ? ?np.dot(mX, vBeta)).reshape(iN, 1)).reshape(iK, 1))
? ??
? ? likelihoodScore(vBeta0, mX, vY).shape # test function call
? ? ?
? ? # optimize the function (without gradient)
? ? optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1.8, .71]),?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? args = (mX, vY), gtol = 1e-3)
? ??
? ? # optimize the function (with gradient)
? ? optimLogit = scipy.optimize.fmin_bfgs(logLikelihoodLogit,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? x0 = np.array([-.1, -.03, -.01, .44, .92, .53,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1.8, .71]), fprime = likelihoodScore,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? args = (mX, vY), gtol = 1e-3)
#=====================================================

* The first optimization (without gradient) ?ends with a whole lot of stuff about division by zero.

* The second optimization (with gradient) ends with a matrices not aligned error, which probably means I have got the way the gradient is to be returned wrong.

Any help with this is appreciated. If anyone wants to try this, the data is included below.

? ? low,age,lwt,race,smoke,ptl,ht,ui
? ? 0,19,182,2,0,0,0,1
? ? 0,33,155,3,0,0,0,0
? ? 0,20,105,1,1,0,0,0
? ? 0,21,108,1,1,0,0,1
? ? 0,18,107,1,1,0,0,1
? ? 0,21,124,3,0,0,0,0
? ? 0,22,118,1,0,0,0,0
? ? 0,17,103,3,0,0,0,0
? ? 0,29,123,1,1,0,0,0
? ? 0,26,113,1,1,0,0,0
? ? 0,19,95,3,0,0,0,0
? ? 0,19,150,3,0,0,0,0
? ? 0,22,95,3,0,0,1,0
? ? 0,30,107,3,0,1,0,1
? ? 0,18,100,1,1,0,0,0
? ? 0,18,100,1,1,0,0,0
? ? 0,15,98,2,0,0,0,0
? ? 0,25,118,1,1,0,0,0
? ? 0,20,120,3,0,0,0,1
? ? 0,28,120,1,1,0,0,0
? ? 0,32,121,3,0,0,0,0
? ? 0,31,100,1,0,0,0,1
? ? 0,36,202,1,0,0,0,0
? ? 0,28,120,3,0,0,0,0
? ? 0,25,120,3,0,0,0,1
? ? 0,28,167,1,0,0,0,0
? ? 0,17,122,1,1,0,0,0
? ? 0,29,150,1,0,0,0,0
? ? 0,26,168,2,1,0,0,0
? ? 0,17,113,2,0,0,0,0
? ? 0,17,113,2,0,0,0,0
? ? 0,24,90,1,1,1,0,0
? ? 0,35,121,2,1,1,0,0
? ? 0,25,155,1,0,0,0,0
? ? 0,25,125,2,0,0,0,0
? ? 0,29,140,1,1,0,0,0
? ? 0,19,138,1,1,0,0,0
? ? 0,27,124,1,1,0,0,0
? ? 0,31,215,1,1,0,0,0
? ? 0,33,109,1,1,0,0,0
? ? 0,21,185,2,1,0,0,0
? ? 0,19,189,1,0,0,0,0
? ? 0,23,130,2,0,0,0,0
? ? 0,21,160,1,0,0,0,0
? ? 0,18,90,1,1,0,0,1
? ? 0,18,90,1,1,0,0,1
? ? 0,32,132,1,0,0,0,0
? ? 0,19,132,3,0,0,0,0
? ? 0,24,115,1,0,0,0,0
? ? 0,22,85,3,1,0,0,0
? ? 0,22,120,1,0,0,1,0
? ? 0,23,128,3,0,0,0,0
? ? 0,22,130,1,1,0,0,0
? ? 0,30,95,1,1,0,0,0
? ? 0,19,115,3,0,0,0,0
? ? 0,16,110,3,0,0,0,0
? ? 0,21,110,3,1,0,0,1
? ? 0,30,153,3,0,0,0,0
? ? 0,20,103,3,0,0,0,0
? ? 0,17,119,3,0,0,0,0
? ? 0,17,119,3,0,0,0,0
? ? 0,23,119,3,0,0,0,0
? ? 0,24,110,3,0,0,0,0
? ? 0,28,140,1,0,0,0,0
? ? 0,26,133,3,1,2,0,0
? ? 0,20,169,3,0,1,0,1
? ? 0,24,115,3,0,0,0,0
? ? 0,28,250,3,1,0,0,0
? ? 0,20,141,1,0,2,0,1
? ? 0,22,158,2,0,1,0,0
? ? 0,22,112,1,1,2,0,0
? ? 0,31,150,3,1,0,0,0
? ? 0,23,115,3,1,0,0,0
? ? 0,16,112,2,0,0,0,0
? ? 0,16,135,1,1,0,0,0
? ? 0,18,229,2,0,0,0,0
? ? 0,25,140,1,0,0,0,0
? ? 0,32,134,1,1,1,0,0
? ? 0,20,121,2,1,0,0,0
? ? 0,23,190,1,0,0,0,0
? ? 0,22,131,1,0,0,0,0
? ? 0,32,170,1,0,0,0,0
? ? 0,30,110,3,0,0,0,0
? ? 0,20,127,3,0,0,0,0
? ? 0,23,123,3,0,0,0,0
? ? 0,17,120,3,1,0,0,0
? ? 0,19,105,3,0,0,0,0
? ? 0,23,130,1,0,0,0,0
? ? 0,36,175,1,0,0,0,0
? ? 0,22,125,1,0,0,0,0
? ? 0,24,133,1,0,0,0,0
? ? 0,21,134,3,0,0,0,0
? ? 0,19,235,1,1,0,1,0
? ? 0,25,95,1,1,3,0,1
? ? 0,16,135,1,1,0,0,0
? ? 0,29,135,1,0,0,0,0
? ? 0,29,154,1,0,0,0,0
? ? 0,19,147,1,1,0,0,0
? ? 0,19,147,1,1,0,0,0
? ? 0,30,137,1,0,0,0,0
? ? 0,24,110,1,0,0,0,0
? ? 0,19,184,1,1,0,1,0
? ? 0,24,110,3,0,1,0,0
? ? 0,23,110,1,0,0,0,0
? ? 0,20,120,3,0,0,0,0
? ? 0,25,241,2,0,0,1,0
? ? 0,30,112,1,0,0,0,0
? ? 0,22,169,1,0,0,0,0
? ? 0,18,120,1,1,0,0,0
? ? 0,16,170,2,0,0,0,0
? ? 0,32,186,1,0,0,0,0
? ? 0,18,120,3,0,0,0,0
? ? 0,29,130,1,1,0,0,0
? ? 0,33,117,1,0,0,0,1
? ? 0,20,170,1,1,0,0,0
? ? 0,28,134,3,0,0,0,0
? ? 0,14,135,1,0,0,0,0
? ? 0,28,130,3,0,0,0,0
? ? 0,25,120,1,0,0,0,0
? ? 0,16,95,3,0,0,0,0
? ? 0,20,158,1,0,0,0,0
? ? 0,26,160,3,0,0,0,0
? ? 0,21,115,1,0,0,0,0
? ? 0,22,129,1,0,0,0,0
? ? 0,25,130,1,0,0,0,0
? ? 0,31,120,1,0,0,0,0
? ? 0,35,170,1,0,1,0,0
? ? 0,19,120,1,1,0,0,0
? ? 0,24,116,1,0,0,0,0
? ? 0,45,123,1,0,0,0,0
? ? 1,28,120,3,1,1,0,1
? ? 1,29,130,1,0,0,0,1
? ? 1,34,187,2,1,0,1,0
? ? 1,25,105,3,0,1,1,0
? ? 1,25,85,3,0,0,0,1
? ? 1,27,150,3,0,0,0,0
? ? 1,23,97,3,0,0,0,1
? ? 1,24,128,2,0,1,0,0
? ? 1,24,132,3,0,0,1,0
? ? 1,21,165,1,1,0,1,0
? ? 1,32,105,1,1,0,0,0
? ? 1,19,91,1,1,2,0,1
? ? 1,25,115,3,0,0,0,0
? ? 1,16,130,3,0,0,0,0
? ? 1,25,92,1,1,0,0,0
? ? 1,20,150,1,1,0,0,0
? ? 1,21,200,2,0,0,0,1
? ? 1,24,155,1,1,1,0,0
? ? 1,21,103,3,0,0,0,0
? ? 1,20,125,3,0,0,0,1
? ? 1,25,89,3,0,2,0,0
? ? 1,19,102,1,0,0,0,0
? ? 1,19,112,1,1,0,0,1
? ? 1,26,117,1,1,1,0,0
? ? 1,24,138,1,0,0,0,0
? ? 1,17,130,3,1,1,0,1
? ? 1,20,120,2,1,0,0,0
? ? 1,22,130,1,1,1,0,1
? ? 1,27,130,2,0,0,0,1
? ? 1,20,80,3,1,0,0,1
? ? 1,17,110,1,1,0,0,0
? ? 1,25,105,3,0,1,0,0
? ? 1,20,109,3,0,0,0,0
? ? 1,18,148,3,0,0,0,0
? ? 1,18,110,2,1,1,0,0
? ? 1,20,121,1,1,1,0,1
? ? 1,21,100,3,0,1,0,0
? ? 1,26,96,3,0,0,0,0
? ? 1,31,102,1,1,1,0,0
? ? 1,15,110,1,0,0,0,0
? ? 1,23,187,2,1,0,0,0
? ? 1,20,122,2,1,0,0,0
? ? 1,24,105,2,1,0,0,0
? ? 1,15,115,3,0,0,0,1
? ? 1,23,120,3,0,0,0,0
? ? 1,30,142,1,1,1,0,0
? ? 1,22,130,1,1,0,0,0
? ? 1,17,120,1,1,0,0,0
? ? 1,23,110,1,1,1,0,0
? ? 1,17,120,2,0,0,0,0
? ? 1,26,154,3,0,1,1,0
? ? 1,20,106,3,0,0,0,0
? ? 1,26,190,1,1,0,0,0
? ? 1,14,101,3,1,1,0,0
? ? 1,28,95,1,1,0,0,0
? ? 1,14,100,3,0,0,0,0
? ? 1,23,94,3,1,0,0,0
? ? 1,17,142,2,0,0,1,0
? ? 1,21,130,1,1,0,1,0

Thanks.


From alan.isaac at gmail.com  Mon Dec 10 08:23:22 2012
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Mon, 10 Dec 2012 08:23:22 -0500
Subject: [SciPy-User] Logistic regression using SciPy
In-Reply-To: <1355115639.66117.YahooMailNeo@web160106.mail.bf1.yahoo.com>
References: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
	<CALc6Xo6A-RqpfFArsW5oueFa53UM=b1E8vCwPubfcdtPvz6n+A@mail.gmail.com>
	<CAMMTP+A_6zMTzsxacdfHYtG3jZUeRVOJZVuQVPs-+Tq4tRVcGA@mail.gmail.com>
	<1355115639.66117.YahooMailNeo@web160106.mail.bf1.yahoo.com>
Message-ID: <50C5E24A.8040201@gmail.com>

On 12/10/2012 12:00 AM, Fg Nu wrote:
> I am aware of the alternatives but am interested in rolling my own.


At least look at statsmodels,
because you are headed down the wrong track.

Alan Isaac



From alicsailmitedu at gmail.com  Mon Dec 10 01:23:07 2012
From: alicsailmitedu at gmail.com (ali rahimi)
Date: Sun, 9 Dec 2012 22:23:07 -0800
Subject: [SciPy-User] computing f and fprime in one evaluation in
	scipy.optimize
Message-ID: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>

When minimizing a function f(x), df/dx can often also be computed with little additional effort. Yet the routines in scipy.optimize accept df/dx as a separate function fprime which must repeat much of the computation of f.

do the scipy.optimize routines allow me to compute f and fprime simultaneous in one call? this is important to me because each evaluation in my case takes ~10 minutes and a few iterations of the optimizer is all i need.

From gregor.thalhammer at gmail.com  Mon Dec 10 09:54:55 2012
From: gregor.thalhammer at gmail.com (Gregor Thalhammer)
Date: Mon, 10 Dec 2012 15:54:55 +0100
Subject: [SciPy-User] computing f and fprime in one evaluation in
	scipy.optimize
In-Reply-To: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>
References: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>
Message-ID: <223DC4FB-F581-4C65-9D37-727A0A0251C3@gmail.com>


Am 10.12.2012 um 07:23 schrieb ali rahimi:

> When minimizing a function f(x), df/dx can often also be computed with little additional effort. Yet the routines in scipy.optimize accept df/dx as a separate function fprime which must repeat much of the computation of f.
> 
> do the scipy.optimize routines allow me to compute f and fprime simultaneous in one call? this is important to me because each evaluation in my case takes ~10 minutes and a few iterations of the optimizer is all i need.

They are called separately. For a more efficient calculation you could save the result of f, e.g. in a class attribute or a global variable, and reuse it in fprime. And don't forget to also store the parameters to check whether fprime is indeed called with the same params.

Gregor

From jsseabold at gmail.com  Mon Dec 10 10:06:27 2012
From: jsseabold at gmail.com (Skipper Seabold)
Date: Mon, 10 Dec 2012 10:06:27 -0500
Subject: [SciPy-User] computing f and fprime in one evaluation in
	scipy.optimize
In-Reply-To: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>
References: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>
Message-ID: <CAKF=DjujnOt1Ee55ZxsLbkecVNN2Q3UEOWCaNcTy7c9=zXvsYA@mail.gmail.com>

On Mon, Dec 10, 2012 at 1:23 AM, ali rahimi <alicsailmitedu at gmail.com> wrote:
> When minimizing a function f(x), df/dx can often also be computed with little additional effort. Yet the routines in scipy.optimize accept df/dx as a separate function fprime which must repeat much of the computation of f.
>
> do the scipy.optimize routines allow me to compute f and fprime simultaneous in one call? this is important to me because each evaluation in my case takes ~10 minutes and a few iterations of the optimizer is all i need.

AFAIK, fmin_l_bfgs_b is the only optimize function written so that f
can also return fprime, but I agree that this could be a nice option
to the other ones.

Skipper


From pav at iki.fi  Mon Dec 10 10:11:03 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 10 Dec 2012 15:11:03 +0000 (UTC)
Subject: [SciPy-User]
	=?utf-8?q?computing_f_and_fprime_in_one_evaluation_i?=
	=?utf-8?q?n=09scipy=2Eoptimize?=
References: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>
	<CAKF=DjujnOt1Ee55ZxsLbkecVNN2Q3UEOWCaNcTy7c9=zXvsYA@mail.gmail.com>
Message-ID: <loom.20121210T160912-642@post.gmane.org>

Skipper Seabold <jsseabold <at> gmail.com> writes:
[clip]
> AFAIK, fmin_l_bfgs_b is the only optimize function written so that f
> can also return fprime, but I agree that this could be a nice option
> to the other ones.

The good news is that minimize(f, .., jac=True) [1] also works like this,
and works for all of the solvers (it caches the jacobian).

So the problem is already solved in Scipy 0.11.0 :)

[1] 
http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html

-- 
Pauli Virtanen



From matthieu.brucher at gmail.com  Mon Dec 10 10:22:16 2012
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 10 Dec 2012 16:22:16 +0100
Subject: [SciPy-User] computing f and fprime in one evaluation in
	scipy.optimize
In-Reply-To: <loom.20121210T160912-642@post.gmane.org>
References: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>
	<CAKF=DjujnOt1Ee55ZxsLbkecVNN2Q3UEOWCaNcTy7c9=zXvsYA@mail.gmail.com>
	<loom.20121210T160912-642@post.gmane.org>
Message-ID: <CAHCaCkKQ-vKGxi7hNif7+iQuPiTTURPqWN3RiVqN5LAuOX4r5w@mail.gmail.com>

Hi Pauli,

What is the Jacobian for a CG or BFGS optimization? There are no Jacobian
in this case for non-least square optimization if I'm not mistaken (at
least in no reference that I know of). Shouldn't this be the gradient
function?

Cheers,



2012/12/10 Pauli Virtanen <pav at iki.fi>

> Skipper Seabold <jsseabold <at> gmail.com> writes:
> [clip]
> > AFAIK, fmin_l_bfgs_b is the only optimize function written so that f
> > can also return fprime, but I agree that this could be a nice option
> > to the other ones.
>
> The good news is that minimize(f, .., jac=True) [1] also works like this,
> and works for all of the solvers (it caches the jacobian).
>
> So the problem is already solved in Scipy 0.11.0 :)
>
> [1]
>
> http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html
>
> --
> Pauli Virtanen
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121210/2e469902/attachment.html>

From fgnu32 at yahoo.com  Mon Dec 10 10:25:36 2012
From: fgnu32 at yahoo.com (Fg Nu)
Date: Mon, 10 Dec 2012 07:25:36 -0800 (PST)
Subject: [SciPy-User] Logistic regression using SciPy
In-Reply-To: <50C5E24A.8040201@gmail.com>
References: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
	<CALc6Xo6A-RqpfFArsW5oueFa53UM=b1E8vCwPubfcdtPvz6n+A@mail.gmail.com>
	<CAMMTP+A_6zMTzsxacdfHYtG3jZUeRVOJZVuQVPs-+Tq4tRVcGA@mail.gmail.com>
	<1355115639.66117.YahooMailNeo@web160106.mail.bf1.yahoo.com>
	<50C5E24A.8040201@gmail.com>
Message-ID: <1355153136.96410.YahooMailNeo@web160101.mail.bf1.yahoo.com>





----- Original Message -----
From: Alan G Isaac <alan.isaac at gmail.com>
To: Fg Nu <fgnu32 at yahoo.com>; SciPy Users List <scipy-user at scipy.org>
Cc: 
Sent: Monday, December 10, 2012 6:53 PM
Subject: Re: [SciPy-User] Logistic regression using SciPy

On 12/10/2012 12:00 AM, Fg Nu wrote:
> I am aware of the alternatives but am interested in rolling my own.


At least look at statsmodels,
because you are headed down the wrong track.

Thanks, but I have fixed my problem, and have posted back to the list.


From sturla at molden.no  Mon Dec 10 11:26:50 2012
From: sturla at molden.no (Sturla Molden)
Date: Mon, 10 Dec 2012 17:26:50 +0100
Subject: [SciPy-User] Logistic regression using SciPy
In-Reply-To: <1355153136.96410.YahooMailNeo@web160101.mail.bf1.yahoo.com>
References: <1355112090.68806.YahooMailNeo@web160105.mail.bf1.yahoo.com>
	<CALc6Xo6A-RqpfFArsW5oueFa53UM=b1E8vCwPubfcdtPvz6n+A@mail.gmail.com>
	<CAMMTP+A_6zMTzsxacdfHYtG3jZUeRVOJZVuQVPs-+Tq4tRVcGA@mail.gmail.com>
	<1355115639.66117.YahooMailNeo@web160106.mail.bf1.yahoo.com>
	<50C5E24A.8040201@gmail.com>
	<1355153136.96410.YahooMailNeo@web160101.mail.bf1.yahoo.com>
Message-ID: <AD9A57CA-433A-40FB-AFA3-7FAD5A6935F7@molden.no>

If it is of any interest, I have working code to do logistic regression with SciPy.

Sturla

Sendt fra min iPad

Den 10. des. 2012 kl. 16:25 skrev Fg Nu <fgnu32 at yahoo.com>:

> 
> 
> 
> 
> ----- Original Message -----
> From: Alan G Isaac <alan.isaac at gmail.com>
> To: Fg Nu <fgnu32 at yahoo.com>; SciPy Users List <scipy-user at scipy.org>
> Cc: 
> Sent: Monday, December 10, 2012 6:53 PM
> Subject: Re: [SciPy-User] Logistic regression using SciPy
> 
> On 12/10/2012 12:00 AM, Fg Nu wrote:
>> I am aware of the alternatives but am interested in rolling my own.
> 
> 
> At least look at statsmodels,
> because you are headed down the wrong track.
> 
> Thanks, but I have fixed my problem, and have posted back to the list.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From pav at iki.fi  Mon Dec 10 11:32:29 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Mon, 10 Dec 2012 16:32:29 +0000 (UTC)
Subject: [SciPy-User]
	=?utf-8?q?computing_f_and_fprime_in_one_evaluation_i?=
	=?utf-8?q?n=09scipy=2Eoptimize?=
References: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>
	<CAKF=DjujnOt1Ee55ZxsLbkecVNN2Q3UEOWCaNcTy7c9=zXvsYA@mail.gmail.com>
	<loom.20121210T160912-642@post.gmane.org>
	<CAHCaCkKQ-vKGxi7hNif7+iQuPiTTURPqWN3RiVqN5LAuOX4r5w@mail.gmail.com>
Message-ID: <loom.20121210T173015-367@post.gmane.org>

Matthieu Brucher <matthieu.brucher <at> gmail.com> writes:
> What is the Jacobian for a CG or BFGS optimization? There are no Jacobian in 
this case for non-least square optimization if I'm not mistaken (at least in no 
reference that I know of). Shouldn't this be the gradient function?

Yeah, it means the gradient. If you nit-pick enough, 
the gradient is the Jacobian of a scalar function.

    Pauli




From matthieu.brucher at gmail.com  Mon Dec 10 11:38:14 2012
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 10 Dec 2012 17:38:14 +0100
Subject: [SciPy-User] computing f and fprime in one evaluation in
	scipy.optimize
In-Reply-To: <loom.20121210T173015-367@post.gmane.org>
References: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>
	<CAKF=DjujnOt1Ee55ZxsLbkecVNN2Q3UEOWCaNcTy7c9=zXvsYA@mail.gmail.com>
	<loom.20121210T160912-642@post.gmane.org>
	<CAHCaCkKQ-vKGxi7hNif7+iQuPiTTURPqWN3RiVqN5LAuOX4r5w@mail.gmail.com>
	<loom.20121210T173015-367@post.gmane.org>
Message-ID: <CAHCaCkLYOFFToBn9ggMzvSGfMgoOKur-_Aj0_LsYZNyxfsN3Dw@mail.gmail.com>

OK, that's what I thought, but perhaps people expecting to find the word
gradient somewhere in the document would not make the connection :/

Cheers,


2012/12/10 Pauli Virtanen <pav at iki.fi>

> Matthieu Brucher <matthieu.brucher <at> gmail.com> writes:
> > What is the Jacobian for a CG or BFGS optimization? There are no
> Jacobian in
> this case for non-least square optimization if I'm not mistaken (at least
> in no
> reference that I know of). Shouldn't this be the gradient function?
>
> Yeah, it means the gradient. If you nit-pick enough,
> the gradient is the Jacobian of a scalar function.
>
>     Pauli
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121210/8e7418ba/attachment.html>

From sturla at molden.no  Mon Dec 10 12:04:15 2012
From: sturla at molden.no (Sturla Molden)
Date: Mon, 10 Dec 2012 18:04:15 +0100
Subject: [SciPy-User] computing f and fprime in one evaluation
	in	scipy.optimize
In-Reply-To: <loom.20121210T160912-642@post.gmane.org>
References: <12E86D26-1E67-4669-A123-E171BDEECDF8@gmail.com>
	<CAKF=DjujnOt1Ee55ZxsLbkecVNN2Q3UEOWCaNcTy7c9=zXvsYA@mail.gmail.com>
	<loom.20121210T160912-642@post.gmane.org>
Message-ID: <01D8D1CE-1910-405D-9AB6-3FA7D2B9CE3F@molden.no>

Den 10. des. 2012 kl. 16:11 skrev Pauli Virtanen <pav at iki.fi>:

> Skipper Seabold <jsseabold <at> gmail.com> writes:
> [clip]
>> AFAIK, fmin_l_bfgs_b is the only optimize function written so that f
>> can also return fprime, but I agree that this could be a nice option
>> to the other ones.
> 
> The good news is that minimize(f, .., jac=True) [1] also works like this,
> and works for all of the solvers (it caches the jacobian).


This is a problem I have run into with sp.optimize.leastsq as well. Caching did not solve the problem, as the Jacobian is rank ndata x nparam, and caching it made me run out of memory. Interestingly, MINPACK lmder.f obtains the residuals and the Jaobian in a single function call, thus the problem is not there in the original Fortran code. It is introduced by the SciPy wrapper. On the other hand, I also prefer to use LAPACK for least-squares solvers. I don't think the built-in QR in MINPACK is very efficient compared to e.g. MKL. (Which is why I seriously consider to write my own LM routine.)

Sturla

From klonuo at gmail.com  Tue Dec 11 07:03:04 2012
From: klonuo at gmail.com (klo uo)
Date: Tue, 11 Dec 2012 13:03:04 +0100
Subject: [SciPy-User] scipy.org is down
Message-ID: <CAA-8Ld8rmnYNi7DWqx3q-av=PUzbPGRZXxE7zYT2Rdongz6J7A@mail.gmail.com>

I can't access scipy.org and it seems it's not just me:

http://www.downforeveryoneorjustme.com/scipy.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121211/cd0038c8/attachment.html>

From klonuo at gmail.com  Tue Dec 11 07:05:48 2012
From: klonuo at gmail.com (klo uo)
Date: Tue, 11 Dec 2012 13:05:48 +0100
Subject: [SciPy-User] scipy.org is down
In-Reply-To: <CAA-8Ld8rmnYNi7DWqx3q-av=PUzbPGRZXxE7zYT2Rdongz6J7A@mail.gmail.com>
References: <CAA-8Ld8rmnYNi7DWqx3q-av=PUzbPGRZXxE7zYT2Rdongz6J7A@mail.gmail.com>
Message-ID: <CAA-8Ld_nCFR1=GS_pKpDXi8ay3J+S5T_hgWn-T4=+PNcn+Jbqw@mail.gmail.com>

Browsing mailing list seems like this is alternative:
http://scipy.github.com/
So perhaps currently there is some transition issue


On Tue, Dec 11, 2012 at 1:03 PM, klo uo <klonuo at gmail.com> wrote:

> I can't access scipy.org and it seems it's not just me:
>
> http://www.downforeveryoneorjustme.com/scipy.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121211/307ba9c9/attachment.html>

From takowl at gmail.com  Tue Dec 11 07:21:42 2012
From: takowl at gmail.com (Thomas Kluyver)
Date: Tue, 11 Dec 2012 12:21:42 +0000
Subject: [SciPy-User] scipy.org is down
In-Reply-To: <CAA-8Ld_nCFR1=GS_pKpDXi8ay3J+S5T_hgWn-T4=+PNcn+Jbqw@mail.gmail.com>
References: <CAA-8Ld8rmnYNi7DWqx3q-av=PUzbPGRZXxE7zYT2Rdongz6J7A@mail.gmail.com>
	<CAA-8Ld_nCFR1=GS_pKpDXi8ay3J+S5T_hgWn-T4=+PNcn+Jbqw@mail.gmail.com>
Message-ID: <CAOvn4qj9ZHm2u+FG_WJ0+N50trygSouQqU=+uuA=_3gede5k9w@mail.gmail.com>

On 11 December 2012 12:05, klo uo <klonuo at gmail.com> wrote:

> Browsing mailing list seems like this is alternative:
> http://scipy.github.com/
> So perhaps currently there is some transition issue
>

That's the new site, I'm hoping we'll be able to put it up as the main site
before long. Another advantage is that it's a static site, so hopefully
there's less to go wrong.

Unfortunately, the server for the current scipy.org site is rather
unreliable. I think a workaround was implemented that involved
automatically restarting the server regularly, so maybe it'll come back in
an hour or two.

Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121211/aa421b70/attachment.html>

From klonuo at gmail.com  Tue Dec 11 07:24:36 2012
From: klonuo at gmail.com (klo uo)
Date: Tue, 11 Dec 2012 13:24:36 +0100
Subject: [SciPy-User] scipy.org is down
In-Reply-To: <CAOvn4qj9ZHm2u+FG_WJ0+N50trygSouQqU=+uuA=_3gede5k9w@mail.gmail.com>
References: <CAA-8Ld8rmnYNi7DWqx3q-av=PUzbPGRZXxE7zYT2Rdongz6J7A@mail.gmail.com>
	<CAA-8Ld_nCFR1=GS_pKpDXi8ay3J+S5T_hgWn-T4=+PNcn+Jbqw@mail.gmail.com>
	<CAOvn4qj9ZHm2u+FG_WJ0+N50trygSouQqU=+uuA=_3gede5k9w@mail.gmail.com>
Message-ID: <CAA-8Ld9Yh3Aa0BV=4r_MUmKvXAxjUy+DLZW4+0nCdKE7Dz+r8A@mail.gmail.com>

OK, thanks Thomas
Google has the cache so it can be browsed that way


On Tue, Dec 11, 2012 at 1:21 PM, Thomas Kluyver <takowl at gmail.com> wrote:

> On 11 December 2012 12:05, klo uo <klonuo at gmail.com> wrote:
>
>> Browsing mailing list seems like this is alternative:
>> http://scipy.github.com/
>> So perhaps currently there is some transition issue
>>
>
> That's the new site, I'm hoping we'll be able to put it up as the main
> site before long. Another advantage is that it's a static site, so
> hopefully there's less to go wrong.
>
> Unfortunately, the server for the current scipy.org site is rather
> unreliable. I think a workaround was implemented that involved
> automatically restarting the server regularly, so maybe it'll come back in
> an hour or two.
>
> Thomas
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121211/f8bf43eb/attachment.html>

From j.l.anderson at phonecoop.coop  Tue Dec 11 10:13:08 2012
From: j.l.anderson at phonecoop.coop (Joseph Anderson)
Date: Tue, 11 Dec 2012 15:13:08 +0000
Subject: [SciPy-User] solving question
Message-ID: <82663A13-C785-400E-87C7-52F941347896@phonecoop.coop>

Hello All,

Hoping for some advice with this somewhat naive question...


I have a set of equations that look like this:

my_f(a0, x0) + my_f(a1, x1) + ... + my_f(aN, xN) - b0 = 0
my_f(a0, x0) + my_f(a1, x1) + ... + my_f(aN, xN) - b1 = 0
.
.
.
my_f(a0, x0) + my_f(a1, x1) + ... + my_f(aN, xN) - bN = 0


my_f() is my own function (some trig, I can post if that will help).

a0... aN and b0... bN are known. I need to find x0... xN.

Any advice on finding x0... xN using scipy?


Thanks in advance for you're help!!



My best,

~~
Joseph Anderson

Artist: 				http://joseph-anderson.org
Ambisonic Toolkit:	http://ambisonictoolkit.net




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121211/4802d59f/attachment.html>

From wkerzendorf at gmail.com  Wed Dec 12 17:08:26 2012
From: wkerzendorf at gmail.com (Wolfgang Kerzendorf)
Date: Wed, 12 Dec 2012 17:08:26 -0500
Subject: [SciPy-User] scipy interpolate.interp1d spline slowness
Message-ID: <6DF4A272-81C3-4A61-BFD0-BB7614448290@gmail.com>

Hello Scipyers,

I've just stumbled across a problem with interpolate.interp1d:
-------------
import numpy as np
from scipy import interpolate
x = arange(1000)
y =  y = np.random.random_integers(0, 900, 1000)

%timeit interp = interpolate.interp1d(x, y, kind='cubic')
1 loops, best of 3: 3.63 s per loop
#the call for the interpolation is really quick afterwards (a couple ms)

tck = interpolate.splrep(x, y, s=0)
%timeit interpolate.splev(x_new, tck, der=0)
100 loops, best of 3: 5.51 ms per loop

------
I do understand that these are different spline interpolations (but that's as far as my knowledge goes). I was just annoyed at the person saying: Ah, you see python is slow - which it is not as shown by the second scipy command. 

Would it be possible to switch the spline interpolator used in interpolate.interp1d to the B-Splines, or to give an option to switch between different spline interpolators (maybe with a warning: slow). 

Ah - a last question: Why don't you  use the issues tab on the github page? 

Thanks in advance,
   Wolfgang

From deshpande.jaidev at gmail.com  Wed Dec 12 17:22:10 2012
From: deshpande.jaidev at gmail.com (Jaidev Deshpande)
Date: Thu, 13 Dec 2012 03:52:10 +0530
Subject: [SciPy-User] scipy interpolate.interp1d spline slowness
In-Reply-To: <6DF4A272-81C3-4A61-BFD0-BB7614448290@gmail.com>
References: <6DF4A272-81C3-4A61-BFD0-BB7614448290@gmail.com>
Message-ID: <CAB=suEm9fvG22oj8c36R2NxBAwiP2Wjs25725n=uDbF3zox=iw@mail.gmail.com>

---------- Forwarded message ----------
From: Wolfgang Kerzendorf <wkerzendorf at gmail.com>
Date: Thu, Dec 13, 2012 at 3:38 AM
Subject: [SciPy-User] scipy interpolate.interp1d spline slowness
To: SciPy Users List <scipy-user at scipy.org>


Hello Scipyers,

I've just stumbled across a problem with interpolate.interp1d:
-------------
import numpy as np
from scipy import interpolate
x = arange(1000)
y =  y = np.random.random_integers(0, 900, 1000)

%timeit interp = interpolate.interp1d(x, y, kind='cubic')
1 loops, best of 3: 3.63 s per loop
#the call for the interpolation is really quick afterwards (a couple ms)

tck = interpolate.splrep(x, y, s=0)
%timeit interpolate.splev(x_new, tck, der=0)
100 loops, best of 3: 5.51 ms per loop

------
I do understand that these are different spline interpolations (but
that's as far as my knowledge goes). I was just annoyed at the person
saying: Ah, you see python is slow - which it is not as shown by the
second scipy command.

Would it be possible to switch the spline interpolator used in
interpolate.interp1d to the B-Splines, or to give an option to switch
between different spline interpolators (maybe with a warning: slow).

Hi,

I don't know how much I can help, but I've noticed that
scipy.interpolate.splrep and scipy.interpolate.splev are slightly
faster than interp1d.

It should suffice for cubic splines.

HTH

Ah - a last question: Why don't you  use the issues tab on the github page?

Thanks in advance,
   Wolfgang
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user


--
JD


From josef.pktd at gmail.com  Wed Dec 12 17:25:00 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 12 Dec 2012 17:25:00 -0500
Subject: [SciPy-User] scipy interpolate.interp1d spline slowness
In-Reply-To: <6DF4A272-81C3-4A61-BFD0-BB7614448290@gmail.com>
References: <6DF4A272-81C3-4A61-BFD0-BB7614448290@gmail.com>
Message-ID: <CAMMTP+APvNFpt_S6Arzx8mx5XQqNcvXfo+Z39pFrD0TZHo94mA@mail.gmail.com>

On Wed, Dec 12, 2012 at 5:08 PM, Wolfgang Kerzendorf
<wkerzendorf at gmail.com> wrote:
> Hello Scipyers,
>
> I've just stumbled across a problem with interpolate.interp1d:
> -------------
> import numpy as np
> from scipy import interpolate
> x = arange(1000)
> y =  y = np.random.random_integers(0, 900, 1000)
>
> %timeit interp = interpolate.interp1d(x, y, kind='cubic')
> 1 loops, best of 3: 3.63 s per loop
> #the call for the interpolation is really quick afterwards (a couple ms)
>
> tck = interpolate.splrep(x, y, s=0)
> %timeit interpolate.splev(x_new, tck, der=0)
> 100 loops, best of 3: 5.51 ms per loop

It looks to me, you are timing two different things here, with
interp1d you time the spline creation with splev you time the
evaluation.

for "cubic", interp1d uses _fitpack._bspleval  so I wouldn't expect
much difference in timing.
But I didn't check whether there is a difference in what the wrappers are doing

Josef



>
> ------
> I do understand that these are different spline interpolations (but that's as far as my knowledge goes). I was just annoyed at the person saying: Ah, you see python is slow - which it is not as shown by the second scipy command.
>
> Would it be possible to switch the spline interpolator used in interpolate.interp1d to the B-Splines, or to give an option to switch between different spline interpolators (maybe with a warning: slow).
>
> Ah - a last question: Why don't you  use the issues tab on the github page?
>
> Thanks in advance,
>    Wolfgang
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From wkerzendorf at gmail.com  Wed Dec 12 18:25:02 2012
From: wkerzendorf at gmail.com (Wolfgang Kerzendorf)
Date: Wed, 12 Dec 2012 18:25:02 -0500
Subject: [SciPy-User] scipy interpolate.interp1d spline slowness
In-Reply-To: <CAMMTP+APvNFpt_S6Arzx8mx5XQqNcvXfo+Z39pFrD0TZHo94mA@mail.gmail.com>
References: <6DF4A272-81C3-4A61-BFD0-BB7614448290@gmail.com>
	<CAMMTP+APvNFpt_S6Arzx8mx5XQqNcvXfo+Z39pFrD0TZHo94mA@mail.gmail.com>
Message-ID: <652C4822-1171-47EE-BE0D-0DFA27BD9DA5@gmail.com>

Hi Josef,

Your'e absolutely right - I just omitted the other timings as they seemed small compared to the interp1d:

%timeit interp(x_new)
100 loops, best of 3: 3.89 ms per loop

%timeit tck = interpolate.splrep(x, y, s=0)
1000 loops, best of 3: 204 us per loop

It would be great if you look into this.

Thanks
   Wolfgang
On 2012-12-12, at 5:25 PM, josef.pktd at gmail.com wrote:

> On Wed, Dec 12, 2012 at 5:08 PM, Wolfgang Kerzendorf
> <wkerzendorf at gmail.com> wrote:
>> Hello Scipyers,
>> 
>> I've just stumbled across a problem with interpolate.interp1d:
>> -------------
>> import numpy as np
>> from scipy import interpolate
>> x = arange(1000)
>> y =  y = np.random.random_integers(0, 900, 1000)
>> 
>> %timeit interp = interpolate.interp1d(x, y, kind='cubic')
>> 1 loops, best of 3: 3.63 s per loop
>> #the call for the interpolation is really quick afterwards (a couple ms)
>> 
>> tck = interpolate.splrep(x, y, s=0)
>> %timeit interpolate.splev(x_new, tck, der=0)
>> 100 loops, best of 3: 5.51 ms per loop
> 
> It looks to me, you are timing two different things here, with
> interp1d you time the spline creation with splev you time the
> evaluation.
> 
> for "cubic", interp1d uses _fitpack._bspleval  so I wouldn't expect
> much difference in timing.
> But I didn't check whether there is a difference in what the wrappers are doing
> 
> Josef
> 
> 
> 
>> 
>> ------
>> I do understand that these are different spline interpolations (but that's as far as my knowledge goes). I was just annoyed at the person saying: Ah, you see python is slow - which it is not as shown by the second scipy command.
>> 
>> Would it be possible to switch the spline interpolator used in interpolate.interp1d to the B-Splines, or to give an option to switch between different spline interpolators (maybe with a warning: slow).
>> 
>> Ah - a last question: Why don't you  use the issues tab on the github page?
>> 
>> Thanks in advance,
>>   Wolfgang
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From wkerzendorf at gmail.com  Wed Dec 12 18:26:49 2012
From: wkerzendorf at gmail.com (Wolfgang Kerzendorf)
Date: Wed, 12 Dec 2012 18:26:49 -0500
Subject: [SciPy-User] scipy interpolate.interp1d spline slowness
In-Reply-To: <652C4822-1171-47EE-BE0D-0DFA27BD9DA5@gmail.com>
References: <6DF4A272-81C3-4A61-BFD0-BB7614448290@gmail.com>
	<CAMMTP+APvNFpt_S6Arzx8mx5XQqNcvXfo+Z39pFrD0TZHo94mA@mail.gmail.com>
	<652C4822-1171-47EE-BE0D-0DFA27BD9DA5@gmail.com>
Message-ID: <B08000B9-641B-4B2A-9535-591DFD62B717@gmail.com>

Sorry again, i should include that interp came from: interp = interpolate.interp1d(x, y, kind='cubic')
Cheers
   W
On 2012-12-12, at 6:25 PM, Wolfgang Kerzendorf <wkerzendorf at gmail.com> wrote:

> Hi Josef,
> 
> Your'e absolutely right - I just omitted the other timings as they seemed small compared to the interp1d:
> 
> %timeit interp(x_new)
> 100 loops, best of 3: 3.89 ms per loop
> 
> %timeit tck = interpolate.splrep(x, y, s=0)
> 1000 loops, best of 3: 204 us per loop
> 
> It would be great if you look into this.
> 
> Thanks
>   Wolfgang
> On 2012-12-12, at 5:25 PM, josef.pktd at gmail.com wrote:
> 
>> On Wed, Dec 12, 2012 at 5:08 PM, Wolfgang Kerzendorf
>> <wkerzendorf at gmail.com> wrote:
>>> Hello Scipyers,
>>> 
>>> I've just stumbled across a problem with interpolate.interp1d:
>>> -------------
>>> import numpy as np
>>> from scipy import interpolate
>>> x = arange(1000)
>>> y =  y = np.random.random_integers(0, 900, 1000)
>>> 
>>> %timeit interp = interpolate.interp1d(x, y, kind='cubic')
>>> 1 loops, best of 3: 3.63 s per loop
>>> #the call for the interpolation is really quick afterwards (a couple ms)
>>> 
>>> tck = interpolate.splrep(x, y, s=0)
>>> %timeit interpolate.splev(x_new, tck, der=0)
>>> 100 loops, best of 3: 5.51 ms per loop
>> 
>> It looks to me, you are timing two different things here, with
>> interp1d you time the spline creation with splev you time the
>> evaluation.
>> 
>> for "cubic", interp1d uses _fitpack._bspleval  so I wouldn't expect
>> much difference in timing.
>> But I didn't check whether there is a difference in what the wrappers are doing
>> 
>> Josef
>> 
>> 
>> 
>>> 
>>> ------
>>> I do understand that these are different spline interpolations (but that's as far as my knowledge goes). I was just annoyed at the person saying: Ah, you see python is slow - which it is not as shown by the second scipy command.
>>> 
>>> Would it be possible to switch the spline interpolator used in interpolate.interp1d to the B-Splines, or to give an option to switch between different spline interpolators (maybe with a warning: slow).
>>> 
>>> Ah - a last question: Why don't you  use the issues tab on the github page?
>>> 
>>> Thanks in advance,
>>>  Wolfgang
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> 



From josef.pktd at gmail.com  Wed Dec 12 19:10:41 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 12 Dec 2012 19:10:41 -0500
Subject: [SciPy-User] scipy interpolate.interp1d spline slowness
In-Reply-To: <B08000B9-641B-4B2A-9535-591DFD62B717@gmail.com>
References: <6DF4A272-81C3-4A61-BFD0-BB7614448290@gmail.com>
	<CAMMTP+APvNFpt_S6Arzx8mx5XQqNcvXfo+Z39pFrD0TZHo94mA@mail.gmail.com>
	<652C4822-1171-47EE-BE0D-0DFA27BD9DA5@gmail.com>
	<B08000B9-641B-4B2A-9535-591DFD62B717@gmail.com>
Message-ID: <CAMMTP+AsRTVXBAh4L8rB01La4FFn0TY-m8afuGP_AY0u1xabvg@mail.gmail.com>

Can you please follow our convention and post at bottom or inline.

On Wed, Dec 12, 2012 at 6:26 PM, Wolfgang Kerzendorf
<wkerzendorf at gmail.com> wrote:
> Sorry again, i should include that interp came from: interp = interpolate.interp1d(x, y, kind='cubic')
> Cheers
>    W
> On 2012-12-12, at 6:25 PM, Wolfgang Kerzendorf <wkerzendorf at gmail.com> wrote:
>
>> Hi Josef,
>>
>> Your'e absolutely right - I just omitted the other timings as they seemed small compared to the interp1d:
>>
>> %timeit interp(x_new)
>> 100 loops, best of 3: 3.89 ms per loop

that looks faster than what you had before :

>> %timeit interpolate.splev(x_new, tck, der=0)
>> 100 loops, best of 3: 5.51 ms per loop

I don't understand the setup code for interp1d cubic, _find_smoothest
for interpolation ?
but that looks slow in your timing.

>>
>> %timeit tck = interpolate.splrep(x, y, s=0)
>> 1000 loops, best of 3: 204 us per loop
>>
>> It would be great if you look into this.

Sorry, not me, I don't have time for this.

I went recently roughly through the source for interp1d "linear", and
so the other parts just on the side.

I have never seen this extra spline code in interpolate.py
there are a lot of potentially nice options
'natural', 'second','clamped', 'endslope', 'first', 'not-a-knot',
'runout', 'parabolic'
and a lot of NotImplementedError and one more wrapper around _fitpack

It looks like a major undertaking to just find one's way around the
splines and fitpack wrapper codes.

I stick to UnivariateSplines for "cubic".

Josef

>>
>> Thanks
>>   Wolfgang
>> On 2012-12-12, at 5:25 PM, josef.pktd at gmail.com wrote:
>>
>>> On Wed, Dec 12, 2012 at 5:08 PM, Wolfgang Kerzendorf
>>> <wkerzendorf at gmail.com> wrote:
>>>> Hello Scipyers,
>>>>
>>>> I've just stumbled across a problem with interpolate.interp1d:
>>>> -------------
>>>> import numpy as np
>>>> from scipy import interpolate
>>>> x = arange(1000)
>>>> y =  y = np.random.random_integers(0, 900, 1000)
>>>>
>>>> %timeit interp = interpolate.interp1d(x, y, kind='cubic')
>>>> 1 loops, best of 3: 3.63 s per loop
>>>> #the call for the interpolation is really quick afterwards (a couple ms)
>>>>
>>>> tck = interpolate.splrep(x, y, s=0)
>>>> %timeit interpolate.splev(x_new, tck, der=0)
>>>> 100 loops, best of 3: 5.51 ms per loop
>>>
>>> It looks to me, you are timing two different things here, with
>>> interp1d you time the spline creation with splev you time the
>>> evaluation.
>>>
>>> for "cubic", interp1d uses _fitpack._bspleval  so I wouldn't expect
>>> much difference in timing.
>>> But I didn't check whether there is a difference in what the wrappers are doing
>>>
>>> Josef
>>>
>>>
>>>
>>>>
>>>> ------
>>>> I do understand that these are different spline interpolations (but that's as far as my knowledge goes). I was just annoyed at the person saying: Ah, you see python is slow - which it is not as shown by the second scipy command.
>>>>
>>>> Would it be possible to switch the spline interpolator used in interpolate.interp1d to the B-Splines, or to give an option to switch between different spline interpolators (maybe with a warning: slow).
>>>>
>>>> Ah - a last question: Why don't you  use the issues tab on the github page?
>>>>
>>>> Thanks in advance,
>>>>  Wolfgang
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From pav at iki.fi  Thu Dec 13 07:18:19 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 13 Dec 2012 12:18:19 +0000 (UTC)
Subject: [SciPy-User] scipy interpolate.interp1d spline slowness
References: <6DF4A272-81C3-4A61-BFD0-BB7614448290@gmail.com>
Message-ID: <loom.20121213T131438-811@post.gmane.org>

Wolfgang Kerzendorf <wkerzendorf <at> gmail.com> writes:
> I've just stumbled across a problem with interpolate.interp1d:
> -------------
> import numpy as np
> from scipy import interpolate
> x = arange(1000)
> y =  y = np.random.random_integers(0, 900, 1000)
> 
> %timeit interp = interpolate.interp1d(x, y, kind='cubic')
> 1 loops, best of 3: 3.63 s per loop

This is indeed slow. The problem is probably that the spline fitting
routine splmake() does not make use of the bandedness of the spline
interpolation matrix, and so it is inefficient.

Switching to FITPACK splines could be a better option --- I think the
duplicated (and partial) spline functionality is not very useful.

-- 
Pauli Virtanen



From perry at stsci.edu  Thu Dec 13 09:37:58 2012
From: perry at stsci.edu (Perry Greenfield)
Date: Thu, 13 Dec 2012 09:37:58 -0500
Subject: [SciPy-User] JOB: positions available at STScI
References: <mailman.71.1355403641.1036.scipy-dev@scipy.org>
Message-ID: <76734032-916A-43E9-A363-F04A94A71426@stsci.edu>



The Science Software Branch at the Space Telescope Science Institute has openings for python developers. If you are interested please apply through this link:

https://rn11.ultipro.com/SPA1004/JobBoard/JobDetails.aspx?__ID=*41BCE81D823F3F8C

The relevant section is the one labeled "Science Data Calibration and Data Analysis"

Although the position is advertised as a junior position, candidates with more experience may be considered for higher level positions.







From bgamari.foss at gmail.com  Thu Dec 13 12:30:41 2012
From: bgamari.foss at gmail.com (Ben Gamari)
Date: Thu, 13 Dec 2012 12:30:41 -0500
Subject: [SciPy-User] scipy.stats.distributions lacking categorical and
	multinomial?
Message-ID: <87txrpuaf2.fsf@gmail.com>

Is there any particular reason for the categorical (generalized
Bernoulli) and multinomial (generalized binomial) distribution not being
included in scipy.stats.distributions?

Cheers,

- Ben




From robert.kern at gmail.com  Thu Dec 13 12:36:50 2012
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 13 Dec 2012 17:36:50 +0000
Subject: [SciPy-User] scipy.stats.distributions lacking categorical and
	multinomial?
In-Reply-To: <87txrpuaf2.fsf@gmail.com>
References: <87txrpuaf2.fsf@gmail.com>
Message-ID: <CAF6FJismnT819QQNLgmpcteAPOddfcY0Jk3c80j68pZgUTuPRA@mail.gmail.com>

On Thu, Dec 13, 2012 at 5:30 PM, Ben Gamari <bgamari.foss at gmail.com> wrote:
> Is there any particular reason for the categorical (generalized
> Bernoulli) and multinomial (generalized binomial) distribution not being
> included in scipy.stats.distributions?

scipy.stats.distributions only provides a framework for univariate
distributions. It's not clear what the interfaces should be for
multivariate distributions, but you are welcome to make proposals.

--
Robert Kern


From jsseabold at gmail.com  Thu Dec 13 12:44:36 2012
From: jsseabold at gmail.com (Skipper Seabold)
Date: Thu, 13 Dec 2012 12:44:36 -0500
Subject: [SciPy-User] scipy.stats.distributions lacking categorical and
	multinomial?
In-Reply-To: <CAF6FJismnT819QQNLgmpcteAPOddfcY0Jk3c80j68pZgUTuPRA@mail.gmail.com>
References: <87txrpuaf2.fsf@gmail.com>
	<CAF6FJismnT819QQNLgmpcteAPOddfcY0Jk3c80j68pZgUTuPRA@mail.gmail.com>
Message-ID: <CAKF=Dju+yufUt0Tc9YutfZopgjr3No_Vtv3dXZ7bEF8XVd1r_A@mail.gmail.com>

On Thu, Dec 13, 2012 at 12:36 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On Thu, Dec 13, 2012 at 5:30 PM, Ben Gamari <bgamari.foss at gmail.com> wrote:
>> Is there any particular reason for the categorical (generalized
>> Bernoulli) and multinomial (generalized binomial) distribution not being
>> included in scipy.stats.distributions?
>
> scipy.stats.distributions only provides a framework for univariate
> distributions. It's not clear what the interfaces should be for
> multivariate distributions, but you are welcome to make proposals.
>

I don't know what your needs are, but FWIW, I've been able to get by
using np.random.multinomial for sampling and either rolling my own
functions or using PyMC for the included categorical_like and
multinomial_like.

Skipper


From jjstickel at gmail.com  Thu Dec 13 15:50:24 2012
From: jjstickel at gmail.com (Jonathan Stickel)
Date: Thu, 13 Dec 2012 13:50:24 -0700
Subject: [SciPy-User] scipy interpolate.interp1d spline slowness
In-Reply-To: <mailman.81.1355419563.1036.scipy-user@scipy.org>
References: <mailman.81.1355419563.1036.scipy-user@scipy.org>
Message-ID: <50CA3F90.3080008@gmail.com>

On 12/13/12 10:26 , scipy-user-request at scipy.org wrote:
> Date: Wed, 12 Dec 2012 17:25:00 -0500
> From:josef
> Subject: Re: [SciPy-User] scipy interpolate.interp1d spline slowness
> To: SciPy Users List<scipy-user at scipy.org>
>
> On Wed, Dec 12, 2012 at 5:08 PM, Wolfgang Kerzendorf
> <wkerzendorf at gmail.com>  wrote:
>> >Hello Scipyers,
>> >
>> >I've just stumbled across a problem with interpolate.interp1d:
>> >-------------
>> >import numpy as np
>> >from scipy import interpolate
>> >x = arange(1000)
>> >y =  y = np.random.random_integers(0, 900, 1000)
>> >
>> >%timeit interp = interpolate.interp1d(x, y, kind='cubic')
>> >1 loops, best of 3: 3.63 s per loop
>> >#the call for the interpolation is really quick afterwards (a couple ms)
>> >
>> >tck = interpolate.splrep(x, y, s=0)
>> >%timeit interpolate.splev(x_new, tck, der=0)
>> >100 loops, best of 3: 5.51 ms per loop
> It looks to me, you are timing two different things here, with
> interp1d you time the spline creation with splev you time the
> evaluation.
>
> for "cubic", interp1d uses _fitpack._bspleval  so I wouldn't expect
> much difference in timing.
> But I didn't check whether there is a difference in what the wrappers are doing
>

I think this is the appropriate comparison:

In [1]: import numpy as np
In [2]: from scipy import interpolate as ipt
In [3]: x = np.arange(1000)
In [4]: y = 100*np.random.rand(1000)
In [5]: xp = x[:-1] + 0.5

In [6]: %timeit yp1 = ipt.interp1d(x,y,kind=3)(xp)
1 loops, best of 3: 5.08 s per loop

In [7]: %timeit yp2 = ipt.splev( xp, ipt.splrep(x,y,k=3) )
1000 loops, best of 3: 326 us per loop

I also don't claim to know the details of the two implementations, but I 
now use splrep/splev for any cubic spline interpolation of more than a 
few points.

Jonathan



From pawel.kw at gmail.com  Fri Dec 14 07:06:09 2012
From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=)
Date: Fri, 14 Dec 2012 13:06:09 +0100
Subject: [SciPy-User] cubic spline interpolation - derivative value in
 an end point
In-Reply-To: <loom.20121127T095758-94@post.gmane.org>
References: <loom.20121123T100600-60@post.gmane.org>
	<CAAih6W=Jzqkz0sqeyQUr=3mTYoe52U1SQAZ1ODCs6f47zWpmWQ@mail.gmail.com>
	<k8oa8j$hkr$1@ger.gmane.org> <loom.20121126T092313-316@post.gmane.org>
	<k90uu6$2qf$1@ger.gmane.org> <loom.20121127T095758-94@post.gmane.org>
Message-ID: <CAAih6Wkk1KeLnceitM61+FUTi86uzjAnm5sQCtRBJ=P4ovFwpA@mail.gmail.com>

Hi,

Actually, I'm having a similar problem now. Your hack should do the job -
thanks Eric.

Cheers,

Pawe?



2012/11/27 Maxim <spamaxx at gmail.com>

> Thank you a lot, Eric!
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121214/2755ffa5/attachment.html>

From philip at semanchuk.com  Fri Dec 14 09:34:57 2012
From: philip at semanchuk.com (Philip Semanchuk)
Date: Fri, 14 Dec 2012 09:34:57 -0500
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
Message-ID: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>

Hi all,
I'm porting some Fortran code that makes use of a number of BLAS and LAPACK functions, including dbdsqr(). I've found all of the functions I need via scipy.linalg.lapack.get_lapack_funcs/get_blas_funcs() except for dbdsqr(). 

I see that the numpy source code (I looked at numpy-1.6.0b2) contains dbdsqr() in numpy/linalg/dlapack_lite.c, but grep can't find that string in the binary distribution on my Mac nor on Linux. If it's buried in a numpy binary somewhere, I'm comfortable with using ctypes to call it, but I suspect it isn't.

Can anyone point me to a cross-platform (OS X, Linux & Windows) way I can call this function without compiling code myself?

I'm unfortunately quite na?ve about the math in the code I'm porting, so I'm porting the code blindly -- if you ask me what problem I'm trying to solve with dbdsqr(), I won't be able to explain.

Thanks in advance for any suggestions,
Philip

PS - Please pardon if you already saw this question on the numpy list. 

From jjhelmus at gmail.com  Fri Dec 14 10:38:35 2012
From: jjhelmus at gmail.com (Jonathan Helmus)
Date: Fri, 14 Dec 2012 10:38:35 -0500
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
Message-ID: <50CB47FB.9090404@gmail.com>

On 12/14/2012 09:34 AM, Philip Semanchuk wrote:
> Hi all,
> I'm porting some Fortran code that makes use of a number of BLAS and LAPACK functions, including dbdsqr(). I've found all of the functions I need via scipy.linalg.lapack.get_lapack_funcs/get_blas_funcs() except for dbdsqr().
>
> I see that the numpy source code (I looked at numpy-1.6.0b2) contains dbdsqr() in numpy/linalg/dlapack_lite.c, but grep can't find that string in the binary distribution on my Mac nor on Linux. If it's buried in a numpy binary somewhere, I'm comfortable with using ctypes to call it, but I suspect it isn't.
>
> Can anyone point me to a cross-platform (OS X, Linux&  Windows) way I can call this function without compiling code myself?
>
> I'm unfortunately quite na?ve about the math in the code I'm porting, so I'm porting the code blindly -- if you ask me what problem I'm trying to solve with dbdsqr(), I won't be able to explain.
>
> Thanks in advance for any suggestions,
> Philip
Philip,

Not sure if this is portable or is possible on all or even most Scipy 
installs but I can find a pointer to the LAPACK dbdsqr function in the 
clapack/flapack shared library using ctypes on my EDP 7.3.1 rh5 install:


In [1]: from ctypes import *

In [2]: import scipy.linalg

In [3]: lib = CDLL(scipy.linalg.lapack.clapack.__file__)

In [4]: lib.dbdsqr
Out[4]: <_FuncPtr object at 0x2d306d0>

This also works with the scipy.linalg.lapack.flapack.__file__ and 
np.linalg.lapack_lite.__file__.  I would think it would be possible to 
create a Python wrapper around this with the ctypes module (>= Python 
2.5) which properly defines the arguments and returned types, etc but it 
is not ideal.

     Cheers,

     - Jonathan Helmus


From sturla at molden.no  Fri Dec 14 10:42:44 2012
From: sturla at molden.no (Sturla Molden)
Date: Fri, 14 Dec 2012 16:42:44 +0100
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
Message-ID: <50CB48F4.3000007@molden.no>

If you are building SciPy from source, just add the interface to 
scipy/linalg/flapack.pyf.src and scipy.linalg.lapack.get_lapack_funcs 
will get it for you. I would be nice if SciPy exposed all of LAPACK, 
even the parts it doesn't use, but currently it don't.

If you're not building SciPy from source, yopu can e.g. write a .pyf 
file for dbdsqr and use f2py and and link with your LAPACK library. If 
you prefer to avoid Fortran, but already have a LAPACK library, you can 
also use LAPACKE with Cython (or ctypes).

Sturla



On 14.12.2012 15:34, Philip Semanchuk wrote:
> Hi all,
> I'm porting some Fortran code that makes use of a number of BLAS and LAPACK functions, including dbdsqr(). I've found all of the functions I need via scipy.linalg.lapack.get_lapack_funcs/get_blas_funcs() except for dbdsqr().
>
> I see that the numpy source code (I looked at numpy-1.6.0b2) contains dbdsqr() in numpy/linalg/dlapack_lite.c, but grep can't find that string in the binary distribution on my Mac nor on Linux. If it's buried in a numpy binary somewhere, I'm comfortable with using ctypes to call it, but I suspect it isn't.
>
> Can anyone point me to a cross-platform (OS X, Linux&  Windows) way I can call this function without compiling code myself?
>
> I'm unfortunately quite na?ve about the math in the code I'm porting, so I'm porting the code blindly -- if you ask me what problem I'm trying to solve with dbdsqr(), I won't be able to explain.
>
> Thanks in advance for any suggestions,
> Philip
>
> PS - Please pardon if you already saw this question on the numpy list.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From philip at semanchuk.com  Fri Dec 14 10:43:52 2012
From: philip at semanchuk.com (Philip Semanchuk)
Date: Fri, 14 Dec 2012 10:43:52 -0500
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <50CB47FB.9090404@gmail.com>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB47FB.9090404@gmail.com>
Message-ID: <1C358F5F-1D30-4ED3-B868-0B6D37E85521@semanchuk.com>


On Dec 14, 2012, at 10:38 AM, Jonathan Helmus wrote:

> On 12/14/2012 09:34 AM, Philip Semanchuk wrote:
>> Hi all,
>> I'm porting some Fortran code that makes use of a number of BLAS and LAPACK functions, including dbdsqr(). I've found all of the functions I need via scipy.linalg.lapack.get_lapack_funcs/get_blas_funcs() except for dbdsqr().
>> 
>> I see that the numpy source code (I looked at numpy-1.6.0b2) contains dbdsqr() in numpy/linalg/dlapack_lite.c, but grep can't find that string in the binary distribution on my Mac nor on Linux. If it's buried in a numpy binary somewhere, I'm comfortable with using ctypes to call it, but I suspect it isn't.
>> 
>> Can anyone point me to a cross-platform (OS X, Linux&  Windows) way I can call this function without compiling code myself?
>> 
>> I'm unfortunately quite na?ve about the math in the code I'm porting, so I'm porting the code blindly -- if you ask me what problem I'm trying to solve with dbdsqr(), I won't be able to explain.
>> 
>> Thanks in advance for any suggestions,
>> Philip
> Philip,
> 
> Not sure if this is portable or is possible on all or even most Scipy 
> installs but I can find a pointer to the LAPACK dbdsqr function in the 
> clapack/flapack shared library using ctypes on my EDP 7.3.1 rh5 install:
> 
> 
> In [1]: from ctypes import *
> 
> In [2]: import scipy.linalg
> 
> In [3]: lib = CDLL(scipy.linalg.lapack.clapack.__file__)
> 
> In [4]: lib.dbdsqr
> Out[4]: <_FuncPtr object at 0x2d306d0>
> 
> This also works with the scipy.linalg.lapack.flapack.__file__ and 
> np.linalg.lapack_lite.__file__.  I would think it would be possible to 
> create a Python wrapper around this with the ctypes module (>= Python 
> 2.5) which properly defines the arguments and returned types, etc but it 
> is not ideal.

Awesome! Believe me, this is ideal compared to the alternatives.

Your code also works on my OS X install using scipy 0.11. It doesn't work under Linux Mint 13 w/scipy 0.9.0 nor WinXP w/scipy 0.10.0.

What's odd is that the latest source code for scipy only has one reference to this function, and it is documentation saying "Not Implemented". I'll have to experiment a little to see if this is coming to scipy via numpy.

Thank you so much, Jonathan. And for nmrglue, too ;)


Cheers
Philip




From philip at semanchuk.com  Fri Dec 14 10:48:09 2012
From: philip at semanchuk.com (Philip Semanchuk)
Date: Fri, 14 Dec 2012 10:48:09 -0500
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <50CB48F4.3000007@molden.no>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB48F4.3000007@molden.no>
Message-ID: <FF148DD9-0C26-4647-A12B-02521E7F4DB2@semanchuk.com>


On Dec 14, 2012, at 10:42 AM, Sturla Molden wrote:

> If you are building SciPy from source, just add the interface to 
> scipy/linalg/flapack.pyf.src and scipy.linalg.lapack.get_lapack_funcs 
> will get it for you. I would be nice if SciPy exposed all of LAPACK, 
> even the parts it doesn't use, but currently it don't.
> 
> If you're not building SciPy from source, yopu can e.g. write a .pyf 
> file for dbdsqr and use f2py and and link with your LAPACK library. If 
> you prefer to avoid Fortran, but already have a LAPACK library, you can 
> also use LAPACKE with Cython (or ctypes).

Thanks, Sturla. Our code is distributed to others as part of an application suite. We don't ask our users to compile anything, nor can we necessarily predict what libraries will be available on the machine outside of our app's prerequisites. Our app requires numpy and scipy, so I was really hoping to get at it from one of those two libraries.

Cheers
Philip


> On 14.12.2012 15:34, Philip Semanchuk wrote:
>> Hi all,
>> I'm porting some Fortran code that makes use of a number of BLAS and LAPACK functions, including dbdsqr(). I've found all of the functions I need via scipy.linalg.lapack.get_lapack_funcs/get_blas_funcs() except for dbdsqr().
>> 
>> I see that the numpy source code (I looked at numpy-1.6.0b2) contains dbdsqr() in numpy/linalg/dlapack_lite.c, but grep can't find that string in the binary distribution on my Mac nor on Linux. If it's buried in a numpy binary somewhere, I'm comfortable with using ctypes to call it, but I suspect it isn't.
>> 
>> Can anyone point me to a cross-platform (OS X, Linux&  Windows) way I can call this function without compiling code myself?
>> 
>> I'm unfortunately quite na?ve about the math in the code I'm porting, so I'm porting the code blindly -- if you ask me what problem I'm trying to solve with dbdsqr(), I won't be able to explain.
>> 
>> Thanks in advance for any suggestions,
>> Philip
>> 
>> PS - Please pardon if you already saw this question on the numpy list.
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From sturla at molden.no  Fri Dec 14 10:54:22 2012
From: sturla at molden.no (Sturla Molden)
Date: Fri, 14 Dec 2012 16:54:22 +0100
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <50CB47FB.9090404@gmail.com>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB47FB.9090404@gmail.com>
Message-ID: <50CB4BAE.1050105@molden.no>

On 14.12.2012 16:38, Jonathan Helmus wrote:

> Not sure if this is portable or is possible on all or even most Scipy
> installs but I can find a pointer to the LAPACK dbdsqr function in the
> clapack/flapack shared library using ctypes on my EDP 7.3.1 rh5 install:
>
>
> In [1]: from ctypes import *
>
> In [2]: import scipy.linalg
>
> In [3]: lib = CDLL(scipy.linalg.lapack.clapack.__file__)

That will not be cross-platform, but depend on ABI for the Fortran 
compiler used on the particular platform. Cross-platform ABI dependency 
is a PITA.

CLAPACK is also abandonware, LAPACKE is the current C/C++ interface. 
Thus with any recent LAPACK, there will be no clapack in SciPy.

I hope that SciPy at some point could start to use LAPACKE.

Another cross-platform solution is to use the Fortran 2003 ISO C binding 
and Cython (i.e. fwrap instead of f2py).

Sturla









From p.zaffino at yahoo.it  Fri Dec 14 10:57:57 2012
From: p.zaffino at yahoo.it (Paolo Zaffino)
Date: Fri, 14 Dec 2012 15:57:57 +0000 (GMT)
Subject: [SciPy-User] Points fitting (non lin)
Message-ID: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>

Dear Scipy community,

I have a set of points (2D) and I would compute a curve that fits them.
The points are ordered in a precise way (not crescent order) and I can't change this order (the curve should fit the points in that order).
I'm interseting in a non linear fit (the ideal case would be more intervals of quadratic curves).
Has anyone any advice about?

Thank you very much.
Regards.
Paolo ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121214/8845cd6e/attachment.html>

From philip at semanchuk.com  Fri Dec 14 11:03:11 2012
From: philip at semanchuk.com (Philip Semanchuk)
Date: Fri, 14 Dec 2012 11:03:11 -0500
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <50CB4BAE.1050105@molden.no>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB47FB.9090404@gmail.com> <50CB4BAE.1050105@molden.no>
Message-ID: <BF9B8C23-B799-4773-9DAD-CC1606F37717@semanchuk.com>


On Dec 14, 2012, at 10:54 AM, Sturla Molden wrote:

> On 14.12.2012 16:38, Jonathan Helmus wrote:
> 
>> Not sure if this is portable or is possible on all or even most Scipy
>> installs but I can find a pointer to the LAPACK dbdsqr function in the
>> clapack/flapack shared library using ctypes on my EDP 7.3.1 rh5 install:
>> 
>> 
>> In [1]: from ctypes import *
>> 
>> In [2]: import scipy.linalg
>> 
>> In [3]: lib = CDLL(scipy.linalg.lapack.clapack.__file__)
> 
> That will not be cross-platform, but depend on ABI for the Fortran 
> compiler used on the particular platform. Cross-platform ABI dependency 
> is a PITA.
> 
> CLAPACK is also abandonware, LAPACKE is the current C/C++ interface. 
> Thus with any recent LAPACK, there will be no clapack in SciPy.

Hmmm, well that's disappointing to hear, but better to know now than finding out the hard way later. 

Maybe a pure-Python + numpy/scipy solution for this just isn't possible right now. 

Tack/Thanks
P




From sturla at molden.no  Fri Dec 14 11:07:44 2012
From: sturla at molden.no (Sturla Molden)
Date: Fri, 14 Dec 2012 17:07:44 +0100
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <FF148DD9-0C26-4647-A12B-02521E7F4DB2@semanchuk.com>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB48F4.3000007@molden.no>
	<FF148DD9-0C26-4647-A12B-02521E7F4DB2@semanchuk.com>
Message-ID: <50CB4ED0.1070203@molden.no>

On 14.12.2012 16:48, Philip Semanchuk wrote:

> Thanks, Sturla. Our code is distributed to others as part of an application suite. We don't ask our users to compile anything, nor can we necessarily predict what libraries will be available on the machine outside of our app's prerequisites. Our app requires numpy and scipy, so I was really hoping to get at it from one of those two libraries.

I know, I constantly run into the same issue.

For example with the latest Enthought Python, I do this to get LAPACK:

intel_mkl = ctypes.CDLL('mk2_rt.dll')

Then I can call LAPACKE functions in the MKL library:

LAPACKE_dbdsqr = intel_mkl.LAPACKE_dbdsqr


And if I want to use it from Cython, I must do this conversion:

    <void*> (<Py_intptr_t> ctypes.cast(LAPACKE_dbdsqr, 
ctypes.c_void_p).value)

which gives me the address of LAPACKE_dbdsqr in a void*.



Sturla



From pawel.kw at gmail.com  Fri Dec 14 11:25:31 2012
From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=)
Date: Fri, 14 Dec 2012 17:25:31 +0100
Subject: [SciPy-User] Points fitting (non lin)
In-Reply-To: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>
References: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>
Message-ID: <CAAih6WmEZVRkmoVD6R0nuXT_kfCS256KZJ+8XmoZ==4RYXN33g@mail.gmail.com>

Dear Paolo,

I'm not sure I understand correctly your problem, but this sounds like a
spline fitting job. You can read more about this here:
http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html

Is this what you are looking for?

Cheers,

Pawe?



2012/12/14 Paolo Zaffino <p.zaffino at yahoo.it>

> Dear Scipy community,
>
> I have a set of points (2D) and I would compute a curve that fits them.
> The points are ordered in a precise way (not crescent order) and I can't
> change this order (the curve should fit the points in that order).
> I'm interseting in a non linear fit (the ideal case would be more
> intervals of quadratic curves).
> Has anyone any advice about?
>
> Thank you very much.
> Regards.
> Paolo
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121214/8948574d/attachment.html>

From pav at iki.fi  Fri Dec 14 11:30:11 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 14 Dec 2012 16:30:11 +0000 (UTC)
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB48F4.3000007@molden.no>
	<FF148DD9-0C26-4647-A12B-02521E7F4DB2@semanchuk.com>
	<50CB4ED0.1070203@molden.no>
Message-ID: <loom.20121214T172921-116@post.gmane.org>

Sturla Molden <sturla <at> molden.no> writes:
> On 14.12.2012 16:48, Philip Semanchuk wrote:
> 
> > Thanks, Sturla. Our code is distributed to others as part of an application 
suite. We don't ask our users to
> compile anything, nor can we necessarily predict what libraries will be 
available on the machine outside
> of our app's prerequisites. Our app requires numpy and scipy, so I was really 
hoping to get at it from one of
> those two libraries.
> 
> I know, I constantly run into the same issue.

I think this means that someone needs to go through the LAPACK
wrappers, and add any functions that are missing.

Pull requests are accepted ;)

-- 
Pauli Virtanen



From pav at iki.fi  Fri Dec 14 11:33:07 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 14 Dec 2012 16:33:07 +0000 (UTC)
Subject: [SciPy-User] Points fitting (non lin)
References: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>
Message-ID: <loom.20121214T173113-88@post.gmane.org>

Paolo Zaffino <p.zaffino <at> yahoo.it> writes:
> I have a set of points (2D) and I would compute a curve that fits them.
> The points are ordered in a precise way (not crescent order) and
> I can't change this order (the curve should fit the points in that order).
> I'm interseting in a non linear fit (the ideal case would be more
> intervals of quadratic curves).
> Has anyone any advice about?

splprep may be the function you are looking for, it fits a parametric
spline to a set of points.

Read the documentation on how to choose the smoothing `s` parameter.

http://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.splprep.htm
l

-- 
Pauli Virtanen



From dmtmakris at gmail.com  Thu Dec 13 14:48:56 2012
From: dmtmakris at gmail.com (maxstirner)
Date: Thu, 13 Dec 2012 11:48:56 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] decimal order bessel function zeros
Message-ID: <34794804.post@talk.nabble.com>


  Hello all,
jn_zeros(n,nt) calculates nt zeros of n order  bessel function(n is
integer).
How i can calculate the nt zeros of n order bessel function when now n is
decimal?
-- 
View this message in context: http://old.nabble.com/decimal-order-bessel-function-zeros-tp34794804p34794804.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From p.zaffino at yahoo.it  Fri Dec 14 11:42:47 2012
From: p.zaffino at yahoo.it (Paolo Zaffino)
Date: Fri, 14 Dec 2012 16:42:47 +0000 (GMT)
Subject: [SciPy-User] Points fitting (non lin)
In-Reply-To: <CAAih6WmEZVRkmoVD6R0nuXT_kfCS256KZJ+8XmoZ==4RYXN33g@mail.gmail.com>
References: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<CAAih6WmEZVRkmoVD6R0nuXT_kfCS256KZJ+8XmoZ==4RYXN33g@mail.gmail.com>
Message-ID: <1355503367.42200.YahooMailNeo@web171602.mail.ir2.yahoo.com>

Dear?Pawe?,
thank you for the reply.
I try to explain better the issue.
I have these points (in this order):

P1 = (1,1)
P2 = (2,2)
P3 = (4,2)

P4 = (3,1)

I need to fit the points in the order P1,P2,P3,P4 even if the x coord of P3 is greater than P4.
I thought to quadratic piecewise curve but other solutions are welcome.

Thanks a lot.
Paolo



________________________________
 Da: Pawe? Kwa?niewski <pawel.kw at gmail.com>
A: Paolo Zaffino <p.zaffino at yahoo.it>; SciPy Users List <scipy-user at scipy.org> 
Inviato: Venerd? 14 Dicembre 2012 17:25
Oggetto: Re: [SciPy-User] Points fitting (non lin)
 

Dear Paolo,

I'm not sure I understand correctly your problem, but this sounds like a spline fitting job. You can read more about this here: http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html 

Is this what you are looking for?

Cheers,



Pawe?



2012/12/14 Paolo Zaffino <p.zaffino at yahoo.it>

Dear Scipy community,
>
>
>I have a set of points (2D) and I would compute a curve that fits them.
>The points are ordered in a precise way (not crescent order) and I can't change this order (the curve should fit the points in that order).
>I'm interseting in a non linear fit (the ideal case would be more intervals of quadratic curves).
>Has anyone any advice about?
>
>
>Thank you very much.
>Regards.Paolo ?
>_______________________________________________
>SciPy-User mailing list
>SciPy-User at scipy.org
>http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121214/df32ac8e/attachment.html>

From sturla at molden.no  Fri Dec 14 11:46:11 2012
From: sturla at molden.no (Sturla Molden)
Date: Fri, 14 Dec 2012 17:46:11 +0100
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <loom.20121214T172921-116@post.gmane.org>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB48F4.3000007@molden.no>
	<FF148DD9-0C26-4647-A12B-02521E7F4DB2@semanchuk.com>
	<50CB4ED0.1070203@molden.no>
	<loom.20121214T172921-116@post.gmane.org>
Message-ID: <50CB57D3.4000800@molden.no>

On 14.12.2012 17:30, Pauli Virtanen wrote:

> I think this means that someone needs to go through the LAPACK
> wrappers, and add any functions that are missing.
>
> Pull requests are accepted ;)

Yes, perferably all of LAPACK should be defined in

   scipy/linalg/flapack.pyf

But now it just has the functions that SciPy needs.

It will take quite some work. And I presume tests will be needed too?


Sturla


From pav at iki.fi  Fri Dec 14 11:47:42 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 14 Dec 2012 16:47:42 +0000 (UTC)
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB47FB.9090404@gmail.com> <50CB4BAE.1050105@molden.no>
	<BF9B8C23-B799-4773-9DAD-CC1606F37717@semanchuk.com>
Message-ID: <loom.20121214T174356-820@post.gmane.org>

Philip Semanchuk <philip <at> semanchuk.com> writes:
[clip]
> > CLAPACK is also abandonware, LAPACKE is the current C/C++ interface. 
> > Thus with any recent LAPACK, there will be no clapack in SciPy.
>
> Hmmm, well that's disappointing to hear, but better to know
> now than finding out the hard way later. 
> 
> Maybe a pure-Python + numpy/scipy solution for this just
> isn't possible right now. 

However, you should still be able to use ctypes to interface with the
Fortran routines in scipy/linalg/flapack.so

While the CLAPACK is not always present, the Fortran-based LAPACK is.

This may however require some fiddling with the calling conventions.
So testing with multiple platforms will be needed, but perhaps you
are OK with doing that...

-- 
Pauli Virtanen




From zachary.pincus at yale.edu  Fri Dec 14 11:50:25 2012
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Fri, 14 Dec 2012 11:50:25 -0500
Subject: [SciPy-User] Points fitting (non lin)
In-Reply-To: <1355503367.42200.YahooMailNeo@web171602.mail.ir2.yahoo.com>
References: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<CAAih6WmEZVRkmoVD6R0nuXT_kfCS256KZJ+8XmoZ==4RYXN33g@mail.gmail.com>
	<1355503367.42200.YahooMailNeo@web171602.mail.ir2.yahoo.com>
Message-ID: <9E04AB38-D38C-4C6D-881E-E294898EA3AC@yale.edu>


On Dec 14, 2012, at 11:42 AM, Paolo Zaffino wrote:

> Dear Pawe?,
> thank you for the reply.
> I try to explain better the issue.
> I have these points (in this order):
> 
> P1 = (1,1)
> P2 = (2,2)
> P3 = (4,2)
> P4 = (3,1)
> 
> I need to fit the points in the order P1,P2,P3,P4 even if the x coord of P3 is greater than P4.
> I thought to quadratic piecewise curve but other solutions are welcome.
> 
> Thanks a lot.
> Paolo

You will want to fit a parametric spline of some degree with some amount (or no) smoothing. I'd look at the splprep function in scipy.interpolate. 

The trick is you associate each point with some monotonic parameter value, and then interpolate along that parameter (say t) to get your x, y coordinates.

E.g.:
t  x  y
0  1  1
1  2  2
2  4  2
3  3  1

Then if you were interpolating linearly, at t=0.5, you would have (1.5, 1.5) as the coordinate.

As above, splprep will generate splines of a desired order (linear, quadratic, cubic, etc.) and with a user-specified smoothing parameter (s), which can be set to zero to get exact interpolation of the input coordinates, potentially at the cost of ringing (sometimes quite bad) away from the input coordinate. So you will need to plot the interpolated values, both at the input t-values, as well as at intermediate t's, to see if the output is sane.

Hopefully this is somewhat clear, or at least enough to get you started. Please read the documentation for splprep and splev for more information.
Zach



> 
> Da: Pawe? Kwa?niewski <pawel.kw at gmail.com>
> A: Paolo Zaffino <p.zaffino at yahoo.it>; SciPy Users List <scipy-user at scipy.org> 
> Inviato: Venerd? 14 Dicembre 2012 17:25
> Oggetto: Re: [SciPy-User] Points fitting (non lin)
> 
> Dear Paolo,
> 
> I'm not sure I understand correctly your problem, but this sounds like a spline fitting job. You can read more about this here: http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html 
> 
> Is this what you are looking for?
> 
> Cheers,
> 
> Pawe?
> 
> 
> 
> 2012/12/14 Paolo Zaffino <p.zaffino at yahoo.it>
> Dear Scipy community,
> 
> I have a set of points (2D) and I would compute a curve that fits them.
> The points are ordered in a precise way (not crescent order) and I can't change this order (the curve should fit the points in that order).
> I'm interseting in a non linear fit (the ideal case would be more intervals of quadratic curves).
> Has anyone any advice about?
> 
> Thank you very much.
> Regards.
> Paolo  
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 
> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From pav at iki.fi  Fri Dec 14 12:03:04 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 14 Dec 2012 17:03:04 +0000 (UTC)
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB48F4.3000007@molden.no>
	<FF148DD9-0C26-4647-A12B-02521E7F4DB2@semanchuk.com>
	<50CB4ED0.1070203@molden.no>
	<loom.20121214T172921-116@post.gmane.org>
	<50CB57D3.4000800@molden.no>
Message-ID: <loom.20121214T174846-826@post.gmane.org>

Sturla Molden <sturla <at> molden.no> writes:
> On 14.12.2012 17:30, Pauli Virtanen wrote:
> > I think this means that someone needs to go through the LAPACK
> > wrappers, and add any functions that are missing.
> >
> > Pull requests are accepted ;)
> 
> Yes, perferably all of LAPACK should be defined in
> 
>    scipy/linalg/flapack.pyf
> 
> But now it just has the functions that SciPy needs.
> 
> It will take quite some work. And I presume tests will be needed too?

We don't have tests for all the existing routines.

Just smoke tests would be good enough, I think --- work space query
and calling the routine on some random data. I'd guess writing these
would be boring, but on the other hand writing the .pyf file probably
take more time.

Probably doesn't make so much sense checking any of the results, as we
trust LAPACK.

-- 
Pauli Virtanen




From pawel.kw at gmail.com  Fri Dec 14 12:13:57 2012
From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=)
Date: Fri, 14 Dec 2012 18:13:57 +0100
Subject: [SciPy-User] Points fitting (non lin)
In-Reply-To: <9E04AB38-D38C-4C6D-881E-E294898EA3AC@yale.edu>
References: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<CAAih6WmEZVRkmoVD6R0nuXT_kfCS256KZJ+8XmoZ==4RYXN33g@mail.gmail.com>
	<1355503367.42200.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<9E04AB38-D38C-4C6D-881E-E294898EA3AC@yale.edu>
Message-ID: <CAAih6WmaPBCw13PFJWzT5DKkGL+LzgOW6j0BD_-RuDjTv4ER_g@mail.gmail.com>

So, as far as I understand, the scipy.interpolate.UnivariateSpline()
function is equivalent to interpolate.splrep - according to the
documentation string it is just a more modern, object oriented
implementation. In practice, it means that the
scipy.interpolate.UnivariateSpline() returns a function which can be then
evaluated on the desired x axis (I actually learned this today...). In the
case of example data you gave, this would be something like this:

from scipy import interpolate
x = array((1,2,4,3))
y = array((1,2,2,1))

# Calculate the spline
f = interpolate.UnivariateSpline(x,y)

# Evaluate the spline
fx = f(x)



Pawe?



2012/12/14 Zachary Pincus <zachary.pincus at yale.edu>

>
> On Dec 14, 2012, at 11:42 AM, Paolo Zaffino wrote:
>
> > Dear Pawe?,
> > thank you for the reply.
> > I try to explain better the issue.
> > I have these points (in this order):
> >
> > P1 = (1,1)
> > P2 = (2,2)
> > P3 = (4,2)
> > P4 = (3,1)
> >
> > I need to fit the points in the order P1,P2,P3,P4 even if the x coord of
> P3 is greater than P4.
> > I thought to quadratic piecewise curve but other solutions are welcome.
> >
> > Thanks a lot.
> > Paolo
>
> You will want to fit a parametric spline of some degree with some amount
> (or no) smoothing. I'd look at the splprep function in scipy.interpolate.
>
> The trick is you associate each point with some monotonic parameter value,
> and then interpolate along that parameter (say t) to get your x, y
> coordinates.
>
> E.g.:
> t  x  y
> 0  1  1
> 1  2  2
> 2  4  2
> 3  3  1
>
> Then if you were interpolating linearly, at t=0.5, you would have (1.5,
> 1.5) as the coordinate.
>
> As above, splprep will generate splines of a desired order (linear,
> quadratic, cubic, etc.) and with a user-specified smoothing parameter (s),
> which can be set to zero to get exact interpolation of the input
> coordinates, potentially at the cost of ringing (sometimes quite bad) away
> from the input coordinate. So you will need to plot the interpolated
> values, both at the input t-values, as well as at intermediate t's, to see
> if the output is sane.
>
> Hopefully this is somewhat clear, or at least enough to get you started.
> Please read the documentation for splprep and splev for more information.
> Zach
>
>
>
> >
> > Da: Pawe? Kwa?niewski <pawel.kw at gmail.com>
> > A: Paolo Zaffino <p.zaffino at yahoo.it>; SciPy Users List <
> scipy-user at scipy.org>
> > Inviato: Venerd? 14 Dicembre 2012 17:25
> > Oggetto: Re: [SciPy-User] Points fitting (non lin)
> >
> > Dear Paolo,
> >
> > I'm not sure I understand correctly your problem, but this sounds like a
> spline fitting job. You can read more about this here:
> http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html
> >
> > Is this what you are looking for?
> >
> > Cheers,
> >
> > Pawe?
> >
> >
> >
> > 2012/12/14 Paolo Zaffino <p.zaffino at yahoo.it>
> > Dear Scipy community,
> >
> > I have a set of points (2D) and I would compute a curve that fits them.
> > The points are ordered in a precise way (not crescent order) and I can't
> change this order (the curve should fit the points in that order).
> > I'm interseting in a non linear fit (the ideal case would be more
> intervals of quadratic curves).
> > Has anyone any advice about?
> >
> > Thank you very much.
> > Regards.
> > Paolo
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121214/2a2a06e3/attachment.html>

From philip at semanchuk.com  Fri Dec 14 14:24:54 2012
From: philip at semanchuk.com (Philip Semanchuk)
Date: Fri, 14 Dec 2012 14:24:54 -0500
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <loom.20121214T174356-820@post.gmane.org>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB47FB.9090404@gmail.com> <50CB4BAE.1050105@molden.no>
	<BF9B8C23-B799-4773-9DAD-CC1606F37717@semanchuk.com>
	<loom.20121214T174356-820@post.gmane.org>
Message-ID: <6DD12DBD-2348-4F66-89B0-7ED338F933CA@semanchuk.com>


On Dec 14, 2012, at 11:47 AM, Pauli Virtanen wrote:

> Philip Semanchuk <philip <at> semanchuk.com> writes:
> [clip]
>>> CLAPACK is also abandonware, LAPACKE is the current C/C++ interface. 
>>> Thus with any recent LAPACK, there will be no clapack in SciPy.
>> 
>> Hmmm, well that's disappointing to hear, but better to know
>> now than finding out the hard way later. 
>> 
>> Maybe a pure-Python + numpy/scipy solution for this just
>> isn't possible right now. 
> 
> However, you should still be able to use ctypes to interface with the
> Fortran routines in scipy/linalg/flapack.so
> 
> While the CLAPACK is not always present, the Fortran-based LAPACK is.
> 
> This may however require some fiddling with the calling conventions.
> So testing with multiple platforms will be needed, but perhaps you
> are OK with doing that...

Hmm, this just gets more and more interesting...

My Python code was already calling a compiled Fortran version of dbdsqr() via ctypes. (This version of dbdsqr() is in our custom library -- the one I'm porting -- and it's the only thing I haven't yet been able to port to Python.) Just now I was able to swap out the reference to our custom library and use Jonathan Helmus' technique to get a scipy reference to dbdsqr. All I changed was the library reference, and it worked! So now I'm totally independent of our compiled custom library under OS X. That's excellent news.

The not-so-excellent news is that on Windows I installed scipy 0.11.0 (the same version that's on my Mac) and it does *not* seem to provide a reference to dbdsqr(). I also looked for _dbdsqr, dbdsqr_, _dbdsqr_, etc. and it's just not there. I can get a reference to e.g. zgelss -- 

>>> hasattr(scipy.linalg.lapack.flapack, 'zgelss')
True

But not dbdsqr. I guess this is an artifact of the LAPACK library against which scipy was compiled?


bye
Philip



From pav at iki.fi  Fri Dec 14 14:38:41 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Fri, 14 Dec 2012 21:38:41 +0200
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <6DD12DBD-2348-4F66-89B0-7ED338F933CA@semanchuk.com>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB47FB.9090404@gmail.com> <50CB4BAE.1050105@molden.no>
	<BF9B8C23-B799-4773-9DAD-CC1606F37717@semanchuk.com>
	<loom.20121214T174356-820@post.gmane.org>
	<6DD12DBD-2348-4F66-89B0-7ED338F933CA@semanchuk.com>
Message-ID: <kafv7v$s1v$1@ger.gmane.org>

14.12.2012 21:24, Philip Semanchuk kirjoitti:
[clip]
> The not-so-excellent news is that on Windows I installed scipy 0.11.0
> (the same version that's on my Mac) and it does *not* seem to provide
> a reference to dbdsqr(). I also looked for _dbdsqr, dbdsqr_, _dbdsqr_,
> etc. and it's just not there. I can get a reference to e.g. zgelss --
> 
>>>> hasattr(scipy.linalg.lapack.flapack, 'zgelss')
> True
> 
> But not dbdsqr. I guess this is an artifact of the LAPACK library
> against which scipy was compiled?

No, the result from hasattr only means that the routine is not wrapped
via f2py. The Scipy LAPACK wrappers always wrap the exact same set of
routines.

However, if a routine is not wrapped, the compiler might not include it
in the built dll file. This probably is platform dependent.

-- 
Pauli Virtanen



From philip at semanchuk.com  Fri Dec 14 15:15:44 2012
From: philip at semanchuk.com (Philip Semanchuk)
Date: Fri, 14 Dec 2012 15:15:44 -0500
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <kafv7v$s1v$1@ger.gmane.org>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB47FB.9090404@gmail.com> <50CB4BAE.1050105@molden.no>
	<BF9B8C23-B799-4773-9DAD-CC1606F37717@semanchuk.com>
	<loom.20121214T174356-820@post.gmane.org>
	<6DD12DBD-2348-4F66-89B0-7ED338F933CA@semanchuk.com>
	<kafv7v$s1v$1@ger.gmane.org>
Message-ID: <26A717B5-2CC5-4A8F-8629-5DB45F46AF6F@semanchuk.com>


On Dec 14, 2012, at 2:38 PM, Pauli Virtanen wrote:

> 14.12.2012 21:24, Philip Semanchuk kirjoitti:
> [clip]
>> The not-so-excellent news is that on Windows I installed scipy 0.11.0
>> (the same version that's on my Mac) and it does *not* seem to provide
>> a reference to dbdsqr(). I also looked for _dbdsqr, dbdsqr_, _dbdsqr_,
>> etc. and it's just not there. I can get a reference to e.g. zgelss --
>> 
>>>>> hasattr(scipy.linalg.lapack.flapack, 'zgelss')
>> True
>> 
>> But not dbdsqr. I guess this is an artifact of the LAPACK library
>> against which scipy was compiled?
> 
> No, the result from hasattr only means that the routine is not wrapped
> via f2py. The Scipy LAPACK wrappers always wrap the exact same set of
> routines.
> 
> However, if a routine is not wrapped, the compiler might not include it
> in the built dll file. This probably is platform dependent.

Oh, I see. 

When I use `dumpbin.exe /SYMBOLS flapack.pyd`, I see dbdsqr referenced, but I can't access it under any name. ISTR that some other LAPACK function in scipy uses dbdsqr(), so it should be in there but perhaps the symbol is not exported. I can't find it in any of the other .pyd files in site-packages\scipy\linalg.

Cheers
Philip

From michael.aye at ucla.edu  Fri Dec 14 19:50:19 2012
From: michael.aye at ucla.edu (Michael Aye)
Date: Fri, 14 Dec 2012 16:50:19 -0800
Subject: [SciPy-User] scipy.interpolate.InterpolatedUnivariateSpline
Message-ID: <kaghg9$itk$1@ger.gmane.org>

The code example for InterpolatedUnivariateSpline here:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.InterpolatedUnivariateSpline.html#scipy.interpolate.InterpolatedUnivariateSpline 


has a copy/paste error.
It imports and works with UnivariateSpline instead of 
InterpolatedUnivariateSpline.

Nice weekend everyone!

Michael





From sturla at molden.no  Sat Dec 15 07:42:21 2012
From: sturla at molden.no (Sturla Molden)
Date: Sat, 15 Dec 2012 13:42:21 +0100
Subject: [SciPy-User] Calling LAPACK function dbdsqr()?
In-Reply-To: <kafv7v$s1v$1@ger.gmane.org>
References: <1208070C-2DA0-4982-BF4E-B85C7ECF67AD@semanchuk.com>
	<50CB47FB.9090404@gmail.com> <50CB4BAE.1050105@molden.no>
	<BF9B8C23-B799-4773-9DAD-CC1606F37717@semanchuk.com>
	<loom.20121214T174356-820@post.gmane.org>
	<6DD12DBD-2348-4F66-89B0-7ED338F933CA@semanchuk.com>
	<kafv7v$s1v$1@ger.gmane.org>
Message-ID: <38F58C46-1690-4A55-AE58-A2324DE5127F@molden.no>

Den 14. des. 2012 kl. 20:38 skrev Pauli Virtanen <pav at iki.fi>:

> 14.12.2012 21:24, Philip Semanchuk kirjoitti:
> [clip]
>> The not-so-excellent news is that on Windows I installed scipy 0.11.0
>> (the same version that's on my Mac) and it does *not* seem to provide
>> a reference to dbdsqr(). I also looked for _dbdsqr, dbdsqr_, _dbdsqr_,
>> etc. and it's just not there. I can get a reference to e.g. zgelss --
>> 
>>>>> hasattr(scipy.linalg.lapack.flapack, 'zgelss')
>> True
>> 
>> But not dbdsqr. I guess this is an artifact of the LAPACK library
>> against which scipy was compiled?
> 
> No, the result from hasattr only means that the routine is not wrapped
> via f2py. The Scipy LAPACK wrappers always wrap the exact same set of
> routines.
> 
> However, if a routine is not wrapped, the compiler might not include it
> in the built dll file. This probably is platform dependent.

Yes, and that typically happen when we link against a static LAPACK library. 

It will help to build a dynamic LAPACK DLL first and link NumPy/SciPy against that.

Sturla






From tmp50 at ukr.net  Sat Dec 15 10:13:31 2012
From: tmp50 at ukr.net (Dmitrey)
Date: Sat, 15 Dec 2012 17:13:31 +0200
Subject: [SciPy-User] [ANN] OpenOpt Suite release 0.43
Message-ID: <1943.1355584411.5558559243999444992@ffe6.ukr.net>




Hi all,


I'm glad to inform you about new OpenOpt release 0.43 (2012-Dec-15):


* interalg now can solve SNLE in 2nd mode (parameter dataHandling =
"raw", before - only "sorted")
* Many other improvements for interalg
* Some improvements for FuncDesigner kernel
* FuncDesigner ODE now has 3 arguments instead of 4 (backward
incompatibility!), e.g. {t: np.linspace(0,1,100)} or mere
np.linspace(0,1,100) if your ODE right side is time-independend
* FuncDesigner stochastic addon now can handle some problems with
gradient-based NLP / NSP solvers
* Many minor improvements and some bugfixes


Visit  openopt.org for more details.


Regards, D.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121215/d0f42046/attachment.html>

From p.zaffino at yahoo.it  Sun Dec 16 09:29:35 2012
From: p.zaffino at yahoo.it (Paolo Zaffino)
Date: Sun, 16 Dec 2012 15:29:35 +0100
Subject: [SciPy-User] Points fitting (non lin)
In-Reply-To: <9E04AB38-D38C-4C6D-881E-E294898EA3AC@yale.edu>
References: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<CAAih6WmEZVRkmoVD6R0nuXT_kfCS256KZJ+8XmoZ==4RYXN33g@mail.gmail.com>
	<1355503367.42200.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<9E04AB38-D38C-4C6D-881E-E294898EA3AC@yale.edu>
Message-ID: <50CDDACF.10305@yahoo.it>

Dear Zach,
I didn't understand the trick...sorry.
Please, can you explain it again?

Thanks a lot.
Paolo


Il 14/12/2012 17:50, Zachary Pincus ha scritto:
> On Dec 14, 2012, at 11:42 AM, Paolo Zaffino wrote:
>
>> Dear Pawe?,
>> thank you for the reply.
>> I try to explain better the issue.
>> I have these points (in this order):
>>
>> P1 = (1,1)
>> P2 = (2,2)
>> P3 = (4,2)
>> P4 = (3,1)
>>
>> I need to fit the points in the order P1,P2,P3,P4 even if the x coord of P3 is greater than P4.
>> I thought to quadratic piecewise curve but other solutions are welcome.
>>
>> Thanks a lot.
>> Paolo
> You will want to fit a parametric spline of some degree with some amount (or no) smoothing. I'd look at the splprep function in scipy.interpolate.
>
> The trick is you associate each point with some monotonic parameter value, and then interpolate along that parameter (say t) to get your x, y coordinates.
>
> E.g.:
> t  x  y
> 0  1  1
> 1  2  2
> 2  4  2
> 3  3  1
>
> Then if you were interpolating linearly, at t=0.5, you would have (1.5, 1.5) as the coordinate.
>
> As above, splprep will generate splines of a desired order (linear, quadratic, cubic, etc.) and with a user-specified smoothing parameter (s), which can be set to zero to get exact interpolation of the input coordinates, potentially at the cost of ringing (sometimes quite bad) away from the input coordinate. So you will need to plot the interpolated values, both at the input t-values, as well as at intermediate t's, to see if the output is sane.
>
> Hopefully this is somewhat clear, or at least enough to get you started. Please read the documentation for splprep and splev for more information.
> Zach
>
>
>
>> Da: Pawe? Kwa?niewski <pawel.kw at gmail.com>
>> A: Paolo Zaffino <p.zaffino at yahoo.it>; SciPy Users List <scipy-user at scipy.org>
>> Inviato: Venerd? 14 Dicembre 2012 17:25
>> Oggetto: Re: [SciPy-User] Points fitting (non lin)
>>
>> Dear Paolo,
>>
>> I'm not sure I understand correctly your problem, but this sounds like a spline fitting job. You can read more about this here: http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html
>>
>> Is this what you are looking for?
>>
>> Cheers,
>>
>> Pawe?
>>
>>
>>
>> 2012/12/14 Paolo Zaffino <p.zaffino at yahoo.it>
>> Dear Scipy community,
>>
>> I have a set of points (2D) and I would compute a curve that fits them.
>> The points are ordered in a precise way (not crescent order) and I can't change this order (the curve should fit the points in that order).
>> I'm interseting in a non linear fit (the ideal case would be more intervals of quadratic curves).
>> Has anyone any advice about?
>>
>> Thank you very much.
>> Regards.
>> Paolo
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From zachary.pincus at yale.edu  Sun Dec 16 14:44:34 2012
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Sun, 16 Dec 2012 14:44:34 -0500
Subject: [SciPy-User] Points fitting (non lin)
In-Reply-To: <50CDDACF.10305@yahoo.it>
References: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<CAAih6WmEZVRkmoVD6R0nuXT_kfCS256KZJ+8XmoZ==4RYXN33g@mail.gmail.com>
	<1355503367.42200.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<9E04AB38-D38C-4C6D-881E-E294898EA3AC@yale.edu>
	<50CDDACF.10305@yahoo.it>
Message-ID: <089C98C7-4E58-46EC-83A7-CDE506C1634C@yale.edu>

> Dear Zach,
> I didn't understand the trick...sorry.
> Please, can you explain it again?

You'll want to read up about parametric representation of curves on wikipedia etc.

Obviously, the coordinates you listed:
P1 = (1,1)
P2 = (2,2)
P3 = (4,2)
P4 = (3,1)
do not trace out y values that are a proper function of, because there are some places where the curve would have multiple y-values for a given x-value (at 3, say). So, as you correctly understood, you can't just try to fit a function y=f(x) to your data and expect that to work.

What you can, however, do is fit TWO functions to your data, using a dummy variable. For example, imagine a point moving along your curve at a given speed, so that at any given time t, the position of the point is given as (x(t), y(t)). 

So for each of your points above P1-P4, associate a value T1-T4.

For example,
t  x  y
0  1  1
1  2  2
2  4  2
3  3  1

is just your data above but with a t parameter indexing each point. So at t=0, x=1 and y=1. At t=1, x=2 and y=2. If you had a functional form for x(t) and y(t), you could then estimate where the point on the curve that was midway between the first (t=0) and second (t=1) points by evaluating x(0.5) and y(0.5).

To do this, one could fit x(t) and y(t) separately (because as long as the t values are monotonic, x(t) and y(t) are proper functions for any smooth curve that you could draw on a plane). Fitting a spline to these data is the most straightforward way to do this, and fortunately, scipy.interpolate.splprep does this all for you and gives you a spline that can be evaluated at different t positions by splev.

If spline fitting is unfamiliar to you, then parametric spline fitting will seem even less comprehensible. So you might want to experiment by fitting other simpler functional curves in scipy with splrep and then moving on. (There should be interpolation examples aplenty online.)

Zach



> Thanks a lot.
> Paolo
> 
> 
> Il 14/12/2012 17:50, Zachary Pincus ha scritto:
>> On Dec 14, 2012, at 11:42 AM, Paolo Zaffino wrote:
>> 
>>> Dear Pawe?,
>>> thank you for the reply.
>>> I try to explain better the issue.
>>> I have these points (in this order):
>>> 
>>> P1 = (1,1)
>>> P2 = (2,2)
>>> P3 = (4,2)
>>> P4 = (3,1)
>>> 
>>> I need to fit the points in the order P1,P2,P3,P4 even if the x coord of P3 is greater than P4.
>>> I thought to quadratic piecewise curve but other solutions are welcome.
>>> 
>>> Thanks a lot.
>>> Paolo
>> You will want to fit a parametric spline of some degree with some amount (or no) smoothing. I'd look at the splprep function in scipy.interpolate.
>> 
>> The trick is you associate each point with some monotonic parameter value, and then interpolate along that parameter (say t) to get your x, y coordinates.
>> 
>> E.g.:
>> t  x  y
>> 0  1  1
>> 1  2  2
>> 2  4  2
>> 3  3  1
>> 
>> Then if you were interpolating linearly, at t=0.5, you would have (1.5, 1.5) as the coordinate.
>> 
>> As above, splprep will generate splines of a desired order (linear, quadratic, cubic, etc.) and with a user-specified smoothing parameter (s), which can be set to zero to get exact interpolation of the input coordinates, potentially at the cost of ringing (sometimes quite bad) away from the input coordinate. So you will need to plot the interpolated values, both at the input t-values, as well as at intermediate t's, to see if the output is sane.
>> 
>> Hopefully this is somewhat clear, or at least enough to get you started. Please read the documentation for splprep and splev for more information.
>> Zach
>> 
>> 
>> 
>>> Da: Pawe? Kwa?niewski <pawel.kw at gmail.com>
>>> A: Paolo Zaffino <p.zaffino at yahoo.it>; SciPy Users List <scipy-user at scipy.org>
>>> Inviato: Venerd? 14 Dicembre 2012 17:25
>>> Oggetto: Re: [SciPy-User] Points fitting (non lin)
>>> 
>>> Dear Paolo,
>>> 
>>> I'm not sure I understand correctly your problem, but this sounds like a spline fitting job. You can read more about this here: http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html
>>> 
>>> Is this what you are looking for?
>>> 
>>> Cheers,
>>> 
>>> Pawe?
>>> 
>>> 
>>> 
>>> 2012/12/14 Paolo Zaffino <p.zaffino at yahoo.it>
>>> Dear Scipy community,
>>> 
>>> I have a set of points (2D) and I would compute a curve that fits them.
>>> The points are ordered in a precise way (not crescent order) and I can't change this order (the curve should fit the points in that order).
>>> I'm interseting in a non linear fit (the ideal case would be more intervals of quadratic curves).
>>> Has anyone any advice about?
>>> 
>>> Thank you very much.
>>> Regards.
>>> Paolo
>>> 
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From pierre.raybaut at gmail.com  Mon Dec 17 08:57:28 2012
From: pierre.raybaut at gmail.com (Pierre Raybaut)
Date: Mon, 17 Dec 2012 14:57:28 +0100
Subject: [SciPy-User] ANN: WinPython v2.7.3.2
Message-ID: <CAKegKuC2uO+d5=kF+x78KixHfjj8=G9ZbORkh3BaKsT1znnp1A@mail.gmail.com>

Hi all,

I'm pleased to announce that WinPython v2.7.3.2 has been released for
32-bit and 64-bit Windows platforms:
http://code.google.com/p/winpython/
This is mainly a maintenance release (many packages have been updated
since v2.7.3.1).

WinPython is a free open-source portable distribution of Python for
Windows, designed for scientists.

It is a full-featured (see
http://code.google.com/p/winpython/wiki/PackageIndex) Python-based
scientific environment:
  * Designed for scientists (thanks to the integrated libraries NumPy,
SciPy, Matplotlib, guiqwt, etc.:
    * Regular *scientific users*: interactive data processing and
visualization using Python with Spyder
    * *Advanced scientific users and software developers*: Python
applications development with Spyder, version control with Mercurial
and other development tools (like gettext)
  * *Portable*: preconfigured, it should run out of the box on any
machine under Windows (without any installation requirements) and the
folder containing WinPython can be moved to any location (local,
network or removable drive)
  * *Flexible*: one can install (or should I write "use" as it's
portable) as many WinPython versions as necessary (like isolated and
self-consistent environments), even if those versions are running
different versions of Python (2.7, 3.x in the near future) or
different architectures (32bit or 64bit) on the same machine
  * *Customizable*: using the integrated package manager (wppm, as
WinPython Package Manager), it's possible to install, uninstall or
upgrade Python packages (see
http://code.google.com/p/winpython/wiki/WPPM for more details on
supported package formats).

*WinPython is not an attempt to replace Python(x,y)*, this is just
something different (see
http://code.google.com/p/winpython/wiki/Roadmap): more flexible,
easier to maintain, movable and less invasive for the OS, but
certainly less user-friendly, with less packages/contents and without
any integration to Windows explorer [*].

[*] Actually there is an optional integration into Windows explorer,
providing the same features as the official Python installer regarding
file associations and context menu entry (this option may be activated
through the WinPython Control Panel).

Enjoy!
-Pierre


From amyla333 at gmail.com  Mon Dec 17 18:22:04 2012
From: amyla333 at gmail.com (Amy Anderson)
Date: Mon, 17 Dec 2012 18:22:04 -0500
Subject: [SciPy-User] running script error (eigen symmetric)
In-Reply-To: <13BFE00357AE78459CE7894943A31CA91BD932BE@MED-CORE07A.med.wayne.edu>
References: <13BFE00357AE78459CE7894943A31CA91BD932BE@MED-CORE07A.med.wayne.edu>
Message-ID: <CAP_FHc_BkPHspPWSw5wdfDxjgiH2=0nrA0DEFPwNJ6yFh7m0ug@mail.gmail.com>

Hi scipy users,

I am trying to run this script in python and the script is now working with
the test data (eigen problem solved) but when I input my own data to be
called by script I got the following errors.

[moriaht-5:~/Desktop/pyClusterROI] moriah% python pyClusterROI_test.py
tcorr connectivity subject2002.nii

Traceback (most recent call last):
  File "pyClusterROI_test.py", line 89, in <module>
    make_local_connectivity_tcorr( infiles[i], maskname, outname, 0.5 )
  File
"/Users/moriah/Desktop/pyClusterROI/make_local_connectivity_tcorr.py", line
114, in make_local_connectivity_tcorr
    msk=csc_matrix((range(1,m+1),(iv,zeros(m))),shape=(prod(sz[1:]),1))
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/compressed.py",
line 44, in __init__
    other = self.__class__( coo_matrix(arg1, shape=shape) )
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/coo.py",
line 188, in __init__
    self._check()
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/coo.py",
line 220, in _check
    raise ValueError('row index exceedes matrix dimensions')
ValueError: row index exceedes matrix dimensions


I was wondering if anyone might know what this error might mean or how to
fix it? I do not know what is meant by "row index exceeds matrix
dimensions" as this is the first time I am using scipy to run a script.

On Mon, Dec 3, 2012 at 11:29 AM, Anderson, Amy <aanderso at med.wayne.edu>wrote:

>  Dear Scipy users,
>
>  I am a new python and scipy user and I am looking for some help to
> troubleshoot an error I have been getting when trying to run a script in
> python.  The script I am trying to run is called pyCluster ROI and it needs
> python, pynifti, scipy, and numpy to run. I have installed all these
> programs successfully using macports on Mac OS 10.8.
>
>  I unarchived the scripts and ran the test script getting this error:
>
> $ python pyClusterROI_test.py****
>
> Traceback (most recent call last):****
>
>   File "pyClusterROI_test.py", line 48, in <module>****
>
>     from make_local_connectivity_scorr import *****
>
>   File
> "/Users/matthewnye/Downloads/pyClusterROI/make_local_connectivity_scorr.py",
> line 40, in <module>****
>
>     from scipy.sparse.linalg.eigen.arpack import eigen_symmetric****
>
> ImportError: cannot import name eigen_symmetric****
>
>
>  I have been trying to hunt down what this error may be and it seems that
> older versions of scipy might have used  ?scipy.sparse.linalg.eigen.arpack.eigen
> symmetric()?  where as new versions it has been renamed as "
> scipy.sparse.linalg.eigen.arpack.eigs()."
>
>  Does anyone know what might be causing this error or if it is a version
> problem is there any way to get an older version of scipy through macports?
> Also I have considered installing all these modules through a manual
> install but since i am a novice user I did not want to attempt that if
> there was a potentially simpler solution.
>
>  Any help would be much appreciated!
>
>  -Amy
>
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121217/17b49aef/attachment.html>

From issa at aims.ac.za  Mon Dec 17 19:12:10 2012
From: issa at aims.ac.za (Issa Karambal)
Date: Tue, 18 Dec 2012 00:12:10 +0000
Subject: [SciPy-User] help: scipy.ode set_integrator dopri5 or dop853
Message-ID: <CAEDf9RCY-Ec_c1A+nSegxsfadf=WiMMGO0zokfow=wQdza_mkQ@mail.gmail.com>

Hi all
I am trying to integrate a simple ode which is succesfull when calling
'vode' integrator. However,
with  'dopri5' or 'dop853' as integrator, I have the following error:
create_cb_arglist: Failed to build argument list (siz) with enough
arguments (tot-opt) required by user-supplied function (siz,tot,opt=2,3,0).
Traceback (most recent call last):
  File "schro1.py", line 45, in <module>
    print Evans(3)
  File "schro1.py", line 24, in Evans
    r.integrate(r.t+dt)
  File "/usr/lib/python2.7/dist-packages/scipy/integrate/ode.py", line 326,
in integrate
    self.f_params,self.jac_params)
  File "/usr/lib/python2.7/dist-packages/scipy/integrate/ode.py", line 749,
in run
    x,y,iwork,idid = self.runner(*((f,t0,y0,t1) + tuple(self.call_args)))
_dop.error: failed in processing argument list for call-back fcn.

I do not understand the above error, so any help is more than welcome.

Thanks

Issa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121218/38592ab9/attachment.html>

From francescoboccacci at libero.it  Tue Dec 18 04:20:46 2012
From: francescoboccacci at libero.it (francescoboccacci at libero.it)
Date: Tue, 18 Dec 2012 10:20:46 +0100 (CET)
Subject: [SciPy-User] kernel method
Message-ID: <8188718.2110881355822446172.JavaMail.root@wmail38>

HI all,
i'm new of the scipy's world and i need some help of course.
I'd like to find some method used in R in scipy. In particular 2 method:

- kernelUD
-lscv.cruncher


anyone can suggest me which are the same methods in scipy?

Thanks

Francesco


From a.klein at science-applied.nl  Tue Dec 18 05:10:42 2012
From: a.klein at science-applied.nl (Almar Klein)
Date: Tue, 18 Dec 2012 11:10:42 +0100
Subject: [SciPy-User] ANN: visvis 1.8 - The object oriented approach to
	visualization
Message-ID: <CAB_T6=kh89+RoWnRytu744QHF3ZCnQ_0YmPwXX_4vAODV79WQg@mail.gmail.com>

Dear all,

On behalf of the Visvis development team, I'm pleased to announce version
1.8 of Visvis - the object oriented approach to visualization.

Essentially, visvis is an object oriented layer of Python on top of OpenGl,
thereby combining the power of OpenGl with the usability of Python. A
Matlab-like interface in the form of a set of functions allows easy
creation of objects (e.g. plot(), imshow(), volshow(), surf()).

The most notable changes are:

   - Better handling of NaN, Inf in line objects, barplots, boxplots, and
   calculation of axes limits.
   - The bar plot now supports the bottom keyword, allowing for stacked bar
   charts.
   - Meshes can now be made (semi)transparent by giving RGBA values to
   mesh.faceColor.
   - Better separation of widget and figure creation, making it easier to
   embed visvis in threaded applications.


Website: https://code.google.com/p/visvis/
Discussion group: http://groups.google.com/group/visvis
Release notes: https://code.google.com/p/visvis/wiki/releaseNotes

Regards,
  Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121218/e4f17502/attachment.html>

From zachary.pincus at yale.edu  Tue Dec 18 10:14:48 2012
From: zachary.pincus at yale.edu (Zachary Pincus)
Date: Tue, 18 Dec 2012 10:14:48 -0500
Subject: [SciPy-User] kernel method
In-Reply-To: <8188718.2110881355822446172.JavaMail.root@wmail38>
References: <8188718.2110881355822446172.JavaMail.root@wmail38>
Message-ID: <3ED7E4DB-7921-4D3A-997A-B53F0850DFC8@yale.edu>

> I'd like to find some method used in R in scipy. In particular 2 method:
> 
> - kernelUD
> -lscv.cruncher


Perhaps you could briefly describe what these functions do? From brief googling, they do not appear to be part of the standard R distribution.

I and likely many others on the list would be happy to help you find replacements for this functionality in scipy (and/or beyond), but not everyone (and especially not me) is expert enough in R (and specifically whatever package the above functions came from) to know what those functions do just based on their names.

Similarly, a bit of description about how you plan to use the functions can be helpful to those suggesting what approaches might be best in scipy, in the case that there is not an exact drop-in replacement. (Indeed, often I find that when I ask a more general question about "what's the best way to solve problem X", rather than just asking about a specific function in scipy, I learn a whole lot from the experts on the list and often come away with a much better understanding of the problem and a better route to solving it...)

Good luck making the R to python transition!

Zach



On Dec 18, 2012, at 4:20 AM, francescoboccacci at libero.it wrote:

> HI all,
> i'm new of the scipy's world and i need some help of course.
> I'd like to find some method used in R in scipy. In particular 2 method:
> 
> - kernelUD
> -lscv.cruncher
> 
> 
> anyone can suggest me which are the same methods in scipy?
> 
> Thanks
> 
> Francesco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From almar.klein at gmail.com  Tue Dec 18 05:08:47 2012
From: almar.klein at gmail.com (Almar Klein)
Date: Tue, 18 Dec 2012 11:08:47 +0100
Subject: [SciPy-User] ANN: visvis 1.8 - The object oriented approach to
	visualization
Message-ID: <CAB_T6==N9iv4zEw7aDWcQd45MJsEyX3-Gi-o4amHCFVtb2Dj+Q@mail.gmail.com>

Dear all,

On behalf of the Visvis development team, I'm pleased to announce version
1.8 of Visvis - the object oriented approach to visualization.

Essentially, visvis is an object oriented layer of Python on top of OpenGl,
thereby combining the power of OpenGl with the usability of Python. A
Matlab-like interface in the form of a set of functions allows easy
creation of objects (e.g. plot(), imshow(), volshow(), surf()).

The most notable changes are:

   - Better handling of NaN, Inf in line objects, barplots, boxplots, and
   calculation of axes limits.
   - The bar plot now supports the bottom keyword, allowing for stacked bar
   charts.
   - Meshes can now be made (semi)transparent by giving RGBA values to
   mesh.faceColor.
   - Better separation of widget and figure creation, making it easier to
   embed visvis in threaded applications.


Website: https://code.google.com/p/visvis/
Discussion group: http://groups.google.com/group/visvis
Release notes: https://code.google.com/p/visvis/wiki/releaseNotes

Regards,
  Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121218/19eab14b/attachment.html>

From aronne.merrelli at gmail.com  Tue Dec 18 11:14:50 2012
From: aronne.merrelli at gmail.com (Aronne Merrelli)
Date: Tue, 18 Dec 2012 10:14:50 -0600
Subject: [SciPy-User] running script error (eigen symmetric)
In-Reply-To: <CAP_FHc_BkPHspPWSw5wdfDxjgiH2=0nrA0DEFPwNJ6yFh7m0ug@mail.gmail.com>
References: <13BFE00357AE78459CE7894943A31CA91BD932BE@MED-CORE07A.med.wayne.edu>
	<CAP_FHc_BkPHspPWSw5wdfDxjgiH2=0nrA0DEFPwNJ6yFh7m0ug@mail.gmail.com>
Message-ID: <CAHNdQ4Ka25vLO8HPUSei+ZJq2gQciDSah0JiffksN0b7fUQjhA@mail.gmail.com>

On Mon, Dec 17, 2012 at 5:22 PM, Amy Anderson <amyla333 at gmail.com> wrote:

> Hi scipy users,
>
> I am trying to run this script in python and the script is now working
> with the test data (eigen problem solved) but when I input my own data to
> be called by script I got the following errors.
>
> [moriaht-5:~/Desktop/pyClusterROI] moriah% python pyClusterROI_test.py
> tcorr connectivity subject2002.nii
>
> Traceback (most recent call last):
>   File "pyClusterROI_test.py", line 89, in <module>
>     make_local_connectivity_tcorr( infiles[i], maskname, outname, 0.5 )
>   File
> "/Users/moriah/Desktop/pyClusterROI/make_local_connectivity_tcorr.py", line
> 114, in make_local_connectivity_tcorr
>     msk=csc_matrix((range(1,m+1),(iv,zeros(m))),shape=(prod(sz[1:]),1))
>   File
> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/compressed.py",
> line 44, in __init__
>     other = self.__class__( coo_matrix(arg1, shape=shape) )
>   File
> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/coo.py",
> line 188, in __init__
>     self._check()
>   File
> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/sparse/coo.py",
> line 220, in _check
>     raise ValueError('row index exceedes matrix dimensions')
> ValueError: row index exceedes matrix dimensions
>
>
> I was wondering if anyone might know what this error might mean or how to
> fix it? I do not know what is meant by "row index exceeds matrix
> dimensions" as this is the first time I am using scipy to run a script.
>
>
Hi,

It is difficult to debug this without seeing the actual code you are
running, but it seems likely that the inputs to the CSC matrix constructor
are in error. Can you set a breakpoint and check the inputs at this call?

msk=csc_matrix((range(1,m+1),(iv,zeros(m))),shape=(prod(sz[1:]),1))

what are the runtime values of m, iv, and sz? I can trigger the same
ValueError that you see, by giving the constructor a set of row, column
entries in the matrix, that are outside the shape of the matrix as
described by the shape keyword. For example:

In [21]: test = scipy.sparse.csc_matrix( ([5,6], ([0,5],[0,0])), shape =
(3,1) )
<snipped various traceback stuff...>
ValueError: row index exceedes matrix dimensions


That call is attempting to put a matrix element at (5,0) while the shape is
only (3,1). So, making shape (6,1) works:

In [22]: test = scipy.sparse.csc_matrix( ([5,6], ([0,5],[0,0])), shape =
(6,1) )

In [23]: test.todense()
Out[23]:
matrix([[5],
        [0],
        [0],
        [0],
        [0],
        [6]])
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121218/0752f6cb/attachment.html>

From sturla at molden.no  Tue Dec 18 13:41:59 2012
From: sturla at molden.no (Sturla Molden)
Date: Tue, 18 Dec 2012 19:41:59 +0100
Subject: [SciPy-User] Fwd: [Numpy-discussion] Support for python 2.4
	dropped. Should we drop 2.5 also?
References: <CAB6mnxKkPwJEsLwpG_ot4RKW109rdq_XqMRzfJYE2Z=vHrwBpg@mail.gmail.com>
Message-ID: <D5C92B7D-BA0E-407D-BAC5-91BBD070B565@molden.no>

Does this apply for SciPy as well?

Sturla

Sendt fra min iPad

Videresendt melding:

> Fra: Charles R Harris <charlesr.harris at gmail.com>
> Dato: 16. desember 2012 18:28:34 CET
> Til: numpy-discussion <numpy-discussion at scipy.org>
> Emne: Re: [Numpy-discussion] Support for python 2.4 dropped. Should we drop 2.5 also?
> Svar til: Discussion of Numerical Python <numpy-discussion at scipy.org>
> 
> 
> 
> On Thu, Dec 13, 2012 at 10:38 AM, Charles R Harris <charlesr.harris at gmail.com> wrote:
>> The previous proposal to drop python 2.4 support garnered no opposition. How about dropping support for python 2.5 also?
> 
> The proposal to drop support for python 2.5 and 2.4 in numpy 1.8 has carried. It is now a todo issue on github.
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121218/3c976e60/attachment.html>

From ralf.gommers at gmail.com  Tue Dec 18 13:47:37 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Tue, 18 Dec 2012 19:47:37 +0100
Subject: [SciPy-User] [SciPy-Dev] Fwd: [Numpy-discussion] Support for
 python 2.4 dropped. Should we drop 2.5 also?
In-Reply-To: <D5C92B7D-BA0E-407D-BAC5-91BBD070B565@molden.no>
References: <CAB6mnxKkPwJEsLwpG_ot4RKW109rdq_XqMRzfJYE2Z=vHrwBpg@mail.gmail.com>
	<D5C92B7D-BA0E-407D-BAC5-91BBD070B565@molden.no>
Message-ID: <CABL7CQjWfOCOX-UTam-Eg35LFSfRZREebssA5v504bY2qpYruA@mail.gmail.com>

On Tue, Dec 18, 2012 at 7:41 PM, Sturla Molden <sturla at molden.no> wrote:

> Does this apply for SciPy as well?
>

I haven't seen any objections to this proposal (
http://thread.gmane.org/gmane.comp.python.scientific.devel/17114/focus=17117)
and several +1's, so yes.

SciPy 0.12.0 will support Python versions 2.6 - 3.3.

Ralf



> Sturla
>
> Sendt fra min iPad
>
> Videresendt melding:
>
> *Fra:* Charles R Harris <charlesr.harris at gmail.com>
> *Dato:* 16. desember 2012 18:28:34 CET
> *Til:* numpy-discussion <numpy-discussion at scipy.org>
> *Emne:* *Re: [Numpy-discussion] Support for python 2.4 dropped. Should we
> drop 2.5 also?*
> *Svar til:* Discussion of Numerical Python <numpy-discussion at scipy.org>
>
>
>
> On Thu, Dec 13, 2012 at 10:38 AM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
>
>> The previous proposal to drop python 2.4 support garnered no opposition.
>> How about dropping support for python 2.5 also?
>>
>>
> The proposal to drop support for python 2.5 and 2.4 in numpy 1.8 has
> carried. It is now a todo issue on github<https://github.com/numpy/numpy/issues/2830>
> .
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121218/d4f21f05/attachment.html>

From klonuo at gmail.com  Tue Dec 18 19:39:48 2012
From: klonuo at gmail.com (klo uo)
Date: Wed, 19 Dec 2012 01:39:48 +0100
Subject: [SciPy-User] Scikits statuses?
Message-ID: <CAA-8Ld9xXJm1P3tYSgvJd7RCy_rc3LbHbSOkebQV0-qC8CdouQ@mail.gmail.com>

I browsed scikits (http://scikits.appspot.com/scikits) and looking at one
of the packages I noticed that it's in poor state - project started long
ago but finished right at it. That give me an idea to try and determine the
state of available scikits.

I copied text from scikits page and saved it as text file. Then I used pypi
as reference and results are in this little script:
  http://nbviewer.ipython.org/url/dl.dropbox.com/u/6735093/scikits.ipynb

As a bonus I attached also a bar plot of pypi download counts per package.


Scikits not hosted on pypi, are mainly hosted on scipy.org and versioned by
subversion:

  delaunay
  eartho
  hydroclimpy
  mlabwrap
  pymat
  rsformats
  scattpy
  timeseries
  umfpack

Their status is questionable, and if I'm not wrong all those are released
years ago, w/o documentation, not even description, neither their names are
correct.
Exceptions are timeseries (author abandoned it), and hydroclimpy which
doesn't look as a scikit to me at all, but universe for itself.

Others, listed by last update (note: two duplicate packages, because of
naming transition):

  scikit-aero           20121125
  scikits.odes          20121122
  scikits.optimization  20121114
  scikits.fitting       20121029
  scikit-fmm            20121016
  scikit-image          20121014
  scikits-image         20121014
  scikit-learn          20121008
  scikits.bvp_solver    20120926
  scikit-rf             20120925
  scikits.bootstrap     20120822
  scikit-vi             20120609
  scikit-commpy         20120404
  scikits.learn         20110921
  scikits.statsmodels   20110824
  scikits.cuda          20110523
  scikits.datasmooth    20110414
  scikits.audiolab      20100723
  scikits.sparse        20091214
  scikits.bvp1lg        20091012
  scikits.talkbox       20090827
  scikits.samplerate    20090722
  scikits.vectorplot    20081202
  scikits.example       20081105
  scikits.ann           20080131

What I want to suggest from this little adventure is to sort scikits
packages by last update, on scikits home page. Then maybe scikits not
updated for more then a year, flag with new status column, describing the
state of the package. I believe it would help general user from start, and
to potential developer for takeover. Also it will show that scikits idea is
active, instead current static display with half packages in unknown state.

As for those in first listing, "abandoned" or "deprecated" is IMHO
suitable, if not necessary.
[image: Inline image 1]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121219/f559e07e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 8910 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121219/f559e07e/attachment.png>

From josef.pktd at gmail.com  Tue Dec 18 20:07:40 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Tue, 18 Dec 2012 20:07:40 -0500
Subject: [SciPy-User] Scikits statuses?
In-Reply-To: <CAA-8Ld9xXJm1P3tYSgvJd7RCy_rc3LbHbSOkebQV0-qC8CdouQ@mail.gmail.com>
References: <CAA-8Ld9xXJm1P3tYSgvJd7RCy_rc3LbHbSOkebQV0-qC8CdouQ@mail.gmail.com>
Message-ID: <CAMMTP+AVq+qoAjMOk+huWhDv8KDwQWqMHaHC6ctoSG9wDn-39w@mail.gmail.com>

On Tue, Dec 18, 2012 at 7:39 PM, klo uo <klonuo at gmail.com> wrote:

> I browsed scikits (http://scikits.appspot.com/scikits) and looking at one
> of the packages I noticed that it's in poor state - project started long
> ago but finished right at it. That give me an idea to try and determine the
> state of available scikits.
>
> I copied text from scikits page and saved it as text file. Then I used
> pypi as reference and results are in this little script:
>   http://nbviewer.ipython.org/url/dl.dropbox.com/u/6735093/scikits.ipynb
>
> As a bonus I attached also a bar plot of pypi download counts per package.
>
>
> Scikits not hosted on pypi, are mainly hosted on scipy.org and versioned
> by subversion:
>
>   delaunay
>   eartho
>   hydroclimpy
>   mlabwrap
>   pymat
>   rsformats
>   scattpy
>   timeseries
>   umfpack
>
> Their status is questionable, and if I'm not wrong all those are released
> years ago, w/o documentation, not even description, neither their names are
> correct.
> Exceptions are timeseries (author abandoned it), and hydroclimpy which
> doesn't look as a scikit to me at all, but universe for itself.
>
> Others, listed by last update (note: two duplicate packages, because of
> naming transition):
>
>   scikit-aero           20121125
>   scikits.odes          20121122
>   scikits.optimization  20121114
>   scikits.fitting       20121029
>   scikit-fmm            20121016
>   scikit-image          20121014
>   scikits-image         20121014
>   scikit-learn          20121008
>   scikits.bvp_solver    20120926
>   scikit-rf             20120925
>   scikits.bootstrap     20120822
>   scikit-vi             20120609
>   scikit-commpy         20120404
>   scikits.learn         20110921
>   scikits.statsmodels   20110824
>

renamed to statsmodels to drop the namespace package problems
Skipper just asked the question whether it's possible to point appspot to
our new pypi page
http://pypi.python.org/pypi/statsmodels

(easy_install statsmodels actually installs the preliminary 0.5. version
https://github.com/statsmodels/statsmodels/downloads )


>   scikits.cuda          20110523
>   scikits.datasmooth    20110414
>   scikits.audiolab      20100723
>   scikits.sparse        20091214
>   scikits.bvp1lg        20091012
>   scikits.talkbox       20090827
>   scikits.samplerate    20090722
>   scikits.vectorplot    20081202
>   scikits.example       20081105
>   scikits.ann           20080131
>
> What I want to suggest from this little adventure is to sort scikits
> packages by last update, on scikits home page. Then maybe scikits not
> updated for more then a year, flag with new status column, describing the
> state of the package. I believe it would help general user from start, and
> to potential developer for takeover. Also it will show that scikits idea is
> active, instead current static display with half packages in unknown state.
>
> As for those in first listing, "abandoned" or "deprecated" is IMHO
> suitable, if not necessary.
> [image: Inline image 1]
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121218/a7a910c0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 8910 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121218/a7a910c0/attachment.png>

From klonuo at gmail.com  Tue Dec 18 21:31:12 2012
From: klonuo at gmail.com (klo uo)
Date: Wed, 19 Dec 2012 03:31:12 +0100
Subject: [SciPy-User] Scikits statuses?
In-Reply-To: <CAMMTP+AVq+qoAjMOk+huWhDv8KDwQWqMHaHC6ctoSG9wDn-39w@mail.gmail.com>
References: <CAA-8Ld9xXJm1P3tYSgvJd7RCy_rc3LbHbSOkebQV0-qC8CdouQ@mail.gmail.com>
	<CAMMTP+AVq+qoAjMOk+huWhDv8KDwQWqMHaHC6ctoSG9wDn-39w@mail.gmail.com>
Message-ID: <CAA-8Ld8HH+80X3JE6Ha6FsV5XO2Tam-xyxSig4QaN9oE7XW9fg@mail.gmail.com>

On Wed, Dec 19, 2012 at 2:07 AM, <josef.pktd at gmail.com> wrote:

>
>   scikits.statsmodels   20110824
>>
>
> renamed to statsmodels to drop the namespace package problems
> Skipper just asked the question whether it's possible to point appspot to
> our new pypi page
> http://pypi.python.org/pypi/statsmodels
>
> (easy_install statsmodels actually installs the preliminary 0.5. version
> https://github.com/statsmodels/statsmodels/downloads )
>

I overlooked that some packages offer multiple releases of same version and
even that I didn't consider counts for each version!
I can correct the first, but I can't make pypi list all versions for each
package, right now. If I make it, I'll post corrected plot, just because it
would be corrected.

I also forgot to include version numbers, and I use this opportunity to do
so:

  scikit-aero           0.1          20121125
  scikits.odes          2.0.2        20121122
  scikits.optimization  0.2          20121114
  scikits.fitting       0.5.1        20121029
  scikit-fmm            0.0.4        20121016
  scikit-image          0.7.2        20121014
  scikits-image         0.7.1        20121014
  scikit-learn          0.12.1       20121008
  scikits.bvp_solver    1.1          20120926
  scikit-rf             0.12         20120925
  scikits.bootstrap     0.2dev       20120822
  statsmodels           0.4.3        20120702
  scikit-vi             0.1          20120609
  scikit-commpy         0.1.0        20120404
  scikits.learn         0.8.1        20110921
  scikits.cuda          0.041        20110523
  scikits.datasmooth    0.61         20110414
  scikits.audiolab      0.11.0       20100723
  scikits.sparse        0.1          20091214
  scikits.bvp1lg        0.2.5        20091012
  scikits.talkbox       0.2.5        20090827
  scikits.samplerate    0.3.3        20090722
  scikits.vectorplot    0.1.1        20081202
  scikits.example       0.1          20081105
  scikits.ann           0.2.dev-r803 20080131
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121219/d7a13e72/attachment.html>

From klonuo at gmail.com  Tue Dec 18 22:00:28 2012
From: klonuo at gmail.com (klo uo)
Date: Wed, 19 Dec 2012 04:00:28 +0100
Subject: [SciPy-User] Scikits statuses?
In-Reply-To: <CAA-8Ld8HH+80X3JE6Ha6FsV5XO2Tam-xyxSig4QaN9oE7XW9fg@mail.gmail.com>
References: <CAA-8Ld9xXJm1P3tYSgvJd7RCy_rc3LbHbSOkebQV0-qC8CdouQ@mail.gmail.com>
	<CAMMTP+AVq+qoAjMOk+huWhDv8KDwQWqMHaHC6ctoSG9wDn-39w@mail.gmail.com>
	<CAA-8Ld8HH+80X3JE6Ha6FsV5XO2Tam-xyxSig4QaN9oE7XW9fg@mail.gmail.com>
Message-ID: <CAA-8Ld9HTwBxFKQ9yAecjsCiGfAUKoKL_B5+5PR49Lqkt4yyaw@mail.gmail.com>

On Wed, Dec 19, 2012 at 3:31 AM, klo uo <klonuo at gmail.com> wrote:

> If I make it, I'll post corrected plot, just because it would be corrected.
>
> Here is code that takes in consideration all available pypi version and
different packaging for each version:

========================================
dat = {}
for kit in kits:
    try:
        if not 'scikit' in kit:  # help pypi find the right package
            kit = 'scikits.' + kit
        k = pypi.search({'name':kit})[-1]
        versions = pypi.package_releases(k['name'], 1)
        dl = 0
        for v in versions:
            d = pypi.release_urls(k['name'], v)
            for package in d:
                dl += package['downloads']
        dat[kit] = {'name':k['name'], 'last_version':k['version'],
'versions':versions,
                    'upload_time':d[0]['upload_time'], 'downloads':dl}
    except IndexError:  # package either not in pypi or w/o release urls
        print kit.replace('scikits.', '')
========================================

I added 'statsmodels' w/o scikit prefix by hand. Here is resultant table
with overall d/l count (and plot is in attachment):

  scikits.fitting       894    ['0.5.1', '0.5']
  scikits.cuda          1765   ['0.041', '0.04']
  scikit-aero           111    ['0.1']
  scikit-commpy         469    ['0.1.0']
  scikits.samplerate    5480   ['0.3.3', '0.3.2', '0.3.1', '0.3.0']
  scikits.bvp_solver    9395   ['1.1', '0.3.0', '0.2.5', '0.2.4', '0.2.3',
'0.2.2']
  scikit-fmm            2357   ['0.0.4', '0.0.3', '0.0.2', '0.0.1']
  scikit-rf             2737   ['0.12', '0.11', '0.1']
  scikits.statsmodels   18853  ['0.3.1', '0.3.0rc1', '0.3.0', '0.2.0',
'0.1.0b1']
  scikits.optimization  2184   ['0.2', '0.1']
  scikits.audiolab      20368  ['0.11.0', '0.10.2', '0.10.1', '0.10.0',
'0.9.0', '0.8']
  scikits.talkbox       7761   ['0.2.5', '0.2.4.dev', '0.2.3', '0.2.2',
'0.2.1', '0.2']
  scikits.datasmooth    1588   ['0.61', '0.5']
  scikit-image          1329   ['0.7.2']
  scikits-image         7743   ['0.7.1', '0.7.0', '0.6.1', '0.6', '0.5',
'0.4.2', '0.4.1']
  scikit-vi             316    ['0.1']
  scikits.vectorplot    2010   ['0.1.1', '0.1']
  scikits.ann           8243   ['0.2.dev-r803', '0.2.dev-r801',
'0.2.dev-r800']
  scikits.scattpy       555    ['0.1.1', '0.1.0']
  scikit-learn          39756  ['0.12.1', '0.12', '0.11', '0.10', '0.9']
  scikits.sparse        1012   ['0.1']
  scikits.example       2833   ['0.1']
  statsmodels           12091  ['0.4.3', '0.4.1', '0.4.0rc2', '0.4.0']
  scikits.bvp1lg        1117   ['0.2.5']
  scikits.learn         7166   ['0.8.1']

[image: Inline image 1]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121219/f938dc81/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 10461 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121219/f938dc81/attachment.png>

From andrewhdmundell at gmail.com  Wed Dec 19 07:16:39 2012
From: andrewhdmundell at gmail.com (amundell)
Date: Wed, 19 Dec 2012 04:16:39 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Two Sample Kolmogorov-Smirnov Test scipy
	vs R
Message-ID: <34814758.post@talk.nabble.com>


I am currently creating a statistical app where I am comparing my hypothesis
test results with R and Python (scipy) libraries. So far so good with most
test. However I have found a discrepancy with the R and Python results for
the Two-Sample Kolmogorov-Smirnov Tests. Below are data vectors I have been
using obviously formatted for both R ks.test and
scipy.stat.msstats.ks_twosamp methods.

sample1=[23.4, 30.9, 18.8, 23.0, 21.4, 1, 24.6, 23.8, 24.1, 18.7, 16.3,
20.3,
             14.9, 35.4, 21.6, 21.2, 21.0, 15.0, 15.6, 24.0, 34.6, 40.9,
30.7, 
             24.5, 16.6, 1, 21.7, 1, 23.6, 1, 25.7, 19.3, 46.9, 23.3, 21.8,
33.3, 
             24.9, 24.4, 1, 19.8, 17.2, 21.5, 25.5, 23.3, 18.6, 22.0, 29.8,
33.3,
             1, 21.3, 18.6, 26.8, 19.4, 21.1, 21.2, 20.5, 19.8, 26.3, 39.3,
21.4, 
             22.6, 1, 35.3, 7.0, 19.3, 21.3, 10.1, 20.2, 1, 36.2, 16.7,
21.1, 39.1,
             19.9, 32.1, 23.1, 21.8, 30.4, 19.62, 15.5]

sample2=[16.5, 1, 22.6, 25.3, 23.7, 1, 23.3, 23.9, 16.2, 23.0, 21.6, 10.8,
12.2,
             23.6, 10.1, 24.4, 16.4, 11.7, 17.7, 34.3, 24.3, 18.7, 27.5,
25.8, 22.5,
             14.2, 21.7, 1, 31.2, 13.8, 29.7, 23.1, 26.1, 25.1, 23.4, 21.7,
24.4, 13.2,
             22.1, 26.7, 22.7, 1, 18.2, 28.7, 29.1, 27.4, 22.3, 13.2, 22.5,
25.0, 1,
             6.6, 23.7, 23.5, 17.3, 24.6, 27.8, 29.7, 25.3, 19.9, 18.2,
26.2, 20.4,
             23.3, 26.7, 26.0, 1, 25.1, 33.1, 35.0, 25.3, 23.6, 23.2, 20.2,
24.7, 22.6,
            39.1, 26.5, 22.7]

Running the tests:
R: 
TT = ks.test(sample1, sample2)
TG = ks.test(sample1, sample2, alternative="greater")
TL = ks.test(sample1, sample2, alternative="less")

TT Result: D = 0.2204, p-value = 0.04205   alternative hypothesis: two-sided 
TG Result: D^+ = 0.2204, p-value = 0.02102  alternative hypothesis: the CDF
of x lies above that of y 
TL Result: D^- = 0.1242, p-value = 0.2933  alternative hypothesis: the CDF
of x lies below that of y 

Scipy:

TT=scipy.stats.mstats.ks_twosamp(sample1, sample2)
TU=scipy.stats.mstats.ks_twosamp(sample1, sample2, alternative='greater')
TL=scipy.stats.mstats.ks_twosamp(sample1, sample2, alternative='less')

TT Result: D= 0.220411392405, p-value= 0.0420492678738 
TU Result: D= 0.124208860759 p-value: 0.293327703926 
TL Result: D=: 0.220411392405, p-value: 0.0210248293393 

So as it can be seen from the results the one tailed upper and lower values
seemed to be reversed. In my app my results were more consistent with R's.
Am I missing something obvious here i.e. with definitions? or is there
potentially a bug in the scipy code?
Any help will be much appreciated. Cheers.




-- 
View this message in context: http://old.nabble.com/Two-Sample-Kolmogorov-Smirnov-Test-scipy-vs-R-tp34814758p34814758.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From josef.pktd at gmail.com  Wed Dec 19 10:11:29 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 19 Dec 2012 10:11:29 -0500
Subject: [SciPy-User] [SciPy-user] Two Sample Kolmogorov-Smirnov Test
 scipy vs R
In-Reply-To: <34814758.post@talk.nabble.com>
References: <34814758.post@talk.nabble.com>
Message-ID: <CAMMTP+CZemt1QHQiw7NXAJ7TdxXQ_Nd-=Fh96nLE-b1ZC10S9Q@mail.gmail.com>

On Wed, Dec 19, 2012 at 7:16 AM, amundell <andrewhdmundell at gmail.com> wrote:
>
> I am currently creating a statistical app where I am comparing my hypothesis
> test results with R and Python (scipy) libraries. So far so good with most
> test. However I have found a discrepancy with the R and Python results for
> the Two-Sample Kolmogorov-Smirnov Tests. Below are data vectors I have been
> using obviously formatted for both R ks.test and
> scipy.stat.msstats.ks_twosamp methods.
>
> sample1=[23.4, 30.9, 18.8, 23.0, 21.4, 1, 24.6, 23.8, 24.1, 18.7, 16.3,
> 20.3,
>              14.9, 35.4, 21.6, 21.2, 21.0, 15.0, 15.6, 24.0, 34.6, 40.9,
> 30.7,
>              24.5, 16.6, 1, 21.7, 1, 23.6, 1, 25.7, 19.3, 46.9, 23.3, 21.8,
> 33.3,
>              24.9, 24.4, 1, 19.8, 17.2, 21.5, 25.5, 23.3, 18.6, 22.0, 29.8,
> 33.3,
>              1, 21.3, 18.6, 26.8, 19.4, 21.1, 21.2, 20.5, 19.8, 26.3, 39.3,
> 21.4,
>              22.6, 1, 35.3, 7.0, 19.3, 21.3, 10.1, 20.2, 1, 36.2, 16.7,
> 21.1, 39.1,
>              19.9, 32.1, 23.1, 21.8, 30.4, 19.62, 15.5]
>
> sample2=[16.5, 1, 22.6, 25.3, 23.7, 1, 23.3, 23.9, 16.2, 23.0, 21.6, 10.8,
> 12.2,
>              23.6, 10.1, 24.4, 16.4, 11.7, 17.7, 34.3, 24.3, 18.7, 27.5,
> 25.8, 22.5,
>              14.2, 21.7, 1, 31.2, 13.8, 29.7, 23.1, 26.1, 25.1, 23.4, 21.7,
> 24.4, 13.2,
>              22.1, 26.7, 22.7, 1, 18.2, 28.7, 29.1, 27.4, 22.3, 13.2, 22.5,
> 25.0, 1,
>              6.6, 23.7, 23.5, 17.3, 24.6, 27.8, 29.7, 25.3, 19.9, 18.2,
> 26.2, 20.4,
>              23.3, 26.7, 26.0, 1, 25.1, 33.1, 35.0, 25.3, 23.6, 23.2, 20.2,
> 24.7, 22.6,
>             39.1, 26.5, 22.7]
>
> Running the tests:
> R:
> TT = ks.test(sample1, sample2)
> TG = ks.test(sample1, sample2, alternative="greater")
> TL = ks.test(sample1, sample2, alternative="less")
>
> TT Result: D = 0.2204, p-value = 0.04205   alternative hypothesis: two-sided
> TG Result: D^+ = 0.2204, p-value = 0.02102  alternative hypothesis: the CDF
> of x lies above that of y
> TL Result: D^- = 0.1242, p-value = 0.2933  alternative hypothesis: the CDF
> of x lies below that of y
>
> Scipy:
>
> TT=scipy.stats.mstats.ks_twosamp(sample1, sample2)
> TU=scipy.stats.mstats.ks_twosamp(sample1, sample2, alternative='greater')
> TL=scipy.stats.mstats.ks_twosamp(sample1, sample2, alternative='less')
>
> TT Result: D= 0.220411392405, p-value= 0.0420492678738
> TU Result: D= 0.124208860759 p-value: 0.293327703926
> TL Result: D=: 0.220411392405, p-value: 0.0210248293393
>
> So as it can be seen from the results the one tailed upper and lower values
> seemed to be reversed. In my app my results were more consistent with R's.
> Am I missing something obvious here i.e. with definitions? or is there
> potentially a bug in the scipy code?
> Any help will be much appreciated. Cheers.

It's not really a bug, since the documentation for mstats.ks_2samp
doesn't specify what is meant by greater or less.

But I think this should be clarified in the documentation and changed.
For stats.kstest I followed the R definition for the one-sided tests, IIRC


Aside:

One part that I always find confusing in this is that having a larger
cdf means that the random values are smaller (in a stochastic
dominance sense)
http://en.wikipedia.org/wiki/Stochastic_dominance#First-order_stochastic_dominance
"A dominating B means that F_A(x) <= F_B(x) for all x, with strict
inequality at some x"

However Kolmogorov-Smirnov only looks at the maximum deviation of the
cdfs in either direction.
In your example we have several intersections

import matplotlib.pyplot as plt
plt.figure()
n1 = len(sample1)
n2 = len(sample2)
plt.step(np.sort(sample1), np.arange(1, n1+1)/(n1+1.), label='sample1')
plt.step(np.sort(sample2), np.arange(1, n2+1)/(n2+1.), label='sample2')
plt.legend()

import statsmodels.graphics.gofplots as smgp
fig2 = smgp.qqplot_2samples(np.asarray(sample1)[:-1],
np.asarray(sample2), line='45') #requires equal length
plt.show()

Josef
>
>
>
>
> --
> View this message in context: http://old.nabble.com/Two-Sample-Kolmogorov-Smirnov-Test-scipy-vs-R-tp34814758p34814758.html
> Sent from the Scipy-User mailing list archive at Nabble.com.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From helmrp at yahoo.com  Wed Dec 19 10:39:30 2012
From: helmrp at yahoo.com (The Helmbolds)
Date: Wed, 19 Dec 2012 07:39:30 -0800 (PST)
Subject: [SciPy-User] Problem with ODE
References: <mailman.247.1355846077.1036.scipy-user@scipy.org>   
Message-ID: <1355931570.2729.YahooMailNeo@web31804.mail.mud.yahoo.com>

On 18 Dec 2012 Issa wrote (heavily abbreviated):

????"I am trying to integrate a simple ode which is succesfull when calling
????'vode' integrator. However,
????with 'dopri5' or 'dop853' as integrator, I have the following error: . . "

I have no idea what the error message means, but the following may be of some help.
Suppose we want to solve the following system of?ODEs:
??????? dx/dt = -Dy
??????? dy/dt = -Ax
for parameters A = 0.01, D = 0.015 and initial
values x(0) = 1000 and y(0) = 1500. We can
put w = (x, y) and proceed as follows:

??? print
??? print "ODE-CLASS Results using DOPRI5"
??? t0 = 0
??? x0 = 1000
??? y0 = 1500
??? w0 = (x0, y0)
??? params = (0.01, 0.015)
??? def func_2(t, w, args):??? # Note reversal: use (t, w, ...) instead of (w, t, ...).
??????? A, D = args???????????? # Note no '*'.
??????? x, y = w
??????? xdot = -D*y
??????? ydot = -A*x
??????? return [xdot, ydot]
??? def J_2(t, w, args):?????? # Jacobian. Note order of (t, w , ...) instead of (w, t, ...).
???????? A, D = args??????????? # Note no '*'.
???????? x, y = w
???????? return [[0, -A], [-D, 0]]
??? import scipy
??? from scipy.integrate import ode

??? # Create an "integrator object" of type "ode"
??? #?? initialized with the above?'func' and?Jacobian.
??? intobj = ode(func_2, J_2)

??? # Set this 'intobj' to use the 'dopri5' integrator,
??? #?? using its defaults.
??? intobj.set_integrator('dopri5')

??? # Set the intobj's initial value and
??? #??? coresponding intial time.
??? intobj.set_initial_value(w0, 0.0)
??? # Set the parameters to be used in intobj's
??? #?? 'func' and Jacobian evaluations.
??? intobj.set_f_params(params)???? # Don't use '*params'.
??? intobj.set_jac_params(params)?? # Don't use '*params'.

??? # Set the final time and time-step to
??? #?? use for reporting the results.
??? tf = 10
??? tstep = 1
??? t0 = 0.0

??? # Display the results, accepting default values
??? #?? for the remaining options.
??? print
??? while intobj.successful() and intobj.t < tf:
??????? intobj.integrate(intobj.t + tstep)
??????? outstr = '%s, %s' %(intobj.t, intobj.y)
??????? print outstr
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++??????? 

??? print
??? print "ODE-CLASS Results using?DOP853"
??? t0 = 0
??? x0 = 1000
??? y0 = 1500
??? w0 = (x0, y0)
??? params = (0.01, 0.015)
??? def func_2(t, w, args):??? # Note reversal: use (t, w, ...) instead of (w, t, ...).
??????? A, D = args???????????? # Note no '*'.
??????? x, y = w
??????? xdot = -D*y
??????? ydot = -A*x
??????? return [xdot, ydot]
??? def J_2(t, w, args):?????? # Jacobian. Note order of (t, w , ...) instead of (w, t, ...).
???????? A, D = args??????????? # Note no '*'.
???????? x, y = w
???????? return [[0, -A], [-D, 0]]
??? import scipy
??? from scipy.integrate import ode

??? # Create an "integrator object" of type "ode"
??? #?? initialized with the above?'func' and?Jacobian.
??? intobj = ode(func_2, J_2)

??? # Set this 'intobj' to use the 'dop853' integrator, 
??? #?? using its defaults.
??? intobj.set_integrator('dop853')

??? # Set the intobj's initial value and
??? #??? coresponding initial time.
??? intobj.set_initial_value(w0, 0.0)
??? # Set the parameters to be used in intobj's
??? #?? 'func' and Jacobian evaluations.
??? intobj.set_f_params(params)???? # Don't use '*params'.
??? intobj.set_jac_params(params)?? # Don't use '*params'.

??? # Set the final time and time-step to
??? #?? use for reporting the results.
??? tf = 10
??? tstep = 1
??? t0 = 0.0

??? # Display the results, accepting default values
??? #?? for the remaining options.
??? print
??? while intobj.successful() and intobj.t < tf:
??????? intobj.integrate(intobj.t + tstep)
??????? outstr = '%s, %s' %(intobj.t, intobj.y)
??????? print outstr
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I get essentially the following?output from either of these, or with the theoretical solution.

?????? 1.0, [? 977.57443843? 1490.1122514 ]
?????? 2.0, [? 955.29551487? 1480.44802244]
?????? 3.0, [? 933.15988742? 1471.00586346]
?????? 4.0, [? 911.1642357?? 1461.78435811]
?????? 5.0, [? 889.30526033? 1452.78212316]
?????? 6.0, [? 867.57968241? 1443.99780825]
?????? 7.0, [? 845.98424307? 1435.43009571]
?????? 8.0, [? 824.51570296? 1427.07770039]
?????? 9.0, [? 803.17084174? 1418.93936939]
?????? 10.0, [? 781.94645766? 1411.01388197]

Hope this helps.
Bob H???????????????????????????????????????


From issa at aims.ac.za  Wed Dec 19 12:31:48 2012
From: issa at aims.ac.za (Issa Karambal)
Date: Wed, 19 Dec 2012 17:31:48 +0000
Subject: [SciPy-User] Problem with ODE
In-Reply-To: <1355931570.2729.YahooMailNeo@web31804.mail.mud.yahoo.com>
References: <mailman.247.1355846077.1036.scipy-user@scipy.org>
	<1355931570.2729.YahooMailNeo@web31804.mail.mud.yahoo.com>
Message-ID: <CAEDf9RBRvDRyozDiZD45fRjcKu8HN4CVn8dOv=9vUfG4cRpqKw@mail.gmail.com>

Hi
Thank to your code I figured out the error I've been getting.
The set_f_params function takes tuple as parameter for the integrators
'dopri5' and 'dopri853' unlike 'vode' integrator.
Thank you very much for your help.

On 19 December 2012 15:39, The Helmbolds <helmrp at yahoo.com> wrote:

> On 18 Dec 2012 Issa wrote (heavily abbreviated):
>
>     "I am trying to integrate a simple ode which is succesfull when calling
>     'vode' integrator. However,
>     with 'dopri5' or 'dop853' as integrator, I have the following error: .
> . "
>
> I have no idea what the error message means, but the following may be of
> some help.
> Suppose we want to solve the following system of ODEs:
>         dx/dt = -Dy
>         dy/dt = -Ax
> for parameters A = 0.01, D = 0.015 and initial
> values x(0) = 1000 and y(0) = 1500. We can
> put w = (x, y) and proceed as follows:
>
>     print
>     print "ODE-CLASS Results using DOPRI5"
>     t0 = 0
>     x0 = 1000
>     y0 = 1500
>     w0 = (x0, y0)
>     params = (0.01, 0.015)
>     def func_2(t, w, args):    # Note reversal: use (t, w, ...) instead of
> (w, t, ...).
>         A, D = args             # Note no '*'.
>         x, y = w
>         xdot = -D*y
>         ydot = -A*x
>         return [xdot, ydot]
>     def J_2(t, w, args):       # Jacobian. Note order of (t, w , ...)
> instead of (w, t, ...).
>          A, D = args            # Note no '*'.
>          x, y = w
>          return [[0, -A], [-D, 0]]
>     import scipy
>     from scipy.integrate import ode
>
>     # Create an "integrator object" of type "ode"
>     #   initialized with the above 'func' and Jacobian.
>     intobj = ode(func_2, J_2)
>
>     # Set this 'intobj' to use the 'dopri5' integrator,
>     #   using its defaults.
>     intobj.set_integrator('dopri5')
>
>     # Set the intobj's initial value and
>     #    coresponding intial time.
>     intobj.set_initial_value(w0, 0.0)
>     # Set the parameters to be used in intobj's
>     #   'func' and Jacobian evaluations.
>     intobj.set_f_params(params)     # Don't use '*params'.
>     intobj.set_jac_params(params)   # Don't use '*params'.
>
>     # Set the final time and time-step to
>     #   use for reporting the results.
>     tf = 10
>     tstep = 1
>     t0 = 0.0
>
>     # Display the results, accepting default values
>     #   for the remaining options.
>     print
>     while intobj.successful() and intobj.t < tf:
>         intobj.integrate(intobj.t + tstep)
>         outstr = '%s, %s' %(intobj.t, intobj.y)
>         print outstr
>
> #+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>     print
>     print "ODE-CLASS Results using DOP853"
>     t0 = 0
>     x0 = 1000
>     y0 = 1500
>     w0 = (x0, y0)
>     params = (0.01, 0.015)
>     def func_2(t, w, args):    # Note reversal: use (t, w, ...) instead of
> (w, t, ...).
>         A, D = args             # Note no '*'.
>         x, y = w
>         xdot = -D*y
>         ydot = -A*x
>         return [xdot, ydot]
>     def J_2(t, w, args):       # Jacobian. Note order of (t, w , ...)
> instead of (w, t, ...).
>          A, D = args            # Note no '*'.
>          x, y = w
>          return [[0, -A], [-D, 0]]
>     import scipy
>     from scipy.integrate import ode
>
>     # Create an "integrator object" of type "ode"
>     #   initialized with the above 'func' and Jacobian.
>     intobj = ode(func_2, J_2)
>
>     # Set this 'intobj' to use the 'dop853' integrator,
>     #   using its defaults.
>     intobj.set_integrator('dop853')
>
>     # Set the intobj's initial value and
>     #    coresponding initial time.
>     intobj.set_initial_value(w0, 0.0)
>     # Set the parameters to be used in intobj's
>     #   'func' and Jacobian evaluations.
>     intobj.set_f_params(params)     # Don't use '*params'.
>     intobj.set_jac_params(params)   # Don't use '*params'.
>
>     # Set the final time and time-step to
>     #   use for reporting the results.
>     tf = 10
>     tstep = 1
>     t0 = 0.0
>
>     # Display the results, accepting default values
>     #   for the remaining options.
>     print
>     while intobj.successful() and intobj.t < tf:
>         intobj.integrate(intobj.t + tstep)
>         outstr = '%s, %s' %(intobj.t, intobj.y)
>         print outstr
> #++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> I get essentially the following output from either of these, or with the
> theoretical solution.
>
>        1.0, [  977.57443843  1490.1122514 ]
>        2.0, [  955.29551487  1480.44802244]
>        3.0, [  933.15988742  1471.00586346]
>        4.0, [  911.1642357   1461.78435811]
>        5.0, [  889.30526033  1452.78212316]
>        6.0, [  867.57968241  1443.99780825]
>        7.0, [  845.98424307  1435.43009571]
>        8.0, [  824.51570296  1427.07770039]
>        9.0, [  803.17084174  1418.93936939]
>        10.0, [  781.94645766  1411.01388197]
>
> Hope this helps.
> Bob H
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121219/8891b1ee/attachment.html>

From andrewhdmundell at gmail.com  Wed Dec 19 13:06:39 2012
From: andrewhdmundell at gmail.com (amundell)
Date: Wed, 19 Dec 2012 10:06:39 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Two Sample Kolmogorov-Smirnov Test
	scipy vs R
In-Reply-To: <CAMMTP+CZemt1QHQiw7NXAJ7TdxXQ_Nd-=Fh96nLE-b1ZC10S9Q@mail.gmail.com>
References: <34814758.post@talk.nabble.com>
	<CAMMTP+CZemt1QHQiw7NXAJ7TdxXQ_Nd-=Fh96nLE-b1ZC10S9Q@mail.gmail.com>
Message-ID: <34816338.post@talk.nabble.com>


Hi Josef,

Thanks for your quick response and information. It does seem a little
confusing as R uses these parameters names as well, but I guess
documentations can clarify this. Incidentally, I did make some comparison
tests with scipy.stat.kstest, with SciPy results appearing more accurate as
confirmed by a third statistical software package. In the two sample test R
and the third software matched closely with their tail data. I am quite new
to working with this type of test and also a little confused with the
meanings of "greater" and "less" in this scenario. 

I am going to make some further investigations into the test theory as well
as the algorithms being used in R and other packages and see if I can come
up with an answer. Will keep you posted.

Thanks again,

Andrew



josef.pktd wrote:
> 
> On Wed, Dec 19, 2012 at 7:16 AM, amundell <andrewhdmundell at gmail.com>
> wrote:
>>
>> I am currently creating a statistical app where I am comparing my
>> hypothesis
>> test results with R and Python (scipy) libraries. So far so good with
>> most
>> test. However I have found a discrepancy with the R and Python results
>> for
>> the Two-Sample Kolmogorov-Smirnov Tests. Below are data vectors I have
>> been
>> using obviously formatted for both R ks.test and
>> scipy.stat.msstats.ks_twosamp methods.
>>
>> sample1=[23.4, 30.9, 18.8, 23.0, 21.4, 1, 24.6, 23.8, 24.1, 18.7, 16.3,
>> 20.3,
>>              14.9, 35.4, 21.6, 21.2, 21.0, 15.0, 15.6, 24.0, 34.6, 40.9,
>> 30.7,
>>              24.5, 16.6, 1, 21.7, 1, 23.6, 1, 25.7, 19.3, 46.9, 23.3,
>> 21.8,
>> 33.3,
>>              24.9, 24.4, 1, 19.8, 17.2, 21.5, 25.5, 23.3, 18.6, 22.0,
>> 29.8,
>> 33.3,
>>              1, 21.3, 18.6, 26.8, 19.4, 21.1, 21.2, 20.5, 19.8, 26.3,
>> 39.3,
>> 21.4,
>>              22.6, 1, 35.3, 7.0, 19.3, 21.3, 10.1, 20.2, 1, 36.2, 16.7,
>> 21.1, 39.1,
>>              19.9, 32.1, 23.1, 21.8, 30.4, 19.62, 15.5]
>>
>> sample2=[16.5, 1, 22.6, 25.3, 23.7, 1, 23.3, 23.9, 16.2, 23.0, 21.6,
>> 10.8,
>> 12.2,
>>              23.6, 10.1, 24.4, 16.4, 11.7, 17.7, 34.3, 24.3, 18.7, 27.5,
>> 25.8, 22.5,
>>              14.2, 21.7, 1, 31.2, 13.8, 29.7, 23.1, 26.1, 25.1, 23.4,
>> 21.7,
>> 24.4, 13.2,
>>              22.1, 26.7, 22.7, 1, 18.2, 28.7, 29.1, 27.4, 22.3, 13.2,
>> 22.5,
>> 25.0, 1,
>>              6.6, 23.7, 23.5, 17.3, 24.6, 27.8, 29.7, 25.3, 19.9, 18.2,
>> 26.2, 20.4,
>>              23.3, 26.7, 26.0, 1, 25.1, 33.1, 35.0, 25.3, 23.6, 23.2,
>> 20.2,
>> 24.7, 22.6,
>>             39.1, 26.5, 22.7]
>>
>> Running the tests:
>> R:
>> TT = ks.test(sample1, sample2)
>> TG = ks.test(sample1, sample2, alternative="greater")
>> TL = ks.test(sample1, sample2, alternative="less")
>>
>> TT Result: D = 0.2204, p-value = 0.04205   alternative hypothesis:
>> two-sided
>> TG Result: D^+ = 0.2204, p-value = 0.02102  alternative hypothesis: the
>> CDF
>> of x lies above that of y
>> TL Result: D^- = 0.1242, p-value = 0.2933  alternative hypothesis: the
>> CDF
>> of x lies below that of y
>>
>> Scipy:
>>
>> TT=scipy.stats.mstats.ks_twosamp(sample1, sample2)
>> TU=scipy.stats.mstats.ks_twosamp(sample1, sample2, alternative='greater')
>> TL=scipy.stats.mstats.ks_twosamp(sample1, sample2, alternative='less')
>>
>> TT Result: D= 0.220411392405, p-value= 0.0420492678738
>> TU Result: D= 0.124208860759 p-value: 0.293327703926
>> TL Result: D=: 0.220411392405, p-value: 0.0210248293393
>>
>> So as it can be seen from the results the one tailed upper and lower
>> values
>> seemed to be reversed. In my app my results were more consistent with
>> R's.
>> Am I missing something obvious here i.e. with definitions? or is there
>> potentially a bug in the scipy code?
>> Any help will be much appreciated. Cheers.
> 
> It's not really a bug, since the documentation for mstats.ks_2samp
> doesn't specify what is meant by greater or less.
> 
> But I think this should be clarified in the documentation and changed.
> For stats.kstest I followed the R definition for the one-sided tests, IIRC
> 
> 
> Aside:
> 
> One part that I always find confusing in this is that having a larger
> cdf means that the random values are smaller (in a stochastic
> dominance sense)
> http://en.wikipedia.org/wiki/Stochastic_dominance#First-order_stochastic_dominance
> "A dominating B means that F_A(x) <= F_B(x) for all x, with strict
> inequality at some x"
> 
> However Kolmogorov-Smirnov only looks at the maximum deviation of the
> cdfs in either direction.
> In your example we have several intersections
> 
> import matplotlib.pyplot as plt
> plt.figure()
> n1 = len(sample1)
> n2 = len(sample2)
> plt.step(np.sort(sample1), np.arange(1, n1+1)/(n1+1.), label='sample1')
> plt.step(np.sort(sample2), np.arange(1, n2+1)/(n2+1.), label='sample2')
> plt.legend()
> 
> import statsmodels.graphics.gofplots as smgp
> fig2 = smgp.qqplot_2samples(np.asarray(sample1)[:-1],
> np.asarray(sample2), line='45') #requires equal length
> plt.show()
> 
> Josef
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Two-Sample-Kolmogorov-Smirnov-Test-scipy-vs-R-tp34814758p34814758.html
>> Sent from the Scipy-User mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 
-- 
View this message in context: http://old.nabble.com/Two-Sample-Kolmogorov-Smirnov-Test-scipy-vs-R-tp34814758p34816338.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From p.zaffino at yahoo.it  Wed Dec 19 19:11:35 2012
From: p.zaffino at yahoo.it (Paolo Zaffino)
Date: Thu, 20 Dec 2012 01:11:35 +0100
Subject: [SciPy-User] Points fitting (non lin)
In-Reply-To: <089C98C7-4E58-46EC-83A7-CDE506C1634C@yale.edu>
References: <1355500677.78276.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<CAAih6WmEZVRkmoVD6R0nuXT_kfCS256KZJ+8XmoZ==4RYXN33g@mail.gmail.com>
	<1355503367.42200.YahooMailNeo@web171602.mail.ir2.yahoo.com>
	<9E04AB38-D38C-4C6D-881E-E294898EA3AC@yale.edu>
	<50CDDACF.10305@yahoo.it>
	<089C98C7-4E58-46EC-83A7-CDE506C1634C@yale.edu>
Message-ID: <50D257B7.8090106@yahoo.it>

Dear Zazh,
thank you very much...I have understood now.

Regards.
Paolo


Il 16/12/2012 20:44, Zachary Pincus ha scritto:
>> Dear Zach,
>> I didn't understand the trick...sorry.
>> Please, can you explain it again?
> You'll want to read up about parametric representation of curves on wikipedia etc.
>
> Obviously, the coordinates you listed:
> P1 = (1,1)
> P2 = (2,2)
> P3 = (4,2)
> P4 = (3,1)
> do not trace out y values that are a proper function of, because there are some places where the curve would have multiple y-values for a given x-value (at 3, say). So, as you correctly understood, you can't just try to fit a function y=f(x) to your data and expect that to work.
>
> What you can, however, do is fit TWO functions to your data, using a dummy variable. For example, imagine a point moving along your curve at a given speed, so that at any given time t, the position of the point is given as (x(t), y(t)).
>
> So for each of your points above P1-P4, associate a value T1-T4.
>
> For example,
> t  x  y
> 0  1  1
> 1  2  2
> 2  4  2
> 3  3  1
>
> is just your data above but with a t parameter indexing each point. So at t=0, x=1 and y=1. At t=1, x=2 and y=2. If you had a functional form for x(t) and y(t), you could then estimate where the point on the curve that was midway between the first (t=0) and second (t=1) points by evaluating x(0.5) and y(0.5).
>
> To do this, one could fit x(t) and y(t) separately (because as long as the t values are monotonic, x(t) and y(t) are proper functions for any smooth curve that you could draw on a plane). Fitting a spline to these data is the most straightforward way to do this, and fortunately, scipy.interpolate.splprep does this all for you and gives you a spline that can be evaluated at different t positions by splev.
>
> If spline fitting is unfamiliar to you, then parametric spline fitting will seem even less comprehensible. So you might want to experiment by fitting other simpler functional curves in scipy with splrep and then moving on. (There should be interpolation examples aplenty online.)
>
> Zach
>
>
>
>> Thanks a lot.
>> Paolo
>>
>>
>> Il 14/12/2012 17:50, Zachary Pincus ha scritto:
>>> On Dec 14, 2012, at 11:42 AM, Paolo Zaffino wrote:
>>>
>>>> Dear Pawe?,
>>>> thank you for the reply.
>>>> I try to explain better the issue.
>>>> I have these points (in this order):
>>>>
>>>> P1 = (1,1)
>>>> P2 = (2,2)
>>>> P3 = (4,2)
>>>> P4 = (3,1)
>>>>
>>>> I need to fit the points in the order P1,P2,P3,P4 even if the x coord of P3 is greater than P4.
>>>> I thought to quadratic piecewise curve but other solutions are welcome.
>>>>
>>>> Thanks a lot.
>>>> Paolo
>>> You will want to fit a parametric spline of some degree with some amount (or no) smoothing. I'd look at the splprep function in scipy.interpolate.
>>>
>>> The trick is you associate each point with some monotonic parameter value, and then interpolate along that parameter (say t) to get your x, y coordinates.
>>>
>>> E.g.:
>>> t  x  y
>>> 0  1  1
>>> 1  2  2
>>> 2  4  2
>>> 3  3  1
>>>
>>> Then if you were interpolating linearly, at t=0.5, you would have (1.5, 1.5) as the coordinate.
>>>
>>> As above, splprep will generate splines of a desired order (linear, quadratic, cubic, etc.) and with a user-specified smoothing parameter (s), which can be set to zero to get exact interpolation of the input coordinates, potentially at the cost of ringing (sometimes quite bad) away from the input coordinate. So you will need to plot the interpolated values, both at the input t-values, as well as at intermediate t's, to see if the output is sane.
>>>
>>> Hopefully this is somewhat clear, or at least enough to get you started. Please read the documentation for splprep and splev for more information.
>>> Zach
>>>
>>>
>>>
>>>> Da: Pawe? Kwa?niewski <pawel.kw at gmail.com>
>>>> A: Paolo Zaffino <p.zaffino at yahoo.it>; SciPy Users List <scipy-user at scipy.org>
>>>> Inviato: Venerd? 14 Dicembre 2012 17:25
>>>> Oggetto: Re: [SciPy-User] Points fitting (non lin)
>>>>
>>>> Dear Paolo,
>>>>
>>>> I'm not sure I understand correctly your problem, but this sounds like a spline fitting job. You can read more about this here: http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html
>>>>
>>>> Is this what you are looking for?
>>>>
>>>> Cheers,
>>>>
>>>> Pawe?
>>>>
>>>>
>>>>
>>>> 2012/12/14 Paolo Zaffino <p.zaffino at yahoo.it>
>>>> Dear Scipy community,
>>>>
>>>> I have a set of points (2D) and I would compute a curve that fits them.
>>>> The points are ordered in a precise way (not crescent order) and I can't change this order (the curve should fit the points in that order).
>>>> I'm interseting in a non linear fit (the ideal case would be more intervals of quadratic curves).
>>>> Has anyone any advice about?
>>>>
>>>> Thank you very much.
>>>> Regards.
>>>> Paolo
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



From michael.aye at ucla.edu  Wed Dec 19 21:52:16 2012
From: michael.aye at ucla.edu (Michael Aye)
Date: Wed, 19 Dec 2012 18:52:16 -0800
Subject: [SciPy-User] spline interpolators silently failing on NaNs
Message-ID: <katuh0$4uf$1@ger.gmane.org>

Hi,

I was just bitten by something nasty.
I am fitting some data from pandas, and as pandas deals so 
fantastically with NaNs, I forgot to check myself before handing the 
data over to InterpolatedUnivariateSpline.
To my surprise and after 2 hours of 'practising' all kinds of 
interpolations, I realised my overlook.
To my next surprise the creation of the interpolator function did not 
complain a single bit (pun not intented). Just the application of the 
interpolator to the new x-data returned only NaNs.

It is also notable, that griddata had no such problem, but I cannot use 
it because I need extrapolation as well.

Should / Could the spline interpolators maybe check for NaNs and 
complain at their existence?

Michael







From andrewhdmundell at gmail.com  Thu Dec 20 06:50:38 2012
From: andrewhdmundell at gmail.com (amundell)
Date: Thu, 20 Dec 2012 03:50:38 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Two Sample Kolmogorov-Smirnov Test
	scipy vs R
In-Reply-To: <34816338.post@talk.nabble.com>
References: <34814758.post@talk.nabble.com>
	<CAMMTP+CZemt1QHQiw7NXAJ7TdxXQ_Nd-=Fh96nLE-b1ZC10S9Q@mail.gmail.com>
	<34816338.post@talk.nabble.com>
Message-ID: <34819082.post@talk.nabble.com>


Hi again Josef, after investigating a few different alogrithm approaches
(most notably in Press, Teukolsky et al. Numerical Recipes) and looking into
R's source), it does seem that 'greater' and 'less' does refer to maximum
(D) deviation of the first sample's (sample1's) cdf 'above' and 'below' the
sample2's cdf. By observing and calculating the maximum values in the plot
of my data, they are consistent with the results that SciPy
(scipy.stats.mstats.ks_twosamp) generate. I have just confirmed I am
generating similar results in a third software package to SciPy. I therefore
believe SciPy's results are accurate.

Andrew


amundell wrote:
> 
> Hi Josef,
> 
> Thanks for your quick response and information. It does seem a little
> confusing as R uses these parameters names as well, but I guess
> documentations can clarify this. Incidentally, I did make some comparison
> tests with scipy.stat.kstest, with SciPy results appearing more accurate
> as confirmed by a third statistical software package. In the two sample
> test R and the third software matched closely with their tail data. I am
> quite new to working with this type of test and also a little confused
> with the meanings of "greater" and "less" in this scenario. 
> 
> I am going to make some further investigations into the test theory as
> well as the algorithms being used in R and other packages and see if I can
> come up with an answer. Will keep you posted.
> 
> Thanks again,
> 
> Andrew
> 
> 
> 
> josef.pktd wrote:
>> 
>> On Wed, Dec 19, 2012 at 7:16 AM, amundell <andrewhdmundell at gmail.com>
>> wrote:
>>>
>>> I am currently creating a statistical app where I am comparing my
>>> hypothesis
>>> test results with R and Python (scipy) libraries. So far so good with
>>> most
>>> test. However I have found a discrepancy with the R and Python results
>>> for
>>> the Two-Sample Kolmogorov-Smirnov Tests. Below are data vectors I have
>>> been
>>> using obviously formatted for both R ks.test and
>>> scipy.stat.msstats.ks_twosamp methods.
>>>
>>> sample1=[23.4, 30.9, 18.8, 23.0, 21.4, 1, 24.6, 23.8, 24.1, 18.7, 16.3,
>>> 20.3,
>>>              14.9, 35.4, 21.6, 21.2, 21.0, 15.0, 15.6, 24.0, 34.6, 40.9,
>>> 30.7,
>>>              24.5, 16.6, 1, 21.7, 1, 23.6, 1, 25.7, 19.3, 46.9, 23.3,
>>> 21.8,
>>> 33.3,
>>>              24.9, 24.4, 1, 19.8, 17.2, 21.5, 25.5, 23.3, 18.6, 22.0,
>>> 29.8,
>>> 33.3,
>>>              1, 21.3, 18.6, 26.8, 19.4, 21.1, 21.2, 20.5, 19.8, 26.3,
>>> 39.3,
>>> 21.4,
>>>              22.6, 1, 35.3, 7.0, 19.3, 21.3, 10.1, 20.2, 1, 36.2, 16.7,
>>> 21.1, 39.1,
>>>              19.9, 32.1, 23.1, 21.8, 30.4, 19.62, 15.5]
>>>
>>> sample2=[16.5, 1, 22.6, 25.3, 23.7, 1, 23.3, 23.9, 16.2, 23.0, 21.6,
>>> 10.8,
>>> 12.2,
>>>              23.6, 10.1, 24.4, 16.4, 11.7, 17.7, 34.3, 24.3, 18.7, 27.5,
>>> 25.8, 22.5,
>>>              14.2, 21.7, 1, 31.2, 13.8, 29.7, 23.1, 26.1, 25.1, 23.4,
>>> 21.7,
>>> 24.4, 13.2,
>>>              22.1, 26.7, 22.7, 1, 18.2, 28.7, 29.1, 27.4, 22.3, 13.2,
>>> 22.5,
>>> 25.0, 1,
>>>              6.6, 23.7, 23.5, 17.3, 24.6, 27.8, 29.7, 25.3, 19.9, 18.2,
>>> 26.2, 20.4,
>>>              23.3, 26.7, 26.0, 1, 25.1, 33.1, 35.0, 25.3, 23.6, 23.2,
>>> 20.2,
>>> 24.7, 22.6,
>>>             39.1, 26.5, 22.7]
>>>
>>> Running the tests:
>>> R:
>>> TT = ks.test(sample1, sample2)
>>> TG = ks.test(sample1, sample2, alternative="greater")
>>> TL = ks.test(sample1, sample2, alternative="less")
>>>
>>> TT Result: D = 0.2204, p-value = 0.04205   alternative hypothesis:
>>> two-sided
>>> TG Result: D^+ = 0.2204, p-value = 0.02102  alternative hypothesis: the
>>> CDF
>>> of x lies above that of y
>>> TL Result: D^- = 0.1242, p-value = 0.2933  alternative hypothesis: the
>>> CDF
>>> of x lies below that of y
>>>
>>> Scipy:
>>>
>>> TT=scipy.stats.mstats.ks_twosamp(sample1, sample2)
>>> TU=scipy.stats.mstats.ks_twosamp(sample1, sample2,
>>> alternative='greater')
>>> TL=scipy.stats.mstats.ks_twosamp(sample1, sample2, alternative='less')
>>>
>>> TT Result: D= 0.220411392405, p-value= 0.0420492678738
>>> TU Result: D= 0.124208860759 p-value: 0.293327703926
>>> TL Result: D=: 0.220411392405, p-value: 0.0210248293393
>>>
>>> So as it can be seen from the results the one tailed upper and lower
>>> values
>>> seemed to be reversed. In my app my results were more consistent with
>>> R's.
>>> Am I missing something obvious here i.e. with definitions? or is there
>>> potentially a bug in the scipy code?
>>> Any help will be much appreciated. Cheers.
>> 
>> It's not really a bug, since the documentation for mstats.ks_2samp
>> doesn't specify what is meant by greater or less.
>> 
>> But I think this should be clarified in the documentation and changed.
>> For stats.kstest I followed the R definition for the one-sided tests,
>> IIRC
>> 
>> 
>> Aside:
>> 
>> One part that I always find confusing in this is that having a larger
>> cdf means that the random values are smaller (in a stochastic
>> dominance sense)
>> http://en.wikipedia.org/wiki/Stochastic_dominance#First-order_stochastic_dominance
>> "A dominating B means that F_A(x) <= F_B(x) for all x, with strict
>> inequality at some x"
>> 
>> However Kolmogorov-Smirnov only looks at the maximum deviation of the
>> cdfs in either direction.
>> In your example we have several intersections
>> 
>> import matplotlib.pyplot as plt
>> plt.figure()
>> n1 = len(sample1)
>> n2 = len(sample2)
>> plt.step(np.sort(sample1), np.arange(1, n1+1)/(n1+1.), label='sample1')
>> plt.step(np.sort(sample2), np.arange(1, n2+1)/(n2+1.), label='sample2')
>> plt.legend()
>> 
>> import statsmodels.graphics.gofplots as smgp
>> fig2 = smgp.qqplot_2samples(np.asarray(sample1)[:-1],
>> np.asarray(sample2), line='45') #requires equal length
>> plt.show()
>> 
>> Josef
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Two-Sample-Kolmogorov-Smirnov-Test-scipy-vs-R-tp34814758p34814758.html
>>> Sent from the Scipy-User mailing list archive at Nabble.com.
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> 
>> 
> 
-- 
View this message in context: http://old.nabble.com/Two-Sample-Kolmogorov-Smirnov-Test-scipy-vs-R-tp34814758p34819082.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From josef.pktd at gmail.com  Thu Dec 20 08:16:48 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 20 Dec 2012 08:16:48 -0500
Subject: [SciPy-User] [SciPy-user] Two Sample Kolmogorov-Smirnov Test
 scipy vs R
In-Reply-To: <34819082.post@talk.nabble.com>
References: <34814758.post@talk.nabble.com>
	<CAMMTP+CZemt1QHQiw7NXAJ7TdxXQ_Nd-=Fh96nLE-b1ZC10S9Q@mail.gmail.com>
	<34816338.post@talk.nabble.com> <34819082.post@talk.nabble.com>
Message-ID: <CAMMTP+BrWM43+TKb0MxHyKhv7E23t7P90FRKESX-wpyYPTWGbw@mail.gmail.com>

On Thu, Dec 20, 2012 at 6:50 AM, amundell <andrewhdmundell at gmail.com> wrote:
>
> Hi again Josef, after investigating a few different alogrithm approaches
> (most notably in Press, Teukolsky et al. Numerical Recipes) and looking into
> R's source), it does seem that 'greater' and 'less' does refer to maximum
> (D) deviation of the first sample's (sample1's) cdf 'above' and 'below' the
> sample2's cdf. By observing and calculating the maximum values in the plot
> of my data, they are consistent with the results that SciPy
> (scipy.stats.mstats.ks_twosamp) generate. I have just confirmed I am
> generating similar results in a third software package to SciPy. I therefore
> believe SciPy's results are accurate.

The results are numerically correct, but the definition of 'greater'
and 'less' is still reversed in scipy.stats.mstats.?

BTW there is a tie-correction in the mstats version, where I don't
know whether it has the same asymptotic distribution for the test
statistic.

Josef


>
> Andrew
>
>
> amundell wrote:
>>
>> Hi Josef,
>>
>> Thanks for your quick response and information. It does seem a little
>> confusing as R uses these parameters names as well, but I guess
>> documentations can clarify this. Incidentally, I did make some comparison
>> tests with scipy.stat.kstest, with SciPy results appearing more accurate
>> as confirmed by a third statistical software package. In the two sample
>> test R and the third software matched closely with their tail data. I am
>> quite new to working with this type of test and also a little confused
>> with the meanings of "greater" and "less" in this scenario.
>>
>> I am going to make some further investigations into the test theory as
>> well as the algorithms being used in R and other packages and see if I can
>> come up with an answer. Will keep you posted.
>>
>> Thanks again,
>>
>> Andrew
>>
>>
>>
>> josef.pktd wrote:
>>>
>>> On Wed, Dec 19, 2012 at 7:16 AM, amundell <andrewhdmundell at gmail.com>
>>> wrote:
>>>>
>>>> I am currently creating a statistical app where I am comparing my
>>>> hypothesis
>>>> test results with R and Python (scipy) libraries. So far so good with
>>>> most
>>>> test. However I have found a discrepancy with the R and Python results
>>>> for
>>>> the Two-Sample Kolmogorov-Smirnov Tests. Below are data vectors I have
>>>> been
>>>> using obviously formatted for both R ks.test and
>>>> scipy.stat.msstats.ks_twosamp methods.
>>>>
>>>> sample1=[23.4, 30.9, 18.8, 23.0, 21.4, 1, 24.6, 23.8, 24.1, 18.7, 16.3,
>>>> 20.3,
>>>>              14.9, 35.4, 21.6, 21.2, 21.0, 15.0, 15.6, 24.0, 34.6, 40.9,
>>>> 30.7,
>>>>              24.5, 16.6, 1, 21.7, 1, 23.6, 1, 25.7, 19.3, 46.9, 23.3,
>>>> 21.8,
>>>> 33.3,
>>>>              24.9, 24.4, 1, 19.8, 17.2, 21.5, 25.5, 23.3, 18.6, 22.0,
>>>> 29.8,
>>>> 33.3,
>>>>              1, 21.3, 18.6, 26.8, 19.4, 21.1, 21.2, 20.5, 19.8, 26.3,
>>>> 39.3,
>>>> 21.4,
>>>>              22.6, 1, 35.3, 7.0, 19.3, 21.3, 10.1, 20.2, 1, 36.2, 16.7,
>>>> 21.1, 39.1,
>>>>              19.9, 32.1, 23.1, 21.8, 30.4, 19.62, 15.5]
>>>>
>>>> sample2=[16.5, 1, 22.6, 25.3, 23.7, 1, 23.3, 23.9, 16.2, 23.0, 21.6,
>>>> 10.8,
>>>> 12.2,
>>>>              23.6, 10.1, 24.4, 16.4, 11.7, 17.7, 34.3, 24.3, 18.7, 27.5,
>>>> 25.8, 22.5,
>>>>              14.2, 21.7, 1, 31.2, 13.8, 29.7, 23.1, 26.1, 25.1, 23.4,
>>>> 21.7,
>>>> 24.4, 13.2,
>>>>              22.1, 26.7, 22.7, 1, 18.2, 28.7, 29.1, 27.4, 22.3, 13.2,
>>>> 22.5,
>>>> 25.0, 1,
>>>>              6.6, 23.7, 23.5, 17.3, 24.6, 27.8, 29.7, 25.3, 19.9, 18.2,
>>>> 26.2, 20.4,
>>>>              23.3, 26.7, 26.0, 1, 25.1, 33.1, 35.0, 25.3, 23.6, 23.2,
>>>> 20.2,
>>>> 24.7, 22.6,
>>>>             39.1, 26.5, 22.7]
>>>>
>>>> Running the tests:
>>>> R:
>>>> TT = ks.test(sample1, sample2)
>>>> TG = ks.test(sample1, sample2, alternative="greater")
>>>> TL = ks.test(sample1, sample2, alternative="less")
>>>>
>>>> TT Result: D = 0.2204, p-value = 0.04205   alternative hypothesis:
>>>> two-sided
>>>> TG Result: D^+ = 0.2204, p-value = 0.02102  alternative hypothesis: the
>>>> CDF
>>>> of x lies above that of y
>>>> TL Result: D^- = 0.1242, p-value = 0.2933  alternative hypothesis: the
>>>> CDF
>>>> of x lies below that of y
>>>>
>>>> Scipy:
>>>>
>>>> TT=scipy.stats.mstats.ks_twosamp(sample1, sample2)
>>>> TU=scipy.stats.mstats.ks_twosamp(sample1, sample2,
>>>> alternative='greater')
>>>> TL=scipy.stats.mstats.ks_twosamp(sample1, sample2, alternative='less')
>>>>
>>>> TT Result: D= 0.220411392405, p-value= 0.0420492678738
>>>> TU Result: D= 0.124208860759 p-value: 0.293327703926
>>>> TL Result: D=: 0.220411392405, p-value: 0.0210248293393
>>>>
>>>> So as it can be seen from the results the one tailed upper and lower
>>>> values
>>>> seemed to be reversed. In my app my results were more consistent with
>>>> R's.
>>>> Am I missing something obvious here i.e. with definitions? or is there
>>>> potentially a bug in the scipy code?
>>>> Any help will be much appreciated. Cheers.
>>>
>>> It's not really a bug, since the documentation for mstats.ks_2samp
>>> doesn't specify what is meant by greater or less.
>>>
>>> But I think this should be clarified in the documentation and changed.
>>> For stats.kstest I followed the R definition for the one-sided tests,
>>> IIRC
>>>
>>>
>>> Aside:
>>>
>>> One part that I always find confusing in this is that having a larger
>>> cdf means that the random values are smaller (in a stochastic
>>> dominance sense)
>>> http://en.wikipedia.org/wiki/Stochastic_dominance#First-order_stochastic_dominance
>>> "A dominating B means that F_A(x) <= F_B(x) for all x, with strict
>>> inequality at some x"
>>>
>>> However Kolmogorov-Smirnov only looks at the maximum deviation of the
>>> cdfs in either direction.
>>> In your example we have several intersections
>>>
>>> import matplotlib.pyplot as plt
>>> plt.figure()
>>> n1 = len(sample1)
>>> n2 = len(sample2)
>>> plt.step(np.sort(sample1), np.arange(1, n1+1)/(n1+1.), label='sample1')
>>> plt.step(np.sort(sample2), np.arange(1, n2+1)/(n2+1.), label='sample2')
>>> plt.legend()
>>>
>>> import statsmodels.graphics.gofplots as smgp
>>> fig2 = smgp.qqplot_2samples(np.asarray(sample1)[:-1],
>>> np.asarray(sample2), line='45') #requires equal length
>>> plt.show()
>>>
>>> Josef
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/Two-Sample-Kolmogorov-Smirnov-Test-scipy-vs-R-tp34814758p34814758.html
>>>> Sent from the Scipy-User mailing list archive at Nabble.com.
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
> --
> View this message in context: http://old.nabble.com/Two-Sample-Kolmogorov-Smirnov-Test-scipy-vs-R-tp34814758p34819082.html
> Sent from the Scipy-User mailing list archive at Nabble.com.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From andrewhdmundell at gmail.com  Thu Dec 20 08:34:40 2012
From: andrewhdmundell at gmail.com (amundell)
Date: Thu, 20 Dec 2012 05:34:40 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] Two Sample Kolmogorov-Smirnov Test
	scipy vs R
In-Reply-To: <CAMMTP+BrWM43+TKb0MxHyKhv7E23t7P90FRKESX-wpyYPTWGbw@mail.gmail.com>
References: <34814758.post@talk.nabble.com>
	<CAMMTP+CZemt1QHQiw7NXAJ7TdxXQ_Nd-=Fh96nLE-b1ZC10S9Q@mail.gmail.com>
	<34816338.post@talk.nabble.com> <34819082.post@talk.nabble.com>
	<CAMMTP+BrWM43+TKb0MxHyKhv7E23t7P90FRKESX-wpyYPTWGbw@mail.gmail.com>
Message-ID: <34819413.post@talk.nabble.com>


The results are numerically correct and the definitions of 'greater' and
'less' are correct and are consistent (not reversed) with R. Not sure if I
made some handling errors yesterday (perhaps samples were reversed) or
changes had been made to the code in scipy.stats.mstats in the meantime. If
it was the former (i.e. from my end), I do apologise for taking up your time
with this.

Andrew


On Thu, Dec 20, 2012 at 6:50 AM, amundell <andrewhdmundell at gmail.com> wrote:
>
> Hi again Josef, after investigating a few different alogrithm approaches
> (most notably in Press, Teukolsky et al. Numerical Recipes) and looking
> into
> R's source), it does seem that 'greater' and 'less' does refer to maximum
> (D) deviation of the first sample's (sample1's) cdf 'above' and 'below'
> the
> sample2's cdf. By observing and calculating the maximum values in the plot
> of my data, they are consistent with the results that SciPy
> (scipy.stats.mstats.ks_twosamp) generate. I have just confirmed I am
> generating similar results in a third software package to SciPy. I
> therefore
> believe SciPy's results are accurate.

The results are numerically correct, but the definition of 'greater'
and 'less' is still reversed in scipy.stats.mstats.?

BTW there is a tie-correction in the mstats version, where I don't
know whether it has the same asymptotic distribution for the test
statistic.

Josef


-- 
View this message in context: http://old.nabble.com/Two-Sample-Kolmogorov-Smirnov-Test-scipy-vs-R-tp34814758p34819413.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From a.klein at science-applied.nl  Thu Dec 20 11:16:41 2012
From: a.klein at science-applied.nl (Almar Klein)
Date: Thu, 20 Dec 2012 17:16:41 +0100
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
Message-ID: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>

Dear all,

We're pleased to announce version 3.1 of the Interactive Editor for Python.

IEP is a cross-platform Python IDE focused on interactivity and
introspection, which makes it very suitable for scientific computing. Its
practical design is aimed at simplicity and efficiency. IEP is written in
Python 3 and Qt.

Binaries are available for Windows, Linux, and Mac.

A selection of new features are:

   - a new powerful menu for the shells widget
   - multiple lines can be copy-pasted (and executed) in the shell
   - run selection (F9) runs the selected statement if on a single line
   - multiple comments/docstring reshaping
   - goto-line
   - the shell widget can now be moved on Mac
   - a wizard to help new users on their way


Website: http://code.google.com/p/iep/
Discussion group: http://groups.google.com/group/iep_
Release notes: h*ttp://code.google.com/p/iep/wiki/Release
#version_3.1_(release_candidate_out_now)*

Regards,
  Rob, Ludo & Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121220/9968549e/attachment.html>

From a.klein at science-applied.nl  Thu Dec 20 16:21:18 2012
From: a.klein at science-applied.nl (Almar Klein)
Date: Thu, 20 Dec 2012 22:21:18 +0100
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
Message-ID: <CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>

A note to Ubuntu 12.10 users:
I just found out (thanks to Thomas Kluyver) that on Ubuntu 12.10, IEP has
some nasty problem between Qt and X, related to scrollbar overlays.
I will fix this asap and come with a 3.1.1 release.

- Almar



On 20 December 2012 17:16, Almar Klein <a.klein at science-applied.nl> wrote:

> Dear all,
>
> We're pleased to announce version 3.1 of the Interactive Editor for
> Python.
>
> IEP is a cross-platform Python IDE focused on interactivity and
> introspection, which makes it very suitable for scientific computing. Its
> practical design is aimed at simplicity and efficiency. IEP is written in
> Python 3 and Qt.
>
> Binaries are available for Windows, Linux, and Mac.
>
> A selection of new features are:
>
>    - a new powerful menu for the shells widget
>    - multiple lines can be copy-pasted (and executed) in the shell
>    - run selection (F9) runs the selected statement if on a single line
>    - multiple comments/docstring reshaping
>    - goto-line
>    - the shell widget can now be moved on Mac
>    - a wizard to help new users on their way
>
>
> Website: http://code.google.com/p/iep/
> Discussion group: http://groups.google.com/group/iep_
> Release notes: h*ttp://code.google.com/p/iep/wiki/Release
> #version_3.1_(release_candidate_out_now)*
>
> Regards,
>   Rob, Ludo & Almar
>
>


-- 
Almar Klein, PhD
Science Applied
phone: +31 6 19268652
e-mail: a.klein at science-applied.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121220/2eea606d/attachment.html>

From a.klein at science-applied.nl  Thu Dec 20 21:35:00 2012
From: a.klein at science-applied.nl (Almar Klein)
Date: Fri, 21 Dec 2012 03:35:00 +0100
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
	<CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
Message-ID: <CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>

Binaries for 3.1.1 are now available (for Linux).
- Almar


On 20 December 2012 22:21, Almar Klein <a.klein at science-applied.nl> wrote:

> A note to Ubuntu 12.10 users:
> I just found out (thanks to Thomas Kluyver) that on Ubuntu 12.10, IEP has
> some nasty problem between Qt and X, related to scrollbar overlays.
> I will fix this asap and come with a 3.1.1 release.
>
> - Almar
>
>
>
> On 20 December 2012 17:16, Almar Klein <a.klein at science-applied.nl> wrote:
>
>> Dear all,
>>
>> We're pleased to announce version 3.1 of the Interactive Editor for
>> Python.
>>
>> IEP is a cross-platform Python IDE focused on interactivity and
>> introspection, which makes it very suitable for scientific computing. Its
>> practical design is aimed at simplicity and efficiency. IEP is written
>> in Python 3 and Qt.
>>
>> Binaries are available for Windows, Linux, and Mac.
>>
>> A selection of new features are:
>>
>>    - a new powerful menu for the shells widget
>>    - multiple lines can be copy-pasted (and executed) in the shell
>>    - run selection (F9) runs the selected statement if on a single line
>>    - multiple comments/docstring reshaping
>>    - goto-line
>>    - the shell widget can now be moved on Mac
>>    - a wizard to help new users on their way
>>
>>
>> Website: http://code.google.com/p/iep/
>> Discussion group: http://groups.google.com/group/iep_
>> Release notes: h*ttp://code.google.com/p/iep/wiki/Release
>> #version_3.1_(release_candidate_out_now)*
>>
>> Regards,
>>   Rob, Ludo & Almar
>>
>>
>
>
> --
> Almar Klein, PhD
> Science Applied
> phone: +31 6 19268652
> e-mail: a.klein at science-applied.nl
>



-- 
Almar Klein, PhD
Science Applied
phone: +31 6 19268652
e-mail: a.klein at science-applied.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121221/87317af0/attachment.html>

From fccoelho at gmail.com  Fri Dec 21 05:26:33 2012
From: fccoelho at gmail.com (Flavio Coelho)
Date: Fri, 21 Dec 2012 08:26:33 -0200
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
	<CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
	<CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>
Message-ID: <CAAbyzbRCngsYjzbCv02-6oG+CQVMy=6wf06G17HeMSFy9fTsow@mail.gmail.com>

Would it be too much to ask for a IEP package in PyPI? it would greatly
facilitate installation... Spyder has one.

cheers,

Fl?vio

On Fri, Dec 21, 2012 at 12:35 AM, Almar Klein <a.klein at science-applied.nl>wrote:

> Binaries for 3.1.1 are now available (for Linux).
> - Almar
>
>
> On 20 December 2012 22:21, Almar Klein <a.klein at science-applied.nl> wrote:
>
>> A note to Ubuntu 12.10 users:
>> I just found out (thanks to Thomas Kluyver) that on Ubuntu 12.10, IEP has
>> some nasty problem between Qt and X, related to scrollbar overlays.
>> I will fix this asap and come with a 3.1.1 release.
>>
>> - Almar
>>
>>
>>
>> On 20 December 2012 17:16, Almar Klein <a.klein at science-applied.nl>wrote:
>>
>>> Dear all,
>>>
>>> We're pleased to announce version 3.1 of the Interactive Editor for
>>> Python.
>>>
>>> IEP is a cross-platform Python IDE focused on interactivity and
>>> introspection, which makes it very suitable for scientific computing. Its
>>> practical design is aimed at simplicity and efficiency. IEP is written
>>> in Python 3 and Qt.
>>>
>>> Binaries are available for Windows, Linux, and Mac.
>>>
>>> A selection of new features are:
>>>
>>>    - a new powerful menu for the shells widget
>>>    - multiple lines can be copy-pasted (and executed) in the shell
>>>    - run selection (F9) runs the selected statement if on a single line
>>>    - multiple comments/docstring reshaping
>>>    - goto-line
>>>    - the shell widget can now be moved on Mac
>>>    - a wizard to help new users on their way
>>>
>>>
>>> Website: http://code.google.com/p/iep/
>>> Discussion group: http://groups.google.com/group/iep_
>>> Release notes: h*ttp://code.google.com/p/iep/wiki/Release
>>> #version_3.1_(release_candidate_out_now)*
>>>
>>> Regards,
>>>   Rob, Ludo & Almar
>>>
>>>
>>
>>
>> --
>> Almar Klein, PhD
>> Science Applied
>> phone: +31 6 19268652
>> e-mail: a.klein at science-applied.nl
>>
>
>
>
> --
> Almar Klein, PhD
> Science Applied
> phone: +31 6 19268652
> e-mail: a.klein at science-applied.nl
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Fl?vio Code?o Coelho
================
+55(21) 3799-5567
Professor
Escola de Matem?tica Aplicada
Funda??o Get?lio Vargas
Praia de Botafogo, 190 sala 312
Rio de Janeiro - RJ
22250-900
Brasil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121221/bec4aceb/attachment.html>

From a.klein at science-applied.nl  Fri Dec 21 07:56:50 2012
From: a.klein at science-applied.nl (Almar Klein)
Date: Fri, 21 Dec 2012 13:56:50 +0100
Subject: [SciPy-User] Introducing Pyzo - scientific computing in Python 3
Message-ID: <CAB_T6=nq0YSWGVKHpFfkWGX7UFeZEre8o57s3At4M9=c+ppfww@mail.gmail.com>

Hi all,

It is with great pleasure that we announce the first version of the Pyzo
distro: a Python distribution for scientific computing, based on Python 3.

For more information, and to give it a try, visit http://www.pyzo.org.

Pyzo is available for Windows and Linux, and we expect a version for Mac
soon. The distribution is portable, thus providing a way to install a
scientific Python stack on computers without the need for admin rights.

Naturally, Pyzo distro is complient with the scipy-stack, and also comes
with packages such as scikit-image and scikit-learn.

With Pyzo we want to make scientific computing in Python easier accessible.
We try to make the user experience coherent. Pyzo uses  IEP as the default
front-end IDE, and IPython (with notebook) is also available.

We especially hope to make it easier for newcomers (such as Matlab
converts) to join our awesome community. We realize that for a large part
of the current community Python 2 is still the most common version.
Nevertheless, we hope that you will give it a try and let us know what you
think.

Happy holidays,

- Almar, Rob and Ludo

PS: sorry for the cross-post, but I think that at least the introduction
would be of interest to both scipy-user as numfocus.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121221/6ba49fb2/attachment.html>

From a.klein at science-applied.nl  Fri Dec 21 08:00:36 2012
From: a.klein at science-applied.nl (Almar Klein)
Date: Fri, 21 Dec 2012 14:00:36 +0100
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <CAAbyzbRCngsYjzbCv02-6oG+CQVMy=6wf06G17HeMSFy9fTsow@mail.gmail.com>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
	<CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
	<CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>
	<CAAbyzbRCngsYjzbCv02-6oG+CQVMy=6wf06G17HeMSFy9fTsow@mail.gmail.com>
Message-ID: <CAB_T6=kRHtZSaAhHMvNaO-e_hyDQOJbZOSd2sDNAQLOf9oHVPA@mail.gmail.com>

No, it would not be too much. In fact there already is one:
http://pypi.python.org/pypi/iep/3.1

(but I do realize now that I should update it to 3.1.1, will do that right
now)

- Almar


On 21 December 2012 11:26, Flavio Coelho <fccoelho at gmail.com> wrote:

> Would it be too much to ask for a IEP package in PyPI? it would greatly
> facilitate installation... Spyder has one.
>
> cheers,
>
> Fl?vio
>
> On Fri, Dec 21, 2012 at 12:35 AM, Almar Klein <a.klein at science-applied.nl>wrote:
>
>> Binaries for 3.1.1 are now available (for Linux).
>> - Almar
>>
>>
>> On 20 December 2012 22:21, Almar Klein <a.klein at science-applied.nl>wrote:
>>
>>> A note to Ubuntu 12.10 users:
>>> I just found out (thanks to Thomas Kluyver) that on Ubuntu 12.10, IEP
>>> has some nasty problem between Qt and X, related to scrollbar overlays.
>>> I will fix this asap and come with a 3.1.1 release.
>>>
>>> - Almar
>>>
>>>
>>>
>>> On 20 December 2012 17:16, Almar Klein <a.klein at science-applied.nl>wrote:
>>>
>>>> Dear all,
>>>>
>>>> We're pleased to announce version 3.1 of the Interactive Editor for
>>>> Python.
>>>>
>>>> IEP is a cross-platform Python IDE focused on interactivity and
>>>> introspection, which makes it very suitable for scientific computing. Its
>>>> practical design is aimed at simplicity and efficiency. IEP is written
>>>> in Python 3 and Qt.
>>>>
>>>> Binaries are available for Windows, Linux, and Mac.
>>>>
>>>> A selection of new features are:
>>>>
>>>>    - a new powerful menu for the shells widget
>>>>    - multiple lines can be copy-pasted (and executed) in the shell
>>>>    - run selection (F9) runs the selected statement if on a single line
>>>>    - multiple comments/docstring reshaping
>>>>    - goto-line
>>>>    - the shell widget can now be moved on Mac
>>>>    - a wizard to help new users on their way
>>>>
>>>>
>>>> Website: http://code.google.com/p/iep/
>>>> Discussion group: http://groups.google.com/group/iep_
>>>> Release notes: h*ttp://code.google.com/p/iep/wiki/Release
>>>> #version_3.1_(release_candidate_out_now)*
>>>>
>>>> Regards,
>>>>   Rob, Ludo & Almar
>>>>
>>>>
>>>
>>>
>>> --
>>> Almar Klein, PhD
>>> Science Applied
>>> phone: +31 6 19268652
>>> e-mail: a.klein at science-applied.nl
>>>
>>
>>
>>
>> --
>> Almar Klein, PhD
>> Science Applied
>> phone: +31 6 19268652
>> e-mail: a.klein at science-applied.nl
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
>
> --
> Fl?vio Code?o Coelho
> ================
> +55(21) 3799-5567
> Professor
> Escola de Matem?tica Aplicada
> Funda??o Get?lio Vargas
> Praia de Botafogo, 190 sala 312
> Rio de Janeiro - RJ
> 22250-900
> Brasil
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Almar Klein, PhD
Science Applied
phone: +31 6 19268652
e-mail: a.klein at science-applied.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121221/304443b2/attachment.html>

From andrewhdmundell at gmail.com  Fri Dec 21 09:21:43 2012
From: andrewhdmundell at gmail.com (amundell)
Date: Fri, 21 Dec 2012 06:21:43 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] One Sample Sign Test for a Median in
	SciPy?
Message-ID: <34823879.post@talk.nabble.com>


Is there a One Sample Sign Test in SciPy?, i.e. non parametric test for
testing the null hypothesis that the median of a population, from which the
tested sample is drawn, is equal to a hypothesised value.
This type of test is included in the BSDA library in R (SIGN.test:
http://rss.acs.unt.edu/Rdoc/library/BSDA/html/sign.test.html) and I was
looking to see if I could make some comparisons with SciPy. . Thanks Andrew.

-- 
View this message in context: http://old.nabble.com/One-Sample-Sign-Test-for-a-Median-in-SciPy--tp34823879p34823879.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From fccoelho at gmail.com  Fri Dec 21 13:11:34 2012
From: fccoelho at gmail.com (Flavio Coelho)
Date: Fri, 21 Dec 2012 16:11:34 -0200
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <CAB_T6=kRHtZSaAhHMvNaO-e_hyDQOJbZOSd2sDNAQLOf9oHVPA@mail.gmail.com>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
	<CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
	<CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>
	<CAAbyzbRCngsYjzbCv02-6oG+CQVMy=6wf06G17HeMSFy9fTsow@mail.gmail.com>
	<CAB_T6=kRHtZSaAhHMvNaO-e_hyDQOJbZOSd2sDNAQLOf9oHVPA@mail.gmail.com>
Message-ID: <CAAbyzbSygpXC=_EnNZk1eSwSnOD_CdZtr-iB79jZgsL6av+UeA@mail.gmail.com>

yes, but the installation is broken (at least for me: ubuntu 12.10):

$ sudo pip install iep
[sudo] password for fccoelho:
Downloading/unpacking iep
  Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
  Running setup.py egg_info for package iep
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
    IOError: [Errno 2] No such file or directory:
'/home/fccoelho/build/iep/setup.py'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in <module>

IOError: [Errno 2] No such file or directory:
'/home/fccoelho/build/iep/setup.py'

----------------------------------------
Command python setup.py egg_info failed with error code 1 in
/home/fccoelho/build/iep
Storing complete log in /home/fccoelho/.pip/pip.log


On Fri, Dec 21, 2012 at 11:00 AM, Almar Klein <a.klein at science-applied.nl>wrote:

> No, it would not be too much. In fact there already is one:
> http://pypi.python.org/pypi/iep/3.1
>
> (but I do realize now that I should update it to 3.1.1, will do that right
> now)
>
> - Almar
>
>
> On 21 December 2012 11:26, Flavio Coelho <fccoelho at gmail.com> wrote:
>
>> Would it be too much to ask for a IEP package in PyPI? it would greatly
>> facilitate installation... Spyder has one.
>>
>> cheers,
>>
>> Fl?vio
>>
>> On Fri, Dec 21, 2012 at 12:35 AM, Almar Klein <a.klein at science-applied.nl
>> > wrote:
>>
>>> Binaries for 3.1.1 are now available (for Linux).
>>> - Almar
>>>
>>>
>>> On 20 December 2012 22:21, Almar Klein <a.klein at science-applied.nl>wrote:
>>>
>>>> A note to Ubuntu 12.10 users:
>>>> I just found out (thanks to Thomas Kluyver) that on Ubuntu 12.10, IEP
>>>> has some nasty problem between Qt and X, related to scrollbar overlays.
>>>> I will fix this asap and come with a 3.1.1 release.
>>>>
>>>> - Almar
>>>>
>>>>
>>>>
>>>> On 20 December 2012 17:16, Almar Klein <a.klein at science-applied.nl>wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> We're pleased to announce version 3.1 of the Interactive Editor for
>>>>> Python.
>>>>>
>>>>> IEP is a cross-platform Python IDE focused on interactivity and
>>>>> introspection, which makes it very suitable for scientific computing. Its
>>>>> practical design is aimed at simplicity and efficiency. IEP is
>>>>> written in Python 3 and Qt.
>>>>>
>>>>> Binaries are available for Windows, Linux, and Mac.
>>>>>
>>>>> A selection of new features are:
>>>>>
>>>>>    - a new powerful menu for the shells widget
>>>>>    - multiple lines can be copy-pasted (and executed) in the shell
>>>>>    - run selection (F9) runs the selected statement if on a single
>>>>>    line
>>>>>    - multiple comments/docstring reshaping
>>>>>    - goto-line
>>>>>    - the shell widget can now be moved on Mac
>>>>>    - a wizard to help new users on their way
>>>>>
>>>>>
>>>>> Website: http://code.google.com/p/iep/
>>>>> Discussion group: http://groups.google.com/group/iep_
>>>>> Release notes: h*ttp://code.google.com/p/iep/wiki/Release
>>>>> #version_3.1_(release_candidate_out_now)*
>>>>>
>>>>> Regards,
>>>>>   Rob, Ludo & Almar
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Almar Klein, PhD
>>>> Science Applied
>>>> phone: +31 6 19268652
>>>> e-mail: a.klein at science-applied.nl
>>>>
>>>
>>>
>>>
>>> --
>>> Almar Klein, PhD
>>> Science Applied
>>> phone: +31 6 19268652
>>> e-mail: a.klein at science-applied.nl
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>>
>> --
>> Fl?vio Code?o Coelho
>> ================
>> +55(21) 3799-5567
>> Professor
>> Escola de Matem?tica Aplicada
>> Funda??o Get?lio Vargas
>> Praia de Botafogo, 190 sala 312
>> Rio de Janeiro - RJ
>> 22250-900
>> Brasil
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
>
> --
> Almar Klein, PhD
> Science Applied
> phone: +31 6 19268652
> e-mail: a.klein at science-applied.nl
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Fl?vio Code?o Coelho
================
+55(21) 3799-5567
Professor
Escola de Matem?tica Aplicada
Funda??o Get?lio Vargas
Praia de Botafogo, 190 sala 312
Rio de Janeiro - RJ
22250-900
Brasil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121221/7369c468/attachment.html>

From osman at fuse.net  Fri Dec 21 13:27:48 2012
From: osman at fuse.net (osman buyukisik)
Date: Fri, 21 Dec 2012 13:27:48 -0500
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <CAAbyzbSygpXC=_EnNZk1eSwSnOD_CdZtr-iB79jZgsL6av+UeA@mail.gmail.com>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
	<CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
	<CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>
	<CAAbyzbRCngsYjzbCv02-6oG+CQVMy=6wf06G17HeMSFy9fTsow@mail.gmail.com>
	<CAB_T6=kRHtZSaAhHMvNaO-e_hyDQOJbZOSd2sDNAQLOf9oHVPA@mail.gmail.com>
	<CAAbyzbSygpXC=_EnNZk1eSwSnOD_CdZtr-iB79jZgsL6av+UeA@mail.gmail.com>
Message-ID: <50D4AA24.2040404@fuse.net>

On 12/21/2012 01:11 PM, Flavio Coelho wrote:
> yes, but the installation is broken (at least for me: ubuntu 12.10):
>
> $ sudo pip install iep
> [sudo] password for fccoelho:
> Downloading/unpacking iep
>   Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
>   Running setup.py egg_info for package iep
>     Traceback (most recent call last):
>       File "<string>", line 14, in <module>
>     IOError: [Errno 2] No such file or directory: 
> '/home/fccoelho/build/iep/setup.py'
>     Complete output from command python setup.py egg_info:
>     Traceback (most recent call last):
>
>   File "<string>", line 14, in <module>
>
> IOError: [Errno 2] No such file or directory: 
> '/home/fccoelho/build/iep/setup.py'
>
> ----------------------------------------
> Command python setup.py egg_info failed with error code 1 in 
> /home/fccoelho/build/iep
> Storing complete log in /home/fccoelho/.pip/pip.log
>
>
With ubuntu 11.10 I have the same error:
  sudo pip install iep
[sudo] password for osman:
Downloading/unpacking iep
   Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
   Running setup.py egg_info for package iep
     Traceback (most recent call last):
       File "<string>", line 14, in <module>
     IOError: [Errno 2] No such file or directory: 
'/home/osman/build/iep/setup.py'
     Complete output from command python setup.py egg_info:
     Traceback (most recent call last):

   File "<string>", line 14, in <module>

IOError: [Errno 2] No such file or directory: 
'/home/osman/build/iep/setup.py'

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in /home/osman/.pip/pip.log
osman at osman-AV019AA-ABA-p6228p:~$



From josef.pktd at gmail.com  Fri Dec 21 13:49:17 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Fri, 21 Dec 2012 13:49:17 -0500
Subject: [SciPy-User] [SciPy-user] One Sample Sign Test for a Median in
	SciPy?
In-Reply-To: <34823879.post@talk.nabble.com>
References: <34823879.post@talk.nabble.com>
Message-ID: <CAMMTP+A6jbYDQ3JzuGdjhANOmRs-0M_bOsVMvngewqQgKMKY2Q@mail.gmail.com>

On Fri, Dec 21, 2012 at 9:21 AM, amundell <andrewhdmundell at gmail.com> wrote:
>
> Is there a One Sample Sign Test in SciPy?, i.e. non parametric test for
> testing the null hypothesis that the median of a population, from which the
> tested sample is drawn, is equal to a hypothesised value.
> This type of test is included in the BSDA library in R (SIGN.test:
> http://rss.acs.unt.edu/Rdoc/library/BSDA/html/sign.test.html) and I was
> looking to see if I could make some comparisons with SciPy. . Thanks Andrew.

I thought it is somewhere, but I don't find any

for the example in the R help, maybe something like this

>>> xs = (7.8, 6.6, 6.5, 7.4, 7.3, 7., 6.4, 7.1, 6.7, 7.6, 6.8)
>>> xs = np.asarray(xs)
>>> med=6.5; stats.binom_test([(xs<med).sum(),(xs>med).sum()], p=0.5)
0.021484374999999997
same p-value as in R

A one-sample version of wilcoxon signed rank test also seems to be missing.

Josef

>
> --
> View this message in context: http://old.nabble.com/One-Sample-Sign-Test-for-a-Median-in-SciPy--tp34823879p34823879.html
> Sent from the Scipy-User mailing list archive at Nabble.com.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From andrewhdmundell at gmail.com  Fri Dec 21 14:37:01 2012
From: andrewhdmundell at gmail.com (amundell)
Date: Fri, 21 Dec 2012 11:37:01 -0800 (PST)
Subject: [SciPy-User] [SciPy-user] One Sample Sign Test for a Median in
	SciPy?
In-Reply-To: <CAMMTP+A6jbYDQ3JzuGdjhANOmRs-0M_bOsVMvngewqQgKMKY2Q@mail.gmail.com>
References: <34823879.post@talk.nabble.com>
	<CAMMTP+A6jbYDQ3JzuGdjhANOmRs-0M_bOsVMvngewqQgKMKY2Q@mail.gmail.com>
Message-ID: <34825074.post@talk.nabble.com>


Thanks Josef,

I tried your suggestion with the data set I was running and getting the same
result (p-value) as R and with a third party software. I will let you know
if I  encounter any furthers issues. 

I will be testing the sample wilcoxon signed rank test at some future date,
so thanks for the letting me know that it's not present.
Cheers, Andrew


josef.pktd wrote:
> 
> On Fri, Dec 21, 2012 at 9:21 AM, amundell <andrewhdmundell at gmail.com>
> wrote:
>>
>> Is there a One Sample Sign Test in SciPy?, i.e. non parametric test for
>> testing the null hypothesis that the median of a population, from which
>> the
>> tested sample is drawn, is equal to a hypothesised value.
>> This type of test is included in the BSDA library in R (SIGN.test:
>> http://rss.acs.unt.edu/Rdoc/library/BSDA/html/sign.test.html) and I was
>> looking to see if I could make some comparisons with SciPy. . Thanks
>> Andrew.
> 
> I thought it is somewhere, but I don't find any
> 
> for the example in the R help, maybe something like this
> 
>>>> xs = (7.8, 6.6, 6.5, 7.4, 7.3, 7., 6.4, 7.1, 6.7, 7.6, 6.8)
>>>> xs = np.asarray(xs)
>>>> med=6.5; stats.binom_test([(xs<med).sum(),(xs>med).sum()], p=0.5)
> 0.021484374999999997
> same p-value as in R
> 
> A one-sample version of wilcoxon signed rank test also seems to be
> missing.
> 
> Josef
> 
>>
>> --
>> View this message in context:
>> http://old.nabble.com/One-Sample-Sign-Test-for-a-Median-in-SciPy--tp34823879p34823879.html
>> Sent from the Scipy-User mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 
> 

-- 
View this message in context: http://old.nabble.com/One-Sample-Sign-Test-for-a-Median-in-SciPy--tp34823879p34825074.html
Sent from the Scipy-User mailing list archive at Nabble.com.



From ralf.gommers at gmail.com  Fri Dec 21 16:33:57 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 21 Dec 2012 22:33:57 +0100
Subject: [SciPy-User] spline interpolators silently failing on NaNs
In-Reply-To: <katuh0$4uf$1@ger.gmane.org>
References: <katuh0$4uf$1@ger.gmane.org>
Message-ID: <CABL7CQg2=-7JGZbzZChR-iGMOyq=zjzXJtjERPhYZc0jnER35Q@mail.gmail.com>

On Thu, Dec 20, 2012 at 3:52 AM, Michael Aye <michael.aye at ucla.edu> wrote:

> Hi,
>
> I was just bitten by something nasty.
> I am fitting some data from pandas, and as pandas deals so
> fantastically with NaNs, I forgot to check myself before handing the
> data over to InterpolatedUnivariateSpline.
> To my surprise and after 2 hours of 'practising' all kinds of
> interpolations, I realised my overlook.
> To my next surprise the creation of the interpolator function did not
> complain a single bit (pun not intented). Just the application of the
> interpolator to the new x-data returned only NaNs.
>
> It is also notable, that griddata had no such problem, but I cannot use
> it because I need extrapolation as well.
>
> Should / Could the spline interpolators maybe check for NaNs and
> complain at their existence?
>

Yes, a check in functions in fitpack2.py for NaNs as well as duplicate
points and monotonicity where applicable would be quite useful.

If anyone would like to have a go at it, that would be great. It wouldn't
be too hard to implement - would be a nice problem to get started with
contributing to scipy perhaps.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121221/61245c72/attachment.html>

From rainexpected at theo.to  Fri Dec 21 16:49:23 2012
From: rainexpected at theo.to (Ted To)
Date: Fri, 21 Dec 2012 16:49:23 -0500
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <50D4AA24.2040404@fuse.net>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
	<CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
	<CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>
	<CAAbyzbRCngsYjzbCv02-6oG+CQVMy=6wf06G17HeMSFy9fTsow@mail.gmail.com>
	<CAB_T6=kRHtZSaAhHMvNaO-e_hyDQOJbZOSd2sDNAQLOf9oHVPA@mail.gmail.com>
	<CAAbyzbSygpXC=_EnNZk1eSwSnOD_CdZtr-iB79jZgsL6av+UeA@mail.gmail.com>
	<50D4AA24.2040404@fuse.net>
Message-ID: <50D4D963.1000401@theo.to>

Same error with a Wheezy box.

On 12/21/2012 01:27 PM, osman buyukisik wrote:
> On 12/21/2012 01:11 PM, Flavio Coelho wrote:
>> yes, but the installation is broken (at least for me: ubuntu 12.10):
>>
>> $ sudo pip install iep
>> [sudo] password for fccoelho:
>> Downloading/unpacking iep
>>   Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
>>   Running setup.py egg_info for package iep
>>     Traceback (most recent call last):
>>       File "<string>", line 14, in <module>
>>     IOError: [Errno 2] No such file or directory:
>> '/home/fccoelho/build/iep/setup.py'
>>     Complete output from command python setup.py egg_info:
>>     Traceback (most recent call last):
>>
>>   File "<string>", line 14, in <module>
>>
>> IOError: [Errno 2] No such file or directory:
>> '/home/fccoelho/build/iep/setup.py'
>>
>> ----------------------------------------
>> Command python setup.py egg_info failed with error code 1 in
>> /home/fccoelho/build/iep
>> Storing complete log in /home/fccoelho/.pip/pip.log
>>
>>
> With ubuntu 11.10 I have the same error:
>  sudo pip install iep
> [sudo] password for osman:
> Downloading/unpacking iep
>   Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
>   Running setup.py egg_info for package iep
>     Traceback (most recent call last):
>       File "<string>", line 14, in <module>
>     IOError: [Errno 2] No such file or directory:
> '/home/osman/build/iep/setup.py'
>     Complete output from command python setup.py egg_info:
>     Traceback (most recent call last):
> 
>   File "<string>", line 14, in <module>
> 
> IOError: [Errno 2] No such file or directory:
> '/home/osman/build/iep/setup.py'
> 
> ----------------------------------------
> Command python setup.py egg_info failed with error code 1
> Storing complete log in /home/osman/.pip/pip.log
> osman at osman-AV019AA-ABA-p6228p:~$
> 
> 


From pav at iki.fi  Fri Dec 21 17:02:09 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 22 Dec 2012 00:02:09 +0200
Subject: [SciPy-User] spline interpolators silently failing on NaNs
In-Reply-To: <CABL7CQg2=-7JGZbzZChR-iGMOyq=zjzXJtjERPhYZc0jnER35Q@mail.gmail.com>
References: <katuh0$4uf$1@ger.gmane.org>
	<CABL7CQg2=-7JGZbzZChR-iGMOyq=zjzXJtjERPhYZc0jnER35Q@mail.gmail.com>
Message-ID: <kb2m8u$7ti$1@ger.gmane.org>

21.12.2012 23:33, Ralf Gommers kirjoitti:
[clip]
> Yes, a check in functions in fitpack2.py for NaNs as well as duplicate
> points and monotonicity where applicable would be quite useful.

The NaN check could be done on the output of the spline fitting routine
--- then, you can just check one of the values (since NaNs always
propagate to fill the whole spline fit), rather than needing to check
all of the input values, which is slower.

-- 
Pauli Virtanen



From a.klein at science-applied.nl  Fri Dec 21 17:09:24 2012
From: a.klein at science-applied.nl (Almar Klein)
Date: Fri, 21 Dec 2012 23:09:24 +0100
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <50D4AA24.2040404@fuse.net>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
	<CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
	<CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>
	<CAAbyzbRCngsYjzbCv02-6oG+CQVMy=6wf06G17HeMSFy9fTsow@mail.gmail.com>
	<CAB_T6=kRHtZSaAhHMvNaO-e_hyDQOJbZOSd2sDNAQLOf9oHVPA@mail.gmail.com>
	<CAAbyzbSygpXC=_EnNZk1eSwSnOD_CdZtr-iB79jZgsL6av+UeA@mail.gmail.com>
	<50D4AA24.2040404@fuse.net>
Message-ID: <CAB_T6=k-uNJEPKbsdWJtfeFN4ASjR62YRusr1P_48vtd==EXXg@mail.gmail.com>

Mmm. Installation works using the code from the code repository, but I
haven't properly tested on the code that distutils uploads to pypi. I more
or less assumed that would work as well. Sorry for the inconvenience.

My suggestion (besides using the binarie) is to install from the cloned
repository, see also https://code.google.com/p/iep/issues/detail?id=139

Thanks for the report!

- Almar


On 21 December 2012 19:27, osman buyukisik <osman at fuse.net> wrote:

> On 12/21/2012 01:11 PM, Flavio Coelho wrote:
> > yes, but the installation is broken (at least for me: ubuntu 12.10):
> >
> > $ sudo pip install iep
> > [sudo] password for fccoelho:
> > Downloading/unpacking iep
> >   Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
> >   Running setup.py egg_info for package iep
> >     Traceback (most recent call last):
> >       File "<string>", line 14, in <module>
> >     IOError: [Errno 2] No such file or directory:
> > '/home/fccoelho/build/iep/setup.py'
> >     Complete output from command python setup.py egg_info:
> >     Traceback (most recent call last):
> >
> >   File "<string>", line 14, in <module>
> >
> > IOError: [Errno 2] No such file or directory:
> > '/home/fccoelho/build/iep/setup.py'
> >
> > ----------------------------------------
> > Command python setup.py egg_info failed with error code 1 in
> > /home/fccoelho/build/iep
> > Storing complete log in /home/fccoelho/.pip/pip.log
> >
> >
> With ubuntu 11.10 I have the same error:
>   sudo pip install iep
> [sudo] password for osman:
> Downloading/unpacking iep
>    Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
>    Running setup.py egg_info for package iep
>      Traceback (most recent call last):
>        File "<string>", line 14, in <module>
>      IOError: [Errno 2] No such file or directory:
> '/home/osman/build/iep/setup.py'
>      Complete output from command python setup.py egg_info:
>      Traceback (most recent call last):
>
>    File "<string>", line 14, in <module>
>
> IOError: [Errno 2] No such file or directory:
> '/home/osman/build/iep/setup.py'
>
> ----------------------------------------
> Command python setup.py egg_info failed with error code 1
> Storing complete log in /home/osman/.pip/pip.log
> osman at osman-AV019AA-ABA-p6228p:~$
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Almar Klein, PhD
Science Applied
phone: +31 6 19268652
e-mail: a.klein at science-applied.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121221/79a029fb/attachment.html>

From a.klein at science-applied.nl  Fri Dec 21 18:09:15 2012
From: a.klein at science-applied.nl (Almar Klein)
Date: Sat, 22 Dec 2012 00:09:15 +0100
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <50D4D963.1000401@theo.to>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
	<CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
	<CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>
	<CAAbyzbRCngsYjzbCv02-6oG+CQVMy=6wf06G17HeMSFy9fTsow@mail.gmail.com>
	<CAB_T6=kRHtZSaAhHMvNaO-e_hyDQOJbZOSd2sDNAQLOf9oHVPA@mail.gmail.com>
	<CAAbyzbSygpXC=_EnNZk1eSwSnOD_CdZtr-iB79jZgsL6av+UeA@mail.gmail.com>
	<50D4AA24.2040404@fuse.net> <50D4D963.1000401@theo.to>
Message-ID: <CAB_T6=mGf2RN_PZh4mSqjw4bMYx6D2VXUvOfsOsBVUmuyJ-HmA@mail.gmail.com>

The archive uploaded to pypi works just fine when extracting and using
"setup.py install"

The problem is that pip tries to pull in the source code from IEPs download
page, and downloads the archive that contains the binaries, which, of
course, does not have a setup.py.

I removed the download_link attribute, but pip's behavior stays the same.
Any ideas why that is?

- Almar


On 21 December 2012 22:49, Ted To <rainexpected at theo.to> wrote:

> Same error with a Wheezy box.
>
> On 12/21/2012 01:27 PM, osman buyukisik wrote:
> > On 12/21/2012 01:11 PM, Flavio Coelho wrote:
> >> yes, but the installation is broken (at least for me: ubuntu 12.10):
> >>
> >> $ sudo pip install iep
> >> [sudo] password for fccoelho:
> >> Downloading/unpacking iep
> >>   Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
> >>   Running setup.py egg_info for package iep
> >>     Traceback (most recent call last):
> >>       File "<string>", line 14, in <module>
> >>     IOError: [Errno 2] No such file or directory:
> >> '/home/fccoelho/build/iep/setup.py'
> >>     Complete output from command python setup.py egg_info:
> >>     Traceback (most recent call last):
> >>
> >>   File "<string>", line 14, in <module>
> >>
> >> IOError: [Errno 2] No such file or directory:
> >> '/home/fccoelho/build/iep/setup.py'
> >>
> >> ----------------------------------------
> >> Command python setup.py egg_info failed with error code 1 in
> >> /home/fccoelho/build/iep
> >> Storing complete log in /home/fccoelho/.pip/pip.log
> >>
> >>
> > With ubuntu 11.10 I have the same error:
> >  sudo pip install iep
> > [sudo] password for osman:
> > Downloading/unpacking iep
> >   Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
> >   Running setup.py egg_info for package iep
> >     Traceback (most recent call last):
> >       File "<string>", line 14, in <module>
> >     IOError: [Errno 2] No such file or directory:
> > '/home/osman/build/iep/setup.py'
> >     Complete output from command python setup.py egg_info:
> >     Traceback (most recent call last):
> >
> >   File "<string>", line 14, in <module>
> >
> > IOError: [Errno 2] No such file or directory:
> > '/home/osman/build/iep/setup.py'
> >
> > ----------------------------------------
> > Command python setup.py egg_info failed with error code 1
> > Storing complete log in /home/osman/.pip/pip.log
> > osman at osman-AV019AA-ABA-p6228p:~$
> >
> >
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Almar Klein, PhD
Science Applied
phone: +31 6 19268652
e-mail: a.klein at science-applied.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121222/3bec375c/attachment.html>

From javgarber at gmail.com  Fri Dec 21 20:06:39 2012
From: javgarber at gmail.com (Javier)
Date: Sat, 22 Dec 2012 01:06:39 +0000 (UTC)
Subject: [SciPy-User] Scipy installation error
Message-ID: <loom.20121222T015833-446@post.gmane.org>

Hi all!

I just bought a new computer and I haven't been able to get scipy working, I've
followed this manual: http://www.scipy.org/Installing_SciPy/Linux and I got the
error below so I tried with pip install and I also got the same error. The
installations don't return any errors, but when I try to use it:

from scipy.stats import *
  File "/usr/lib/python2.7/dist-packages/scipy/stats/__init__.py", line 321, in
<module>
    from stats import *
  File "/usr/lib/python2.7/dist-packages/scipy/stats/stats.py", line 194, in
<module>
    import scipy.linalg as linalg
  File "/usr/lib/python2.7/dist-packages/scipy/linalg/__init__.py", line 133, in
<module>
    from basic import *
  File "/usr/lib/python2.7/dist-packages/scipy/linalg/basic.py", line 12, in
<module>
    from lapack import get_lapack_funcs
  File "/usr/lib/python2.7/dist-packages/scipy/linalg/lapack.py", line 14, in
<module>
    from scipy.linalg import flapack
ImportError: /usr/lib/python2.7/dist-packages/scipy/linalg/flapack.so: undefined
symbol: sgges_

Hope somebody can help me, it's driving me crazy... Thanks!



From takowl at gmail.com  Sat Dec 22 06:57:16 2012
From: takowl at gmail.com (Thomas Kluyver)
Date: Sat, 22 Dec 2012 11:57:16 +0000
Subject: [SciPy-User] Scipy installation error
In-Reply-To: <loom.20121222T015833-446@post.gmane.org>
References: <loom.20121222T015833-446@post.gmane.org>
Message-ID: <CAOvn4qhnZ3d=pxsi7XLk2xqRWDJ0mkCY17=7LEeo8eazhA+CuA@mail.gmail.com>

Hi Javier,

On 22 December 2012 01:06, Javier <javgarber at gmail.com> wrote:

> I just bought a new computer and I haven't been able to get scipy working,
> I've
> followed this manual: http://www.scipy.org/Installing_SciPy/Linux and I
> got the
> error below so I tried with pip install and I also got the same error. The
> installations don't return any errors, but when I try to use it:


Which distribution do you use? In most cases, it's easiest to install numpy
& scipy from the repositories for your distribution. E.g. on Ubuntu, "sudo
apt-get install python-scipy". It's probably only worth installing from
source if you need something that was added in the newest version of Scipy.

Best wishes,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121222/df760233/attachment.html>

From pawel.kw at gmail.com  Sat Dec 22 07:51:02 2012
From: pawel.kw at gmail.com (pawel.kw at gmail.com)
Date: Sat, 22 Dec 2012 13:51:02 +0100
Subject: [SciPy-User] spline interpolators silently failing on NaNs
In-Reply-To: <kb2m8u$7ti$1@ger.gmane.org>
Message-ID: <50d5acb6.236bb40a.031f.643d@mx.google.com>

Hi,

I have also encountered this problem on several occasions-every time I forget about this and it takes some time before I realise what's wrong. I could try to implement a NaN check, resulting with an error message. Can someone point me to some instructions on how to contribute properly? I guess there's an official way of submitting code - I've never done this. I have some experience with svn and git (I use it regularly for some private projects), I'd be happy to contribute.

Cheers,

Pawel 


On Dec 21, 2012 23:02, Pauli Virtanen <pav at iki.fi> wrote: 
21.12.2012 23:33, Ralf Gommers kirjoitti:

[clip]

> Yes, a check in functions in fitpack2.py for NaNs as well as duplicate

> points and monotonicity where applicable would be quite useful.



The NaN check could be done on the output of the spline fitting routine

--- then, you can just check one of the values (since NaNs always

propagate to fill the whole spline fit), rather than needing to check

all of the input values, which is slower.



-- 

Pauli Virtanen



_______________________________________________

SciPy-User mailing list

SciPy-User at scipy.org

http://mail.scipy.org/mailman/listinfo/scipy-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121222/52344a59/attachment.html>

From pav at iki.fi  Sat Dec 22 08:21:39 2012
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 22 Dec 2012 15:21:39 +0200
Subject: [SciPy-User] spline interpolators silently failing on NaNs
In-Reply-To: <50d5acb6.236bb40a.031f.643d@mx.google.com>
References: <kb2m8u$7ti$1@ger.gmane.org>
	<50d5acb6.236bb40a.031f.643d@mx.google.com>
Message-ID: <kb4c51$qkn$1@ger.gmane.org>

22.12.2012 14:51, pawel.kw at gmail.com kirjoitti:
> I have also encountered this problem on several occasions-every time I
> forget about this and it takes some time before I realise what's wrong.
> I could try to implement a NaN check, resulting with an error message.
> Can someone point me to some instructions on how to contribute properly?
> I guess there's an official way of submitting code - I've never done
> this. I have some experience with svn and git (I use it regularly for
> some private projects), I'd be happy to contribute.

You can take a look at here:

https://github.com/scipy/scipy/blob/master/HACKING.rst.txt

These checks would go around here:

https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack2.py#L147

https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack.py#L469

https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack.py#L898

https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack2.py#L634

https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack2.py#L706

https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack2.py#L766

and so on...

(Yes, there are two separate fitpack wrappers.)

-- 
Pauli Virtanen



From kevin.gullikson.signup at gmail.com  Sat Dec 22 11:22:38 2012
From: kevin.gullikson.signup at gmail.com (Kevin Gullikson)
Date: Sat, 22 Dec 2012 10:22:38 -0600
Subject: [SciPy-User] spline interpolators silently failing on NaNs
In-Reply-To: <50d5acb6.236bb40a.031f.643d@mx.google.com>
References: <kb2m8u$7ti$1@ger.gmane.org>
	<50d5acb6.236bb40a.031f.643d@mx.google.com>
Message-ID: <CAEUX0XcGjeJTtiGmJ8Vn-3v7Q+=sF9AaoXWzciLt_bkRjGQoxg@mail.gmail.com>

We may also want to check that the x array is monotonic. I had an issue
once where one of the x points was repeated and it made the whole spline
return NaNs (also silently)


On Sat, Dec 22, 2012 at 6:51 AM, <pawel.kw at gmail.com> wrote:

> Hi,
>
> I have also encountered this problem on several occasions-every time I
> forget about this and it takes some time before I realise what's wrong. I
> could try to implement a NaN check, resulting with an error message. Can
> someone point me to some instructions on how to contribute properly? I
> guess there's an official way of submitting code - I've never done this. I
> have some experience with svn and git (I use it regularly for some private
> projects), I'd be happy to contribute.
>
> Cheers,
>
> Pawel
>
>
> ------------------------------
> On Dec 21, 2012 23:02, Pauli Virtanen <pav at iki.fi> wrote:
> 21.12.2012 23:33, Ralf Gommers kirjoitti:
> [clip]
> > Yes, a check in functions in fitpack2.py for NaNs as well as duplicate
> > points and monotonicity where applicable would be quite useful.
>
> The NaN check could be done on the output of the spline fitting routine
> --- then, you can just check one of the values (since NaNs always
> propagate to fill the whole spline fit), rather than needing to check
> all of the input values, which is slower.
>
> --
> Pauli Virtanen
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121222/3cc06913/attachment.html>

From pawel.kw at gmail.com  Sat Dec 22 12:28:12 2012
From: pawel.kw at gmail.com (=?ISO-8859-2?Q?Pawe=B3_Kwa=B6niewski?=)
Date: Sat, 22 Dec 2012 18:28:12 +0100
Subject: [SciPy-User] spline interpolators silently failing on NaNs
In-Reply-To: <CAEUX0XcGjeJTtiGmJ8Vn-3v7Q+=sF9AaoXWzciLt_bkRjGQoxg@mail.gmail.com>
References: <kb2m8u$7ti$1@ger.gmane.org>
	<50d5acb6.236bb40a.031f.643d@mx.google.com>
	<CAEUX0XcGjeJTtiGmJ8Vn-3v7Q+=sF9AaoXWzciLt_bkRjGQoxg@mail.gmail.com>
Message-ID: <CAAih6Wn-x5FeChYND=ymt5nQfcYvfHKUUqGfhK19oxveWznbYw@mail.gmail.com>

OK, I'll try to work on that, but I must warn you that it won't be quick -
I'm starting my Christmas travel tomorrow...


Pawe?


2012/12/22 Kevin Gullikson <kevin.gullikson.signup at gmail.com>

> We may also want to check that the x array is monotonic. I had an issue
> once where one of the x points was repeated and it made the whole spline
> return NaNs (also silently)
>
>
> On Sat, Dec 22, 2012 at 6:51 AM, <pawel.kw at gmail.com> wrote:
>
>> Hi,
>>
>> I have also encountered this problem on several occasions-every time I
>> forget about this and it takes some time before I realise what's wrong. I
>> could try to implement a NaN check, resulting with an error message. Can
>> someone point me to some instructions on how to contribute properly? I
>> guess there's an official way of submitting code - I've never done this. I
>> have some experience with svn and git (I use it regularly for some private
>> projects), I'd be happy to contribute.
>>
>> Cheers,
>>
>> Pawel
>>
>>
>> ------------------------------
>> On Dec 21, 2012 23:02, Pauli Virtanen <pav at iki.fi> wrote:
>> 21.12.2012 23:33, Ralf Gommers kirjoitti:
>> [clip]
>> > Yes, a check in functions in fitpack2.py for NaNs as well as duplicate
>> > points and monotonicity where applicable would be quite useful.
>>
>> The NaN check could be done on the output of the spline fitting routine
>> --- then, you can just check one of the values (since NaNs always
>> propagate to fill the whole spline fit), rather than needing to check
>> all of the input values, which is slower.
>>
>> --
>> Pauli Virtanen
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121222/03ed24ed/attachment.html>

From a.klein at science-applied.nl  Sat Dec 22 18:27:07 2012
From: a.klein at science-applied.nl (Almar Klein)
Date: Sun, 23 Dec 2012 00:27:07 +0100
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
In-Reply-To: <CAB_T6=mGf2RN_PZh4mSqjw4bMYx6D2VXUvOfsOsBVUmuyJ-HmA@mail.gmail.com>
References: <CAB_T6=nnNcuaj-Gv9sK3NaN+pc2O9we2EhTN6O+Qsx_ovVyh+Q@mail.gmail.com>
	<CAB_T6==DzuPe3ced=Ogm8mFb5zy+jC6yr94CcadzHvh7S1s7EA@mail.gmail.com>
	<CAB_T6=kh8ip5rncq1=gMMSW1HfJUisudN5ZWQN=-OdGtf0XGOw@mail.gmail.com>
	<CAAbyzbRCngsYjzbCv02-6oG+CQVMy=6wf06G17HeMSFy9fTsow@mail.gmail.com>
	<CAB_T6=kRHtZSaAhHMvNaO-e_hyDQOJbZOSd2sDNAQLOf9oHVPA@mail.gmail.com>
	<CAAbyzbSygpXC=_EnNZk1eSwSnOD_CdZtr-iB79jZgsL6av+UeA@mail.gmail.com>
	<50D4AA24.2040404@fuse.net> <50D4D963.1000401@theo.to>
	<CAB_T6=mGf2RN_PZh4mSqjw4bMYx6D2VXUvOfsOsBVUmuyJ-HmA@mail.gmail.com>
Message-ID: <CAB_T6=mNikXE39pQ4LxmOBKpqO97k3DvwTOJQK_fF9bjU4jonw@mail.gmail.com>

Mmm, the problem was that pip looks in http://pypi.python.org/*simple*/iep/,
which also listed the download page from version 3.1 of iep.
I removed the 3.1 entry of IEP from pypi, and pip seems to work fine now...


On 22 December 2012 00:09, Almar Klein <a.klein at science-applied.nl> wrote:

> The archive uploaded to pypi works just fine when extracting and using
> "setup.py install"
>
> The problem is that pip tries to pull in the source code from IEPs
> download page, and downloads the archive that contains the binaries, which,
> of course, does not have a setup.py.
>
> I removed the download_link attribute, but pip's behavior stays the same.
> Any ideas why that is?
>
> - Almar
>
>
> On 21 December 2012 22:49, Ted To <rainexpected at theo.to> wrote:
>
>> Same error with a Wheezy box.
>>
>> On 12/21/2012 01:27 PM, osman buyukisik wrote:
>> > On 12/21/2012 01:11 PM, Flavio Coelho wrote:
>> >> yes, but the installation is broken (at least for me: ubuntu 12.10):
>> >>
>> >> $ sudo pip install iep
>> >> [sudo] password for fccoelho:
>> >> Downloading/unpacking iep
>> >>   Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
>> >>   Running setup.py egg_info for package iep
>> >>     Traceback (most recent call last):
>> >>       File "<string>", line 14, in <module>
>> >>     IOError: [Errno 2] No such file or directory:
>> >> '/home/fccoelho/build/iep/setup.py'
>> >>     Complete output from command python setup.py egg_info:
>> >>     Traceback (most recent call last):
>> >>
>> >>   File "<string>", line 14, in <module>
>> >>
>> >> IOError: [Errno 2] No such file or directory:
>> >> '/home/fccoelho/build/iep/setup.py'
>> >>
>> >> ----------------------------------------
>> >> Command python setup.py egg_info failed with error code 1 in
>> >> /home/fccoelho/build/iep
>> >> Storing complete log in /home/fccoelho/.pip/pip.log
>> >>
>> >>
>> > With ubuntu 11.10 I have the same error:
>> >  sudo pip install iep
>> > [sudo] password for osman:
>> > Downloading/unpacking iep
>> >   Downloading iep-3.1.1.linux64.tar.gz (17.3Mb): 17.3Mb downloaded
>> >   Running setup.py egg_info for package iep
>> >     Traceback (most recent call last):
>> >       File "<string>", line 14, in <module>
>> >     IOError: [Errno 2] No such file or directory:
>> > '/home/osman/build/iep/setup.py'
>> >     Complete output from command python setup.py egg_info:
>> >     Traceback (most recent call last):
>> >
>> >   File "<string>", line 14, in <module>
>> >
>> > IOError: [Errno 2] No such file or directory:
>> > '/home/osman/build/iep/setup.py'
>> >
>> > ----------------------------------------
>> > Command python setup.py egg_info failed with error code 1
>> > Storing complete log in /home/osman/.pip/pip.log
>> > osman at osman-AV019AA-ABA-p6228p:~$
>> >
>> >
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
>
> --
> Almar Klein, PhD
> Science Applied
> phone: +31 6 19268652
> e-mail: a.klein at science-applied.nl
>



-- 
Almar Klein, PhD
Science Applied
phone: +31 6 19268652
e-mail: a.klein at science-applied.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121223/bea8c51a/attachment.html>

From almar.klein at gmail.com  Thu Dec 20 11:14:05 2012
From: almar.klein at gmail.com (Almar Klein)
Date: Thu, 20 Dec 2012 17:14:05 +0100
Subject: [SciPy-User] ANN: IEP 3.1 - the Interactive Editor for Python
Message-ID: <CAB_T6=khEX7i8ckWuYtF560uQBs2yRyZ2hP8TTiOwEVu_HrR1Q@mail.gmail.com>

Dear all,

We're pleased to announce version 3.1 of the Interactive Editor for Python.

IEP is a cross-platform Python IDE focused on interactivity and
introspection, which makes it very suitable for scientific computing. Its
practical design is aimed at simplicity and efficiency. IEP is written in
Python 3 and Qt.

Binaries are available for Windows, Linux, and Mac.

A selection of new features are:

   - a new powerful menu for the shells widget
   - multiple lines can be copy-pasted (and executed) in the shell
   - run selection (F9) runs the selected statement if on a single line
   - multiple comments/docstring reshaping
   - goto-line
   - the shell widget can now be moved on Mac
   - a wizard to help new users on their way


Website: http://code.google.com/p/iep/
Discussion group: http://groups.google.com/group/iep_
Release notes: h*ttp://code.google.com/p/iep/wiki/Release
#version_3.1_(release_candidate_out_now)*

Regards,
  Rob, Ludo & Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121220/1993c034/attachment.html>

From btemperton at gmail.com  Thu Dec 20 17:44:43 2012
From: btemperton at gmail.com (Ben Temperton)
Date: Thu, 20 Dec 2012 14:44:43 -0800 (PST)
Subject: [SciPy-User] Identifying variables driving divergence in Scipy
Message-ID: <7f28c3ba-46cf-42c5-a9fe-6fa7e6e27c93@googlegroups.com>

Hi there,

I have a matrix of 37 samples and ~1300 variables that are clearly 
clustering into 3 distinct groups using:

import scipy.cluster.hierarchy as sch
import scipy.spatial.distance as dist

d1= dist.pdist(mtx2, 'braycurtis')
Y1 = sch.linkage(d1, method='average', metric='braycurtis')
Z1 = sch.dendrogram(Y1, labels=mtx2.index)

What I would like to do is find out which variables are driving the 
divergence in this clustering, i.e. which ones are greater in group A than 
group B and vice versa. In tools like Primer, this would be done with a 
SIMPER test, and I am hoping someone can point me in the direction of a 
Scipy equivalent as I am trying to keep all my data analysis in ipython 
notebook.

Many thanks,

Ben



<https://lh5.googleusercontent.com/-YdYGFVthrYk/UNOUWRM4OsI/AAAAAAAAAkI/VBJvYtIbPJo/s1600/clust.png>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121220/4fc0fbcd/attachment.html>

From l.j.j.vandervelden at gmail.com  Sat Dec 22 10:35:42 2012
From: l.j.j.vandervelden at gmail.com (Luuk van der Velden)
Date: Sat, 22 Dec 2012 07:35:42 -0800 (PST)
Subject: [SciPy-User] Avoiding inner for loops??
In-Reply-To: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com>
References: <82d5dfed-e68c-44b6-94f9-7f2557ddece4@googlegroups.com>
Message-ID: <af2b0ffb-462a-4ba5-b4e3-697502290aa4@googlegroups.com>

Consider using broadcasting to three 3D parameters matrices (A,B,C), then 
creating a 'ufunc' that takes three parameter at a time (a,b,c) for every 
identical position in the three arrays (A,B,C). So giving the broadcasting 
arrays as input to a ufunction which maps the model function on the three 
paramater arrays.

greets,
Luuk 

On Sunday, August 19, 2012 11:07:59 AM UTC+2, Martin De Kauwe wrote:
>
> Hi, 
>
> I need to avoid (at least) two inner for loops in what I am trying to do 
> otherwise my processing takes forever. What is the best way to transfer 
> what I am doing into a more "numpy way"? Essentially I am trying to call a 
> model again for various different parameter combinations. The example is 
> fictional, by the grid_size would ideally grow > 500 and by doing so the 
> processing speed becomes very slow the way I have set things up.. 
>
> thanks. 
>
> example. 
>
>
> import numpy as np 
>
> def fake_model(data1, data2, p1, p2, p3): 
>     """ complete nonsense """ 
>     return data1 + data2 * p1 * p2 * p3 
>
> data1 = np.random.rand(10) # the size of this arrays varies might be 10 
> might be 15 etc 
> data2 = np.random.rand(10) # the size of this arrays varies might be 10 
> might be 15 etc 
> obs = np.random.rand(10) # the size of this arrays varies might be 10 
> might be 15 etc 
>
> grid_size = 10 # Ideally this would be a large number 
> param1 = np.linspace(5.0, 350, grid_size) 
> param2 = np.linspace(5.0, 550, grid_size) 
> param3 = np.linspace(1E-8, 10.5, grid_size) 
> ss = np.zeros(0) 
>
> for p1 in param1: 
>     for p2 in param2: 
>         for p3 in param3: 
>             ans = fake_model(data1, data2, p1, p2, p3) 
>             
>             ss = np.append(ss, np.sum(obs - ans)**2) 
>             print np.sum(obs - ans)**2
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121222/56d83158/attachment.html>

From toogad at gmail.com  Sun Dec 23 17:25:00 2012
From: toogad at gmail.com (Kai Zhang)
Date: Sun, 23 Dec 2012 16:25:00 -0600
Subject: [SciPy-User] Anyone Interested in Developing Simulation Software
 for Biological Wastewater Treatment?
Message-ID: <CAJJGNdSKWkYPy9Yw9c0+60iUO6oRy=wBYyjOg=+w=tZac90iag@mail.gmail.com>

Hi All,

I am an environmental engineer who focus on biological wastewater treatment
system design, including both municipal and industrial applications.

I am interested in expanding the application of Python and Scipy into
biological wastewater treatment modeling. There are expensive commercial
softwares like BioWin and GPS-X, but they are proprietary, not available
for Linux (not sure about MacOSX), and many engineers in small design firms
don't have access to them.

Does anyone have the same interest? Maybe we can form a team to work
something out.

I have to admit I am fairly new to Python and Scipy. But I am very
confident that this can be done nicely with the great support from this
community and the easy use of Python. And it will be great benefit to both
students and engineers that are interested in wastewater treatment modeling.

Thank you all in advance!

Best Regards,

Kai Zhang

-- 
Kai Zhang

----------------------------------------------------------
Coincidence is God's way of remaining anonymous.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121223/ab169dec/attachment.html>

From josef.pktd at gmail.com  Mon Dec 24 15:26:38 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 24 Dec 2012 15:26:38 -0500
Subject: [SciPy-User] scoreatpercentile vectorized
Message-ID: <CAMMTP+B-Zo+1PA8qe-VVEdBbjt62Ty5sTxVzRA_prBdUJaDwzw@mail.gmail.com>

An exercise to see if scoreatpercentile can be vectorized

https://gist.github.com/4370540

Josef


From programmercplusplus at yahoo.com  Tue Dec 25 11:16:49 2012
From: programmercplusplus at yahoo.com (Cplusplus Programmer)
Date: Tue, 25 Dec 2012 16:16:49 +0000 (GMT)
Subject: [SciPy-User] fancy indexing lil_matrix extremely slow
Message-ID: <1356452209.74565.YahooMailNeo@web171505.mail.ir2.yahoo.com>

Hello All,

I am using scipy version 0.9. I have a sparse matrix, of which I do not know beforehand what values it will get.
What I did was first generating a scipy.sparse.lil_matrix of zeros (N x N matrix).

To assign values to the sparse matrix I use fancy slicing e.g

for i in range( N ):
??? myLilMatrix[ 0: N-4, i ] = myArrayWithValuesToAssign

I works fine, however, it is very very slow. I could not use other sparse matrix types, since those does not support fancy indexing for assignment. I was wondering if there is a way to speed up sparse matrix assignements ?

If scipy.sparse module does not support this, does anyone know another module which will be faster ?

regards,

pcpp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121225/c3b5855d/attachment.html>

From cmutel at gmail.com  Tue Dec 25 11:27:42 2012
From: cmutel at gmail.com (Christopher Mutel)
Date: Tue, 25 Dec 2012 17:27:42 +0100
Subject: [SciPy-User] fancy indexing lil_matrix extremely slow
In-Reply-To: <1356452209.74565.YahooMailNeo@web171505.mail.ir2.yahoo.com>
References: <1356452209.74565.YahooMailNeo@web171505.mail.ir2.yahoo.com>
Message-ID: <CAOc=Qn+Z=h=hwOhh=Mex1C-PtPyBUfwozoQ3VaN8wNNGNvX33w@mail.gmail.com>

It sounds like you would be better served using a COO matrix constructor
instead of LIL (
http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.coo_matrix.html).
>From your example code, you don't actually need slicing.

On Tue, Dec 25, 2012 at 5:16 PM, Cplusplus Programmer <
programmercplusplus at yahoo.com> wrote:

> Hello All,
>
> I am using scipy version 0.9. I have a sparse matrix, of which I do not
> know beforehand what values it will get.
> What I did was first generating a scipy.sparse.lil_matrix of zeros (N x N
> matrix).
>
> To assign values to the sparse matrix I use fancy slicing e.g
>
> for i in range( N ):
>     myLilMatrix[ 0: N-4, i ] = myArrayWithValuesToAssign
>
> I works fine, however, it is very very slow. I could not use other sparse
> matrix types, since those does not support fancy indexing for assignment. I
> was wondering if there is a way to speed up sparse matrix assignements ?
>
> If scipy.sparse module does not support this, does anyone know another
> module which will be faster ?
>
> regards,
>
> pcpp
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121225/423df4b4/attachment.html>

From pranava at lsi.upc.edu  Tue Dec 25 13:34:11 2012
From: pranava at lsi.upc.edu (Pranava Swaroop Madhyastha)
Date: Tue, 25 Dec 2012 19:34:11 +0100
Subject: [SciPy-User] Help: Multiplying two sparse matrices
Message-ID: <CA+6_nTddWLFj3r9-3gpRX8EPWeK0enQjNyP6SjcXVoLoXLkZyw@mail.gmail.com>

Hi,

I have a problem multiplying two sparse matrices. Here is a sample:

import scipy.sparse as ss

D = ss.rand(49837, 1000, format='csr', density=0.1)

E = ss.rand(1000, 49837, format='csr', density=0.1)

D * E

This is the error that I get:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-c94d439b5bf5> in <module>()
----> 1 D * E

/usr/local/lib/python2.7/dist-packages/scipy/sparse/base.pyc in
__mul__(self, other)
    254             if self.shape[1] != other.shape[0]:
    255                 raise ValueError('dimension mismatch')
--> 256             return self._mul_sparse_matrix(other)
    257
    258         try:

/usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py in
_mul_sparse_matrix(self, other)
    296
    297         nnz = indptr[-1]
--> 298         indices = np.empty(nnz, dtype=np.intc)
    299         data    = np.empty(nnz, dtype=upcast(self.dtype,other.dtype))
    300

ValueError: negative dimensions are not allowed


Apparently, the error seems to be because of dtype=np.intc. Is there a
work around for this?

Thanks and Regards,
Pranava


From sebastian at sipsolutions.net  Tue Dec 25 13:47:06 2012
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Tue, 25 Dec 2012 19:47:06 +0100
Subject: [SciPy-User] Help: Multiplying two sparse matrices
In-Reply-To: <CA+6_nTddWLFj3r9-3gpRX8EPWeK0enQjNyP6SjcXVoLoXLkZyw@mail.gmail.com>
References: <CA+6_nTddWLFj3r9-3gpRX8EPWeK0enQjNyP6SjcXVoLoXLkZyw@mail.gmail.com>
Message-ID: <1356461226.26515.3.camel@sebastian-laptop>

On Tue, 2012-12-25 at 19:34 +0100, Pranava Swaroop Madhyastha wrote:
> Hi,
> 
> I have a problem multiplying two sparse matrices. Here is a sample:
> 
> import scipy.sparse as ss
> 
> D = ss.rand(49837, 1000, format='csr', density=0.1)
> 
> E = ss.rand(1000, 49837, format='csr', density=0.1)
> 
> D * E
> 
> This is the error that I get:
> 
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call last)
> <ipython-input-4-c94d439b5bf5> in <module>()
> ----> 1 D * E
> 
> /usr/local/lib/python2.7/dist-packages/scipy/sparse/base.pyc in
> __mul__(self, other)
>     254             if self.shape[1] != other.shape[0]:
>     255                 raise ValueError('dimension mismatch')
> --> 256             return self._mul_sparse_matrix(other)
>     257
>     258         try:
> 
> /usr/local/lib/python2.7/dist-packages/scipy/sparse/compressed.py in
> _mul_sparse_matrix(self, other)
>     296
>     297         nnz = indptr[-1]
> --> 298         indices = np.empty(nnz, dtype=np.intc)
>     299         data    = np.empty(nnz, dtype=upcast(self.dtype,other.dtype))
>     300
> 
> ValueError: negative dimensions are not allowed
> 

This is because the result has too many nonzero elements. Scipy seems to
use 32bit integers to save positions, and if you have too many non-zero
elements, 32bit integers can overflow easily... Maybe if you feel like
it, you could do a PR against scipy to change the error message with
something understandable. I don't know if there is a great reason to use
32bit integers, though I guess it saves quite a bit of RAM if it is
large enough.

However, if your result has so many nonzero elements, it seems sounds
like the sparsity is probably so low, you probably could just as well
use dense matrix (for the result, if possible), for this example.

Regards,

Sebastian

> 
> Apparently, the error seems to be because of dtype=np.intc. Is there a
> work around for this?
> 
> Thanks and Regards,
> Pranava
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 




From pranava at lsi.upc.edu  Tue Dec 25 14:02:22 2012
From: pranava at lsi.upc.edu (Pranava Swaroop Madhyastha)
Date: Tue, 25 Dec 2012 20:02:22 +0100
Subject: [SciPy-User] Help: Multiplying two sparse matrices
In-Reply-To: <1356461226.26515.3.camel@sebastian-laptop>
References: <CA+6_nTddWLFj3r9-3gpRX8EPWeK0enQjNyP6SjcXVoLoXLkZyw@mail.gmail.com>
	<1356461226.26515.3.camel@sebastian-laptop>
Message-ID: <CA+6_nTe_NYUdVB+v+v1OpvqT=MFXNKP1m-QXOi+akPDmgvSk+w@mail.gmail.com>

On Tue, Dec 25, 2012 at 7:47 PM, Sebastian Berg
<sebastian at sipsolutions.net> wrote:
> This is because the result has too many nonzero elements. Scipy seems to
> use 32bit integers to save positions, and if you have too many non-zero
> elements, 32bit integers can overflow easily... Maybe if you feel like
> it, you could do a PR against scipy to change the error message with
> something understandable. I don't know if there is a great reason to use
> 32bit integers, though I guess it saves quite a bit of RAM if it is
> large enough.

I will look into it.


>
> However, if your result has so many nonzero elements, it seems sounds
> like the sparsity is probably so low, you probably could just as well
> use dense matrix (for the result, if possible), for this example.

I am actually constrained by RAM. Is there any other way around
without converting it to dense matrix?

>
> Regards,
>
> Sebastian


From ralf.gommers at gmail.com  Wed Dec 26 09:17:32 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Wed, 26 Dec 2012 15:17:32 +0100
Subject: [SciPy-User] scoreatpercentile vectorized
In-Reply-To: <CAMMTP+B-Zo+1PA8qe-VVEdBbjt62Ty5sTxVzRA_prBdUJaDwzw@mail.gmail.com>
References: <CAMMTP+B-Zo+1PA8qe-VVEdBbjt62Ty5sTxVzRA_prBdUJaDwzw@mail.gmail.com>
Message-ID: <CABL7CQgk8Coj6=OEb=PFMqKv6Q4PpOAn7gX9Va23PUUiSrFMSA@mail.gmail.com>

On Mon, Dec 24, 2012 at 9:26 PM, <josef.pktd at gmail.com> wrote:

> An exercise to see if scoreatpercentile can be vectorized
>
> https://gist.github.com/4370540
>

Vectorized `per` already part of https://github.com/scipy/scipy/pull/374

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121226/b525de75/attachment.html>

From josef.pktd at gmail.com  Wed Dec 26 09:37:52 2012
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Wed, 26 Dec 2012 09:37:52 -0500
Subject: [SciPy-User] scoreatpercentile vectorized
In-Reply-To: <CABL7CQgk8Coj6=OEb=PFMqKv6Q4PpOAn7gX9Va23PUUiSrFMSA@mail.gmail.com>
References: <CAMMTP+B-Zo+1PA8qe-VVEdBbjt62Ty5sTxVzRA_prBdUJaDwzw@mail.gmail.com>
	<CABL7CQgk8Coj6=OEb=PFMqKv6Q4PpOAn7gX9Va23PUUiSrFMSA@mail.gmail.com>
Message-ID: <CAMMTP+BkU-FNy1_1JjYuLGgOtH68tCLkjb+FOcqpFxThaE=2tg@mail.gmail.com>

On Wed, Dec 26, 2012 at 9:17 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Mon, Dec 24, 2012 at 9:26 PM, <josef.pktd at gmail.com> wrote:
>>
>> An exercise to see if scoreatpercentile can be vectorized
>>
>> https://gist.github.com/4370540
>
>
> Vectorized `per` already part of https://github.com/scipy/scipy/pull/374

That, like the numpy version, is vectorized by looping over all per

Josef

>
> Ralf
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From ndbecker2 at gmail.com  Wed Dec 26 11:38:01 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 26 Dec 2012 11:38:01 -0500
Subject: [SciPy-User] suggestions for modeling
Message-ID: <kbf958$th4$1@ger.gmane.org>

I'm looking for some suggestions for modeling some data.

Suppose I'm given a set of black boxes, which have some nonlinear input/output 
relations.  Each box will be similar, but there will be unit-to-unit variations.

I want to model the general input/output relation as some curve having 
relatively few parameters to approximate the unit-to-unit variations.

Any ideas?



From francescoboccacci at libero.it  Wed Dec 26 11:47:26 2012
From: francescoboccacci at libero.it (francescoboccacci at libero.it)
Date: Wed, 26 Dec 2012 17:47:26 +0100 (CET)
Subject: [SciPy-User] UD utilization distribution kernel function
Message-ID: <16364885.3721151356540446303.JavaMail.defaultUser@defaultHost>

Hi all,
i'd like to found some kernel function in scipy to replicate the same behavior 
of UD calculation of the package adehabitat (R).
Is there some kernel functions that can i use?

Thanks

Francesco


From bahtiyor_zohidov at mail.ru  Tue Dec 25 11:55:45 2012
From: bahtiyor_zohidov at mail.ru (=?UTF-8?B?SGFwcHltYW4=?=)
Date: Tue, 25 Dec 2012 20:55:45 +0400
Subject: [SciPy-User] =?utf-8?q?WHICH_ONE_IS_BETTER=3F=3F_Loadtxt=28=29_or?=
 =?utf-8?b?IG9wZW4oJ2ZpbGVuYW1lJywgJ3InKSBtZXRob2Q=?=
Message-ID: <1356454545.292651328@f305.mail.ru>


I had 120x150 matrix which I have to read Rows by Cols... 
for example: 
M=(120,150) . This matrix exists under name: BigFile.txt
So, M=loadtxt ( 'BigFile.txt' ). 

After processing M matrix I have M1 ?matrix which has the same Row and Cols (120 x 150 )

I saved it: savetxt ( ' Bigprocess.txt ', M1 ). The problem is I can not control Rows and Cols to be the same order of rows and cols as in the M matrix

The second way: or is it better to use open ('filename.txt') ?method ??????
QUESTION: 
1) WHich one (loadtxt and savetxt) or (open ('filename.txt') and write (data)) faster to write my matrix
2) How can I control over M1 matrix using loadtxt method??????

Any answers will be appreciated!!!!

--
????? ????????? ?????? @Mail.Ru.
??? ??????????? ? ????? ????????? - m.mail.ru
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121225/223d1d68/attachment.html>

From amschne at umich.edu  Wed Dec 26 15:31:36 2012
From: amschne at umich.edu (Adam Schneider)
Date: Wed, 26 Dec 2012 12:31:36 -0800
Subject: [SciPy-User] numpy.test() fail mac with homebrew python 2.7
Message-ID: <CAAiJ8FKNhNxtLKtgRATYu-xDGJ+2Cekj67iFfNx3My60CyCC+g@mail.gmail.com>

After installing numpy version 1.6.2 and scipy version 0.11.0 via pip 1.2.1
for python 2.7.3 (from homebrew) on my MacBook Air with core i7 processors
on OS X 10.8.2, I ran tests using numpy.test() and scipy.test() which both
failed.  Below are three failure lines from the numpy.test, and attached is
the full log.

FAIL: Test automatically generated assignments which overlap in memory.
FAIL: test_nditer.test_iter_broadcasting_errors
FAIL: test_nditer.test_iter_array_cast

Will you please let me know how big of an issue this is and the best way to
fix it?

Thank you,

Adam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121226/61c03c1b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpy_fail.log
Type: application/octet-stream
Size: 57998 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121226/61c03c1b/attachment.obj>

From renato.fabbri at gmail.com  Wed Dec 26 16:17:52 2012
From: renato.fabbri at gmail.com (Renato Fabbri)
Date: Wed, 26 Dec 2012 19:17:52 -0200
Subject: [SciPy-User] WHICH ONE IS BETTER?? Loadtxt() or open('filename',
 'r') method
In-Reply-To: <1356454545.292651328@f305.mail.ru>
References: <1356454545.292651328@f305.mail.ru>
Message-ID: <CALqdYOy=dfTaoDJuy2NQvD_=nYX=VEBYXfM1x3y2+XtFzAHPfg@mail.gmail.com>

try it with ipython's timeit ;-)
or make loops and measure them with time.time() itself.

paste back results if any pls

cheers, happyman

2012/12/25 Happyman <bahtiyor_zohidov at mail.ru>:
>
> I had 120x150 matrix which I have to read Rows by Cols...
> for example:
> M=(120,150) . This matrix exists under name: BigFile.txt
> So, M=loadtxt ( 'BigFile.txt' ).
>
> After processing M matrix I have M1  matrix which has the same Row and Cols
> (120 x 150 )
>
> I saved it: savetxt ( ' Bigprocess.txt ', M1 ). The problem is I can not
> control Rows and Cols to be the same order of rows and cols as in the M
> matrix
>
> The second way: or is it better to use open ('filename.txt')  method ??????
> QUESTION:
> 1) WHich one (loadtxt and savetxt) or (open ('filename.txt') and write
> (data)) faster to write my matrix
> 2) How can I control over M1 matrix using loadtxt method??????
>
> Any answers will be appreciated!!!!
>
> --
> ????? ????????? ?????? @Mail.Ru.
> ??? ??????????? ? ????? ????????? - m.mail.ru
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
GNU/Linux User #479299
labmacambira.sf.net


From ralf.gommers at gmail.com  Thu Dec 27 14:37:47 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Thu, 27 Dec 2012 20:37:47 +0100
Subject: [SciPy-User] numpy.test() fail mac with homebrew python 2.7
In-Reply-To: <CAAiJ8FKNhNxtLKtgRATYu-xDGJ+2Cekj67iFfNx3My60CyCC+g@mail.gmail.com>
References: <CAAiJ8FKNhNxtLKtgRATYu-xDGJ+2Cekj67iFfNx3My60CyCC+g@mail.gmail.com>
Message-ID: <CABL7CQhoOmfrLWuyS06QdVzwskNG5y+SBWAVgegNMwzq9q9gtA@mail.gmail.com>

On Wed, Dec 26, 2012 at 9:31 PM, Adam Schneider <amschne at umich.edu> wrote:

> After installing numpy version 1.6.2 and scipy version 0.11.0 via pip
> 1.2.1 for python 2.7.3 (from homebrew) on my MacBook Air with core i7
> processors on OS X 10.8.2, I ran tests using numpy.test() and scipy.test()
> which both failed.  Below are three failure lines from the numpy.test, and
> attached is the full log.
>
> FAIL: Test automatically generated assignments which overlap in memory.
> FAIL: test_nditer.test_iter_broadcasting_errors
> FAIL: test_nditer.test_iter_array_cast
>
> Will you please let me know how big of an issue this is and the best way
> to fix it?
>

Hi Adam,

It looks like all the errors come from files left over from a previous
numpy install. The basis and cast methods were added in 1.7.x.

An "rm -rf numpy/" and clean reinstall should fix it.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121227/9d2e3467/attachment.html>

From ralf.gommers at gmail.com  Thu Dec 27 16:39:33 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Thu, 27 Dec 2012 22:39:33 +0100
Subject: [SciPy-User] numpy.test() fail mac with homebrew python 2.7
In-Reply-To: <CAAiJ8FKtSL_G1hLsginUv4SV4c71J4X7VFDE6rZzANu1c0rv1Q@mail.gmail.com>
References: <CABL7CQhoOmfrLWuyS06QdVzwskNG5y+SBWAVgegNMwzq9q9gtA@mail.gmail.com>
	<190AEC12-2F22-461D-8ED0-378801B65ADF@umich.edu>
	<CAAiJ8FKtSL_G1hLsginUv4SV4c71J4X7VFDE6rZzANu1c0rv1Q@mail.gmail.com>
Message-ID: <CABL7CQgzgwcP93J6iMedKP0zuLq3bMTcv662Gyp3LD4KTdk7Kg@mail.gmail.com>

On Thu, Dec 27, 2012 at 10:09 PM, Adam Schneider <amschne at umich.edu> wrote:

> Hey Ralf, thanks for the quick response.
>
> I tried a clean install, but scipy still fails to install with this error:
> numpy.distutils.npy_pkg_config.PkgNotFound: Could not find file(s)
> ['/usr/local/lib/python2.7/site-packages/numpy/core/lib/npy-pkg-config/npymath.ini']
>
> It looks like it is the same issue that people are running into here:
> http://stackoverflow.com/questions/12574604/scipy-install-on-mountain-lion-failing
>
> Trying the workaround suggested on SO and installing numpy from top of
> tree allows scipy to install from pip, but numpy.test() fails the following
> test:
>
> FAIL: Test numpy dot with different order C, F
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in
> runTest
>     self.test(*self.arg)
>   File
> "/usr/local/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py",
> line 119, in test_dot_array_order
>     assert_array_equal(a.dot(a_T), a.dot(a.T))
>   File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py",
> line 719, in assert_array_equal
>     verbose=verbose, header='Arrays are not equal')
>   File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py",
> line 645, in assert_array_compare
>     raise AssertionError(msg)
> AssertionError:
> Arrays are not equal
>
> (mismatch 11.0%)
>  x: array([[  8.33980975,  -1.73402744,   8.39918067,  -0.92846214,
>          -1.42408847,   3.07286699,   2.08420574,  -0.48730198,
>           0.1639567 ,  -1.04598212],...
>  y: array([[  8.33980975,  -1.73402744,   8.39918067,  -0.92846214,
>          -1.42408847,   3.07286699,   2.08420574,  -0.48730198,
>           0.1639567 ,  -1.04598212],...
>
> ----------------------------------------------------------------------
> Ran 4801 tests in 18.344s
>
> FAILED (KNOWNFAIL=5, SKIP=5, failures=1)
> <nose.result.TextTestResult run=4801 errors=0 failures=1>
>
> Should this be a known fail?
>

No, it shouldn't. Doesn't look serious but should be fixed. Could you open
an issue on Github, and list the compilers and build command used?


> Also, should scipy from pip (0.11.0) install cleanly with numpy from pip
> (1.6.2)?
>

It should, but apparently it's broken in a not very reproducible way.
That's going to take some time to fix probably. This sort of thing is why I
usually avoid pip & co by the way.

Ralf


> On Thu, Dec 27, 2012 at 12:17 PM, Jordan Schneider <jbschne at umich.edu>wrote:
>
>>
>>
>> Sent from my iPad
>>
>> Begin forwarded message:
>>
>> *From:* Ralf Gommers <ralf.gommers at gmail.com>
>> *Date:* December 27, 2012, 2:37:47 PM EST
>> *To:* SciPy Users List <scipy-user at scipy.org>
>> *Cc:* Jordan Schneider <jbschne at umich.edu>
>> *Subject:* *Re: [SciPy-User] numpy.test() fail mac with homebrew python
>> 2.7*
>>
>>
>>
>>
>> On Wed, Dec 26, 2012 at 9:31 PM, Adam Schneider <amschne at umich.edu>wrote:
>>
>>> After installing numpy version 1.6.2 and scipy version 0.11.0 via pip
>>> 1.2.1 for python 2.7.3 (from homebrew) on my MacBook Air with core i7
>>> processors on OS X 10.8.2, I ran tests using numpy.test() and scipy.test()
>>> which both failed.  Below are three failure lines from the numpy.test, and
>>> attached is the full log.
>>>
>>> FAIL: Test automatically generated assignments which overlap in memory.
>>> FAIL: test_nditer.test_iter_broadcasting_errors
>>> FAIL: test_nditer.test_iter_array_cast
>>>
>>> Will you please let me know how big of an issue this is and the best way
>>> to fix it?
>>>
>>
>> Hi Adam,
>>
>> It looks like all the errors come from files left over from a previous
>> numpy install. The basis and cast methods were added in 1.7.x.
>>
>> An "rm -rf numpy/" and clean reinstall should fix it.
>>
>> Ralf
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121227/c9928813/attachment.html>

From amschne at umich.edu  Thu Dec 27 16:09:20 2012
From: amschne at umich.edu (Adam Schneider)
Date: Thu, 27 Dec 2012 13:09:20 -0800
Subject: [SciPy-User] numpy.test() fail mac with homebrew python 2.7
In-Reply-To: <190AEC12-2F22-461D-8ED0-378801B65ADF@umich.edu>
References: <CABL7CQhoOmfrLWuyS06QdVzwskNG5y+SBWAVgegNMwzq9q9gtA@mail.gmail.com>
	<190AEC12-2F22-461D-8ED0-378801B65ADF@umich.edu>
Message-ID: <CAAiJ8FKtSL_G1hLsginUv4SV4c71J4X7VFDE6rZzANu1c0rv1Q@mail.gmail.com>

Hey Ralf, thanks for the quick response.

I tried a clean install, but scipy still fails to install with this error:
numpy.distutils.npy_pkg_config.PkgNotFound: Could not find file(s)
['/usr/local/lib/python2.7/site-packages/numpy/core/lib/npy-pkg-config/npymath.ini']

It looks like it is the same issue that people are running into here:
http://stackoverflow.com/questions/12574604/scipy-install-on-mountain-lion-failing

Trying the workaround suggested on SO and installing numpy from top of tree
allows scipy to install from pip, but numpy.test() fails the following test:

FAIL: Test numpy dot with different order C, F
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in
runTest
    self.test(*self.arg)
  File
"/usr/local/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py",
line 119, in test_dot_array_order
    assert_array_equal(a.dot(a_T), a.dot(a.T))
  File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py",
line 719, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py",
line 645, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not equal

(mismatch 11.0%)
 x: array([[  8.33980975,  -1.73402744,   8.39918067,  -0.92846214,
         -1.42408847,   3.07286699,   2.08420574,  -0.48730198,
          0.1639567 ,  -1.04598212],...
 y: array([[  8.33980975,  -1.73402744,   8.39918067,  -0.92846214,
         -1.42408847,   3.07286699,   2.08420574,  -0.48730198,
          0.1639567 ,  -1.04598212],...

----------------------------------------------------------------------
Ran 4801 tests in 18.344s

FAILED (KNOWNFAIL=5, SKIP=5, failures=1)
<nose.result.TextTestResult run=4801 errors=0 failures=1>

Should this be a known fail? Also, should scipy from pip (0.11.0) install
cleanly with numpy from pip (1.6.2)?

Thanks again,

Adam


On Thu, Dec 27, 2012 at 12:17 PM, Jordan Schneider <jbschne at umich.edu>wrote:

>
>
> Sent from my iPad
>
> Begin forwarded message:
>
> *From:* Ralf Gommers <ralf.gommers at gmail.com>
> *Date:* December 27, 2012, 2:37:47 PM EST
> *To:* SciPy Users List <scipy-user at scipy.org>
> *Cc:* Jordan Schneider <jbschne at umich.edu>
> *Subject:* *Re: [SciPy-User] numpy.test() fail mac with homebrew python
> 2.7*
>
>
>
>
> On Wed, Dec 26, 2012 at 9:31 PM, Adam Schneider <amschne at umich.edu> wrote:
>
>> After installing numpy version 1.6.2 and scipy version 0.11.0 via pip
>> 1.2.1 for python 2.7.3 (from homebrew) on my MacBook Air with core i7
>> processors on OS X 10.8.2, I ran tests using numpy.test() and scipy.test()
>> which both failed.  Below are three failure lines from the numpy.test, and
>> attached is the full log.
>>
>> FAIL: Test automatically generated assignments which overlap in memory.
>> FAIL: test_nditer.test_iter_broadcasting_errors
>> FAIL: test_nditer.test_iter_array_cast
>>
>> Will you please let me know how big of an issue this is and the best way
>> to fix it?
>>
>
> Hi Adam,
>
> It looks like all the errors come from files left over from a previous
> numpy install. The basis and cast methods were added in 1.7.x.
>
> An "rm -rf numpy/" and clean reinstall should fix it.
>
> Ralf
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121227/853ea7b3/attachment.html>

From amschne at umich.edu  Thu Dec 27 21:21:12 2012
From: amschne at umich.edu (Adam Schneider)
Date: Thu, 27 Dec 2012 18:21:12 -0800
Subject: [SciPy-User] numpy.test() fail mac with homebrew python 2.7
In-Reply-To: <CABL7CQgzgwcP93J6iMedKP0zuLq3bMTcv662Gyp3LD4KTdk7Kg@mail.gmail.com>
References: <CABL7CQhoOmfrLWuyS06QdVzwskNG5y+SBWAVgegNMwzq9q9gtA@mail.gmail.com>
	<190AEC12-2F22-461D-8ED0-378801B65ADF@umich.edu>
	<CAAiJ8FKtSL_G1hLsginUv4SV4c71J4X7VFDE6rZzANu1c0rv1Q@mail.gmail.com>
	<CABL7CQgzgwcP93J6iMedKP0zuLq3bMTcv662Gyp3LD4KTdk7Kg@mail.gmail.com>
Message-ID: <CAAiJ8F+BHwGifjMqHHCmJdDG0Y_mB8q80bFT1ucoar5-36Qi2w@mail.gmail.com>

Thanks for the help. numpy 1.7.0b2 and 1.6.2 are passing tests. I filed an
issue for 1.8.0.dev-a8c641f: https://github.com/numpy/numpy/issues/2860

However, scipy 0.11.0 and top of tree (0.12.0.dev-b69fe18) are not passing
tests (installing directly from distutils and from pip). Attached is the
output from 0.11.0 tests (72 failures). Any idea why this might be
happening?

Thanks,
Adam


On Thu, Dec 27, 2012 at 1:39 PM, Ralf Gommers <ralf.gommers at gmail.com>wrote:

>
>
>
> On Thu, Dec 27, 2012 at 10:09 PM, Adam Schneider <amschne at umich.edu>wrote:
>
>> Hey Ralf, thanks for the quick response.
>>
>> I tried a clean install, but scipy still fails to install with this error:
>> numpy.distutils.npy_pkg_config.PkgNotFound: Could not find file(s)
>> ['/usr/local/lib/python2.7/site-packages/numpy/core/lib/npy-pkg-config/npymath.ini']
>>
>> It looks like it is the same issue that people are running into here:
>> http://stackoverflow.com/questions/12574604/scipy-install-on-mountain-lion-failing
>>
>> Trying the workaround suggested on SO and installing numpy from top of
>> tree allows scipy to install from pip, but numpy.test() fails the following
>> test:
>>
>> FAIL: Test numpy dot with different order C, F
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197,
>> in runTest
>>     self.test(*self.arg)
>>   File
>> "/usr/local/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.py",
>> line 119, in test_dot_array_order
>>     assert_array_equal(a.dot(a_T), a.dot(a.T))
>>   File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py",
>> line 719, in assert_array_equal
>>     verbose=verbose, header='Arrays are not equal')
>>   File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py",
>> line 645, in assert_array_compare
>>     raise AssertionError(msg)
>> AssertionError:
>> Arrays are not equal
>>
>> (mismatch 11.0%)
>>  x: array([[  8.33980975,  -1.73402744,   8.39918067,  -0.92846214,
>>          -1.42408847,   3.07286699,   2.08420574,  -0.48730198,
>>           0.1639567 ,  -1.04598212],...
>>  y: array([[  8.33980975,  -1.73402744,   8.39918067,  -0.92846214,
>>          -1.42408847,   3.07286699,   2.08420574,  -0.48730198,
>>           0.1639567 ,  -1.04598212],...
>>
>> ----------------------------------------------------------------------
>> Ran 4801 tests in 18.344s
>>
>> FAILED (KNOWNFAIL=5, SKIP=5, failures=1)
>> <nose.result.TextTestResult run=4801 errors=0 failures=1>
>>
>> Should this be a known fail?
>>
>
> No, it shouldn't. Doesn't look serious but should be fixed. Could you open
> an issue on Github, and list the compilers and build command used?
>
>
>> Also, should scipy from pip (0.11.0) install cleanly with numpy from pip
>> (1.6.2)?
>>
>
> It should, but apparently it's broken in a not very reproducible way.
> That's going to take some time to fix probably. This sort of thing is why I
> usually avoid pip & co by the way.
>
> Ralf
>
>
>> On Thu, Dec 27, 2012 at 12:17 PM, Jordan Schneider <jbschne at umich.edu>wrote:
>>
>>>
>>>
>>> Sent from my iPad
>>>
>>> Begin forwarded message:
>>>
>>> *From:* Ralf Gommers <ralf.gommers at gmail.com>
>>> *Date:* December 27, 2012, 2:37:47 PM EST
>>> *To:* SciPy Users List <scipy-user at scipy.org>
>>> *Cc:* Jordan Schneider <jbschne at umich.edu>
>>> *Subject:* *Re: [SciPy-User] numpy.test() fail mac with homebrew python
>>> 2.7*
>>>
>>>
>>>
>>>
>>> On Wed, Dec 26, 2012 at 9:31 PM, Adam Schneider <amschne at umich.edu>wrote:
>>>
>>>> After installing numpy version 1.6.2 and scipy version 0.11.0 via pip
>>>> 1.2.1 for python 2.7.3 (from homebrew) on my MacBook Air with core i7
>>>> processors on OS X 10.8.2, I ran tests using numpy.test() and scipy.test()
>>>> which both failed.  Below are three failure lines from the numpy.test, and
>>>> attached is the full log.
>>>>
>>>> FAIL: Test automatically generated assignments which overlap in memory.
>>>> FAIL: test_nditer.test_iter_broadcasting_errors
>>>> FAIL: test_nditer.test_iter_array_cast
>>>>
>>>> Will you please let me know how big of an issue this is and the best
>>>> way to fix it?
>>>>
>>>
>>> Hi Adam,
>>>
>>> It looks like all the errors come from files left over from a previous
>>> numpy install. The basis and cast methods were added in 1.7.x.
>>>
>>> An "rm -rf numpy/" and clean reinstall should fix it.
>>>
>>> Ralf
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121227/0c23cb89/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scipy_0.11.0_failuretests.log
Type: application/octet-stream
Size: 110394 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121227/0c23cb89/attachment.obj>

From bahtiyor_zohidov at mail.ru  Sat Dec 29 22:32:13 2012
From: bahtiyor_zohidov at mail.ru (=?UTF-8?B?SGFwcHltYW4=?=)
Date: Sun, 30 Dec 2012 07:32:13 +0400
Subject: [SciPy-User] =?utf-8?q?3D_array_problem_in_Python?=
Message-ID: <1356838333.805656881@f211.mail.ru>

 Hello

I have 3 dimensional array ?which I want ?to calculate in a huge process. Everything is working well if I use ordinary way which is unsuitable in Python like the following:

nums=32
rows=120
cols=150

for k in range(0,nums):
? ? ? ? ? for i in range(0,rows):
? ? ? ? ? ? ? ? ? ? ?for j in range(0,cols):
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? if float ( R[ k ] [ i ] [ j ] ) == 0.0:
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?val11 [ i ] =0.0
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? else:
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?val11[ i ] [ j ], val22[ i ][ j ] = integrate.quad( lambda x : ?F1(x)*F2(x) , 0 , pi)

But, this calculation takes so long time, let's say about ?1 hour (theoretically)... Is there any better way to easily and fast calculate the process such as [ F( i ) for i in xlist ] or something like that rather than using for loop?


?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/72f77b63/attachment.html>

From guziy.sasha at gmail.com  Sun Dec 30 13:32:09 2012
From: guziy.sasha at gmail.com (Oleksandr Huziy)
Date: Sun, 30 Dec 2012 13:32:09 -0500
Subject: [SciPy-User] 3D array problem in Python
In-Reply-To: <1356838333.805656881@f211.mail.ru>
References: <1356838333.805656881@f211.mail.ru>
Message-ID: <CAN3p1sWC0b==6X5xf8M3y7ub8HJq790PyeaM-1g9ubMzgBH+bw@mail.gmail.com>

Hi,
I understand you've simplified the example here, but just in case you did
not,

your integral does not depend on i,j or k, so you could calculate it only
once.


import numpy as np
a, b = integrate.quad( lambda x :  F1(x)*F2(x) , 0 , pi)

val11 = np.array(val11)
val22 = np.array(val22)
R = np.array(R)

val11 = 0
val22 = 0
for k in range(nums):
    nonz = R[k] > 0
    val11[nonz] = a
    val22[nonz] = b


Or if the above is not possible try this

pairs = [
[
[integrate.quad( lambda x :  F1(x)*F2(x) , 0 , pi) if R[k][i][j] > 0 else
(0,0) for j in range(cols) ]
for i in range(rows)]
for k in range(nums)]


But then from pairs you have to work how to extract vals11 and vals22

HTH

Cheers
--
Oleksandr (Sasha) Huziy


2012/12/29 Happyman <bahtiyor_zohidov at mail.ru>

> Hello
>
> I have 3 dimensional array  which I want  to calculate in a huge process.
> Everything is working well if I use ordinary way which is unsuitable in
> Python like the following:
>
> nums=32
> rows=120
> cols=150
>
> for k in range(0,nums):
>           for i in range(0,rows):
>                      for j in range(0,cols):
>                                       if float ( R[ k ] [ i ] [ j ] ) ==
> 0.0:
>                                                  val11 [ i ] =0.0
>                                       else:
>                                                  val11[ i ] [ j ], val22[
> i ][ j ] = integrate.quad( lambda x :  F1(x)*F2(x) , 0 , pi)
>
> But, this calculation takes so long time, let's say about  1 hour
> (theoretically)... Is there any better way to easily and fast calculate the
> process such as [ F( i ) for i in xlist ] or something like that rather
> than using for loop?
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/3cbf031e/attachment.html>

From vaggi.federico at gmail.com  Sun Dec 30 14:09:37 2012
From: vaggi.federico at gmail.com (federico vaggi)
Date: Sun, 30 Dec 2012 20:09:37 +0100
Subject: [SciPy-User] 3D array problem in Python (Happyman)
Message-ID: <CAGvd0=hrQCxZV2h9uOqFmovtck-kD4b3G8uXLPSkd355kVw8qw@mail.gmail.com>

There are faster ways to loop over the array using array slicing:

http://www.scipy.org/Tentative_NumPy_Tutorial#head-864862d3f2bb4c32f04260fac61eb4ef34788c4c

If, however, you have to apply a slow function to all non-zero elements,
that will probably be your bottleneck, and fast access using array slicing
won't help you.

By the way, as a quick hint, an easy way to access all non-zero elements of
an array is using logical indexing, which you will be familiar with if
you've used MATLAB before.  You appear however to be using python lists and
not numpy arrays judging by your syntax.

> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 30 Dec 2012 07:32:13 +0400
> From: Happyman <bahtiyor_zohidov at mail.ru>
> Subject: [SciPy-User] 3D array problem in Python
> To: scipy-user <scipy-user at scipy.org>
> Message-ID: <1356838333.805656881 at f211.mail.ru>
> Content-Type: text/plain; charset="utf-8"
>
>  Hello
>
> I have 3 dimensional array ?which I want ?to calculate in a huge process.
> Everything is working well if I use ordinary way which is unsuitable in
> Python like the following:
>
> nums=32
> rows=120
> cols=150
>
> for k in range(0,nums):
> ? ? ? ? ? for i in range(0,rows):
> ? ? ? ? ? ? ? ? ? ? ?for j in range(0,cols):
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? if float ( R[ k ] [ i ] [ j ] ) ==
> 0.0:
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?val11 [ i ] =0.0
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? else:
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?val11[ i ] [ j ], val22[
> i ][ j ] = integrate.quad( lambda x : ?F1(x)*F2(x) , 0 , pi)
>
> But, this calculation takes so long time, let's say about ?1 hour
> (theoretically)... Is there any better way to easily and fast calculate the
> process such as [ F( i ) for i in xlist ] or something like that rather
> than using for loop?
>
>
> ?
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/72f77b63/attachment-0001.html
>
> ------------------------------
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> End of SciPy-User Digest, Vol 112, Issue 44
> *******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/c6e88d7b/attachment.html>

From thoger.emil at gmail.com  Sun Dec 30 10:47:07 2012
From: thoger.emil at gmail.com (=?ISO-8859-1?Q?Th=F8ger_Rivera-Thorsen?=)
Date: Sun, 30 Dec 2012 16:47:07 +0100
Subject: [SciPy-User] 3D array problem in Python
In-Reply-To: <1356838333.805656881@f211.mail.ru>
References: <1356838333.805656881@f211.mail.ru>
Message-ID: <50E061FB.2090507@gmail.com>

Use np.where() or logical indexing (same thing, really) to mask your 
array, then perform the operations. Let's say your array is called A:

A[float(A) == 0.0] = 0.0

A[float(A) != 0.0] = [...etc.]


This, of course, only works if the operation for an entry doesn't depend 
on other entries in the array; but it should give you a great speed gain.

Cheers;

Emil



On 12/30/2012 04:32 AM, Happyman wrote:
> Hello
>
> I have 3 dimensional array  which I want  to calculate in a huge 
> process. Everything is working well if I use ordinary way which is 
> unsuitable in Python like the following:
>
> nums=32
> rows=120
> cols=150
>
> for k in range(0,nums):
>           for i in range(0,rows):
>                      for j in range(0,cols):
>     if float ( R[ k ] [ i ] [ j ] ) == 0.0:
>                val11 [ i ] =0.0
>     else:
>                val11[ i ] [ j ], val22[ i ][ j ] = integrate.quad( 
> lambda x :  F1(x)*F2(x) , 0 , pi)
>
> But, this calculation takes so long time, let's say about  1 hour 
> (theoretically)... Is there any better way to easily and fast 
> calculate the process such as [ F( i ) for i in xlist ] or something 
> like that rather than using for loop?
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/423baf68/attachment.html>

From thoger.emil at gmail.com  Sun Dec 30 12:03:09 2012
From: thoger.emil at gmail.com (=?UTF-8?B?VGjDuGdlciBSaXZlcmEtVGhvcnNlbg==?=)
Date: Sun, 30 Dec 2012 18:03:09 +0100
Subject: [SciPy-User] 3D array problem in Python
In-Reply-To: <1356884214.944243056@f294.mail.ru>
References: <1356838333.805656881@f211.mail.ru> <50E061FB.2090507@gmail.com>
	<1356884214.944243056@f294.mail.ru>
Message-ID: <50E073CD.6090402@gmail.com>

Oops, we should probably reply to the list again;

On 12/30/2012 05:16 PM, Happyman wrote:
> Hi
>
> Thanks for your answer... I also tried to do so unfortunately I could 
> not manage that because of no very good experience in Python...
> The way you propose is really good indeed but how can i do it???
>
> Two huge problems I always encounter in Python are:
> 1) Array with scalars...always!!!!!
> 2) for loop      I can understand Python is script language but how to 
> deal with it???
>

The idea behind NumPy  is exactly that it performs element-wise 
operations on arrays of N dimensions. Loops should be reserved only to 
the cases where an operation on an element depends on the state of other 
elements.
To perform an element-wise operation on two arrays, they need to be of 
the same dimensions (and the same orientation - that is, a size (3, 6, 
3) is not compatible with a size (3, 3, 6) until you've transposed it)

But suppose you have two arrays:

import numpy as np

A = np.array([1., 2., 3., 4., 5., 6., 7., 8., 9., 19., 11., 12.]).reshape((3, 4))

B = np.random.random(A.shape)

C = A + B

D = A**B

But if you change the orientation of one, it goes wrong: C*A.transpose() 
will not work.

And you can perform operations on only elements in an array that meet 
some logical condition, for example:

E = np.zeros_like(A)  # Just to create the array

E[B<0.5] = A[B<0.5]

E[B>0.5] = C[B>0.5]

Or simply:

B[B>5] += 1.

Note, again, tyhat there is *no* looping going on here.

> Especially, when I create some function in which arguments can take 
> one value are messed up with negotiation wit outer coming variables 
> into my function!!
> for example,
> arg1, arg2 are arrays!
> def f( arg1, arg2 ):
>        some process, loops everthing
>    return val
>
I am not sure I understand the last question; what exactly is it that 
goes wrong?

Cheers,
Emil


>
>
>
>
> ???????????, 30 ??????? 2012, 16:47 +01:00 ?? Th?ger Rivera-Thorsen 
> <thoger.emil at gmail.com>:
>
>     Use np.where() or logical indexing (same thing, really) to mask
>     your array, then perform the operations. Let's say your array is
>     called A:
>
>     A[float(A) == 0.0] = 0.0
>
>     A[float(A) != 0.0] = [...etc.]
>
>
>     This, of course, only works if the operation for an entry doesn't
>     depend on other entries in the array; but it should give you a
>     great speed gain.
>
>     Cheers;
>
>     Emil
>
>
>
>     On 12/30/2012 04:32 AM, Happyman wrote:
>>     Hello
>>
>>     I have 3 dimensional array  which I want  to calculate in a huge
>>     process. Everything is working well if I use ordinary way which
>>     is unsuitable in Python like the following:
>>
>>     nums=32
>>     rows=120
>>     cols=150
>>
>>     for k in range(0,nums):
>>               for i in range(0,rows):
>>      for j in range(0,cols):
>>                     if float ( R[ k ] [ i ] [ j ] ) == 0.0:
>>                                val11 [ i ] =0.0
>>                     else:
>>                                val11[ i ] [ j ], val22[ i ][ j ] =
>>     integrate.quad( lambda x :  F1(x)*F2(x) , 0 , pi)
>>
>>     But, this calculation takes so long time, let's say about  1 hour
>>     (theoretically)... Is there any better way to easily and fast
>>     calculate the process such as [ F( i ) for i in xlist ] or
>>     something like that rather than using for loop?
>>
>>
>>
>>
>>     _______________________________________________
>>     SciPy-User mailing list
>>     SciPy-User at scipy.org  <sentmsg?mailto=mailto%3aSciPy%2dUser at scipy.org>
>>     http://mail.scipy.org/mailman/listinfo/scipy-user
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/a04be7a7/attachment.html>

From thoger.emil at gmail.com  Sun Dec 30 12:11:16 2012
From: thoger.emil at gmail.com (=?UTF-8?B?VGjDuGdlciBSaXZlcmEtVGhvcnNlbg==?=)
Date: Sun, 30 Dec 2012 18:11:16 +0100
Subject: [SciPy-User] 3D array problem in Python
In-Reply-To: <1356884214.944243056@f294.mail.ru>
References: <1356838333.805656881@f211.mail.ru> <50E061FB.2090507@gmail.com>
	<1356884214.944243056@f294.mail.ru>
Message-ID: <50E075B4.5020209@gmail.com>

By the way;

I don't know if they are useful to you, but here are some introductory 
slides from a workshop I gave for our undergraduate studemts at 
Stockholm University. The Python part starts at slide 18.

Emil


On 12/30/2012 05:16 PM, Happyman wrote:
> Hi
>
> Thanks for your answer... I also tried to do so unfortunately I could 
> not manage that because of no very good experience in Python...
> The way you propose is really good indeed but how can i do it???
>
> Two huge problems I always encounter in Python are:
> 1) Array with scalars...always!!!!!
> 2) for loop      I can understand Python is script language but how to 
> deal with it???
>
> Especially, when I create some function in which arguments can take 
> one value are messed up with negotiation wit outer coming variables 
> into my function!!
> for example,
> arg1, arg2 are arrays!
> def f( arg1, arg2 ):
>        some process, loops everthing
>    return val
>
>
>
>
>
> ???????????, 30 ??????? 2012, 16:47 +01:00 ?? Th?ger Rivera-Thorsen 
> <thoger.emil at gmail.com>:
>
>     Use np.where() or logical indexing (same thing, really) to mask
>     your array, then perform the operations. Let's say your array is
>     called A:
>
>     A[float(A) == 0.0] = 0.0
>
>     A[float(A) != 0.0] = [...etc.]
>
>
>     This, of course, only works if the operation for an entry doesn't
>     depend on other entries in the array; but it should give you a
>     great speed gain.
>
>     Cheers;
>
>     Emil
>
>
>
>     On 12/30/2012 04:32 AM, Happyman wrote:
>>     Hello
>>
>>     I have 3 dimensional array  which I want  to calculate in a huge
>>     process. Everything is working well if I use ordinary way which
>>     is unsuitable in Python like the following:
>>
>>     nums=32
>>     rows=120
>>     cols=150
>>
>>     for k in range(0,nums):
>>               for i in range(0,rows):
>>      for j in range(0,cols):
>>                     if float ( R[ k ] [ i ] [ j ] ) == 0.0:
>>                                val11 [ i ] =0.0
>>                     else:
>>                                val11[ i ] [ j ], val22[ i ][ j ] =
>>     integrate.quad( lambda x :  F1(x)*F2(x) , 0 , pi)
>>
>>     But, this calculation takes so long time, let's say about  1 hour
>>     (theoretically)... Is there any better way to easily and fast
>>     calculate the process such as [ F( i ) for i in xlist ] or
>>     something like that rather than using for loop?
>>
>>
>>
>>
>>     _______________________________________________
>>     SciPy-User mailing list
>>     SciPy-User at scipy.org  <sentmsg?mailto=mailto%3aSciPy%2dUser at scipy.org>
>>     http://mail.scipy.org/mailman/listinfo/scipy-user
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/69720c08/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Tools-Print-OneOnOne.pdf
Type: application/pdf
Size: 1847695 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/69720c08/attachment.pdf>

From thoger.emil at gmail.com  Sun Dec 30 14:29:42 2012
From: thoger.emil at gmail.com (=?UTF-8?B?VGjDuGdlciBSaXZlcmEtVGhvcnNlbg==?=)
Date: Sun, 30 Dec 2012 20:29:42 +0100
Subject: [SciPy-User] 3D array problem in Python
In-Reply-To: <1356889571.638175686@f257.mail.ru>
References: <1356838333.805656881@f211.mail.ru>
	<1356884214.944243056@f294.mail.ru> <50E073CD.6090402@gmail.com>
	<1356889571.638175686@f257.mail.ru>
Message-ID: <50E09626.5070500@gmail.com>

On 12/30/2012 06:46 PM, Happyman wrote:
> Great thanks Emil
>
> because of being in a new step in python may be I am not realizing the 
> "advantages" of it!!!
>
>
> I am going to try to do so according to your advice hopefully.
>
> What I wanted to explain by showing the function which you did not get 
> is I have two separately independent args (arg1, arg2) which contains 
> arrays, one and 3 dimensional respectively..
>
> arg1=A -> 1-Dimen.
> arg2=B -> 3 Dimen.
>
> def F(arg1, arg2):
>         # this is just example we can put any function or process here!!
>       return array(value)  # as an example
>
> I wanted to get faster result by avoiding loops inside of function 
> here...it seems really (of course for me) confusing...to me. I tried 
> but could not do it!
>

I think the answer depends on the specific operations you want the 
function to do. As an example, you could imagine that A is a (25,) 1D 
array, and B is a (33, 25) 2D array. Remember,m the latter means 33 
rows, 25 columns. Let's say we want to add A to each of the rows of B. 
We could of course loop, but we don't want to do that. Instead we can 
create a (33, 25) array where each row is A, and then add this array to B:

import numpy as np

A = np.random.random(25)
B = np.ones((33, 25))

def F(Arg1, Arg2):

     # We want it to be a 2D array so we can define if it's a row or a column.
     #  Default is that it's a row:
     C = np.array(Arg1, ndmin=2)
     
     print C.shape            # Just to see it
     C =C.repeat(33, axis=0)  # Stack 33 rows on top of each other
     out = B + C              # Now, element-wise addition.
     return out

my_sum = F(A, B)
print my_sum

Hope this helps...
A shortcut to avoid some of these steps is to define A as a row or 
column explicitly:

A = np.random.random((1, 25))  # Or ((25, 1)) if you want it to be a column.


Then you can just A.transpose() or A.T  if you need that.

Try googling "Numpy for matlab users" and "NumPy for IDL users", even if 
you don't use matlab or IDL. They have some pretty good list of basic 
array handling functionality (and NumPy is designed to take you a very 
long way with basic functionality).

/Emil



>
> ???????????, 30 ??????? 2012, 18:03 +01:00 ?? Th?ger 
> Rivera-Thorsen<thoger.emil at gmail.com>:
>
>     Oops, we should probably reply to the list again;
>
>     On 12/30/2012 05:16 PM, Happyman wrote:
>>     Hi
>>
>>     Thanks for your answer... I also tried to do so unfortunately I
>>     could not manage that because of no very good experience in Python...
>>     The way you propose is really good indeed but how can i do it???
>>
>>     Two huge problems I always encounter in Python are:
>>     1) Array with scalars...always!!!!!
>>     2) for loop      I can understand Python is script language but
>>     how to deal with it???
>>
>
>     The idea behind NumPy  is exactly that it performs element-wise
>     operations on arrays of N dimensions. Loops should be reserved
>     only to the cases where an operation on an element depends on the
>     state of other elements.
>     To perform an element-wise operation on two arrays, they need to
>     be of the same dimensions (and the same orientation - that is, a
>     size (3, 6, 3) is not compatible with a size (3, 3, 6) until
>     you've transposed it)
>
>     But suppose you have two arrays:
>
>     import numpy as np
>
>     A = np.array([1., 2., 3., 4., 5., 6., 7., 8., 9., 19., 11., 12.]).reshape((3, 4))
>
>     B = np.random.random(A.shape)
>
>     C = A + B
>
>     D = A**B
>
>     But if you change the orientation of one, it goes wrong:
>     C*A.transpose() will not work.
>
>     And you can perform operations on only elements in an array that
>     meet some logical condition, for example:
>
>     E = np.zeros_like(A)  # Just to create the array
>
>     E[B<0.5] = A[B<0.5]
>
>     E[B>0.5] = C[B>0.5]
>
>     Or simply:
>
>     B[B>5] += 1.
>
>     Note, again, tyhat there is *no* looping going on here.
>
>>     Especially, when I create some function in which arguments can
>>     take one value are messed up with negotiation wit outer coming
>>     variables into my function!!
>>     for example,
>>     arg1, arg2 are arrays!
>>     def f( arg1, arg2 ):
>>            some process, loops everthing
>>        return val
>>
>     I am not sure I understand the last question; what exactly is it
>     that goes wrong?
>
>     Cheers,
>     Emil
>
>
>>
>>
>>
>>
>>     ???????????, 30 ??????? 2012, 16:47 +01:00 ?? Th?ger
>>     Rivera-Thorsen <thoger.emil at gmail.com>
>>     <sentmsg?mailto=mailto%3athoger.emil at gmail.com>:
>>
>>         Use np.where() or logical indexing (same thing, really) to
>>         mask your array, then perform the operations. Let's say your
>>         array is called A:
>>
>>         A[float(A) == 0.0] = 0.0
>>
>>         A[float(A) != 0.0] = [...etc.]
>>
>>
>>         This, of course, only works if the operation for an entry
>>         doesn't depend on other entries in the array; but it should
>>         give you a great speed gain.
>>
>>         Cheers;
>>
>>         Emil
>>
>>
>>
>>         On 12/30/2012 04:32 AM, Happyman wrote:
>>>         Hello
>>>
>>>         I have 3 dimensional array  which I want  to calculate in a
>>>         huge process. Everything is working well if I use ordinary
>>>         way which is unsuitable in Python like the following:
>>>
>>>         nums=32
>>>         rows=120
>>>         cols=150
>>>
>>>         for k in range(0,nums):
>>>         for i in range(0,rows):
>>>                    for j in range(0,cols):
>>>                                     if float ( R[ k ] [ i ] [ j ] )
>>>         == 0.0:
>>>          val11 [ i ] =0.0
>>>                                     else:
>>>          val11[ i ] [ j ], val22[ i ][ j ] = integrate.quad( lambda
>>>         x :  F1(x)*F2(x) , 0 , pi)
>>>
>>>         But, this calculation takes so long time, let's say about  1
>>>         hour (theoretically)... Is there any better way to easily
>>>         and fast calculate the process such as [ F( i ) for i in
>>>         xlist ] or something like that rather than using for loop?
>>>
>>>
>>>
>>>
>>>         _______________________________________________
>>>         SciPy-User mailing list
>>>         SciPy-User at scipy.org
>>>         http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/769763f5/attachment.html>

From amschne at umich.edu  Sun Dec 30 19:16:30 2012
From: amschne at umich.edu (Adam Schneider)
Date: Sun, 30 Dec 2012 19:16:30 -0500
Subject: [SciPy-User] Fwd: scipy.test() fails for 0.11.0 on OS X 10.8.2
In-Reply-To: <mailman.456.1356912113.1036.scipy-dev@scipy.org>
References: <mailman.456.1356912113.1036.scipy-dev@scipy.org>
Message-ID: <CAAiJ8F+vHdVnseAWTAAqg532nD+hXwcOOBqAn2W_Zrt9d+C1Bg@mail.gmail.com>

Hello,

I'm trying to install scipy on my macbook and am getting the following
failure(s) when running scipy.test():

On OS X 10.8.2 (Core i7), scipy.test() for scipy version 0.11.0 fails the
following test:

```
In [2]: import scipy

In [3]: scipy.test()
Running unit tests for scipy
NumPy version 1.8.0.dev-a8c641f
NumPy is installed in /usr/local/lib/python2.7/site-packages/numpy
SciPy version 0.11.0
SciPy is installed in /usr/local/lib/python2.7/site-packages/scipy
Python version 2.7.3 (default, Dec  3 2012, 00:18:53) [GCC 4.2.1 Compatible
Apple Clang 4.1 ((tags/Apple/clang-421.11.66))]
nose version 1.2.1
```
...(see https://gist.github.com/4416094 for full failure log)

```
Ran 5481 tests in 58.156s

FAILED (KNOWNFAIL=13, SKIP=42, errors=48, failures=72)
Out[3]: <nose.result.TextTestResult run=5481 errors=48 failures=72>
```
Here's my compiler info:
```
    build_src
    building py_modules sources
    building library "npymath" sources
    customize Gnu95FCompiler
    Found executable /usr/local/bin/gfortran
    customize Gnu95FCompiler
    customize Gnu95FCompiler using config
    C compiler: cc -fno-strict-aliasing -fno-common -dynamic
-I/usr/local/include -DNDEBUG -g -O3 -Wall -Wstrict-prototypes

    compile options:
'-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/private
-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src
-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core
-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/npymath
-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/multiarray
-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/umath
-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/npysort
-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/include
-I/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/include/python2.7
-c'
```
```
$ cc --version
Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin12.2.1
Thread model: posix

$ gfortran --version
GNU Fortran (GCC) 4.7.2

$ python --version
Python 2.7.3
```

This version of scipy was installed using pip.  What can I do to eliminate
these failures and get scipy working properly?

Thank you for the help,

Adam




On Sun, Dec 30, 2012 at 7:01 PM, <scipy-dev-owner at scipy.org> wrote:

> This is a members-only list.  Your message has been automatically
> rejected, since it came from a non-member's email address.  Please
> make sure to use the email account that you used to join this list.
>
>
>
> ---------- Forwarded message ----------
> From: Adam Schneider <amschne at umich.edu>
> To: scipy-dev at scipy.org
> Cc: Jordan Schneider <jbschne at umich.edu>
> Date: Sun, 30 Dec 2012 19:06:34 -0500
> Subject: scipy.test() fails for 0.11.0 on OS X 10.8.2
> Hello,
>
> I'm trying to install scipy on my macbook and am getting the following
> failure(s) when running scipy.test():
>
> On OS X 10.8.2 (Core i7), scipy.test() for scipy version 0.11.0 fails the
> following test:
>
> ```
> In [2]: import scipy
>
> In [3]: scipy.test()
> Running unit tests for scipy
> NumPy version 1.8.0.dev-a8c641f
> NumPy is installed in /usr/local/lib/python2.7/site-packages/numpy
> SciPy version 0.11.0
> SciPy is installed in /usr/local/lib/python2.7/site-packages/scipy
> Python version 2.7.3 (default, Dec  3 2012, 00:18:53) [GCC 4.2.1
> Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))]
> nose version 1.2.1
> ```
> ...(see https://gist.github.com/4416094 for full failure log)
>
> ```
> Ran 5481 tests in 58.156s
>
> FAILED (KNOWNFAIL=13, SKIP=42, errors=48, failures=72)
> Out[3]: <nose.result.TextTestResult run=5481 errors=48 failures=72>
> ```
> Here's my compiler info:
> ```
>     build_src
>     building py_modules sources
>     building library "npymath" sources
>     customize Gnu95FCompiler
>     Found executable /usr/local/bin/gfortran
>     customize Gnu95FCompiler
>     customize Gnu95FCompiler using config
>     C compiler: cc -fno-strict-aliasing -fno-common -dynamic
> -I/usr/local/include -DNDEBUG -g -O3 -Wall -Wstrict-prototypes
>
>     compile options:
> '-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/private
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/npymath
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/multiarray
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/umath
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/npysort
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/include
> -I/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/include/python2.7
> -c'
> ```
> ```
> $ cc --version
> Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)
> Target: x86_64-apple-darwin12.2.1
> Thread model: posix
>
> $ gfortran --version
> GNU Fortran (GCC) 4.7.2
>
> $ python --version
> Python 2.7.3
> ```
>
> This version of scipy was installed using pip.  What can I do to eliminate
> these failures and get scipy working properly?
>
> Thank you for the help,
>
> Adam
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121230/779d694e/attachment.html>

From ralf.gommers at gmail.com  Mon Dec 31 12:14:50 2012
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 31 Dec 2012 18:14:50 +0100
Subject: [SciPy-User] Fwd: scipy.test() fails for 0.11.0 on OS X 10.8.2
In-Reply-To: <CAAiJ8F+vHdVnseAWTAAqg532nD+hXwcOOBqAn2W_Zrt9d+C1Bg@mail.gmail.com>
References: <mailman.456.1356912113.1036.scipy-dev@scipy.org>
	<CAAiJ8F+vHdVnseAWTAAqg532nD+hXwcOOBqAn2W_Zrt9d+C1Bg@mail.gmail.com>
Message-ID: <CABL7CQjr6jPdxNvS478vk0Ho2b8-tR_8p1pEJBhP3z=4v=YXpQ@mail.gmail.com>

On Mon, Dec 31, 2012 at 1:16 AM, Adam Schneider <amschne at umich.edu> wrote:

> Hello,
>
> I'm trying to install scipy on my macbook and am getting the following
> failure(s) when running scipy.test():
>
> On OS X 10.8.2 (Core i7), scipy.test() for scipy version 0.11.0 fails the
> following test:
>
> ```
> In [2]: import scipy
>
> In [3]: scipy.test()
> Running unit tests for scipy
> NumPy version 1.8.0.dev-a8c641f
> NumPy is installed in /usr/local/lib/python2.7/site-packages/numpy
> SciPy version 0.11.0
> SciPy is installed in /usr/local/lib/python2.7/site-packages/scipy
> Python version 2.7.3 (default, Dec  3 2012, 00:18:53) [GCC 4.2.1
> Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))]
> nose version 1.2.1
> ```
> ...(see https://gist.github.com/4416094 for full failure log)
>
> ```
> Ran 5481 tests in 58.156s
>
> FAILED (KNOWNFAIL=13, SKIP=42, errors=48, failures=72)
> Out[3]: <nose.result.TextTestResult run=5481 errors=48 failures=72>
> ```
> Here's my compiler info:
> ```
>     build_src
>     building py_modules sources
>     building library "npymath" sources
>     customize Gnu95FCompiler
>     Found executable /usr/local/bin/gfortran
>     customize Gnu95FCompiler
>     customize Gnu95FCompiler using config
>     C compiler: cc -fno-strict-aliasing -fno-common -dynamic
> -I/usr/local/include -DNDEBUG -g -O3 -Wall -Wstrict-prototypes
>
>     compile options:
> '-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/private
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/npymath
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/multiarray
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/umath
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/npysort
> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/include
> -I/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/include/python2.7
> -c'
> ```
> ```
> $ cc --version
> Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)
> Target: x86_64-apple-darwin12.2.1
> Thread model: posix
>
> $ gfortran --version
> GNU Fortran (GCC) 4.7.2
>
> $ python --version
> Python 2.7.3
> ```
>
> This version of scipy was installed using pip.  What can I do to eliminate
> these failures and get scipy working properly?
>

The "DeprecationWarning: non-integer scalar index" errors are due to a very
recent change in numpy master for which scipy still has to be updated.
They're not a problem, the code being tested will only stop working with
numpy 1.9.

For the failures there's no good solution on OS X 10.8 except for not
building against the Accelerate Framework but against Netlib BLAS/LAPACK
instead. A description of how to do that can be found at the bottom of
http://projects.scipy.org/scipy/ticket/1737.

Ralf



> Thank you for the help,
>
> Adam
>
>
>
>
> On Sun, Dec 30, 2012 at 7:01 PM, <scipy-dev-owner at scipy.org> wrote:
>
>> This is a members-only list.  Your message has been automatically
>> rejected, since it came from a non-member's email address.  Please
>> make sure to use the email account that you used to join this list.
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Adam Schneider <amschne at umich.edu>
>> To: scipy-dev at scipy.org
>> Cc: Jordan Schneider <jbschne at umich.edu>
>> Date: Sun, 30 Dec 2012 19:06:34 -0500
>> Subject: scipy.test() fails for 0.11.0 on OS X 10.8.2
>> Hello,
>>
>> I'm trying to install scipy on my macbook and am getting the following
>> failure(s) when running scipy.test():
>>
>> On OS X 10.8.2 (Core i7), scipy.test() for scipy version 0.11.0 fails the
>> following test:
>>
>> ```
>> In [2]: import scipy
>>
>> In [3]: scipy.test()
>> Running unit tests for scipy
>> NumPy version 1.8.0.dev-a8c641f
>> NumPy is installed in /usr/local/lib/python2.7/site-packages/numpy
>> SciPy version 0.11.0
>> SciPy is installed in /usr/local/lib/python2.7/site-packages/scipy
>> Python version 2.7.3 (default, Dec  3 2012, 00:18:53) [GCC 4.2.1
>> Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))]
>> nose version 1.2.1
>> ```
>> ...(see https://gist.github.com/4416094 for full failure log)
>>
>> ```
>> Ran 5481 tests in 58.156s
>>
>> FAILED (KNOWNFAIL=13, SKIP=42, errors=48, failures=72)
>> Out[3]: <nose.result.TextTestResult run=5481 errors=48 failures=72>
>> ```
>> Here's my compiler info:
>> ```
>>     build_src
>>     building py_modules sources
>>     building library "npymath" sources
>>     customize Gnu95FCompiler
>>     Found executable /usr/local/bin/gfortran
>>     customize Gnu95FCompiler
>>     customize Gnu95FCompiler using config
>>     C compiler: cc -fno-strict-aliasing -fno-common -dynamic
>> -I/usr/local/include -DNDEBUG -g -O3 -Wall -Wstrict-prototypes
>>
>>     compile options:
>> '-I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/private
>> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src
>> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core
>> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/npymath
>> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/multiarray
>> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/umath
>> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/src/npysort
>> -I/private/var/folders/sz/fc0nh0x16fv3ytl31cff6tq80000gn/T/pip-build/numpy-dev/numpy/core/include
>> -I/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/include/python2.7
>> -c'
>> ```
>> ```
>> $ cc --version
>> Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM
>> 3.1svn)
>> Target: x86_64-apple-darwin12.2.1
>> Thread model: posix
>>
>> $ gfortran --version
>> GNU Fortran (GCC) 4.7.2
>>
>> $ python --version
>> Python 2.7.3
>> ```
>>
>> This version of scipy was installed using pip.  What can I do to
>> eliminate these failures and get scipy working properly?
>>
>> Thank you for the help,
>>
>> Adam
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20121231/212b0862/attachment.html>