[SciPy-Dev] scipy.linalg asarray_chkfinite

[Christopher Jordan-Squire]

On Wed, Jul 27, 2011 at 9:04 AM, Bruce Southey <bsouthey at gmail.com> wrote:

> On 07/25/2011 05:20 PM, Travis Oliphant wrote:
> > Probably not,  but perhaps their should be.  If I recall correctly, non
> finite values give some bad behavior --- sometimes aborting the process to
> most of the routines and so the default is to check.
> >
> > Travis
> >
> > --
> > (mobile phone of)
> > Travis Oliphant
> > Enthought, Inc.
> > http://www.enthought.com
> >
> > On Jul 25, 2011, at 4:16 PM, Christopher Jordan-Squire<cjordan1 at uw.edu>
>  wrote:
> >
> >> Hi,
> >>
> >> I've been trying to speed up a linear least squares solve in scipy and
> running into some problems. I was hoping someone could help.
> >>
> >> I've been using scipy.linalg.lstsq. But it's slower than I want.
> Profiling showed A LOT of time in lstsq was spent just checking that the
> entries of my input matrices were finite using asarray_chkfinite. (Typically
> around 1/3 of the lstsq function call seems to be checking that.) But I know
> that the matrices I'm passing in have finite values.
> >>
> >> Also, lstsq uses an svd-based solution method instead of the faster qr
> solution method. I had hoped I could write my own qr-based least squares
> solver to circumvent both problems, but I found the asarray_chkfinite calls
> seem to be in most scipy.linalg functions.
> >>
> >> Is there a way around these time consuming checks when calling
> scipy.linalg routines?
> >>
> >> Thanks,
> >> Chris Jordan-Squire
> Without knowing what you actually need to solve, have you tried numpy's
> version of lstsq (in numpy.linalg)?
> Alternatively, since you know your inputs, create a stripped down
> version of scipy's lstsq function as there are very few lines you would
> need from it.
>
> There is a thread on QR:
> http://mail.scipy.org/pipermail/scipy-dev/2011-July/016366.html
>
> The more general issue is that scipy can not automatically assume that
> the input is correct. Note that the scipy.linalg.lstsqr docstring is not
> completely correct since it accept array-like inputs not just arrays
> (due to the fact that numpy.asarray_chkfinite accepts array-like
> inputs). So if you can find a faster way to do this then please file a
> ticket so the community can benefit from it and it not get lost in the
> list.
>
>
Thanks for the link. It's instructive, but it looks like they're mostly
talking about a different qr algorithm.

I hadn't thought of using the numpy version, but I prefer to use the scipy
version to make sure the functions are using the correct lapack.

Later this week I'm going to add a flag to several of the scipy.linalg
functions to allow the caller to disable the checking if they desire. This
will replace the asarray_chkfinite with asarray, which will also take
array-likes, but will pass pre-existing numpy arrays straight through. This
should speed things up a lot if you're passing in a numpy array that's
already been verified as having finite entries.

I experimented a bit yesterday, and it appears that the following is true.
If you pass a numpy array with a NaN in it to lstsq (after removing the
chkfinite command), then it fails and gives back arrays full of nan's. But
if you pass a numpy array with inf's in it, then it hangs. With qr, if you
pass a NaN you again get an array of NaN's back, but if you pass in an inf
you get something back (i.e. no error), but it has NaN's and Inf's in the
matrix. So the user can check for and catch that error.

-Chris Jordan-Squire










> Bruce
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20110727/3e3dbe74/attachment.html>