[SciPy-Dev] Slow moment function in scipy.stats

[Ralf Gommers]

On Tue, Feb 24, 2015 at 12:42 PM, stefan <stefan.peterson at rubico.com> wrote:

>  Hi Ralf and Julian
>
> I have created a pull request (I hope, very limited experience with git and github). After doing so, I saw Julian's post. Indeed, pow is very accurate. I would argue that for all realistic uses of the moment function, the inaccuracies will be insignificant, but this is certainly up for discussion.
>
> I would agree - if you have to care about precision of some ulps for
statistical functions like these then you are very likely using the wrong
approach.

BR,
> Stefan
>
> On 02/24/2015 08:17 AM, Ralf Gommers wrote:
> >* Hi Stefan,
> *
> > > >* On Thu, Feb 19, 2015 at 12:55 PM, stefan <stefan.peterson at rubico.com <http://mail.scipy.org/mailman/listinfo/scipy-dev>
> *>* <mailto:stefan.peterson at rubico.com <http://mail.scipy.org/mailman/listinfo/scipy-dev>>> wrote:
> *> >*     Hello list,
> *> >*     First time poster here. Anyway, some time ago I noticed that the
> *>*     scipy skewness function was a major bottleneck in an algorithm of
> *>*     mine. Back then, I typed up my own replacement and thought no more
> *>*     about it. Today, for some unknown reason, I decided to dig a little
> *>*     deeper in this and found the major culprit to be the way moments are
> *>*     computed, specifically the use of np.power.
> *> > >* np.power is indeed slow, see for explanations:
> *>* http://stackoverflow.com/questions/25254541/why-is-numpy-power-60x-slower-than-in-lining <http://stackoverflow.com/questions/25254541/why-is-numpy-power-60x-slower-than-in-lining>
> *>* http://stackoverflow.com/questions/26770996/why-is-numpy-power-slower-for-integer-exponents <http://stackoverflow.com/questions/26770996/why-is-numpy-power-slower-for-integer-exponents>
> *>  > >* I'd say that replacing one call to np.power with 6 lines of code to
> *>* achieve a ~10x speedup is a good tradeoff. Pull request is welcome:)
> *>
> its faster but also less accurate, the reason pow is so slow is that it
> has an accuracy of 0.5 ulp regardless of input, which a multiplication
> does not.
> But the argument can certainly be made that this is irrelevant
> especially as the moment computation performed here is numerically not
> that stable in the first place (uncorrected loss of significance in the
> subtraction, mean has an error in the order of the array size ...)
>
> maybe numpy should have a special integer case in power for such situations?
>
> Probably not very high prio, but it would be nice to have that.

Ralf



>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20150225/f87a58a9/attachment.html>