[SciPy-dev] code for incremental least squares

[josef.pktd at gmail.com]

On Wed, Feb 17, 2010 at 12:31 AM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Feb 16, 2010 at 8:51 PM,  <josef.pktd at gmail.com> wrote:
>> I found this comment in incremental_ls
>>
>>        # R and Scipy disagree by about 3 orders of magnitude here:
>>        #  pf(4704.7767809675142416, 4, 12, lower.tail=FALSE) == 4.68-19
>>        #  1 - f.cdf(4704.7767809675142416, 4, 12) == 1.11-16
>>        # which I guess is most likely Scipy having limited resolution in the
>>        # tails.
>>
>>>>> from scipy import stats
>>>>> stats.f.sf(4704.7767809675142416, 4, 12)
>> 4.6848221938640787e-019
>>
>> some of our tails are also pretty good
>
> Oh, awesome, I'd missed .sf() -- thanks for the pointer! It's
> aesthetically annoying to plot p-values on a graph and always have
> weird discontinuities where they underflow...

I'm glad to help, (especially if I can show off things that work)

There are still some strange things left in the range smaller than
1e-8, but every once in a while we catch one and stats.distributions
is slowly improving.
Filing a ticket for these cases is useful, sometimes it's easy to
improve when we find the loss in precision, sometimes it's buried in
the c or fortran code in scipy.special and we have to live with 1e-8
or whichever precision we get. (Although it won't make much difference
to the statistical significance of any test.)

Josef

>
> -- Nathaniel
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>