[pypy-dev] speed.pypy.org launched

[Miquel Torres]

The paper is right, and the unladen swallow runner does the right thing.

What I meant was: use the statistically right method (like we are
doing now!), but don't show deviation bars if the deviation is
acceptable. Check after the run whether the deviation is not
"acceptable". If it isn't, rerun later, check that nothing in the
background is affecting performance, reevaluate reproducibility of the
given benchmark, etc.

But it doesn't change the fact that speed could save the deviation
data for later use.

What do you think?
Miquel

2010/2/26 Antonio Cuni <anto.cuni at gmail.com>:
> Carl Friedrich Bolz wrote:
>
>> I disagree with this, please read the following paper:
>>
>> http://buytaert.net/files/oopsla07-georges.pdf
>
> +1 for this paper. I was about to link it as well, but cf has been faster.
> I looked at the unladen swallow runner when I was writing my thesis, and I can
> confirm that it does the "statistically right" thing as described in the paper
> to compute error bars.
>
> ciao,
> Anto
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>