[pypy-dev] speed.pypy.org launched

[Miquel Torres]

You may also consider that a benchmark that varies greatly between
runs may be a flawed benchmark.

I think it should be considered, but only on the running side, and act
accordingly (too high a deviation: discard run, reconsider benchmark,
reconsider environment or whatever).

Miquel


2010/2/26 Stefan Behnel <stefan_ml at behnel.de>:
> Carl Friedrich Bolz, 25.02.2010 18:38:
>> On 02/25/2010 04:10 PM, Miquel Torres wrote:
>>> As some of you already know, there is a new performance site online:
>>> speed.pypy.org.
>> [...]
>>
>> I'm quite impressed, this is very cool work and a good improvement over
>> the current plots. Thanks for doing this!
>>
>> One thing that I would really find useful are error bars in the timeline
>> view. This would help us judge whether up-and-down movements are within
>> the margin of randomness, or whether it is a real effect. I don't know
>> how annoying they are to implement though, no clue whether the plot lib
>> supports that. There should be enough information about errors in the
>> json files, as far as I remember.
>
> The standard deviation is commonly considered meaningless for benchmark
> results. All that really matters is the fastest run, everything else is
> just fog.
>
> Read the docs on timeit, for example.
>
> Stefan
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>