[pypy-dev] Update on MyHDL benchmarks with pypy

[Bengt Richter]

On 06/16/2012 09:57 PM Jan Decaluwe wrote:
> On 06/16/2012 05:20 PM, Maciej Fijalkowski wrote:
>
>> Have a look at our current benchmarks. They're on
>> bitbucket.org/pypy/benchmarks<http://bitbucket.org/pypy/benchmarks>.
>> Look at the format. Ideally they would execute a few different kinds
>> of examples (say 5) each running for about a second?
>
> At only a second for a benchmark, how do you account for
> the JIT warming-up phase?
>
> In MyHDL simulations, I have found the JIT warming-up phase
> to be quite significant (which is OK, because I get such
> good results afterwards).
>
> I run the benchmarks for around 1 minute to remove
> the influence of the JIT warming-up phase from the speed-up
> factor. Probably you run the benchmarks on a much faster
> machine than my i3 laptop, but still.
>
Re faster and slower machines, I wonder if it wouldn't be interesting
to have an option to report results in gigabogoinst instead of,
or in addition to, seconds, (by multiplying seconds by bogogips (== bogomips/1000 ;-)), e.g.,

 >>> bogogips=(lambda b=[float(line.split(':')[1].strip()) for line in open('/proc/cpuinfo') if 
line.startswith('bogomips')]:sum(b)/len(b))()/1000.0
 >>> bogogips
3.99242
would be the factor on my laptop, averaging the bogomips of all (two) CPUs.

If the jit processing had optional callback hooks returning ('<hook id>', time.time()) that
a detailed benchmarker could use to see what activities were taking what time, that would be cool too.
Maybe you already have something like that?

Regards,
Bengt Richter