[pypy-dev] benchmarking pypy with pybench

[Carl Friedrich Bolz]

Hi Jonathan!

Thanks for doing this, we used pybench a while ago but didn't pursue if
further. This is indeed very interesting!

Jonathan Doda wrote:
> - From what I understand, neither pystone nor richards are very
> representative of idiomatic python, but pybench is (as much as is
> possible for a synthetic benchmark, in any case). So, I thought it would
> be interesting to see how pypy compared to cpython using pybench.
>
> Pybench 2.0 was used, downloaded from the python.org svn repo. Pybench
> reports details of the machine and python installation using the various
> functions in the platform module, but these functions don't appear to
> work in pypy, so the reporting was disabled. The UnicodeProperties test
> also had to be disabled due to pypy not having a unicodedata module. All
> other tests ran successfully, and pybench was otherwise unmodified.

PyPy has a unicodedata module, it is not compiled in by default, though.
It's not clear to me why the platform module does not work, though.

> pypy revision 32952 was used, but pypy was translated using the default
> options, so I imagine these results could be improved on.
>
> The tests are run ten times, the first group of three columns shows the
> best times, and the second group shows the average times. Within each
> group, the first column is the pypy time, the second the python 2.4.3
> time, and the third is the ratio.
>
> I hope these results are of interest, and if anyone has any advice on
> how to improve the test setup, or translation time options I should try,
> please let me know.

You could try to pass in some translation-options that try to optimize
common string operations and small integers:

python translate.py targetpypystandalone.py --objspace-std-withsmallint
--objspace-std-withstrslice --objspace-std-withstrjoin

Other interesting options are --objspace-std-withstrdict (to enable
optimizations for dictionaries with string keys),
--usemodules=unicodedata (to enable the unicode data module). The
options are mutually compatible, so you can do a super-build with all of
them (which unfortunately increases executable size quite a bit).

Cheers,

Carl Friedrich Bolz