[Cython] speed.pypy.org

[Stefan Behnel]

Vitja Makarov, 30.04.2011 08:08:
> Can I run this tests at home?

Don't expect it to come shrink wrapped. It took me a while to get the setup 
running on Hudson/Jenkins, and the scripts that do it aren't completely 
trivial. You can take a look at the config of the job that runs them:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/

I'm currently using the PyPy test suite at

https://bitbucket.org/pypy/benchmarks/

instead of the future CPython one at

http://hg.python.org/benchmarks/

simply because the PyPy suite has more benchmarks (that work for us ;).

The major problem with the test suite is that the unladen swallow people 
who wrote the test runner somehow got the paranoid idea that they had to 
control which environment variables were set during the run, so they chose 
to drop the entire environment and use their own one. They clearly never 
tried to run distutils in a benchmark, which requires some intimate 
knowledge about the environment it runs in.

So you basically need a wrapper around cythonrun that properly sets up the 
environment for you. Take a look at what I did in the build job, it 
generates a script that sets up the environment, and then passes that into 
the benchmark runner.

You also need a "usercustomize.py" script that installs pyximport (but only 
for the Cython runs, not for plain CPython!), so that it's not only the 
benchmark script that gets compiled but also its dependencies. I'm 
currently blacklisting the stdlib modules ['threading', 'socket', 
'gettext', 'locale'] in pyximport.PyxImporter.blocked_modules when I run it 
locally, simply because they don't work but have a substantial impact on 
the running code.

The basic command I'm using to run the suite is:

python runner.py -a ",-Xauto_cpdef=True" \
     -p ./cythonrun.sh --baseline=/path/to/python \
     --fast -o results.json

The CPython test runner is a tad better here, because it allows you to 
provide at least a whitelist of environment variables that are being passed 
through. I'm currently trying to set up a second benchmark job to run the 
(smaller) CPython benchmark suite in Py3 mode. The command to run that 
suite is more like

python perf.py -a ',-Xauto_cpdef=true' \
     --inherit_env=PYTHONPATH,PYTHONHOME,PYTHON,CFLAGS,OPT,PATH \
     /path/to/python ./cythonrun.sh \
    --fast

and it doesn't seem to have json output. (Seriously, I can't believe they 
even broke "PATH" in their runner script!).

There is also a GSoC project that aims to fix up the benchmark suite for 
the different Python implementations in order to set up a 
"speed.python.org" site that compares them. We should stay involved in that.

Stefan