[Speed] New benchmark suite for Python

Victor Stinner victor.stinner at gmail.com
Wed Aug 17 20:37:11 EDT 2016


Hi,

After a few months of work, I created a "new" benchmark suite for Python:

   https://github.com/python/benchmarks

It's based on:
   https://hg.python.org/sandbox/benchmarks_perf
which is my fork the CPython benchmark suite:
   https://hg.python.org/benchmarks
which is based on Unladen Swallow's benchmark suite (if I understood correctly).

Major differences:

* Use the perf module to run benchmarks in multiple processes and
store results as JSON
* Create virtual environments using requirements.txt to download
dependencies from PyPI (rather than using old copies of libraries)
* Many libraries have been upgraded: see requirements.txt

The project works on Python 2 and Python 3 (I tested 2.7 and 3.6).

Known regressions:

* Memory tracking is broken
* run_compare command is currently broken: use run (store result into
a file) + compare manually
* Some benchmarks have been removed: rietveld, spitfire (not on PyPI),
pystone, gcbench, tuple_gc_hell
* I only tested Linux, I expect issues on Windows. (I didn't try my
perf module on Windows yet.)

I already allowed all Python core developers to push to the GitHub
project. We can create a new "benchmarks" (or "Performance" maybe?)
team if we want to allow more contributors who are not core
developers.

PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to
take a look at my project and test it ;-) Tell me what is broken, what
is missing, and I will try to help you to move your project to this
new benchmark suite!

As requested (suggested?) by Brett Canon, the Git repository has no
history, it only contains 1 commit! I'm really sorry of loosing all
the history and all authors, but it allows to start with a much
smaller repository: around 2 MB. The current benchmark repository is
more around 200 MB!

TODO:

* continue to upgrade libraries in requirements.txt. I failed to
upgrade Django to 1.10, it complains about a missing template engine
config setting.
* convert more code to the perf module, like "startup" tests
* run benchmarks and analyze results ;-)
* write more documentation explaining how to run reliable benchmarks
* ...

Victor


More information about the Speed mailing list