[Python-Dev] Information about how cpython in benchmarked

[Jesse Noller]

On Tue, Mar 29, 2011 at 7:00 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Tue, Mar 29, 2011 at 8:01 PM, Tennessee Leeuwenburg
> <tleeuwenburg at gmail.com> wrote:
>> PyPy maintains http://speed.pypy.org/, which provides very clear information
>> about the relative performance of PyPy trunk against some version of cpython
>> (presumably 2.6 or 2.7). I'm not aware of a similar site for cpython, but
>> that could easily just be my ignorance speaking.
>> My interest is that I'm looking at building a benchmarking solution at work.
>> and I can't think of a better way to build something good and general than
>> to try and write something that could potentially be released as open source
>> and be useful to others. As such I thought that benchmarking cpython would
>> be a great use case, but I want to find out as much as I can about how
>> people currently go about benchmarking Python. Initially I'm just looking at
>> CPU profiling since it's easiest.
>
> One of the points coming out of the VM summit at Pycon is actually
> that we want to create a shared benchmarking site for CPython, PyPy,
> Jython, IronPython (and possibly Stackless) under the python.org
> banner (either speed.python.org, or possibly performance.python.org,
> since we want to do memory profiling as well).
>
> speed.pypy.org will be the reference site for this, but Maciej
> indicated at the VM summit that the code that runs that site needs
> some improvements before it will really be up to the task of
> effectively benchmarking multiple targets.
>
> So, according to http://speed.pypy.org/about/, the place to start with
> your benchmarking system would probably be
> https://github.com/tobami/codespeed.
>
> Cheers,
> Nick.

Essentially echoing what nick said. I'm currently working on getting
the HW for this together.