[Speed] Buildbot Status

[Mark Shannon]

Maciej Fijalkowski wrote:
> On Wed, Feb 1, 2012 at 1:24 PM, Jesse Noller <jnoller at gmail.com> wrote:
>>
>> On Feb 1, 2012, at 4:43 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>
>>>> I think pickle was mostly for unladen's pickle performance patches (trying
>>>> saying that three times fast =), so I don't really care about that one.
>>> This is up for discussion whether pickle's performance matters or not
>>> (we have it disabled for example, but we might reenable it one day)
>>>
>>>> Would it make sense to change the pypy repo to make the unladen_swallow
>>>> directory an external repo from hg.python.org/benchmarks? Because as it
>>>> stands right now there are two mako benchmarks that are not identical.
>>>> Otherwise we should talk at PyCon and figure this all out before we end up
>>>> with two divergent benchmark suites that are being independently maintained
>>>> (since we are all going to be running the same benchmarks on
>>>> speed.python.org).
>>> No, I think it's a bad idea. First benchmarks should not change. It's
>>> fine to have a py3k benchmark next to py2 one, but we have 0 checkins
>>> to US benchmarks once we imported them.
>>>
>>> Second, I find some of US benchmarks (those with hand unrolled loops,
>>> like json from the newer one) complete nonsense. If it does make any
>>> sense, it makes it only for cpython so we have no interest in having
>>> those benchmarks at all. If someone unrolls loops by hand and have a
>>> performance problem on pypy it's his own problem :)

Doesn't PyPy have a loop re-rolling optimisation? :)

>> Great attitude for a shared, common set of benchmarks. Remind me not to provide additional resources.
> 
> Jesse, do you really write code like that:
> 
>             json.dumps(DICT)

[snip - Manual loop unrolling (1970s style) :( ]

>             json.dumps(TUPLE)
> 

I think the PyPy benchmarks are better than the US ones,
some of which may have been aimed at a few particular cases
where Google wanted to speed up CPython.

Try running the json or pickle benchmarks with a profiler
(oprofile or similar)
and you will see that they spend most of their time in a single
C module, not much of a benchmark for testing the VM.

It might seem that the PyPy folks were cherry picking,
but I don't think that is the case.

Cheers,
Mark.