[Speed] Buildbot Status

[Brett Cannon]

On Wed, Feb 1, 2012 at 06:33, Maciej Fijalkowski <fijall at gmail.com> wrote:

> On Wed, Feb 1, 2012 at 1:24 PM, Jesse Noller <jnoller at gmail.com> wrote:
> >
> >
> > On Feb 1, 2012, at 4:43 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> >
> >>>
> >>> I think pickle was mostly for unladen's pickle performance patches
> (trying
> >>> saying that three times fast =), so I don't really care about that one.
> >>
> >> This is up for discussion whether pickle's performance matters or not
> >> (we have it disabled for example, but we might reenable it one day)
> >>
> >>>
> >>> Would it make sense to change the pypy repo to make the unladen_swallow
> >>> directory an external repo from hg.python.org/benchmarks? Because as
> it
> >>> stands right now there are two mako benchmarks that are not identical.
> >>> Otherwise we should talk at PyCon and figure this all out before we
> end up
> >>> with two divergent benchmark suites that are being independently
> maintained
> >>> (since we are all going to be running the same benchmarks on
> >>> speed.python.org).
> >>
> >> No, I think it's a bad idea. First benchmarks should not change. It's
> >> fine to have a py3k benchmark next to py2 one, but we have 0 checkins
> >> to US benchmarks once we imported them.
> >>
> >> Second, I find some of US benchmarks (those with hand unrolled loops,
> >> like json from the newer one) complete nonsense. If it does make any
> >> sense, it makes it only for cpython so we have no interest in having
> >> those benchmarks at all. If someone unrolls loops by hand and have a
> >> performance problem on pypy it's his own problem :)
> >
> > Great attitude for a shared, common set of benchmarks. Remind me not to
> provide additional resources.
>
> Jesse, do you really write code like that:
>
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(DICT)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>            json.dumps(TUPLE)
>
> sorry for the long paste, but this is how the benchmark looks like. I
> would like to have a common set of benchmarks, but a *reasonable* one.
> Don't complain if I call it nonsense.
>

I think Jesse's point has nothing to do with thinking the unladen
benchmarks are perfect as-is and everything to do with divergence; instead
of diverging from the unladen benchmarks it would have been nicer to simply
fix them instead of having PyPy containing some changes that are legitimate
but never added back to the unladen benchmark repo that the work started
from. We have revision history so there is no need to keep a pristine
version anywhere if that was the thinking. Otherwise I'm going to assume
it's because of unladen being on svn back in the day or some historical
reason.

So, to prevent this from either ending up in a dead-end because of this, we
need to first decide where the canonical set of Python VM benchmarks are
going to live. I say hg.python.org/benchmarks for two reasons. One is that
Antoine has already done work there to port some of the benchmarks so there
is at least some there that are ready to be  run under Python 3 (and the
tooling is in place to create separate Python 2 and Python 3 benchmark
suites). Two, this can be a test of having the various VM contributors work
out of hg.python.org if we are ever going to break the stdlib out for
shared development. At worst we can simply take the changes made at
pypy/benchmarks that apply to just the unladen benchmarks that exists, and
at best merge the two sets (manually) into one benchmark suite so PyPy
doesn't lose anything for Python 2 measurements that they have written and
CPython doesn't lose any of its Python 3 benchmarks that it has created.

How does that sound?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20120201/db4d217e/attachment.html>