[Speed] Adapting pypy benchmark suite

[Maciej Fijalkowski]

On Thu, Apr 14, 2016 at 6:51 PM, Brett Cannon <brett at python.org> wrote:
>
>
> On Wed, 13 Apr 2016 at 13:33 Zachary Ware <zachary.ware+pydev at gmail.com>
> wrote:
>>
>> On Wed, Apr 13, 2016 at 1:57 PM, Maciej Fijalkowski <fijall at gmail.com>
>> wrote:
>> > Hi
>> >
>> > I have a radical idea: to take a pypy benchmark suite, update the
>> > libraries to newer ones and replace python benchmarks with that. The
>> > main reason being that pypy has a much better coverage of things that
>> > are not microbenchmarks, the list (in json):
>> >
>> > http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/
>> >
>> > Which is much more extensive than this:
>> >
>> > https://hg.python.org/benchmarks/file/tip/performance
>> >
>> > I'm willing to put *some* effort, what do people think?
>>
>> I'm in favor.  My support has two conditions, though:
>>
>> 1) at least a majority of the benchmarks must be Python3 compatible.
>> Preferably 2/3 compatible, but I assume all of the PyPy benchmarks are
>> 2 compatible anyway.
>
>
> Agreed (although I don't care about the 2/3 compatibility, just the 3 compat
> ;) .
>
>>
>>
>> 2) and there should be an easy way to run the benchmarks against
>> exactly 1 interpreter (for use with speed.python.org).  I initially
>> tried to set up speed.python.org using the PyPy benchmarks, but
>> quickly ran into issues with trying to use 'nullpython.py' as the
>> baseline Python.  When I switched to using h.p.o/benchmarks, I added
>> the '--raw' flag to perf.py which allows the benchmarks to be run on
>> one interpreter instead of two.  It was just a quick hack, though; I
>> have no problems with that feature completely changing (even invoking
>> it a different way is ok), so long as it exists.
>>
>> This project could probably start its life as
>> github.com/python/benchmarks and save us from having to migrate
>> h.p.o/benchmarks to GitHub.
>
>
> Yep, I'm willing to postpone moving the benchmarks repo from hg.python.org
> if works starts on this idea and then not move the old repo at all if this
> succeeds. We could then make the people who care about the benchmarks the
> maintainers of the new repository and contribute to it directly (which means
> interested people from CPython, PyPy, Pyston, IronPython, and Jython). That
> way we all get exposure to everyone's benchmarks and there's no more
> benchmark fragmentation because some people disagree with another's approach
> (i.e. no more benchmark silos among the Python implementations).
>
> And just because we're talking a new repo, would it be worth considering
> relying on pip to grab libraries instead of embedding them in the
> repository, hence shrinking the overall size of the repo?
>

Both make sense - to benchmark against the latest lib X and to
benchmark against a pinned lib X