[Speed] Cython's view on a common benchmark suite (was: Re: Buildbot Status)

[Maciej Fijalkowski]

On Thu, Feb 2, 2012 at 1:09 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Thu, Feb 2, 2012 at 10:21 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Brett Cannon, 01.02.2012 18:25:
>>> to prevent this from either ending up in a dead-end because of this, we
>>> need to first decide where the canonical set of Python VM benchmarks are
>>> going to live. I say hg.python.org/benchmarks for two reasons. One is that
>>> Antoine has already done work there to port some of the benchmarks so there
>>> is at least some there that are ready to be  run under Python 3 (and the
>>> tooling is in place to create separate Python 2 and Python 3 benchmark
>>> suites). Two, this can be a test of having the various VM contributors work
>>> out of hg.python.org if we are ever going to break the stdlib out for
>>> shared development. At worst we can simply take the changes made at
>>> pypy/benchmarks that apply to just the unladen benchmarks that exists, and
>>> at best merge the two sets (manually) into one benchmark suite so PyPy
>>> doesn't lose anything for Python 2 measurements that they have written and
>>> CPython doesn't lose any of its Python 3 benchmarks that it has created.
>>>
>>> How does that sound?
>>
>> +1
>>
>> FWIW, Cython currently uses both benchmark suites, that of PyPy (in Py2.7)
>> and that of hg.python.org (in Py2.7 and 3.3), but without codespeed
>> integration and also without a dedicated server for benchmark runs. So the
>> results are unfortunately not accurate enough to spot minor changes even
>> over time.
>>
>> https://sage.math.washington.edu:8091/hudson/view/bench/
>>
>> We would like to join in on speed.python.org, once it's clear how the
>> benchmarks will be run and how the data uploads work and all that. It
>> already proved a bit tricky to get Cython integrated with the benchmark
>> runner on our side, and I'm planning to rewrite that integration at some
>> point, but it should already be doable to get "something" to work now.
>
> Can you come up with a script that does "cython <a python program>"?
> that would simplify a lot
>
>>
>> I should also note that we don't currently support the whole benchmark
>> suite, so there must be a way to record individual benchmark results even
>> in the face of failures in other benchmarks. Basically, speed.python.org
>> would be useless for us if a failure in a single benchmark left us without
>> any performance data at all, because it will still take us some time to get
>> to 100% compliance and we would like to know if anything on that road has a
>> performance impact. Currently, we apply a short patch that adds a
>> try-except to the benchmark runner's main loop before starting the
>> measurements, because otherwise it would just bail out completely on a
>> single failure. Oh, and we also patch the benchmarks to remove references
>> to __file__ because of CPython issue 13429, although we may be able to work
>> around that at some point, specifically when doing on-the-fly compilation
>> during imports.
>
> I think it's fine to mark certain benchmarks not to be runnable under
> certain platforms. For example it's not like jython will run twisted
> stuff.
>
>>
>> http://bugs.python.org/issue13429
>>
>> Also note that benchmarks that only test C implemented stdlib modules (re,
>> pickle, json) are useless for Cython because they would only end up timing
>> the exact same code as for plain CPython.
>>
>> Another test that is useless for us is the "mako" benchmark, because most
>> of what it does is to run generated code. There is currently no way for
>> Cython to hook into that, so we're out of the game here.
>
> Well, if you want cython to be considered python I think this is a
> pretty crucial feature no?
>
>>
>> We also don't care about program startup tests, obviously, because we know
>> that Cython's compiler overhead plus an optimising gcc run will render them
>> meaningless anyway. I like the fact that there's still an old hg_startup
>> timing result lingering around from the time before I disabled that test,
>> telling us that Cython runs it 99.68% slower than CPython. Got to beat
>> that. 8-)
>
> That's probably okish.

Stefan, can you please not cross-post between mailing lists? Not
everyone is subscribed and people reading would get a confusing
half-of-the-world view.

Cheers,
fijal