[Speed] Cython's view on a common benchmark suite

[Alex Gaynor]

On Thu, Feb 2, 2012 at 12:06 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Thu, Feb 2, 2012 at 08:42, Maciej Fijalkowski <fijall at gmail.com> wrote:
>
>> On Thu, Feb 2, 2012 at 3:40 PM, Stefan Behnel <stefan_ml at behnel.de>
>> wrote:
>> > Maciej Fijalkowski, 02.02.2012 14:35:
>> >>> Oh, we have that feature, it's called CPython. The thing is that
>> Cython
>> >>> doesn't get to see the generated sources, so it won't compile them and
>> >>> instead, CPython ends up executing the code at normal interpreted
>> speed. So
>> >>> there's nothing gained by running the benchmark at all. And even if we
>> >>> found a way to hook into this machinery, I doubt that the static
>> compiler
>> >>> overhead would make this any useful. The whole purpose of generating
>> code
>> >>> is that it likely will not look the same the next time you do it
>> (well,
>> >>> outside of benchmarks, that is), so even a cache is unlikely to help
>> much
>> >>> for real code. It's like PyPy running code in interpreting mode
>> before it
>> >>> gets compiled, except that Cython will never compile this code, even
>> if it
>> >>> turns out to be worth it.
>> >>>
>> >>> Personally, I rather consider it a feature that users can employ
>> exec()
>> >>> from their Cython code to run code in plain CPython (for whatever
>> reason).
>> >>
>> >> Yes, ok, but I believe this should mean "Cython does not give speedups
>> >> on this benchmark" and not "we should modify the benchmark".
>> >
>> > Oh, I hadn't suggested to modify it. I was merely stating (as part of a
>> > longer list) that it's of no use specifically to Cython. I.e., if
>> there's
>> > something to gain from having the benchmark runs take less time by
>> > disabling benchmarks for specific runtimes, it's one of the candidates
>> on
>> > our side.
>> >
>> > Stefan
>>
>> Oh ok, I misread you then, sorry.
>>
>> I think having a dedicated speed.python machine is specifically so
>> that we can run benchmarks however much we want :) At least Cython
>> does not compile for like an hour...
>
>
> Yeah, we have tried to make sure  the machine we have for all of this is
> fast enough that we can run all the benchmarks on all measured VMs once a
> day. If that ever becomes an issue we would probably prune the benchmarks
> rather than turn them on/off selectively.
>
> But speaking of benchmarks that won't work on other VMs (e.g. twisted
> under Jython), we will obviously try to minimize how many of those we have.
> Twisted is somewhat of a special case because (a) PyPy has already put the
> time into creating the benchmarks and (b) it is used by so many people that
> measuring its speed is a good thing. Otherwise I would argue that all
> future benchmarks should be runnable on any VM and not just CPython or any
> other VMs that support C extensions (numpy is the only exception to this
> that I can think of because of its popularity and numpypy will extend its
> reach once the work is complete).
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> http://mail.python.org/mailman/listinfo/speed
>
>
I think, ideally, the benchmarks should all be valid pure-Python.  Without
wanted to beat up on Jython, I don't think that twisted doesn't run there
is an argument against including it.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20120202/d5d1c823/attachment.html>