[Python-Dev] [GSoC] Developing a benchmark suite (for Python 3.x)

[Stefan Behnel]

Maciej Fijalkowski, 08.04.2011 11:41:
> On Fri, Apr 8, 2011 at 11:22 AM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> Jesse Noller, 07.04.2011 22:28:
>>>
>>> On Thu, Apr 7, 2011 at 3:54 PM, Anthony Scopatz wrote:
>>>>
>>>> Hi Daniel,
>>>> Thanks for putting this together.  I am a huge supporter of benchmarking
>>>> efforts.  My brief comment is below.
>>>>
>>>> On Wed, Apr 6, 2011 at 11:52 AM, DasIch wrote:
>>>>>
>>>>> 1. Definition of the benchmark suite. This will entail contacting
>>>>> developers of Python implementations (CPython, PyPy, IronPython and
>>>>> Jython), via discussion on the appropriate mailing lists. This might
>>>>> be achievable as part of this proposal.
>>>>>
>>>>
>>>> If you are reaching out to other projects at this stage, I think you
>>>> should
>>>> also be in touch with the Cython people  (even if its 'implementation'
>>>> sits on top of CPython).
>>>> As a scientist/engineer what I care about is how Cython benchmarks to
>>>> CPython.  I believe that they have some ideas on benchmarking and have
>>>> also explored this space.  Their inclusion would be helpful to me
>>>> thinking
>>>> this GSoC successful at the end of the day (summer).
>>>> Thanks for your consideration.
>>>> Be Well
>>>> Anthony
>>>
>>> Right now, we are talking about building "speed.python.org" to test
>>> the speed of python interpreters, over time, and alongside one another
>>> - cython *is not* an interpreter.
>>
>> Would you also want to exclude Psyco then? It clearly does not qualify as a
>> Python interpreter.
>
> Why not? it does run those benchmarks just fine.

Sure.


>>> Cython is out of scope for this.
>>
>> Why? It should be easy to integrate Cython using pyximport. Basically, all
>> you have to do is register the pyximport module as an import hook. Cython
>> will then try to compile the imported Python modules and fall back to the
>> normal .py file import if the compilation fails for some reason.
>
> then it's fine to include it. we can even include it now in
> speed.pypy.org that way. would it compile django?

Never tried. Likely not completely, but surely some major parts of it. 
That's the beauty of it - it just falls back to CPython. :) If we're lucky, 
it will manage to compile some performance critical parts without 
modifications. In any case, it'll be trying to compile each module.


>> So, once CPython is up and running in the benchmark test, adding Cython
>> should be as easy as copying the configuration, installing Cython and adding
>> two lines to site.py.
>
> can you provide a simple command line tool for that? I want
> essentially to run ./cython-importing-stuff some-file.py

You can try

     python -c 'import pyximport; \
                pyximport.install(pyimport=True); \
                exec("somefile.py")'

You may want to configure the output directory for the binary modules, 
though, see

https://github.com/cython/cython/blob/master/pyximport/pyximport.py#L343

Please also take care to provide suitable gcc CFLAGS, e.g. "-O3 
-march=native" etc.


>> Obviously, we'd have to integrate a build of the latest Cython development
>> sources as well, but it's not like installing a distutils enabled Python
>> package from sources is so hard that it pushes Cython out of scope for this
>> GSoC.
>
> no, that's fine. My main concern is - will cython run those
> benchmarks?

In the worst case, they will run at CPython speed with uncompiled modules.


> and will you complain if we don't provide a custom cython
> hacks? (like providing extra type information)

I don't consider providing extra type information a hack. Remember that 
they are only used for additional speed-ups in cases where the author is 
smarter than the compiler. It will work just fine without them.

Stefan