[Python-Dev] [GSoC] Developing a benchmark suite (for Python 3.x)

[Stefan Behnel]

Maciej Fijalkowski, 11.04.2011 11:39:
> On Fri, Apr 8, 2011 at 11:22 AM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> Jesse Noller, 07.04.2011 22:28:
>>>
>>> On Thu, Apr 7, 2011 at 3:54 PM, Anthony Scopatz wrote:
>>>>
>>>> Hi Daniel,
>>>> Thanks for putting this together.  I am a huge supporter of benchmarking
>>>> efforts.  My brief comment is below.
>>>>
>>>> On Wed, Apr 6, 2011 at 11:52 AM, DasIch wrote:
>>>>>
>>>>> 1. Definition of the benchmark suite. This will entail contacting
>>>>> developers of Python implementations (CPython, PyPy, IronPython and
>>>>> Jython), via discussion on the appropriate mailing lists. This might
>>>>> be achievable as part of this proposal.
>>>>>
>>>>
>>>> If you are reaching out to other projects at this stage, I think you
>>>> should
>>>> also be in touch with the Cython people  (even if its 'implementation'
>>>> sits on top of CPython).
>>>> As a scientist/engineer what I care about is how Cython benchmarks to
>>>> CPython.  I believe that they have some ideas on benchmarking and have
>>>> also explored this space.  Their inclusion would be helpful to me
>>>> thinking
>>>> this GSoC successful at the end of the day (summer).
>>>> Thanks for your consideration.
>>>> Be Well
>>>> Anthony
>>>
>>> Right now, we are talking about building "speed.python.org" to test
>>> the speed of python interpreters, over time, and alongside one another
>>> - cython *is not* an interpreter.
>>
>> Would you also want to exclude Psyco then? It clearly does not qualify as a
>> Python interpreter.
>
> Just to clarify - the crucial word here is Python and not the
> interpreter.

Psyco is also not a Python implementation. It doesn't work without CPython, 
just like Cython. But I doubt that anyone would seriously argue for 
excluding Psyco from a Python speed comparison. That was my point here.


> I don't care myself if it's an interpreter or a compiler,
> I do care if it can pass the python test suite (modulo things that are
> known to be implementation details and agreed upon).
>
> How far is Cython from passing the full test suite?

According to our CI server, we currently have 255 failing tests out of 7094 
in Python 2.7.

https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-pyregr-py27-c/

This is not completely accurate as a) it only includes compiling the test 
module, and e.g. not the stdlib modules that are being tested, and b) the 
total number of tests we see depends on how many test modules we can 
compile in order to import and run the contained tests. It also doesn't 
mean that we have >200 compatibility problems, the majority of failures 
tends to be because of just a hand full of bugs.

Another measure is that Cython can currently compile some 160 modules out 
of a bit less than 200 in Django (almost all failures due to one bug about 
incompatibilities between PyCFunction and Python functions) and an 
(untested!) 1219 out of 1538 modules in the stdlib. We haven't put that 
together yet in order to actually test the compiled stdlib modules. That'll 
come.


> Are there known incompatibilities that would be considered wontfix?

There are known incompatibilities that are considered bugs. There are no 
"wontfix" bugs when it comes to Python compatibility. But there are 
obviously developer priorities when it comes to fixing bugs. Cython is a 
lot more than just a Python compiler (such as a programming language that 
keeps people from writing C code), so there are also bugs and feature 
requests apart from Python semantics that we consider more important to 
fix. It's not like all bugs on CPython's bug tracker would get closed 
within a day or so.

Stefan