[Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibility Requirements

[R. David Murray]

On Sun, 17 Apr 2011 12:14:51 +0200, Stefan Krah <stefan at bytereef.org> wrote:
> I'm not sure that I understand the duplication of effort: If there
> is a C module without a Python implementation in the stdlib, then
> the PyPy, Jython, and IronPython developers are free to cooperate
> and implement a single Python version. I would not consider this
> a duplication of effort.

Yes, that's exactly what we are trying to encourage.  If the Python
standard library is seen as common property of all Python implementations,
then this is *much* more likely to happen.

> If, on the other hand, they choose to provide three individual
> implementations in C#, Java and (?), then that is their own choice
> and surely not the fault of the C module developer.

Right.

> By contrast, this PEP puts a great burden on the developers of
> new C modules. If this PEP is accepted, it is the C module developers
> who will have to do duplicate work.

This is true only because of the current "blessed" position of CPython
in the Python ecosystem.  If a separate Python stdlib is the common
property of all Python implementations, then the same double burden
would apply to, say, an IronPython developer writing a module in C#
and wanting it included in the stdlib.

> In my view, the PEP should have a clause that *active* participation
> of PyPy, Jython, and IronPython developers is expected if they want
> pure compatible Python versions to exist.

> >     Re-implementing parts (or all) of a module in C (in the case
> >     of CPython) is still allowed for performance reasons, but any such
> >     accelerated code must pass the same test suite (sans VM- or C-specific
> >     tests) to verify semantics and prevent divergence. To accomplish this,
> >     the test suite for the module must have 100% branch coverage of the
> >     pure Python implementation before the acceleration code may be added.
> 
> Raymond has pointed out that the PEP seems to discourage C modules. This
> is one of the examples. Since implementing C modules takes a lot of time,
> I'd appreciate to know if they are just tolerated or actually welcome.

I believe they are welcome, but that they are a CPython implementation
detail, and the PEP is trying to make that distinction clear.

One can also imagine a C module getting accepted to the stdblib
because everybody agrees that (a) it can't be implemented in Python and
(b) every Python implementation should support it.  In that case only
the test suite will be part of the implementation-independent part of
the stdlib.  I do think that such modules (and we already have several)
should have a higher bar to cross to get in to the stdlib than modules
that have a pure Python implementation.

> If all possible exceptions must match, then in the case of decimal the
> PEP should give permission to change the published API of an existing
> Python module (in this case decimal.py). Otherwise, I see no way of
> accomplishing this goal.

This may well be what needs to be done, both for CPython and for other
implementations.  When we agree that some test covers something that is
an implementation detail, the tests should be so marked.  Making changes
to the API and tests to accommodate specific Python implementations
(including CPython) will be the right thing to do in some cases.
Obviously these will have to be considered on a case by case basis.

The Python sdtlib and its tests is already the standard that other
implementations need to conform to.  The PEP is trying to lay out some
rules so that CPython has to conform on equal footing with the
other implementations.

> It is possible to give many frivolous examples:
> 
> >>> from decimal import *
> >>>
> 
> >>> class C():
> ...     def __init__(self):
> ...         self.traps = 'invalid'
> ...
> 
> >>> # No exception
> ... setcontext(C())
> >>> 
> 
> 
> 
> >>> from cdecimal import *
> >>> class C():
> ...     def __init__(self):
> ...         self.traps = 'invalid'
> ... 
> 
> >>> setcontext(C())
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: argument must be a context.
> >>>
>
> In the case of duck typing, the only solution I see is to lock down the
> types in decimal.py, thus changing the API. This is one of the things that
> should be decided *before* the PEP is accepted.

Here you perceive the burden we are currently placing on the other
implementations.  That's the world they live in *now*.  The PEP is asking
CPython to share this pain equally.

I agree that this is a concrete example that the PEP could address.
I myself don't know enough about decimal/cdecimal or the Python C API
to know why cdecimal can't duck type here, but it certainly sounds
like a good example to use to clarify the requirements being advocated
by the PEP.  I won't be surprised to find that the issues involved are
the same issues that an accelerator module for the other Python
implementations would face.

--
R. David Murray           http://www.bitdance.com