[Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Jesse Noller]

On Tue, Apr 19, 2011 at 1:06 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
[snip]
> This PEP has received a lengthy discussion by now, so here's why I think
> it's being fought so heavily by several CPython core developers,
> specifically those who have traditionally carried a large part of the
> optimisation load in the project.
>
> I think the whole point of this PEP is that, having agreed that a shared
> standard library for all Python implementations is a good thing, the amount
> of shareable code should be maximised. I doubt that anyone will argue
> against this goal.
>
> But that obviously includes all sides. If other implementations are free to
> cherry pick the targets of their own effort geared by the optimisation of
> their own implementation, and leave the whole burden of compatibility and
> code reusability on CPython, in addition to the CPython efforts of improving
> and optimising its own core code base and its own stdlib version, it's not
> an equal matter.
>

I am going to go out on a limb here and state that once the stdlib is
shared, it is all of the VM's responsibility to help maintaining it,
meaning the PEP 399 is binding to all of the VMs. If Jython wants to
write an accelerator module in Java for something in the stdlib, they
have to follow the same guidelines, same applies to PyPy, etc.

I think this is an equal matter, and if needed, we should make note of
it in the PEP. The goal here is to make it easier to share the code
base of the stdlib and not pull the rug out from other implementations
by having a stdlib module written only in highly optimized C with no
Python fallback, leaving them with the unsavory duty of reimplementing
it in Python|Java|C#, etc.

Pure Python is the coin of the realm.

> That's what makes the PEP feel so unfair to CPython developers, because they
> are the ones who carry most of the burden of maintaining the stdlib in the
> first place, and who will most likely continue to carry it, because other
> implementations will continue to be occupied with their own core development
> for another while or two. It is nice to read that other implementations are
> contributing back patches that simplify their own reuse of the stdlib code.
> However, that does not yet make them equal contributors to the development
> and the maintenance of the stdlib, and is of very little worth to the
> CPython project. It often even runs counter to the interest of CPython
> itself.

Sure, at first glance this seems to place an unfair burden on CPython
- because we're just as guilty as being "closed" to other
implementation as the other implementations are to us. We're trying to
change that, and someone (us, as the reference implementation) need to
take the first responsible step.

Once this move is made/accepted, I would expect the other
implementation to rapidly move away from their custom implementations
of the stdlib and contribute to the shared code base and
documentation. Yes, this places a burden on CPython, but in the long
term in benefits *all* of the projects equally by simply having more
active contributors.

We have over 200 stdlib modules, and far, far less than that in active
developers focused or working on the stdlib. Making it a shared
property (in theory) means that the other VMs have a shared interest
in that property. We're effectively spreading the load.

> I think this social problem of the PEP can only be solved if the CPython
> project stops doing the major share of the stdlib maintenance, thus freeing
> its own developer capacities to focus on CPython related improvements and
> optimisations, just like the other implementations currently do. I'm not
> sure we want that at this point.

That's not going to happen. CPython will continue to do the bulk of
the maintenance until we break it out, and the other implementations
have time to adapt and pull in the shared code base.

I don't see this as such a large burden as you seem to be making it
out to be: CPython is the reference implementation, and our stdlib is
the reference stdlib. We can break out the stdlib and share it amongst
the implementations therefore making it more than the reference stdlib
- we can make it the defacto stdlib for the language as a whole.

We also, long term, want spread the maintenance load beyond CPython,
but right now we are the primary caretakers, so yes - this adds load
to us in the short term, but benefits us in the long term.

jesse