[python-committers] Redoing the C API?

[Larry Hastings]

p

On 03/03/2016 09:58 AM, Eric Snow wrote:
> On Thu, Mar 3, 2016 at 10:39 AM, Brett Cannon <brett at python.org> wrote:
>> ...what would we need to do to our C API to make
>> it so that anyone following a new API wouldn't be broken if we dropped the
>> GIL?
> If I recall correctly, this was one key topic that Larry discussed at
> the language summit latest year.

Kinda, yeah.  Certainly it's a topic I've thought a lot about.

Consider this.  With almost no exceptions*, none of the popular new 
languages have a C API.  Instead they'll have a foreign function 
interface allowing you to call C from inside the language.  So you don't 
write extensions in C and call into the language, you write your 
extensions natively in the language and call out.

One advantage of this technique is that it allows most implementation 
details of the language to remain hidden.  CPython can't drop reference 
counting and move solely to tracing garbage collection, because the C 
API lets external callers deal with Python objects, which means Python's 
internal object lifetime management approach must be visible, which 
means it's implicitly part of the API.  In short, if we change from 
reference counting to tracing GC, we break every C extension in 
existence, kablooey, oblivion.

But!  If there were no C extensions--if all Python programs talked to 
native libraries through FFIs like ctypes and cffi--then this would be a 
private implementation detail and we could iterate however we liked on 
object lifetime management.  I've asked Armin Rigo about PyPy here.  
Pardon me if my memory is faulty, but what I think he said was this: 
they started with GC, then went to generational GC, then went to 
incremental generational GC.  If they'd had a C API, going to 
generational probably wouldn't have broken all their extensions, but 
going to incremental absolutely would have.  Since PyPy doesn't have a C 
API, naturally they can change it all they like.

If we could wave a magic wand and get all extension authors to switch to 
writing their extensions in Python and using cffi, we should absolutely 
do it.  That'd be great for cross-implementation compatibility; your 
extension would (hopefully) run unchanged in CPython and PyPy today, and 
I heard a rumor that Jython and IronPython want to support cffi too, so 
hey! someday it might run unchanged in those too.  This would also make 
it possible for CPython to declare that the C API was dead and free us 
up to make some radical but welcome changes to CPython's innards. 
Unfortunately, we don't have such a magic wand, and I don't think 
there's any workable path to convince extension authors to switch en 
masse.  And if we're stuck with the C API, we're stuck with a lot of the 
implementation details that are baked into it.

I'm hoping to present on this subject at this year's summit.  I hope all 
the interested core devs can make it!


//arry/


* The only exception I know of is Lua--are there more?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-committers/attachments/20160304/0727edd6/attachment-0001.html>