[Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

[Nathaniel Smith]

On May 24, 2017 20:31, "Guido van Rossum" <guido at python.org> wrote:

Hm... Curiously, I've heard a few people at PyCon mention they thought
subinterpreters were broken and not useful (and they share the GIL anyways)
and should be taken out. So we should at least have clarity on which
direction we want to take...


My impression is that the code to support them inside CPython is fine, but
they're broken and not very useful in the sense that lots of C extensions
don't really support them, so in practice you can't reliably use them to
run arbitrary code. Numpy for example definitely has lots of
subinterpreter-related bugs, and when they get reported we close them as
WONTFIX.

Based on conversations at last year's pycon, my impression is that numpy
probably *could* support subinterpreters (i.e. the required apis exist),
but none of us really understand the details, it's the kind of problem that
requires a careful whole-codebase audit, and a naive approach might make
numpy's code slower and more complicated for everyone. (For example, there
are lots of places where numpy keeps a little global cache that I guess
should instead be per-subinterpreter caches, which would mean adding an
extra lookup operation to fast paths.)

Or maybe it'd be fine, but no one is motivated to figure it out, because
the other side of the cost/benefit analysis is that almost nobody actually
uses subinterpreters. I think the only two projects that do are mod_wsgi
and jep [1].

So yeah, the status quo is broken. But there are two possible ways to fix
it: IMHO either subinterpreters should be removed *or* they should have
some compelling features added to make them actually worth the effort of
fixing c extensions to support them. If Eric can pull off this multi-core
idea then that would be pretty compelling :-). (And my impression is that
the things that break under subinterpreters are essentially the same as
would break under any GIL-removal plan.)

The problem is that we don't actually know yet whether the multi-core idea
will work, so it seems like a bad time to double down on committing to
subinterpreter support and pressuring C extensions to keep up. Eric- do you
have a plan written down somewhere? I'm wondering what ​the critical path
from here to a multi-core proof of concept looks like.

-n

[1] https://github.com/mrj0/jep/wiki/How-Jep-Works
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170525/4053ce3e/attachment-0001.html>