[Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Nick Coghlan]

On 13 September 2017 at 00:35, Koos Zevenhoven <k7hoven at gmail.com> wrote:
> On Tue, Sep 12, 2017 at 1:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 11 September 2017 at 18:02, Koos Zevenhoven <k7hoven at gmail.com> wrote:
>> > On Mon, Sep 11, 2017 at 8:32 AM, Nick Coghlan <ncoghlan at gmail.com>
>> > wrote:
>> >> The line between it and the "CPython Runtime" is fuzzy for both
>> >> practical and historical reasons, but the regular Python CLI will
>> >> always have a "first created, last destroyed" main interpreter, simply
>> >> because we don't really gain anything significant from eliminating it
>> >> as a concept.
>> >
>> > I fear that emphasizing the main interpreter will lead to all kinds of
>> > libraries/programs that somehow unnecessarily rely on some or all tasks
>> > being performed in the main interpreter. Then you'll have a hard time
>> > running two of them in parallel in the same process, because you don't
>> > have
>> > two main interpreters.
>>
>> You don't need to fear this scenario, since it's a description of the
>> status quo (and it's the primary source of overstated claims about
>> subinterpreters being "fundamentally broken").
>>
>
> Well, if that's true, it's hardly a counter-argument to what I said. Anyway,
> there is no status quo about what is proposed in the PEP.

Yes, there is, since subinterpreters are an existing feature of the
CPython implementation.

What's new in the PEP is the idea of giving that feature a Python
level API so that it's available to regular Python programs, rather
than only being available to embedding applications that choose to use
it (e.g. mod_wsgi).

> And as long as the existing APIs are preserved, why not make the new one
> less susceptible to overstated fundamental brokenness?

Having a privileged main interpreter isn't fundamentally broken, since
you aren't going to run __main__ in more than one interpreter, just as
you don't run __main__ in more than one thread (and multiprocessing
deliberately avoids running the "if __name__ == '__main__'" sections
of it in more than one process).

>> So no, not everything will be subinterpreter-friendly, just as not
>> everything in Python is thread-safe, and not everything is portable
>> across platforms.
>
> I don't see how the situation benefits from calling something the "main
> interpreter". Subinterpreters can be a way to take something non-thread-safe
> and make it thread-safe, because in an interpreter-per-thread scheme, most
> of the state, like module globals, are thread-local. (Well, this doesn't
> help for async concurrency, but anyway.)

"The interpreter that runs __main__" is never going to go away as a
concept for the regular CPython CLI.

Right now, its also a restriction even for applications like mod_wsgi,
since the GIL state APIs always register C created threads with the
main interpreter.

>> That's OK - it just means we'll aim to make as many
>> things as possible implicitly subinterpreter-friendly, and for
>> everything else, we'll aim to minimise the adjustments needed to
>> *make* things subinterpreter friendly.
>>
>
> And that's exactly what I'm after here!

No, you're after deliberately making the proposed API
non-representative of how the reference implementation actually works
because of a personal aesthetic preference rather than asking yourself
what the practical benefit of hiding the existence of the main
interpreter would be.

The fact is that the main interpreter *is* special (just as the main
thread is special), and your wishing that things were otherwise won't
magically make it so.

> I'm mostly just worried about the `get_main()` function. Maybe it should be
> called `asdfjaosjnoijb()`, so people wouldn't use it. Can't the first
> running interpreter just introduce itself to its children? And if that's too
> much to ask, maybe there could be a `get_parent()` function, which would
> give you the interpreter that spawned the current subinterpreter.

If the embedding application never calls
"_Py_ConfigureMainInterpreter", then get_main() could conceivably
return None. However, we don't expose that as a public API yet, so for
the time being, Py_Initialize() will always call it, and hence there
will always be a main interpreter (even in things like mod_wsgi).

Whether we invest significant effort in making configuring the main
interpreter genuinely optional is still an open question - since most
applications are free to just not use the main interpreter for code
execution if they don't want to, we haven't found a real world use
case that would benefit meaningfully from its non-existence (just as
the vast majority of applications don't care about the various ways in
which the main thread that runs Py_Initialize() and Py_Finalize() is
given special treatment, and for those that do, they're free to avoid
using it).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia