[Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Nathaniel Smith]

On Thu, Sep 7, 2017 at 8:11 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> First of all, thanks for the feedback and encouragement!  Responses
> in-line below.

I hope it's helpful! More responses in-line as well.

> On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> My concern about this is the same as it was last time -- the work
>> looks neat, but right now, almost no-one uses subinterpreters
>> (basically it's Jep and mod_wsgi and that's it?), and therefore many
>> packages get away with ignoring subinterpreters.
>
> My concern is that this is a chicken-and-egg problem.  The situation
> won't improve until subinterpreters are more readily available.

Okay, but you're assuming that "more libraries work well with
subinterpreters" is in fact an improvement. I'm asking you to convince
me of that :-). Are there people saying "oh, if only subinterpreters
had a Python API and less weird interactions with C extensions, I
could do <something awesome>"? So far they haven't exactly taken the
world by storm...

>> Numpy is the one I'm
>> most familiar with: when we get subinterpreter bugs we close them
>> wontfix, because supporting subinterpreters properly would require
>> non-trivial auditing, add overhead for non-subinterpreter use cases,
>> and benefit a tiny tiny fraction of our users.
>
> The main problem of which I'm aware is C globals in libraries and
> extension modules.  PEPs 489 and 3121 are meant to help but I know
> that there is at least one major situation which is still a blocker
> for multi-interpreter-safe module state.  Other than C globals, is
> there some other issue?

That's the main one I'm aware of, yeah, though I haven't looked into it closely.

>> If we add a friendly python-level API like this, then we're committing
>> to this being a part of Python for the long term and encouraging
>> people to use it, which puts pressure on downstream packages to do
>> that work... but it's still not clear whether any benefits will
>> actually materialize.
>
> I'm fine with Nick's idea about making this a "provisional" module.
> Would that be enough to ease your concern here?

Potentially, yeah -- basically I'm fine with anything that doesn't end
up looking like python-dev telling everyone "subinterpreters are the
future! go forth and yell at any devs who don't support them!".

What do you think the criteria for graduating to non-provisional
status should be, in this case?

[snip]
>> So the only case I can see where I'd expect subinterpreters to make
>> communication dramatically more efficient is if you have a "deeply
>> immutable" type
>> [snip]
>> However, it seems impossible to support user-defined deeply-immutable
>> types in Python:
>> [snip]
>
> I agree that it is currently not an option.  That is part of the
> exercise.  There are a number of possible solutions to explore once we
> get to that point.  However, this PEP isn't about that.  I'm confident
> enough about the possibilities that I'm comfortable with moving
> forward here.

I guess I would be much more confident in the possibilities here if
you could give:

- some hand-wavy sketch for how subinterpreter A could call a function
that as originally defined in subinterpreter B without the GIL, which
seems like a precondition for sharing user-defined classes

- some hand-wavy sketch for how refcounting will work for objects
shared between multiple subinterpreters without the GIL, without
majorly impacting single-thread performance (I actually forgot about
this problem in my last email, because PyPy has already solved this
part!)

These are the two problems where I find it most difficult to have faith.

[snip]
>> I hope so -- a really useful subinterpreter multi-core stor[y] would be
>> awesome.
>
> Agreed!  Thanks for the encouragement. :)

Thanks for attempting such an ambitious project :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org