[Python-Dev] PEP 554 v2 (new "interpreters" module)

[Antoine Pitrou]

On Fri, 8 Sep 2017 16:04:27 -0700
Eric Snow <ericsnowcurrently at gmail.com> wrote:
> 
> The module provides the following functions:
> 
> ``list()``::
> 
>    Return a list of all existing interpreters.

It's called ``enumerate()`` in the threading module.  Not sure there's
a point in choosing a different name here.

> The module also provides the following classes:
> 
> ``Interpreter(id)``::

>    run(code):
> 
>       Run the provided Python code in the interpreter, in the current
>       OS thread.  If the interpreter is already running then raise
>       RuntimeError in the interpreter that called ``run()``.
> 
>       The current interpreter (which called ``run()``) will block
> until the subinterpreter finishes running the requested code.  Any
>       uncaught exception in that code will bubble up to the current
>       interpreter.

Why does it block?  How is concurrency supposed to be achieved in that
model?  It would be more flexible if run(code) returned an object that
can later be waited on.  Something like... a Future :-)

And why guarantee that it executes in the "current OS thread"?
I would say you don't want to specify where it executes exactly, as it
opens the door for more sophisticated implementations (such as
automatic assignment of subinterpreters inside a pool of threads).

>    get_fifo(name):
> 
>       Return the FIFO object with the given name that is associated
>       with this interpreter.  If no such FIFO exists then raise
>       KeyError.  The FIFO will be either a "FIFOReader" or a
>       "FIFOWriter", depending on which "add_*_fifo()" was called.
> 
>    list_fifos():
> 
>       Return a list of all fifos associated with the interpreter.

If fifos are uniquely named, why not return a name->fifo mapping?

> ``FIFOReader(name)``::
> [...]

I don't think the method naming choice is very adequate here.  The API
model for the FIFO objects can either be a (threading or
multiprocessing) Queue or a multiprocessing Pipe.

- if a Queue, then it should have a get() / put() pair of methods
- if a Pipe, then it should have a recv() / send() pair of methods

Now, since Queues are multi-producer multi-consumer, while Pipes are
single-producer single-consumer (they aren't "synchronized"), the
better analogy seems to the multiprocessing Pipe here, so I would vote
for recv() / send().

But, in any case, definitely not a pop() / push() pair.


Has any thought been given to how FIFOs could integrate with async code
driven by an event loop (e.g. asyncio)?  I think the model of executing
several asyncio (or Tornado) applications each in their own
subinterpreter may prove quite interesting to reconcile multi-core
concurrency with ease of programming.  That would require the FIFOs to
be able to synchronize on something an event loop can wait on (probably
a file descriptor?).

Regards

Antoine.