[Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

[Eric Snow]

On Mon, Jul 16, 2018 at 11:08 AM Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 16 Jul 2018 18:00:37 +0100
> MRAB <python at mrabarnett.plus.com> wrote:
> > Could you explicitly share an object in a similar way to how you
> > explicitly open a file?
> >
> > The shared object's refcount would be incremented and the sharing
> > function would return a proxy to the shared object.
> >
> > Refcounting in the thread/process would be done on the proxy.
> >
> > When the proxy is closed or garbage-collected, the shared object's
> > refcount would be decremented.
> >
> > The shared object could be garbage-collected when its refcount drops to
> > zero.
>
> Yes, I'm assuming that would be how shareable buffers could be
> implemented: a per-interpreter proxy (with a regular Python refcount)
> mediating access to a shared object (which could have an atomic /
> thread-safe refcount).

Nice! That's exactly how I'm doing it. :)  The buffer protocol makes
it easier, but the idea could apply to arbitrary objects generally.
That's something I'll look into in a later phase of the project.

In both cases the tricky part is ensuring that the proxy does not
directly mutate the object (especially the refcount).  In fact, the
decref part above is the trickiest.  The trickiness is a consequence
of our goals.  In my multi-core project we're aiming for not sharing
the GIL between interpreters.  That means reaching and keeping proper
separation between interpreters.  Notably, without a GIL shared by
interpreters, refcount operations are not thread-safe.  Also, in the
decref case GC would happen under the wrong interpreter (which is
problematic for several reasons).

With this in mind, here's how I'm approaching the problem:

1. interp A "shares" an object with interp B (e.g. through a channel)
    * the object is incref'ed under A before it is sent to B
2. the object is wrapped in a proxy owned by B
    * the proxy may not make C-API calls that would mutate the object
or even cause an incref/decref
3. when the proxy is GC'd, the original object is decref'ed
    * the decref must happen in a thread in which A is running

In order to make all this work the missing piece is a mechanism by
which the decref (#3) happens under the original interpreter.  At the
moment Emily Morehouse and I are pursuing an approach that extends the
existing ceval "pending call" machinery currently used for handling
signals (see Py_AddPendingCall).  The new [*private*] API would work
the same way but on a per-interpreter basis rather than just the main
interpreter.  This would allow one interpreter to queue up a decref to
happen later under another interpreter.

FWIW, this ability to decref an object under a different interpreter
is a blocker right now for a number of things, including supporting
buffers in PEP 554 channels.

-eric