[Python-Dev] PEP 554 v3 (new interpreters module)

[Nick Coghlan]

On 5 October 2017 at 18:45, Eric Snow <ericsnowcurrently at gmail.com> wrote:

> After we move to not sharing the GIL between interpreters:
>
> Channel.send(obj):  # in interp A
>     incref(obj)
>     if type(obj).tp_share == NULL:
>         raise ValueError("not a shareable type")
>     set_owner(obj)  # obj.owner or add an obj -> interp entry to global
> table
>     ch.objects.append(obj)
>
> Channel.recv():  # in interp B
>     orig = ch.objects.pop(0)
>     obj = orig.tp_share()
>     set_shared(obj, orig)  # add to a global table
>     return obj
>

This would be hard to get to work reliably, because "orig.tp_share()" would
be running in the receiving interpreter, but all the attributes of "orig"
would have been allocated by the sending interpreter. It gets more reliable
if it's *Channel.send* that calls tp_share() though, but moving the call to
the sending side makes it clear that a tp_share protocol would still need
to rely on a more primitive set of "shareable objects" that were the
permitted return values from the tp_share call.

And that's the real pay-off that comes from defining this in terms of the
memoryview protocol: Py_buffer structs *aren't* Python objects, so it's
only a regular C struct that gets passed across the interpreter boundary
(the reference to the original objects gets carried along passively as part
of the CIV - it never gets *used* in the receiving interpreter).


> bytes.tp_share():
>     obj = blank_bytes(len(self))
>     obj.ob_sval = self.ob_sval # hand-wavy memory sharing
>     return obj
>

This is effectively reinventing memoryview, while trying to pretend it's an
ordinary bytes object. Don't reinvent memoryview :)


> bytes.tp_free():  # under no-shared-GIL:
>     # most of this could be pulled into a macro for re-use
>     orig = lookup_shared(self)
>     if orig != NULL:
>         current = release_LIL()
>         interp = lookup_owner(orig)
>         acquire_LIL(interp)
>         decref(orig)
>         release_LIL(interp)
>         acquire_LIL(current)
>         # clear shared/owner tables
>         # clear/release self.ob_sval
>     free(self)
>

I don't think we should be touching the behaviour of core builtins solely
to enable message passing to subinterpreters without a shared GIL.

The simplest possible variant of CIVs that I can think of would be able to
avoid that outcome by being a memoryview subclass, since they just need to
hold the extra reference to the original interpreter, and include some
logic to swtich interpreters at the appropriate time.

That said, I think there's definitely a useful design question to ask in
this area, not about bytes (which can be readily represented by a
memoryview variant in the receiving interpreter), but about *strings*: they
have a more complex internal layout than bytes objects, but as long as the
receiving interpreter can make sure that the original string continues to
exist, then you could usefully implement a "strview" type to avoid having
to go through an encode/decode cycle just to pass a string to another
subinterpreter.

That would provide a reasonable compelling argument that CIVs *shouldn't*
be implemented as memoryview subclasses, but instead defined as
*containing* a managed view of an object owned by a different interpreter.

That way, even if the initial implementation only supported CIVs that
contained a memoryview instance, we'd have the freedom to define other
kinds of views later (such as strview), while being able to reuse the same
CIV machinery.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20171005/2c041f5c/attachment.html>