[Python-ideas] solving multi-core Python

[Eric Snow]

On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
>
> On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently at gmail.com> wrote:
> >
> > tl;dr Let's exploit multiple cores by fixing up subinterpreters,
> > exposing them in Python, and adding a mechanism to safely share
> > objects between them.
>
> This all sounds really cool if you can pull it off, and shared-nothing
threads do seem like the least impossible model to pull off.

Agreed.

> But "least impossible" and "possible" are different :-). From your email
I can't tell whether this plan is viable while preserving backcompat and
memory safety.

I agree that those issues must be clearly solved in the proposal before it
can be approved.  I'm confident the approach I'm pursuing will afford us
the necessary guarantees.  I'll address those specific points directly when
I can sit down and organize my thoughts.

>
> Suppose I have a queue between two subinterpreters, and on this queue I
place a list of dicts of user-defined-in-python objects, each of which
holds a reference to a user-defined-via-the-C-api object. What happens next?

You've hit upon exactly the trickiness involved and why I'm thinking the
best approach initially is to only allow *strictly* immutable objects to
pass between interpreters.  Admittedly, my description of channels is very
vague.:)  There are a number of possibilities with them that I'm still
exploring (CSP has particular opinions...), but immutability is a
characteristic that may provide the simplest *initial* approach.  Going
that route shouldn't preclude adding some sort of support for mutable
objects later.

Keep in mind that by "immutability" I'm talking about *really* immutable,
perhaps going so far as treating the full memory space associated with an
object as frozen.  For instance, we'd have to ensure that "immutable"
Python objects like strings, ints, and tuples do not change (i.e. via the C
API).  The contents of involved tuples/containers would have to be likewise
immutable.  Even changing refcounts could be too much, hence the idea of
moving refcounts out to a separate table.

This level of immutability would be something new to Python.  We'll see if
it's necessary.  If it isn't too much work it might be a good idea
regardless of the multi-core proposal.

Also note that Barry has a (rejected) PEP from a number of years ago about
freezing objects...  That idea is likely out of scope as relates to my
proposal, but it certainly factors in the problem space.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150620/974ef380/attachment.html>