[Python-ideas] solving multi-core Python

[Wes Turner]

Exciting!

*
http://zero-buffer.readthedocs.org/en/latest/api-reference/#zero_buffer.BufferView
* https://www.google.com/search?q=python+channels
  * https://docs.python.org/2/library/asyncore.html#module-asyncore
  * https://chan.readthedocs.org/en/latest/
  * https://goless.readthedocs.org/en/latest/
* other approaches to the problem (with great APIs):
  * http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords
  * http://discodb.readthedocs.org/en/latest/
On Jun 20, 2015 5:55 PM, "Eric Snow" <ericsnowcurrently at gmail.com> wrote:

>
> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
> >
> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently at gmail.com>
> wrote:
> > >
> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters,
> > > exposing them in Python, and adding a mechanism to safely share
> > > objects between them.
> >
> > This all sounds really cool if you can pull it off, and shared-nothing
> threads do seem like the least impossible model to pull off.
>
> Agreed.
>
> > But "least impossible" and "possible" are different :-). From your email
> I can't tell whether this plan is viable while preserving backcompat and
> memory safety.
>
> I agree that those issues must be clearly solved in the proposal before it
> can be approved.  I'm confident the approach I'm pursuing will afford us
> the necessary guarantees.  I'll address those specific points directly when
> I can sit down and organize my thoughts.
>
> >
> > Suppose I have a queue between two subinterpreters, and on this queue I
> place a list of dicts of user-defined-in-python objects, each of which
> holds a reference to a user-defined-via-the-C-api object. What happens next?
>
> You've hit upon exactly the trickiness involved and why I'm thinking the
> best approach initially is to only allow *strictly* immutable objects to
> pass between interpreters.  Admittedly, my description of channels is very
> vague.:)  There are a number of possibilities with them that I'm still
> exploring (CSP has particular opinions...), but immutability is a
> characteristic that may provide the simplest *initial* approach.  Going
> that route shouldn't preclude adding some sort of support for mutable
> objects later.
>
> Keep in mind that by "immutability" I'm talking about *really* immutable,
> perhaps going so far as treating the full memory space associated with an
> object as frozen.  For instance, we'd have to ensure that "immutable"
> Python objects like strings, ints, and tuples do not change (i.e. via the C
> API).  The contents of involved tuples/containers would have to be likewise
> immutable.  Even changing refcounts could be too much, hence the idea of
> moving refcounts out to a separate table.
>
> This level of immutability would be something new to Python.  We'll see if
> it's necessary.  If it isn't too much work it might be a good idea
> regardless of the multi-core proposal.
>
> Also note that Barry has a (rejected) PEP from a number of years ago about
> freezing objects...  That idea is likely out of scope as relates to my
> proposal, but it certainly factors in the problem space.
>
> -eric
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150621/e4aecde0/attachment-0001.html>