[Python-ideas] Python-ideas Digest, Vol 103, Issue 100
jean-charles.douet at laposte.net
jean-charles.douet at laposte.net
Sun Jun 21 13:03:37 CEST 2015
Hello,
My dix cents may not seem to be neither very clear, nor too argumented, but I wanted to let you know that there seem to be already interesting pythonic studies about how to release the so-called GIL and about, all, for which pragmatic use cases.
Reading the following article, I discovered that Python by itself seems not to be a language, but language specification :
http://www.toptal.com/python/why-are-there-so-many-pythons
Thus, it justifies that PyPy is a more general approach than CPython, which looks like a particular case of Python, even if the most frequently used (?)
Now, there is a specific study of PyPy aimed at removing the GIL, called "Software Transactional Memory". Here it is :
http://doc.pypy.org/en/latest/stm.html
Hope it helps :
Best regards,
Jean-Charles.
----- Mail original -----
De: python-ideas-request at python.org
À: python-ideas at python.org
Envoyé: Dimanche 21 Juin 2015 08:41:24
Objet: Python-ideas Digest, Vol 103, Issue 100
Send Python-ideas mailing list submissions to
python-ideas at python.org
To subscribe or unsubscribe via the World Wide Web, visit
https://mail.python.org/mailman/listinfo/python-ideas
or, via email, send a message with subject or body 'help' to
python-ideas-request at python.org
You can reach the person managing the list at
python-ideas-owner at python.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Python-ideas digest..."
Today's Topics:
1. Re: solving multi-core Python (Nathaniel Smith)
2. Re: solving multi-core Python (Nick Coghlan)
3. Re: solving multi-core Python (Wes Turner)
----------------------------------------------------------------------
Message: 1
Date: Sat, 20 Jun 2015 22:25:07 -0700
From: Nathaniel Smith <njs at pobox.com>
To: Eric Snow <ericsnowcurrently at gmail.com>
Cc: python-ideas <python-ideas at python.org>
Subject: Re: [Python-ideas] solving multi-core Python
Message-ID:
<CAPJVwBkjoK31m2-ynrGF_AmYFL0ULL3LdX6r7+d+B7RienQh7A at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
On Jun 20, 2015 3:54 PM, "Eric Snow" <ericsnowcurrently at gmail.com> wrote:
>
>
> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
> >
> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently at gmail.com>
wrote:
> > >
> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters,
> > > exposing them in Python, and adding a mechanism to safely share
> > > objects between them.
> >
> > This all sounds really cool if you can pull it off, and shared-nothing
threads do seem like the least impossible model to pull off.
>
> Agreed.
>
> > But "least impossible" and "possible" are different :-). From your
email I can't tell whether this plan is viable while preserving backcompat
and memory safety.
>
> I agree that those issues must be clearly solved in the proposal before
it can be approved. I'm confident the approach I'm pursuing will afford us
the necessary guarantees. I'll address those specific points directly when
I can sit down and organize my thoughts.
I'd love to see just a hand wavy, verbal proof-of-concept walking through
how this might work in some simple but realistic case. To me a single
compelling example could make this proposal feel much more concrete and
achievable.
> > Suppose I have a queue between two subinterpreters, and on this queue I
place a list of dicts of user-defined-in-python objects, each of which
holds a reference to a user-defined-via-the-C-api object. What happens next?
>
> You've hit upon exactly the trickiness involved and why I'm thinking the
best approach initially is to only allow *strictly* immutable objects to
pass between interpreters. Admittedly, my description of channels is very
vague.:) There are a number of possibilities with them that I'm still
exploring (CSP has particular opinions...), but immutability is a
characteristic that may provide the simplest *initial* approach. Going
that route shouldn't preclude adding some sort of support for mutable
objects later.
There aren't really many options for mutable objects, right? If you want
shared nothing semantics, then transmitting a mutable object either needs
to make a copy, or else be a real transfer, where the sender no longer has
it (cf. Rust).
I guess for the latter you'd need some new syntax for send-and-del, that
requires the object to be self contained (all mutable objects reachable
from it are only referenced by each other) and have only one reference in
the sending process (which is the one being sent and then destroyed).
> Keep in mind that by "immutability" I'm talking about *really* immutable,
perhaps going so far as treating the full memory space associated with an
object as frozen. For instance, we'd have to ensure that "immutable"
Python objects like strings, ints, and tuples do not change (i.e. via the C
API).
This seems like a red herring to me. It's already the case that you can't
legally use the c api to mutate tuples, ints, for any object that's ever
been, say, passed to a function. So for these objects, the subinterpreter
setup doesn't actually add any new constraints on user code.
C code is always going to be *able* to break memory safety so long as
you're using shared-memory threading at the c level to implement this
stuff. We just need to make it easy not to.
Refcnts and garbage collection are another matter, of course.
-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150620/c2b41fe0/attachment-0001.html>
------------------------------
Message: 2
Date: Sun, 21 Jun 2015 16:31:33 +1000
From: Nick Coghlan <ncoghlan at gmail.com>
To: Nathaniel Smith <njs at pobox.com>
Cc: Eric Snow <ericsnowcurrently at gmail.com>, python-ideas
<python-ideas at python.org>
Subject: Re: [Python-ideas] solving multi-core Python
Message-ID:
<CADiSq7cv538UBK9BE3e8eAakFB=njwHB-qnMu3m=qzLADzpsOg at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
On 21 June 2015 at 15:25, Nathaniel Smith <njs at pobox.com> wrote:
> On Jun 20, 2015 3:54 PM, "Eric Snow" <ericsnowcurrently at gmail.com> wrote:
>>
>>
>> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
>> >
>> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently at gmail.com>
>> > wrote:
>> > >
>> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters,
>> > > exposing them in Python, and adding a mechanism to safely share
>> > > objects between them.
>> >
>> > This all sounds really cool if you can pull it off, and shared-nothing
>> > threads do seem like the least impossible model to pull off.
>>
>> Agreed.
>>
>> > But "least impossible" and "possible" are different :-). From your email
>> > I can't tell whether this plan is viable while preserving backcompat and
>> > memory safety.
>>
>> I agree that those issues must be clearly solved in the proposal before it
>> can be approved. I'm confident the approach I'm pursuing will afford us the
>> necessary guarantees. I'll address those specific points directly when I
>> can sit down and organize my thoughts.
>
> I'd love to see just a hand wavy, verbal proof-of-concept walking through
> how this might work in some simple but realistic case. To me a single
> compelling example could make this proposal feel much more concrete and
> achievable.
I was one of the folks pushing Eric in this direction, and that's
because it's a possibility that was conceived of a few years back, but
never tried due to lack of time (and inclination for those of us that
are using Python primarily as an orchestration tool and hence spend
most of our time on IO bound problems rather than CPU bound ones):
http://www.curiousefficiency.org/posts/2012/07/volunteer-supported-free-threaded-cross.html
As mentioned there, I've at least spent some time with Graham
Dumpleton over the past few years figuring out (and occasionally
trying to address) some of the limitations of mod_wsgi's existing
subinterpreter based WSGI app separation:
https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading#Python_Sub_Interpreters
The fact that mod_wsgi can run most Python web applications in a
subinterpreter quite happily means we already know the core mechanism
works fine, and there don't appear to be any insurmountable technical
hurdles between the status quo and getting to a point where we can
either switch the GIL to a read/write lock where a write lock is only
needed for inter-interpreter communications, or else find a way for
subinterpreters to release the GIL entirely by restricting them
appropriately.
For inter-interpreter communication, the worst case scenario is having
to rely on a memcpy based message passing system (which would still be
faster than multiprocessing's serialisation + IPC overhead), but there
don't appear to be any insurmountable barriers to setting up an object
ownership based system instead (code that accesses PyObject_HEAD
fields directly rather than through the relevant macros and functions
seems to be the most likely culprit for breaking, but I think "don't
do that" is a reasonable answer there).
There's plenty of prior art here (including a system I once wrote in C
myself atop TI's DSP/BIOS MBX and TSK APIs), so I'm comfortable with
Eric's "simple matter of engineering" characterisation of the problem
space.
The main reason that subinterpreters have never had a Python API
before is that they have enough rough edges that having to write a
custom C extension module to access the API is the least of your
problems if you decide you need them. At the same time, not having a
Python API not only makes them much harder to test, which means
various aspects of their operation are more likely to be broken, but
also makes them inherently CPython specific.
Eric's proposal essentially amounts to three things:
1. Filing off enough of the rough edges of the subinterpreter support
that we're comfortable giving them a public Python level API that
other interpreter implementations can reasonably support
2. Providing the primitives needed for safe and efficient message
passing between subinterpreters
3. Allowing subinterpreters to truly execute in parallel on multicore machines
All 3 of those are useful enhancements in their own right, which
offers the prospect of being able to make incremental progress towards
the ultimate goal of native Python level support for distributing
across multiple cores within a single process.
Cheers,
Nick.
--
Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
------------------------------
Message: 3
Date: Sun, 21 Jun 2015 01:41:21 -0500
From: Wes Turner <wes.turner at gmail.com>
To: Eric Snow <ericsnowcurrently at gmail.com>
Cc: Nathaniel Smith <njs at pobox.com>, Python-Ideas
<python-ideas at python.org>
Subject: Re: [Python-ideas] solving multi-core Python
Message-ID:
<CACfEFw_1JVpUmZwFVkye-fbEsfH_NXVSo6WDMi1azhnXdY6PcA at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Exciting!
*
http://zero-buffer.readthedocs.org/en/latest/api-reference/#zero_buffer.BufferView
* https://www.google.com/search?q=python+channels
* https://docs.python.org/2/library/asyncore.html#module-asyncore
* https://chan.readthedocs.org/en/latest/
* https://goless.readthedocs.org/en/latest/
* other approaches to the problem (with great APIs):
* http://celery.readthedocs.org/en/latest/userguide/canvas.html#chords
* http://discodb.readthedocs.org/en/latest/
On Jun 20, 2015 5:55 PM, "Eric Snow" <ericsnowcurrently at gmail.com> wrote:
>
> On Jun 20, 2015 4:08 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
> >
> > On Jun 20, 2015 2:42 PM, "Eric Snow" <ericsnowcurrently at gmail.com>
> wrote:
> > >
> > > tl;dr Let's exploit multiple cores by fixing up subinterpreters,
> > > exposing them in Python, and adding a mechanism to safely share
> > > objects between them.
> >
> > This all sounds really cool if you can pull it off, and shared-nothing
> threads do seem like the least impossible model to pull off.
>
> Agreed.
>
> > But "least impossible" and "possible" are different :-). From your email
> I can't tell whether this plan is viable while preserving backcompat and
> memory safety.
>
> I agree that those issues must be clearly solved in the proposal before it
> can be approved. I'm confident the approach I'm pursuing will afford us
> the necessary guarantees. I'll address those specific points directly when
> I can sit down and organize my thoughts.
>
> >
> > Suppose I have a queue between two subinterpreters, and on this queue I
> place a list of dicts of user-defined-in-python objects, each of which
> holds a reference to a user-defined-via-the-C-api object. What happens next?
>
> You've hit upon exactly the trickiness involved and why I'm thinking the
> best approach initially is to only allow *strictly* immutable objects to
> pass between interpreters. Admittedly, my description of channels is very
> vague.:) There are a number of possibilities with them that I'm still
> exploring (CSP has particular opinions...), but immutability is a
> characteristic that may provide the simplest *initial* approach. Going
> that route shouldn't preclude adding some sort of support for mutable
> objects later.
>
> Keep in mind that by "immutability" I'm talking about *really* immutable,
> perhaps going so far as treating the full memory space associated with an
> object as frozen. For instance, we'd have to ensure that "immutable"
> Python objects like strings, ints, and tuples do not change (i.e. via the C
> API). The contents of involved tuples/containers would have to be likewise
> immutable. Even changing refcounts could be too much, hence the idea of
> moving refcounts out to a separate table.
>
> This level of immutability would be something new to Python. We'll see if
> it's necessary. If it isn't too much work it might be a good idea
> regardless of the multi-core proposal.
>
> Also note that Barry has a (rejected) PEP from a number of years ago about
> freezing objects... That idea is likely out of scope as relates to my
> proposal, but it certainly factors in the problem space.
>
> -eric
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150621/e4aecde0/attachment.html>
------------------------------
Subject: Digest Footer
_______________________________________________
Python-ideas mailing list
Python-ideas at python.org
https://mail.python.org/mailman/listinfo/python-ideas
------------------------------
End of Python-ideas Digest, Vol 103, Issue 100
**********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150621/df712e90/attachment-0001.html>
More information about the Python-ideas
mailing list