The Future of Python Threading

[Jean-Paul Calderone]

On Fri, 10 Aug 2007 10:01:51 -0000, "Justin T." <jmtulloss at gmail.com> wrote:
>Hello,
>
>While I don't pretend to be an authority on the subject, a few days of
>research has lead me to believe that a discussion needs to be started
>(or continued) on the state and direction of multi-threading python.
>
> [snip - threading in Python doesn't exploit hardware level parallelism
>  well, we should incorporate stackless and remove the GIL to fix this]
>

I think you have a misunderstanding of what greenlets are.  Greenlets are
essentially a non-preemptive user-space threading mechanism.  They do not
allow hardware level parallelism to be exploited.

In a hypothetical Python interpreter, equivalent to current CPython except
for the removal of the GIL, you could certainly have greenlets running in
different pthreads, and thus take care of hardware parallelism, but you
could do the same without greenlets too.

So Stackless Python is an unrelated matter.

>There has been much discussion on this in the past [2]. Those
>discussions, I feel, were premature. Now that stackless is mature (and
>continuation free!), Py3k is in full swing, and parallel programming
>has been fully realized as THE next big problem for computer science,
>the time is ripe for discussing how we will approach multi-threading
>in the future.

Many of the discussions rehash the same issues as previous ones.  Many
of them are started based on false assumptions or are discussions between
people who don't have a firm grasp of the relevant issues.

I don't intend to suggest that no improvements can be made in this area of
Python interpreter development, but it is a complex issue and cheerleading
will only advance the cause so far.  At some point, someone needs to write
some code.  Stackless is great, but it's not the code that will solve this
problem.

In the mean time, you might consider some multi-process solutions.  There
are a number of tools for getting concurrency like that.

Jean-Paul