The Future of Python Threading

[Jean-Paul Calderone]

On Fri, 10 Aug 2007 16:37:19 -0000, "Justin T." <jmtulloss at gmail.com> wrote:
>On Aug 10, 3:52 am, Jean-Paul Calderone <exar... at divmod.com> wrote:
>> On Fri, 10 Aug 2007 10:01:51 -0000, "Justin T." <jmtull... at gmail.com> wrote:
>> >Hello,
>>
>> >While I don't pretend to be an authority on the subject, a few days of
>> >research has lead me to believe that a discussion needs to be started
>> >(or continued) on the state and direction of multi-threading python.
>>
>> > [snip - threading in Python doesn't exploit hardware level parallelism
>> >  well, we should incorporate stackless and remove the GIL to fix this]
>>
>> I think you have a misunderstanding of what greenlets are.  Greenlets are
>> essentially a non-preemptive user-space threading mechanism.  They do not
>> allow hardware level parallelism to be exploited.
>
>I'm not an expert, but I understand that much. What greenlets do is
>force the programmer to think about concurrent programming. It doesn't
>force them to think about real threads, which is good, because a
>computer should take care of that for you.Greenlets are nice because
>they can run concurrently, but they don't have to. This means you can
>safely divide them up among many threads. You could not safely do this
>with just any old python program.

There may be something to this.  On the other hand, there's no _guarantee_
that code written with greenlets will work with pre-emptive threading instead
of cooperative threading.  There might be a tendency on the part of developers
to try to write code which will work with pre-emptive threading, but it's just
that - a mild pressure towards a particular behavior.  That's not sufficient
to successfully write correct software (where "correct" in this context means
"works when used with pre-emptive threads", of course).

One also needs to consider the tasks necessary to really get this integration
done.  It won't change very much if you just add greenlets to the standard
library.  For there to be real consequences for real programmers, you'd
probably want to replace all of the modules which do I/O (and maybe some
that do computationally intensive things) with versions implemented using
greenlets.  Otherwise you end up with a pretty hard barrier between greenlets
and all existing software that will probably prevent most people from changing
how they program.

Then you have to worry about the other issues greenlets introduce, like
invisible context switches, which can make your code which _doesn't_ use
pre-emptive threading broken.

All in all, it seems like a wash to me.  There probably isn't sufficient
evidence to answer the question definitively either way, though.  And trying
to make it work is certainly one way to come up with such evidence. :)

Jean-Paul