The Future of Python Threading

[Steve Holden]

Justin T. wrote:
> Hello,
> 
> While I don't pretend to be an authority on the subject, a few days of
> research has lead me to believe that a discussion needs to be started
> (or continued) on the state and direction of multi-threading python.
[...]
> What these seemingly unrelated thoughts come down to is a perfect
> opportunity to become THE next generation language. It is already far
> more advanced than almost every other language out there. By
> integrating stackless into an architecture where tasklets can be
> divided over several parallelizable threads, it will be able to
> capitalize on performance gains that will have people using python
> just for its performance, rather than that being the excuse not to use
> it.
> 
Aah, the path to world domination. You know you don't *have* to use 
Python for *everything*.

> The nice thing is that this requires a fairly doable amount of work.
> First, stackless should be integrated into the core. Then there should
> be an effort to remove the reliance on the GIL for python threading.
> After that, advanced features like moving tasklets amongst threads
> should be explored. I can imagine a world where a single python web
> application is able to redistribute its millions of requests amongst
> thousands of threads without the developer ever being aware that the
> application would eventually scale. An efficient and natively multi-
> threaded implementation of python will be invaluable as cores continue
> to multiply like rabbits.
> 
Be my guest, if it's so simple.

> There has been much discussion on this in the past [2]. Those
> discussions, I feel, were premature. Now that stackless is mature (and
> continuation free!), Py3k is in full swing, and parallel programming
> has been fully realized as THE next big problem for computer science,
> the time is ripe for discussing how we will approach multi-threading
> in the future.
> 
I doubt that a thread on c.l.py is going to change much. It's the 
python-dev and py3k lists where you'll need to take up the cudgels, 
because I can almost guarantee nobody is going to take the GIL out of 
2.6 or 2.7.

> 
> [1] I haven't actually looked at the GIL code. It's possible that it
> creates a bunch of wait queues for each nice level that a python
> thread is running at and just wakes up the higher priority threads
> first, thus maintaining the nice values determined by the scheduler,
> or something. I highly doubt it. I bet every python thread gets an
> equal chance of getting that lock despite whatever patterns the
> scheduler may have noticed.
> 
It's possible that a little imp tosses a coin and decides which thread 
gets to run next. The point of open source is that it's easy to dispel 
ignorance and make your own changes if you are competent to do so. Talk 
is cheap, code is costly. Your bet is worth nothing. Is it even possible 
to run threads of the same process at different priority levels on all 
platforms?

> [2]
> http://groups.google.com/group/comp.lang.python/browse_thread/thread/7d16083cf34a706f/858e64a2a6a5d976?q=stackless%2C+thread+paradigm&lnk=ol&
> http://www.stackless.com/pipermail/stackless/2003-June/000742.html
> ... More that I lost, just search this group.
> 
Anyone who's been around Python for a while is familiar with the issues. 
Have you actually asked Chris Tismer whether he's happy to see Stackless 
go into the core? He was far from certain that would be a good idea at 
PyCon earlier this year. He probably has his reasons ...

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------