The Future of Python Threading

[Bryan Olson]

Justin T. wrote:
> True, but Python seems to be the *best* place to tackle this problem,
> at least to me. It has a large pool of developers, a large standard
> library, it's evolving, and it's a language I like :). Languages that
> seamlessly support multi-threaded programming are coming, as are
> extensions that make it easier on every existent platform. Python has
> the opportunity to lead that change.

I have to disagree. A dynamic scripting language, even a great
one such as Python, is not the vehicle to lead an advance of
pervasive threading. The features that draw people to Python
do not play nicely with optimizing multi-core efficiency; they
didn't play all that well with single-core efficiency either.
Python is about making good use of our time, not our machines'
time.

Finer-grain locking sounds good, but realize how many items
need concurrency control. Python offers excellent facilities
for abstracting away complexity, so we programmers do not
mentally track all the possible object interactions at once.
In a real application, objects are more intertwingled than
we realize. Whatever mistakes a Python programmer makes,
the interpreter must protect its own data structures
against all possible race errors.

Run-time flexibility is a key Python feature, and it
necessarily implies that most bindings are subject to
change. Looking at a Python function, we tend to focus on
the named data objects, and forget how many names Python
is looking up at run time. How often do we take and release
locks on all the name-spaces that might effect execution?

The GIL both sucks and rocks. Obviously cores are becoming
plentiful and the GIL limits how we can exploit them. On
the other hand, correctness must dominate efficiency. We
lack exact reasoning on what we must lock; with the GIL,
we err on the side of caution and correctness. Instead of
trying to figure out what we need to lock, we default to
locking everything, and try to figure out what we can
safely release.

[Steve Holden:]
>> Be my guest, if it's so simple.
>>
> I knew somebody was going to say that! I'm pretty busy, but I'll see
> if I can find some time to look into it.

If you want to lead fixing Python's threading, consider
first delivering a not-so-grand but definite
improvement. Have you seen how the 'threading' module
in Python's standard library implements timeouts?


-- 
--Bryan