Status of Python threading support (GIL removal)?

[Jure Erznožnik]

On Jun 19, 11:45 pm, OdarR <Olivier.Da... at gmail.com> wrote:
> On 19 juin, 21:05, Christian Heimes <li... at cheimes.de> wrote:
>
> > I've seen a single Python process using the full capacity of up to 8
> > CPUs. The application is making heavy use of lxml for large XSL
> > transformations, a database adapter and my own image processing library
> > based upon FreeImage.
>
> interesting...
>
> > Of course both lxml and my library are written with the GIL in mind.
> > They release the GIL around every call to C libraries that don't touch
> > Python objects. PIL releases the lock around ops as well (although it
> > took me a while to figure it out because PIL uses its own API instead of
> > the standard macros). reportlab has some optional C libraries that
> > increase the speed, too. Are you using them?
>
> I don't. Or maybe I did, but I have no clue what to test.
> Do you have a real example, some code snippet to can prove/show
> activity on multiple core ?
> I accept your explanation, but I also like experiencing :)
>
> > By the way threads are evil
> > (http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf) and
> > not *the* answer to concurrency.
>
> I don't see threads as evil from my little experience on the subject,
> but we need them.
> I'm reading what's happening in the java world too, it can be
> interesting.
>
> Olivier

Olivier,
What Christian is saying is that you can write a C/C++ Python plugin,
release the GIL inside it and then process stuff in threads inside the
plugin.
All this is possible if the progammer doesn't use any Python objects
and it's fairly easy to write such a plugin. Any counting example will
do just fine.

The problem with this solution is that you have to write the code in C
which quite defeats the purpose of using an interpreter in the first
place...
Of course, no pure python code will currently utilize multiple cores
(because of GIL).

I do aggree though that threading is important. Regardless of any
studies showing that threads suck, they are here and they offer
relatively simple concurrency. IMHO they should never have been
crippled like this. Even though GIL solves access violations, it's not
the right approach. It simply kills all threading benefits except for
the situation where you work with multiple I/O blocking threads.
That's just about the only situation where this problem is not
apparent.

We're way past single processor single core computers now. An
important product like Python should support these architectures
properly even if only 1% of applications written in it use threading.

But as Guido himself said; I should not complain but instead try to
contribute to solution. That's the hard part, especially since there's
lots of code that actually need the locking.