Parallelization on muli-CPU hardware?

[Nicolas Lehuen]

aleaxit at yahoo.com (Alex Martelli) wrote in message news:<1glbyrn.4jt08j144wg1dN%aleaxit at yahoo.com>...
> > Guido did not impose the GIL lightly.  This is a hard problem.
> 
> Sure.  I wonder what (e.g.) IronPython or Ruby do about it -- never
> studied the internals of either, yet.
> 
> 
> Alex

I've just asked the question on the IronPython mailing list - I'll
post the answer here. I've also asked about Jython since Jim Hugunin
worked on it.

Just as a note, coming from a Java background (I switched to Python
two years ago and nearly never looked back - my current job allows
this, I'm an happy guy), I must say that there is a certain deficit of
thread-awareness in the Python community at large (not offense meant).

The stdlib is actually quite poor on the thread-safe side, and whereas
I see people rush to implement Doug Lea's concurrent package
[http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html]
on new languages like D, it seems that it is not a priority to the
Python community (maybe a point in the 'develop the stdlib vs develop
the interpreter and language' debate). A symptom of this is that
must-have classes like threading.RLock() are still implemented in
Python ! This should definitely be written in C for the sake of
performance.

I ran into many multi-threading problem in mod_python as well, and the
problem was the same ; people expected mod_python to run in a
multi-process context, not a multi-threaded context (I guess this is
due to a Linux-centered mindset, forgetting about BSD, MacosX or Win32
OSes). When I asked questions and pointed problem, the answer was 'Duh
- use Linux with the forking MPM, not Windows with the threading MPM'.
Well, this is not going to take us anywhere, especially with all the
multicore CPUs coming.

Java, C#, D and many other languages prove that threading is not
necessarily a complicated issue. I can be tricky, granted, but where
would be the fun it if it was not ? In a former life, I have
implemented an application server and had to write a big bunch of
code, always with thread awareness in mind. I guarantee you that
provided your language gives you the proper support (I was using
Java), it's a real treat :).

With all respect due to the Founding Fathers (and most of all Guido),
there are quite a few languages out there that do not require a GIL...
I understand that historical implementation issues meant that a GIL
was required (we talk about a language whose implementation began in
the early nineties), but maybe it is time to reconsider this in Python
3000 ? And while I'm on this iconoclast binge, why not forget about
reference counting and just switch to plain garbage collecting
everywhere, thus clearing a lot of the mess we have to deal with when
writing C extensions ? Plus, a full garbage collection model can
actually help with threading issues, and vice versa.

I know a lot of people who fleed Python for two reasons. The first
reason is a bad one, it's the indentation-as-syntax reason. My answer
is 'just try it'. The second reason is concerns about performance and
scalability, especially from those who heard about the GIL. Well, all
I can answer is 'That will improve over time, look at Psyco, for
instance'. But as far as the GIL is concerned, well... I'm as worried
as them. Please, please, could we change this ?

Regards,

Nicolas Lehuen