GIL in the new glossary

[John J. Lee]

lcaamano at mindspring.com (Luis P Caamano) writes:

> jjl at pobox.com (John J. Lee) wrote in message news:<87oewzyv81.fsf at pobox.com>...
> > "Luis P Caamano" <lpc at racemi.com> writes:
> > [...]
> > > application in any other language, ever.  As it is right
> > > now, if I need to run a server app on a big MP machine
> > > and I need to scale accordingly, I have two choices:
> > > 
> > > a) play with IPC again (as in if we were back in
> > >    in the 80s before pthreads) to have separate interpreters
> > >    per processor or 
> > [...]
> > 
> > Sounds like a great idea to me.  :-)
> 
> Not if you need several hundred threads in your app.

Well, I don't think one starts from that point.  Your statement is
more an implementation decision than a requirement.


> > Did you get the earlier references to POSH?
> 
> No, I didn't.
> 
> > 
> > http://poshmodule.sourceforge.net/posh/html/
> > 
> 
> I just read the docs.  It's pretty cool.  Thanks for
> the pointer as it is something we could definitely use
> right away.

One other thing you should look at is Stackless Python.  Actually,
that's probably three things: Stackless past, Stackless present, and
Stackless future.  It has changed drastically recently, mostly because
Guido dictated that it was never going to merge with mainstream Python
in its old form (it wasn't portable to the Java platform, for Jython).


> However, although this is a great solution for some
> cases, it's not a solution for all cases.
> There's a table at the end of the doc that shows
> a performance comparison between using python
> threads and multiple processes, clearly showing how
> much faster the posh implementation is when using
> more than one processor.  It would be really nice to
> see how it would compare to the same app written in
> C running on an OS with an MxN pthread library.  :-)
> I'd expect the results to prove my point.

Which point is that?  How would it be instructive to compare a
multithreaded C app with a POSH Python app?

[...]
> There are many applications where threads are definitely
> better than multi-processes, IPC, and shared memory.  In
> some cases, threads are the only way, e.g., if the app
> requires thousands of threads.  Or do you think threads
> came to be just because they were cool?  

Came to be?  No.  Came to be as widely used as they are now?  Yes.


> I agree that using mp, IPC, and shared memory is
> a way to get around the GIL, but just because python
> doesn't have free threading doesn't mean threading
> is bad.

I'm sure everyone will agree with you that "is threading bad" is too
simple-minded a question!


> Having to use shared memory and multiple
> processes is a step backwards and one BIG reason,
> if not the biggest, that holds some people back from
> using python for large scale apps.  It is a tough
> problem to solve, I agree and I understand why it
> hasn't been solved and why it will take a while to
> tackle it.  However, it will have to be solved eventually
> and any argument saying it doesn't need a solution is
> just like trying to cover then sun with a thumb.

The problem we're faced with is simply that getting rid of the GIL
entails costs that have to be paid by everyone.  So, it's just not
going to happen until somebody forks Python and puts in the work.  If
you *really* find you need massive multi-threading taking advantage of
multiprocessors, you either do the fork, or you use Java (or
whatever).  Nobody here (that I've noticed...) says "threads are bad"
in a sour-grapes kind of way, they just point out that 1. the GIL is
arguably not a bug to be fixed, but a sensible trade-off and
2. threads are rarely the only, or even the best, option.

I agree with what others have said: the natural fit of threads and
Python is in programs that are simpler when written that way, not
primarily as an optimisation technique (with the exception of
applications like making GUIs responsive, which could be regarded as
an optimisation, I suppose -- but the GIL isn't a problem there).  And
it's worth repeating yet again that the GIL *is* released in many C
extensions, which (so I'm told) does allow some practical, useful,
optimisations.


> The GIL is a convenience that hurts only a few.  When
> that few grows to a critical number, the GIL will
> disappear.  Until then though ...

I'm not holding my breath <wink>.


John