[Python-Dev] Free threading

[Guido van Rossum]

> > Nothing stops you from doing that -- the intention is that it's
> > possible. 
> 
> I didn't ask the question properly. You can run multiple interpreters in
> multiple threads but they share a GIL, right? I'm really asking whether
> multiple interpreters can be as independent of each other as any other
> bit of carefuly written re-entrant C code.

The share the GIL to protect the other shared resources.  Since
immutable objects can be shared, there's no way to have completely
independent interpreters.

Anyway, I don't see the point: if you want two 100% independent
interpreters, you can get that easily by forking two separate
processes.  That's what separate processes are *for*...

> > ... Immutable objects (like strings and ints) and all built-in
> > type objects (including extension types) are shared between all
> > interpreters.  Interned strings are shared.  Exceptions are shared
> > (there may actually be a bug related to initializing exceptions that I
> > still have to track down).  Extension modules that have global state
> > may cause additional confusion.  Operating system resources like file
> > descriptors are shared.
> 
> It might be interesting if there were a module that allowed the Python
> programmer to spawn new interpreters on new threads from within Python
> and a way to share primitive data structures between them. My
> understanding is that Perl and Tcl both use this "independent threads"
> model which scales really well on SMP but seems to me to be pretty
> brutal to program for. Nevertheless I can see how it would be useful
> where all you care about is SMP performance.
> 
> http://search.cpan.org/doc/ABERGMAN/threads-shared-0.02/shared.pm

I bet the level of sharing exposed here can be implemented easily on
top of the mmap module.  Essentially, no matter what you do on an SMP
machine, it seems you have to minimize sharing because it slows things
down too much.  Every solution somehow ends up making private data (to
a process/thread) the default and makes you request sharing
explicitly.  In Python you won't be able to share *variables*, but it
should be easy enough to share objects via some memory-based RPC
protocol.

That sounds to me a quicker road to an efficient SMP solution than
trying to get rid of the GIL.

--Guido van Rossum (home page: http://www.python.org/~guido/)