[Baypiggies] Guido's Blog: It isn't Easy to Remove the

[Warren DeLano]

Granted, much existing C code is not thread-safe, but that begs the
question, since with concurrent interpreters, one could *only* expect to
interact with modern thread-safe extensions that encapsulate state into
object handles.

However, Guido's comments do help shape a possible solution:  there
would still have to be a primary interpreter responsible for interacting
with thread-unsafe "legacy" extensions, and all such extensions would
have be strictly unavailable to secondary (concurrent) interpreters.  

Indeed, this really isn't all that different than the status quo, where
sub-interpreters are not able to perform certain tasks.  Concurrent
sub-interpreters would have to be even more restricted in what they
could do.

But the real issue is the thread-unsafe nature of C/Python itself, and
what might or might not be done about it.  So...

Assuming that TLS is adopted as the means of identifying an active
concurrent interpreter bound to a native thread, does it strike anyone
as potentially feasible to conditionally intercept and redirect all
access to existing C/Python interpreter global state over to
instance-specific equivalents (for concurrent interpreters *only*)?  

Regardless, outside of a concurrent interpreter TLS-bound thread, all
existing code paths would remain unchanged, and thus, there would be no
need to rewrite "all existing C extensions" as Guido suggests.  

Nevertheless, thread-safe C extensions would need to be modified to
self-identify as such in order to be accessible from concurrent
interpreters, and so the capabilities of concurrent interpreters would
start out small and only grow slowly as thread-safety permits.

Thoughts?  Opinions?

> -----Original Message-----
> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On 
> Behalf Of Guido van Rossum
> Sent: Friday, September 14, 2007 5:34 PM
> To: Warren DeLano
> Cc: baypiggies at python.org
> Subject: Re: [Baypiggies] Guido's Blog: It isn't Easy to Remove the
> 
> Well, it would require rewriting all existing C extensions, 
> except for the ones that have zero mutable global data. 
> Worse, some C extensions wrap around C libraries that have 
> global mutable data -- for those there is no fix. 
> Realistically, most C code uses global mutable state, and 
> realistically, most Python programs use C extensions.
> 
> On 9/14/07, Warren DeLano <warren at delsci.com> wrote:
> > Hmm...
> >
> > Call me crazy, but it seems to me that the problem isn't so 
> much the 
> > GIL, but rather, it is our inability to simultaneously run multiple 
> > Python interpreters within a single process, with each interpreter 
> > playing nicely with native C extensions and libraries.
> >
> > In other words, it is not the interpreter lock that is so limiting, 
> > rather, it is the fact that the interpreter acts like a global 
> > singleton with respect to the C/API and other native code 
> extensions.
> >
> > To restate this yet another way:  In my perfect 
> Python-embedded world, 
> > the Python interpreter would itself become an object you can 
> > instantiate many times, and the bulk of the C/API would then become 
> > methods called on that object.  Each interpreter instance 
> would still 
> > have its own GIL
> > -- you wouldn't need to deal with microscopic locking, and you 
> > wouldn't lose any of Python's existing performance.
> >
> > Of course, most C/API methods would however need to take 
> "self" as an
> > argument:
> >
> > PyObject *PySome_Action(PythonInterpreterInstance *pythonSelf, ...);
> >
> > However, we could use TLS (thread-local-state) to 
> dynamically bind a 
> > specific interpreter instance to a specific thread.
> >
> > PythonInterpreterInstance *P = ...some extant interpeter 
> instance...;
> >
> > if(PyInstanceViaTLS_BindThread(P)) {
> >
> >   PyGILState_STATE gstate;
> >   gstate = PyGILState_Ensure();
> >
> >   /* global calls within this thread are now directed at a specific 
> > interpreter */
> >
> >   PySome_Action (...);
> >
> >   /* meanwhile, other threads could simultaneously be 
> messaging other 
> > interpreters running in parallel within the same process */
> >
> >   PyGILState_Release(gstate);
> >
> >   PyInstanceViaTLS_UnbindThread();
> > }
> >
> > That simple workaround could preserve compatibility will existing 
> > C/API code, while still freeing us from this crushing 
> global singularity...
> >
> > So is this just crazy talk?  Or might something like this 
> actually be 
> > worth a try?
> >
> > Cheers,
> > Warren
> > warren at delsci.com
> >
> >
> >
> > _______________________________________________
> > Baypiggies mailing list
> > Baypiggies at python.org
> > To change your subscription options or unsubscribe:
> > http://mail.python.org/mailman/listinfo/baypiggies
> >
> 
> 
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> 
> 
> 
>