2.6, 3.0, and truly independent intepreters

[Rhamphoryncus]

On Oct 28, 9:30 am, "Andy O'Meara" <and... at gmail.com> wrote:
> On Oct 25, 9:46 am, "M.-A. Lemburg" <m... at egenix.com> wrote:
>
> > These discussion pop up every year or so and I think that most of them
> > are not really all that necessary, since the GIL isn't all that bad.
>
> Thing is, if the topic keeps coming up, then that may be an indicator
> that change is truly needed.  Someone much wiser than me once shared
> that a measure of the usefulness and quality of a package (or API) is
> how easily it can be added to an application--of any flavors--without
> the application needing to change.
>
> So in the rising world of idle cores and worker threads, I do see an
> increasing concern over the GIL.  Although I recognize that the debate
> is lengthy, heated, and has strong arguments on both sides, my reading
> on the issue makes me feel like there's a bias for the pro-GIL side
> because of the volume of design and coding work associated with
> considering various alternatives (such as Glenn's "Py*" concepts).
> And I DO respect and appreciate where the pro-GIL people come from:
> who the heck wants to do all that work and recoding so that a tiny
> percent of developers can benefit?  And my best response is that as
> unfortunate as it is, python needs to be more multi-threaded app-
> friendly if we hope to attract the next generation of app developers
> that want to just drop python into their app (and not have to change
> their app around python).  For example, Lua has that property, as
> evidenced by its rapidly growing presence in commercial software
> (Blizzard uses it heavily, for example).
>
>
>
> > Furthermore, there are lots of ways to tune the CPython VM to make
> > it more or less responsive to thread switches via the various sys.set*()
> > functions in the sys module.
>
> > Most computing or I/O intense C extensions, built-in modules and object
> > implementations already release the GIL for you, so it usually doesn't
> > get in the way all that often.
>
> The main issue I take there is that it's often highly useful for C
> modules to make subsequent calls back into the interpreter. I suppose
> the response to that is to call the GIL before reentry, but it just
> seems to be more code and responsibility in scenarios where it's no
> necessary.  Although that code and protocol may come easy to veteran
> CPython developers, let's not forget that an important goal is to
> attract new developers and companies to the scene, where they get
> their thread-independent code up and running using python without any
> unexpected reengineering.  Again, why are companies choosing Lua over
> Python when it comes to an easy and flexible drop-in interpreter?  And
> please take my points here to be exploratory, and not hostile or
> accusatory, in nature.
>
> Andy

Okay, here's the bottom line:
* This is not about the GIL.  This is about *completely* isolated
interpreters; most of the time when we want to remove the GIL we want
a single interpreter with lots of shared data.
* Your use case, although not common, is not extraordinarily rare
either.  It'd be nice to support.
* If CPython had supported it all along we would continue to maintain
it.
* However, since it's not supported today, it's not worth the time
invested, API incompatibility, and general breakage it would imply.
* Although it's far more work than just solving your problem, if I
were to remove the GIL I'd go all the way and allow shared objects.

So there's really only two options here:
* get a short-term bodge that works, like hacking the 3rd party
library to use your shared-memory allocator.  Should be far less work
than hacking all of CPython.
* invest yourself in solving the *entire* problem (GIL removal with
shared python objects).