2.6, 3.0, and truly independent intepreters

[Andy O'Meara]

>
> The Global Interpreter Lock is fundamentally designed to make the
> interpreter easier to maintain and safer: Developers do not need to
> worry about other code stepping on their namespace. This makes things
> thread-safe, inasmuch as having multiple PThreads within the same
> interpreter space modifying global state and variable at once is,
> well, bad. A c-level module, on the other hand, can sidestep/release
> the GIL at will, and go on it's merry way and process away.

...Unless part of the C module execution involves the need do CPU-
bound work on another thread through a different python interpreter,
right? (even if the interpreter is 100% independent, yikes).  For
example, have a python C module designed to programmatically generate
images (and video frames) in RAM for immediate and subsequent use in
animation.  Meanwhile, we'd like to have a pthread with its own
interpreter with an instance of this module and have it dequeue jobs
as they come in (in fact, there'd be one of these threads for each
excess core present on the machine).  As far as I can tell, it seems
CPython's current state can't CPU bound parallelization in the same
address space (basically, it seems that we're talking about the
"embarrassingly parallel" scenario raised in that paper).  Why does it
have to be in same address space?  Convenience and simplicity--the
same reasons that most APIs let you hang yourself if the app does dumb
things with threads.  Also, when the data sets that you need to send
to and from each process is large, using the same address space makes
more and more sense.


> So, just to clarify - Andy, do you want one interpreter, $N threads
> (e.g. PThreads) or the ability to fork multiple "heavyweight"
> processes?

Sorry if I haven't been clear, but we're talking the app starting a
pthread, making a fresh/clean/independent interpreter, and then being
responsible for its safety at the highest level (with the payoff of
each of these threads executing without hinderance).  No different
than if you used most APIs out there where step 1 is always to make
and init a context object and the final step is always to destroy/take-
down that context object.

I'm a lousy writer sometimes, but I feel bad if you took the time to
describe threads vs processes.  The only reason I raised IPC with my
"messaging isn't very attractive" comment was to respond to Glenn
Linderman's points regarding tradeoffs of shared memory vs no.


Andy