2.6, 3.0, and truly independent intepreters

[Andy O'Meara]

On Oct 24, 9:52 pm, "Martin v. Löwis" <mar... at v.loewis.de> wrote:
> >> A c-level module, on the other hand, can sidestep/release
> >> the GIL at will, and go on it's merry way and process away.
>
> > ...Unless part of the C module execution involves the need do CPU-
> > bound work on another thread through a different python interpreter,
> > right?
>
> Wrong.
>
> > (even if the interpreter is 100% independent, yikes).
>
> Again, wrong.
>
> > For
> > example, have a python C module designed to programmatically generate
> > images (and video frames) in RAM for immediate and subsequent use in
> > animation.  Meanwhile, we'd like to have a pthread with its own
> > interpreter with an instance of this module and have it dequeue jobs
> > as they come in (in fact, there'd be one of these threads for each
> > excess core present on the machine).
>
> I don't understand how this example involves multiple threads. You
> mention a single thread (running the module), and you mention designing
> a  module. Where is the second thread?

Glenn seems to be following me here...  The point is to have any many
threads as the app wants, each in it's own world, running without
restriction (performance wise).  Maybe the app wants to run a thread
for each extra core on the machine.

Perhaps the disconnect here is that when I've been saying "start a
thread", I mean the app starts an OS thread (e.g. pthread) with the
given that any contact with other threads is managed at the app level
(as opposed to starting threads through python).  So, as far as python
knows, there's zero mention or use of threading in any way,
*anywhere*.


> > As far as I can tell, it seems
> > CPython's current state can't CPU bound parallelization in the same
> > address space.
>
> That's not true.
>

Um...  So let's say you have a opaque object ref from the OS that
represents hundreds of megs of data (e.g. memory-resident video).  How
do you get that back to the parent process without serialization and
IPC?  What should really happen is just use the same address space so
just a pointer changes hands.  THAT's why I'm saying that a separate
address space is  generally a deal breaker when you have large or
intricate data sets (ie. when performance matters).

Andy