Exploiting Dual Core's with Py_NewInterpreter's separated GIL ?

[grahamd at dscpl.com.au]

Shane Hathaway wrote:
> robert wrote:
> > I'd like to use multiple CPU cores for selected time consuming Python
> > computations (incl. numpy/scipy) in a frictionless manner.
> >
> > Interprocess communication is tedious and out of question, so I
> > thought about simply using a more Python interpreter instances
> > (Py_NewInterpreter) with extra GIL in the same process. I expect to
> > be able to directly push around Python Object-Trees between the 2 (or
> > more) interpreters by doing some careful locking.
> >
> > Any hope to come through? If possible, what are the main dangers? Is
> > there an example / guideline around for that task? - using ctypes or
> > so.
> >
> > Or is there even a ready made Python module which makes it easy to
> > setup and deal with extra Interpreter instances? If not, would it be
> > an idea to create such thing in the Python std libs to make Python
> > multi-processor-ready. I guess Python will always have a GIL -
> > otherwise it would loose lots of comfort in threaded programming
>
> I'd like to mention mod_python, which creates multiple interpreters
> inside Apache.  It does this transparently and works well.  There is no
> apparent lock contention between the interpreters, because no Python
> objects are shared.

In relation to mod_python, your statement isn't strictly true. One can
create scenarios in mod_python where a Python object created in the
context of one interpreter is then used in another interpreter at a
later point. Generally this isn't an issue because it is part of a
recursive sequence of processing and thus while the second interpreter
is doing stuff with the object the first is stalled waiting for the
second to return. It wouldn't be totally inconceivable though for a
user to create threads which may operate on the object at the same time
in the first interpreter as the object is being used in the second
interpreter.

The situation gets even more complicated when you start to use third
party modules which may be used from multiple interpreters and a
multithreaded Apache MPM is used and where the third party module
performs callbacks into interpreters with some common object.

Thus, mod_python may not be able to benefit from each Python
interpreter instance having its own GIL if that was your point in
referencing mod_python.

Graham